SONARJAVA-7229 Add telemetry for Java semantic degradation and classpath configuration

docs/semantic-model-telemetry.md

@@ -0,0 +1,43 @@
+# Java semantic model telemetry
+
+These counters describe observed semantic degradation and possible configuration causes. They do not classify a project as misconfigured automatically. All counters aggregate across modules and contain no source paths, type names, or diagnostic messages.
+
+## Semantic observations
+
+The prefix `java.analysis.<scope>.semantic.` applies separately to `main`, `test`, and `generated` sources.
+
+| Suffix | Meaning |
+| --- | --- |
+| `files_evaluated` | Files for which parsing and AST conversion returned a semantic model and its diagnostics were examined. |
+| `files_with_resolution_errors` | Evaluated files with at least one `UNDEFINED_TYPE` diagnostic. |
+| `resolution_error_count` | Sum of `UNDEFINED_TYPE` diagnostics, deduplicated by message within each file. Preview-feature diagnostics are excluded. |
+| `modules_evaluated` | Modules with at least one evaluated file in this scope. |
+| `modules_with_resolution_errors` | Modules with at least one affected file in this scope. |
+
+Observations are recorded before rule execution, so a later visitor failure does not erase the semantic observation. The existing `success.type_error_count` remains unchanged, including its preview-feature diagnostics and restriction to successful analysis.
+
+The affected-file ratio is `files_with_resolution_errors / files_evaluated`. An absent or zero denominator means unobserved. Zero resolution errors means no degradation was detected by these diagnostics; it does not prove that every binding resolved correctly. No explicit model-absence counter is emitted because normal Java parsing constructs a semantic model even with an incomplete classpath.
+
+## Analysis coverage
+
+Under `java.analysis.<scope>.`, the counters `success.file_count`, `parse_errors.file_count`, and `exceptions.file_count` count file processing outcomes, alongside the existing size and time measurements. `cached_file_count` applies only to `main` and `test`: it counts files successfully processed without parsing through server-side caching and is emitted when that optimization is attempted. Generated sources always go through parsing and do not emit this counter. Cached files have no new semantic observation. A parse failure has no semantic observation either. An exception may occur before or after semantic observation, so these counters must not be added to `files_evaluated` as disjoint populations.
+
+## Configuration signals
+
+The prefix `java.classpath.<scope>.` applies separately to `main` and `test`. Only modules containing Java sources in the corresponding scope contribute.
+
+| Suffix | Meaning |
+| --- | --- |
+| `module_count` | Modules whose classpath configuration was examined. |
+| `modules_missing_binaries` | Modules without the corresponding binaries property. |
+| `modules_without_binary_entries` | Modules whose binaries property resolves to no entries. |
+| `modules_missing_libraries` | Modules without the corresponding libraries property. |
+| `modules_without_library_entries` | Modules whose libraries property resolves to no entries, excluding JDK entries. |
+| `modules_with_unmatched_entries` | Modules with a configured binaries or libraries pattern that produces the existing unmatched-entry warning, including partial classpaths with other valid entries. |
+| `modules_with_invalid_jdk` | Modules whose configured `sonar.java.jdkHome` path is missing or is not a directory, causing fallback to the runtime JDK. |
+
+These signals are collected for AutoScan too; existing warning suppression is unchanged. Missing libraries may be legitimate for JDK-only code. A resolved directory is an entry even if empty; these metrics do not inspect bytecode contents or establish freshness. Main and test configuration counters overlap when both scopes exist, so their module counts must not be summed to obtain distinct project modules.
+
+Use affected-file and affected-module ratios to measure degradation, then correlate with configuration signals and existing scanner, AutoScan, language-version, and analyzer-version context. Project aggregates cannot prove that a configuration problem and a resolution error occurred in the same module. Track distinct projects using their latest analysis and distinguish full observations from partial or cache-only observations.
+
+Invalid classpath entries that abort analysis, binding-level completeness, diagnostic categories beyond the existing parser diagnostics, and telemetry for failed scanner executions are outside this first iteration. The existing project-end sensor submits these counters on completed analyses.

its/plugin/tests/src/test/java/com/sonar/it/java/JspTest.java

@@ -94,15 +94,47 @@ public void should_transpile_jsp() throws Exception {
     // size of the generated files varies depending on the environment line endings
     assertThat(extractTelemetryLogs(buildResult))
       .matches(patternWithLiteralDot("""
+        Telemetry java.analysis.generated.semantic.files_evaluated: 4
+        Telemetry java.analysis.generated.semantic.files_with_resolution_errors: 0
+        Telemetry java.analysis.generated.semantic.modules_evaluated: 1
+        Telemetry java.analysis.generated.semantic.modules_with_resolution_errors: 0
+        Telemetry java.analysis.generated.semantic.resolution_error_count: 0
+        Telemetry java.analysis.generated.success.file_count: 4
         Telemetry java.analysis.generated.success.size_chars: \\d{5}
         Telemetry java.analysis.generated.success.time_ms: \\d+
         Telemetry java.analysis.generated.success.type_error_count: 0
+        Telemetry java.analysis.main.semantic.files_evaluated: 1
+        Telemetry java.analysis.main.semantic.files_with_resolution_errors: 0
+        Telemetry java.analysis.main.semantic.modules_evaluated: 1
+        Telemetry java.analysis.main.semantic.modules_with_resolution_errors: 0
+        Telemetry java.analysis.main.semantic.resolution_error_count: 0
+        Telemetry java.analysis.main.success.file_count: 1
         Telemetry java.analysis.main.success.size_chars: 969
         Telemetry java.analysis.main.success.time_ms: \\d+
         Telemetry java.analysis.main.success.type_error_count: 0
+        Telemetry java.analysis.test.semantic.files_evaluated: 1
+        Telemetry java.analysis.test.semantic.files_with_resolution_errors: 0
+        Telemetry java.analysis.test.semantic.modules_evaluated: 1
+        Telemetry java.analysis.test.semantic.modules_with_resolution_errors: 0
+        Telemetry java.analysis.test.semantic.resolution_error_count: 0
+        Telemetry java.analysis.test.success.file_count: 1
         Telemetry java.analysis.test.success.size_chars: 20
         Telemetry java.analysis.test.success.time_ms: \\d+
         Telemetry java.analysis.test.success.type_error_count: 0
+        Telemetry java.classpath.main.module_count: 1
+        Telemetry java.classpath.main.modules_missing_binaries: 0
+        Telemetry java.classpath.main.modules_missing_libraries: 0
+        Telemetry java.classpath.main.modules_with_invalid_jdk: 0
+        Telemetry java.classpath.main.modules_with_unmatched_entries: 0
+        Telemetry java.classpath.main.modules_without_binary_entries: 0
+        Telemetry java.classpath.main.modules_without_library_entries: 0
+        Telemetry java.classpath.test.module_count: 1
+        Telemetry java.classpath.test.modules_missing_binaries: 0
+        Telemetry java.classpath.test.modules_missing_libraries: 0
+        Telemetry java.classpath.test.modules_with_invalid_jdk: 0
+        Telemetry java.classpath.test.modules_with_unmatched_entries: 0
+        Telemetry java.classpath.test.modules_without_binary_entries: 0
+        Telemetry java.classpath.test.modules_without_library_entries: 0
         Telemetry java.dependency.lombok: absent
         Telemetry java.dependency.spring-boot: absent
         Telemetry java.dependency.spring-web: absent

its/plugin/tests/src/test/java/com/sonar/it/java/suite/JavaTutorialTest.java

@@ -73,9 +73,22 @@ private void executeAndAssertBuild(MavenBuild build, String projectKey) {
 
     assertThat(extractTelemetryLogs(buildResult))
       .matches(patternWithLiteralDot("""
+        Telemetry java.analysis.main.semantic.files_evaluated: 8
+        Telemetry java.analysis.main.semantic.files_with_resolution_errors: 0
+        Telemetry java.analysis.main.semantic.modules_evaluated: 1
+        Telemetry java.analysis.main.semantic.modules_with_resolution_errors: 0
+        Telemetry java.analysis.main.semantic.resolution_error_count: 0
+        Telemetry java.analysis.main.success.file_count: 8
         Telemetry java.analysis.main.success.size_chars: \\d{4}
         Telemetry java.analysis.main.success.time_ms: \\d+
         Telemetry java.analysis.main.success.type_error_count: 0
+        Telemetry java.classpath.main.module_count: 1
+        Telemetry java.classpath.main.modules_missing_binaries: 0
+        Telemetry java.classpath.main.modules_missing_libraries: 0
+        Telemetry java.classpath.main.modules_with_invalid_jdk: 0
+        Telemetry java.classpath.main.modules_with_unmatched_entries: 0
+        Telemetry java.classpath.main.modules_without_binary_entries: 0
+        Telemetry java.classpath.main.modules_without_library_entries: 0
         Telemetry java.dependency.lombok: absent
         Telemetry java.dependency.spring-boot: absent
         Telemetry java.dependency.spring-web: 5.3.18

its/plugin/tests/src/test/java/com/sonar/it/java/suite/MultiModuleTelemetryTest.java

@@ -44,6 +44,13 @@ public void test() {
     assertThat(buildResult.getLogs())
       .containsOnlyOnce("Telemetry java.language.version: 8")
       .containsOnlyOnce("Telemetry java.module_count: 2")
+      .containsOnlyOnce("Telemetry java.analysis.main.success.file_count: 2")
+      .containsOnlyOnce("Telemetry java.analysis.main.semantic.files_evaluated: 2")
+      .containsOnlyOnce("Telemetry java.analysis.main.semantic.modules_evaluated: 2")
+      .containsOnlyOnce("Telemetry java.analysis.test.semantic.files_evaluated: 2")
+      .containsOnlyOnce("Telemetry java.analysis.test.semantic.modules_evaluated: 2")
+      .containsOnlyOnce("Telemetry java.classpath.main.module_count: 2")
+      .containsOnlyOnce("Telemetry java.classpath.test.module_count: 2")
       .containsOnlyOnce("Telemetry java.scanner_app: ScannerMaven")
       .containsOnlyOnce("Telemetry java.is_autoscan: false")
       .containsOnlyOnce("Telemetry java.dependency.lombok: 1.18.30,1.18.38");

java-frontend/src/main/java/org/sonar/java/JavaFrontend.java

@@ -107,6 +107,7 @@ public JavaFrontend(JavaVersion javaVersion, SonarComponents sonarComponents, Me
     testCodeVisitors.addAll(sonarComponents.testChecks());
     List<JavaCheck> jspCodeVisitors = sonarComponents.jspChecks();
     boolean inAndroidContext = sonarComponents.inAndroidContext();
+    sonarComponents.recordClasspathTelemetry(telemetry);
 
     globalClasspath = Stream.of(classpath, testClasspath, jspClasspath)
       .flatMap(Collection::stream).distinct().toList();

java-frontend/src/main/java/org/sonar/java/SonarComponents.java

@@ -68,6 +68,8 @@
 import org.sonar.java.model.LineUtils;
 import org.sonar.java.reporting.AnalyzerMessage;
 import org.sonar.java.reporting.JavaIssue;
+import org.sonar.java.telemetry.Telemetry;
+import org.sonar.java.telemetry.TelemetryKey;
 import org.sonar.java.utils.ModuleMetadataUtils;
 import org.sonar.plugins.java.api.CheckRegistrar;
 import org.sonar.plugins.java.api.JavaCheck;
@@ -252,6 +254,13 @@ public NewHighlighting highlightableFor(InputFile inputFile) {
     return context.newHighlighting().onFile(inputFile);
   }
 
+  public void recordClasspathTelemetry(Telemetry telemetry) {
+    if (javaClasspath != null) {
+      javaClasspath.recordTelemetry(telemetry, TelemetryKey.JAVA_CLASSPATH_MAIN);
+    }
+    javaTestClasspath.recordTelemetry(telemetry, TelemetryKey.JAVA_CLASSPATH_TEST);
+  }
+
   public List<File> getJavaClasspath() {
     if (javaClasspath == null) {
       return new ArrayList<>();

java-frontend/src/main/java/org/sonar/java/ast/JavaAstScanner.java

@@ -62,6 +62,8 @@ public class JavaAstScanner {
   private final Telemetry telemetry;
   private final JavaAnalysisKeys javaAnalysisKeys;
   private VisitorsBridge visitor;
+  private boolean semanticModelEvaluated;
+  private boolean resolutionErrorsDetected;
   private boolean reportedMisconfiguredVersion = false;
 
   public JavaAstScanner(@Nullable SonarComponents sonarComponents, Telemetry telemetry, JavaAnalysisKeys javaAnalysisKeys) {
@@ -82,9 +84,10 @@ public List<File> getClasspath() {
    * under the {@code false} key, files that need to be parsed for further analysis.
    */
   public Map<Boolean, List<InputFile>> scanWithoutParsing(Iterable<? extends InputFile> inputFiles) {
-    return StreamSupport.stream(inputFiles.spliterator(), false)
-      // Split files between successfully scanned without parsing and failed to scan without parsing
+    Map<Boolean, List<InputFile>> files = StreamSupport.stream(inputFiles.spliterator(), false)
       .collect(Collectors.partitioningBy(visitor::scanWithoutParsing));
+    telemetry.aggregateAsCounter(javaAnalysisKeys.cachedFileCountKey(), files.get(true).size());
+    return files;
   }
 
   public void scan(Iterable<? extends InputFile> inputFiles) {
@@ -163,6 +166,7 @@ public void simpleScan(InputFile inputFile, JParserConfig.Result result, Consume
     var telemetryAnalysisKeys = javaAnalysisKeys.exceptions();
     try {
       JavaTree.CompilationUnitTreeImpl ast = result.get();
+      recordSemanticTelemetry(ast.sema.undefinedTypes());
       modifyCompilationUnit.accept(ast);
       visitor.visitFile(ast, sonarComponents != null && sonarComponents.fileCanBeSkipped(inputFile));
       String path = inputFile.toString();
@@ -189,11 +193,29 @@ public void simpleScan(InputFile inputFile, JParserConfig.Result result, Consume
         throw error;
       }
     } finally {
+      telemetry.aggregateAsCounter(telemetryAnalysisKeys.fileCountKey(), 1);
       telemetry.aggregateAsCounter(telemetryAnalysisKeys.sizeCharsKey(), InputFileUtils.charCount(inputFile, 0));
       telemetry.aggregateAsCounter(telemetryAnalysisKeys.timeMsKey(), currentTimeMillis() - startTime);
     }
   }
 
+  private void recordSemanticTelemetry(Set<JProblem> problems) {
+    var keys = javaAnalysisKeys.semantic();
+    long resolutionErrors = problems.stream().filter(problem -> problem.type() == JProblem.Type.UNDEFINED_TYPE).count();
+    telemetry.aggregateAsCounter(keys.filesEvaluated(), 1);
+    telemetry.aggregateAsCounter(keys.filesWithResolutionErrors(), resolutionErrors > 0 ? 1 : 0);
+    telemetry.aggregateAsCounter(keys.resolutionErrorCount(), resolutionErrors);
+    if (!semanticModelEvaluated) {
+      telemetry.aggregateAsCounter(keys.modulesEvaluated(), 1);
+      telemetry.aggregateAsCounter(keys.modulesWithResolutionErrors(), 0);
+      semanticModelEvaluated = true;
+    }
+    if (resolutionErrors > 0 && !resolutionErrorsDetected) {
+      telemetry.aggregateAsCounter(keys.modulesWithResolutionErrors(), 1);
+      resolutionErrorsDetected = true;
+    }
+  }
+
   private static void cleanUpAst(JavaTree.CompilationUnitTreeImpl ast) {
     // release environment used for semantic resolution
     ast.sema.getEnvironmentCleaner().run();

java-frontend/src/main/java/org/sonar/java/classpath/AbstractClasspath.java

@@ -49,6 +49,8 @@
 import org.sonar.api.config.Configuration;
 import org.sonar.java.AnalysisWarningsWrapper;
 import org.sonar.java.SonarComponents;
+import org.sonar.java.telemetry.Telemetry;
+import org.sonar.java.telemetry.TelemetryKey.ClasspathKeys;
 import org.sonarsource.api.sonarlint.SonarLintSide;
 
 @ScannerSide
@@ -70,6 +72,9 @@ public abstract class AbstractClasspath {
   protected List<File> elements;
   protected boolean validateLibraries;
   protected boolean initialized;
+  private int libraryEntryCount;
+  private boolean invalidJdkHome;
+  private boolean unmatchedClasspathEntries;
   private boolean inAndroidContext = false;
   private final Set<String> classpathWarnings;
   protected final AnalysisWarningsWrapper analysisWarnings;
@@ -96,6 +101,7 @@ protected void init() {
       binaries.addAll(getFilesFromProperty(binariesProperty));
       Set<File> libraries = new LinkedHashSet<>(getJdkJars());
       Set<File> extraLibraries = getFilesFromProperty(librariesProperty);
+      libraryEntryCount = extraLibraries.size();
       libraries.addAll(extraLibraries);
       logResolvedFiles(binariesProperty, binaries);
       logResolvedFiles(librariesProperty, libraries);
@@ -145,6 +151,7 @@ private Optional<File> existingDirectoryOrLog(String path) {
     LOG.debug("Property '{}' set with: {}", ClasspathProperties.SONAR_JAVA_JDK_HOME, path);
     File file = new File(path);
     if (!file.exists() || !file.isDirectory()) {
+      invalidJdkHome = true;
       classpathWarnings.add("Invalid value '" + file.getAbsolutePath() + "' for '" + ClasspathProperties.SONAR_JAVA_JDK_HOME + "' property, defaulting to runtime JDK.");
       return Optional.empty();
     }
@@ -171,6 +178,7 @@ protected Set<File> getFilesFromProperty(String property) {
       for (String pathPattern : fileNames) {
         Set<File> libraryFilesForPattern = getFilesForPattern(baseDir.toPath(), pathPattern, isLibraryProperty);
         if (validateLibraries && libraryFilesForPattern.isEmpty() && hasJavaFiles) {
+          unmatchedClasspathEntries = true;
           classpathWarnings.add("Invalid value for '" + property + "', no files nor directories matching '" + pathPattern + "'.");
         }
         validateLibraries = validateLibs;
@@ -371,6 +379,20 @@ private static Path resolvePath(Path baseDir, String fileName) {
     return filePath.normalize();
   }
 
+  public void recordTelemetry(Telemetry telemetry, ClasspathKeys keys) {
+    if (!hasJavaFiles()) {
+      return;
+    }
+    init();
+    telemetry.aggregateAsCounter(keys.moduleCount(), 1);
+    telemetry.aggregateAsCounter(keys.modulesMissingBinaries(), settings.hasKey(binariesProperty) ? 0 : 1);
+    telemetry.aggregateAsCounter(keys.modulesWithoutBinaryEntries(), binaries.isEmpty() ? 1 : 0);
+    telemetry.aggregateAsCounter(keys.modulesMissingLibraries(), settings.hasKey(librariesProperty) ? 0 : 1);
+    telemetry.aggregateAsCounter(keys.modulesWithoutLibraryEntries(), libraryEntryCount == 0 ? 1 : 0);
+    telemetry.aggregateAsCounter(keys.modulesWithInvalidJdk(), invalidJdkHome ? 1 : 0);
+    telemetry.aggregateAsCounter(keys.modulesWithUnmatchedEntries(), unmatchedClasspathEntries ? 1 : 0);
+  }
+
   public List<File> getElements() {
     init();
     return elements;