2.0: performance measurement windows and per-metric state (#769, #770)

CHANGELOG.md

@@ -19,6 +19,8 @@ and verification for each change is in the
 
 ### Changed
 
+- **Performance:** Each core metric (`performance.lcp`, `fcp`, `cls`, `tbt`) and sub-analysis reports its collection state; a throttled profile without all four core metrics has no score (`score: null`) and is not adopted as the canonical measurement.
+- **Performance:** LCP is the last candidate before the read, also one painted after `load`; TBT counts long tasks over FCP→TTI (Lighthouse window) instead of up to load + 100 ms. TTI is the end of the last long task before the first 5 s quiet window.
 - **Scoring:** A module that lost a collection its score depends on is shown as partial and no longer counts toward the overall score; derived indicators are not computed over a failed module.
 - **Scoring:** The overall score always uses the published module weights, including dual-viewport audits. Incomplete performance measurements and unchecked HTML conformance do not contribute; estimated Speed Index remains diagnostic. Principle coverage uses execution records instead of assuming every catalogued criterion passed.
 - **Security:** Security-only audits also collect the JavaScript library checks. The known Handlebars advisory range includes versions below 4.7.7 and describes conditional exploitability.
@@ -42,11 +44,13 @@ and verification for each change is in the
 
 ### Removed
 
+- **JSON:** The `tbt_zero_heavy_page` measurement warning. A static page on a slow network has a late LCP and no long tasks; TBT 0 is a valid measurement there.
 - **Library:** Breaking: the unfinished `audit::Baseline` API (`Baseline`, `BaselineDiff`, `BaselineViolation`, `WaivedViolation`). It was not wired into the CLI and identified findings by criterion and AX node id; a baseline comparison will build on the stable finding and pattern ids.
 - **JSON:** Breaking: `summary.template_clusters`, `summary.top_recurring_rules` and `internal_comparison.root_causes`. Recurring element patterns are `summary.root_causes` with their evidence grade; the former lists either duplicated it or called anything on two pages a "likely template or component".
 
 ### Fixed
 
+- **Performance:** A failed resource or coverage read no longer counts as 0 bytes, no blocking resources or 100 % JS used (`unused_js.used_pct` is `null` without measured bytes); a throttled pass whose CPU throttling or cache disabling failed is not stored as that profile.
 - **Collection:** A request or script that did not answer no longer becomes a finding (e.g. "HTTP/1.1 only", "www not consolidated", "invalid manifest", "broken link") or a clean result (e.g. "no cookies", "no console errors", 100 image-efficiency points).
 - **Content visibility:** The SSR content-gap check loads the page itself instead of measuring the blank tab left after interactive journeys.
 - **Accessibility:** Text spacing, reflow, page title, on-input and orientation report a technical rule failure instead of passing when the page gives no answer.

locales/de/report.ftl

@@ -346,7 +346,6 @@ perf-lab-data-body = LCP, CLS und TBT werden lokal in Headless-Chrome unter eine
 perf-render-blocking-analysis = Render-Blocking Analyse
 perf-measurement-warnings-title = Messtechnische Einschränkungen
 perf-warning-lcp-missing = LCP nicht erfasst: Der PerformanceObserver hat keinen LCP-Eintrag geliefert (typisch bei starker Netzwerkdrosselung) — LCP-Score fehlt in der Bewertung.
-perf-warning-tbt-zero = TBT = 0 ms bei schwerer Seite (LCP > 3 s): Headless-Chrome registriert keine Long Tasks — TBT ist wahrscheinlich unterschätzt.
 perf-warning-si-fallback = Speed Index entspricht LCP: Formelschätzung ohne unabhängigen Messwert (0,35 × FCP + 0,65 × LCP).
 perf-warning-tti-fallback = TTI entspricht LCP: Formelschätzung ohne direkten Interaktivitätswert.
 perf-warning-inp-missing = INP nicht messbar: Headless-Browser erzeugen keine Nutzereingaben — Interaktionslatenz konnte nicht erfasst werden.
@@ -597,6 +596,7 @@ pdf-perf-sub-resources = Ressourcen & Datenmenge
 pdf-perf-sub-bottlenecks = Lade-Engpässe & Rendering
 pdf-perf-throttled-title = Performance unter gedrosselten Bedingungen
 pdf-perf-throttled-profile = Profil
+pdf-perf-throttled-incomplete = Unvollständig: nicht alle Kernmetriken gemessen
 pdf-perf-cls-title = CLS — Layout-Verschiebungen
 pdf-perf-cls-value = Wert
 pdf-perf-cls-time = Zeitpunkt

locales/en/report.ftl

@@ -346,7 +346,6 @@ perf-lab-data-body = LCP, CLS and TBT are measured locally in headless Chrome un
 perf-render-blocking-analysis = Render-blocking analysis
 perf-measurement-warnings-title = Measurement limitations
 perf-warning-lcp-missing = LCP not captured: the PerformanceObserver returned no LCP entry (typical under heavy network throttling) — LCP score is absent from the rating.
-perf-warning-tbt-zero = TBT = 0 ms on a heavy page (LCP > 3 s): headless Chrome does not register long tasks — TBT is likely underestimated.
 perf-warning-si-fallback = Speed Index equals LCP: formula estimate without an independent measurement (0.35 × FCP + 0.65 × LCP).
 perf-warning-tti-fallback = TTI equals LCP: formula estimate without a direct interactivity value.
 perf-warning-inp-missing = INP not measurable: headless browsers generate no user input — interaction latency could not be captured.
@@ -597,6 +596,7 @@ pdf-perf-sub-resources = Resources & Payload Size
 pdf-perf-sub-bottlenecks = Loading Bottlenecks & Rendering
 pdf-perf-throttled-title = Performance under throttled conditions
 pdf-perf-throttled-profile = Profile
+pdf-perf-throttled-incomplete = Incomplete: not all core metrics measured
 pdf-perf-cls-title = CLS — Layout Shifts
 pdf-perf-cls-value = Shift
 pdf-perf-cls-time = Time

src/audit/collection.rs

@@ -413,6 +413,22 @@ pub static COLLECTORS: &[&CollectorSpec] = &[
     &crate::best_practices::console_errors::CONSOLE,
     &crate::best_practices::vulnerable_libs::LIBRARIES,
     &crate::design_quality::CANDIDATES,
+    &crate::performance::vitals::OBSERVERS,
+    &crate::performance::vitals::CDP_METRICS,
+    &crate::performance::vitals::NAVIGATION_TIMING,
+    &crate::performance::vitals::LCP,
+    &crate::performance::vitals::FCP,
+    &crate::performance::vitals::CLS,
+    &crate::performance::vitals::TBT,
+    &crate::performance::module::CONTENT_WEIGHT,
+    &crate::performance::module::RENDER_BLOCKING,
+    &crate::performance::module::THIRD_PARTY,
+    &crate::performance::module::CRITICAL_CHAIN,
+    &crate::performance::module::MINIFICATION,
+    &crate::performance::module::ANIMATIONS,
+    &crate::performance::coverage::JS_COVERAGE,
+    &crate::performance::coverage::CSS_COVERAGE,
+    &crate::performance::third_party::ISOLATION,
     &crate::dark_mode::STATIC_SUPPORT,
     &crate::dark_mode::PRINT_STYLESHEET,
     &crate::dark_mode::PRINT_LAYOUT,
@@ -627,6 +643,58 @@ mod tests {
                 "design_quality.candidates",
                 crate::design_quality::counter_cases,
             ),
+            (
+                "performance.observers",
+                crate::performance::vitals::counter_cases,
+            ),
+            (
+                "performance.cdp_metrics",
+                crate::performance::vitals::counter_cases,
+            ),
+            (
+                "performance.navigation_timing",
+                crate::performance::vitals::counter_cases,
+            ),
+            ("performance.lcp", crate::performance::vitals::counter_cases),
+            ("performance.fcp", crate::performance::vitals::counter_cases),
+            ("performance.cls", crate::performance::vitals::counter_cases),
+            ("performance.tbt", crate::performance::vitals::counter_cases),
+            (
+                "performance.content_weight",
+                crate::performance::content_weight::counter_cases,
+            ),
+            (
+                "performance.render_blocking",
+                crate::performance::render_blocking::counter_cases,
+            ),
+            (
+                "performance.third_party",
+                crate::performance::third_party::counter_cases,
+            ),
+            (
+                "performance.critical_chain",
+                crate::performance::critical_chain::counter_cases,
+            ),
+            (
+                "performance.minification",
+                crate::performance::minification::counter_cases,
+            ),
+            (
+                "performance.animations",
+                crate::performance::animations::counter_cases,
+            ),
+            (
+                "performance.js_coverage",
+                crate::performance::coverage::counter_cases,
+            ),
+            (
+                "performance.css_coverage",
+                crate::performance::coverage::counter_cases,
+            ),
+            (
+                "performance.third_party_isolation",
+                crate::performance::third_party::counter_cases,
+            ),
             ("dark_mode.static_support", crate::dark_mode::counter_cases),
             (
                 "dark_mode.print_stylesheet",
@@ -663,6 +731,46 @@ mod tests {
         }
     }
 
+    /// Plan 69 / #777: a lost collection makes the run partial, all of them
+    /// lost makes it failed — not "completed" with an empty, passing result.
+    #[test]
+    fn fold_marks_partial_and_failed_runs() {
+        use crate::audit::{ExecutionStatus, ModuleRun};
+        let record = |collector: &str, state| CollectionRecord {
+            collector: collector.into(),
+            module: "best_practices".into(),
+            state,
+            method: CollectionMethod::JavaScript,
+            method_version: 1,
+            scoring: true,
+            reason_code: None,
+            gaps: vec![],
+        };
+        let status = |states: [CollectionState; 2]| {
+            let mut runs = vec![ModuleRun {
+                module: "best_practices".into(),
+                status: ExecutionStatus::Completed,
+                ..Default::default()
+            }];
+            fold_records(
+                &mut runs,
+                vec![
+                    record("best_practices.console", states[0]),
+                    record("best_practices.libraries", states[1]),
+                ],
+            );
+            (runs[0].status, runs[0].subchecks.len())
+        };
+        use CollectionState::{Empty, Failed, Measured, NotApplicable};
+        assert_eq!(status([Measured, Empty]), (ExecutionStatus::Completed, 2));
+        assert_eq!(
+            status([Measured, NotApplicable]),
+            (ExecutionStatus::Completed, 2)
+        );
+        assert_eq!(status([Failed, Measured]), (ExecutionStatus::Partial, 2));
+        assert_eq!(status([Failed, Failed]), (ExecutionStatus::Failed, 2));
+    }
+
     #[test]
     fn inventory_ids_are_unique_and_scoped_to_their_module() {
         let mut seen = std::collections::HashSet::new();

src/audit/pipeline/assembly.rs

@@ -262,7 +262,6 @@ pub(super) fn aggregate_report(
 
     if let Some(performance) = snapshot.performance.clone() {
         report = report.with_performance(performance);
-        attach_performance_subchecks(&mut report);
     }
     if let Some(seo) = snapshot.seo.clone() {
         report = report.with_seo(seo);
@@ -363,68 +362,6 @@ pub(super) fn reconcile_image_alt_count(report: &mut AuditReport) {
     }
 }
 
-fn attach_performance_subchecks(report: &mut AuditReport) {
-    let Some(performance) = report.performance.as_ref() else {
-        return;
-    };
-    let checks = [
-        ("lcp", performance.vitals.lcp.is_some()),
-        ("fcp", performance.vitals.fcp.is_some()),
-        ("cls", performance.vitals.cls.is_some()),
-        ("tbt", performance.vitals.tbt.is_some()),
-        ("content_weight", performance.content_weight.is_some()),
-        ("render_blocking", performance.render_blocking.is_some()),
-        ("third_party", performance.third_party.is_some()),
-        ("critical_chain", performance.critical_chain.is_some()),
-        ("minification", performance.minification.is_some()),
-        ("animations", performance.animations.is_some()),
-        ("coverage", performance.coverage.is_some()),
-    ];
-    // The score comes from the Web Vitals, not these sub-analyses: losing
-    // all of them leaves a partial run, not a failed one.
-    attach_subchecks(report, "performance", &checks, false);
-}
-
-/// Records each sub-analysis of `module` as completed or failed. One failed
-/// sub-analysis makes the module run `Partial`; when the sub-analyses are the
-/// whole module (`whole_module`), all of them failing makes it `Failed` — an
-/// empty result from a collector that never ran is no measurement (plan 69).
-fn attach_subchecks(
-    report: &mut AuditReport,
-    module: &str,
-    checks: &[(&str, bool)],
-    whole_module: bool,
-) {
-    if let Some(run) = report
-        .accessibility
-        .execution
-        .module_runs
-        .iter_mut()
-        .find(|run| run.module == module)
-    {
-        run.subchecks = checks
-            .iter()
-            .map(|(name, available)| crate::audit::SubcheckRun {
-                subcheck: (*name).to_string(),
-                status: if *available {
-                    crate::audit::ExecutionStatus::Completed
-                } else {
-                    crate::audit::ExecutionStatus::Failed
-                },
-                reason_code: (!available).then(|| "subanalysis_unavailable".to_string()),
-                ..Default::default()
-            })
-            .collect();
-        if whole_module && checks.iter().all(|(_, available)| !available) {
-            run.status = crate::audit::ExecutionStatus::Failed;
-            run.reason_code = Some("all_subanalyses_unavailable".to_string());
-        } else if checks.iter().any(|(_, available)| !available) {
-            run.status = crate::audit::ExecutionStatus::Partial;
-            run.reason_code = Some("one_or_more_subanalyses_unavailable".to_string());
-        }
-    }
-}
-
 fn audit_scope_from_config(config: &PipelineConfig) -> crate::audit::AuditScope {
     let catalog = AuditCatalog::standard();
     let mut requested_modules = vec!["accessibility".to_string()];
@@ -874,28 +811,4 @@ mod tests {
         });
         report
     }
-
-    /// Plan 69: a collector that never ran makes the run partial, none at
-    /// all makes it failed — not "completed" with an empty, passing result.
-    #[test]
-    fn subchecks_mark_partial_and_failed_runs() {
-        let status = |checks: &[(&str, bool)]| {
-            let mut report = report_with_run("best_practices");
-            attach_subchecks(&mut report, "best_practices", checks, true);
-            let run = &report.accessibility.execution.module_runs[0];
-            (run.status, run.subchecks.len())
-        };
-        assert!(matches!(
-            status(&[("console_errors", true), ("vulnerable_libraries", true)]),
-            (ExecutionStatus::Completed, 2)
-        ));
-        assert!(matches!(
-            status(&[("console_errors", false), ("vulnerable_libraries", true)]),
-            (ExecutionStatus::Partial, 2)
-        ));
-        assert!(matches!(
-            status(&[("console_errors", false), ("vulnerable_libraries", false)]),
-            (ExecutionStatus::Failed, 2)
-        ));
-    }
 }

src/audit/pipeline/mod.rs

@@ -615,14 +615,25 @@ async fn audit_on_page(
         });
         if let Some((baseline_tbt_ms, origins)) = baseline {
             if !origins.is_empty() {
-                let impacts = crate::performance::isolate_third_party_impact(
-                    page,
-                    browser,
-                    url,
-                    &origins,
-                    baseline_tbt_ms,
-                )
-                .await;
+                let log = crate::audit::collection::CollectionLog::new();
+                let impacts = log
+                    .record(
+                        &crate::performance::third_party::ISOLATION,
+                        crate::performance::isolate_third_party_impact(
+                            page,
+                            browser,
+                            url,
+                            &origins,
+                            baseline_tbt_ms,
+                        )
+                        .await,
+                    )
+                    .into_value()
+                    .unwrap_or_default();
+                crate::audit::collection::fold_records(
+                    &mut report.accessibility.execution.module_runs,
+                    log.into_records(),
+                );
                 if let Some(perf) = report.performance.as_mut() {
                     if let Some(third_party) = perf.third_party.as_mut() {
                         third_party.isolated_impact = impacts;