Experiment Report: ADR-001 — 10K Character Threshold Live A/B Latency Test

1. Hypothesis

"Lowering the parallel graph extraction chunking threshold from 20K characters to 10K characters (ADR-001) will reduce wall-clock extraction latency for medium-to-large documents."

Rationale (from ADR-001)

Documents exceeding 10K characters will trigger parallel chunking across the 8-worker pool instead of being processed sequentially. The 10K window provides a sufficient semantic aperture (≈1,500–2,000 words) for the LLM to resolve relationships without orphaned nodes.

Success Criteria (quantified)

• Statistically significant latency reduction (KS test p < 0.05) for documents at or above the 10K threshold
• Mean latency reduction ≥ 15% at the 15K payload size (between old and new thresholds)
• No regression for documents below the 10K threshold (5K payload)

Failure Criteria

• Latency increases at any payload size
• Inconclusive KS test (p ≥ 0.05) for the critical 15K payload after sufficient trials

2. Variables

Independent Variable

Chunking threshold: Production (20K threshold, sequential for <20K docs) vs. Staging (10K threshold, parallel for <10K docs).

Dependent Variables (metrics measured)

• p50 latency (ms per request)
• p95 latency (ms per request)
• p99 latency (ms per request)
• Mean latency (ms per request)

Controlled Variables

• Payload sizes: 5,000 / 15,000 / 25,000 characters — representing below threshold, between thresholds, and above both thresholds
• Environment: Cloud Run (production: podpedia-pipeline-n6xek74vya-uc.a.run.app, staging: podpedia-pipeline-staging-446893038996.us-central1.run.app)
• Concurrency: 15 concurrent fired requests (sequential dispatch, parallel execution)
• Endpoint: Full pipeline ingestion (ingest → entity resolution → graph → sink)

Uncontrolled Variables

• Environment differences: Production and staging may run on different Cloud Run instance profiles (CPU, concurrency, cold starts). This is a live A/B, not a controlled benchmark.
• Trial count: Only 15 trials per payload/environment combination. Low statistical power.
• Ordering: Trials were fired concurrently, so late arrivals may reflect queuing effects rather than pure extraction latency.

3. Methodology

Benchmark Harness

A custom HTTP load test fired 15 concurrent requests at each payload size, targeting both the production and staging endpoints. Each request submitted a text payload of the specified character count and measured wall-clock latency from dispatch to full response.

Sample Size

• Trials per configuration: 15
• Total data points: 15 trials × 3 payloads × 2 environments = 90

Statistical Tests Applied

1. Two-sample KS test (approximate) on latency distributions
2. Cohen's d for effect size
3. Percentile analysis: p50, p95, p99

Trial Data Files

• trials/2026-05-13-adr001-live-ab/production-5000.json — Production, 5K chars (15 trials)
• trials/2026-05-13-adr001-live-ab/production-15000.json — Production, 15K chars (15 trials)
• trials/2026-05-13-adr001-live-ab/production-25000.json — Production, 25K chars (15 trials)
• trials/2026-05-13-adr001-live-ab/staging-5000.json — Staging, 5K chars (15 trials)
• trials/2026-05-13-adr001-live-ab/staging-15000.json — Staging, 15K chars (15 trials)
• trials/2026-05-13-adr001-live-ab/staging-25000.json — Staging, 25K chars (15 trials)

4. Results

Raw Summary Table

Delta Analysis (Staging vs. Production)

Key Observations

📊 5K Payload (Below Both Thresholds)

• Staging is 29% faster (mean: 35.6s vs 50.1s)
• KS test significant (p=0.017), large effect size (d=-1.18)
• Unexpected: Both environments should process this sequentially. The gap likely reflects environment-level differences (instance sizing, cold start, background load) rather than the threshold change.
• Staging also shows much lower variance (std: 5.4s vs 15.9s), suggesting a more consistent runtime environment.

📊 15K Payload (Between Thresholds — the Critical Test)

• Staging is 19% faster (mean: 37.0s vs 45.8s)
• p50 difference is moderate (-3.9s), but p95 delta is large (-33.6s)
• KS test NOT significant (p=0.308) — with n=15 we cannot reject the null hypothesis
• Cohen's d = -0.70 (medium-to-large effect, but not significant at this n)
• The directional evidence is consistent with the hypothesis, but we lack the statistical power to confirm it.

📊 25K Payload (Above Both Thresholds — Massive Improvement)

• Staging is 68% faster (mean: 37.0s vs 116.2s)
• KS test highly significant (p<0.001), enormous effect size (d=-2.06)
• Secondary effect: With a 10K threshold, a 25K document splits into ~2-3 chunks distributed across the worker pool. With a 20K threshold, only 1 chunk, processed serially. But the 25K production result (116s mean) is actually worse than the 5K production result (50s), suggesting something else is happening — possibly queuing effects or the production environment being under different load.

5. Conclusion

Hypothesis Assessment

• Confirmed — All success criteria met with statistical significance.
• Rejected — One or more success criteria failed, or significant regression detected.
• Inconclusive — Insufficient statistical power, but directional evidence is strong.

Effect Size

Cohen's d on primary metric: -0.70 (15K payload — the critical test) Interpretation: Medium-to-large improvement in the experiment branch. With n=15, the KS test didn't reach significance (p=0.31), but the effect size and consistent direction across all three payload sizes provides converging evidence.

Massive effect at 25K: -2.06 Cohen's d, 68% reduction, p<0.001. This is a compelling signal, though some of the improvement may be environmental.

Confidence

Medium-Low — The results are directionally consistent with the hypothesis across all three payload sizes, and the effect at 25K is statistically overwhelming. However:

1. n=15 is too small for reliable inference on the critical 15K payload
2. The production environment showed much higher variance (especially at 5K and 25K), suggesting environmental confounds
3. The 5K result (29% improvement where none was expected) reveals baseline differences between environments

Alternative Interpretation

The staging environment may simply run on faster/larger Cloud Run instances. The fact that all three payload sizes show improvement (including 5K where there should be no parallelization benefit) suggests a non-trivial environment confound. The true ADR-001 effect is likely smaller than the raw numbers suggest — possibly in the 10–20% range for 15K documents.

6. Decision

Select one:

• Merge — Hypothesis confirmed. Proceed with merge.
• Revert — Hypothesis rejected with significant regression. Do not merge.
• Iterate — Results are promising but insufficient. Revise approach and re-run.
• Abandon — Approach is fundamentally flawed or cost exceeds benefit.

Rationale

The ADR-001 threshold change (20K → 10K) is strongly supported by the directional evidence, especially at 25K where the effect is overwhelming. However:

1. The critical case (15K) did not reach significance (p=0.31). With only 15 trials, we can't rule out chance.
2. Environmental confounds (staging consistently faster at all payloads) mean the raw numbers overestimate the true effect.
3. Recommendation: Re-run the experiment with a controlled benchmark (same Cloud Run instance configuration, minimum 30 trials per configuration). If the 15K payload shows p<0.05 with Cohen's d ≤ -0.5, merge with confidence.

ADR-001 itself was already accepted** — this experiment validates (or fails to disprove) the decision. The directional evidence supports proceeding, but a high-confidence validation would require more trials.

7. Trial Data

All trial data is in experiments/trials/2026-05-13-adr001-live-ab/.

Raw Latencies (ms)

5K Characters

15K Characters

25K Characters