Add one mandatory parameter to the most-used method in a production backend. Both runs find what breaks. Only one of them stays reliable getting there.
IConnectionFactory.CreateConnectionAsync() is the database access primitive: almost every repository, sync service, API endpoint and background job calls it. Making the parameter mandatory ripples through the whole codebase. The question every engineer asks first: what exactly breaks?C# / Roslyn tier ~3,500 symbols Claude Opus 4.8
Add a non-optional CancellationToken to the interface method and its single implementation. Every call site that omits it stops compiling. The same prompt, the same model, the same codebase. The only variable: does the agent have ctxo, or does it work the file system by hand?
With ctxo: one deterministic tool chain - search_symbols, get_symbol_importance, find_importers, get_blast_radius, get_why_context.
Without ctxo: grep, ripgrep, file reads, and parallel sub-agents - many rounds, manual reconciliation, no graph to lean on.
Tool-call and sub-agent counts are exact (read off the transcripts). Wall-clock was not stopwatched, so we describe it directionally: ctxo runs as a single fast pass; the manual route needs many sequential rounds plus parallel sub-agent dispatch.
| Metric | With ctxo | Without ctxo |
|---|---|---|
| Tool calls to the answer | 6 (one chain) | ~15 + 2 sub-agents |
| First-pass result | correct | grep misfired - 1 match, "looks like dead code" |
| Centrality of the symbol | PageRank #1 of 3,500, instantly | not computable by grep |
| Call sites found | 155 / 54 files | 155 / 54 files (after recovery) |
| Vs compiler ground truth | 155 = 154 build errors + 1 in a skipped test project. Exact. | same set, reached the expensive way |
| Cross-cutting signal | 18 clusters - multi-team review | flat list, no boundary view |
| Git history / anti-patterns | checked in one call (clean) | not checked |
| Reproducibility | identical every run | depends on tool + sub-agent luck |
Ground truth was produced by actually applying the change on a throwaway branch and running dotnet build. The compiler reported 154 missing-argument errors plus 14 downstream type-inference cascades; the one remaining predicted site lived in a test project that was skipped once the core project failed to compile. ctxo's prediction matched the compiler exactly.
At Opus 4.8 the manual route eventually arrives at the correct set. That is the honest result, and it is exactly why the difference matters: getting there meant a misfired grep that almost flagged the codebase's most-used method as unused, two sub-agents whose answers disagreed, and a stack of rounds reconciling it all by hand. ctxo returned the same set as one deterministic chain, named it the #1 symbol by PageRank, mapped 18 clusters, and checked the git history - with nothing to fabricate.
Indexed facts, symbol-scoped, identical every run. No misfires, no fabricated call sites.
PageRank centrality, cluster blast radius and git intent - signals grep structurally cannot produce.
Its prediction matched the compiler exactly, and it has caught real bugs in real production history.