The setup
Zaguán Blade has a Symbols Index: a local, on-disk map of a codebase, functions, classes, methods, relationships, anchors, that lets an AI coding agent find the right code before it reads everything. The whole philosophy is to send targeted context to the model instead of turning every task into a token furnace.
The index worked. It worked on our benchmark corpora. It worked on the Linux kernel, slowly. So we did the obvious thing: we pointed it at all of Firefox. 469,627 files. Thirty years of history. C++, Rust, JavaScript, a menagerie of test fixtures, and a 5.4 GB .git directory.
Two decisions made this the most productive debugging stretch I’ve had in a long time.
The first was inheriting ideas from an inspiration project: codebase-memory-mcp by DeusData. That project’s approach to generated-file handling and per-file parse budgets seeded a couple of our guardrails, the “index the file anchor-only if it looks generated” heuristic and a wall-clock cap on a single tree-sitter parse. Good ideas travel.
The second decision was, on the face of it, a mistake: I ran the whole thing off a 2 TB spinning hard drive attached over USB. Not because it was a good idea for performance. Because my main NVMe was almost full and an extra 2 TB is a Good Thing™.
That slow, seeky, high-latency USB spinner turned out to be the best debugging tool in the whole exercise. It exaggerated every stall just enough to make the real bottlenecks impossible to hide. A fast disk would have papered over at least three of the bugs below. This is the thesis of the whole post: slow hardware is a fuzzer for your assumptions.
Here’s everything that happened. The awful, the grueling, the genuinely great, and the parts where I was confidently, embarrassingly wrong.
Act I: The crashes (or, “something bad happened”)
The first full-Firefox run didn’t finish. It didn’t even fail cleanly. It sat at 100% CPU on a single file and stopped making progress. The best description I could muster in my own notes at the time was, verbatim, “something bad happened… and I don’t know what or why.”
The index workers run with panic = "abort" in the release build, chosen to strip panic-cleanup code for a leaner, faster binary. The consequence: any panic in any worker takes down the whole process. One pathological file out of 469,000 crashes the entire index. Over the next few days, Firefox’s sheer variety found us three of these, one at a time. Whack-a-mole.
To hunt them, I built a small throwaway harness, a watchdog that walks the corpus, indexes each file on a detached thread with a timeout, and records the current file to a tracker before every call so that even a panic=abort crash names the culprit. It earned its keep immediately.
Bug #1: Infinite recursion on cyclic re-exports
The first “hang” wasn’t slow, it was unbounded recursion. When the indexer enriches a file’s relationships, it resolves export * from './x' by indexing the target module, which enriches its re-exports, and so on. Firefox is full of barrel files (index.js re-exporting siblings) and Python __init__.py packages that re-export each other. A cycle in that graph, a re-exports b, b re-exports a, recurses forever.
On the 2 MB stacks in my reproducer it overflowed instantly. On the app’s 256 MB worker stacks it didn’t overflow, it just ground at 100% CPU, looking exactly like a hang. A gdb backtrace showed the tell: the same five frames repeating forever.
The fix was a classic cycle-breaker: a per-thread re-entrancy guard. If a file is already being indexed higher up the current thread’s stack, don’t recurse into it again. The forward edge is still recorded; only a redundant back-edge in a true cycle is skipped.
Bug #2: A single @ character
The next one was beautiful. The scan froze, and this time I grabbed the live process ID while it was still stuck. top -H showed one thread pegged; gdb showed it deep inside serde_yaml. I bisected a 596 KB file, modules/libpref/init/StaticPrefList.yaml, down to a single line:
value: @IS_XP_MACOSX@@ is a reserved indicator character in YAML. That token is a Firefox build-time placeholder, the file is only valid YAML after preprocessing. So serde_yaml hit a hard parse error. And here’s the trap: our config scanner iterated documents with
for document in serde_yaml::Deserializer::from_str(content) {
let Ok(value) = serde_yaml::Value::deserialize(document) else { continue };
...
}serde_yaml’s Deserializer does not advance past a parse error, it re-yields the same Err on every poll. continue there is an infinite loop that allocates and drops an error object forever, which is exactly what the profiler kept showing. The fix was one word: break instead of continue.
Bug #3: range start index 153 out of range for slice of length 30
The third crash was a real panic, aborting the whole run on a web-platform-tests file:
thread '<unnamed>' panicked at tree-sitter-0.26.9/binding_rust/lib.rs:2010:31:
range start index 153 out of range for slice of length 30tree-sitter can, during parse-error recovery on malformed input, hand back a node whose byte range lies outside the source buffer. Node::utf8_text then slices &source[153..] on a 30-byte string and panics before the ubiquitous .ok()? can catch it. With panic=abort, that’s the whole index dead.
We couldn’t catch_unwind it (abort doesn’t unwind), so the fix had to be prevention: a safe_text wrapper that bounds-checks the node’s range against the source length before ever calling utf8_text, applied to all 52 extraction call sites. I validated it by running the reproducer across the entire testing/web-platform/tests subtree, 160,256 files, with no panic.
The lesson from Act I: these were not “big repo is slow” problems. Each was a specific, latent correctness bug that only a corpus with Firefox’s diversity would surface. And panic=abort turned each one into a full crash. We discussed switching to panic=unwind + catch_unwind for general robustness; the call was to keep the lean binary and fix each bug as it appears. Fair, but it’s the kind of decision worth making on purpose, not by accident.
Act II: The great “it’s slow” misdiagnosis
With the crashes fixed, Firefox indexed end to end. In about two and a half hours. That’s not shippable; it looks amateurish.
My gut said: it’s the CPU. We have a 16-core / 32-thread machine and the extraction is parallel, surely we’re not saturating it. I’d already bumped the worker pool to use every core. And yet CPU utilization was low. So it must be extraction, or enrichment, or the sheer symbol volume, right?
Wrong. Completely wrong. And the only reason I found out is that I profiled the live process instead of theorizing.
top -H on the running index showed one thread busy and 48 asleep. The machine was 87.8% idle with 9.2% I/O-wait. One core of 32. Then the gdb backtrace of the one busy thread:
commit_staged_file_indexes
→ sqlite3_step
→ sqlite3WalDefaultHook
→ sqlite3_wal_checkpoint_v2
→ sqlite3WalCheckpoint
→ fsync ← blocked hereIt was never CPU-bound. It was fsync-bound. sqlite3WalDefaultHook is SQLite’s automatic WAL checkpoint, firing at its default threshold of 1000 pages, 4 MB. Over a multi-gigabyte index, that’s roughly a thousand synchronous checkpoint-and-fsync cycles, each one blocking the single writer thread while every extraction worker sat idle on backpressure.
This is where the slow disk earns its billing. On an NVMe, a thousand tiny fsyncs are annoying but survivable, you might never notice. On a USB spinner, each one is a real seek-and-flush, and they added up to hours. The disk didn’t cause the bug; it revealed it, loudly.
The fix was one line, raise wal_autocheckpoint to 1 GiB so a handful of big checkpoints fire instead of a thousand tiny ones, and let the final wal_checkpoint(TRUNCATE) consolidate at the end. Combined with a couple of pipeline changes (an overlapping committer thread so extraction and commit run concurrently, and larger batches), the run went from 2.5 hours to about 30 minutes.
The lesson: I would have “optimized” extraction for a week and moved the needle by nothing. The profiler took ten minutes and pointed at the disk. Measure the thing. Do not optimize your guess.
Act III: Value density, what does the model actually need?
While chasing speed, we also looked at the output. The Firefox index was 8.8 GB. I had to reframe what I was even optimizing for: a big DB isn’t inherently bad, it just means the codebase is big. What matters is whether every byte earns its place by being valuable to the model.
So we audited. The index held 5.87 million semantic anchors, more than the 5.16 million symbols. When I looked at what they actually were:
config_keyanchors turned out to be C++ class names, already stored as symbols. Pure redundancy.translation_keyanchors were header filenames (nsIURI.h) misclassified because they contain a dot.css_tokenanchors were mostly C++--decrementexpressions, not CSS variables.
The anchor system was designed for web code, routes, i18n keys, CSS custom properties, where string literals are genuine cross-references. On a C++/systems codebase it mostly captured noise and duplicated the symbols. So we gated the generic literal scan to front-end / style / markup languages and left systems code to the symbols that already own it. That alone cut about 2.66 million anchors (45%) and eliminated every misclassification, all of which were C++.
Then we applied the index’s north star to the query results: references, not text. The old symbol serializer embedded each symbol’s full docstring (up to ~6.7 KB) in every node of every bulk search or graph result. A single query could balloon into a wall of doc comments. The vision is the opposite: return a precise reference, file plus byte range, and let the model read the exact bytes on demand. So bulk results now carry the compact signature and the locating range, with a has_docstring flag; the full text is fetched only when the model asks for one specific symbol.
A word of caution that came out of this act, and one I had to keep reminding myself of: web frameworks matter. A huge share of AI-coding users are building SaaS and sites in Next.js, Astro, and friends. It would have been easy to over-trim anchors and gut the value for exactly those users. That’s a thin line, and we deliberately stayed on the generous side of it for web code.
Act IV: The ten wasted minutes, and a relic
Even at 30 minutes, there was a tail: after the scan “finished,” the app sat for 10+ minutes looking idle. And I had a nagging hunch: if we already have line numbers, that second pass is completely wasted, it smelled like a relic. It was.
A live gdb trace showed a thread stuck in IndexerManager::new → FileMetadata::from_path → count_lines → read_to_string, in uninterruptible I/O wait, reading every one of the 469,000 files’ full contents. Why? To compute a line_count field that was written once and never read by anything. Dead I/O, duplicating a line count the symbols index already stored. On the spinner, that dead read cost ten minutes. (There it is again: fast disk hides it, slow disk screams.)
Removing the field made that pass stat-only. But pulling the thread revealed the whole thing was a pre-symbols-index relic: a second indexer that walked the entire workspace, wrote a 130 MiB index.json on every launch, and produced a project overview, none of which anything still consumed. The GUI file tree reads the filesystem directly. The legacy overview writer was off by default. The one modern tool that produces a project overview had already been quietly migrated to the symbols index. So we deleted it: ~1,680 lines, nine files, gone. The symbols index is now genuinely the index.
Act V: Telling the user what’s happening
The last thread was UX. Throughout that multi-minute finalization, global relationship back-fill, receiver-type mining, the final checkpoint, the status bar said, unhelpfully, “Indexing symbols.” It looked frozen even though one core was busy doing real work.
Profiling confirmed it wasn’t a hang (it was the back-fill, a big set-based SQL pass), so the fix was purely to narrate it. And here I nearly shipped a mistake I’d have caught in review anyway: I’d almost forgotten that the app supports two languages, English and Spanish. The existing status label was hardcoded English, an i18n gap. So the fix both closed the gap and added a distinct, localized “Finalizing index… / Finalizando índice…” phase for the post-scan window, so a legitimately busy tail no longer reads as a stall.
What we actually learned
Profile the live process; don’t optimize your gut. Nearly every real fix in this saga came from top -H plus a gdb backtrace on a running PID, and every one of them debunked the obvious theory. The hangs were recursion and an error-loop, not slowness. The “CPU” problem was fsync. The “wasted” minutes were a dead line-count read. My intuition batted roughly .000.
Slow, honest hardware is a feature. The USB spinner didn’t cause a single bug, but it surfaced at least three that a fast NVMe would have buried under acceptable-looking latency: the checkpoint stalls, the dead file reads, the general “is it working or hung?” ambiguity. If you only ever test on fast disks, you’re testing on a difficulty setting that hides your worst mistakes.
Diversity is the real stress test. Firefox didn’t break the index because it’s big. It broke it because it contains barrel-file cycles, YAML with build-time placeholders, deliberately malformed web-platform-tests, and generated data, the long tail of “valid input we never imagined.” Benchmarks of well-behaved repos told us none of this.
panic=abort is a real tradeoff, not a default. A lean binary is nice; one bad file crashing a 469k-file job is not. Whatever you choose, choose it knowing that a single unhandled panic in a worker is a full outage.
Value density beats size. The index got faster and leaner and more useful by asking, of every anchor and every field, “does the model actually need this, or can it read the range itself?” References over text. Symbols over redundant literals. Web-relevant anchors over C++ noise.
Delete the relic. The single most satisfying commit was −1,680 lines: a whole subsystem the symbols index had already superseded, still paying its full cost on every launch. If two things claim to be the index, one of them is wrong.
The numbers
| Before | After | |
|---|---|---|
| Full Firefox cold index | ~2.5 hours, then crash/hang | ~30 minutes, clean |
| Distinct single-file crashes | 3 (recursion, YAML loop, tree-sitter panic) | 0 |
| Semantic anchors | 5.87M (81% noise/redundant) | −45%, misclassification eliminated |
| Peak RAM during index | climbing unbounded (~2 GB of dead cache) | flat, under 1.5 GiB |
| CPU during commit | ~1 core of 32 (fsync-bound) | cores kept busy |
| Post-scan “dead” window | 10+ minutes reading every file | stat-only |
| Legacy subsystem | ~1,680 lines, 130 MiB cache/launch | deleted |
Giving it back
We’ve been through the whole spectrum over these last few days, the awful first crash report, the grueling multi-hour runs, the confidently wrong guesses, and the genuinely great moment when a one-line pragma change cut two hours off the clock. None of it was clever in isolation. It was a slow disk, a live debugger, an inspiration project (codebase-memory-mcp), and the discipline to keep asking “what is it actually doing?” instead of “what do I think it’s doing?”
If there’s one thing to take away: the next time your build or your indexer feels slow, resist the urge to optimize the part you already suspect. Attach a profiler. Run it on your worst hardware. Let the machine tell you the truth. It’s usually more interesting, and more embarrassing, than your theory.
Now, if you’ll excuse me, there’s an NVMe test to run.