Measuring Mutation Effectiveness¶

A structure-aware fuzzer is only as good as the strategies it mutates with. Crucible ships ~60 GGUF mutation strategies; crucible mutate-stats measures which of them actually earn their keep so the engine can invest in what works and prune what does not. It is purely additive telemetry: it consumes the strategy names the mutator already reports and the stack hashes triage already computes. It never changes how mutation works, and it never decides anything a human gate owns.

The honest part first: attribution is co-occurrence¶

The mutator applies 1-3 strategies per input, so in general you cannot isolate an effect to a single strategy. The ledger therefore uses co-occurrence credit: an input that finds new coverage or a crash credits every strategy that fired on it. To recover a cleaner, causal-leaning signal, it also tracks the solo subset (inputs where exactly one strategy fired) in separate SOLO / SOLO-COV columns. Read the numbers as evidence for where to invest, not as proof any single strategy caused a bug.

Two ways to measure¶

1. Go-side campaign (no target) — exploration breadth¶

mutate-stats run drives the real mutation engine over a seed corpus and records, per strategy, how often it fired (APPLIED / SOLO) and how often it produced a structurally-novel output (NOVEL). Novelty is an exploration-breadth proxy, not code coverage, but it needs no build and is deterministic under --seed:

crucible mutate-stats run --corpus ./corpus --iterations 20000 --seed 42 \
  --out mutation-stats.json
crucible mutate-stats report --in mutation-stats.json

The ledger accumulates: run it again (another seed, another corpus) against the same --out and the counts merge.

2. Provenance join (real target) — coverage & crash yield¶

To attribute real new-coverage inputs and crashes to strategies, fuzz with the provenance-logging mutator, then join its log against what the fuzzer kept.

The traced mutator (cmd/crucible-mutator-traced) is byte-for-byte identical in output to the normal custom mutator; when CRUCIBLE_PROVENANCE_LOG is set it additionally appends output-hash -> [strategy names] for each mutant. Build it like the normal mutator archive and link it into any harness:

go build -buildmode=c-archive -o libcruciblemut_traced.a ./cmd/crucible-mutator-traced
# ...link libcruciblemut_traced.a into the harness exactly like libcruciblemut.a...

# Run a BOUNDED campaign (the log is one line per exec — cap the budget):
CRUCIBLE_PROVENANCE_LOG=prov.jsonl \
  ./harness corpus/ -artifact_prefix=crashes/ -max_total_time=120

The whole loop — build check, restart-loop campaign, join, report — is wrapped by scripts/traced-mutation-stats.sh <harness> <seeds> <workdir> [budget], which refuses to run if the harness has the weak-undefined mutator.

Then join the log against the corpus the fuzzer kept (each kept file = a new-coverage input) and the crash artifacts:

crucible mutate-stats join \
  --provenance prov.jsonl \
  --corpus corpus/ \
  --crashes crashes/ \
  --harness ./harness \
  --out mutation-stats.json
crucible mutate-stats report --in mutation-stats.json

--harness is optional: with it, each crash is replayed and deduplicated by stack hash (bug-level); without it, crashes are counted per unique crashing input (input-level). The join tolerates a truncated log from a killed campaign, and any input it cannot find in the provenance log is reported as unmatched rather than misattributed.

Reading the report¶

STRATEGY                  APPLIED  SOLO  COV  SOLO-COV  COV%  NOVEL  CRASHES
metadata.int_overflow     324      51    2    1         0.6   317    1
tensorinfo.offset         713      128   1    1         0.1   672    1
header.version            305      59    1    0         0.3   299    0

• APPLIED / SOLO — how much budget the strategy consumed (and the single-strategy subset).
• COV / SOLO-COV / COV% — new-coverage inputs it co-occurred with (and the solo subset, and yield-per-application).
• NOVEL — structurally-distinct outputs (Go-side breadth proxy).
• CRASHES — unique crash stack-hashes (or crashing inputs without --harness) credited to it.

Strategies are ranked crashes-first, then coverage, then novelty. The report also prints prune candidates: strategies applied many times with zero coverage and zero crashes, the "drop what does not earn it" list. Use --json for machine-readable output.

Where this fits¶

This is Track A of the fuzzing-expansion plan: make the engine measurable before adding more strategies, so new strategies (Track B) are added where the stats show gaps and the weak ones are pruned. It changes nothing about the two human gates — exploitability-tier ratification and disclosure still stay with the operator.