Evidence and Validation¶

Crucible's output is not crashes. It is claims -- "this parser has an out-of-bounds read reachable from an untrusted model file, and here is a fix." A claim is only worth what its evidence is worth, so the evidence layer is part of the architecture rather than a process bolted on afterwards.

RRR¶

Every claim, finding, patch and capability is held to three properties:


Replicability	Someone else, on their own machine, from the written artifact alone, gets the same result. If the reproduction needs a detail that was not published, it is not replicable.
Robustness	The result holds outside the one configuration that was tested -- across architectures, build types, optimisation levels, and inputs that were not hand-picked.
Reproducibility	The same result can be obtained again on demand from recorded inputs: pinned commits, retained artifacts, a command that runs. Not "it crashed once".

These are the bar, not aspirations. A finding that fails any one of them is not ready, regardless of how real the underlying bug is.

The evidence ladder¶

Not all witnesses are equal, and the difference decides what a finding may claim.

flowchart TD
    S["<b>source-confirmed</b><br/>read the code, formed a hypothesis"]
    D["<b>detorch</b><br/>faithfully extracted code path<br/>reproduced under a sanitiser"]
    R["<b>real-artifact</b><br/>the project's own binary or module,<br/>control vs crafted"]
    E["<b>end-to-end</b><br/>the deployed entry point,<br/>no harness substitution"]

    S -->|"compile the real path"| D
    D -->|"build the shipped artifact"| R
    R -->|"drive the real entry point"| E

    S -. "a hypothesis,<br/>never a finding" .-> X1[" "]
    D -. "proves the code is unsafe.<br/><b>Does NOT prove it is<br/>compiled, linked or reached.</b>" .-> X2[" "]
    R -. "proves it runs<br/>in the real thing" .-> X3[" "]

    style S fill:#4a2020,stroke:#a55,color:#fff
    style D fill:#4a3a20,stroke:#a85,color:#fff
    style R fill:#204a2a,stroke:#5a8,color:#fff
    style E fill:#20304a,stroke:#58a,color:#fff
    style X1 fill:none,stroke:none
    style X2 fill:none,stroke:none
    style X3 fill:none,stroke:none

The rung that causes trouble is detorch. It establishes that the code as written is unsafe. It says nothing about whether that code is compiled into the shipped artifact, linked, or reachable from any entry point. A finding may be genuinely correct at the detorch rung and still describe code that never runs.

This is why reachability is a separate claim from the bug, and why a reachability claim resting on a source read is recorded as source-confirmed, never as established.

Why the remedy needs its own evidence¶

A finding says "this is broken." A patch says "this is now fixed, and nothing else broke." The second is a strictly stronger claim, and it needs evidence the first does not:

which copy of the code actually runs -- a duplicated loader means a correct patch to the wrong file records the bug as fixed while leaving it live
what legitimate input looks like -- a bound that looks obviously right can reject a real, shipped artifact, and a security fix that breaks users gets reverted
what happens downstream of the new check -- a guard can pass while the original fault still reproduces further along
every site that reaches the sink -- fixing one call site and staying silent about the others reads as completeness

A detorch supplies none of those. So the evidence bar rises for the remedy even when the finding was already established.

The validation layer¶

Three tools answer these questions mechanically. All three report evidence and never decide: none can represent "by-design", "fixed", "known" or "submit", and all exit successfully regardless of outcome so that none can silently gate a pipeline.

flowchart LR
    subgraph EV["Validation layer (tools/)"]
        direction TB
        LIVE["<b>crucible-live</b><br/>is this code compiled,<br/>linked and referenced?"]
        PC["<b>crucible-patchcheck</b><br/>does this patch close the bug<br/>without breaking real input?"]
        LED["<b>build-ledger</b><br/>what do we actually have?"]
    end

    F["Finding<br/>(evidence tier)"] --> LIVE
    LIVE --> PC
    P["Candidate patch"] --> PC
    PC --> G{"Operator<br/>gate"}
    LED --> REP["Generated records<br/>(single source of truth)"]

    style LIVE fill:#20304a,stroke:#58a,color:#fff
    style PC fill:#204a2a,stroke:#5a8,color:#fff
    style LED fill:#3a2a4a,stroke:#85a,color:#fff
    style G fill:#4a2020,stroke:#a55,color:#fff

Tristate, never boolean¶

Every check returns PRESENT / ABSENT / UNKNOWN, or PASS / FAIL / UNKNOWN.

UNKNOWN is a normal, frequent and correct answer. A high unknown rate is a pass. The failure mode these tools are built to avoid is UNKNOWN silently reading as ABSENT, because that is how a validation tool starts killing real findings. A lone negative from a weak signal is therefore downgraded to UNKNOWN unless a second, independent signal corroborates it.

crucible-live¶

Four independent checks against a real built artifact:

Check	Answers
`symbol_defined`	is the function in the binary at all
`symbol_xref`	is it referenced, or compiled with zero surviving callers
`preproc_live`	does the source line survive the preprocessor with the build's own flags
`tu_built`	is the translation unit compiled in this configuration

Deliberately cheap -- symbol tables, relocations, the preprocessor, and the build description. No DWARF and no static call graph, because indirect calls through function pointers defeat call-graph extraction anyway, and the build-description check is both cheaper and more decisive.

crucible-patchcheck¶

flowchart TD
    A["1 · applies<br/><i>a diff, not a sketch</i>"] --> B["2 · compiles"]
    B --> C["3 · PoC differential<br/><i>faults pristine, rejected patched</i>"]
    C --> D["4 · corpus regression<br/><i>genuine artifacts still work</i>"]
    D --> E["5 · corpus reaches patch<br/><i>the corpus actually executed<br/>the patched lines</i>"]
    E --> F["6 · build attribution<br/><i>the patched file is built</i>"]
    F --> G["7 · sibling sweep<br/><i>other sites hitting the same sink</i>"]
    G --> H{"Operator reads<br/>the evidence"}

    style E fill:#204a2a,stroke:#5a8,color:#fff
    style H fill:#4a2020,stroke:#a55,color:#fff

Check 5 is the one worth having. Checks 3 and 4 can both pass on a patch that is still wrong: a regression corpus proves nothing if it never executes the lines that changed. "The shipped models still load" is a vacuous statement when every shipped model skips the patched branch. Check 5 sets a breakpoint on the patched lines and requires evidence they were hit -- and reports UNKNOWN, not FAIL, when the binary lacks the debug information needed to tell.

build-ledger¶

Counts are generated from the tree, never typed by hand, and every exclusion is enumerated with a written reason. A hand-typed count in a report is treated as a defect: it is how a project ends up asserting several mutually contradictory numbers about its own work.

What this layer does not do¶

It does not decide whether to disclose. It does not rate severity. It does not conclude that anything is "by design" or "already known". Those judgements stay with the operator, and the tools exist to make sure the evidence in front of that judgement is real.