Crash Triage¶

When the fuzzer finds a crash, the work is only half done. Crucible automates the triage pipeline — classifying crash types, deduplicating by stack trace, scoring severity, and generating actionable reports.

Triage Pipeline¶

flowchart LR
    A[Crash Files] --> B[Classify]
    B --> C[Deduplicate]
    C --> D[Map CWE]
    D --> E[Score]
    E --> F[Generate Reports]
    F --> G[Export SARIF]

1. Classify — Parse the sanitizer output to determine the crash type
2. Deduplicate — Hash the stack trace to group identical crashes
3. Map CWE — Assign a CWE identifier based on the vulnerability class
4. Score — Assign a CVSS score based on the vulnerability class
5. Generate Reports — Produce markdown reports with all details needed for CVE filing
6. Export SARIF — Optionally write SARIF 2.1.0 output for CI/security tooling integration

Crash Classification¶

Crucible parses AddressSanitizer (ASAN), MemorySanitizer (MSAN), and UndefinedBehaviorSanitizer (UBSAN) output to classify each crash into a known vulnerability type:

Severity Rationale¶

Stack Hash Deduplication¶

A single bug often produces thousands of crash files as the fuzzer continues running. Crucible deduplicates by hashing the crash's stack trace.

How It Works¶

1. Parse the ASAN/MSAN/UBSAN output from stderr
2. Extract the top 5 stack frames (function name, source file, line number)
3. Normalize by removing memory addresses, build paths, and frame numbers
4. Hash the normalized frames with SHA-256
5. Group crashes with identical hashes as the same unique bug

Raw ASAN frame:
    #0 0x55a3c2f1e4b7 in gguf_init_from_file /src/llama.cpp/ggml/src/gguf.c:387:21

Normalized:
    gguf_init_from_file|gguf.c|387

Final hash input (top 5 frames joined):
    gguf_init_from_file|gguf.c|387
    ggml_tensor_overhead|ggml.c|2104
    ggml_new_tensor_impl|ggml.c|2285
    ggml_new_tensor|ggml.c|2342
    llama_model_load|llama.cpp:4517

Dedup in Practice¶

crashes/
├── heap_overflow_a1b2c3d4/
│   ├── report.md           ← generated report
│   ├── first.gguf          ← first reproducer found
│   ├── 002.gguf            ← additional reproducers (same hash)
│   └── 003.gguf
├── integer_overflow_e5f6a7b8/
│   ├── report.md
│   └── first.gguf
└── null_deref_c9d0e1f2/
    ├── report.md
    └── first.gguf

Each unique stack hash gets its own directory. The first crash file is preserved as the primary reproducer. Subsequent duplicates are stored but not re-triaged.

Bulk Crash Deduplication¶

After a long campaign, crash directories can accumulate thousands of duplicate files. The crucible triage dedup command deduplicates an entire directory in one pass:

crucible triage dedup \
  --crashes ./crashes \
  --harness ./crucible-libfuzzer \
  --delete

crucible triage dedup \
  --crashes ./crashes \
  --fast \
  --delete

Both modes default to dry-run (report only). Pass --delete to actually remove duplicates.

See the CLI reference for all flags.

Automatic CVSS Scoring¶

Each crash type maps to a base CVSS v3.1 vector string:

var cvssVectors = map[CrashType]string{
    HeapOverflow:      "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",  // (1)!
    GlobalOverflow:    "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",  // (2)!
    UseAfterFree:      "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",  // (3)!
    DoubleFree:        "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",  // (4)!
    IntegerOverflow:   "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H",  // (5)!
    StackOverflow:     "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H",  // (6)!
    DivByZero:         "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H",  // (7)!
    AllocTooBig:       "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H",  // (8)!
    NullDeref:         "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (9)!
    AssertionFailure:  "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (10)!
    UseAfterPoison:    "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (11)!
    Timeout:           "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (12)!
    OOM:               "CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (13)!
    EnumLoad:          "CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (14)!
    MisalignedAccess:  "CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:N/I:N/A:L",  // (15)!
    Unknown:           "",                                                 // (16)!
}

1. 9.8 Critical — heap corruption enables arbitrary code execution
2. 9.8 Critical — global buffer overflow enables arbitrary code execution
3. 9.8 Critical — use-after-free enables arbitrary code execution
4. 9.8 Critical — double-free enables arbitrary code execution
5. 8.8 High — integer overflow often leads to heap corruption
6. 7.5 High — stack smashing can enable code execution
7. 7.5 High — hardware divide-by-zero causes unconditional crash (x86_64)
8. 7.5 High — oversized allocation causes denial of service
9. 5.3 Medium — null dereference causes denial of service
10. 5.3 Medium — assertion failure causes denial of service
11. 5.3 Medium — use-after-poison indicates memory safety violation
12. 5.3 Medium — execution timeout indicates algorithmic complexity issue
13. 5.3 Medium — out-of-memory indicates unbounded allocation
14. 3.3 Low — undefined behavior from invalid enum value load
15. 3.3 Low — misaligned memory access (crashes on strict-alignment architectures)
16. 0.0 Unknown — unrecognized crash signal

The attack vector is Network with User Interaction Required because the typical attack scenario is: attacker publishes a malicious GGUF model, victim downloads and loads it.

CWE Classification¶

Every crash type maps to a CWE (Common Weakness Enumeration) identifier. The CWE is included in generated reports and SARIF output.

The CWEForType() function in pkg/triage performs this mapping. See the API reference for details.

Report Generation¶

For each unique crash, Crucible generates a structured markdown report containing everything needed to file a CVE or security advisory:

# Crash Report: HeapOverflow in gguf_init_from_file

| Field | Value |
|-------|-------|
| **Type** | Heap Buffer Overflow |
| **CWE** | CWE-122 — Heap-based Buffer Overflow |
| **Location** | gguf.c:387 |
| **Function** | gguf_init_from_file |
| **CVSS Score** | 9.8 Critical |
| **CVSS Vector** | CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H |
| **Stack Hash** | a1b2c3d4e5f6a7b8 |
| **Target** | gguf |
| **Harness** | ./harness/libfuzzer/crucible-libfuzzer |
| **Minimized** | reports/minimized/crash-a1b2c3d4 |
| **Found** | 2026-04-01T14:23:07Z |
| **Seed** | 8827361950234 |
| **Crucible Version** | 0.1.0 |

## Reproducer

    ./crucible-libfuzzer crashes/heap_overflow_a1b2c3d4/first.gguf

## ASAN Output

    ==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address ...
    READ of size 4 at 0x... thread T0
        #0 gguf_init_from_file gguf.c:387
        #1 ggml_tensor_overhead ggml.c:2104
        ...

## CVE Template

**Title:** Heap buffer overflow in llama.cpp GGUF metadata parsing

**Description:** A heap buffer overflow exists in the GGUF file parser
of llama.cpp. A specially crafted .gguf file can trigger an out-of-bounds
read/write in gguf_init_from_file when parsing metadata values. An attacker
could exploit this by distributing a malicious model file.

**Affected:** llama.cpp (version), Ollama, LM Studio, and any application
using the ggml GGUF parser.

SARIF Export¶

Crucible can export triage results in SARIF 2.1.0 (Static Analysis Results Interchange Format), a standard consumed by GitHub Code Scanning, VS Code SARIF Viewer, and other security tools.

crucible-triage --crashes ./crashes --output ./reports --sarif ./results.sarif

The SARIF file contains:

• Tool metadata — Crucible name and version as the analysis tool
• Rules — One rule per crash type (e.g., heap-buffer-overflow), with full description and CWE reference
• Results — One result per unique crash, with:
  • Source location (file path, line number)
  • Severity level (error, warning, note) mapped from CVSS score
  • Fingerprints: crucible/stackHash/v1 (stack hash) and crucible/target (target surface) for cross-run deduplication
  • CWE reference for the crash type
• Rule tags — Each rule includes security, CWE ID, and target/<surface> tags

flowchart LR
    A[Crash Reports] --> B[WriteSARIF]
    B --> C[results.sarif]
    C --> D[GitHub Code Scanning]
    C --> E[VS Code SARIF Viewer]
    C --> F[Other SARIF Consumers]

Integration with the Fuzzer¶

Crash triage runs automatically during fuzzing. When the target process exits with a non-zero status and produces sanitizer output:

flowchart TD
    A[Target Process Exits] --> B{Exit Code != 0?}
    B -->|No| C[Continue Fuzzing]
    B -->|Yes| D{Sanitizer Output?}
    D -->|No| E[Log as Timeout/Unknown]
    D -->|Yes| F[Parse Crash Type]
    F --> G[Compute Stack Hash]
    G --> H{Hash Seen Before?}
    H -->|Yes| I[Store as Duplicate]
    H -->|No| J[Create Report Directory]
    J --> K[Generate Report]
    K --> L[Log New Unique Crash]
    L --> C
    I --> C

The fuzzer tracks unique crash count in real time. The summary printed at the end of a run includes:

• Total crashes found
• Unique crashes (by stack hash)
• Breakdown by crash type and severity
• Paths to all generated reports