Skip to content

RTV Operator Checklist

This checklist keeps the DEF CON tactic aligned with the accepted benchmark format: attendees scan the estate, follow the guided Ray -> MLflow -> TGI credential chain, run score.py, then improve coverage across the remaining benchmark surfaces.

Week-1 Organizer Questions

Ask RTV organizers and record the answers before final range sizing:

  • Who hosts the tactic infrastructure?
  • Is Ludus available or expected? If yes, what version and resource limits?
  • If Ludus is not available, are AWS-hosted ranges acceptable?
  • Expected group count and maximum concurrent attendees.
  • CPU, RAM, disk, and network budget per group.
  • Whether target VMs may have outbound internet.
  • How attendees receive WireGuard configs.
  • Whether RTV provides WireGuard onboarding support.
  • Required reset SLA between waves.
  • Acceptable reset failure procedure.
  • On-site support contact during the tactic.
  • Whether a self-hosted fallback is allowed or expected.

Default decision rule: use Ludus if RTV provides or prefers it; use AWS/Terraform if cloud-hosted ranges are required; keep raw Proxmox/bash for rehearsal and emergency rebuilds.

Live-Wave Runbook

Before each wave:

  • One-command reset (preferred). From the Proxmox host, reset-wave.sh rolls back every VM to lab-ready, waits for boot, re-arms the disk-only-snapshot gaps (re-seed Ray jobs + MLflow + ChromaDB on ailab-ml, restart acme-mcp on ailab-dev), then runs both verifiers and prints PASS + elapsed per estate. Estates reset in parallel, so N estates fit roughly one estate's wall-clock.
bash lab-scripts/reset-wave.sh                    # base estate
bash lab-scripts/reset-wave.sh --groups "0 1 2"   # several estates in parallel (see Multi-Estate Standup)
bash lab-scripts/reset-wave.sh --dry-run          # preview the plan; touch nothing

Measured wall-clock vs. the 10-min SLA: validated on the live lab (2026-06-29) — a single-estate reset completed in 54s end-to-end (rollback → boot-wait → re-seed → verify), landing verify-lab 58/0/0 and verify-chain 13/13. That's well inside the 600s SLA; the re-seed is the long pole, and estates reset in parallel so N estates ≈ one estate's wall-clock. A parallel reset-wave --groups "0 1 2" reset and re-verified all three estates in 99s (capacity §4). Net: the demo is validated both ways — on a warm lab (170/170 coverage, verifiers green) and across a full redeploy (snapshot rollback → re-seed → re-verify), so a between-waves reset reliably re-lands the same green state.

  • Manual fallback (drive the steps by hand if the one-command path misbehaves):
    • Restore the snapshot: bash lab-scripts/lab-snapshots.sh --yes restore lab-ready (~2 min to boot).
    • Re-arm (required): the snapshot is disk-only, so Ray's in-memory seeded jobs (which carry the chain's looted creds) do NOT survive a restore. ssh labadmin@<ml> 'sudo bash ~/lab/ml-platform/seed.sh' (restores Ray jobs) and, if verify-lab flags MCP, ssh labadmin@<dev> 'sudo systemctl restart acme-mcp'. (A full re-arm is deploy-all.sh --skip-base.)
    • bash lab-scripts/verify-lab.sh (expect 58 passed / 0 failed; the MCP tools probe may warn even when mcp enum works — confirm with an actual aipostex mcp ... enum).
    • bash lab-scripts/verify-chain.sh (expect 13/13 — proves the re-arm restored the chain).
  • Confirm the attack box has aipostex, jq, curl, ssh, and lab-pivot.sh.
  • Confirm score.py default output is non-spoiler.
  • Confirm WireGuard access for each group.

During the wave:

  • Keep the guided chain flat and auth-gated; pivoting is advanced-only.
  • Use score.py as the attendee-facing benchmark report.
  • Use verify-chain.sh only as an operator diagnostic.
  • Do not paste exact manifest values unless deliberately switching to after-action mode.

Failure recovery:

  • Service down: restart the named systemd unit, then rerun the relevant verifier section.
  • Chain auth broken: rerun MLflow/Ray seed phase, then verify-chain.sh.
  • WireGuard failure: reissue the group config or move the group to a spare range.
  • Range drift: restore clean snapshot; if restore fails, rebuild from the golden image.
  • Demo failure: use the recorded backup demo and continue with scoring/take-home material.

Benchmark Scoring (operator)

To confirm the lab supports full coverage before a wave (the "answer-key" pass), run the coverage-complete collection from the attack box, then score it:

bash ~/lab/attack-box/collect-findings.sh ~/lab-results
python3 ~/lab/scoring/score.py ~/lab-results --strict --contracts --rrr-require-covered

collect-findings.sh wraps verify-aipostex.sh (preserving its per-command JSON via AIPOSTEX_KEEP_RESULTS) and adds the few surfaces it does not run by default (pgvector, tfserving model metadata, A2A seeded-task history, the fraud-detection W&B run-config secrets, and the on-host filesystem secret scans on dev + ml). It ships the binary to ailab-dev and ailab-ml so discover files can read /home/devuser and /opt/litellm.

Validated baseline (2026-06-28, full deploy): 100% coverage (170/170), service inventory 100%, 0 false positives, contracts 87 passed / 0 failed / 15 skipped, RRR 0 over-claims / 0 uncovered. --rrr-require-covered enforces the honesty matrix: GPU-bound inference fixtures (triton/tfserving/torchserve/bentoml) are capped at reachable (detection) — they return canned responses, so the tool claims detection, not execution.

Attendees do NOT run this — they explore interactively and score their own (smaller) output. Keep the default score.py report non-spoiler; --verbose/--json reveal answer-key detail for after-action review.

Advanced Layer

The segmented production TGI finale is optional. It must never block the benchmark path.

Required proof when enabled:

  • Direct production TGI access fails from the attack box.
  • lab-pivot.sh up creates the SOCKS endpoint.
  • aipostex reaches production TGI with --proxy socks5://....
  • Production TGI still rejects missing/wrong tokens.
  • The Layer-1 HF token succeeds after the pivot.