Skip to content

Scoring Manifest

lab-scripts/scoring/manifest.json is the lab’s answer key. It now reflects the expanded topology and the Tier 2 verification contracts.

What Changed (v0.4.0)

  • W&B mock added on ailab-ml (port 8444) — 5 REST endpoints, 8 planted credential findings
  • A2A fixtures expanded — 6 seeded tasks (was 4): added credential-replay and tool-inject tasks
  • 170 total sensitive findings in the current manifest
  • 101 contract expectations
  • 55 verify-lab checks on the full validation path
  • score.py updated with wandb and a2a source→module mappings

What Changed (v0.3.0)

  • A2A fixtures added on ailab-app (ports 8100, 8101, 8102)
  • 3 agents × 5 skills each + expanded agent card schema
  • 4 seeded multi-turn task histories with planted credentials (~15 new findings)
  • 138 total sensitive findings (was 122 + 16 new A2A/internal-admin)
  • Lab listener added on ailab-attack (port 9000) — callback/exfil sink
  • Post-Ex Oracle added on ailab-app (port 8765) — credential replay + sentinel/heartbeat
  • Lateral movement target added on ailab-dev (127.0.0.1:9999) + token breadcrumb
  • 2 new A2A templates: push-notification-validated, multiturn-history-leak
  • Walkthrough expanded with Acts 7–10 covering credential replay, persistence validation, execution oracle, and lateral movement

Invariants

  • 170 planted sensitive findings tracked by manifest.json
  • A2A agents and POX are scoring targets — aipostex should find A2A findings via templates
  • The lab listener and POX are validation infrastructure — not scored as exploit targets
  • The manifest still acts as the strict-mode answer key for host attribution, service attribution, and workflow metadata expectations

Host Coverage

Host Role
ailab-dev developer workstation
ailab-ml ML platform
ailab-ds data science
ailab-app shared AI apps

The attack box is not scored as a target host, but it does participate in verification through local MCP fixtures and the stdio MCP server.