Skip to content

Scenario 09: ML Pipeline Run Injection

Difficulty: Advanced Time: ~30 minutes Prerequisites: Complete Scenario 08 Target: ailab-ml:9000 (Kubeflow)

Background

Kubeflow Pipelines is a platform for building and deploying ML workflows. Each pipeline defines a series of steps — data ingestion, preprocessing, training, evaluation, deployment. Pipeline runs accept parameters that control behavior. If an attacker can submit pipeline runs, they can inject malicious parameters, modify training data sources, or redirect model output to attacker-controlled storage.

Objective

Enumerate Kubeflow pipelines, extract credentials from pipeline parameters, and submit a pipeline run with modified parameters to demonstrate injection capability.

Commands

# Enumerate pipelines and their parameters
aipostex kubeflow --target 172.16.50.20:9000 enum

# Fingerprint to understand the pipeline structure
aipostex discover network --target 172.16.50.20:9000 --discovery-only

Expected Finding

Pipeline Enumeration: - acme-training-pipeline: Parameters include HF_TOKEN, SNOWFLAKE_CONN_STR - acme-data-ingest-pipeline: Parameters include AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

Existing Runs: - 3 completed runs with full parameter values visible - Run logs may contain additional credentials or internal URLs

Experiments: - production-training and staging-experiments experiments - Experiment metadata reveals team structure and workflow patterns

Notebooks: - 2 Kubeflow notebooks running in the kubeflow-user namespace - Notebooks have network access to other cluster services

Example finding: 

{
  "finding_type": "credential",
  "service": "kubeflow",
  "detail": "Pipeline parameter contains HuggingFace token",
  "pipeline": "acme-training-pipeline",
  "key": "HF_TOKEN",
  "value": "hf_kfp_prod_..."
}

Scoring objective: Extract all 4 credentials from Kubeflow pipeline parameters (HF_TOKEN, SNOWFLAKE_CONN_STR, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY).

Real-World Impact

Kubeflow pipelines are the automation backbone of enterprise ML. Compromising the pipeline system means an attacker can: - Poison training data by modifying data source parameters - Exfiltrate models by redirecting output storage - Harvest credentials that are passed as pipeline parameters - Backdoor models by injecting malicious steps into training pipelines

This is supply chain compromise at the ML workflow level.

Follow-On

  • Scenario 10: Use pipeline access to tamper with model artifacts
  • Scenario 12: Chain pipeline injection into a full campaign