hemlock validate¶

Simulate text extraction by a RAG framework and check whether a hidden payload survives processing.

Synopsis¶

hemlock validate --file <path> [flags]
hemlock validate --dir <path> [flags]

Flags¶

Flag	Type	Default	Description
`--file`	`string`		Input file path to validate (required unless `--dir` is set)
`--dir`	`string`		Directory of files to validate against all frameworks
`--framework`	`string`	`generic`	RAG framework to simulate: `langchain`, `llamaindex`, `unstructured`, `haystack`, `generic`
`--payload`	`string`		Expected payload text to search for in extracted content
`--json`	`bool`	`false`	Output results as JSON

Description¶

validate processes a document through a simulated extraction pipeline that mimics how a specific RAG framework loads and chunks documents. It then searches the extracted text for the expected payload string.

The validation pipeline:

Detects the file format from the extension
Applies the framework-specific extraction logic (document loader simulation)
Searches the extracted text for --payload
Reports PASS if the payload is found, FAIL if it is not

What validation does not do

validate does not send documents to an actual RAG pipeline or LLM. It simulates the text extraction stage only. A PASS result means the payload text survives extraction and would be present in the context window---it does not guarantee the LLM will follow the injected instruction.

Framework Options¶

Framework	Simulates	Notes
`generic`	Basic text extraction	Default. No framework-specific handling.
`langchain`	LangChain document loaders	Simulates `UnstructuredHTMLLoader`, `Docx2txtLoader`, `PyPDFLoader`, etc.
`llamaindex`	LlamaIndex readers	Simulates `SimpleDirectoryReader` and format-specific readers.
`unstructured`	Unstructured.io partitioners	Simulates `partition_html`, `partition_docx`, `partition_pdf`, etc.
`haystack`	Haystack converters	Simulates `HTMLToDocument`, `DOCXToDocument`, `PDFMinerToDocument`, etc.

Unsupported format/framework combinations

Some format/framework combinations are not supported by the real framework (e.g., PNG with LangChain). When this occurs, validate reports Unsupported: true with confidence n/a instead of a false negative.

Different frameworks handle the same document differently. A payload that survives LangChain extraction may not survive Unstructured.io, and vice versa. Always validate against the framework your target uses.

Examples¶

Basic validation¶

hemlock validate \
  --file ./output/poisoned-css-hide-001.html \
  --payload "Ignore all previous instructions"

PASS resultFAIL result

[hemlock] Validating: ./output/poisoned-css-hide-001.html
[hemlock] Framework:  generic
[hemlock] Payload:    found in extracted text
[hemlock] Result:     PASS

[hemlock] Validating: ./output/poisoned-comment-001.html
[hemlock] Framework:  unstructured
[hemlock] Payload:    not found in extracted text
[hemlock] Result:     FAIL

Validate against a specific framework¶

hemlock validate \
  --file ./output/poisoned-fontzero-001.docx \
  --framework langchain \
  --payload "Ignore all previous instructions"

Validate without a payload check¶

When --payload is omitted, validate displays the extracted text (truncated to 500 characters) without performing a pass/fail check. This is useful for inspecting what the framework actually sees:

hemlock validate \
  --file ./output/poisoned-zero-width-001.txt \
  --framework llamaindex

Batch Validation¶

Combine validate with shell utilities to check an entire output directory:

for f in ./output/poisoned-*.html; do
  hemlock validate \
    --file "$f" \
    --framework langchain \
    --payload "Ignore all previous instructions"
done

Use the exit code in scripts

validate returns exit code 0 on PASS and exit code 2 on FAIL (payload not found), making it suitable for use in CI pipelines and shell conditionals:

if hemlock validate --file doc.html --framework langchain --payload "PWNED"; then
  echo "Payload survived extraction"
else
  echo "Payload was stripped"
fi

Tips¶

Match the payload text exactly

The --payload flag performs a substring match against the extracted text. If you used a preset payload (e.g., --payload override in craft), you need to provide the actual injection text that hemlock embedded---not the preset name.

Run hemlock list-payloads to see the exact text for each preset variant, or use validate without --payload to inspect the raw extracted content first.

Framework differences matter

Technique survival varies significantly across frameworks. For example:

HTML comments are stripped by most frameworks (low survival rate)
CSS-hidden divs often survive LangChain but are stripped by Unstructured.io
Zero-width characters in TXT files have high survival rates across all frameworks

Always validate against the framework your target actually uses.