Skip to content

01 — AHTL Specification Conformance (PDF vs Markdown vs Code)

Workstream W1. Sources compared: docs/ahtl/AHTL_v1_Complete_Technical_Specification_Volume.pdf (58 pages), docs/ahtl/README.md and 01–08, docs/alien-holographic-codec.md, ref-docs/ahtl-exactness-review-and-analysis.md, and crates/icx-codec/src/{ahtl,braille,redact}.rs at c176d03.

1. Provenance of the PDF volume

Fact Evidence
The PDF is generated by docs/ahtl/build_ahtl_volume_pdf.py from README.md + 01–08 (Python-Markdown + WeasyPrint). build_ahtl_volume_pdf.py:17-27
The cover and table of contents are hard-coded HTML, not generated from Markdown, and carry their own claims ("3-tier SecretRedactor", "≥94% KV hit", "12 ns VPOPCNTDQ", "197.85 µs"). build_ahtl_volume_pdf.py:1099-1246
The PDF was last rebuilt in 41c451b (v0.3.0). 08-v1-reference-implementation-blueprint.md changed after that, in c176d03 (v1.0.0). git log -- docs/ahtl/*.pdf, git log -1 -- docs/ahtl/08*.md
PDF Chapter 08 (pp. 53–56) still says v0.3.0 and shows the old struct (handle_hex, splitn(5, '│')), while the cover says v1.0.0. pdftotext lines 2903, 3542, 3625, 3642, 3695

Chapters 01–07 of the PDF match the current Markdown. Spot checks covered the EBNF, the 320-byte Profile A budget, the %23L comment, and both test vectors.

ICX-AUD-030 — The normative PDF is stale and self-contradictory about its version (Medium)

  • Claim: "AHTL v1.0 Complete Technical Specification Volume" (cover). to-dos/ROADMAP.md Phase 2 says the code was cross-referenced against it.
  • Reality: Chapter 08 documents the v0.3.0 data model, not the shipped one. The cover's hard-coded claims are not regenerated from the chapters.
  • Impact: Any reader treating the PDF as the specification implements the wrong field set and parser arity.
  • Remediation: Generate the cover and TOC claims from the Markdown, not inline HTML. Make the PDF a build artifact checked in CI: rebuild it and fail if git diff --exit-code is non-empty. Alternatively, stop committing the 19 MB PDF and publish it as a release asset.

2. Normative clause matrix

Legend: OK = code matches spec. DIFF = mismatch. UNTESTED = no test pins the clause.

# Spec clause (ch01 / PDF p.) Code Status
G1 Delimiters ⟦⟧⟪⟫│⊢⋮↦⇒ with fixed code points (ch01 §3, p.11) ahtl.rs render/parse OK
G2 EpochHeader = "⟪ICX:v1│ep=" Hex4 … (p.9) parser ahtl.rs:812-835 reads hex; producer dag.rs:736 writes {epoch_index:04} decimal DIFF (ICX-AUD-024)
G3 Epoch count kinds nodes= / τ= parser also accepts folded= (not in EBNF) DIFF (doc)
G4 HexDigit lowercase only parser accepts uppercase and normalizes DIFF (lenient)
G5 SafeChar forbids ⟦⟧⟪⟫│⊢⋮↦\n\r in subjects subject escaping covers % #L ⟦ ⟧ ⊢ ⋮ ↦; │ ⟪ ⟫ \n \r are not escaped (removed by sanitize_hook_label upstream instead, mmu.rs:966-972) PARTIAL
G6 Clause escaping ("bijective clause escaping", CHANGELOG v1.0.0) escapes % : ⋮ ⇒ ↦ ⟦ ⟧ \n \r; not ⊢ │ ⟪ ⟫ PARTIAL; EBNF in ch01 documents none of it
G7 Profile A #icx:<hex>(SliceMap) where SliceMap = "L" UInt "-" UInt \| "B" UInt ":" UInt slice_hint is free text, rendered unescaped (ahtl.rs:449-454, 636-641) DIFF (ICX-AUD-025)
G8 Clause [Op] Key [":" Value] ["⇒" Clause] recursive parse, unbounded depth (ahtl.rs:133-135) DIFF (ICX-AUD-026)
G9 Braille φ(b) = U+2800 + b, 8-glyph Profile B sketch, 64-glyph Profile A braille.rs OK, round-trip tested
G10 Handle widths 12 or 16 hex lib.rs:84-105 OK; but docs use 8-hex examples (architecture.md:69-73, agent-adapters-and-interception.md:88,107, deployment guide :656-661), which the grammar and parser reject
G11 Hologram byte B[7] = RetentionTier (ch02:108-113) codec doc and review say TIER_MASK bits 0–1 + FLAG_* bits 2–7 DIFF between documents
G12 Test vector 2 fc:4000c000… labelled Q15 (ch01:296) the values decode to exact ±1.0/±0.5 in Q1.14 (scale 16,384), not Q15 DIFF (doc)
G13 Profile A totals "336–580 B" (ch01) component minimum is 339 B; "880+ bits" omits the 56-bit header DIFF (arithmetic)
G14 Header productions STUB, TAIL_SLOT│conv=, EPOCH_FOLDED, HERMES_EPOCH, PROVENANCE, STATUS, LCM_CHECKPOINT are emitted or documented but have no production DIFF (grammar incomplete)
G15 "Single BPE token per delimiter" (ch07) not verified for Anthropic tokenizers (only checkable via the token-counting API); see §5 UNVERIFIED

3. Reproduced defects

ICX-AUD-024 — Epoch headers are written in decimal and parsed as hex (Low, latent)

PoC (seal_epoch then EpochHeader::parse):

seal_epoch(10)    header = ⟪ICX:v1│ep=0010│root=39b25e0c│nodes=1⟫  -> parse epoch_id = 16
seal_epoch(10000) header = ⟪ICX:v1│ep=10000│root=cf709ef2│nodes=1⟫ -> Err(MalformedEpochHeader)
  • Status: no production code parses DAG-emitted headers yet (EpochHeader::parse has no callers outside ahtl.rs), so this is latent. It becomes live as soon as a consumer re-reads headers, and a model reading ep=0010 cannot tell the two conventions apart.
  • Fix: write {epoch_index:04x} in dag.rs:736, or change the grammar to decimal. Widen past 4 digits or document the wrap at 65,535.
  • Regression test: seal_epoch(n) → EpochHeader::parse must return n for n ∈ {0, 9, 10, 15, 16, 255, 65535}.

ICX-AUD-025 — Unescaped slice_hint allows stream-level packet forgery (Low, latent)

AhtlPacket::parse(render(p)) == p holds for adversarial hints, because the parser splits from the right. The pre-audit lead "not bijective" is therefore wrong at the packet level. At the stream level, AhtlStream::parse_stream splits on the first ⟧. A hint carrying …⟧\n⟦Ω:…⟧ therefore produces an extra, forged Invariant packet (⟦Ω:forged⊢admin:true…⟧).

  • Reachability: slice_hint is only populated by parse() and by tests. Exposure is limited to parse → re-render relays, which no current code path performs.
  • Fix: validate slice_hint against the SliceMap production at construction and in parse, or percent-escape it like clauses.
  • Regression test: property test asserting that parse_stream(render(p)) yields exactly one packet equal to p for arbitrary slice_hint: String.

ICX-AUD-026 — Unbounded recursion in AhtlClause::parse aborts the process (Low)

A clause chain of k⇒k⇒… parses at 100,000 links but aborts at 200,000 links, which is about 600 KB of input. The abort (fatal runtime error: stack overflow, aborting) happens on the 8 MB main stack and cannot be caught. icx decode-ahtl is protected only incidentally, by ARG_MAX (a 40k-link argument already fails with E2BIG). The library API, and any future stdin or MCP path, is exposed.

  • Fix: iterative parse with a maximum chain depth (for example 64), returning CodecError::ChainTooDeep.
  • Regression test: 1,000,000-link chain returns Err within a 2 MB thread stack.

4. Exactness-review cross-check (ref-docs/ahtl-exactness-review-and-analysis.md)

The review marks all seven "minimum exactness standards" as ENFORCED (§2.5, :188-196). Audit verdicts:

Standard Review verdict Audit verdict
Commit to CAS before skeletonizing ENFORCED Holds, but what is committed is the sanitized text, not the raw bytes (ICX-AUD-021)
Verify the full 256-bit digest ENFORCED Holds for deref_bytes; but mmu.rs:585-588 falls back to unchecked put_bytes on any error, including a collision (ICX-AUD-022)
Ambiguous short handles fail closed ENFORCED Holds (cas.rs:106-113)
Pinned round-trip tests ENFORCED Packet-level only; no stream-level, property or fuzz tests (ICX-AUD-039)
CAS durability / access control independent ENFORCED Not met: the workspace is a repo-relative directory with no ownership or trust check (see 05-security-findings.md, ICX-AUD-010)
Retrieval separated from authoritative recovery ENFORCED Partially; recall output and deref output carry no untrusted-content provenance (ICX-AUD-011)
Adversarial test coverage ENFORCED Not met (ICX-AUD-039)

The review also contradicts itself:

  • ICX-AUD-032 (Low): §2.2.5 (:95) says clauses are unescaped, while §2.5 (:193) says escaping is enforced.
  • Wrong figure: the 48-bit birthday figure (:82, "≈1.18×10⁷ (≈2^24.5)") is wrong. The 50% point is 1.1774·2^24 ≈ 1.98×10⁷; 2^24.5 is 2.37×10⁷.
  • Unresolved items under "ENFORCED": several items raised in §2.3.4 and §2.4 have no recorded resolution yet are covered by the blanket "ENFORCED": diff misrouting on : error TS, the brace scanner missing block comments, standalone 64-hex secrets, split tokens, FAIL@L functional keys, and the writer spin-drop.
  • Inconsistent line citations: the same function is cited at different lines in different sections (mmu.rs:342-357 vs 744-760, cas.rs:202-225 vs 256-282).

5. Items that could not be verified

  • BPE alignment: "each delimiter is a single BPE token" for Anthropic models (ch07) needs the Messages token-counting API with a key, so it was not run. Treat the claim as unverified.
  • Test vector 1: its BLAKE3 digest (4f3598d9…) was not recomputed independently. The codec's own test reproduces it, but that test is self-referential.

6. Recommendations

  1. Make the ch01 EBNF the single normative grammar. Generate parser test vectors from it, and add every emitted header kind (STUB, TAIL_SLOT, PROVENANCE, …) as a production.
  2. Fix all documentation examples to valid 12- or 16-hex handles, and add a doc test that extracts every ⟦…⟧ / ⟪…⟫ sample from docs/**/*.md and parses it.
  3. Replace "bijective" with a precise statement. Escaping is bijective for clause keys and values; Tier-0 delimiter replacement (⟪→<<) is lossy and non-injective (ICX-AUD-021).
  4. Add proptest round-trip properties for packet, stream, epoch-header and Braille layers, and a cargo fuzz target for AhtlStream::parse_stream and WirePacket::parse_auto.