01 — AHTL Specification Conformance (PDF vs Markdown vs Code)¶
Workstream W1. Sources compared: docs/ahtl/AHTL_v1_Complete_Technical_Specification_Volume.pdf
(58 pages), docs/ahtl/README.md and 01–08, docs/alien-holographic-codec.md,
ref-docs/ahtl-exactness-review-and-analysis.md, and crates/icx-codec/src/{ahtl,braille,redact}.rs
at c176d03.
1. Provenance of the PDF volume¶
| Fact | Evidence |
|---|---|
The PDF is generated by docs/ahtl/build_ahtl_volume_pdf.py from README.md + 01–08 (Python-Markdown + WeasyPrint). |
build_ahtl_volume_pdf.py:17-27 |
| The cover and table of contents are hard-coded HTML, not generated from Markdown, and carry their own claims ("3-tier SecretRedactor", "≥94% KV hit", "12 ns VPOPCNTDQ", "197.85 µs"). | build_ahtl_volume_pdf.py:1099-1246 |
The PDF was last rebuilt in 41c451b (v0.3.0). 08-v1-reference-implementation-blueprint.md changed after that, in c176d03 (v1.0.0). |
git log -- docs/ahtl/*.pdf, git log -1 -- docs/ahtl/08*.md |
PDF Chapter 08 (pp. 53–56) still says v0.3.0 and shows the old struct (handle_hex, splitn(5, '│')), while the cover says v1.0.0. |
pdftotext lines 2903, 3542, 3625, 3642, 3695 |
Chapters 01–07 of the PDF match the current Markdown. Spot checks covered the EBNF, the 320-byte
Profile A budget, the %23L comment, and both test vectors.
ICX-AUD-030 — The normative PDF is stale and self-contradictory about its version (Medium)¶
- Claim: "AHTL v1.0 Complete Technical Specification Volume" (cover).
to-dos/ROADMAP.mdPhase 2 says the code was cross-referenced against it. - Reality: Chapter 08 documents the v0.3.0 data model, not the shipped one. The cover's hard-coded claims are not regenerated from the chapters.
- Impact: Any reader treating the PDF as the specification implements the wrong field set and parser arity.
- Remediation: Generate the cover and TOC claims from the Markdown, not inline HTML. Make the
PDF a build artifact checked in CI: rebuild it and fail if
git diff --exit-codeis non-empty. Alternatively, stop committing the 19 MB PDF and publish it as a release asset.
2. Normative clause matrix¶
Legend: OK = code matches spec. DIFF = mismatch. UNTESTED = no test pins the clause.
| # | Spec clause (ch01 / PDF p.) | Code | Status |
|---|---|---|---|
| G1 | Delimiters ⟦⟧⟪⟫│⊢⋮↦⇒ with fixed code points (ch01 §3, p.11) |
ahtl.rs render/parse |
OK |
| G2 | EpochHeader = "⟪ICX:v1│ep=" Hex4 … (p.9) |
parser ahtl.rs:812-835 reads hex; producer dag.rs:736 writes {epoch_index:04} decimal |
DIFF (ICX-AUD-024) |
| G3 | Epoch count kinds nodes= / τ= |
parser also accepts folded= (not in EBNF) |
DIFF (doc) |
| G4 | HexDigit lowercase only |
parser accepts uppercase and normalizes | DIFF (lenient) |
| G5 | SafeChar forbids ⟦⟧⟪⟫│⊢⋮↦\n\r in subjects |
subject escaping covers % #L ⟦ ⟧ ⊢ ⋮ ↦; │ ⟪ ⟫ \n \r are not escaped (removed by sanitize_hook_label upstream instead, mmu.rs:966-972) |
PARTIAL |
| G6 | Clause escaping ("bijective clause escaping", CHANGELOG v1.0.0) | escapes % : ⋮ ⇒ ↦ ⟦ ⟧ \n \r; not ⊢ │ ⟪ ⟫ |
PARTIAL; EBNF in ch01 documents none of it |
| G7 | Profile A #icx:<hex>(SliceMap) where SliceMap = "L" UInt "-" UInt \| "B" UInt ":" UInt |
slice_hint is free text, rendered unescaped (ahtl.rs:449-454, 636-641) |
DIFF (ICX-AUD-025) |
| G8 | Clause [Op] Key [":" Value] ["⇒" Clause] |
recursive parse, unbounded depth (ahtl.rs:133-135) |
DIFF (ICX-AUD-026) |
| G9 | Braille φ(b) = U+2800 + b, 8-glyph Profile B sketch, 64-glyph Profile A | braille.rs |
OK, round-trip tested |
| G10 | Handle widths 12 or 16 hex | lib.rs:84-105 |
OK; but docs use 8-hex examples (architecture.md:69-73, agent-adapters-and-interception.md:88,107, deployment guide :656-661), which the grammar and parser reject |
| G11 | Hologram byte B[7] = RetentionTier (ch02:108-113) |
codec doc and review say TIER_MASK bits 0–1 + FLAG_* bits 2–7 |
DIFF between documents |
| G12 | Test vector 2 fc:4000c000… labelled Q15 (ch01:296) |
the values decode to exact ±1.0/±0.5 in Q1.14 (scale 16,384), not Q15 | DIFF (doc) |
| G13 | Profile A totals "336–580 B" (ch01) | component minimum is 339 B; "880+ bits" omits the 56-bit header | DIFF (arithmetic) |
| G14 | Header productions | STUB, TAIL_SLOT│conv=, EPOCH_FOLDED, HERMES_EPOCH, PROVENANCE, STATUS, LCM_CHECKPOINT are emitted or documented but have no production |
DIFF (grammar incomplete) |
| G15 | "Single BPE token per delimiter" (ch07) | not verified for Anthropic tokenizers (only checkable via the token-counting API); see §5 | UNVERIFIED |
3. Reproduced defects¶
ICX-AUD-024 — Epoch headers are written in decimal and parsed as hex (Low, latent)¶
PoC (seal_epoch then EpochHeader::parse):
seal_epoch(10) header = ⟪ICX:v1│ep=0010│root=39b25e0c│nodes=1⟫ -> parse epoch_id = 16
seal_epoch(10000) header = ⟪ICX:v1│ep=10000│root=cf709ef2│nodes=1⟫ -> Err(MalformedEpochHeader)
- Status: no production code parses DAG-emitted headers yet (
EpochHeader::parsehas no callers outsideahtl.rs), so this is latent. It becomes live as soon as a consumer re-reads headers, and a model readingep=0010cannot tell the two conventions apart. - Fix: write
{epoch_index:04x}indag.rs:736, or change the grammar to decimal. Widen past 4 digits or document the wrap at 65,535. - Regression test:
seal_epoch(n)→EpochHeader::parsemust returnnfor n ∈ {0, 9, 10, 15, 16, 255, 65535}.
ICX-AUD-025 — Unescaped slice_hint allows stream-level packet forgery (Low, latent)¶
AhtlPacket::parse(render(p)) == p holds for adversarial hints, because the parser splits from the
right. The pre-audit lead "not bijective" is therefore wrong at the packet level. At the
stream level, AhtlStream::parse_stream splits on the first ⟧. A hint carrying …⟧\n⟦Ω:…⟧
therefore produces an extra, forged Invariant packet (⟦Ω:forged⊢admin:true…⟧).
- Reachability:
slice_hintis only populated byparse()and by tests. Exposure is limited to parse → re-render relays, which no current code path performs. - Fix: validate
slice_hintagainst theSliceMapproduction at construction and in parse, or percent-escape it like clauses. - Regression test: property test asserting that
parse_stream(render(p))yields exactly one packet equal topfor arbitraryslice_hint: String.
ICX-AUD-026 — Unbounded recursion in AhtlClause::parse aborts the process (Low)¶
A clause chain of k⇒k⇒… parses at 100,000 links but aborts at 200,000 links, which is about
600 KB of input. The abort (fatal runtime error: stack overflow, aborting) happens on the 8 MB
main stack and cannot be caught. icx decode-ahtl is protected only incidentally, by ARG_MAX
(a 40k-link argument already fails with E2BIG). The library API, and any future stdin or MCP
path, is exposed.
- Fix: iterative parse with a maximum chain depth (for example 64), returning
CodecError::ChainTooDeep. - Regression test: 1,000,000-link chain returns
Errwithin a 2 MB thread stack.
4. Exactness-review cross-check (ref-docs/ahtl-exactness-review-and-analysis.md)¶
The review marks all seven "minimum exactness standards" as ENFORCED (§2.5, :188-196).
Audit verdicts:
| Standard | Review verdict | Audit verdict |
|---|---|---|
| Commit to CAS before skeletonizing | ENFORCED | Holds, but what is committed is the sanitized text, not the raw bytes (ICX-AUD-021) |
| Verify the full 256-bit digest | ENFORCED | Holds for deref_bytes; but mmu.rs:585-588 falls back to unchecked put_bytes on any error, including a collision (ICX-AUD-022) |
| Ambiguous short handles fail closed | ENFORCED | Holds (cas.rs:106-113) |
| Pinned round-trip tests | ENFORCED | Packet-level only; no stream-level, property or fuzz tests (ICX-AUD-039) |
| CAS durability / access control independent | ENFORCED | Not met: the workspace is a repo-relative directory with no ownership or trust check (see 05-security-findings.md, ICX-AUD-010) |
| Retrieval separated from authoritative recovery | ENFORCED | Partially; recall output and deref output carry no untrusted-content provenance (ICX-AUD-011) |
| Adversarial test coverage | ENFORCED | Not met (ICX-AUD-039) |
The review also contradicts itself:
- ICX-AUD-032 (Low): §2.2.5 (
:95) says clauses are unescaped, while §2.5 (:193) says escaping is enforced. - Wrong figure: the 48-bit birthday figure (
:82, "≈1.18×10⁷ (≈2^24.5)") is wrong. The 50% point is 1.1774·2^24 ≈ 1.98×10⁷; 2^24.5 is 2.37×10⁷. - Unresolved items under "ENFORCED": several items raised in §2.3.4 and §2.4 have no recorded
resolution yet are covered by the blanket "ENFORCED": diff misrouting on
: error TS, the brace scanner missing block comments, standalone 64-hex secrets, split tokens,FAIL@Lfunctional keys, and the writer spin-drop. - Inconsistent line citations: the same function is cited at different lines in different
sections (
mmu.rs:342-357vs744-760,cas.rs:202-225vs256-282).
5. Items that could not be verified¶
- BPE alignment: "each delimiter is a single BPE token" for Anthropic models (ch07) needs the Messages token-counting API with a key, so it was not run. Treat the claim as unverified.
- Test vector 1: its BLAKE3 digest (
4f3598d9…) was not recomputed independently. The codec's own test reproduces it, but that test is self-referential.
6. Recommendations¶
- Make the ch01 EBNF the single normative grammar. Generate parser test vectors from it, and add
every emitted header kind (
STUB,TAIL_SLOT,PROVENANCE, …) as a production. - Fix all documentation examples to valid 12- or 16-hex handles, and add a doc test that
extracts every
⟦…⟧/⟪…⟫sample fromdocs/**/*.mdand parses it. - Replace "bijective" with a precise statement. Escaping is bijective for clause keys and
values; Tier-0 delimiter replacement (
⟪→<<) is lossy and non-injective (ICX-AUD-021). - Add
proptestround-trip properties for packet, stream, epoch-header and Braille layers, and acargo fuzztarget forAhtlStream::parse_streamandWirePacket::parse_auto.