Alien Holographic Token Lattice (AHTL) & Structural Skeletonizer Specification¶
Applies to: icx v1.1.0
References: docs/ahtl/README.md, docs/ahtl/AHTL_v1_Complete_Technical_Specification_Volume.pdf, ref-docs/novel-methods/non-human-neural-codecs-and-holographic-context.md §1, ref-docs/research-report.md §4.2
Target Crate: crates/icx-codec
1. Purpose¶
crates/icx-codec transforms bloated human-readable text, source code, diffs, compiler outputs, and conversation turns into AHTL (Alien Holographic Token Lattice)—a non-human, BPE-aligned Dual-Profile symbolic representation (Profile A AhtlWirePacketV1 and Profile B AhtlPacket) intended to be read by the model rather than by people. A packet is a structural skeleton (identifiers, line ranges, error codes, failing tests, causal clauses) plus a dereference pointer (#icx:<16-hex> or #icx:<12-hex>) to the stored bytes. It is not a lossless encoding: the full text is recovered by dereferencing the pointer. The size reduction depends on the input; icx benchmark measures 214x on a repetitive 1,200-line test log and 159.8x on a varied log, both synthetic (docs/performance.md).
2. Formal AHTL v1.0 Dual-Profile Grammar (EBNF)¶
AhtlStream ::= EpochHeader ("\n" WirePacket)+ ;
EpochHeader ::= "⟪ICX:v1│ep=" Hex4to8 "│root=" ( Hex8 | Hex16 ) "│" EpochCount "⟫" ;
EpochCount ::= ( "τ=" | "nodes=" | "folded=" ) UInt32 ; (* epoch id is hexadecimal: ep=000a is 10 *)
WirePacket ::= ProfileAPacket | ProfileBPacket ;
ProfileAPacket ::= "⟦AHTL v1│" VirtualHandle "│" FractalCoord "│" HdcDigest
"│" BrailleSpectrum64 "│" SkeletonPayload "⟧" ;
ProfileBPacket ::= "⟦" PacketKind ":" Subject ( "⊢" ClauseList )? ( "↦" DerefHandle )? ( "⋮Ψ:" Hologram )? "⟧" ;
PacketKind ::= "Δ" (* Code/AST/File mutation or read *)
| "Ψ" (* Tool execution / diagnostic stub *)
| "Ω" (* Architectural invariant / hard rule *)
| "Θ" (* Cross-agent telepathy broadcast *)
| "Φ" (* Folded sub-trajectory branch commit *) ;
Subject ::= PathOrSymbol ( "#L" LineRange )? ;
ClauseList ::= Clause ( "⋮" Clause )* ;
Clause ::= OpPrefix? Identifier ( ":" ValueExpr )? ( "⇒" Clause )? ; (* at most 64 "⇒" links *)
OpPrefix ::= "⊕" (* Added / defined *)
| "⊖" (* Removed / invalidated *)
| "¬" (* Negative constraint / forbidden *)
| "!" (* High-surprise anomaly / error / panic *)
| "✓" (* Verified / test passed *) ;
DerefHandle ::= "#icx:" ( Hex16 | Hex12 ) ( "(" SliceMap ")" )? ;
SliceMap ::= "L" UInt "-" UInt | "B" UInt ":" UInt ;
Hologram ::= BrailleSketch ( "[" Hex4 "]" )? ;
BrailleSketch ::= { U+2800 .. U+28FF }+ ; (* 8 bits per Unicode Braille codepoint *)
2.1 Parser Bounds and Validation (v1.1.0)¶
- Epoch ids are
u32, written and parsed as 4 to 8 hexadecimal digits everywhere (seal, fold, retarget). v1.0.0 wrote some headers in decimal, so epoch 10 read back as 16 (ICX-AUD-024). - Slice hints must match
SliceMap(is_valid_slice_map). An invalidslice_hintis never rendered and is rejected when parsing, which prevents a hint from injecting packet structure into a stream (ICX-AUD-025). - Clause chains are parsed iteratively and materialize at most
MAX_CLAUSE_CHAIN = 64⇒links, so arbitrarily long chains cannot exhaust the stack (ICX-AUD-026). hv:digests (hv_digest_u64, Profile AHdcDigest) are a linear, non-cryptographic sketch of the hypervector. They are not an identity and must not be used for integrity (ICX-AUD-053).
3. Dual-Channel Encoding Mechanics¶
3.1 Channel A: Attention-Queryable Relational Skeleton¶
Standard BPE tokenizers (o200k_base, cl100k_base, Claude BPE) split punctuation-heavy JSON ({"file": "src/main.rs", "line": 42}) into 14+ tokens. AHTL replaces JSON structural keys with single-codepoint mathematical operators (⊢, ⇒, ↦, ⋮, ⊕, ⊖, ¬, !, ✓) and strips all whitespace between clauses.
- Crucial Invariant: Identifiers (
alloc_page,E0502,state_5.sqlite) are emitted verbatim without internal character mangling so that the model's token embeddings for those identifiers match user queries identically in attention space.
3.2 Channel B: Braille-256 Holographic Sketch (U+2800..U+28FF)¶
Each compressed Profile B packet appends a compact 8-character Braille-256 sketch (Ψ:⠒⠴⡖⡸⢚⢼⣞⣰) encoding the 64-bit ChannelBHologram metadata fingerprint (B[0..=3] 32-bit big-endian HDC XOR parity fold, B[4..=5] 16-bit big-endian quantized IFS quadrant signature, B[6] 8-bit quantized surprise \(S_t\) from the Titans-inspired gate, B[7] flags byte: bits 0..=1 retention tier (TIER_MASK), bit 2 FLAG_HAS_ERROR, bit 3 FLAG_HAS_PANIC, bit 4 FLAG_HAS_DIFF, bit 5 FLAG_IS_FOLDED, bit 6 FLAG_IS_EPOCH, bit 7 FLAG_IS_REDACTED), while Profile A carries a 64-character (512-bit) 1-Lipschitz HDC spectral band projection (Hypervector::spectral_bands_64, encode_braille_64 / decode_braille_64).
Encoding byte \(b \in [0, 255]\) to Unicode Braille (U+2800 + b) and its exact 3-byte UTF-8 triple ([0xE2, 0xA0 | (b >> 6), 0x80 | (b & 0x3F)]) is a branchless constant-time bijection (byte_to_braille, braille_to_byte, byte_to_braille_utf8, braille_utf8_to_byte).
4. Domain-Specific Structural Skeletonizers (StructuralSkeletonizer)¶
Before AHTL packet assembly, StructuralSkeletonizer extracts high-entropy structural skeletons from raw text in \(O(N)\) single-pass scans:
- Source Code Skeletonizer (
Rust,TypeScript/JavaScript,Python,Go,C/C++): - Extracts module/package headers (
mod@L<n>), folds contiguous header imports (imports:L<a>-<b>(N)including Goimport (...)blocks), and extractsfn/def/func/struct/enum/trait/impl/class/interface/typesignatures with quote-, Rust-lifetime- ('a), and comment-awareL<start>-<end>line spans, plusTODO/FIXME/SAFETYnotes. - Compiler & Test Log Skeletonizer (
cargo test/clippy,pytest,vitest/jest,tsc,go test): - Collapses \(N\) passing test lines into a single aggregate counter (
✓pass:N_tests). - Preserves 100% of failing test names (
!FAIL:<test_name>, deduplicated across summary blocks),error[E...],FAILED,panicked at, and panic locations (!PANIC@...). - Unified Diff Skeletonizer (
git diff): - Pairs adjacent
--- a/<path>/+++ b/<path>headers (file:<path>), extracts hunk headers (@@ -a,b +c,d @@), removed lines (⊖), and added lines (⊕), and stores the full patch in CAS.
5. Secret Redaction (SecretRedactor)¶
Per Module 60, secrets are removed before content enters the CAS, the HDC and bi-temporal indexes, or the telepathy ring. v1.1.0 separates two operations (ICX-AUD-021):
SecretRedactor::redact(storage). Removes secrets and preserves every other byte: line endings (including CRLF), trailing newlines and literal delimiters. For secret-free input the output equals the input, which is what makes CAS dereferencing lossless. Redacted artifacts setFLAG_IS_REDACTED(bit 7 ofB[7]).SecretRedactor::sanitize(model-facing text).redactfollowed by Tier-0 delimiter escaping. Used for packets, skeletons, tail slots and MCP/CLI deref output, never for stored bytes.
Tiers:
- Tier 0 (delimiter neutralization,
sanitizeonly): escapes literal⟪(<<),⟫(>>),⟦([[) and⟧(]]) so untrusted text cannot imitate an AHTL packet or epoch header. Text that is byte-identical to a blockicxemitted is recognized by digest, not by prefix (docs/security-and-threat-model.md §4.3). - Tier 1 (private-key blocks): multi-line
-----BEGIN ... PRIVATE KEY-----and PGP private key blocks; single-line (JSON-escaped) blocks are redacted inline; an unterminated block stops consuming after 200 lines (ICX-AUD-016). - Tier 2 (known formats and key names): vendor token prefixes (Anthropic, OpenAI, Stripe,
GitHub, GitLab, AWS, GCP, Hugging Face, NPM, PyPI, Slack bot/app tokens and webhooks, SendGrid,
DigitalOcean, Shopify, Twilio,
ai-memory), 3-segment JWTs, URL user-info passwords for every scheme, CLI secret flags (--api-key=,--token=,--password=,--secret=),BearerandBasicheaders, and.env/JSON/YAML assignments to sensitive key names (including multi-line JSON/YAML values after a sensitive key) (ICX-AUD-015). After a sensitive key, a value holding any quote is hidden from its first quote to the end of the line; only a call head right before that quote stays visible (v1.3.1, #29). A line that is one complete JSON object or array is redacted structurally instead: only the values of sensitive keys are replaced, and other strings are redacted with their key as context. - Tier 3 (entropy gate): unprefixed high-entropy Base64 tokens (\(|S| \ge 24\), \(H(S) > 4.35\) bits/char) and hex assignment values (\(|S| \ge 32\), \(H(S) > 3.50\) bits/char).
Redaction is pattern-based. Secrets in unrecognized formats can pass through, and high-entropy non-secrets can be removed; see docs/security-and-threat-model.md §5.
6. Test Plan¶
- Property tests (
proptest,crates/icx-codec/tests/properties.rs):redactis the identity on secret-free text and idempotent;sanitizeremoves every delimiter; packets built from hostile field text render to exactly one packet and round-trip; clause parsing is total. - Codec regressions (
crates/icx-codec/tests/codec_conformance.rs): hex epoch ids,SliceMapvalidation, bounded clause chains. - Redaction fixtures (
crates/icx-codec/tests/redact_recall_and_losslessness.rs): URL user-info for every scheme, additional vendor formats and key names, multi-line JSON/YAML values, single-line and unterminated PEM blocks, losslessness on secret-free input. - Compression tests (
crates/icx-codec/src/skeleton.rs):>= 25xon a synthetic passing test log and>= 15xon a synthetic pytest log, with failing test names, error codes and handles preserved. - Fuzzing (
fuzz/fuzz_targets/ahtl_stream.rs,redact.rs): the stream parser and the redactor must not panic on arbitrary input.