--- wo: 1 title: Write RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md -- the parity-protected storage container programme: OPALFLAME-ARCHIVE-CONTAINER depends_on: [] blocks: [2, 3, 4, 5] --- # WO 1 -- the container RFC ## Doing Write `docs/RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md`: the normative definition of the parity-protected, optionally encrypted and signed archive container that carries a `.ofp3.tgz` (or any payload) through long-term storage. Sibling to `docs/RFC-OPALFLAME-FORMAT-STANDARD-V3.md`, and deliberately separate from it. That RFC defines the logical document; this one defines the storage container, which is indifferent to its payload and reusable for anything else worth archiving. Same standard as the v3 RFC: RFC 2119 keywords, complete enough that an independent implementer could build a conforming packer and unpacker from it alone. ## The pipeline this specifies ``` project/ -> project.tar uncompressed -> project.tar.sha256 plaintext hash, NEVER encrypted -> [encrypt project.tar] optional, BEFORE split -> split -b 1440K -> project.part.000 .. NNN -> project.part.sfv CRC32 over the parts (authoritative list) -> project.nfo release/packaging notes, extract / repair / install instructions, registration, dev contact, user guide, TS FAQ (recovery) -> rm project.tar, project.tar.enc originals removed -> par2 create -r10 over the ENTIRE remaining dir -> full_packaged_asset-.tar uncompressed outer container ``` ## Parameters -- SETTLED BY MEASUREMENT, do not re-derive These were measured on haven-4346 with par2cmdline 1.2.0 (probe: `bin/par2_probe.sh`) against real export sizes. Record them in the RFC with the evidence, and state that they are measured rather than assumed. **Chunk size: 1.44MB (`split -b 1440K`).** **Redundancy: `par2 create -r10`.** **One size tier.** Every real project export is under 100MB, so the earlier DVD/CD tier table is deleted, not carried forward. Measured corpus: | project | size | |---|---| | same_water ACT1 vols | 0.8-0.9 MB | | jade mask | 1.1-9.2 MB | | pgtl-imperial_shield / fifa_fraud | 11-12 MB | | fahu-template | 17 MB | | pgtl-fix_iran | 35 MB | | FAHU video packages (x4) | 39-50 MB | | baseline-v2 fixture | 48 MB | | pgtl-toll_booth | 57 MB | | pgtl-seahorse_syndicate | 75 MB (largest found) | Exports carry images, clips and audio -- NOT a final render -- which is why they stay small. **Why these values.** Measured on a 50MB archive at 1.44MB chunks (34 chunks), `-r10` produced a 6.4MB recovery set in ~1s and survived: | damage | outcome | |---|---| | 3 whole chunks deleted | recovered, payload sha256 exact | | 4 whole chunks deleted | failed ("need 36 more recovery blocks") | | 5 whole chunks deleted | failed ("need 95 more") | | scattered corruption across 12 chunks | recovered, payload sha256 exact | par2 real overhead is ~1.25x nominal: `-r5`=7.3%, `-r10`=12.9%, `-r15`=18.5%, `-r20`=23.7% of payload. The governing relationship, which the RFC MUST state so the numbers are re-derivable if sizes ever change: > Surviving `K` entirely-lost chunks requires redundancy exceeding > `K x (chunk_size / total_size) x 100` percent, plus headroom for par2 > overhead. Sitting *at* that ratio FAILS -- measured: a chunk sized at exactly > 10% of the archive was unrecoverable at `-r10`, short by 9 blocks. At 1.44MB each chunk is ~2.9% of a 50MB archive, which is why `-r10` clears 3 whole chunks instead of zero. This is the entire reason the chunk size is small. ## Required contents 1. **Scope, terminology, conformance.** What makes a container, a packer and an unpacker conforming. 2. **The pipeline**, step by step, normatively, in the order above. 3. **Uncompressed throughout, and why.** Payload is already-compressed media so compression buys almost nothing; and unrepairable damage in a compressed stream destroys everything downstream of it, where damage in a raw tar stays local and surrounding members remain salvageable by hand. Graceful degradation is the design goal. The outer container is uncompressed too. 4. **`split`, not split-zip, and why.** Split-zip keeps the central directory in the final volume, so losing that one volume can make the set unopenable by standard tools; raw split output is sequential bytes that `cat` back together, and a missing middle chunk still leaves the rest walkable. 5. **Integrity layers, and what each is for.** State plainly that these are three different jobs, not redundancy: - `project.part.sfv` (CRC32) -- **triage**: which chunk moved, and the authoritative ordered file list for reassembly. CRC32 is correct here: the job is detection of media decay, not defence against an adversary, and par2 carries its own hashes underneath. - par2 -- **repair**. - `project.tar.sha256` -- **the integrity authority** for the payload. 6. **`project.tar.sha256` stays unencrypted, and there is NO ciphertext hash.** Keyless verification of the stored bytes is already provided by the SFV, with par2 restoring parts to that state when they fail. The plaintext hash has no subject until decryption produces one -- hash and data become available at the same moment, which is the soonest it could be checked. There is no gap to close. Record this reasoning so it is not "fixed" later. 7. **Encryption: BEFORE split, so parity covers ciphertext.** Never parity-then-encrypt: an AEAD tag fails over the whole file on a single flipped bit, and parity sealed inside the ciphertext can never be reached to repair the thing blocking access to it. Parity repairs the stored bytes first; decryption and authentication then run on data already whole. Name ONE cipher path (gpg and openssl are present on beacon; `age` is not) and specify key handling: no key, passphrase or recipient identity is ever persisted in the project, database, `.nfo`, or repo. 8. **Signing.** Sign `project.tar.sha256` (the inner content digest), so a recovered archive can prove it is authentic and not merely intact. Specify whether the outer container is also signed. 9. **`project.nfo`** -- required fields, populated from real data, never a static template: project name, export timestamp, studio version/commit, payload schema version, chunk size, part count, pinned tool names + exact versions + checksums, and the literal verify / repair / reassemble / decrypt commands. This is what a stranger reads in ten years; a placeholder in it is a defect. 10. **Reassembly is driven by the SFV.** The SFV is the authoritative ordered file list. Note the operational consequence: par2 renames each damaged file to a `.1` backup on repair, so **repair needs disk headroom beyond the package size** (one extra copy of every damaged chunk). Specify whether the tool purges them (`par2 -p`) or cleans up after a verified reassembly. 11. **Edge case: payload smaller than one chunk.** Projects under 1.44MB (real: same_water volumes, jade mask at 1.1MB) produce a single part. The split is a no-op and this MUST be valid, not an error. 12. **Error taxonomy** -- every refusal a conforming unpacker can raise, with stable identifiers. WO 3 consumes this. 13. **Recovery procedure**, written for a human with none of our tooling: the exact ordered commands to verify, repair, reassemble, decrypt and check. 14. **Worked example** -- a minimal complete container. 15. **Conformance checklist**, mechanically checkable. ## Hard constraints - **Use par2, do not implement Reed-Solomon.** par2cmdline is mature and specified; a bespoke codec is a large, subtle, security-relevant undertaking and would be exactly the second definition this fleet keeps paying for. - Every par2 claim in the RFC MUST come from the spec or from measurement, with the source named. Do not restate remembered defaults: par2cmdline 1.2.0 documents `-b` block count default **2000**, `-r` redundancy default **5%**, `-s` block size (mutually exclusive with `-b`), `-n` recovery files (max 31), `-R` recurse. par2 derives block size from block count and never inherits input file sizes; damage cost is per BLOCK, so several corrupt regions in one file repair independently at block cost. - The container is payload-agnostic. It MUST NOT reach into `.ofp3.tgz` internals or reference beat/scene semantics. ## Done means - `docs/RFC-OPALFLAME-ARCHIVE-CONTAINER-V1.md` written, all 15 sections. - Chunk size, redundancy and the single tier recorded WITH their measured evidence and the governing relationship. - Encryption ordering, the no-ciphertext-hash decision, and the SFV-as-list rule each recorded with their reasoning. - Error taxonomy enumerated (WO 3 depends on it). - `.nfo` required-field list complete (WO 5 depends on it). - No code written in this WO -- specification only. - Summary appended, ending with `complete` on its own last line, then shipped.