diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index 84a816de01..a1bba7b213 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -623,6 +623,13 @@ semantics are unsupported. `m68k` requires an even entry and decodes the 68040 superset of the 68000 family; Amiga/Atari containers and vector-table entry discovery are unsupported. Windows Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md). +Intel HEX, S-record, UF2 and DfuSe files are containers, not raw images: +`inspect_artifact` lists address regions (UF2 per familyID, DfuSe per element +with its alternate setting), and `extract_artifact` writes each region. Open an +extracted region as `raw-binary` with its start address as `entry_point`. UF2 +numbering gaps leave coverage unknown; a DfuSe suffix CRC mismatch follows +`integrity_policy`. See [UF2 and DfuSe](https://github.com/morluto/rea/blob/main/docs/uf2-dfuse.md). + ### Function annotations in Ghidra Fresh-import sessions admit annotations; existing-project reuse is read-only. diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index b1eb73651e..40409b89eb 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -17,6 +17,7 @@ notices supplied with the software you actually obtain or redistribute. | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | npm dependencies | [Complete inventory](licenses/npm-dependencies.txt) of every entry in `package-lock.json`, including transitive, optional platform, development, and website dependencies. [Runtime notices](licenses/npm-runtime-notices.txt) preserve the copyright, license, and NOTICE documents from integrity-verified release archives, plus the MIT notice for Rolldown's generated runtime helpers. Development dependencies are inventoried but are not all distributed in REA's npm package. | | pefile ordinal tables | Import-hash ordinal names are derived from [pefile 2024.8.26](https://github.com/erocarrera/pefile/tree/4b3b1e2e568a88d4f1897d694d684f23d9e270c4), copyright 2004–2024 Ero Carrera. Preserve the [MIT license](licenses/pefile-2024.8.26-LICENSE.txt). The Python package is not a runtime dependency. | +| UF2 family names | UF2 family IDs and short names are copied from [microsoft/uf2 `utils/uf2families.json`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json), copyright Microsoft Corporation. Preserve the [MIT license](licenses/microsoft-uf2-MIT.txt). | | NativeAOT parser | REA's derived parser uses the MIT-licensed [ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot/tree/effeb734fc570c32650f88b159608979dc7b423e) work of Washi and the NativeAOT layout definitions in [dotnet/runtime](https://github.com/dotnet/runtime/tree/a3b2d40328be0be3f8dd43f0e5cc925b8827b044). Preserve [Washi's MIT notice](licenses/nativeaot-parser-MIT.txt) and the [.NET Foundation MIT notice](licenses/dotnet-runtime-MIT.txt). | | EVM analysis | `evmole@0.9.3`, MIT, copyright 2023 Maxim Andreev. The published npm archive omits its license document; REA retains the [original upstream license](third_party/evmole/LICENSE) and [provenance](third_party/evmole/README.md). It is also included in the runtime notices. | | Windows native addon | The release addon is compiled with MinGW-w64 and statically links GCC runtime libraries. Preserve the [MinGW-w64 runtime notices](licenses/MinGW-w64-runtime.txt), [GPLv3 text](licenses/GPL-3.0.txt), and [GCC Runtime Library Exception 3.1](licenses/GCC-Runtime-Library-Exception-3.1.txt). The exception governs eligible compiled combinations; it does not relicense the runtime itself under MIT. The Node-API build headers are from Node 22.19.0; its [original license and component notices](licenses/Node-22.19.0-LICENSE.txt) are retained. Node itself is supplied by the user. Snapshot provenance and release-toolchain review requirements are in [licenses/README.md](licenses/README.md). | diff --git a/docs/cli.md b/docs/cli.md index bbb5c11983..251b7cd102 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -329,6 +329,16 @@ overlap and archive expansion checks. The native streaming decoder keeps the existing 128 MiB Zstandard window cap; frames requiring a larger window are refused. Deprecated method 20 and other unregistered ZIP codecs remain unsupported. +Inventory and extract UF2 or DfuSe regions: + +```sh +rea inspect-artifact ./firmware.uf2 --json +rea extract-artifact ./firmware.dfu --json +``` + +See [UF2 and DfuSe images](uf2-dfuse.md) for recognition, families, numbering, +the DfuSe suffix CRC and limits. + Inventory and extract U-Boot FIT images: ```sh diff --git a/docs/error-contract.schema.json b/docs/error-contract.schema.json index a5b4ccf324..cc7611087a 100644 --- a/docs/error-contract.schema.json +++ b/docs/error-contract.schema.json @@ -728,6 +728,32 @@ "unpacked" ], "additionalProperties": {} + }, + { + "type": "object", + "properties": { + "logical_path": { "type": "string" }, + "algorithm": { "type": "string", "const": "dfu-crc32" }, + "declared_crc32": { + "type": "string", + "pattern": "^[a-f0-9]{8}$" + }, + "calculated_crc32": { + "anyOf": [ + { "type": "string", "pattern": "^[a-f0-9]{8}$" }, + { "type": "null" } + ] + }, + "unpacked": { "type": "boolean" } + }, + "required": [ + "logical_path", + "algorithm", + "declared_crc32", + "calculated_crc32", + "unpacked" + ], + "additionalProperties": {} } ] } diff --git a/docs/firmware-analysis.md b/docs/firmware-analysis.md index 921886ba06..28d70d4f27 100644 --- a/docs/firmware-analysis.md +++ b/docs/firmware-analysis.md @@ -340,6 +340,15 @@ payloads stay as stored. Intel HEX/S-record and FIT images are separate readers and are not covered here. See [U-Boot legacy uImage](uimage.md) for component padding, preserved names, resource policies and verification scope. +## Inspect UF2 and DfuSe images + +Use `inspect-artifact` / `inspect_artifact` on a UF2 file for each familyID's +address regions and its block numbering, or on a DfuSe file for its targets, +elements and DFU suffix, then `extract-artifact` / `extract_artifact` for one +raw file per region or element. A DfuSe suffix CRC mismatch follows +`integrity_policy`. Decoding is portable and requires no external tool. See +[UF2 and DfuSe images](uf2-dfuse.md). + ## Inspect U-Boot FIT images Use `inspect-artifact` / `inspect_artifact` on a U-Boot FIT (`.itb`) for its diff --git a/docs/mcp-contracts.md b/docs/mcp-contracts.md index 046c46aaf0..4dbc5735e9 100644 --- a/docs/mcp-contracts.md +++ b/docs/mcp-contracts.md @@ -1317,12 +1317,20 @@ bytes and returns the `integrity_contradictions` records for the extracted files. An active ASAR unpacked entry whose companion file is absent cannot be materialized; extraction fails as `unavailable` with that logical path. +A container can also declare a checksum over its own bytes; today that is the +DFU suffix CRC of a DfuSe file. A mismatch is a contradiction of the root +occurrence (logical path `.`), recorded as a `dfu-crc32` pair. Under +`record-and-continue` the root becomes `mismatched` while its elements stay +listed and extractable as observed-untrusted bytes, and `extract_artifact` +returns the root record with the extracted files' records. + If native DMG child traversal is unavailable after carrier identification, the typed failure retains a revalidated root-only inventory under `details.partial_observation.inventory`, with unknown child coverage and the producer's diagnostics/provenance. It does not borrow or detach an existing attachment. Native checksum verification failures still refuse traversal; -`record-and-continue` recovery applies to structured entry contradictions. +`record-and-continue` recovery applies to structured entry and container +contradictions. Occurrences use normalized logical paths for identity and lookup. Optional `source_path` retains the original path within the owning reader when Unicode diff --git a/docs/testing.md b/docs/testing.md index dde7318eac..94e5abde6f 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2038,6 +2038,13 @@ tool before building and warns when esptool is not the pinned v5.6.0. `REA_XTENSA_ESP32_GCC` adds an ESP32 Xtensa build. See [ESP images](esp-image.md). +`npm run verify:uf2` needs only a Python that runs esptool, named by +`REA_ESPTOOL_PYTHON` (default `python3`). It writes inputs at known addresses, +builds UF2 files with esptool's `merge-bin --format uf2` (v4 `merge_bin`) with +and without MD5 and with 256-byte payloads, and concatenates two chips' +outputs. Inspection and extraction through the built CLI must return each input +at its address under the familyID esptool itself reports. It is not part of CI. +See [UF2 and DfuSe images](uf2-dfuse.md). `npm run verify:esp-flash` needs the same tools and an ESP-IDF checkout (`REA_ESP_IDF_PATH`) for `gen_esp32part.py` and `otatool.py`. It builds partition tables, assembles dumps with `esptool merge-bin`, writes OTA records diff --git a/docs/uf2-dfuse.md b/docs/uf2-dfuse.md new file mode 100644 index 0000000000..6eadfd3c43 --- /dev/null +++ b/docs/uf2-dfuse.md @@ -0,0 +1,224 @@ +# UF2 and DfuSe images + +REA inventories UF2 (Microsoft's USB Flashing Format) and DfuSe (ST's DFU +update container, `.dfu`) files through `inspect-artifact` / +`inspect_artifact`, and materializes their decoded bytes through +`extract-artifact` / `extract_artifact`. UF2 is the drag-and-drop format of the +RP2040 and RP2350 bootloaders, of TinyUF2 on ESP32-S2 and ESP32-S3, and of +`esptool merge-bin --format uf2`. Decoding is static and needs no external tool. + +```sh +rea inspect-artifact ./firmware.uf2 --json +rea inspect-artifact ./firmware.dfu --detail summary --evidence_output evidence.json --json +rea extract-artifact ./firmware.uf2 --json +``` + +For MCP, open the file with `open_binary`, then call `inspect_artifact`. Summary +delivery retains the complete Evidence; use `parent_evidence_id` with +`inspect_analysis_view` to read selected regions and fields. `extract_artifact` +returns a fresh `output_root` containing one raw file per region. + +## Recognition + +A file is UF2 only when its first 512 bytes are a complete block with all three +magic numbers: `0x0A324655` and `0x9E5D5157` at the start and `0x0AB16F30` at +the end. A file is DfuSe only when it begins with `DfuSe` and format version 1 +and ends with a 16-byte DFU 1.1a suffix: `bcdDFU` `0x011A`, the `UFD` signature +and `bLength` 16. The suffix CRC is not part of recognition. The file name does +not matter, and a file with only one of the DfuSe markers stays unrecognized. +The container formats are `uf2` and `dfuse`. + +## UF2 regions and metadata + +Each entry is one address-contiguous region of one family's main-flash +payloads, named `family-0x/0x-0x.bin` with an inclusive +end address, for example `family-0xe48bff56/0x10000000-0x1000ffff.bin`. Blocks +without the familyID flag (`0x2000`) go under `no-family/`. Families are +separate address spaces: a multi-family file, such as a Pico SDK image for both +RP2040 and RP2350, has a region per family even at the same addresses. Gaps are +never filled. + +Region `format_metadata` (`uf2-region`) records `family_id` (or `null`), +`family_name`, `start_address`, `end_address`, `length`, `block_count`, and +`first_file_block`/`last_file_block`: the zero-based positions in the file of +the contributing blocks, not their UF2 block numbers. `family_name` is REA's +label, not file content. It comes from a small table copied from Microsoft's +`uf2families.json` at commit +[`cf3bd7f9`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json), +which the root's `family_name_registry` names. The table covers the ESP32 +family, ESP8266, RP2040, the RP2350 and RP2XXX download families, STM32F4, +nRF52840, SAMD21 and SAMD51. It also covers the families that LibreTiny builds +for and the registry lists: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, +BK7231N, BK7251 and BL602. Other IDs, LibreTiny's LN882H among them, keep their +value with `family_name: null`. A familyID names a target family; it does not +establish the CPU state, memory map or entry point. + +Root `format_metadata` (`uf2`) records: + +| Field | Meaning | +| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `block_count`, `trailing_bytes` | Complete 512-byte blocks, and bytes after the last complete block | +| `invalid_block_count` | Blocks with a wrong magic number, a payload over 476 bytes, or a payload that wraps past 4 GiB | +| `flash_block_count` | Valid main-flash blocks, the only source of region bytes | +| `not_main_flash_block_count`, `file_container_block_count` | Blocks with flag `0x1`, and blocks with flag `0x1000` | +| `md5_block_count`, `extension_tag_block_count` | Blocks that declare an MD5 flash region (`0x4000`) or extension tags (`0x8000`) | +| `family_count`, `family_name_registry` | Distinct familyIDs among main-flash blocks, and the registry revision of `family_name` | +| `numbering_group_count` | Groups of valid blocks that share a familyID (or none) and a declared block count | +| `missing_block_count` | Block numbers below a group's block count that no valid block carries | +| `ambiguous_numbering_group_count` | Groups whose numbers appear at unequal counts of target addresses | +| `repeated_block_count`, `out_of_order_block_count` | Blocks whose number already appeared in their group, and blocks numbered at or below the previous block of their group | +| `block_number_out_of_range_count` | Blocks numbered at or above their block count, or with a block count of 0 | +| `region_count`, `data_bytes`, `duplicate_bytes` | Regions, unique decoded bytes, and bytes rewritten with identical values | + +Numbering is grouped by familyID because a bootloader ignores every field of +another family's block, including its numbering. Within a group, a block number +seen at several target addresses belongs to several copies of a sequence: +esptool numbers each merged input file from 0, so inputs with equal block +counts share a group, and each number then appears once per input. A block sent +twice to the same address is one block. A group is complete when every number +below its block count is present at the same count of addresses. If some numbers +appear at more addresses than others, a copy may have lost a block, so coverage +is unknown and a limitation names the group. + +## UF2 facts, limitations and failures + +| Situation | Outcome | +| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| Blocks out of address order | Sorted into regions | +| Blocks numbered at or below the previous block of their group | Counted in `out_of_order_block_count`; the spec lets blocks arrive in any order | +| An address written twice with the same value in one family | Accepted and counted in `duplicate_bytes` | +| An address written twice with different values in one family | Fails, naming the family, the address and the block spans of both writes | +| The same addresses in different families | Separate regions; never a conflict | +| Numbered blocks absent from a group | Regions are returned; coverage is unknown and a limitation names the groups | +| Reused numbers at unequal address counts | Regions are returned; coverage is unknown and a limitation names the groups | +| An invalid block | Excluded; coverage is unknown and a limitation names the first one and its offset | +| A partial final block | Excluded; coverage is unknown and the file may be truncated | +| Block numbers at or above the block count | Payloads are kept; coverage is unknown | +| Not-main-flash blocks (`0x1`) | Excluded from regions; their count and payload byte count are reported; coverage is unaffected | +| File-container blocks (`0x1000`) | Excluded from regions; their files are not reconstructed, so coverage is unknown | +| MD5 (`0x4000`) and extension-tag (`0x8000`) blocks | Counted and reported as a limitation; neither is decoded or verified | + +Unlike a malformed Intel HEX record, which fails the whole file, an invalid UF2 +block is excluded and reported: each block carries its own framing and address, +so its neighbours stay readable. + +The MD5 structure is not checked because it is not a checksum of the block's +bytes in the file. It names a region of device flash, usually aligned to erase +pages, whose MD5 a flashing tool compares to skip writing. Producers also +disagree on its place: the specification puts it in the last 24 bytes of the +data area, while esptool v4.12.0 and v5.6.0 write it right after a shorter final +payload, so those 24 bytes hold zeros or part of that structure. + +## DfuSe regions and metadata + +Each element is one entry at its own file range, named +`target-/element--0x-0x.bin` with zero-based target and +element indexes, for example `target-0/element-0-0x08000000-0x0800ffff.bin`. +Elements are never merged. Each occurrence's `source_location` is the byte range +of the element data in the file. + +Element `format_metadata` (`dfuse-element`) records `target_index`, +`alternate_setting`, `target_name`, `element_index`, `start_address`, +`end_address` and `length`. `target_name` is `null` when `bTargetNamed` is 0; +otherwise it is `szTargetName` up to its first NUL, one character per byte. +`compare_artifacts` treats an element's alternate setting and target name as +its declaration, so the same bytes moved to another device memory are +`changed` in `metadata`. + +Root `format_metadata` (`dfuse`) records: + +| Field | Meaning | +| ---------------------------------------------------- | ------------------------------------------------------------------------------- | +| `image_size`, `target_count` | `DFUImageSize` and `bTargets` from the prefix | +| `element_count`, `empty_element_count`, `data_bytes` | All elements, elements of 0 bytes (which have no entry), and element data bytes | +| `overlapping_element_count` | Elements that overlap an earlier element of the same alternate setting | +| `device_version`, `product_id`, `vendor_id` | `bcdDevice`, `idProduct` and `idVendor` from the suffix | +| `suffix_crc`, `computed_crc` | The stored `dwCRC`, and the same CRC computed over every byte before it | + +Recognition already requires `bVersion` 1 and `bcdDFU` `0x011A`, so they are +not repeated. The DFU suffix CRC is CRC-32 without the final inversion, as +dfu-util stores and checks it. + +Producers disagree on `DFUImageSize`. UM0391 calls it the file length, and +dfu-util's `dfuse-pack.py` writes that value since commit `ee6b393d` +(2020-10-08). Earlier `dfuse-pack.py` and MicroPython's `tools/dfu.py` +(checked at `cb99ca98`) write the length before the suffix, and an ST community +report describes ST's DfuFileMgr the same way; REA has not run DfuFileMgr. +dfu-util's own parser ignores the field. REA accepts either value and keeps the +declared one in `image_size`. + +## DfuSe facts, limitations and failures + +| Situation | Outcome | +| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| `dwCRC` differs from the computed CRC | Follows `integrity_policy`, below | +| A target without the `Target` signature | Fails with its byte offset | +| `DFUImageSize` that equals neither the file length nor the bytes before the suffix | Fails with the declared size, the suffix offset, the file length and whether the suffix CRC matches | +| A target size or an element size that disagrees with the bytes that hold it | Fails with the byte offset and the declared and available sizes | +| An element whose address range wraps past `0xFFFFFFFF` | Fails with its byte offset | +| Overlapping elements of one alternate setting | Kept as separate entries; a limitation names the first four; coverage is unaffected | +| The same addresses in different alternate settings | Separate entries; an alternate setting selects a device memory | +| Elements of 0 bytes, and targets with no elements | No entry; counted; a limitation names the first four, with address and alternate setting | + +A CRC mismatch is an integrity contradiction, never proof of a cause. With the +default `integrity_policy` `fail`, inspection and extraction fail with +`artifact_integrity_mismatch`, whose details carry `algorithm: "dfu-crc32"`, +`declared_crc32` and `calculated_crc32`. With `record-and-continue` +(CLI `--integrity-policy record-and-continue`), the root occurrence becomes +`mismatched`, `integrity_contradictions` records the pair for logical path `.`, +and extraction carries that contradiction, because every element came from the +contradicted container. The root occurrence has no `declared_integrity`: that +field holds entry digests, so the declared CRC appears in the contradiction and, +as a number, in `suffix_crc`. Under `fail`, the elements are hashed before the +mismatch is raised. + +## Limits + +Each file must be at most 128 MiB. The UF2 decoder reserves at most 476 bytes +per complete block before reading, merges payloads into regions of at most the +same total size, and keeps one numbering entry per valid block, so its peak is +about twice the decoded bytes. It retains at most 65,536 regions as entries; the +order of blocks does not count against that bound. The DfuSe decoder holds the +file once, serves elements as views of it, and retains at most 65,536 elements +with bytes as entries; empty elements are only counted. These are fixed +resource bounds, not format limits, and fail as limits (`file-size` and +`capacity`) with no runtime override. Configurable ceilings are tracked in +[#6176](https://github.com/morluto/rea/issues/6176). + +Decoded regions are cached only for the reader's lifetime, and each open checks +the admitted carrier identity again. These readers share the Intel HEX +reader's region streams, cancellation and close behavior. + +## Analyzing a region + +Inventory does not establish the CPU, byte order or memory map, or an entry +point. To disassemble a region, extract it and open the region file with +`open_binary` using format `raw-binary`, an architecture you choose, and +`entry_point` set to the region start address; raw-binary maps file offset 0 at +`entry_point`. Each region is opened separately, so references between regions +are not resolved. + +## Verification + +Fixtures written from the UF2 specification and from ST UM0391 cover block and +element layouts, numbering groups and reused numbers, families, rewrites and +conflicts, invalid and partial blocks, not-main-flash and file-container blocks, +structural faults, overlaps, both integrity policies, accepted boundaries beside +each refusal, resource refusals, inventory, extraction, comparison and reader +identity checks. The reader lifecycle (cancellation, close and stream +ownership) is covered by the Intel HEX reader tests, which exercise the shared +reader. Two DfuSe fixtures are byte-identical, CRC included, to what the +`build()` functions of dfu-util's `dfuse-pack.py` (at `f9a537c9`) and +MicroPython's `tools/dfu.py` (at `cb99ca98`) produce for the same image, one for +each `DFUImageSize` convention. Those functions were transcribed and run under +Python 3; the scripts themselves were not run. CLI and stdio MCP tests exercise +the same artifact provider and advertised schemas. + +`npm run verify:uf2` runs esptool's `merge-bin --format uf2` on generated +inputs at known addresses, with and without MD5, with 256-byte payloads, with +two inputs of equal block count, and on a concatenation of two chips' outputs. +Inspection and extraction must return every input at its address under +esptool's own familyID. It passed with esptool v4.12.0 and v5.6.0 on macOS. Set +`REA_ESPTOOL_PYTHON` to a Python that runs `-m esptool`. Pico SDK, picotool, +`uf2conv.py`, TinyUF2 and real DfuSe producers remain unverified, and no device +accepted these files in REA's tests. diff --git a/licenses/README.md b/licenses/README.md index fa698f449e..b3d1b7f184 100644 --- a/licenses/README.md +++ b/licenses/README.md @@ -9,6 +9,7 @@ upstream texts; they do not change [REA's MIT license](../LICENSE). | `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. | | `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. | | `protobuf-es-v2.16.0-LICENSE.txt` | Original Apache license omitted from the protobuf-es npm archive, preserved from its v2.16.0 source commit. Its embedded Google BSD notice is retained in `npm-runtime-notices.txt`. | +| `microsoft-uf2-MIT.txt` | Unchanged `LICENSE.txt` from microsoft/uf2 revision `cf3bd7f9c274b949528f8cd6a19eef2502a687d2`, for the UF2 family names copied from its `utils/uf2families.json`. | | `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. | | `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. | | `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. | diff --git a/licenses/microsoft-uf2-MIT.txt b/licenses/microsoft-uf2-MIT.txt new file mode 100644 index 0000000000..d3e3605f81 --- /dev/null +++ b/licenses/microsoft-uf2-MIT.txt @@ -0,0 +1,25 @@ +Microsoft UF2 + +The MIT License (MIT) + +Copyright (c) Microsoft Corporation + +All rights reserved. + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/licenses/upstream-sources.json b/licenses/upstream-sources.json index c42bb91821..ce4b8d626e 100644 --- a/licenses/upstream-sources.json +++ b/licenses/upstream-sources.json @@ -78,5 +78,10 @@ "file": "licenses/pefile-2024.8.26-LICENSE.txt", "source": "https://raw.githubusercontent.com/erocarrera/pefile/4b3b1e2e568a88d4f1897d694d684f23d9e270c4/LICENSE", "sha256": "b44409c067c6da52bfb54bb6624fb11a7b52157c3a13ae1400232f8196e86ad3" + }, + { + "file": "licenses/microsoft-uf2-MIT.txt", + "source": "https://raw.githubusercontent.com/microsoft/uf2/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/LICENSE.txt", + "sha256": "c581accf1e5449482b70ac5fcdfb171599317ccff559cf0794d771671486f800" } ] diff --git a/package.json b/package.json index 08d0c30856..c9b4e8aac5 100644 --- a/package.json +++ b/package.json @@ -239,6 +239,7 @@ "verify:javascript:retained-facts": "npm run build:cached && node scripts/verify/javascript/retained-facts.mjs", "verify:application-workflows": "npm run build:cached && node scripts/verify/javascript/application-workflows.mjs", "verify:managed": "npm run build:cached && node scripts/verify/managed/conformance.mjs", + "verify:uf2": "npm run build:cached && node scripts/verify/firmware/uf2-esptool.mjs", "verify:firmware": "npm run build:cached && node scripts/verify/firmware/analysis.mjs", "verify:devicetree": "npm run build:cached && node scripts/verify/firmware/devicetree.mjs", "verify:esp-images": "npm run build:cached && node scripts/verify/firmware/esp-image.mjs", diff --git a/scripts/verify/firmware/uf2-esptool.mjs b/scripts/verify/firmware/uf2-esptool.mjs new file mode 100644 index 0000000000..6467595c60 --- /dev/null +++ b/scripts/verify/firmware/uf2-esptool.mjs @@ -0,0 +1,284 @@ +#!/usr/bin/env node +import assert from "node:assert/strict"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { createVerifierCommands } from "../../lib/verify-commands.mjs"; +import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs"; + +// Optional verification tool only; the UF2 reader starts no tool. +const python = process.env.REA_ESPTOOL_PYTHON ?? "python3"; +const entrypoint = fileURLToPath(new URL("../../rea.mjs", import.meta.url)); +const options = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 }; +/** Every tool runs as an owned process tree that close() stops. */ +const commands = await createVerifierCommands(); + +let workspace; +const path = (name) => join(workspace, name); + +/** Deterministic input bytes; lengths include one that is not word-aligned. */ +const input = (length, seed) => + Buffer.from(Array.from({ length }, (_, index) => (index * 13 + seed) & 0xff)); + +// `otadata` matches the table's block count, so esptool's per-input numbering +// from 0 puts both inputs in one numbering group. +const FILES = { + bootloader: input(1000, 1), + table: input(3072, 2), + otadata: input(3072, 4), + app: input(501, 3), +}; + +const CASES = [ + { + name: "esp32s3.uf2", + chip: "esp32s3", + layout: [ + [0x0, "bootloader"], + [0x8000, "table"], + [0xd000, "otadata"], + [0x10000, "app"], + ], + extra: [], + md5: true, + payload: 452, + }, + { + name: "esp32s2-tinyuf2.uf2", + chip: "esp32s2", + layout: [ + [0x1000, "bootloader"], + [0x8000, "table"], + ], + extra: ["--chunk-size", "256", "--md5-disable"], + md5: false, + payload: 256, + }, +]; + +const tool = async (args) => { + const { stdout, stderr } = await commands.exec( + python, + ["-m", "esptool", ...args], + options, + ); + return `${stdout}${stderr}`; +}; + +/** esptool's own family ID for a chip: the oracle REA's table must match. */ +const familyOf = async (chip) => { + const { stdout } = await commands.exec( + python, + [ + "-c", + "import sys, esptool.targets as t; print(t.CHIP_DEFS[sys.argv[1]].UF2_FAMILY_ID)", + chip, + ], + options, + ); + return Number(stdout.trim()); +}; + +const rea = async (command, file) => { + const { stdout } = await commands.exec( + process.execPath, + [entrypoint, command, file, "--json"], + { + ...options, + env: { + ...process.env, + REA_LOG_LEVEL: "silent", + TMPDIR: workspace, + TMP: workspace, + TEMP: workspace, + }, + }, + ); + return JSON.parse(stdout).normalized_result; +}; + +const familyDirectory = (familyId) => + `family-0x${familyId.toString(16).padStart(8, "0")}`; +const hex32 = (value) => `0x${value.toString(16).padStart(8, "0")}`; +const blocksOf = (entry, name) => Math.ceil(FILES[name].length / entry.payload); +const inventoryOf = async (file) => + (await rea("inspect-artifact", path(file))).substeps[0].evidence + .normalized_result; +const rootOf = (inventory) => + inventory.occurrences.find((row) => row.logical_path === ".").format_metadata; + +/** Write the inputs, merge each case with esptool, and concatenate the cases. */ +const produce = async (v5) => { + for (const [name, bytes] of Object.entries(FILES)) + await writeFile(path(`${name}.bin`), bytes); + for (const entry of CASES) { + await tool([ + "--chip", + entry.chip, + v5 ? "merge-bin" : "merge_bin", + "--format", + "uf2", + ...entry.extra, + "-o", + path(entry.name), + ...entry.layout.flatMap(([address, name]) => [ + hex32(address), + path(`${name}.bin`), + ]), + ]); + entry.familyId = await familyOf(entry.chip); + } + // The UF2 specification makes a multi-family file by concatenation. + await writeFile( + path("concatenated.uf2"), + Buffer.concat( + await Promise.all(CASES.map((entry) => readFile(path(entry.name)))), + ), + ); +}; + +/** Root facts a complete file from these cases must report. */ +const expectedRoot = (included) => { + const blocks = included.flatMap((entry) => + entry.layout.map(([, name]) => ({ + group: `${entry.chip}:${String(blocksOf(entry, name))}`, + count: blocksOf(entry, name), + md5: entry.md5, + })), + ); + const total = blocks.reduce((sum, { count }) => sum + count, 0); + // Inputs of one chip with equal block counts form one numbering group, + // and every input after the first in a group repeats its numbers. + const groups = new Map(); + for (const { group, count } of blocks) + groups.set(group, [...(groups.get(group) ?? []), count]); + return { + block_count: total, + invalid_block_count: 0, + flash_block_count: total, + md5_block_count: blocks + .filter(({ md5 }) => md5) + .reduce((sum, { count }) => sum + count, 0), + family_count: included.length, + numbering_group_count: groups.size, + missing_block_count: 0, + ambiguous_numbering_group_count: 0, + repeated_block_count: [...groups.values()].reduce( + (sum, counts) => sum + (counts.length - 1) * (counts[0] ?? 0), + 0, + ), + duplicate_bytes: 0, + }; +}; + +/** Every input must come back at its address under esptool's familyID. */ +const check = async (file, included) => { + const inventory = await inventoryOf(file); + assert.equal(inventory.manifest.root_format, "uf2", file); + assert.equal(inventory.coverage.status, "complete", file); + const root = rootOf(inventory); + const expected = expectedRoot(included); + assert.deepEqual( + Object.fromEntries(Object.keys(expected).map((key) => [key, root[key]])), + expected, + file, + ); + const inputs = included.flatMap((entry) => + entry.layout.map(([address, name]) => ({ + path: `${familyDirectory(entry.familyId)}/${hex32(address)}-${hex32(address + FILES[name].length - 1)}.bin`, + familyId: entry.familyId, + bytes: FILES[name], + })), + ); + const regions = inventory.occurrences.filter( + (row) => row.format_metadata?.kind === "uf2-region", + ); + assert.deepEqual( + regions.map((row) => row.logical_path).sort(), + inputs.map((region) => region.path).sort(), + file, + ); + for (const region of regions) + assert.equal( + region.format_metadata.family_id, + inputs.find((row) => row.path === region.logical_path).familyId, + `${file} ${region.logical_path}`, + ); + const extracted = await rea("extract-artifact", path(file)); + assert.equal(extracted.containment_verified, true, file); + for (const region of inputs) + assert.equal( + Buffer.compare( + await readFile(join(extracted.output_root, region.path)), + region.bytes, + ), + 0, + `${file} ${region.path}`, + ); + return { + file, + families: included.map((entry) => ({ + chip: entry.chip, + family_id: hex32(entry.familyId), + family_name: regions.find( + (row) => row.format_metadata.family_id === entry.familyId, + ).format_metadata.family_name, + })), + blocks: root.block_count, + regions: regions.length, + md5_blocks: root.md5_block_count, + }; +}; + +/** + * Losing one block of one of two equal-count inputs must not read as + * complete: the other input still carries every block number. + */ +const checkLostBlock = async () => { + const [s3] = CASES; + const position = s3.layout.findIndex(([, name]) => name === "otadata"); + const lostIndex = + s3.layout + .slice(0, position) + .reduce((sum, [, name]) => sum + blocksOf(s3, name), 0) + 1; + const whole = await readFile(path(s3.name)); + await writeFile( + path("lost-block.uf2"), + Buffer.concat([ + whole.subarray(0, lostIndex * 512), + whole.subarray((lostIndex + 1) * 512), + ]), + ); + const inventory = await inventoryOf("lost-block.uf2"); + const root = rootOf(inventory); + assert.equal(inventory.coverage.status, "unknown", "lost-block.uf2"); + assert.equal(root.ambiguous_numbering_group_count, 1, "lost-block.uf2"); + assert.equal(root.missing_block_count, 0, "lost-block.uf2"); + return { + file: "lost-block.uf2", + removed_block: lostIndex, + coverage: inventory.coverage.status, + ambiguous_numbering_groups: root.ambiguous_numbering_group_count, + }; +}; + +await withVerifierCleanup(async () => { + const versionText = await tool(["version"]); + const version = /\bv?(\d+)\.(\d+)\.(\d+)\b/u.exec(versionText); + assert.ok(version, `Unrecognized esptool version: ${versionText}`); + workspace = await mkdtemp(join(tmpdir(), "rea-uf2-esptool-")); + await produce(Number(version[1]) >= 5); + const report = { esptool: version[0], cases: [] }; + for (const entry of CASES) + report.cases.push(await check(entry.name, [entry])); + report.cases.push(await check("concatenated.uf2", CASES)); + report.cases.push(await checkLostBlock()); + process.stdout.write(`${JSON.stringify(report, null, 2)}\n`); +}, [ + () => commands.close(), + () => + workspace === undefined + ? undefined + : rm(workspace, { recursive: true, force: true }), +]); diff --git a/src/application/BinaryTargetResolver.ts b/src/application/BinaryTargetResolver.ts index b544d3adb5..a8a4472327 100644 --- a/src/application/BinaryTargetResolver.ts +++ b/src/application/BinaryTargetResolver.ts @@ -86,6 +86,12 @@ import { hasGodotPckHeader } from "../domain/godotPckFormat.js"; import { hasUImageMagic } from "../domain/uimageFormat.js"; import { hasUnityFsHeader } from "../domain/unityfsFormat.js"; import { detectHexRecordEncoding } from "../domain/firmware/hexRecords.js"; +import { hasUf2Block } from "../domain/firmware/uf2.js"; +import { + DFUSE_SUFFIX_BYTES, + hasDfuSePrefix, + isDfuSeFile, +} from "../domain/firmware/dfuse.js"; import { hasFitImagesNode } from "../domain/firmware/fitImage.js"; import { locateEspFlashLayout, @@ -777,6 +783,22 @@ const detectArtifactFormat = async ( if (hasUImageMagic(magic)) return "uimage"; const hexRecords = detectHexRecordEncoding(magic); if (hexRecords !== undefined) return hexRecords; + if (hasUf2Block(magic)) return "uf2"; + if ( + hasDfuSePrefix(magic) && + initial.size >= DFUSE_SUFFIX_BYTES && + isDfuSeFile( + magic, + await readHandleRange( + handle, + initial.size - DFUSE_SUFFIX_BYTES, + DFUSE_SUFFIX_BYTES, + signal, + ), + initial.size, + ) + ) + return "dfuse"; // A C3 or S3 dump begins with its bootloader, so a layout comes first. const flash = await locateEspFlashLayout( initial.size, @@ -870,6 +892,8 @@ const isArchiveFormat = ( "intel-hex", "srec", "esp-image", + "uf2", + "dfuse", "esp-flash-image", "fit", ].includes(format); diff --git a/src/application/binary/TargetFormatVocabularies.test.ts b/src/application/binary/TargetFormatVocabularies.test.ts index cf35a6d8e6..8aef2e212f 100644 --- a/src/application/binary/TargetFormatVocabularies.test.ts +++ b/src/application/binary/TargetFormatVocabularies.test.ts @@ -121,6 +121,8 @@ const TARGETS: TargetFixtures = { "godot-pck": { ...archive, format: "godot-pck" }, unityfs: { ...archive, format: "unityfs" }, uimage: { ...archive, format: "uimage" }, + uf2: { ...archive, format: "uf2" }, + dfuse: { ...archive, format: "dfuse" }, dmg: { ...archive, format: "dmg" }, pkg: { ...archive, format: "pkg" }, "intel-hex": { ...archive, format: "intel-hex" }, diff --git a/src/artifacts/ArtifactClassificationProbe.ts b/src/artifacts/ArtifactClassificationProbe.ts index 65626988e7..582e221315 100644 --- a/src/artifacts/ArtifactClassificationProbe.ts +++ b/src/artifacts/ArtifactClassificationProbe.ts @@ -19,13 +19,18 @@ import { observeMzWindowsHeader, type MzWindowsHeaderSignatureEvidence, } from "../domain/dosMz.js"; +import { DFUSE_SUFFIX_BYTES } from "../domain/firmware/dfuse.js"; /** Initial bytes required by the other bounded format classifiers. */ export const ARTIFACT_CLASSIFICATION_PREFIX_BYTES = 8_192; +/** Final bytes kept for formats identified by a trailer: the DFU suffix. */ +const ARTIFACT_CLASSIFICATION_SUFFIX_BYTES = DFUSE_SUFFIX_BYTES; /** Retained prefix plus bounded format-specific observations from the stream. */ export interface ArtifactClassificationEvidence { readonly prefix: Buffer; + /** The last bytes of the stream, when the producer observed its end. */ + readonly suffix?: Buffer; readonly asarHeader: FinalAsarHeaderProbeObservation; readonly fatSliceHeaders?: FatSliceHeaderEvidence; readonly mzWindowsHeaderSignature?: MzWindowsHeaderSignatureEvidence; @@ -76,6 +81,7 @@ const observeFatSliceHeaders = (input: { const finishClassificationEvidence = ( prefix: Buffer, + suffix: Buffer | undefined, archiveBytes: number, finishAsar: (archiveBytes: number) => FinalAsarHeaderProbeObservation, sliceHeadersByOffset: Map, @@ -106,6 +112,7 @@ const finishClassificationEvidence = ( : undefined; return { prefix, + ...(suffix === undefined ? {} : { suffix }), asarHeader: finishAsar(archiveBytes), ...(fatSliceHeaders === undefined ? {} : { fatSliceHeaders }), ...(mzWindowsHeaderSignature === undefined @@ -132,6 +139,18 @@ export const createArtifactClassificationProbe = ( let mzWindowsHeaderSignature: MzWindowsHeaderSignatureEvidence | undefined; let mzWindowsHeaderOffset: number | undefined; let mzHeaderChecked = false; + let suffix = Buffer.alloc(0); + const observeSuffix = (bytes: Uint8Array): void => { + const joined = + bytes.length >= ARTIFACT_CLASSIFICATION_SUFFIX_BYTES + ? bytes + : Buffer.concat([suffix, bytes]); + suffix = Buffer.from( + joined.subarray( + Math.max(0, joined.length - ARTIFACT_CLASSIFICATION_SUFFIX_BYTES), + ), + ); + }; const observeSignatureWindow = ( offset: number, sourceOffset: number, @@ -258,11 +277,14 @@ export const createArtifactClassificationProbe = ( } if (mzWindowsHeaderOffset !== undefined) observeSignatureWindow(mzWindowsHeaderOffset, streamOffset, bytes); + observeSuffix(bytes); observedBytes += bytes.length; }, finish(archiveBytes: number): ArtifactClassificationEvidence { return finishClassificationEvidence( retainedPrefix(), + // Bytes the probe did not observe could still follow its last write. + observedBytes === archiveBytes ? suffix : undefined, archiveBytes, (bytes) => asar.finish(bytes), fatSliceHeadersByOffset, diff --git a/src/artifacts/ArtifactReader.ts b/src/artifacts/ArtifactReader.ts index c9abf679c6..e2fc1b6219 100644 --- a/src/artifacts/ArtifactReader.ts +++ b/src/artifacts/ArtifactReader.ts @@ -4,6 +4,7 @@ import type { ArtifactLimitBoundary, } from "../domain/artifactOperationError.js"; import type { + ArtifactContainerIntegrityPair, ArtifactIntegrityClaim, ArtifactIntegrityDetails, } from "../domain/artifactIntegrity.js"; @@ -204,6 +205,8 @@ export interface ArtifactReader { | "intel-hex" | "srec" | "esp-image" + | "uf2" + | "dfuse" | "esp-flash-image" | "fit" | "file"; @@ -214,6 +217,8 @@ export interface ArtifactReader { open(entry: ArtifactEntry, signal?: AbortSignal): Promise; provenance(): readonly ArtifactCommand[]; formatMetadata?(): ArtifactFormatMetadata | undefined; + /** A checksum the container declares over its own bytes, with the observed value. */ + containerIntegrity?(): ArtifactContainerIntegrityPair | undefined; /** Bind materialization to the exact inventoried carrier before creating outputs. */ prepareContainer?( expectedSha256: string, diff --git a/src/artifacts/DecodedRegionArtifactReader.ts b/src/artifacts/DecodedRegionArtifactReader.ts new file mode 100644 index 0000000000..be7a689554 --- /dev/null +++ b/src/artifacts/DecodedRegionArtifactReader.ts @@ -0,0 +1,297 @@ +import { Readable } from "node:stream"; +import { setImmediate } from "node:timers/promises"; + +import type { ArtifactFormatMetadata } from "../domain/artifactFormatMetadata.js"; +import type { ArtifactContainerIntegrityPair } from "../domain/artifactIntegrity.js"; +import type { Result } from "../domain/result.js"; +import type { + RegionDecodeFailure, + RegionDecoder, +} from "../domain/firmware/regionDecoder.js"; +import { + ArtifactReaderFailure, + copyArtifactEntry, + sameArtifactEntry, + type ArtifactEntry, + type ArtifactReader, + type ArtifactReaderTraversal, + type ArtifactTraversalOptions, +} from "./ArtifactReader.js"; +import { ArtifactResourceScope } from "./ArtifactResourceScope.js"; +import { assertArtifactReadActive } from "./ArtifactFileOperation.js"; +import { ArtifactFileSource } from "./ArtifactFileSource.js"; +import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js"; +import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js"; + +/** One extractable region: its inventory facts and the decoded bytes behind them. */ +export interface DecodedRegion { + readonly path: string; + /** The file offset of the bytes when they are one contiguous file range. */ + readonly byteOffset: number | null; + readonly formatMetadata: ArtifactFormatMetadata; + readonly bytes: Uint8Array; +} + +/** What one format's decoder established about the whole file. */ +export interface DecodedRegions { + readonly metadata: ArtifactFormatMetadata; + readonly regions: readonly DecodedRegion[]; + /** Omitted when every byte was examined and no limitation applies. */ + readonly coverage?: ArtifactReaderTraversal; + readonly containerIntegrity?: ArtifactContainerIntegrityPair; +} + +/** How one firmware format turns admitted file bytes into regions. */ +export interface RegionDecoding { + readonly label: string; + readonly keyPrefix: string; + createDecoder( + fileBytes: number, + ): Result, RegionDecodeFailure>; + describe(image: Image): DecodedRegions; +} + +const addressText = (address: number): string => + `0x${address.toString(16).padStart(8, "0")}`; + +/** The shared region file name: inclusive start and end as eight hex digits. */ +export const regionFileName = (start: number, end: number): string => + `${addressText(start)}-${addressText(end)}.bin`; + +/** + * Decode a whole firmware file once, retain its region bytes for the reader's + * lifetime, and serve each region as an inventory entry. + */ +export class DecodedRegionArtifactReader implements ArtifactReader { + readonly #scope = new ArtifactResourceScope(); + readonly #entries = new Map(); + readonly #regions = new Map(); + readonly #streams = new Set(); + readonly #source: ArtifactFileSource; + readonly #decoding: RegionDecoding; + #prepared: Promise | undefined; + /** Whole-file facts only; the region map is the sole owner of region bytes. */ + #described: Omit | undefined; + #closing = false; + + constructor( + path: string, + readonly format: "intel-hex" | "srec" | "uf2" | "dfuse", + decoding: RegionDecoding, + admitted?: StableRegularFileDescriptor, + ) { + this.#decoding = decoding; + this.#source = new ArtifactFileSource(path, decoding.label, admitted); + } + + formatMetadata(): ArtifactFormatMetadata | undefined { + const metadata = this.#described?.metadata; + return metadata === undefined ? undefined : { ...metadata }; + } + + containerIntegrity(): ArtifactContainerIntegrityPair | undefined { + const integrity = this.#described?.containerIntegrity; + return integrity === undefined ? undefined : { ...integrity }; + } + + async prepareContainer( + expectedSha256: string, + signal?: AbortSignal, + ): Promise { + await this.#scope.run(async () => { + this.#assertOpen(signal); + await this.#source.bindSha256(expectedSha256, signal); + }); + } + + async *entries( + signal?: AbortSignal, + options?: ArtifactTraversalOptions, + ): AsyncIterable { + await this.#scope.run(async () => { + this.#assertOpen(signal); + this.#prepared ??= this.#prepare(signal); + await this.#prepared; + await this.#source.read(signal); + }); + const coverage = this.#described?.coverage; + if (coverage !== undefined) + options?.onCoverage?.({ + ...coverage, + limitations: [...coverage.limitations], + }); + let index = 0; + for (const entry of this.#entries.values()) { + if (index % 256 === 0) await setImmediate(); + index += 1; + this.#assertOpen(signal); + yield copyArtifactEntry(entry); + } + } + + async open(entry: ArtifactEntry, signal?: AbortSignal): Promise { + return this.#scope.run(async () => { + this.#assertOpen(signal); + const stored = this.#entries.get(entry.adapterKey); + const bytes = this.#regions.get(entry.adapterKey); + if ( + stored === undefined || + bytes === undefined || + !sameArtifactEntry(entry, stored) + ) + throw new ArtifactReaderFailure( + "integrity", + `${this.#decoding.label} entry does not match its owning inventory`, + ); + // Decoded bytes are retained, so only the carrier identity is rechecked. + await this.#source.read(signal); + this.#assertOpen(signal); + const output = Readable.from(this.#regionChunks(bytes, signal), { + objectMode: false, + }); + const onAbort = (): void => { + output.destroy( + new ArtifactReaderFailure( + "cancelled", + `${this.#decoding.label} region read cancelled`, + { cause: signal?.reason }, + ), + ); + }; + signal?.addEventListener("abort", onAbort, { once: true }); + output.once("close", () => signal?.removeEventListener("abort", onAbort)); + output.on("error", () => undefined); + this.#streams.add(output); + output.once("close", () => this.#streams.delete(output)); + // Draining the scope must wait for the readable side, including an + // abandoned consumer and an in-flight asynchronous chunk producer. + void this.#scope + .run( + () => + new Promise((resolve) => { + output.once("close", () => resolve()); + }), + ) + .catch((cause: unknown) => + output.destroy( + cause instanceof Error + ? cause + : new ArtifactReaderFailure( + "unavailable", + `${this.#decoding.label} stream ownership failed`, + { cause }, + ), + ), + ); + return output; + }); + } + + async *#regionChunks( + bytes: Uint8Array, + signal?: AbortSignal, + ): AsyncIterable { + // Keep cached bytes isolated from consumers without copying a whole + // region for every open. Match descriptor reads' 64 KiB chunk size. + for (let offset = 0; offset < bytes.length; offset += 64 * 1024) { + await setImmediate(); + this.#assertOpen(signal); + yield Buffer.from(bytes.subarray(offset, offset + 64 * 1024)); + } + } + + async close(): Promise { + this.#closing = true; + for (const stream of this.#streams) + stream.destroy( + new ArtifactReaderFailure( + "unavailable", + `${this.#decoding.label} reader closed`, + ), + ); + await this.#scope.close(); + await this.#source.close(); + this.#entries.clear(); + this.#regions.clear(); + } + + provenance(): readonly [] { + return []; + } + + #assertOpen(signal?: AbortSignal): void { + assertArtifactReadActive(signal); + if (this.#closing) + throw new ArtifactReaderFailure( + "unavailable", + `${this.#decoding.label} reader is closed`, + ); + } + + async #prepare(signal?: AbortSignal): Promise { + const described = this.#decoding.describe(await this.#decode(signal)); + let index = 0; + for (const region of described.regions) { + if (index % 256 === 0) await setImmediate(); + this.#assertOpen(signal); + const entry = { + path: region.path, + kind: "file" as const, + declaredSize: region.bytes.length, + compressedSize: null, + executable: false, + encrypted: false, + byteOffset: region.byteOffset, + declaredIntegrity: null, + unpacked: false, + limitations: [], + adapterKey: `${this.#decoding.keyPrefix}:${String(index)}`, + formatMetadata: region.formatMetadata, + } satisfies ArtifactEntry; + this.#entries.set(entry.adapterKey, entry); + this.#regions.set(entry.adapterKey, region.bytes); + index += 1; + } + const { regions: _regions, ...facts } = described; + this.#described = facts; + await this.#source.read(signal); + } + + async #decode(signal?: AbortSignal): Promise { + const { handle, initial } = await this.#source.read(signal); + const decoder = this.#decoding.createDecoder(initial.size); + if (!decoder.ok) throw this.#failure(decoder.error); + // Reading only the admitted extent keeps decoded bytes within the arena. + if (initial.size > 0) + for await (const chunk of readFileHandleChunks(handle, { + end: initial.size - 1, + signal, + cancellationFailure: (reason) => + new ArtifactReaderFailure( + "cancelled", + `${this.#decoding.label} read cancelled`, + { cause: reason }, + ), + }) as AsyncIterable) { + this.#assertOpen(signal); + const written = decoder.value.write(chunk); + if (!written.ok) throw this.#failure(written.error); + } + const image = decoder.value.finish(); + if (!image.ok) throw this.#failure(image.error); + return image.value; + } + + #failure(failure: RegionDecodeFailure): ArtifactReaderFailure { + return failure.kind === "format" + ? new ArtifactReaderFailure( + "format", + `${this.#decoding.label} is malformed: ${failure.message}`, + ) + : new ArtifactReaderFailure( + "limit", + `${this.#decoding.label} exceeds a fixed limit: ${failure.message}`, + failure.boundary === undefined ? {} : { boundary: failure.boundary }, + ); + } +} diff --git a/src/artifacts/DfuSeArtifactReader.test.ts b/src/artifacts/DfuSeArtifactReader.test.ts new file mode 100644 index 0000000000..6240e2c31a --- /dev/null +++ b/src/artifacts/DfuSeArtifactReader.test.ts @@ -0,0 +1,299 @@ +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it } from "vitest"; + +import { ArtifactResourceScope } from "./ArtifactResourceScope.js"; +import { extractArtifact } from "./extraction/ArtifactExtraction.js"; +import { + dfuseFile, + type DfuSeFixture, +} from "../domain/firmware/dfuse.fixture.js"; +import { ARTIFACT_COMPARISON_EXAMPLE } from "../contracts/artifactComparisonExample.js"; +import { compareArtifacts } from "../domain/artifactComparison.js"; +import { artifactInventoryResultSchema } from "../domain/artifactGraph.js"; +import { createEvidence } from "../domain/evidence.js"; +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; +import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js"; + +const FIRMWARE: DfuSeFixture = { + targets: [ + { + alternateSetting: 0, + name: "Internal Flash", + elements: [ + { address: 0x0800_0000, data: [1, 2, 3, 4] }, + { address: 0x0800_4000, data: [5, 6] }, + ], + }, + { + alternateSetting: 1, + elements: [{ address: 0x1fff_c000, data: [0xaa, 0x55] }], + }, + ], +}; + +const writeImage = async (name: string, bytes: Uint8Array): Promise => { + const directory = await createTestTempDirectory("rea-dfuse-"); + const path = join(directory, name); + await writeFile(path, bytes); + return path; +}; + +it("inventories one region per element with its file range and target facts", async () => { + const snapshot = await scanCanonicalArtifactInventory( + await writeImage("firmware.dfu", dfuseFile(FIRMWARE)), + ); + expect(snapshot.manifest.root_format).toBe("dfuse"); + expect(snapshot.coverage.status).toBe("complete"); + expect(snapshot.integrity_contradictions).toEqual([]); + expect( + snapshot.occurrences.find((row) => row.logical_path === "."), + ).toMatchObject({ + artifact_kind: "container", + hash_status: "verified", + format_metadata: { kind: "dfuse", target_count: 2, element_count: 3 }, + }); + expect( + snapshot.occurrences + .filter((row) => row.logical_path !== ".") + .map((row) => [ + row.logical_path, + row.source_location, + row.format_metadata, + ]), + ).toEqual([ + [ + "target-0/element-0-0x08000000-0x08000003.bin", + { offset: 293, length: 4 }, + { + kind: "dfuse-element", + target_index: 0, + alternate_setting: 0, + target_name: "Internal Flash", + element_index: 0, + start_address: 0x0800_0000, + end_address: 0x0800_0003, + length: 4, + }, + ], + [ + "target-0/element-1-0x08004000-0x08004001.bin", + { offset: 305, length: 2 }, + { + kind: "dfuse-element", + target_index: 0, + alternate_setting: 0, + target_name: "Internal Flash", + element_index: 1, + start_address: 0x0800_4000, + end_address: 0x0800_4001, + length: 2, + }, + ], + [ + "target-1/element-0-0x1fffc000-0x1fffc001.bin", + { offset: 589, length: 2 }, + { + kind: "dfuse-element", + target_index: 1, + alternate_setting: 1, + target_name: null, + element_index: 0, + start_address: 0x1fff_c000, + end_address: 0x1fff_c001, + length: 2, + }, + ], + ]); +}); + +it("refuses a suffix CRC mismatch under the default integrity policy", async () => { + const path = await writeImage( + "bad-crc.dfu", + dfuseFile({ ...FIRMWARE, crc: 0x1234_5678 }), + ); + const expected = ( + await scanCanonicalArtifactInventory( + await writeImage("good.dfu", dfuseFile(FIRMWARE)), + ) + ).occurrences.find((row) => row.logical_path === ".")?.format_metadata; + if (expected?.kind !== "dfuse") throw new Error("Missing DfuSe metadata"); + const computed = expected.computed_crc.toString(16).padStart(8, "0"); + await expect(scanCanonicalArtifactInventory(path)).rejects.toMatchObject({ + reason: "integrity", + details: { + logicalPath: ".", + unpacked: false, + algorithm: "dfu-crc32", + declaredCrc32: "12345678", + calculatedCrc32: computed, + }, + }); + const recorded = await scanCanonicalArtifactInventory(path, { + integrity: "record-and-continue", + }); + const root = recorded.occurrences.find((row) => row.logical_path === "."); + expect(root).toMatchObject({ + hash_status: "mismatched", + limitations: ["Declared integrity metadata contradicts observed bytes."], + format_metadata: { + suffix_crc: 0x1234_5678, + computed_crc: expected.computed_crc, + }, + }); + expect(recorded.integrity_contradictions).toEqual([ + expect.objectContaining({ + occurrence_id: root?.occurrence_id, + logical_path: ".", + algorithm: "dfu-crc32", + declared_crc32: "12345678", + observed_crc32: computed, + trust: "observed-untrusted", + }), + ]); + expect(recorded.limitations).toContain( + "1 integrity contradiction(s) were recorded; mismatched content is observed-untrusted.", + ); + expect(recorded.occurrences).toHaveLength(4); +}); + +it("extracts elements and carries a root contradiction into the extraction", async () => { + const directory = await createTestTempDirectory("rea-dfuse-extract-"); + const path = join(directory, "firmware.dfu"); + await writeFile(path, dfuseFile({ ...FIRMWARE, crc: 0 })); + const resourceScope = new ArtifactResourceScope(); + try { + const extracted = await extractArtifact({ + inputPath: path, + inputFormat: "dfuse", + outputRoot: join(directory, "out"), + environment: process.env, + integrityPolicy: "record-and-continue", + resourceScope, + }); + expect([ + ...(await readFile( + join( + extracted.output_root, + "target-1/element-0-0x1fffc000-0x1fffc001.bin", + ), + )), + ]).toEqual([0xaa, 0x55]); + expect(extracted.artifacts).toHaveLength(3); + expect(extracted.integrity_contradictions).toEqual([ + expect.objectContaining({ logical_path: ".", algorithm: "dfu-crc32" }), + ]); + expect(extracted.limitations).toContain( + "The container contradicts its declared integrity metadata; every extracted file is observed-untrusted.", + ); + expect(extracted.limitations.join("\n")).not.toContain( + "extracted file(s) contradict", + ); + } finally { + await resourceScope.close(); + } +}); + +it("recognizes DfuSe only with its suffix, including inside a directory", async () => { + const directory = await createTestTempDirectory("rea-dfuse-nested-"); + await mkdir(join(directory, "updates")); + const bytes = dfuseFile(FIRMWARE); + await writeFile(join(directory, "updates", "image.bin"), bytes); + await writeFile( + join(directory, "updates", "no-suffix.dfu"), + bytes.subarray(0, bytes.length - 16), + ); + const snapshot = await scanCanonicalArtifactInventory(directory); + expect( + snapshot.occurrences + .filter((row) => row.entry_kind === "file") + .map((row) => [row.logical_path, row.artifact_kind, row.artifact_format]), + ).toEqual([ + ["updates/image.bin", "container", "dfuse"], + ["updates/no-suffix.dfu", "resource", "file"], + ]); + const root = await scanCanonicalArtifactInventory( + await writeImage("no-suffix.dfu", bytes.subarray(0, bytes.length - 16)), + ); + expect(root.manifest.root_format).toBe("file"); +}); + +it("reads the suffix of a DfuSe root longer than the classification prefix", async () => { + const large = dfuseFile({ + targets: [ + { + alternateSetting: 0, + elements: [ + { address: 0x0800_0000, data: new Uint8Array(20_000).fill(0x5a) }, + ], + }, + ], + }); + const snapshot = await scanCanonicalArtifactInventory( + await writeImage("large.dfu", large), + ); + expect(snapshot.manifest.root_format).toBe("dfuse"); + expect( + snapshot.occurrences.find((row) => row.logical_path !== ".") + ?.source_location, + ).toEqual({ offset: 293, length: 20_000 }); + // The same prefix without the suffix stays a plain file. + const unsuffixed = await scanCanonicalArtifactInventory( + await writeImage( + "large-no-suffix.dfu", + large.subarray(0, large.length - 1), + ), + ); + expect(unsuffixed.manifest.root_format).toBe("file"); +}); + +it("compares an element's target memory, not only its bytes and path", async () => { + const evidence = async (alternateSetting: number, name: string) => { + const inventory = artifactInventoryResultSchema.parse( + await scanCanonicalArtifactInventory( + await writeImage( + "firmware.dfu", + dfuseFile({ + targets: [ + { + alternateSetting, + name, + elements: [{ address: 0x0800_0000, data: [1, 2, 3, 4] }], + }, + ], + }), + ), + ), + ); + return createEvidence( + { + path: "firmware.dfu", + sha256: inventory.manifest.root_sha256, + format: "dfuse", + }, + ARTIFACT_COMPARISON_EXAMPLE.left.provider, + { + operation: "inventory_artifact", + parameters: {}, + result: inventory, + confidence: "observed", + authority: "shipped-artifact", + }, + ); + }; + const left = await evidence(0, "Internal Flash"); + const path = "target-0/element-0-0x08000000-0x08000003.bin"; + for (const right of [ + await evidence(1, "Internal Flash"), + await evidence(0, "Option Bytes"), + ]) + expect( + compareArtifacts(left, right).changes.find( + (change) => change.logical_path === path, + ), + ).toMatchObject({ classification: "changed", dimensions: ["metadata"] }); + expect( + compareArtifacts(left, await evidence(0, "Internal Flash")).changes, + ).toEqual([]); +}); diff --git a/src/artifacts/DfuSeArtifactReader.ts b/src/artifacts/DfuSeArtifactReader.ts new file mode 100644 index 0000000000..7982db76fe --- /dev/null +++ b/src/artifacts/DfuSeArtifactReader.ts @@ -0,0 +1,55 @@ +import { + createDfuSeDecoder, + type DfuSeImage, +} from "../domain/firmware/dfuse.js"; +import { + DecodedRegionArtifactReader, + regionFileName, + type RegionDecoding, +} from "./DecodedRegionArtifactReader.js"; +import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js"; + +const crcText = (value: number): string => value.toString(16).padStart(8, "0"); + +const dfuSeRegions: RegionDecoding = { + label: "DfuSe", + keyPrefix: "dfuse", + createDecoder: createDfuSeDecoder, + describe: (image) => ({ + metadata: image.metadata, + // Each element stays one region at its own file range, even beside others. + regions: image.elements.map((element) => { + const end = element.address + element.bytes.length - 1; + return { + path: `target-${String(element.targetIndex)}/element-${String(element.elementIndex)}-${regionFileName(element.address, end)}`, + byteOffset: element.dataOffset, + formatMetadata: { + kind: "dfuse-element", + target_index: element.targetIndex, + alternate_setting: element.alternateSetting, + target_name: element.targetName, + element_index: element.elementIndex, + start_address: element.address, + end_address: end, + length: element.bytes.length, + }, + bytes: element.bytes, + }; + }), + ...(image.limitations.length === 0 + ? {} + : { coverage: { status: "complete", limitations: image.limitations } }), + containerIntegrity: { + algorithm: "dfu-crc32", + declared_crc32: crcText(image.metadata.suffix_crc), + observed_crc32: crcText(image.metadata.computed_crc), + }, + }), +}; + +/** Decode DfuSe targets into one entry per element, verifying the suffix CRC. */ +export class DfuSeArtifactReader extends DecodedRegionArtifactReader { + constructor(path: string, admitted?: StableRegularFileDescriptor) { + super(path, "dfuse", dfuSeRegions, admitted); + } +} diff --git a/src/artifacts/HexRecordArtifactReader.ts b/src/artifacts/HexRecordArtifactReader.ts index b5eb489ed0..3f5628c2a8 100644 --- a/src/artifacts/HexRecordArtifactReader.ts +++ b/src/artifacts/HexRecordArtifactReader.ts @@ -1,267 +1,56 @@ -import { Readable } from "node:stream"; -import { setImmediate } from "node:timers/promises"; - -import type { HexRecordsMetadata } from "../domain/artifactFormatMetadata.js"; import { HEX_RECORD_ENCODING_LABELS, createHexRecordDecoder, - type HexRecordFailure, type HexRecordImage, } from "../domain/firmware/hexRecords.js"; import type { HexRecordEncoding } from "../domain/firmware/hexRecordSyntax.js"; import { - ArtifactReaderFailure, - copyArtifactEntry, - sameArtifactEntry, - type ArtifactEntry, - type ArtifactReader, - type ArtifactReaderTraversal, - type ArtifactTraversalOptions, -} from "./ArtifactReader.js"; -import { ArtifactResourceScope } from "./ArtifactResourceScope.js"; -import { assertArtifactReadActive } from "./ArtifactFileOperation.js"; -import { ArtifactFileSource } from "./ArtifactFileSource.js"; -import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js"; + DecodedRegionArtifactReader, + regionFileName, + type RegionDecoding, +} from "./DecodedRegionArtifactReader.js"; import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js"; -const addressText = (address: number): string => - `0x${address.toString(16).padStart(8, "0")}`; +const hexRecordRegions = ( + format: HexRecordEncoding, +): RegionDecoding => ({ + label: HEX_RECORD_ENCODING_LABELS[format], + keyPrefix: "hex-records", + createDecoder: (fileBytes) => createHexRecordDecoder(format, fileBytes), + describe: (image) => ({ + metadata: image.metadata, + regions: image.regions.map((region) => ({ + path: regionFileName(region.start, region.end), + byteOffset: null, + formatMetadata: { + kind: "hex-record-region", + start_address: region.start, + end_address: region.end, + length: region.bytes.length, + record_count: region.recordCount, + first_line: region.firstLine, + last_line: region.lastLine, + }, + bytes: region.bytes, + })), + ...(image.terminated && image.limitations.length === 0 + ? {} + : { + coverage: { + status: image.terminated ? "complete" : "unknown", + limitations: image.limitations, + }, + }), + }), +}); /** Decode Intel HEX or S-record text into address-contiguous region entries. */ -export class HexRecordArtifactReader implements ArtifactReader { - readonly #scope = new ArtifactResourceScope(); - readonly #entries = new Map(); - readonly #regions = new Map(); - readonly #streams = new Set(); - readonly #source: ArtifactFileSource; - readonly #label: string; - #prepared: Promise | undefined; - #metadata: HexRecordsMetadata | undefined; - #coverage: ArtifactReaderTraversal | undefined; - #closing = false; - +export class HexRecordArtifactReader extends DecodedRegionArtifactReader { constructor( path: string, - readonly format: HexRecordEncoding, + format: HexRecordEncoding, admitted?: StableRegularFileDescriptor, ) { - this.#label = HEX_RECORD_ENCODING_LABELS[format]; - this.#source = new ArtifactFileSource(path, this.#label, admitted); - } - - formatMetadata(): HexRecordsMetadata | undefined { - return this.#metadata === undefined ? undefined : { ...this.#metadata }; - } - - async prepareContainer( - expectedSha256: string, - signal?: AbortSignal, - ): Promise { - await this.#scope.run(async () => { - this.#assertOpen(signal); - await this.#source.bindSha256(expectedSha256, signal); - }); - } - - async *entries( - signal?: AbortSignal, - options?: ArtifactTraversalOptions, - ): AsyncIterable { - await this.#scope.run(async () => { - this.#assertOpen(signal); - this.#prepared ??= this.#prepare(signal); - await this.#prepared; - await this.#source.read(signal); - }); - if (this.#coverage !== undefined) - options?.onCoverage?.({ - ...this.#coverage, - limitations: [...this.#coverage.limitations], - }); - let index = 0; - for (const entry of this.#entries.values()) { - if (index % 256 === 0) await setImmediate(); - index += 1; - this.#assertOpen(signal); - yield copyArtifactEntry(entry); - } - } - - async open(entry: ArtifactEntry, signal?: AbortSignal): Promise { - return this.#scope.run(async () => { - this.#assertOpen(signal); - const stored = this.#entries.get(entry.adapterKey); - const bytes = this.#regions.get(entry.adapterKey); - if ( - stored === undefined || - bytes === undefined || - !sameArtifactEntry(entry, stored) - ) - throw new ArtifactReaderFailure( - "integrity", - `${this.#label} entry does not match its owning inventory`, - ); - // Decoded bytes are retained, so only the carrier identity is rechecked. - await this.#source.read(signal); - this.#assertOpen(signal); - const output = Readable.from(this.#regionChunks(bytes, signal), { - objectMode: false, - }); - const onAbort = (): void => { - output.destroy( - new ArtifactReaderFailure( - "cancelled", - `${this.#label} region read cancelled`, - { cause: signal?.reason }, - ), - ); - }; - signal?.addEventListener("abort", onAbort, { once: true }); - output.once("close", () => signal?.removeEventListener("abort", onAbort)); - output.on("error", () => undefined); - this.#streams.add(output); - output.once("close", () => this.#streams.delete(output)); - // Draining the scope must wait for the readable side, including an - // abandoned consumer and an in-flight asynchronous chunk producer. - void this.#scope - .run( - () => - new Promise((resolve) => { - output.once("close", () => resolve()); - }), - ) - .catch((cause: unknown) => - output.destroy( - cause instanceof Error - ? cause - : new ArtifactReaderFailure( - "unavailable", - `${this.#label} stream ownership failed`, - { cause }, - ), - ), - ); - return output; - }); - } - - async *#regionChunks( - bytes: Uint8Array, - signal?: AbortSignal, - ): AsyncIterable { - // Keep cached bytes isolated from consumers without copying a whole - // region for every open. Match descriptor reads' 64 KiB chunk size. - for (let offset = 0; offset < bytes.length; offset += 64 * 1024) { - await setImmediate(); - this.#assertOpen(signal); - yield Buffer.from(bytes.subarray(offset, offset + 64 * 1024)); - } - } - - async close(): Promise { - this.#closing = true; - for (const stream of this.#streams) - stream.destroy( - new ArtifactReaderFailure( - "unavailable", - `${this.#label} reader closed`, - ), - ); - await this.#scope.close(); - await this.#source.close(); - this.#entries.clear(); - this.#regions.clear(); - } - - provenance(): readonly [] { - return []; - } - - #assertOpen(signal?: AbortSignal): void { - assertArtifactReadActive(signal); - if (this.#closing) - throw new ArtifactReaderFailure( - "unavailable", - `${this.#label} reader is closed`, - ); - } - - async #prepare(signal?: AbortSignal): Promise { - const image = await this.#decode(signal); - let index = 0; - for (const region of image.regions) { - if (index % 256 === 0) await setImmediate(); - this.#assertOpen(signal); - const entry = { - path: `${addressText(region.start)}-${addressText(region.end)}.bin`, - kind: "file" as const, - declaredSize: region.bytes.length, - compressedSize: null, - executable: false, - encrypted: false, - byteOffset: null, - declaredIntegrity: null, - unpacked: false, - limitations: [], - adapterKey: `hex-records:${String(index)}`, - formatMetadata: { - kind: "hex-record-region" as const, - start_address: region.start, - end_address: region.end, - length: region.bytes.length, - record_count: region.recordCount, - first_line: region.firstLine, - last_line: region.lastLine, - }, - } satisfies ArtifactEntry; - this.#entries.set(entry.adapterKey, entry); - this.#regions.set(entry.adapterKey, region.bytes); - index += 1; - } - this.#metadata = image.metadata; - if (!image.terminated || image.limitations.length > 0) - this.#coverage = { - status: image.terminated ? "complete" : "unknown", - limitations: image.limitations, - }; - await this.#source.read(signal); - } - - async #decode(signal?: AbortSignal): Promise { - const { handle, initial } = await this.#source.read(signal); - const decoder = createHexRecordDecoder(this.format, initial.size); - if (!decoder.ok) throw this.#failure(decoder.error); - // Reading only the admitted extent keeps decoded bytes within the arena. - if (initial.size > 0) - for await (const chunk of readFileHandleChunks(handle, { - end: initial.size - 1, - signal, - cancellationFailure: (reason) => - new ArtifactReaderFailure( - "cancelled", - `${this.#label} read cancelled`, - { cause: reason }, - ), - }) as AsyncIterable) { - this.#assertOpen(signal); - const written = decoder.value.write(chunk); - if (!written.ok) throw this.#failure(written.error); - } - const image = decoder.value.finish(); - if (!image.ok) throw this.#failure(image.error); - return image.value; - } - - #failure(failure: HexRecordFailure): ArtifactReaderFailure { - return failure.kind === "format" - ? new ArtifactReaderFailure( - "format", - `${this.#label} is malformed: ${failure.message}`, - ) - : new ArtifactReaderFailure( - "limit", - `${this.#label} exceeds a fixed limit: ${failure.message}`, - failure.boundary === undefined ? {} : { boundary: failure.boundary }, - ); + super(path, format, hexRecordRegions(format), admitted); } } diff --git a/src/artifacts/Uf2ArtifactReader.test.ts b/src/artifacts/Uf2ArtifactReader.test.ts new file mode 100644 index 0000000000..3cf1cafe38 --- /dev/null +++ b/src/artifacts/Uf2ArtifactReader.test.ts @@ -0,0 +1,231 @@ +import { readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; + +import { expect, it } from "vitest"; + +import { Uf2ArtifactReader } from "./Uf2ArtifactReader.js"; +import { ArtifactResourceScope } from "./ArtifactResourceScope.js"; +import { extractArtifact } from "./extraction/ArtifactExtraction.js"; +import { + RP2040_FAMILY, + RP2XXX_ABSOLUTE_FAMILY, + patternBytes, + uf2Block, + uf2File, + uf2Image, +} from "../domain/firmware/uf2.fixture.js"; +import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js"; +import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js"; + +const INVENTORY_LIMITATION = + "UF2 inventory decodes main-flash block payloads into address-contiguous regions per familyID. It does not establish CPU, byte order, memory map, or an entry point; a familyID names a target family, and gaps are unwritten, not zero."; + +const writeImage = async (name: string, bytes: Uint8Array): Promise => { + const directory = await createTestTempDirectory("rea-uf2-"); + const path = join(directory, name); + await writeFile(path, bytes); + return path; +}; + +const APP = patternBytes(600, 5); +const ABSOLUTE = uf2Block({ + address: 0x10ff_ff00, + payload: new Uint8Array(256).fill(0xef), + blockNo: 0, + numBlocks: 2, + familyId: RP2XXX_ABSOLUTE_FAMILY, +}); +const UNNAMED = uf2Block({ + address: 0x2000_0000, + payload: [1, 2, 3], + blockNo: 0, + numBlocks: 1, + familyId: 0x1234_5678, +}); + +it("inventories each family's regions with family and block facts", async () => { + const snapshot = await scanCanonicalArtifactInventory( + await writeImage( + "firmware.bin", + uf2File([...uf2Image(RP2040_FAMILY, 0x1000_0000, APP), UNNAMED]), + ), + ); + expect(snapshot.manifest.root_format).toBe("uf2"); + expect(snapshot.coverage).toEqual({ + scope: "container-entries", + status: "complete", + }); + expect(snapshot.limitations).toContain(INVENTORY_LIMITATION); + expect( + snapshot.occurrences.find((row) => row.logical_path === "."), + ).toMatchObject({ + artifact_kind: "container", + artifact_format: "uf2", + format_metadata: { + kind: "uf2", + block_count: 4, + family_count: 2, + region_count: 2, + data_bytes: 603, + }, + }); + expect( + snapshot.occurrences + .filter((row) => row.logical_path !== ".") + .map((row) => [ + row.logical_path, + row.declared_size, + row.source_location, + row.format_metadata, + ]), + ).toEqual([ + [ + "family-0x12345678/0x20000000-0x20000002.bin", + 3, + null, + { + kind: "uf2-region", + family_id: 0x1234_5678, + family_name: null, + start_address: 0x2000_0000, + end_address: 0x2000_0002, + length: 3, + block_count: 1, + first_file_block: 3, + last_file_block: 3, + }, + ], + [ + "family-0xe48bff56/0x10000000-0x10000257.bin", + 600, + null, + { + kind: "uf2-region", + family_id: RP2040_FAMILY, + family_name: "RP2040", + start_address: 0x1000_0000, + end_address: 0x1000_0257, + length: 600, + block_count: 3, + first_file_block: 0, + last_file_block: 2, + }, + ], + ]); +}); + +it("leaves coverage unknown when numbered blocks are absent", async () => { + const snapshot = await scanCanonicalArtifactInventory( + await writeImage( + "universal.uf2", + uf2File([ABSOLUTE, ...uf2Image(RP2040_FAMILY, 0x1000_0000, APP)]), + ), + ); + expect(snapshot.coverage.status).toBe("unknown"); + expect(snapshot.limitations).toContain( + "UF2 numbering declares 1 block(s) that are absent: family 0xe48bff57 (RP2XXX_ABSOLUTE) with 2 numbered blocks has 1 of them.", + ); + expect( + snapshot.occurrences + .filter((row) => row.logical_path !== ".") + .map((row) => row.logical_path), + ).toEqual([ + "family-0xe48bff56/0x10000000-0x10000257.bin", + "family-0xe48bff57/0x10ffff00-0x10ffffff.bin", + ]); +}); + +it("fails closed on a conflicting rewrite and leaves short files unrecognized", async () => { + const rewrite = uf2Block({ + address: 0x1000_0010, + payload: [0xff], + blockNo: 0, + numBlocks: 1, + familyId: RP2040_FAMILY, + }); + await expect( + scanCanonicalArtifactInventory( + await writeImage( + "conflict.uf2", + uf2File([...uf2Image(RP2040_FAMILY, 0x1000_0000, APP), rewrite]), + ), + ), + ).rejects.toMatchObject({ + reason: "format", + message: + "UF2 is malformed: family 0xe48bff56 (RP2040) address 0x10000010 is written with different values by blocks 0-2 (0x75) and block 3 (0xff)", + }); + const truncated = await scanCanonicalArtifactInventory( + await writeImage("short.uf2", ABSOLUTE.subarray(0, 511)), + ); + expect(truncated.manifest.root_format).toBe("file"); +}); + +it("extracts byte-exact regions into family directories", async () => { + const directory = await createTestTempDirectory("rea-uf2-extract-"); + const path = join(directory, "firmware.uf2"); + await writeFile( + path, + uf2File([ABSOLUTE, ...uf2Image(RP2040_FAMILY, 0x1000_0000, APP)]), + ); + const resourceScope = new ArtifactResourceScope(); + try { + const extracted = await extractArtifact({ + inputPath: path, + inputFormat: "uf2", + outputRoot: join(directory, "out"), + environment: process.env, + integrityPolicy: "fail", + resourceScope, + }); + expect(extracted.containment_verified).toBe(true); + expect(extracted.coverage.status).toBe("unknown"); + expect( + Buffer.compare( + await readFile( + join( + extracted.output_root, + "family-0xe48bff56/0x10000000-0x10000257.bin", + ), + ), + APP, + ), + ).toBe(0); + expect( + Buffer.compare( + await readFile( + join( + extracted.output_root, + "family-0xe48bff57/0x10ffff00-0x10ffffff.bin", + ), + ), + Buffer.alloc(256, 0xef), + ), + ).toBe(0); + } finally { + await resourceScope.close(); + } +}); + +it("rejects a forged region entry and a changed carrier", async () => { + const path = await writeImage( + "identity.uf2", + uf2File(uf2Image(RP2040_FAMILY, 0x1000_0000, APP)), + ); + const reader = new Uf2ArtifactReader(path); + try { + const entries = []; + for await (const entry of reader.entries()) entries.push(entry); + const entry = entries[0]; + if (entry === undefined) throw new Error("Missing region"); + await expect( + reader.open({ ...entry, adapterKey: "uf2:1" }), + ).rejects.toMatchObject({ reason: "integrity" }); + await writeFile(path, uf2File(uf2Image(RP2040_FAMILY, 0, APP))); + await expect(reader.open(entry)).rejects.toMatchObject({ + reason: "integrity", + }); + } finally { + await reader.close(); + } +}); diff --git a/src/artifacts/Uf2ArtifactReader.ts b/src/artifacts/Uf2ArtifactReader.ts new file mode 100644 index 0000000000..df0a93c091 --- /dev/null +++ b/src/artifacts/Uf2ArtifactReader.ts @@ -0,0 +1,53 @@ +import { createUf2Decoder, type Uf2Image } from "../domain/firmware/uf2.js"; +import { + DecodedRegionArtifactReader, + regionFileName, + type RegionDecoding, +} from "./DecodedRegionArtifactReader.js"; +import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js"; + +/** Families are separate address spaces, so each names its own directory. */ +const familyDirectory = (familyId: number | null): string => + familyId === null + ? "no-family" + : `family-0x${familyId.toString(16).padStart(8, "0")}`; + +const uf2Regions: RegionDecoding = { + label: "UF2", + keyPrefix: "uf2", + createDecoder: createUf2Decoder, + describe: (image) => ({ + metadata: image.metadata, + regions: image.regions.map((region) => ({ + path: `${familyDirectory(region.familyId)}/${regionFileName(region.start, region.end)}`, + byteOffset: null, + formatMetadata: { + kind: "uf2-region", + family_id: region.familyId, + family_name: region.familyName, + start_address: region.start, + end_address: region.end, + length: region.bytes.length, + block_count: region.blockCount, + first_file_block: region.firstFileBlock, + last_file_block: region.lastFileBlock, + }, + bytes: region.bytes, + })), + ...(image.complete && image.limitations.length === 0 + ? {} + : { + coverage: { + status: image.complete ? "complete" : "unknown", + limitations: image.limitations, + }, + }), + }), +}; + +/** Decode UF2 blocks into each family's address-contiguous region entries. */ +export class Uf2ArtifactReader extends DecodedRegionArtifactReader { + constructor(path: string, admitted?: StableRegularFileDescriptor) { + super(path, "uf2", uf2Regions, admitted); + } +} diff --git a/src/artifacts/artifactFormats.test.ts b/src/artifacts/artifactFormats.test.ts index 78c99e3ad4..da351dc0c8 100644 --- a/src/artifacts/artifactFormats.test.ts +++ b/src/artifacts/artifactFormats.test.ts @@ -38,6 +38,8 @@ type FormatsWithReaders = | "intel-hex" | "srec" | "esp-image" + | "uf2" + | "dfuse" | "esp-flash-image" | "fit"; diff --git a/src/artifacts/artifactFormats.ts b/src/artifacts/artifactFormats.ts index 930209b918..aeb7dc8bff 100644 --- a/src/artifacts/artifactFormats.ts +++ b/src/artifacts/artifactFormats.ts @@ -17,6 +17,7 @@ import type { ArtifactReaderAdmission } from "./ArtifactReaderAdmission.js"; import { AsarArtifactReader } from "./AsarArtifactReader.js"; import type { BunStandaloneAdmission } from "./bun/BunStandaloneAdmission.js"; import { BunStandaloneArtifactReader } from "./BunStandaloneArtifactReader.js"; +import { DfuSeArtifactReader } from "./DfuSeArtifactReader.js"; import { DirectoryArtifactReader } from "./DirectoryArtifactReader.js"; import type { DotnetBundleAdmission } from "./dotnet/DotnetBundleAdmission.js"; import { DotnetBundleArtifactReader } from "./DotnetBundleArtifactReader.js"; @@ -36,6 +37,7 @@ import type { NodeSeaAdmission } from "./nodeSea/NodeSeaAdmission.js"; import { NodeSeaArtifactReader } from "./NodeSeaArtifactReader.js"; import type { PyInstallerAdmission } from "./pyinstaller/PyInstallerAdmission.js"; import { PyInstallerArtifactReader } from "./PyInstallerArtifactReader.js"; +import { Uf2ArtifactReader } from "./Uf2ArtifactReader.js"; import { UImageArtifactReader } from "./UImageArtifactReader.js"; import { UnityFsArtifactReader } from "./UnityFsArtifactReader.js"; import { ZipArtifactReader } from "./ZipArtifactReader.js"; @@ -369,6 +371,26 @@ const NON_ZIP_FORMATS = { createReader: (path, options) => new EspImageArtifactReader(path, sourceOf(options)), }), + uf2: container("uf2", { + roleContainer: true, + retainClassifiedSource: true, + extractable: true, + absentInventory: "unavailable", + readerLimitation: + "UF2 inventory decodes main-flash block payloads into address-contiguous regions per familyID. It does not establish CPU, byte order, memory map, or an entry point; a familyID names a target family, and gaps are unwritten, not zero.", + createReader: (path, options) => + new Uf2ArtifactReader(path, sourceOf(options)), + }), + dfuse: container("dfuse", { + roleContainer: true, + retainClassifiedSource: true, + extractable: true, + absentInventory: "unavailable", + readerLimitation: + "DfuSe inventory lists each target element as one region at its declared address. It does not establish CPU, byte order, memory map, or an entry point; an alternate setting selects a device memory, not an address space REA can interpret.", + createReader: (path, options) => + new DfuSeArtifactReader(path, sourceOf(options)), + }), "esp-flash-image": container("esp-flash-image", { roleContainer: true, retainClassifiedSource: true, diff --git a/src/artifacts/extraction/ArtifactExtraction.ts b/src/artifacts/extraction/ArtifactExtraction.ts index 40ed1de772..d51a52c083 100644 --- a/src/artifacts/extraction/ArtifactExtraction.ts +++ b/src/artifacts/extraction/ArtifactExtraction.ts @@ -534,9 +534,18 @@ const createExtractionResult = ( const selectedIds = new Set( selected.map(({ occurrence }) => occurrence.occurrence_id), ); - const integrityContradictions = inventory.integrity_contradictions.filter( - ({ occurrence_id: id }) => selectedIds.has(id), + // A root contradiction covers the container bytes every extracted file came from. + const rootId = inventory.occurrences.find( + (row) => row.logical_path === ".", + )?.occurrence_id; + const rootContradicted = inventory.integrity_contradictions.some( + ({ occurrence_id: id }) => id === rootId, ); + const integrityContradictions = inventory.integrity_contradictions.filter( + ({ occurrence_id: id }) => selectedIds.has(id) || id === rootId, + ); + const extractedContradictions = + integrityContradictions.length - (rootContradicted ? 1 : 0); const extractionSemantic = { source_manifest_id: inventory.manifest.manifest_id, selected_occurrence_ids: selected @@ -572,11 +581,16 @@ const createExtractionResult = ( ? "All regular files exposed by the active artifact reader were materialized." : "All inventoried regular files exposed by the active reader were materialized; source traversal coverage is unknown, so the output is not established as a complete source copy.", "Publication verifies the selected inventoried regular files; it does not establish complete content decoding.", - ...(integrityContradictions.length === 0 + ...(extractedContradictions === 0 ? [] : [ - `${String(integrityContradictions.length)} extracted file(s) contradict declared integrity metadata; their bytes are observed-untrusted.`, + `${String(extractedContradictions)} extracted file(s) contradict declared integrity metadata; their bytes are observed-untrusted.`, ]), + ...(rootContradicted + ? [ + "The container contradicts its declared integrity metadata; every extracted file is observed-untrusted.", + ] + : []), ], }); }; diff --git a/src/artifacts/inventory/ArtifactGraphConstruction.test.ts b/src/artifacts/inventory/ArtifactGraphConstruction.test.ts index fff099b2bb..3075f65f91 100644 --- a/src/artifacts/inventory/ArtifactGraphConstruction.test.ts +++ b/src/artifacts/inventory/ArtifactGraphConstruction.test.ts @@ -17,6 +17,13 @@ import { createArtifactClassificationProbe } from "../ArtifactClassificationProb import { artifactIdForContent } from "../../domain/artifactIdentity.js"; import { pe } from "../../domain/binaryTarget.fixture.js"; import type { ArtifactEntry } from "../ArtifactReader.js"; +import { dfuseFile } from "../../domain/firmware/dfuse.fixture.js"; +import { + RP2040_FAMILY, + patternBytes, + uf2File, + uf2Image, +} from "../../domain/firmware/uf2.fixture.js"; import { classifyArtifactBytesAtPath } from "../../../tests/fixtures/artifactClassification.js"; import { @@ -493,6 +500,41 @@ it("classifies nested ESP images as containers by bytes", () => { }); }); +it("classifies nested UF2 by block magics and DfuSe by prefix and suffix", () => { + const uf2 = uf2File(uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(300))); + expect(classifyArtifactBytesAtPath("firmware/update.bin", uf2)).toEqual({ + kind: "container", + format: "uf2", + }); + expect( + classifyArtifactBytesAtPath("firmware/short.uf2", uf2.subarray(0, 511)), + ).toEqual({ kind: "resource", format: "file" }); + const dfu = dfuseFile({ + targets: [{ alternateSetting: 0, elements: [{ address: 0, data: [1] }] }], + }); + // The prefix alone is not DfuSe; the suffix is observed at the stream end. + expect(classifyArtifactBytesAtPath("firmware/update.dfu", dfu)).toEqual({ + kind: "resource", + format: "file", + }); + const probe = createArtifactClassificationProbe(); + for (let offset = 0; offset < dfu.length; offset += 5) + probe.write(dfu.subarray(offset, offset + 5)); + expect( + classifyArtifactContent( + "firmware/update.bin", + probe.finish(dfu.length), + dfu.length, + ), + ).toMatchObject({ kind: "container", format: "dfuse" }); + // Bytes the probe never saw may follow, so it reports no suffix at all. + const partial = probe.finish(dfu.length + 1); + expect(partial.suffix).toBeUndefined(); + expect( + classifyArtifactContent("firmware/update.bin", partial, dfu.length + 1), + ).toMatchObject({ format: "file" }); +}); + it("classifies a FIT by a /images node within the first 4 KiB", () => { const images = [{ name: "kernel", data: Buffer.from("kernel") }]; const late = devicetreeFixture({ diff --git a/src/artifacts/inventory/ArtifactGraphConstruction.ts b/src/artifacts/inventory/ArtifactGraphConstruction.ts index dd0892c6d5..75614bcb2c 100644 --- a/src/artifacts/inventory/ArtifactGraphConstruction.ts +++ b/src/artifacts/inventory/ArtifactGraphConstruction.ts @@ -4,6 +4,8 @@ import { hasUImageMagic } from "../../domain/uimageFormat.js"; import { hasUnityFsHeader } from "../../domain/unityfsFormat.js"; import { detectHexRecordEncoding } from "../../domain/firmware/hexRecords.js"; import { recognizeEspImage } from "../../domain/firmware/espImage.js"; +import { hasUf2Block } from "../../domain/firmware/uf2.js"; +import { isDfuSeFile } from "../../domain/firmware/dfuse.js"; import { hasFitImagesNode } from "../../domain/firmware/fitImage.js"; import type { ArtifactClassificationEvidence } from "../ArtifactClassificationProbe.js"; import { asarHeaderAdmissionFailure } from "../AsarHeader.js"; @@ -591,7 +593,7 @@ export const classifyArtifactBytes = ( evidence: ArtifactClassificationEvidence, fileSize = evidence.prefix.length, ): ArtifactNode["format"] => { - const { prefix, mzWindowsHeaderSignature } = evidence; + const { prefix, suffix, mzWindowsHeaderSignature } = evidence; // A header-only region has the signature but not the declared cartridge // extent. Keep it as a file instead of inventing another container. if ( @@ -605,6 +607,9 @@ export const classifyArtifactBytes = ( if (hasUImageMagic(prefix)) return "uimage"; const hexRecords = detectHexRecordEncoding(prefix); if (hexRecords !== undefined) return hexRecords; + if (hasUf2Block(prefix)) return "uf2"; + if (suffix !== undefined && isDfuSeFile(prefix, suffix, fileSize)) + return "dfuse"; if (recognizeEspImage(prefix, fileSize) === "esp-image") return "esp-image"; if (hasFitImagesNode(prefix)) return "fit"; if (hasBinaryPlistSignature(prefix)) return "plist"; diff --git a/src/artifacts/inventory/classify.ts b/src/artifacts/inventory/classify.ts index 8939a90b55..0100df88b0 100644 --- a/src/artifacts/inventory/classify.ts +++ b/src/artifacts/inventory/classify.ts @@ -52,6 +52,10 @@ import { import type { ArtifactInventoryOptions } from "./types.js"; import { readMachoHeaderPrefixFacts } from "../../domain/apple/machoContainer.js"; import { hasDiskImageSignature } from "../../domain/apple/diskImageFormat.js"; +import { + DFUSE_SUFFIX_BYTES, + hasDfuSePrefix, +} from "../../domain/firmware/dfuse.js"; import { observeMzWindowsHeader, type MzWindowsHeaderSignatureEvidence, @@ -317,10 +321,21 @@ const classifyRootFormat = async ( archiveBytes, signal, ); - const classificationEvidence = - mzWindowsHeaderSignature === undefined - ? fatEvidence - : { ...fatEvidence, mzWindowsHeaderSignature }; + // The probe stops at its bounded prefix, so a DfuSe trailer is read from + // the file's own end unless the probe already observed every byte. + const suffix = + fatEvidence.suffix === undefined && + archiveBytes >= DFUSE_SUFFIX_BYTES && + hasDfuSePrefix(evidence.prefix) + ? await read(archiveBytes - DFUSE_SUFFIX_BYTES, DFUSE_SUFFIX_BYTES) + : fatEvidence.suffix; + const classificationEvidence = { + ...fatEvidence, + ...(suffix === undefined ? {} : { suffix }), + ...(mzWindowsHeaderSignature === undefined + ? {} + : { mzWindowsHeaderSignature }), + }; const format = classifyArtifactContent( classificationPath, classificationEvidence, diff --git a/src/artifacts/inventory/scanCanonical.ts b/src/artifacts/inventory/scanCanonical.ts index 58994677f1..f8d5e14d12 100644 --- a/src/artifacts/inventory/scanCanonical.ts +++ b/src/artifacts/inventory/scanCanonical.ts @@ -53,6 +53,7 @@ import { type PendingIntegrityContradiction, } from "./scanReader.js"; import type { ArtifactInventoryOptions } from "./types.js"; +import type { ArtifactContainerIntegrityPair } from "../../domain/artifactIntegrity.js"; export interface TransferredArtifactInventorySource { readonly owner: ArtifactResourceOwner; @@ -147,6 +148,7 @@ const scanCanonical = async ( inesRegionScan, unknownParentOccurrences, traversal, + rootContradiction, } = await scanReader( reader, ownedReaders, @@ -172,6 +174,7 @@ const scanCanonical = async ( inesRegionScan, unknownParentOccurrences, traversal, + rootContradiction, }), }; } catch (cause: unknown) { @@ -203,6 +206,7 @@ const scanCanonical = async ( caseCollisionLimitation: undefined, inesRegionScan: undefined, unknownParentOccurrences: new Set(), + rootContradiction: undefined, traversal: { status: "unknown", limitations: [ @@ -405,6 +409,7 @@ interface SnapshotBuildInput { readonly inesRegionScan: InesRegionScan | undefined; readonly unknownParentOccurrences: ReadonlySet; readonly traversal: ArtifactReaderTraversal; + readonly rootContradiction: ArtifactContainerIntegrityPair | undefined; } const buildInventorySnapshot = async ( @@ -433,6 +438,18 @@ const buildInventorySnapshot = async ( ) occurrence.parent_occurrence_id = rootOccurrence.occurrence_id; input.occurrences.unshift(rootOccurrence); + if (input.rootContradiction !== undefined) { + rootOccurrence.hash_status = "mismatched"; + rootOccurrence.limitations.push( + "Declared integrity metadata contradicts observed bytes.", + ); + input.pendingContradictions.push({ + occurrence: rootOccurrence, + integrity: input.rootContradiction, + entryKind: "file", + unpacked: false, + }); + } const occurrenceById = new Map( input.occurrences.map((occurrence) => [ diff --git a/src/artifacts/inventory/scanReader.ts b/src/artifacts/inventory/scanReader.ts index 8ce6b2b816..5210d9733e 100644 --- a/src/artifacts/inventory/scanReader.ts +++ b/src/artifacts/inventory/scanReader.ts @@ -1,7 +1,9 @@ import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js"; import { + artifactContainerIntegrityDetails, artifactIntegrityPair, artifactIntegrityDetails, + type ArtifactContainerIntegrityPair, type ArtifactIntegrityPair, } from "../../domain/artifactIntegrity.js"; import { isAbsolute } from "node:path"; @@ -89,6 +91,8 @@ export const scanReader = async ( readonly inesRegionScan: InesRegionScan | undefined; readonly unknownParentOccurrences: ReadonlySet; readonly traversal: ArtifactReaderTraversal; + /** A recorded disagreement between the root's own checksum and its bytes. */ + readonly rootContradiction: ArtifactContainerIntegrityPair | undefined; }> => { const { nodes, occurrences, pendingContradictions } = emptyScan(); if (reader === undefined) @@ -100,6 +104,7 @@ export const scanReader = async ( inesRegionScan: undefined, unknownParentOccurrences: new Set(), traversal: { status: "complete", limitations: [] }, + rootContradiction: undefined, }; const context: ScanContext = { reader, @@ -120,6 +125,7 @@ export const scanReader = async ( traversal: new ArtifactTraversalObservation(), }; await visitArtifactEntries(context, reader, ""); + const rootContradiction = containerContradiction(reader, integrity); // Archive directories may appear after their children. Resolve containment // against the complete index before directory identities are materialized. const byPath = indexDestinationOccurrencesByPath(occurrences); @@ -148,9 +154,28 @@ export const scanReader = async ( }, unknownParentOccurrences: parentResolution.unknownParentOccurrences, traversal: context.traversal.snapshot(), + rootContradiction, }; }; +/** Apply the integrity policy to a checksum the root declares over its own bytes. */ +const containerContradiction = ( + reader: ArtifactReader, + integrity: ArtifactIntegrityPolicyName, +): ArtifactContainerIntegrityPair | undefined => { + const pair = reader.containerIntegrity?.(); + if (pair === undefined || pair.declared_crc32 === pair.observed_crc32) + return undefined; + if (integrity === "fail") + throw new ArtifactReaderFailure( + "integrity", + "Artifact integrity metadata disagrees with content: .", + undefined, + artifactContainerIntegrityDetails(".", pair), + ); + return pair; +}; + const inventoryEntries = ( context: ScanContext, reader: ArtifactReader, diff --git a/src/contracts/errorSchemas.ts b/src/contracts/errorSchemas.ts index 6e3839fcc7..4395468939 100644 --- a/src/contracts/errorSchemas.ts +++ b/src/contracts/errorSchemas.ts @@ -71,6 +71,18 @@ export const analysisErrorProjectionSchema = z.discriminatedUnion("code", [ unpacked: z.boolean(), }) .catchall(jsonValueSchema), + z + .object({ + logical_path: z.string(), + algorithm: z.literal("dfu-crc32"), + declared_crc32: z.string().regex(/^[a-f0-9]{8}$/u), + calculated_crc32: z + .string() + .regex(/^[a-f0-9]{8}$/u) + .nullable(), + unpacked: z.boolean(), + }) + .catchall(jsonValueSchema), ]), }) .strict(), diff --git a/src/domain/analysisSnapshot.ts b/src/domain/analysisSnapshot.ts index 9bd44537d1..1dcea4e47a 100644 --- a/src/domain/analysisSnapshot.ts +++ b/src/domain/analysisSnapshot.ts @@ -81,6 +81,8 @@ const formatSchema = z.enum([ "intel-hex", "srec", "esp-image", + "uf2", + "dfuse", "esp-flash-image", "fit", "plist", diff --git a/src/domain/analysisView/artifactInventoryView.ts b/src/domain/analysisView/artifactInventoryView.ts index 083402868d..50fa587ce1 100644 --- a/src/domain/analysisView/artifactInventoryView.ts +++ b/src/domain/analysisView/artifactInventoryView.ts @@ -282,6 +282,8 @@ export const projectArtifactInventoryView = ( inventory.manifest.root_format === "intel-hex" || inventory.manifest.root_format === "srec" || inventory.manifest.root_format === "esp-image" || + inventory.manifest.root_format === "uf2" || + inventory.manifest.root_format === "dfuse" || inventory.manifest.root_format === "esp-flash-image" || inventory.manifest.root_format === "fit"; const counts = inventoryCounts(source); diff --git a/src/domain/artifactComparison.ts b/src/domain/artifactComparison.ts index d670e49c06..214b68cbe6 100644 --- a/src/domain/artifactComparison.ts +++ b/src/domain/artifactComparison.ts @@ -14,6 +14,7 @@ import { type InventorySet, } from "./artifactInventoryEvidence.js"; import { + dfuSeElementDeclaration, espFlashPartitionDeclaration, espImageSegmentDeclaration, fitImageDeclaration, @@ -469,6 +470,7 @@ interface OccurrenceMetadata { readonly espFlashPartition: MetadataComparison; readonly fitImage: MetadataComparison; readonly fitImageAvailability: MetadataComparison; + readonly dfuSeElement: MetadataComparison; readonly pyInstaller: MetadataComparison; } @@ -478,7 +480,8 @@ const declarationChanged = (metadata: OccurrenceMetadata): boolean => metadata.nodeSea === "changed" || metadata.espImageSegment === "changed" || metadata.espFlashPartition === "changed" || - metadata.fitImage === "changed"; + metadata.fitImage === "changed" || + metadata.dfuSeElement === "changed"; /** Unknown classifications and absent node metadata cannot establish a difference. */ const differsWithUnknown = ( @@ -584,6 +587,15 @@ const occurrenceMetadata = (input: OccurrencePair): OccurrenceMetadata => { : undefined, espImageSegmentDeclaration, ), + dfuSeElement: compareOptionalMetadata( + left?.format_metadata?.kind === "dfuse-element" + ? left.format_metadata + : undefined, + right?.format_metadata?.kind === "dfuse-element" + ? right.format_metadata + : undefined, + dfuSeElementDeclaration, + ), espFlashPartition: compareOptionalMetadata( left?.format_metadata?.kind === "esp-flash-partition" ? left.format_metadata diff --git a/src/domain/artifactFormatMetadata.ts b/src/domain/artifactFormatMetadata.ts index b39e3e6602..d6f8e95479 100644 --- a/src/domain/artifactFormatMetadata.ts +++ b/src/domain/artifactFormatMetadata.ts @@ -163,6 +163,86 @@ export const nodeSeaEntryMetadataSchema = z.strictObject({ index: uint32, }); +/** + * UF2 facts derived from every 512-byte block. Regions hold only valid + * main-flash payloads; block numbering is grouped by familyID and block count. + */ +export const uf2MetadataSchema = z.strictObject({ + kind: z.literal("uf2"), + block_count: offset, + trailing_bytes: offset, + invalid_block_count: offset, + flash_block_count: offset, + not_main_flash_block_count: offset, + file_container_block_count: offset, + md5_block_count: offset, + extension_tag_block_count: offset, + family_count: offset, + /** The registry revision REA's region `family_name` values come from. */ + family_name_registry: z.string(), + numbering_group_count: offset, + missing_block_count: offset, + ambiguous_numbering_group_count: offset, + repeated_block_count: offset, + out_of_order_block_count: offset, + block_number_out_of_range_count: offset, + region_count: offset, + data_bytes: offset, + duplicate_bytes: offset, +}); +export type Uf2Metadata = z.infer; + +/** One family's address-contiguous region; file blocks are zero-based positions. */ +export const uf2RegionMetadataSchema = z.strictObject({ + kind: z.literal("uf2-region"), + family_id: uint32.nullable(), + /** REA's label for `family_id`, from `family_name_registry`; not file content. */ + family_name: z.string().nullable(), + start_address: uint32, + end_address: uint32, + length: offset, + block_count: offset, + first_file_block: offset, + last_file_block: offset, +}); + +/** DfuSe prefix and DFU suffix facts; both CRCs use the suffix's stored form. */ +export const dfuSeMetadataSchema = z.strictObject({ + kind: z.literal("dfuse"), + image_size: uint32, + target_count: uint32, + element_count: offset, + empty_element_count: offset, + overlapping_element_count: offset, + data_bytes: offset, + device_version: uint32, + product_id: uint32, + vendor_id: uint32, + suffix_crc: uint32, + computed_crc: uint32, +}); +export type DfuSeMetadata = z.infer; + +/** One DfuSe element with its target's alternate setting and name. */ +export const dfuSeElementMetadataSchema = z.strictObject({ + kind: z.literal("dfuse-element"), + target_index: uint32, + alternate_setting: uint32, + target_name: z.string().nullable(), + element_index: uint32, + start_address: uint32, + end_address: uint32, + length: offset, +}); + +/** The device memory an element targets, which its path and bytes do not show. */ +export const dfuSeElementDeclaration = ( + metadata: z.infer, +) => ({ + alternate_setting: metadata.alternate_setting, + target_name: metadata.target_name, +}); + /** * Intel HEX or Motorola S-record facts derived from every record. Start * addresses are declared values, not proven entry points. @@ -763,6 +843,10 @@ export const artifactFormatMetadataSchema = z.discriminatedUnion("kind", [ espImageMetadataSchema, espImageSegmentMetadataSchema, espImageSignatureBlockMetadataSchema, + uf2MetadataSchema, + uf2RegionMetadataSchema, + dfuSeMetadataSchema, + dfuSeElementMetadataSchema, espFlashImageMetadataSchema, espFlashPartitionMetadataSchema, fitMetadataSchema, diff --git a/src/domain/artifactGraph.ts b/src/domain/artifactGraph.ts index b2ef951b84..9528ba6885 100644 --- a/src/domain/artifactGraph.ts +++ b/src/domain/artifactGraph.ts @@ -1,4 +1,5 @@ import { + artifactDfuCrc32PairSchema, artifactIntegrityClaimSchema, artifactSha256PairSchema, artifactMd5PairSchema, @@ -95,6 +96,8 @@ export const ARTIFACT_FORMAT_IDS = [ "intel-hex", "srec", "esp-image", + "uf2", + "dfuse", "esp-flash-image", "fit", "unknown", @@ -114,6 +117,8 @@ const artifactContentFormatSchema = artifactFormatSchema.extract([ "intel-hex", "srec", "esp-image", + "uf2", + "dfuse", "fit", "mach-o-universal", "mach-o", @@ -198,6 +203,7 @@ const integrityContradictionBaseSchema = z.object({ export const integrityContradictionSchema = z.union([ integrityContradictionBaseSchema.extend(artifactSha256PairSchema.shape), integrityContradictionBaseSchema.extend(artifactMd5PairSchema.shape), + integrityContradictionBaseSchema.extend(artifactDfuCrc32PairSchema.shape), ]); /** diff --git a/src/domain/artifactInspection.ts b/src/domain/artifactInspection.ts index 7ccd5fe86b..6671fca9d1 100644 --- a/src/domain/artifactInspection.ts +++ b/src/domain/artifactInspection.ts @@ -6,6 +6,7 @@ import { immutableJsonSnapshotMember, } from "./immutableJson.js"; import { + artifactDfuCrc32PairSchema, artifactIntegrityPairOf, artifactSha256PairSchema, artifactMd5PairSchema, @@ -61,6 +62,7 @@ const inspectionContradictionBaseSchema = z.strictObject({ const inspectionContradictionSchema = z.union([ inspectionContradictionBaseSchema.extend(artifactSha256PairSchema.shape), inspectionContradictionBaseSchema.extend(artifactMd5PairSchema.shape), + inspectionContradictionBaseSchema.extend(artifactDfuCrc32PairSchema.shape), ]); const nextProbeSchema = z.strictObject({ @@ -307,7 +309,10 @@ const contradictions = ( ): ArtifactInspectionResult["contradictions"] => inventory.integrity_contradictions.map((value) => { const semantic = { - statement: `Declared integrity for ${value.logical_path} contradicts observed bytes.`, + statement: + value.logical_path === "." + ? "The container's declared integrity contradicts its observed bytes." + : `Declared integrity for ${value.logical_path} contradicts observed bytes.`, occurrence_id: value.occurrence_id, ...artifactIntegrityPairOf(value), evidence_id: evidenceId, @@ -340,9 +345,17 @@ const probesFor = ( }, ] : []; - if (format === "intel-hex" || format === "srec") + if ( + format === "intel-hex" || + format === "srec" || + format === "uf2" || + format === "dfuse" + ) return inventory.occurrences.some( - (row) => row.format_metadata?.kind === "hex-record-region", + (row) => + row.format_metadata?.kind === "hex-record-region" || + row.format_metadata?.kind === "uf2-region" || + row.format_metadata?.kind === "dfuse-element", ) ? [ { diff --git a/src/domain/artifactIntegrity.ts b/src/domain/artifactIntegrity.ts index 3fc3904ddf..e1cad5fce2 100644 --- a/src/domain/artifactIntegrity.ts +++ b/src/domain/artifactIntegrity.ts @@ -2,6 +2,7 @@ import { z } from "zod"; const sha256 = z.string().regex(/^[a-f0-9]{64}$/u); const md5 = z.string().regex(/^[a-f0-9]{32}$/u); +const crc32 = z.string().regex(/^[a-f0-9]{8}$/u); /** A container's claim is distinct from the SHA-256 identity of observed bytes. */ export const artifactIntegrityClaimSchema = z.discriminatedUnion("algorithm", [ @@ -22,9 +23,23 @@ export const artifactMd5PairSchema = z.object({ declared_md5: md5, observed_md5: md5, }); +/** + * A container's DFU suffix checksum over its own bytes before `dwCRC`: CRC-32 + * without the final inversion, as the suffix stores it. + */ +export const artifactDfuCrc32PairSchema = z.object({ + algorithm: z.literal("dfu-crc32"), + declared_crc32: crc32, + observed_crc32: crc32, +}); +/** A checksum a container declares over its own bytes, with the observed value. */ +export type ArtifactContainerIntegrityPair = z.infer< + typeof artifactDfuCrc32PairSchema +>; export type ArtifactIntegrityPair = | z.infer - | z.infer; + | z.infer + | ArtifactContainerIntegrityPair; /** The canonical portable fields shared by inventory and inspection findings. */ export const artifactIntegrityPair = ( @@ -38,17 +53,24 @@ export const artifactIntegrityPair = ( /** Select only digest facts from an authenticated inventory contradiction. */ export const artifactIntegrityPairOf = ( pair: ArtifactIntegrityPair, -): ArtifactIntegrityPair => - "declared_sha256" in pair +): ArtifactIntegrityPair => { + if ("declared_sha256" in pair) + return { + declared_sha256: pair.declared_sha256, + observed_sha256: pair.observed_sha256, + }; + return pair.algorithm === "md5" ? { - declared_sha256: pair.declared_sha256, - observed_sha256: pair.observed_sha256, - } - : { algorithm: pair.algorithm, declared_md5: pair.declared_md5, observed_md5: pair.observed_md5, + } + : { + algorithm: pair.algorithm, + declared_crc32: pair.declared_crc32, + observed_crc32: pair.observed_crc32, }; +}; export type ArtifactIntegrityDetails = { readonly logicalPath: string; @@ -63,6 +85,11 @@ export type ArtifactIntegrityDetails = { readonly declaredMd5: string; readonly calculatedMd5: string | null; } + | { + readonly algorithm: "dfu-crc32"; + readonly declaredCrc32: string; + readonly calculatedCrc32: string | null; + } ); /** Preserve the existing error carrier while naming non-SHA digest algorithms. */ @@ -83,19 +110,41 @@ export const artifactIntegrityDetails = ( : { declaredSha256: claim?.digest ?? null, calculatedSha256: observed }), }); +/** Error details for a container checksum that disagrees with its own bytes. */ +export const artifactContainerIntegrityDetails = ( + logicalPath: string, + pair: ArtifactContainerIntegrityPair, +): ArtifactIntegrityDetails => ({ + logicalPath, + unpacked: false, + algorithm: pair.algorithm, + declaredCrc32: pair.declared_crc32, + calculatedCrc32: pair.observed_crc32, +}); + +const projectedDigests = (details: ArtifactIntegrityDetails) => { + if ("declaredSha256" in details) + return { + declared_sha256: details.declaredSha256, + calculated_sha256: details.calculatedSha256, + }; + return details.algorithm === "md5" + ? { + algorithm: details.algorithm, + declared_md5: details.declaredMd5, + calculated_md5: details.calculatedMd5, + } + : { + algorithm: details.algorithm, + declared_crc32: details.declaredCrc32, + calculated_crc32: details.calculatedCrc32, + }; +}; + export const projectArtifactIntegrityDetails = ( details: ArtifactIntegrityDetails, ) => ({ logical_path: details.logicalPath, - ...("declaredSha256" in details - ? { - declared_sha256: details.declaredSha256, - calculated_sha256: details.calculatedSha256, - } - : { - algorithm: details.algorithm, - declared_md5: details.declaredMd5, - calculated_md5: details.calculatedMd5, - }), + ...projectedDigests(details), unpacked: details.unpacked, }); diff --git a/src/domain/binaryTargetTypes.ts b/src/domain/binaryTargetTypes.ts index eec169b766..743a92c9a2 100644 --- a/src/domain/binaryTargetTypes.ts +++ b/src/domain/binaryTargetTypes.ts @@ -299,6 +299,8 @@ export type BinaryTarget = | "intel-hex" | "srec" | "esp-image" + | "uf2" + | "dfuse" | "fit"; }) | (BinaryTargetIdentity & diff --git a/src/domain/evidenceSubject.ts b/src/domain/evidenceSubject.ts index 2c0457049e..5c0994b32c 100644 --- a/src/domain/evidenceSubject.ts +++ b/src/domain/evidenceSubject.ts @@ -46,6 +46,8 @@ export const evidenceSubjectFormatSchema = z.enum([ "intel-hex", "srec", "esp-image", + "uf2", + "dfuse", "esp-flash-image", "fit", ]); diff --git a/src/domain/firmware/addressRuns.ts b/src/domain/firmware/addressRuns.ts new file mode 100644 index 0000000000..7eb778460f --- /dev/null +++ b/src/domain/firmware/addressRuns.ts @@ -0,0 +1,121 @@ +import { err, ok, type Result } from "../result.js"; + +/** Bytes written at consecutive addresses, held at `offset` in a decoder arena. */ +export interface ArenaRun { + readonly address: number; + readonly offset: number; + readonly length: number; +} + +/** One address-contiguous range and the runs that wrote it, in address order. */ +export interface MergedRuns { + readonly start: number; + /** The range's bytes; its inclusive end is `start + bytes.length - 1`. */ + readonly bytes: Uint8Array; + readonly runs: readonly Run[]; +} + +/** The first address two overlapping runs write with different values. */ +export interface RunConflict { + readonly address: number; + /** The earlier covering run in address order, which supplied the kept byte. */ + readonly writer: Run; + readonly run: Run; + readonly existing: number; + readonly incoming: number; +} + +const runEnd = (run: ArenaRun | undefined): number => + run === undefined ? 0 : run.address + run.length; + +/** + * Merge runs into address-contiguous ranges, never filling gaps. Runs in + * different address spaces never merge; `space` orders them before address. + * Overlapping runs must repeat the same bytes, which count as duplicates. + */ +export const mergeArenaRuns = ( + runs: readonly Run[], + arena: Uint8Array, + space: (run: Run) => number = () => 0, +): Result< + { readonly ranges: MergedRuns[]; readonly duplicateBytes: number }, + RunConflict +> => { + const ordered = [...runs].sort( + (left, right) => space(left) - space(right) || left.address - right.address, + ); + const ranges: MergedRuns[] = []; + let duplicateBytes = 0; + let first = 0; + while (first < ordered.length) { + const head = ordered[first]; + if (head === undefined) break; + let next = first + 1; + let end = runEnd(head); + while (next < ordered.length) { + const candidate = ordered[next]; + if ( + candidate === undefined || + space(candidate) !== space(head) || + candidate.address > end + ) + break; + end = Math.max(end, runEnd(candidate)); + next += 1; + } + const range = mergeRange(ordered.slice(first, next), end, arena); + if (!range.ok) return range; + ranges.push(range.value.range); + duplicateBytes += range.value.duplicateBytes; + first = next; + } + return ok({ ranges, duplicateBytes }); +}; + +/** Copy one contiguous group; written addresses always form [start, written). */ +const mergeRange = ( + group: readonly Run[], + end: number, + arena: Uint8Array, +): Result< + { readonly range: MergedRuns; readonly duplicateBytes: number }, + RunConflict +> => { + const start = group[0]?.address ?? 0; + const bytes = new Uint8Array(end - start); + let written = start; + let duplicateBytes = 0; + for (const run of group) { + const overlapEnd = Math.min(written, runEnd(run)); + for (let address = run.address; address < overlapEnd; address += 1) { + const existing = bytes[address - start] ?? 0; + const incoming = arena[run.offset + address - run.address] ?? 0; + if (existing !== incoming) + return err({ + address, + writer: + group.find( + (candidate) => + candidate !== run && + candidate.address <= address && + address < runEnd(candidate), + ) ?? run, + run, + existing, + incoming, + }); + } + duplicateBytes += Math.max(overlapEnd - run.address, 0); + if (runEnd(run) > written) { + bytes.set( + arena.subarray( + run.offset + written - run.address, + run.offset + run.length, + ), + written - start, + ); + written = runEnd(run); + } + } + return ok({ range: { start, bytes, runs: group }, duplicateBytes }); +}; diff --git a/src/domain/firmware/dfuse.fixture.ts b/src/domain/firmware/dfuse.fixture.ts new file mode 100644 index 0000000000..6d874f5845 --- /dev/null +++ b/src/domain/firmware/dfuse.fixture.ts @@ -0,0 +1,93 @@ +import { crc32 } from "node:zlib"; + +export interface DfuSeFixtureElement { + readonly address: number; + readonly data: Uint8Array | readonly number[]; + /** A dwElementSize that differs from the data, for malformed elements. */ + readonly declaredSize?: number; +} + +export interface DfuSeFixtureTarget { + readonly alternateSetting: number; + /** Omitted or null writes bTargetNamed 0; raw bytes cover non-text names. */ + readonly name?: string | Uint8Array | null; + readonly elements: readonly DfuSeFixtureElement[]; + readonly declaredSize?: number; + readonly declaredElements?: number; + readonly signature?: string; +} + +export interface DfuSeFixture { + readonly targets: readonly DfuSeFixtureTarget[]; + readonly vendorId?: number; + readonly productId?: number; + readonly deviceVersion?: number; + readonly declaredTargets?: number; + /** Replaces DFUImageSize, which otherwise counts the prefix and targets. */ + readonly imageSize?: number; + /** Count the suffix in DFUImageSize too, as dfu-util's dfuse-pack.py does. */ + readonly imageSizeIncludesSuffix?: boolean; + /** Bytes between the last target and the suffix. */ + readonly gap?: Uint8Array; + readonly dfuVersion?: number; + /** Replaces the computed dwCRC. */ + readonly crc?: number; +} + +const targetBytes = (target: DfuSeFixtureTarget): Buffer => { + const elements = Buffer.concat( + target.elements.map((element) => { + const data = Uint8Array.from(element.data); + const header = Buffer.alloc(8); + header.writeUInt32LE(element.address, 0); + header.writeUInt32LE(element.declaredSize ?? data.length, 4); + return Buffer.concat([header, data]); + }), + ); + const prefix = Buffer.alloc(274); + prefix.write(target.signature ?? "Target", 0, "latin1"); + prefix.writeUInt8(target.alternateSetting, 6); + if (target.name !== undefined && target.name !== null) { + prefix.writeUInt32LE(1, 7); + const name = + typeof target.name === "string" + ? Buffer.from(target.name, "latin1") + : Buffer.from(target.name); + name.copy(prefix, 11, 0, Math.min(name.length, 255)); + } + prefix.writeUInt32LE(target.declaredSize ?? elements.length, 266); + prefix.writeUInt32LE(target.declaredElements ?? target.elements.length, 270); + return Buffer.concat([prefix, elements]); +}; + +/** + * A DfuSe file written from ST UM0391 and the DFU 1.1 suffix, with the suffix + * CRC computed as dfu-util's dfuse-pack.py does: the inverted zlib CRC-32. + * DFUImageSize excludes the suffix unless `imageSizeIncludesSuffix` is set. + */ +export const dfuseFile = (fixture: DfuSeFixture): Buffer => { + const targets = Buffer.concat(fixture.targets.map(targetBytes)); + const gap = Buffer.from(fixture.gap ?? []); + const prefix = Buffer.alloc(11); + prefix.write("DfuSe", 0, "latin1"); + prefix.writeUInt8(1, 5); + prefix.writeUInt32LE( + fixture.imageSize ?? + prefix.length + + targets.length + + gap.length + + (fixture.imageSizeIncludesSuffix === true ? 16 : 0), + 6, + ); + prefix.writeUInt8(fixture.declaredTargets ?? fixture.targets.length, 10); + const suffix = Buffer.alloc(16); + suffix.writeUInt16LE(fixture.deviceVersion ?? 0xffff, 0); + suffix.writeUInt16LE(fixture.productId ?? 0xdf11, 2); + suffix.writeUInt16LE(fixture.vendorId ?? 0x0483, 4); + suffix.writeUInt16LE(fixture.dfuVersion ?? 0x011a, 6); + suffix.write("UFD", 8, "latin1"); + suffix.writeUInt8(16, 11); + const body = Buffer.concat([prefix, targets, gap, suffix.subarray(0, 12)]); + suffix.writeUInt32LE(fixture.crc ?? ~crc32(body) >>> 0, 12); + return Buffer.concat([body, suffix.subarray(12)]); +}; diff --git a/src/domain/firmware/dfuse.test.ts b/src/domain/firmware/dfuse.test.ts new file mode 100644 index 0000000000..b174392654 --- /dev/null +++ b/src/domain/firmware/dfuse.test.ts @@ -0,0 +1,406 @@ +import { createHash } from "node:crypto"; + +import { expect, it } from "vitest"; + +import { + DFUSE_MAX_FILE_BYTES, + createDfuSeDecoder, + isDfuSeFile, + type DfuSeImage, +} from "./dfuse.js"; +import { dfuseFile, type DfuSeFixture } from "./dfuse.fixture.js"; + +const decode = (bytes: Uint8Array, chunkSize = 64) => { + const decoder = createDfuSeDecoder(bytes.length); + if (!decoder.ok) throw new Error(decoder.error.message); + for (let offset = 0; offset < bytes.length; offset += chunkSize) { + const written = decoder.value.write( + bytes.subarray(offset, offset + chunkSize), + ); + if (!written.ok) return written; + } + return decoder.value.finish(); +}; + +const image = (bytes: Uint8Array): DfuSeImage => { + const decoded = decode(bytes); + if (!decoded.ok) throw new Error(decoded.error.message); + return decoded.value; +}; + +const recognized = (bytes: Uint8Array): boolean => + isDfuSeFile(bytes.subarray(0, 4096), bytes.subarray(-16), bytes.length); + +const elementFacts = ({ elements }: DfuSeImage) => + elements.map(({ bytes, ...facts }) => ({ ...facts, length: bytes.length })); + +/** One 16-byte image at 0x08000000 in target "ST...", as the packers below write it. */ +const DFUSE_PACK_EQUIVALENT: DfuSeFixture = { + targets: [ + { + alternateSetting: 0, + name: "ST...", + elements: [{ address: 0x0800_0000, data: [...Array(16).keys()] }], + }, + ], + deviceVersion: 0, +}; + +it("decodes files packed with and without the suffix in DFUImageSize", () => { + // Oracle: build() and compute_crc() of dfu-util's dfuse-pack.py at f9a537c9, + // transcribed and run under Python 3. Since ee6b393d it writes DFUImageSize + // as PREFIX_SIZE + len(data) + SUFFIX_SIZE, the whole file. + const packed = dfuseFile({ + ...DFUSE_PACK_EQUIVALENT, + imageSizeIncludesSuffix: true, + }); + expect(createHash("sha256").update(packed).digest("hex")).toBe( + "4d9f8c231291504484745f0b0586031c8f804d141e964cb33a9b82965f8a69d0", + ); + expect(image(packed).metadata).toMatchObject({ + image_size: 325, + suffix_crc: 0x2c4c_4f44, + computed_crc: 0x2c4c_4f44, + }); + const bytes = dfuseFile(DFUSE_PACK_EQUIVALENT); + // Oracle: build() and compute_crc() of MicroPython's tools/dfu.py at + // cb99ca98, transcribed and run under Python 3. It writes len(data) + 11, + // leaving the suffix out, as dfuse-pack.py did before ee6b393d. + expect(bytes.length).toBe(325); + expect(createHash("sha256").update(bytes).digest("hex")).toBe( + "65ad96452f1f82cc8b44d30dfe94f268f4fd230a32b30a8826a639b269f7034d", + ); + const decoded = image(bytes); + expect(decoded.metadata).toEqual({ + kind: "dfuse", + image_size: 309, + target_count: 1, + element_count: 1, + empty_element_count: 0, + overlapping_element_count: 0, + data_bytes: 16, + device_version: 0, + product_id: 0xdf11, + vendor_id: 0x0483, + suffix_crc: 0x7a37_11da, + computed_crc: 0x7a37_11da, + }); + expect(elementFacts(decoded)).toEqual([ + { + targetIndex: 0, + alternateSetting: 0, + targetName: "ST...", + elementIndex: 0, + address: 0x0800_0000, + dataOffset: 293, + length: 16, + }, + ]); + expect(decoded.limitations).toEqual([]); +}); + +it("keeps each target's alternate setting and name on its elements", () => { + const decoded = image( + dfuseFile({ + targets: [ + { + alternateSetting: 0, + name: "Internal Flash", + elements: [ + { address: 0x0800_0000, data: [1, 2, 3, 4] }, + { address: 0x0800_4000, data: [5, 6] }, + ], + }, + { + alternateSetting: 1, + elements: [{ address: 0x1fff_c000, data: [0xaa, 0x55] }], + }, + { + alternateSetting: 2, + name: Uint8Array.from([0x4f, 0xe9, 0, 0x58]), + elements: [{ address: 0x1fff_7800, data: [] }], + }, + ], + }), + ); + expect( + decoded.elements.map((element) => [ + element.targetIndex, + element.alternateSetting, + element.targetName, + element.elementIndex, + element.address, + [...element.bytes], + ]), + ).toEqual([ + [0, 0, "Internal Flash", 0, 0x0800_0000, [1, 2, 3, 4]], + [0, 0, "Internal Flash", 1, 0x0800_4000, [5, 6]], + [1, 1, null, 0, 0x1fff_c000, [0xaa, 0x55]], + ]); + expect(decoded.metadata).toMatchObject({ + target_count: 3, + element_count: 4, + empty_element_count: 1, + data_bytes: 8, + }); + // A named target's field is read to its first NUL, one character per byte. + const named = image( + dfuseFile({ + targets: [ + { + alternateSetting: 3, + name: Uint8Array.from([0x4f, 0xe9, 0, 0x58]), + elements: [{ address: 0, data: [1] }], + }, + ], + }), + ); + expect(named.elements[0]?.targetName).toBe("Oé"); +}); + +it("reports a suffix CRC mismatch as facts without refusing the file", () => { + const decoded = image(dfuseFile({ ...DFUSE_PACK_EQUIVALENT, crc: 0 })); + expect(decoded.metadata).toMatchObject({ + suffix_crc: 0, + computed_crc: 0x7a37_11da, + }); + expect(decoded.elements).toHaveLength(1); +}); + +it("names overlapping elements within one alternate setting only", () => { + const decoded = image( + dfuseFile({ + targets: [ + { + alternateSetting: 0, + elements: [ + { address: 0x0800_0000, data: [1, 2, 3, 4] }, + { address: 0x0800_0002, data: [3, 4] }, + ], + }, + { + alternateSetting: 1, + elements: [{ address: 0x0800_0000, data: [9] }], + }, + { + alternateSetting: 0, + elements: [{ address: 0x0800_0003, data: [4, 5] }], + }, + ], + }), + ); + expect(decoded.metadata.overlapping_element_count).toBe(2); + expect(decoded.limitations).toEqual([ + "2 DfuSe element(s) overlap an earlier element of the same alternate setting; each stays a separate entry and no bytes were merged: target 0 element 1 (0x08000002-0x08000003) overlaps target 0 element 0 (0x08000000-0x08000003) in alternate setting 0; target 2 element 0 (0x08000003-0x08000004) overlaps target 0 element 0 (0x08000000-0x08000003) in alternate setting 0.", + ]); +}); + +it("accepts boundary declarations beside the malformed ones", () => { + // A valid file may hold no targets at all. + expect(image(dfuseFile({ targets: [] })).metadata).toMatchObject({ + image_size: 11, + target_count: 0, + element_count: 0, + }); + const name = Uint8Array.from({ length: 255 }, () => 0x41); + const decoded = image( + dfuseFile({ + targets: [ + { + alternateSetting: 4, + // A name filling all 255 bytes has no NUL. + name, + elements: [ + // An element may end exactly at the top of the address space. + { address: 0xffff_fffe, data: [1, 2] }, + { address: 0x2000_0000, data: [] }, + ], + }, + { alternateSetting: 5, elements: [] }, + ], + }), + ); + expect(decoded.elements.map((element) => element.targetName)).toEqual([ + "A".repeat(255), + ]); + expect(decoded.metadata).toMatchObject({ + target_count: 2, + element_count: 2, + empty_element_count: 1, + }); + expect(decoded.limitations).toEqual([ + "DfuSe declarations without bytes have no entry: target 0 element 1 at 0x20000000 (alternate setting 4) holds no bytes; target 1 (alternate setting 5) declares no elements.", + ]); +}); + +it("recognizes a file only by the DfuSe prefix together with the DFU suffix", () => { + const bytes = dfuseFile(DFUSE_PACK_EQUIVALENT); + expect(recognized(bytes)).toBe(true); + const changed = (offset: number, value: number): Buffer => { + const copy = Buffer.from(bytes); + copy[offset] = value; + return copy; + }; + expect(recognized(changed(5, 2))).toBe(false); + expect(recognized(changed(0, 0x64))).toBe(false); + expect(recognized(changed(bytes.length - 10, 0x00))).toBe(false); + expect(recognized(changed(bytes.length - 8, 0x58))).toBe(false); + expect(recognized(changed(bytes.length - 5, 15))).toBe(false); + expect(recognized(bytes.subarray(0, bytes.length - 1))).toBe(false); + // The CRC is checked after recognition, so a mismatch still recognizes. + expect(recognized(dfuseFile({ ...DFUSE_PACK_EQUIVALENT, crc: 1 }))).toBe( + true, + ); +}); + +it.each([ + [ + "a target without its signature", + { + targets: [{ alternateSetting: 0, signature: "Tarxet", elements: [] }], + }, + 'byte offset 11: target 0 does not begin with the "Target" signature', + ], + [ + "an element larger than its target", + { + targets: [ + { + alternateSetting: 0, + elements: [{ address: 0, data: [1, 2], declaredSize: 3 }], + }, + ], + }, + "byte offset 289: target 0 element 0 declares 3 bytes, but its target ends at byte 295", + ], + [ + "a target larger than the image", + { + targets: [ + { + alternateSetting: 0, + declaredSize: 100, + elements: [{ address: 0, data: [1] }], + }, + ], + }, + "byte offset 277: target 0 declares 100 bytes, but the image ends at byte 294", + ], + [ + "fewer elements than the target size", + { + targets: [ + { + alternateSetting: 0, + declaredElements: 0, + elements: [{ address: 0, data: [1] }], + }, + ], + }, + "byte offset 285: target 0 declares 9 bytes, but its 0 element(s) end 9 bytes earlier", + ], + [ + "an image size that matches neither packer convention", + { targets: [], imageSize: 12 }, + "byte offset 6: DFUImageSize declares 12 bytes, but the DFU suffix begins at byte 11 and the file holds 27 bytes; the suffix CRC matches the file", + ], + [ + "bytes between the last target and the suffix", + { targets: [], gap: Uint8Array.from([0]) }, + "byte offset 11: 0 target(s) end at byte 11, but the DFU suffix begins at byte 12", + ], + [ + "more targets than the image holds", + { targets: [], declaredTargets: 1 }, + "byte offset 11: target 0 prefix needs 274 bytes, but the image ends at byte 11", + ], + [ + "more elements than the target holds", + { + targets: [ + { + alternateSetting: 0, + declaredElements: 2, + elements: [{ address: 0, data: [1] }], + }, + ], + }, + "byte offset 294: target 0 element 1 header lies beyond the target's end at byte 294", + ], + [ + "an element that wraps the address space", + { + targets: [ + { + alternateSetting: 0, + elements: [{ address: 0xffff_ffff, data: [1, 2] }], + }, + ], + }, + "byte offset 285: target 0 element 0 at 0xffffffff with 2 bytes wraps past 0xffffffff", + ], +] as const)("refuses %s with its byte offset", (_name, fixture, message) => { + expect(decode(dfuseFile(fixture))).toEqual({ + ok: false, + error: { kind: "format", message }, + }); +}); + +it("refuses files above the fixed ceiling and more elements than retained", () => { + expect(createDfuSeDecoder(DFUSE_MAX_FILE_BYTES + 1)).toMatchObject({ + ok: false, + error: { kind: "limit", boundary: "file-size" }, + }); + const elements = (count: number, data: readonly number[]) => + Array.from({ length: count }, (_, index) => ({ + address: index * 2, + data, + })); + const accepted = decode( + dfuseFile({ + targets: [{ alternateSetting: 0, elements: elements(65_536, [1]) }], + }), + 1 << 20, + ); + if (!accepted.ok) throw new Error(accepted.error.message); + expect(accepted.value.elements).toHaveLength(65_536); + expect( + decode( + dfuseFile({ + targets: [{ alternateSetting: 0, elements: elements(65_537, [1]) }], + }), + 1 << 20, + ), + ).toMatchObject({ + ok: false, + error: { kind: "limit", boundary: "capacity" }, + }); +}); + +it("counts empty elements past the entry cap and names only the first four", () => { + // Empty elements have no entry, so they are not capped; 8 header bytes each + // let a file hold millions, and only a count and four descriptions are kept. + const empty = Array.from({ length: 65_537 }, (_, index) => ({ + address: index * 2, + data: [], + })); + const decoded = decode( + dfuseFile({ + targets: [ + { alternateSetting: 0, elements: empty }, + { alternateSetting: 1, elements: [] }, + ], + }), + 1 << 20, + ); + if (!decoded.ok) throw new Error(decoded.error.message); + expect(decoded.value.elements).toEqual([]); + expect(decoded.value.metadata).toMatchObject({ + element_count: 65_537, + empty_element_count: 65_537, + data_bytes: 0, + }); + expect(decoded.value.limitations).toEqual([ + "DfuSe declarations without bytes have no entry: target 0 element 0 at 0x00000000 (alternate setting 0) holds no bytes; target 0 element 1 at 0x00000002 (alternate setting 0) holds no bytes; target 0 element 2 at 0x00000004 (alternate setting 0) holds no bytes; target 0 element 3 at 0x00000006 (alternate setting 0) holds no bytes; and 65534 more.", + ]); +}); diff --git a/src/domain/firmware/dfuse.ts b/src/domain/firmware/dfuse.ts new file mode 100644 index 0000000000..3a0f01dfb3 --- /dev/null +++ b/src/domain/firmware/dfuse.ts @@ -0,0 +1,412 @@ +import { crc32 } from "node:zlib"; + +import type { DfuSeMetadata } from "../artifactFormatMetadata.js"; +import { err, ok, type Result } from "../result.js"; +import type { RegionDecodeFailure, RegionDecoder } from "./regionDecoder.js"; + +/** "DfuSe", bVersion, DFUImageSize and bTargets (ST UM0391). */ +const PREFIX_BYTES = 11; +/** "Target", bAlternateSetting, bTargetNamed, szTargetName, dwTargetSize, dwNbElements. */ +const TARGET_PREFIX_BYTES = 274; +const TARGET_NAME_OFFSET = 11; +const TARGET_NAME_BYTES = 255; +/** dwElementAddress and dwElementSize. */ +const ELEMENT_HEADER_BYTES = 8; +/** bcdDevice, idProduct, idVendor, bcdDFU, "UFD", bLength and dwCRC (DFU 1.1). */ +export const DFUSE_SUFFIX_BYTES = 16; +const DFUSE_VERSION = 1; +const DFU_SUFFIX_VERSION = 0x011a; +const ADDRESS_SPACE = 0x1_0000_0000; +/** Elements named in one limitation before the rest are counted. */ +const NAMED_ELEMENTS = 4; +const SIGNATURE = [0x44, 0x66, 0x75, 0x53, 0x65]; +const TARGET_SIGNATURE = [0x54, 0x61, 0x72, 0x67, 0x65, 0x74]; +const SUFFIX_SIGNATURE = [0x55, 0x46, 0x44]; + +/** + * Fixed whole-file ceiling, as for Intel HEX; configurable ceilings are + * tracked in #6176. The decoder holds the file once and slices elements from it. + */ +export const DFUSE_MAX_FILE_BYTES = 128 * 1024 * 1024; +/** Non-empty elements retained as inventory entries, as for UF2 regions. */ +export const DFUSE_MAX_ELEMENTS = 65_536; + +/** One element: bytes the file declares for an address in one alternate setting. */ +export interface DfuSeElement { + readonly targetIndex: number; + readonly alternateSetting: number; + /** szTargetName up to its first NUL, one character per byte; null when unnamed. */ + readonly targetName: string | null; + readonly elementIndex: number; + readonly address: number; + /** The file offset of the element data. */ + readonly dataOffset: number; + readonly bytes: Uint8Array; +} + +export interface DfuSeImage { + readonly metadata: DfuSeMetadata; + /** In file order; empty elements are counted in metadata but not listed. */ + readonly elements: readonly DfuSeElement[]; + readonly limitations: readonly string[]; +} + +const readUint16 = (bytes: Uint8Array, offset: number): number => + (bytes[offset] ?? 0) | ((bytes[offset + 1] ?? 0) << 8); + +const readUint32 = (bytes: Uint8Array, offset: number): number => + (readUint16(bytes, offset) | (readUint16(bytes, offset + 2) << 16)) >>> 0; + +const startsWith = ( + bytes: Uint8Array, + offset: number, + signature: readonly number[], +): boolean => signature.every((byte, index) => bytes[offset + index] === byte); + +const hex32 = (value: number): string => + `0x${value.toString(16).padStart(8, "0")}`; + +/** Whether the leading bytes begin a DfuSe version 1 image. */ +export const hasDfuSePrefix = (prefix: Uint8Array): boolean => + startsWith(prefix, 0, SIGNATURE) && prefix[5] === DFUSE_VERSION; + +/** Whether the final 16 bytes are a DFU 1.1a suffix, as DfuSe requires. */ +const hasDfuSeSuffix = (suffix: Uint8Array): boolean => + suffix.length === DFUSE_SUFFIX_BYTES && + readUint16(suffix, 6) === DFU_SUFFIX_VERSION && + startsWith(suffix, 8, SUFFIX_SIGNATURE) && + suffix[11] === DFUSE_SUFFIX_BYTES; + +/** + * Recognize a file only by the DfuSe prefix and the DFU suffix together. The + * suffix CRC is not part of recognition: a mismatch is an integrity fact. + */ +export const isDfuSeFile = ( + prefix: Uint8Array, + suffix: Uint8Array, + fileBytes: number, +): boolean => + fileBytes >= PREFIX_BYTES + DFUSE_SUFFIX_BYTES && + hasDfuSePrefix(prefix) && + hasDfuSeSuffix(suffix); + +/** Start a decoder that holds at most the admitted file size. */ +export const createDfuSeDecoder = ( + fileBytes: number, +): Result, RegionDecodeFailure> => + Number.isSafeInteger(fileBytes) && + fileBytes >= 0 && + fileBytes <= DFUSE_MAX_FILE_BYTES + ? ok(new RetainingDfuSeDecoder(fileBytes)) + : err({ + kind: "limit", + boundary: "file-size", + message: `file is ${String(fileBytes)} bytes, exceeding the fixed ${String(DFUSE_MAX_FILE_BYTES)}-byte (128 MiB) ceiling`, + }); + +/** The DFU suffix CRC: CRC-32 without the final inversion, over every byte before dwCRC. */ +const dfuCrc = (bytes: Uint8Array): number => ~crc32(bytes) >>> 0; + +const formatFailure = ( + offset: number, + detail: string, +): RegionDecodeFailure => ({ + kind: "format", + message: `byte offset ${String(offset)}: ${detail}`, +}); + +/** At most NAMED_ELEMENTS descriptions, then a count of the rest of `total`. */ +const named = (items: readonly string[], total = items.length): string => + `${items.slice(0, NAMED_ELEMENTS).join("; ")}${total > NAMED_ELEMENTS ? `; and ${String(total - NAMED_ELEMENTS)} more` : ""}`; + +class RetainingDfuSeDecoder implements RegionDecoder { + readonly #bytes: Uint8Array; + #length = 0; + + constructor(fileBytes: number) { + this.#bytes = new Uint8Array(fileBytes); + } + + write(chunk: Uint8Array): Result { + if (this.#length + chunk.length > this.#bytes.length) + return err({ + kind: "limit", + boundary: "file-size", + message: "file grew beyond its admitted size while being read", + }); + this.#bytes.set(chunk, this.#length); + this.#length += chunk.length; + return ok(undefined); + } + + finish(): Result { + return parseDfuSe(this.#bytes.subarray(0, this.#length)); + } +} + +const parseDfuSe = ( + bytes: Uint8Array, +): Result => { + const suffixStart = bytes.length - DFUSE_SUFFIX_BYTES; + if ( + bytes.length < PREFIX_BYTES + DFUSE_SUFFIX_BYTES || + !hasDfuSePrefix(bytes) || + !hasDfuSeSuffix(bytes.subarray(suffixStart)) + ) + return err( + formatFailure( + 0, + "the file does not begin with a DfuSe prefix and end with a DFU 1.1a suffix", + ), + ); + // UM0391 calls DFUImageSize the file length, which dfu-util's dfuse-pack.py + // writes since ee6b393d (2020-10-08). Earlier dfuse-pack.py and MicroPython's + // tools/dfu.py write the length before the suffix. + const imageSize = readUint32(bytes, 6); + const suffixCrc = readUint32(bytes, suffixStart + 12); + const computedCrc = dfuCrc(bytes.subarray(0, bytes.length - 4)); + if (imageSize !== suffixStart && imageSize !== bytes.length) + return err( + formatFailure( + 6, + `DFUImageSize declares ${String(imageSize)} bytes, but the DFU suffix begins at byte ${String(suffixStart)} and the file holds ${String(bytes.length)} bytes; the suffix CRC ${suffixCrc === computedCrc ? "matches" : "does not match"} the file`, + ), + ); + const targetCount = bytes[10] ?? 0; + const walk = new TargetWalk(bytes, suffixStart); + let offset = PREFIX_BYTES; + for (let targetIndex = 0; targetIndex < targetCount; targetIndex += 1) { + const next = walk.target(targetIndex, offset); + if (!next.ok) return next; + offset = next.value; + } + if (offset !== suffixStart) + return err( + formatFailure( + offset, + `${String(targetCount)} target(s) end at byte ${String(offset)}, but the DFU suffix begins at byte ${String(suffixStart)}`, + ), + ); + const { elements } = walk; + const overlaps = overlappingElements(elements); + return ok({ + metadata: { + kind: "dfuse", + image_size: imageSize, + target_count: targetCount, + element_count: walk.elementCount, + empty_element_count: walk.emptyElements, + overlapping_element_count: overlaps.length, + data_bytes: elements.reduce( + (sum, element) => sum + element.bytes.length, + 0, + ), + device_version: readUint16(bytes, suffixStart), + product_id: readUint16(bytes, suffixStart + 2), + vendor_id: readUint16(bytes, suffixStart + 4), + suffix_crc: suffixCrc, + computed_crc: computedCrc, + }, + elements, + limitations: [ + ...(overlaps.length === 0 + ? [] + : [ + `${String(overlaps.length)} DfuSe element(s) overlap an earlier element of the same alternate setting; each stays a separate entry and no bytes were merged: ${named(overlaps)}.`, + ]), + ...(walk.unlistedCount === 0 + ? [] + : [ + `DfuSe declarations without bytes have no entry: ${named(walk.unlisted, walk.unlistedCount)}.`, + ]), + ], + }); +}; + +/** Targets and elements in file order; every size is checked against its container. */ +class TargetWalk { + readonly elements: DfuSeElement[] = []; + /** + * The first NAMED_ELEMENTS empty elements and element-less targets, which no + * entry can show. A file can hold millions of empty elements, so the rest + * are only counted in `unlistedCount`. + */ + readonly unlisted: string[] = []; + unlistedCount = 0; + elementCount = 0; + emptyElements = 0; + + constructor( + private readonly bytes: Uint8Array, + private readonly suffixStart: number, + ) {} + + /** Parse one target at `offset`; the result is the offset after its elements. */ + target( + targetIndex: number, + offset: number, + ): Result { + const { bytes } = this; + if (offset + TARGET_PREFIX_BYTES > this.suffixStart) + return err( + formatFailure( + offset, + `target ${String(targetIndex)} prefix needs ${String(TARGET_PREFIX_BYTES)} bytes, but the image ends at byte ${String(this.suffixStart)}`, + ), + ); + if (!startsWith(bytes, offset, TARGET_SIGNATURE)) + return err( + formatFailure( + offset, + `target ${String(targetIndex)} does not begin with the "Target" signature`, + ), + ); + const targetSize = readUint32(bytes, offset + 266); + const declaredElements = readUint32(bytes, offset + 270); + const bodyStart = offset + TARGET_PREFIX_BYTES; + const bodyEnd = bodyStart + targetSize; + if (bodyEnd > this.suffixStart) + return err( + formatFailure( + offset + 266, + `target ${String(targetIndex)} declares ${String(targetSize)} bytes, but the image ends at byte ${String(this.suffixStart)}`, + ), + ); + const target = { + targetIndex, + alternateSetting: bytes[offset + 6] ?? 0, + targetName: + readUint32(bytes, offset + 7) === 0 + ? null + : latin1UntilNul( + bytes.subarray( + offset + TARGET_NAME_OFFSET, + offset + TARGET_NAME_OFFSET + TARGET_NAME_BYTES, + ), + ), + }; + if (declaredElements === 0) + this.#unlisted( + () => + `target ${String(targetIndex)} (alternate setting ${String(target.alternateSetting)}) declares no elements`, + ); + let position = bodyStart; + for (let index = 0; index < declaredElements; index += 1) { + const next = this.#element(target, index, position, bodyEnd); + if (!next.ok) return next; + position = next.value; + } + return position === bodyEnd + ? ok(bodyEnd) + : err( + formatFailure( + position, + `target ${String(targetIndex)} declares ${String(targetSize)} bytes, but its ${String(declaredElements)} element(s) end ${String(bodyEnd - position)} bytes earlier`, + ), + ); + } + + #element( + target: Pick< + DfuSeElement, + "targetIndex" | "alternateSetting" | "targetName" + >, + elementIndex: number, + position: number, + bodyEnd: number, + ): Result { + const name = `target ${String(target.targetIndex)} element ${String(elementIndex)}`; + if (position + ELEMENT_HEADER_BYTES > bodyEnd) + return err( + formatFailure( + position, + `${name} header lies beyond the target's end at byte ${String(bodyEnd)}`, + ), + ); + const address = readUint32(this.bytes, position); + const size = readUint32(this.bytes, position + 4); + const dataOffset = position + ELEMENT_HEADER_BYTES; + if (dataOffset + size > bodyEnd) + return err( + formatFailure( + position + 4, + `${name} declares ${String(size)} bytes, but its target ends at byte ${String(bodyEnd)}`, + ), + ); + if (address + size > ADDRESS_SPACE) + return err( + formatFailure( + position, + `${name} at ${hex32(address)} with ${String(size)} bytes wraps past 0xffffffff`, + ), + ); + if (size > 0 && this.elements.length >= DFUSE_MAX_ELEMENTS) + return err({ + kind: "limit", + boundary: "capacity", + message: `byte offset ${String(position)}: the file holds more than ${String(DFUSE_MAX_ELEMENTS)} elements with bytes, the most retained as entries`, + }); + this.elementCount += 1; + if (size === 0) { + this.emptyElements += 1; + this.#unlisted( + () => + `${name} at ${hex32(address)} (alternate setting ${String(target.alternateSetting)}) holds no bytes`, + ); + } else + this.elements.push({ + ...target, + elementIndex, + address, + dataOffset, + bytes: this.bytes.subarray(dataOffset, dataOffset + size), + }); + return ok(dataOffset + size); + } + + /** Count a declaration without bytes; describe only the first few. */ + #unlisted(describe: () => string): void { + this.unlistedCount += 1; + if (this.unlisted.length < NAMED_ELEMENTS) this.unlisted.push(describe()); + } +} + +/** szTargetName is a fixed field; bytes after the first NUL are padding. */ +const latin1UntilNul = (field: Uint8Array): string => { + const end = field.indexOf(0); + return String.fromCharCode( + ...field.subarray(0, end < 0 ? field.length : end), + ); +}; + +const elementText = (element: DfuSeElement): string => + `target ${String(element.targetIndex)} element ${String(element.elementIndex)} (${hex32(element.address)}-${hex32(element.address + element.bytes.length - 1)})`; + +/** Elements that overlap an earlier one in address order within one alternate setting. */ +const overlappingElements = (elements: readonly DfuSeElement[]): string[] => { + const ordered = [...elements].sort( + (left, right) => + left.alternateSetting - right.alternateSetting || + left.address - right.address, + ); + const overlaps: string[] = []; + let reach: DfuSeElement | undefined; + for (const element of ordered) { + if ( + reach !== undefined && + reach.alternateSetting === element.alternateSetting && + element.address < reach.address + reach.bytes.length + ) + overlaps.push( + `${elementText(element)} overlaps ${elementText(reach)} in alternate setting ${String(element.alternateSetting)}`, + ); + if ( + reach === undefined || + reach.alternateSetting !== element.alternateSetting || + element.address + element.bytes.length > + reach.address + reach.bytes.length + ) + reach = element; + } + return overlaps; +}; diff --git a/src/domain/firmware/hexRecords.ts b/src/domain/firmware/hexRecords.ts index 8b477d428b..d1e8fdd690 100644 --- a/src/domain/firmware/hexRecords.ts +++ b/src/domain/firmware/hexRecords.ts @@ -1,5 +1,7 @@ import type { HexRecordsMetadata } from "../artifactFormatMetadata.js"; import { err, ok, type Result } from "../result.js"; +import { mergeArenaRuns, type RunConflict } from "./addressRuns.js"; +import type { RegionDecodeFailure, RegionDecoder } from "./regionDecoder.js"; import { COLON, CR, @@ -28,13 +30,6 @@ export const HEX_RECORD_ENCODING_LABELS: Readonly< Record > = { "intel-hex": "Intel HEX", srec: "S-record" }; -/** A whole-file refusal; no partial regions accompany it. */ -export interface HexRecordFailure { - readonly kind: "format" | "limit"; - readonly message: string; - readonly boundary?: "file-size" | "capacity"; -} - /** Decoded bytes for one address-contiguous range; `end` is inclusive. */ export interface HexRecordRegion { readonly start: number; @@ -53,13 +48,6 @@ export interface HexRecordImage { readonly limitations: readonly string[]; } -export interface HexRecordDecoder { - /** Decode the next file bytes; a line may span chunks. */ - write(chunk: Uint8Array): Result; - /** Decode a final unterminated line, then merge runs into regions. */ - finish(): Result; -} - /** Consecutive records whose addresses continue each other. */ interface RecordRun { readonly address: number; @@ -99,7 +87,7 @@ export const detectHexRecordEncoding = ( export const createHexRecordDecoder = ( encoding: HexRecordEncoding, fileBytes: number, -): Result => +): Result, RegionDecodeFailure> => Number.isSafeInteger(fileBytes) && fileBytes >= 0 && fileBytes <= HEX_RECORD_MAX_FILE_BYTES @@ -110,12 +98,12 @@ export const createHexRecordDecoder = ( message: `file is ${String(fileBytes)} bytes, exceeding the fixed ${String(HEX_RECORD_MAX_FILE_BYTES)}-byte (128 MiB) ceiling`, }); -const formatFailure = (line: number, detail: string): HexRecordFailure => ({ +const formatFailure = (line: number, detail: string): RegionDecodeFailure => ({ kind: "format", message: `line ${String(line)}: ${detail}`, }); -class StreamingHexRecordDecoder implements HexRecordDecoder { +class StreamingHexRecordDecoder implements RegionDecoder { readonly #encoding: HexRecordEncoding; readonly #line: Uint8Array; readonly #runs: RecordRun[] = []; @@ -143,7 +131,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { this.#arena = new Uint8Array(Math.floor(fileBytes / 2)); } - write(chunk: Uint8Array): Result { + write(chunk: Uint8Array): Result { let from = 0; while (from < chunk.length) { const lf = chunk.indexOf(LF, from); @@ -160,7 +148,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { return ok(undefined); } - finish(): Result { + finish(): Result { if (this.#lineLength > 0) { const ended = this.#endLine(false); if (!ended.ok) return ended; @@ -177,14 +165,14 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { }); } - #overlong(line: number): HexRecordFailure { + #overlong(line: number): RegionDecodeFailure { return formatFailure( line, `line exceeds the ${String(maxLineChars(this.#encoding))}-character ${HEX_RECORD_ENCODING_LABELS[this.#encoding]} maximum`, ); } - #endLine(lfTerminated: boolean): Result { + #endLine(lfTerminated: boolean): Result { this.#lineNumber += 1; let length = this.#lineLength; this.#lineLength = 0; @@ -208,11 +196,11 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { : this.#applySrec(parsed.value); } - #failure(detail: string): HexRecordFailure { + #failure(detail: string): RegionDecodeFailure { return formatFailure(this.#lineNumber, detail); } - #applyIntel(record: HexRecord): Result { + #applyIntel(record: HexRecord): Result { const value = readBigEndian(record.data); switch (record.type) { case "00": @@ -236,7 +224,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { } } - #intelData({ address, data }: HexRecord): Result { + #intelData({ address, data }: HexRecord): Result { this.#dataRecordCount += 1; // The specification wraps within 64 KiB, but some producers continue // linearly, so the address of the wrapped bytes is ambiguous. @@ -256,7 +244,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { return this.#addData(start, data); } - #applySrec(record: HexRecord): Result { + #applySrec(record: HexRecord): Result { switch (record.type) { case "S0": if (this.#header !== undefined) @@ -291,7 +279,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { type, address, data, - }: HexRecord): Result { + }: HexRecord): Result { this.#dataRecordCount += 1; this.#dataTypes.add(type); const limit = SREC_DATA_LIMITS[type] ?? 0; @@ -304,7 +292,10 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { return this.#addData(address, data); } - #declareStart(value: number, record: string): Result { + #declareStart( + value: number, + record: string, + ): Result { const existing = this.#start; if (existing !== undefined && existing.value !== value) return err( @@ -318,7 +309,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { #declareStartSegment( value: readonly [number, number], - ): Result { + ): Result { this.#segmentRecords = true; const existing = this.#startSegment; if ( @@ -334,7 +325,10 @@ class StreamingHexRecordDecoder implements HexRecordDecoder { return ok(undefined); } - #addData(address: number, data: Uint8Array): Result { + #addData( + address: number, + data: Uint8Array, + ): Result { if (data.length === 0) return ok(undefined); if (this.#arenaUsed + data.length > this.#arena.length) return err({ @@ -455,68 +449,14 @@ const mergeRuns = ( arena: Uint8Array, ): Result< { regions: HexRecordRegion[]; duplicateBytes: number }, - HexRecordFailure + RegionDecodeFailure > => { - const ordered = [...runs].sort((left, right) => left.address - right.address); - const regions: HexRecordRegion[] = []; - let duplicateBytes = 0; - let first = 0; - while (first < ordered.length) { - let next = first + 1; - let end = runEnd(ordered[first]); - while (next < ordered.length && (ordered[next]?.address ?? 0) <= end) { - end = Math.max(end, runEnd(ordered[next])); - next += 1; - } - const region = mergeRegion(ordered.slice(first, next), end, arena); - if (!region.ok) return region; - regions.push(region.value.region); - duplicateBytes += region.value.duplicateBytes; - first = next; - } - return ok({ regions, duplicateBytes }); -}; - -const runEnd = (run: RecordRun | undefined): number => - run === undefined ? 0 : run.address + run.length; - -/** Copy one contiguous group; written addresses always form [start, written). */ -const mergeRegion = ( - group: readonly RecordRun[], - end: number, - arena: Uint8Array, -): Result< - { region: HexRecordRegion; duplicateBytes: number }, - HexRecordFailure -> => { - const start = group[0]?.address ?? 0; - const bytes = new Uint8Array(end - start); - let written = start; - let duplicateBytes = 0; - for (const run of group) { - const overlapEnd = Math.min(written, runEnd(run)); - for (let address = run.address; address < overlapEnd; address += 1) { - const existing = bytes[address - start] ?? 0; - const incoming = arena[run.offset + address - run.address] ?? 0; - if (existing !== incoming) - return err(conflict(group, run, address, [existing, incoming])); - } - duplicateBytes += Math.max(overlapEnd - run.address, 0); - if (runEnd(run) > written) { - bytes.set( - arena.subarray( - run.offset + written - run.address, - run.offset + run.length, - ), - written - start, - ); - written = runEnd(run); - } - } + const merged = mergeArenaRuns(runs, arena); + if (!merged.ok) return err(conflict(merged.error)); return ok({ - region: { + regions: merged.value.ranges.map(({ start, bytes, runs: group }) => ({ start, - end: end - 1, + end: start + bytes.length - 1, bytes, recordCount: group.reduce((sum, run) => sum + run.recordCount, 0), firstLine: group.reduce( @@ -524,27 +464,18 @@ const mergeRegion = ( Number.MAX_SAFE_INTEGER, ), lastLine: group.reduce((line, run) => Math.max(line, run.lastLine), 0), - }, - duplicateBytes, + })), + duplicateBytes: merged.value.duplicateBytes, }); }; -const conflict = ( - group: readonly RecordRun[], - run: RecordRun, - address: number, - [existing, incoming]: readonly [number, number], -): HexRecordFailure => { - // The first covering run in address order supplied the retained byte. - const writer = - group.find( - (candidate) => - candidate !== run && - candidate.address <= address && - address < runEnd(candidate), - ) ?? run; - return { - kind: "format", - message: `address 0x${hex(address, 8)} written with different values at ${lineSpan(writer)} (0x${hex(existing, 2)}) and ${lineSpan(run)} (0x${hex(incoming, 2)})`, - }; -}; +const conflict = ({ + address, + writer, + run, + existing, + incoming, +}: RunConflict): RegionDecodeFailure => ({ + kind: "format", + message: `address 0x${hex(address, 8)} written with different values at ${lineSpan(writer)} (0x${hex(existing, 2)}) and ${lineSpan(run)} (0x${hex(incoming, 2)})`, +}); diff --git a/src/domain/firmware/regionDecoder.ts b/src/domain/firmware/regionDecoder.ts new file mode 100644 index 0000000000..9c3b964775 --- /dev/null +++ b/src/domain/firmware/regionDecoder.ts @@ -0,0 +1,14 @@ +import type { Result } from "../result.js"; + +/** A whole-file refusal from a firmware decoder; no partial regions accompany it. */ +export interface RegionDecodeFailure { + readonly kind: "format" | "limit"; + readonly message: string; + readonly boundary?: "file-size" | "capacity"; +} + +/** A streaming firmware decoder, fed the admitted file bytes in order. */ +export interface RegionDecoder { + write(chunk: Uint8Array): Result; + finish(): Result; +} diff --git a/src/domain/firmware/uf2.fixture.ts b/src/domain/firmware/uf2.fixture.ts new file mode 100644 index 0000000000..90804b2bf3 --- /dev/null +++ b/src/domain/firmware/uf2.fixture.ts @@ -0,0 +1,78 @@ +/** UF2 flag values from Microsoft's UF2 specification. */ +export const UF2_FLAGS = { + notMainFlash: 0x0000_0001, + fileContainer: 0x0000_1000, + familyId: 0x0000_2000, + md5: 0x0000_4000, + extensionTags: 0x0000_8000, +} as const; + +export const RP2040_FAMILY = 0xe48b_ff56; +export const RP2XXX_ABSOLUTE_FAMILY = 0xe48b_ff57; +export const RP2350_ARM_S_FAMILY = 0xe48b_ff59; + +export interface Uf2BlockFields { + readonly address: number; + readonly payload: Uint8Array | readonly number[]; + readonly blockNo: number; + readonly numBlocks: number; + /** Sets flag 0x2000 and the familyID word unless `fileSize` is given. */ + readonly familyId?: number; + readonly fileSize?: number; + readonly flags?: number; + /** A payload size field that differs from the payload, for malformed blocks. */ + readonly declaredSize?: number; + readonly magics?: readonly [number, number, number]; +} + +/** One 512-byte block written from the specification's field layout. */ +export const uf2Block = (fields: Uf2BlockFields): Uint8Array => { + const block = Buffer.alloc(512); + const [start0, start1, end] = fields.magics ?? [ + 0x0a32_4655, 0x9e5d_5157, 0x0ab1_6f30, + ]; + const payload = Uint8Array.from(fields.payload); + const flags = + (fields.flags ?? 0) | + (fields.familyId === undefined ? 0 : UF2_FLAGS.familyId); + block.writeUInt32LE(start0, 0); + block.writeUInt32LE(start1, 4); + block.writeUInt32LE(flags >>> 0, 8); + block.writeUInt32LE(fields.address, 12); + block.writeUInt32LE(fields.declaredSize ?? payload.length, 16); + block.writeUInt32LE(fields.blockNo, 20); + block.writeUInt32LE(fields.numBlocks, 24); + block.writeUInt32LE(fields.familyId ?? fields.fileSize ?? 0, 28); + block.set(payload.subarray(0, 476), 32); + block.writeUInt32LE(end, 508); + return block; +}; + +/** Numbered 256-byte blocks for one family, as RP2040 producers write them. */ +export const uf2Image = ( + familyId: number | undefined, + address: number, + bytes: Uint8Array, + payloadSize = 256, +): Uint8Array[] => { + const numBlocks = Math.ceil(bytes.length / payloadSize); + return Array.from({ length: numBlocks }, (_, blockNo) => + uf2Block({ + address: address + blockNo * payloadSize, + payload: bytes.subarray( + blockNo * payloadSize, + (blockNo + 1) * payloadSize, + ), + blockNo, + numBlocks, + ...(familyId === undefined ? {} : { familyId }), + }), + ); +}; + +/** Deterministic, address-dependent bytes so misplaced payloads are visible. */ +export const patternBytes = (length: number, seed = 0): Uint8Array => + Uint8Array.from({ length }, (_, index) => (index * 7 + seed) & 0xff); + +export const uf2File = (blocks: readonly Uint8Array[]): Buffer => + Buffer.concat(blocks); diff --git a/src/domain/firmware/uf2.test.ts b/src/domain/firmware/uf2.test.ts new file mode 100644 index 0000000000..9719b2470b --- /dev/null +++ b/src/domain/firmware/uf2.test.ts @@ -0,0 +1,567 @@ +import { expect, it } from "vitest"; + +import { + createUf2Decoder, + hasUf2Block, + UF2_MAX_FILE_BYTES, + type Uf2Image, +} from "./uf2.js"; +import { + RP2040_FAMILY, + RP2350_ARM_S_FAMILY, + RP2XXX_ABSOLUTE_FAMILY, + UF2_FLAGS, + patternBytes, + uf2Block, + uf2File, + uf2Image, +} from "./uf2.fixture.js"; + +const decode = (bytes: Uint8Array, chunkSize = bytes.length || 1) => { + const decoder = createUf2Decoder(bytes.length); + if (!decoder.ok) throw new Error(decoder.error.message); + for (let offset = 0; offset < bytes.length; offset += chunkSize) { + const written = decoder.value.write( + bytes.subarray(offset, offset + chunkSize), + ); + if (!written.ok) return written; + } + return decoder.value.finish(); +}; + +const image = (bytes: Uint8Array, chunkSize?: number): Uf2Image => { + const decoded = decode(bytes, chunkSize); + if (!decoded.ok) throw new Error(decoded.error.message); + return decoded.value; +}; + +const regionFacts = ({ regions }: Uf2Image) => + regions.map(({ bytes, ...facts }) => ({ ...facts, length: bytes.length })); + +it("recognizes only a complete first block with all three magic numbers", () => { + const block = uf2Block({ + address: 0x1000_0000, + payload: [1], + blockNo: 0, + numBlocks: 1, + }); + expect(hasUf2Block(block)).toBe(true); + expect(hasUf2Block(block.subarray(0, 511))).toBe(false); + for (const magics of [ + [0x0a32_4656, 0x9e5d_5157, 0x0ab1_6f30], + [0x0a32_4655, 0x9e5d_5156, 0x0ab1_6f30], + [0x0a32_4655, 0x9e5d_5157, 0x0ab1_6f31], + ] as const) + expect( + hasUf2Block( + uf2Block({ + address: 0, + payload: [1], + blockNo: 0, + numBlocks: 1, + magics, + }), + ), + ).toBe(false); +}); + +it("joins one family's numbered blocks into a region across chunk boundaries", () => { + const bytes = patternBytes(1000); + const file = uf2File(uf2Image(RP2040_FAMILY, 0x1000_0000, bytes)); + for (const chunkSize of [file.length, 512, 100, 7]) { + const decoded = image(file, chunkSize); + expect(decoded.complete).toBe(true); + expect(decoded.limitations).toEqual([]); + expect(regionFacts(decoded)).toEqual([ + { + familyId: RP2040_FAMILY, + familyName: "RP2040", + start: 0x1000_0000, + end: 0x1000_03e7, + length: 1000, + blockCount: 4, + firstFileBlock: 0, + lastFileBlock: 3, + }, + ]); + expect(Buffer.compare(decoded.regions[0]!.bytes, bytes)).toBe(0); + } + expect(image(file).metadata).toEqual({ + kind: "uf2", + block_count: 4, + trailing_bytes: 0, + invalid_block_count: 0, + flash_block_count: 4, + not_main_flash_block_count: 0, + file_container_block_count: 0, + md5_block_count: 0, + extension_tag_block_count: 0, + family_count: 1, + family_name_registry: "microsoft/uf2 cf3bd7f9 utils/uf2families.json", + numbering_group_count: 1, + missing_block_count: 0, + ambiguous_numbering_group_count: 0, + repeated_block_count: 0, + out_of_order_block_count: 0, + block_number_out_of_range_count: 0, + region_count: 1, + data_bytes: 1000, + duplicate_bytes: 0, + }); +}); + +it("keeps gaps between regions and sorts blocks that arrive out of order", () => { + const payload = patternBytes(768); + const [first, second, third, late] = [0, 1, 2, 3].map((blockNo) => + uf2Block({ + address: blockNo === 3 ? 0x1000_1000 : 0x1000_0000 + blockNo * 256, + payload: + blockNo === 3 + ? [0xaa, 0xbb] + : payload.subarray(blockNo * 256, (blockNo + 1) * 256), + blockNo, + numBlocks: 4, + familyId: RP2040_FAMILY, + }), + ); + const decoded = image(uf2File([late!, third!, first!, second!])); + expect(Buffer.compare(decoded.regions[0]!.bytes, payload)).toBe(0); + expect(regionFacts(decoded)).toEqual([ + expect.objectContaining({ + start: 0x1000_0000, + end: 0x1000_02ff, + blockCount: 3, + firstFileBlock: 1, + lastFileBlock: 3, + }), + expect.objectContaining({ + start: 0x1000_1000, + end: 0x1000_1001, + blockCount: 1, + firstFileBlock: 0, + lastFileBlock: 0, + }), + ]); + expect(decoded.metadata).toMatchObject({ + out_of_order_block_count: 2, + missing_block_count: 0, + region_count: 2, + data_bytes: 770, + }); + expect(decoded.complete).toBe(true); +}); + +it("counts identical rewrites and refuses a conflicting rewrite of one address", () => { + const first = uf2Block({ + address: 0x2000_0000, + payload: [1, 2, 3, 4], + blockNo: 0, + numBlocks: 2, + familyId: RP2040_FAMILY, + }); + const same = uf2Block({ + address: 0x2000_0002, + payload: [3, 4, 5], + blockNo: 1, + numBlocks: 2, + familyId: RP2040_FAMILY, + }); + const decoded = image(uf2File([first, same])); + expect(decoded.metadata).toMatchObject({ + duplicate_bytes: 2, + data_bytes: 5, + }); + expect( + Buffer.compare(decoded.regions[0]!.bytes, Uint8Array.from([1, 2, 3, 4, 5])), + ).toBe(0); + const different = uf2Block({ + address: 0x2000_0002, + payload: [3, 9], + blockNo: 1, + numBlocks: 2, + familyId: RP2040_FAMILY, + }); + expect(decode(uf2File([first, different]))).toEqual({ + ok: false, + error: { + kind: "format", + message: + "family 0xe48bff56 (RP2040) address 0x20000003 is written with different values by block 0 (0x04) and block 1 (0x09)", + }, + }); +}); + +it("keeps families apart even when they target the same addresses", () => { + // A Pico SDK universal image: an RP2350 absolute block, then RP2040 and RP2350 images. + const absolute = uf2Block({ + address: 0x10ff_ff00, + payload: new Uint8Array(256).fill(0xef), + blockNo: 0, + numBlocks: 2, + familyId: RP2XXX_ABSOLUTE_FAMILY, + }); + const decoded = image( + uf2File([ + absolute, + ...uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(512, 1)), + ...uf2Image(RP2350_ARM_S_FAMILY, 0x1000_0000, patternBytes(512, 2)), + ]), + ); + expect( + decoded.regions.map(({ familyId, start, end }) => [familyId, start, end]), + ).toEqual([ + [RP2040_FAMILY, 0x1000_0000, 0x1000_01ff], + [RP2XXX_ABSOLUTE_FAMILY, 0x10ff_ff00, 0x10ff_ffff], + [RP2350_ARM_S_FAMILY, 0x1000_0000, 0x1000_01ff], + ]); + expect(decoded.metadata).toMatchObject({ + family_count: 3, + numbering_group_count: 3, + missing_block_count: 1, + duplicate_bytes: 0, + }); + expect(decoded.complete).toBe(false); + expect(decoded.limitations).toEqual([ + "UF2 numbering declares 1 block(s) that are absent: family 0xe48bff57 (RP2XXX_ABSOLUTE) with 2 numbered blocks has 1 of them.", + ]); +}); + +it("names the LibreTiny families that the cited registry lists", () => { + // LibreTiny also builds LN882H (0xa38090a8), which that registry omits. + const families = new Map([ + [0x9fff_d543, "RTL8710A"], + [0x22e0_d6fc, "RTL8710B"], + [0xe08f_7564, "RTL8720C"], + [0x3379_cfe2, "RTL8720D"], + [0x675a_40b0, "BK7231U"], + [0x7b3e_f230, "BK7231N"], + [0x6a82_cc42, "BK7251"], + [0xde12_70b7, "BL602"], + [0xa380_90a8, null], + ]); + const decoded = image( + uf2File( + [...families.keys()].flatMap((familyId) => + uf2Image(familyId, 0x0, patternBytes(256)), + ), + ), + ); + expect( + new Map( + decoded.regions.map(({ familyId, familyName }) => [familyId, familyName]), + ), + ).toEqual(families); +}); + +it("reports numbered blocks that are absent and leaves coverage unknown", () => { + const blocks = uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(1024)); + const decoded = image(uf2File([blocks[0]!, blocks[1]!, blocks[3]!])); + expect(decoded.complete).toBe(false); + expect(decoded.metadata).toMatchObject({ + missing_block_count: 1, + region_count: 2, + }); + expect(decoded.limitations).toEqual([ + "UF2 numbering declares 1 block(s) that are absent: family 0xe48bff56 (RP2040) with 4 numbered blocks has 3 of them.", + ]); +}); + +it("keeps equal-sized esptool copies complete and an identical resend complete", () => { + // esptool numbers each merged file from 0, so equal-sized files reuse numbers. + const decoded = image( + uf2File([ + ...uf2Image(0xc47e_5767, 0x0, patternBytes(512, 3)), + ...uf2Image(0xc47e_5767, 0x8000, patternBytes(512, 4)), + ]), + ); + expect(decoded.metadata).toMatchObject({ + numbering_group_count: 1, + repeated_block_count: 2, + out_of_order_block_count: 1, + missing_block_count: 0, + ambiguous_numbering_group_count: 0, + family_count: 1, + region_count: 2, + }); + expect(decoded.complete).toBe(true); + expect(decoded.regions[1]?.familyName).toBe("ESP32S3"); + // A block sent twice to its own address is one block, not a second copy. + const blocks = uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(512)); + const resent = image(uf2File([blocks[0]!, blocks[1]!, blocks[1]!])); + expect(resent.complete).toBe(true); + expect(resent.metadata).toMatchObject({ + repeated_block_count: 1, + duplicate_bytes: 256, + ambiguous_numbering_group_count: 0, + }); +}); + +it("leaves coverage unknown when one copy of reused numbering lost a block", () => { + const first = uf2Image(0xc47e_5767, 0x0, patternBytes(768, 5)); + const second = uf2Image(0xc47e_5767, 0x8000, patternBytes(768, 6)); + for (const lost of [first[1]!, second[2]!]) { + const decoded = image( + uf2File([...first, ...second].filter((block) => block !== lost)), + ); + expect(decoded.complete).toBe(false); + expect(decoded.metadata).toMatchObject({ + missing_block_count: 0, + ambiguous_numbering_group_count: 1, + }); + expect(decoded.limitations).toEqual([ + "UF2 numbering cannot show that every copy of a block sequence is complete: family 0xc47e5767 (ESP32S3) with 3 numbered blocks has some numbers at 1 address(es) and others at 2.", + ]); + } +}); + +it("excludes not-main-flash and file-container blocks from regions", () => { + const flash = uf2Block({ + address: 0x0, + payload: [1, 2], + blockNo: 0, + numBlocks: 3, + }); + const comment = uf2Block({ + address: 0x0, + payload: [9, 9, 9], + blockNo: 1, + numBlocks: 3, + flags: UF2_FLAGS.notMainFlash, + }); + const decoded = image(uf2File([flash, comment])); + expect(regionFacts(decoded)).toEqual([ + { + familyId: null, + familyName: null, + start: 0, + end: 1, + length: 2, + blockCount: 1, + firstFileBlock: 0, + lastFileBlock: 0, + }, + ]); + expect(decoded.metadata).toMatchObject({ + not_main_flash_block_count: 1, + flash_block_count: 1, + family_count: 0, + missing_block_count: 1, + }); + expect(decoded.limitations).toContain( + "1 UF2 block(s) are flagged not main flash (flag 0x1); their 3 payload bytes are in no region.", + ); + const file = uf2Block({ + address: 0, + payload: [0x41], + blockNo: 2, + numBlocks: 3, + fileSize: 1, + flags: UF2_FLAGS.fileContainer | UF2_FLAGS.notMainFlash, + }); + const contained = image(uf2File([flash, comment, file])); + expect(contained.complete).toBe(false); + expect(contained.metadata).toMatchObject({ + file_container_block_count: 1, + not_main_flash_block_count: 1, + missing_block_count: 0, + region_count: 1, + }); + expect(contained.limitations).toContain( + "1 UF2 block(s) belong to file containers (flag 0x1000); their files were not reconstructed.", + ); +}); + +it("excludes invalid blocks and a truncated final block, leaving coverage unknown", () => { + const blocks = uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(768)); + const badEnd = uf2Block({ + address: 0x1000_0100, + payload: patternBytes(256), + blockNo: 1, + numBlocks: 3, + familyId: RP2040_FAMILY, + magics: [0x0a32_4655, 0x9e5d_5157, 0], + }); + const oversized = uf2Block({ + address: 0x1000_0300, + payload: [1], + declaredSize: 477, + blockNo: 2, + numBlocks: 3, + familyId: RP2040_FAMILY, + }); + const wrapping = uf2Block({ + address: 0xffff_ffff, + payload: [1, 2], + blockNo: 0, + numBlocks: 1, + }); + const decoded = image( + uf2File([ + blocks[0]!, + badEnd, + blocks[2]!, + oversized, + wrapping, + blocks[1]!.subarray(0, 100), + ]), + ); + expect(decoded.metadata).toMatchObject({ + block_count: 5, + trailing_bytes: 100, + invalid_block_count: 3, + flash_block_count: 2, + missing_block_count: 1, + region_count: 2, + }); + expect(decoded.complete).toBe(false); + expect(decoded.limitations).toEqual([ + "UF2 file ends with 100 bytes after its last complete 512-byte block; the file may be truncated.", + "3 UF2 block(s) are invalid and were excluded; the first is block 1 at byte offset 512: end magic number does not match.", + "UF2 numbering declares 1 block(s) that are absent: family 0xe48bff56 (RP2040) with 3 numbered blocks has 2 of them.", + ]); +}); + +it("reports block numbers outside their declared count", () => { + const decoded = image( + uf2File([ + uf2Block({ address: 0, payload: [1], blockNo: 1, numBlocks: 1 }), + uf2Block({ address: 1, payload: [2], blockNo: 0, numBlocks: 0 }), + ]), + ); + expect(decoded.metadata).toMatchObject({ + block_number_out_of_range_count: 2, + numbering_group_count: 0, + region_count: 1, + data_bytes: 2, + }); + expect(decoded.complete).toBe(false); +}); + +it("counts MD5 and extension-tag flags as declared, unverified facts", () => { + const decoded = image( + uf2File([ + uf2Block({ + address: 0, + payload: [1], + blockNo: 0, + numBlocks: 2, + flags: UF2_FLAGS.md5, + }), + uf2Block({ + address: 1, + payload: [2], + blockNo: 1, + numBlocks: 2, + flags: UF2_FLAGS.extensionTags, + }), + ]), + ); + expect(decoded.metadata).toMatchObject({ + md5_block_count: 1, + extension_tag_block_count: 1, + region_count: 1, + }); + expect(decoded.complete).toBe(true); + expect(decoded.limitations).toEqual([ + "1 UF2 block(s) declare an MD5 flash region (flag 0x4000) and 1 declare extension tags (flag 0x8000); both are counted, not decoded or verified.", + ]); +}); + +it("accepts boundary blocks beside the invalid ones", () => { + const decoded = image( + uf2File([ + // A full 476-byte payload ends exactly at the top of the address space. + uf2Block({ + address: 0xffff_ffff - 475, + payload: patternBytes(476), + blockNo: 0, + numBlocks: 3, + familyId: RP2040_FAMILY, + }), + // A main-flash block may carry no payload. + uf2Block({ + address: 0x1000_0000, + payload: [], + blockNo: 1, + numBlocks: 3, + familyId: RP2040_FAMILY, + }), + // Blocks without a familyID form their own address space. + uf2Block({ + address: 0xffff_ffff - 475, + payload: patternBytes(476, 1), + blockNo: 0, + numBlocks: 1, + }), + uf2Block({ + address: 0x1000_0000, + payload: [7], + blockNo: 2, + numBlocks: 3, + familyId: RP2040_FAMILY, + }), + ]), + ); + expect(decoded.complete).toBe(true); + expect( + decoded.regions.map(({ familyId, start, end }) => [familyId, start, end]), + ).toEqual([ + [null, 0xffff_fe24, 0xffff_ffff], + [RP2040_FAMILY, 0x1000_0000, 0x1000_0000], + [RP2040_FAMILY, 0xffff_fe24, 0xffff_ffff], + ]); + expect(decoded.metadata).toMatchObject({ + invalid_block_count: 0, + flash_block_count: 4, + family_count: 1, + numbering_group_count: 2, + }); +}); + +it("refuses a larger file than its decoder admitted with a typed limit", () => { + const block = uf2Block({ + address: 0, + payload: [1], + blockNo: 0, + numBlocks: 1, + }); + const decoder = createUf2Decoder(511); + if (!decoder.ok) throw new Error(decoder.error.message); + expect(decoder.value.write(block)).toMatchObject({ + ok: false, + error: { kind: "limit", boundary: "file-size" }, + }); +}); + +it("refuses files above the fixed ceiling and more regions than retained", () => { + expect(createUf2Decoder(UF2_MAX_FILE_BYTES + 1)).toMatchObject({ + ok: false, + error: { kind: "limit", boundary: "file-size" }, + }); + const blocks = (count: number) => + Array.from({ length: count }, (_, index) => + uf2Block({ address: index * 2, payload: [1], blockNo: 0, numBlocks: 1 }), + ); + const accepted = decode(uf2File(blocks(65_536)), 512 * 1024); + if (!accepted.ok) throw new Error(accepted.error.message); + expect(accepted.value.metadata.region_count).toBe(65_536); + expect(decode(uf2File(blocks(65_537)), 512 * 1024)).toMatchObject({ + ok: false, + error: { kind: "limit", boundary: "capacity" }, + }); + // Runs are not capped: blocks in reverse order still merge into one region. + const reversed = Array.from({ length: 70_000 }, (_, index) => + uf2Block({ + address: (69_999 - index) * 4, + payload: [1, 2, 3, 4], + blockNo: 0, + numBlocks: 1, + }), + ); + const merged = decode(uf2File(reversed), 512 * 1024); + if (!merged.ok) throw new Error(merged.error.message); + expect(merged.value.metadata).toMatchObject({ + region_count: 1, + data_bytes: 280_000, + }); +}); diff --git a/src/domain/firmware/uf2.ts b/src/domain/firmware/uf2.ts new file mode 100644 index 0000000000..c906123ab6 --- /dev/null +++ b/src/domain/firmware/uf2.ts @@ -0,0 +1,475 @@ +import type { Uf2Metadata } from "../artifactFormatMetadata.js"; +import { err, ok, type Result } from "../result.js"; +import { mergeArenaRuns, type RunConflict } from "./addressRuns.js"; +import type { RegionDecodeFailure, RegionDecoder } from "./regionDecoder.js"; +import { UF2_FAMILY_REGISTRY, uf2FamilyName } from "./uf2Families.js"; + +/** Every UF2 block is 512 bytes: a 32-byte header, 476 data bytes, an end magic. */ +export const UF2_BLOCK_BYTES = 512; +const UF2_DATA_OFFSET = 32; +const UF2_DATA_BYTES = 476; +const UF2_MAGIC_START0 = 0x0a32_4655; +const UF2_MAGIC_START1 = 0x9e5d_5157; +const UF2_MAGIC_END = 0x0ab1_6f30; +const UF2_FLAG_NOT_MAIN_FLASH = 0x0000_0001; +const UF2_FLAG_FILE_CONTAINER = 0x0000_1000; +const UF2_FLAG_FAMILY_ID = 0x0000_2000; +const UF2_FLAG_MD5 = 0x0000_4000; +const UF2_FLAG_EXTENSION_TAGS = 0x0000_8000; +const ADDRESS_SPACE = 0x1_0000_0000; +/** Numbering groups named in one limitation before the rest are counted. */ +const NAMED_GROUPS = 4; + +/** + * Fixed whole-file ceiling, as for Intel HEX; configurable ceilings are + * tracked in #6176. Decoded bytes stay below 476/512 of it. + */ +export const UF2_MAX_FILE_BYTES = 128 * 1024 * 1024; +/** Regions retained as inventory entries, as for Intel HEX record runs. */ +export const UF2_MAX_REGIONS = 65_536; + +/** Decoded bytes for one family's address-contiguous range; `end` is inclusive. */ +export interface Uf2Region { + /** The declared familyID, or null when the block did not set flag 0x2000. */ + readonly familyId: number | null; + /** The registry short name REA knows for `familyId`, or null. */ + readonly familyName: string | null; + readonly start: number; + readonly end: number; + readonly bytes: Uint8Array; + readonly blockCount: number; + /** Zero-based positions of the contributing blocks in the file. */ + readonly firstFileBlock: number; + readonly lastFileBlock: number; +} + +export interface Uf2Image { + readonly metadata: Uf2Metadata; + /** Ordered by family, with undeclared families first, then by address. */ + readonly regions: readonly Uf2Region[]; + readonly complete: boolean; + readonly limitations: readonly string[]; +} + +/** Consecutive main-flash blocks of one family whose addresses continue each other. */ +interface BlockRun { + readonly familyId: number | null; + readonly address: number; + readonly offset: number; + length: number; + blockCount: number; + readonly firstFileBlock: number; + lastFileBlock: number; +} + +/** + * Valid blocks sharing a familyID (or none) and a declared block count. A + * number seen at several addresses belongs to several copies, as when esptool + * numbers each merged file from 0; a repeat at one address is the same block. + */ +interface NumberingGroup { + readonly familyId: number | null; + readonly declared: number; + /** The first target address seen for each block number. */ + readonly first: Map; + /** Further distinct target addresses, only for numbers that have them. */ + readonly further: Map>; + previous: number; +} + +/** What block numbering establishes about completeness. */ +interface NumberingFacts { + readonly missing: number; + readonly absent: readonly string[]; + readonly ambiguous: readonly string[]; +} + +const readUint32 = (bytes: Uint8Array, offset: number): number => + ((bytes[offset] ?? 0) | + ((bytes[offset + 1] ?? 0) << 8) | + ((bytes[offset + 2] ?? 0) << 16) | + ((bytes[offset + 3] ?? 0) << 24)) >>> + 0; + +const hex32 = (value: number): string => + `0x${value.toString(16).padStart(8, "0")}`; + +const familyText = (familyId: number | null): string => { + if (familyId === null) return "no familyID"; + const name = uf2FamilyName(familyId); + return name === null + ? `family ${hex32(familyId)}` + : `family ${hex32(familyId)} (${name})`; +}; + +/** At most NAMED_GROUPS descriptions, then a count of the rest. */ +const named = (items: readonly string[]): string => + `${items.slice(0, NAMED_GROUPS).join("; ")}${items.length > NAMED_GROUPS ? `; and ${String(items.length - NAMED_GROUPS)} more` : ""}`; + +/** Recognize a file only by a complete first block with all three magic numbers. */ +export const hasUf2Block = (prefix: Uint8Array): boolean => + prefix.length >= UF2_BLOCK_BYTES && + readUint32(prefix, 0) === UF2_MAGIC_START0 && + readUint32(prefix, 4) === UF2_MAGIC_START1 && + readUint32(prefix, UF2_BLOCK_BYTES - 4) === UF2_MAGIC_END; + +/** Start a block decoder whose arena is bounded by the admitted file size. */ +export const createUf2Decoder = ( + fileBytes: number, +): Result, RegionDecodeFailure> => + Number.isSafeInteger(fileBytes) && + fileBytes >= 0 && + fileBytes <= UF2_MAX_FILE_BYTES + ? ok(new StreamingUf2Decoder(fileBytes)) + : err({ + kind: "limit", + boundary: "file-size", + message: `file is ${String(fileBytes)} bytes, exceeding the fixed ${String(UF2_MAX_FILE_BYTES)}-byte (128 MiB) ceiling`, + }); + +/** Absent numbers, and groups whose numbers appear at unequal address counts. */ +const numberingFacts = (groups: Iterable): NumberingFacts => { + let missing = 0; + const absent: string[] = []; + const ambiguous: string[] = []; + for (const group of groups) { + const seen = group.first.size; + const name = `${familyText(group.familyId)} with ${String(group.declared)} numbered blocks`; + if (seen < group.declared) { + missing += group.declared - seen; + absent.push(`${name} has ${String(seen)} of them`); + } + // Every seen number holds one address unless another copy was seen. + let most = 1; + let fewest = group.further.size < seen ? 1 : Number.MAX_SAFE_INTEGER; + for (const further of group.further.values()) { + most = Math.max(most, further.size + 1); + fewest = Math.min(fewest, further.size + 1); + } + if (most !== fewest) + ambiguous.push( + `${name} has some numbers at ${String(fewest)} address(es) and others at ${String(most)}`, + ); + } + return { missing, absent, ambiguous }; +}; + +class StreamingUf2Decoder implements RegionDecoder { + readonly #block = new Uint8Array(UF2_BLOCK_BYTES); + readonly #runs: BlockRun[] = []; + readonly #groups = new Map(); + readonly #families = new Set(); + #arena: Uint8Array; + #arenaUsed = 0; + #blockLength = 0; + #blockCount = 0; + #invalid = 0; + #firstInvalid: + | { readonly block: number; readonly reason: string } + | undefined; + #flash = 0; + #notMainFlash = 0; + #notMainFlashBytes = 0; + #fileContainer = 0; + #md5 = 0; + #extensionTags = 0; + #repeated = 0; + #outOfOrder = 0; + #outOfRange = 0; + + constructor(fileBytes: number) { + this.#arena = new Uint8Array( + Math.floor(fileBytes / UF2_BLOCK_BYTES) * UF2_DATA_BYTES, + ); + } + + write(chunk: Uint8Array): Result { + let from = 0; + while (from < chunk.length) { + const count = Math.min( + UF2_BLOCK_BYTES - this.#blockLength, + chunk.length - from, + ); + this.#block.set(chunk.subarray(from, from + count), this.#blockLength); + this.#blockLength += count; + from += count; + if (this.#blockLength < UF2_BLOCK_BYTES) break; + this.#blockLength = 0; + const decoded = this.#decodeBlock(this.#block); + if (!decoded.ok) return decoded; + } + return ok(undefined); + } + + finish(): Result { + const merged = mergeRuns(this.#runs, this.#arena); + this.#arena = new Uint8Array(0); + if (!merged.ok) return merged; + const { regions, duplicateBytes } = merged.value; + if (regions.length > UF2_MAX_REGIONS) + return err({ + kind: "limit", + boundary: "capacity", + message: `payloads form ${String(regions.length)} address-discontiguous regions, more than the ${String(UF2_MAX_REGIONS)} retained as entries`, + }); + const numbering = numberingFacts(this.#groups.values()); + const limitations = this.#limitations(numbering); + return ok({ + metadata: this.#metadata(regions, duplicateBytes, numbering), + regions, + complete: limitations.unknown.length === 0, + limitations: [...limitations.unknown, ...limitations.known], + }); + } + + #decodeBlock(block: Uint8Array): Result { + const index = this.#blockCount; + this.#blockCount += 1; + const flags = readUint32(block, 8); + const address = readUint32(block, 12); + const size = readUint32(block, 16); + const fault = + readUint32(block, 0) !== UF2_MAGIC_START0 || + readUint32(block, 4) !== UF2_MAGIC_START1 + ? "start magic numbers do not match" + : readUint32(block, UF2_BLOCK_BYTES - 4) !== UF2_MAGIC_END + ? "end magic number does not match" + : size > UF2_DATA_BYTES + ? `payload size ${String(size)} exceeds the ${String(UF2_DATA_BYTES)}-byte data area` + : address + size > ADDRESS_SPACE + ? `payload at ${hex32(address)} with ${String(size)} bytes wraps past 0xffffffff` + : undefined; + if (fault !== undefined) { + this.#invalid += 1; + this.#firstInvalid ??= { block: index, reason: fault }; + return ok(undefined); + } + const familyId = + (flags & UF2_FLAG_FAMILY_ID) !== 0 ? readUint32(block, 28) : null; + if ((flags & UF2_FLAG_MD5) !== 0) this.#md5 += 1; + if ((flags & UF2_FLAG_EXTENSION_TAGS) !== 0) this.#extensionTags += 1; + this.#number( + familyId, + readUint32(block, 24), + readUint32(block, 20), + address, + ); + // The file-container flag overrides not-main-flash: such blocks hold files. + if ((flags & UF2_FLAG_FILE_CONTAINER) !== 0) { + this.#fileContainer += 1; + return ok(undefined); + } + if ((flags & UF2_FLAG_NOT_MAIN_FLASH) !== 0) { + this.#notMainFlash += 1; + this.#notMainFlashBytes += size; + return ok(undefined); + } + this.#flash += 1; + if (familyId !== null) this.#families.add(familyId); + return this.#addData( + index, + familyId, + address, + block.subarray(UF2_DATA_OFFSET, UF2_DATA_OFFSET + size), + ); + } + + /** Numbering is per familyID and block count, as bootloaders ignore other families. */ + #number( + familyId: number | null, + declared: number, + blockNo: number, + address: number, + ): void { + if (declared === 0 || blockNo >= declared) { + this.#outOfRange += 1; + return; + } + const key = `${familyId === null ? "none" : String(familyId)}:${String(declared)}`; + let group = this.#groups.get(key); + if (group === undefined) { + group = { + familyId, + declared, + first: new Map(), + further: new Map(), + previous: -1, + }; + this.#groups.set(key, group); + } + const first = group.first.get(blockNo); + if (first === undefined) group.first.set(blockNo, address); + else { + this.#repeated += 1; + if (first !== address) { + let further = group.further.get(blockNo); + if (further === undefined) { + further = new Set(); + group.further.set(blockNo, further); + } + further.add(address); + } + } + if (blockNo <= group.previous) this.#outOfOrder += 1; + group.previous = blockNo; + } + + #addData( + block: number, + familyId: number | null, + address: number, + data: Uint8Array, + ): Result { + if (data.length === 0) return ok(undefined); + if (this.#arenaUsed + data.length > this.#arena.length) + return err({ + kind: "limit", + boundary: "file-size", + message: `block ${String(block)}: payload bytes exceed ${String(UF2_DATA_BYTES)} per complete block of the admitted file size`, + }); + const run = this.#runs.at(-1); + if ( + run !== undefined && + run.familyId === familyId && + run.address + run.length === address + ) { + run.length += data.length; + run.blockCount += 1; + run.lastFileBlock = block; + } else + this.#runs.push({ + familyId, + address, + offset: this.#arenaUsed, + length: data.length, + blockCount: 1, + firstFileBlock: block, + lastFileBlock: block, + }); + this.#arena.set(data, this.#arenaUsed); + this.#arenaUsed += data.length; + return ok(undefined); + } + + /** Limitations that leave coverage unknown, then facts that do not. */ + #limitations(numbering: NumberingFacts): { + readonly unknown: string[]; + readonly known: string[]; + } { + const unknown: string[] = []; + const known: string[] = []; + if (this.#blockLength > 0) + unknown.push( + `UF2 file ends with ${String(this.#blockLength)} bytes after its last complete 512-byte block; the file may be truncated.`, + ); + if (this.#firstInvalid !== undefined) + unknown.push( + `${String(this.#invalid)} UF2 block(s) are invalid and were excluded; the first is block ${String(this.#firstInvalid.block)} at byte offset ${String(this.#firstInvalid.block * UF2_BLOCK_BYTES)}: ${this.#firstInvalid.reason}.`, + ); + if (numbering.missing > 0) + unknown.push( + `UF2 numbering declares ${String(numbering.missing)} block(s) that are absent: ${named(numbering.absent)}.`, + ); + if (numbering.ambiguous.length > 0) + unknown.push( + `UF2 numbering cannot show that every copy of a block sequence is complete: ${named(numbering.ambiguous)}.`, + ); + if (this.#outOfRange > 0) + unknown.push( + `${String(this.#outOfRange)} UF2 block(s) have a block number at or above their block count, so their numbering cannot establish completeness.`, + ); + if (this.#fileContainer > 0) + unknown.push( + `${String(this.#fileContainer)} UF2 block(s) belong to file containers (flag 0x1000); their files were not reconstructed.`, + ); + if (this.#notMainFlash > 0) + known.push( + `${String(this.#notMainFlash)} UF2 block(s) are flagged not main flash (flag 0x1); their ${String(this.#notMainFlashBytes)} payload bytes are in no region.`, + ); + if (this.#md5 > 0 || this.#extensionTags > 0) + known.push( + `${String(this.#md5)} UF2 block(s) declare an MD5 flash region (flag 0x4000) and ${String(this.#extensionTags)} declare extension tags (flag 0x8000); both are counted, not decoded or verified.`, + ); + return { unknown, known }; + } + + #metadata( + regions: readonly Uf2Region[], + duplicateBytes: number, + numbering: NumberingFacts, + ): Uf2Metadata { + return { + kind: "uf2", + block_count: this.#blockCount, + trailing_bytes: this.#blockLength, + invalid_block_count: this.#invalid, + flash_block_count: this.#flash, + not_main_flash_block_count: this.#notMainFlash, + file_container_block_count: this.#fileContainer, + md5_block_count: this.#md5, + extension_tag_block_count: this.#extensionTags, + family_count: this.#families.size, + family_name_registry: UF2_FAMILY_REGISTRY, + numbering_group_count: this.#groups.size, + missing_block_count: numbering.missing, + ambiguous_numbering_group_count: numbering.ambiguous.length, + repeated_block_count: this.#repeated, + out_of_order_block_count: this.#outOfOrder, + block_number_out_of_range_count: this.#outOfRange, + region_count: regions.length, + data_bytes: regions.reduce((sum, region) => sum + region.bytes.length, 0), + duplicate_bytes: duplicateBytes, + }; + } +} + +const blockSpan = (run: BlockRun): string => + run.firstFileBlock === run.lastFileBlock + ? `block ${String(run.firstFileBlock)}` + : `blocks ${String(run.firstFileBlock)}-${String(run.lastFileBlock)}`; + +/** Merge each family's runs by address; overlaps must repeat the same bytes. */ +const mergeRuns = ( + runs: readonly BlockRun[], + arena: Uint8Array, +): Result< + { readonly regions: Uf2Region[]; readonly duplicateBytes: number }, + RegionDecodeFailure +> => { + // Undeclared families sort first; other families never merge with them. + const merged = mergeArenaRuns(runs, arena, (run) => run.familyId ?? -1); + if (!merged.ok) return err(conflict(merged.error)); + return ok({ + regions: merged.value.ranges.map(({ start, bytes, runs: group }) => { + const familyId = group[0]?.familyId ?? null; + return { + familyId, + familyName: familyId === null ? null : uf2FamilyName(familyId), + start, + end: start + bytes.length - 1, + bytes, + blockCount: group.reduce((sum, run) => sum + run.blockCount, 0), + firstFileBlock: group.reduce( + (block, run) => Math.min(block, run.firstFileBlock), + Number.MAX_SAFE_INTEGER, + ), + lastFileBlock: group.reduce( + (block, run) => Math.max(block, run.lastFileBlock), + 0, + ), + }; + }), + duplicateBytes: merged.value.duplicateBytes, + }); +}; + +const conflict = ({ + address, + writer, + run, + existing, + incoming, +}: RunConflict): RegionDecodeFailure => ({ + kind: "format", + message: `${familyText(run.familyId)} address ${hex32(address)} is written with different values by ${blockSpan(writer)} (0x${existing.toString(16).padStart(2, "0")}) and ${blockSpan(run)} (0x${incoming.toString(16).padStart(2, "0")})`, +}); diff --git a/src/domain/firmware/uf2Families.ts b/src/domain/firmware/uf2Families.ts new file mode 100644 index 0000000000..e5147601f1 --- /dev/null +++ b/src/domain/firmware/uf2Families.ts @@ -0,0 +1,53 @@ +/** + * UF2 family IDs that REA names, copied from Microsoft's registry + * `utils/uf2families.json` at microsoft/uf2 + * `cf3bd7f9c274b949528f8cd6a19eef2502a687d2` (2026-01-26, MIT licence; file + * SHA-256 `9d8f561e050507c5909063005cd6f5b157e6fe6db1e8d25b5d9027d57064c194`). + * + * The subset covers the families named by REA's embedded work. The Espressif + * IDs equal esptool v4.12.0 and v5.6.0 `UF2_FAMILY_ID` values. Any other ID is + * reported by value only; a missing name never means an unknown family is + * invalid. + */ +/** The registry revision the names below come from, recorded with each inventory. */ +export const UF2_FAMILY_REGISTRY = + "microsoft/uf2 cf3bd7f9 utils/uf2families.json"; + +const UF2_FAMILY_NAMES: ReadonlyMap = new Map([ + [0x1c5f_21b0, "ESP32"], + [0xbfdd_4eee, "ESP32S2"], + [0xc47e_5767, "ESP32S3"], + [0x3101_f7c1, "ESP32S31"], + [0x2b88_d29c, "ESP32C2"], + [0xd42b_a06c, "ESP32C3"], + [0xf71c_0343, "ESP32C5"], + [0x540d_df62, "ESP32C6"], + [0x77d8_50c4, "ESP32C61"], + [0x3327_26f6, "ESP32H2"], + [0xb6dd_00af, "ESP32H21"], + [0x9e0b_aa8a, "ESP32H4"], + [0x3d30_8e94, "ESP32P4"], + [0x7eab_61ed, "ESP8266"], + [0xe48b_ff56, "RP2040"], + [0xe48b_ff57, "RP2XXX_ABSOLUTE"], + [0xe48b_ff58, "RP2XXX_DATA"], + [0xe48b_ff59, "RP2350_ARM_S"], + [0xe48b_ff5a, "RP2350_RISCV"], + [0xe48b_ff5b, "RP2350_ARM_NS"], + [0x5775_5a57, "STM32F4"], + [0xada5_2840, "NRF52840"], + [0x68ed_2b88, "SAMD21"], + [0x5511_4460, "SAMD51"], + [0x9fff_d543, "RTL8710A"], + [0x22e0_d6fc, "RTL8710B"], + [0xe08f_7564, "RTL8720C"], + [0x3379_cfe2, "RTL8720D"], + [0x675a_40b0, "BK7231U"], + [0x7b3e_f230, "BK7231N"], + [0x6a82_cc42, "BK7251"], + [0xde12_70b7, "BL602"], +]); + +/** The registry short name for a family ID, or null when REA does not name it. */ +export const uf2FamilyName = (familyId: number): string | null => + UF2_FAMILY_NAMES.get(familyId) ?? null; diff --git a/src/domain/liveBinaryTargetSummary.ts b/src/domain/liveBinaryTargetSummary.ts index ba97c867a5..ab1efe39f5 100644 --- a/src/domain/liveBinaryTargetSummary.ts +++ b/src/domain/liveBinaryTargetSummary.ts @@ -79,6 +79,8 @@ export const liveBinaryTargetSummaryVariants = [ "intel-hex", "srec", "esp-image", + "uf2", + "dfuse", "fit", ]), architecture: z.null(), diff --git a/tests/acceptance/firmwareContainersCli.test.ts b/tests/acceptance/firmwareContainersCli.test.ts new file mode 100644 index 0000000000..eca7b2cf4a --- /dev/null +++ b/tests/acceptance/firmwareContainersCli.test.ts @@ -0,0 +1,209 @@ +import { readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, onTestFinished } from "vitest"; +import { cliTest } from "../support/cli/cliFixture.js"; +import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js"; +import { parseEvidence } from "../../src/domain/evidence.js"; +import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js"; +import { + artifactExtractionResultSchema, + artifactInventoryResultSchema, +} from "../../src/domain/artifactGraph.js"; +import { dfuseFile } from "../../src/domain/firmware/dfuse.fixture.js"; +import { + RP2040_FAMILY, + RP2350_ARM_S_FAMILY, + patternBytes, + uf2Block, + uf2File, + uf2Image, +} from "../../src/domain/firmware/uf2.fixture.js"; + +const RP2040_APP = patternBytes(300, 1); +const RP2350_APP = patternBytes(300, 2); +const FLASH = Uint8Array.from([1, 2, 3, 4]); +const OPTION_BYTES = Uint8Array.from([0xaa, 0x55]); + +const cases = [ + { + format: "uf2", + name: "universal.uf2", + bytes: uf2File([ + ...uf2Image(RP2040_FAMILY, 0x1000_0000, RP2040_APP), + ...uf2Image(RP2350_ARM_S_FAMILY, 0x1000_0000, RP2350_APP), + ]), + root: { kind: "uf2", family_count: 2, region_count: 2 }, + regions: { + "family-0xe48bff56/0x10000000-0x1000012b.bin": RP2040_APP, + "family-0xe48bff59/0x10000000-0x1000012b.bin": RP2350_APP, + }, + }, + { + format: "dfuse", + name: "firmware.dfu", + bytes: dfuseFile({ + targets: [ + { + alternateSetting: 0, + name: "Internal Flash", + elements: [{ address: 0x0800_0000, data: FLASH }], + }, + { + alternateSetting: 1, + name: "Option Bytes", + elements: [{ address: 0x1fff_c000, data: OPTION_BYTES }], + }, + ], + }), + root: { kind: "dfuse", target_count: 2, element_count: 2 }, + regions: { + "target-0/element-0-0x08000000-0x08000003.bin": FLASH, + "target-1/element-0-0x1fffc000-0x1fffc001.bin": OPTION_BYTES, + }, + }, +] as const; + +const graphFrom = (json: unknown) => + artifactInventoryResultSchema.parse( + artifactInspectionResultSchema.parse(parseEvidence(json).normalized_result) + .substeps[0]?.evidence.normalized_result, + ); + +cliTest.for(cases)( + "inventories and extracts $format regions through the compiled CLI", + async ({ format, name, bytes, root, regions }, { cli }) => { + const directory = await createTestTempDirectory("rea-firmware-cli-"); + const source = join(directory, name); + await writeFile(source, bytes); + const inspected = await cli.run({ + arguments: ["inspect-artifact", source, "--json"], + }); + expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0); + const inspection = artifactInspectionResultSchema.parse( + parseEvidence(inspected.json).normalized_result, + ); + const graph = graphFrom(inspected.json); + expect(graph.manifest.root_format).toBe(format); + expect(graph.coverage.status).toBe("complete"); + expect( + graph.occurrences.find((row) => row.logical_path === ".") + ?.format_metadata, + ).toMatchObject(root); + expect( + graph.occurrences + .filter((row) => row.logical_path !== ".") + .map((row) => row.logical_path), + ).toEqual(Object.keys(regions)); + expect(inspection.next_probes.map((probe) => probe.operation)).toEqual([ + "open_binary", + ]); + const extracted = await cli.run({ + arguments: ["extract-artifact", source, "--json"], + environment: { TMPDIR: directory, TMP: directory, TEMP: directory }, + }); + expect(extracted.exitCode, extracted.stdout + extracted.stderr).toBe(0); + const result = artifactExtractionResultSchema.parse( + parseEvidence(extracted.json).normalized_result, + ); + onTestFinished(() => + rm(result.output_root, { recursive: true, force: true }), + ); + expect(result.containment_verified).toBe(true); + for (const [path, expected] of Object.entries(regions)) + expect( + Buffer.compare( + await readFile(join(result.output_root, path)), + expected, + ), + ).toBe(0); + }, +); + +cliTest( + "applies strict and record-and-continue policies to a DfuSe suffix CRC", + async ({ cli }) => { + const directory = await createTestTempDirectory("rea-dfuse-crc-cli-"); + const source = join(directory, "firmware.dfu"); + const valid = dfuseFile({ + targets: [ + { alternateSetting: 0, elements: [{ address: 0, data: FLASH }] }, + ], + }); + const observed = valid.readUInt32LE(valid.length - 4); + const corrupt = Buffer.from(valid); + corrupt.writeUInt32LE(0x1234_5678, corrupt.length - 4); + await writeFile(source, corrupt); + const rejected = await cli.run({ + arguments: ["inspect-artifact", source, "--json"], + }); + expect(rejected.exitCode).toBe(1); + expect(rejected.json).toMatchObject({ + code: "artifact_integrity_mismatch", + details: { + logical_path: ".", + algorithm: "dfu-crc32", + declared_crc32: "12345678", + calculated_crc32: observed.toString(16).padStart(8, "0"), + unpacked: false, + }, + }); + const inspected = await cli.run({ + arguments: [ + "inspect-artifact", + source, + "--integrity-policy", + "record-and-continue", + "--json", + ], + }); + expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0); + expect(graphFrom(inspected.json).integrity_contradictions).toEqual([ + expect.objectContaining({ + logical_path: ".", + algorithm: "dfu-crc32", + declared_crc32: "12345678", + observed_crc32: observed.toString(16).padStart(8, "0"), + trust: "observed-untrusted", + }), + ]); + // Extraction applies the same default policy and leaves no output behind. + const refused = await cli.run({ + arguments: ["extract-artifact", source, "--json"], + environment: { TMPDIR: directory, TMP: directory, TEMP: directory }, + }); + expect(refused.exitCode).toBe(1); + expect(refused.json).toMatchObject({ + code: "artifact_integrity_mismatch", + details: { logical_path: ".", algorithm: "dfu-crc32" }, + }); + expect(await readdir(directory)).toEqual(["firmware.dfu"]); + }, +); + +cliTest( + "refuses a conflicting UF2 rewrite through the CLI", + async ({ cli }) => { + const directory = await createTestTempDirectory("rea-uf2-conflict-cli-"); + const source = join(directory, "firmware.uf2"); + await writeFile( + source, + uf2File([ + ...uf2Image(RP2040_FAMILY, 0x1000_0000, RP2040_APP), + uf2Block({ + address: 0x1000_0000, + payload: [0xff], + blockNo: 0, + numBlocks: 1, + familyId: RP2040_FAMILY, + }), + ]), + ); + const result = await cli.run({ + arguments: ["inspect-artifact", source, "--json"], + }); + expect(result.exitCode).toBe(1); + expect(result.stdout + result.stderr).toContain( + "UF2 is malformed: family 0xe48bff56 (RP2040) address 0x10000000 is written with different values by blocks 0-1 (0x01) and block 2 (0xff)", + ); + }, +); diff --git a/tests/boundary/filesystem/binaryTarget.test.ts b/tests/boundary/filesystem/binaryTarget.test.ts index 47309bc5b0..1452dfb509 100644 --- a/tests/boundary/filesystem/binaryTarget.test.ts +++ b/tests/boundary/filesystem/binaryTarget.test.ts @@ -41,6 +41,12 @@ import { } from "../../fixtures/espImage.js"; import { espFlashDump } from "../../../src/domain/firmware/espFlash.fixture.js"; import { javaClassFile } from "../../fixtures/javaClassFile.js"; +import { dfuseFile } from "../../../src/domain/firmware/dfuse.fixture.js"; +import { + patternBytes, + uf2File, + uf2Image, +} from "../../../src/domain/firmware/uf2.fixture.js"; import { MAX_JVM_CLASS_IDENTITY_BYTES, JVM_CLASS_IDENTITY_LIMIT_REASON, @@ -603,6 +609,47 @@ describe("binary target I/O: image classification", () => { }); }); +describe("binary target I/O: firmware containers", () => { + it("classifies UF2 by block magics and DfuSe by its prefix and suffix", async () => { + const directory = await createTestTempDirectory("rea-firmware-target-"); + const uf2 = uf2File(uf2Image(0xe48b_ff56, 0x1000_0000, patternBytes(300))); + const dfu = dfuseFile({ + targets: [{ alternateSetting: 0, elements: [{ address: 0, data: [1] }] }], + }); + const paths = { + uf2: join(directory, "firmware.bin"), + dfuse: join(directory, "firmware.img"), + badCrc: join(directory, "bad-crc.dfu"), + shortUf2: join(directory, "short.uf2"), + noSuffix: join(directory, "no-suffix.dfu"), + }; + // The suffix CRC is checked by inventory, not by recognition. + const badCrc = Buffer.from(dfu); + badCrc.writeUInt32LE(0x1234_5678, badCrc.length - 4); + await Promise.all([ + writeFile(paths.uf2, uf2), + writeFile(paths.dfuse, dfu), + writeFile(paths.badCrc, badCrc), + writeFile(paths.shortUf2, uf2.subarray(0, 511)), + writeFile(paths.noSuffix, dfu.subarray(0, dfu.length - 16)), + ]); + for (const [path, format] of [ + [paths.uf2, "uf2"], + [paths.dfuse, "dfuse"], + [paths.badCrc, "dfuse"], + ] as const) + expect(await parseBinaryTarget(path)).toMatchObject({ + ok: true, + value: { kind: "archive", format }, + }); + for (const path of [paths.shortUf2, paths.noSuffix]) + expect(await parseBinaryTarget(path)).toMatchObject({ + ok: false, + error: { _tag: "BinaryTargetError" }, + }); + }); +}); + describe("binary target I/O: ESP flash layouts", () => { it("detects a flash layout before the bootloader it begins with", async () => { const directory = await createTestTempDirectory("rea-esp-flash-target-"); diff --git a/tests/boundary/mcp/firmwareContainersMcp.test.ts b/tests/boundary/mcp/firmwareContainersMcp.test.ts new file mode 100644 index 0000000000..2d9a42dbb8 --- /dev/null +++ b/tests/boundary/mcp/firmwareContainersMcp.test.ts @@ -0,0 +1,158 @@ +import { readFile, rm, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, it } from "vitest"; +import { z } from "zod"; +import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js"; +import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js"; +import { parseMcpToolError } from "../../fixtures/mcpToolError.js"; +import { parseEvidence } from "../../../src/domain/evidence.js"; +import { + artifactExtractionResultSchema, + artifactInventoryResultSchema, +} from "../../../src/domain/artifactGraph.js"; +import { artifactInspectionResultSchema } from "../../../src/domain/artifactInspection.js"; +import { dfuseFile } from "../../../src/domain/firmware/dfuse.fixture.js"; +import { + RP2040_FAMILY, + RP2XXX_ABSOLUTE_FAMILY, + patternBytes, + uf2Block, + uf2File, + uf2Image, +} from "../../../src/domain/firmware/uf2.fixture.js"; + +const APP = patternBytes(300, 9); +const REGION = "family-0xe48bff56/0x10000000-0x1000012b.bin"; + +// The compiled stdio fixture checks advertised schemas against their dialect and +// validates actual tool responses, avoiding a second source server module graph. +it.each(["full", "compact"] as const)( + "exposes UF2 facts, retained regions and extraction through MCP (%s)", + async (profile) => { + const root = await createTestTempDirectory("rea-uf2-mcp-"); + const path = join(root, "firmware.uf2"); + await writeFile( + path, + uf2File([ + uf2Block({ + address: 0x10ff_ff00, + payload: new Uint8Array(256).fill(0xef), + blockNo: 0, + numBlocks: 2, + familyId: RP2XXX_ABSOLUTE_FAMILY, + }), + ...uf2Image(RP2040_FAMILY, 0x1000_0000, APP), + ]), + ); + const { call } = await connectLocalToolsMcp({ + REA_MCP_INPUT_SCHEMA_PROFILE: profile, + }); + let outputRoot: string | undefined; + try { + const opened = await call("open_binary", { path }); + expect(opened.isError, JSON.stringify(opened)).not.toBe(true); + const reply = await call("inspect_artifact", { detail: "summary" }); + expect(reply.isError, JSON.stringify(reply)).not.toBe(true); + const summary = z + .object({ + parent_evidence_id: z.string(), + summary: z.object({ + root_format_metadata: z.unknown(), + source_coverage: z.unknown(), + }), + }) + .parse(parseEvidence(reply.structuredContent).normalized_result); + expect(summary.summary.root_format_metadata).toMatchObject({ + kind: "uf2", + family_count: 2, + missing_block_count: 1, + region_count: 2, + }); + expect(summary.summary.source_coverage).toMatchObject({ + status: "unknown", + }); + const retained = await call("inspect_analysis_view", { + source: { + kind: "retained-evidence", + evidence_id: summary.parent_evidence_id, + }, + view: { + kind: "inventory", + collection: "occurrences", + logical_paths: [REGION], + fields: ["format_metadata"], + }, + }); + expect(retained.isError, JSON.stringify(retained)).not.toBe(true); + expect(JSON.stringify(retained.structuredContent)).toContain( + '"family_name":"RP2040"', + ); + const extracted = await call("extract_artifact", {}); + expect(extracted.isError, JSON.stringify(extracted)).not.toBe(true); + const result = artifactExtractionResultSchema.parse( + parseEvidence(extracted.structuredContent).normalized_result, + ); + outputRoot = result.output_root; + expect( + Buffer.compare(await readFile(join(result.output_root, REGION)), APP), + ).toBe(0); + await call("close_binary", {}); + } finally { + if (outputRoot !== undefined) + await rm(outputRoot, { recursive: true, force: true }); + } + }, +); + +it("delivers a DfuSe CRC contradiction under both integrity policies through MCP", async () => { + const root = await createTestTempDirectory("rea-dfuse-mcp-"); + const path = join(root, "firmware.dfu"); + const valid = dfuseFile({ + targets: [ + { + alternateSetting: 0, + name: "Internal Flash", + elements: [{ address: 0x0800_0000, data: [1, 2, 3, 4] }], + }, + ], + }); + const corrupt = Buffer.from(valid); + corrupt.writeUInt32LE(0, corrupt.length - 4); + await writeFile(path, corrupt); + const { call } = await connectLocalToolsMcp({}); + const opened = await call("open_binary", { path }); + expect(opened.isError, JSON.stringify(opened)).not.toBe(true); + const rejected = parseMcpToolError(await call("inspect_artifact", {})); + expect(rejected.error).toMatchObject({ + code: "artifact_integrity_mismatch", + details: { + logical_path: ".", + algorithm: "dfu-crc32", + declared_crc32: "00000000", + calculated_crc32: valid + .readUInt32LE(valid.length - 4) + .toString(16) + .padStart(8, "0"), + }, + }); + const recorded = await call("inspect_artifact", { + integrity_policy: "record-and-continue", + }); + expect(recorded.isError, JSON.stringify(recorded)).not.toBe(true); + const inspection = artifactInspectionResultSchema.parse( + parseEvidence(recorded.structuredContent).normalized_result, + ); + expect(inspection.contradictions).toEqual([ + expect.objectContaining({ + algorithm: "dfu-crc32", + declared_crc32: "00000000", + }), + ]); + const graph = artifactInventoryResultSchema.parse( + inspection.substeps[0]?.evidence.normalized_result, + ); + expect( + graph.occurrences.find((row) => row.logical_path === ".")?.hash_status, + ).toBe("mismatched"); + await call("close_binary", {}); +});