feat(artifacts): inventory and extract UF2 and DfuSe firmware containers (#8230)
* feat(artifacts): inventory and extract UF2 and DfuSe regions open_binary refused UF2 and DfuSe firmware containers as an unsupported format, so inspect_artifact and extract_artifact could not show which bytes a drag-and-drop or DFU update writes at which addresses. UF2 is recognized by a complete first block with all three magic numbers. Valid main-flash payloads are joined into address-contiguous regions per familyID, named family-0x<id>/0x<start>-0x<end>.bin, and known IDs are named from a small table copied from Microsoft's uf2families.json at a pinned commit. Numbering is counted per familyID and block count: absent, repeated, out-of-order and out-of-range block numbers are reported, and absent blocks leave coverage unknown. Not-main-flash blocks are excluded and reported; file-container blocks are reported and leave coverage unknown; invalid blocks and a partial final block are excluded with their position. A rewrite of one address with different bytes fails closed, naming both block spans. DfuSe is recognized only by its prefix together with a DFU 1.1a suffix. Each element becomes one region at its file range with its target's alternate setting and name. Structural faults fail closed with a byte offset; overlapping elements stay separate and are reported. A suffix CRC mismatch follows integrity_policy: fail refuses with a dfu-crc32 integrity error, and record-and-continue records a root contradiction that extraction also carries. HEX, UF2 and DfuSe share one retained-region reader and one run merger, so the HEX decoder and reader lose their private copies. * docs(artifacts): document UF2 and DfuSe images and their esptool lane Describe recognition, region names, metadata, numbering, failures, the DfuSe suffix CRC under integrity_policy, and limits, and link the page from the CLI and firmware guides and the agent skill reference. npm run verify:uf2 runs esptool merge-bin --format uf2 on generated inputs at known addresses, with and without MD5, with 256-byte payloads, and on a concatenation of two chips' outputs. Inspection and extraction through the built CLI must return every input at its address under the familyID esptool itself reports. * fix(artifacts): accept both DfuSe image size conventions UM0391 calls DFUImageSize the file length, and dfu-util's dfuse-pack.py writes that value, while ST's DfuFileMgr, MicroPython's dfu.py and pydfu leave the 16-byte suffix out. The decoder accepted only the second form, so it refused dfuse-pack.py output as malformed. Accept either value and keep the declared one in image_size; any other value still fails with the suffix offset and the file length. The tests pin both producers' output for one image, CRC included, from their build() functions transcribed and run under Python 3. * fix(artifacts): keep reused UF2 block numbers from hiding missing blocks esptool numbers every merged input from 0, so two inputs with the same block count share one numbering group. A block lost from one input left its number present in the other, and the file reported complete coverage. Each block number now records the target addresses it was seen at: equal address counts are complete copies, a resend to the same address is one block, and unequal counts make coverage unknown with a limitation and ambiguous_numbering_group_count. sequence_count becomes numbering_group_count, and region block positions become first_file_block and last_file_block. Review follow-ups in the same change: - compare_artifacts compares a DfuSe element's alternate setting and target name, so bytes moved to another device memory are changed; - the shared reader keeps no region bytes outside its map after close; - one domain type replaces four identical decoder failure types; - the UF2 decoder returns a typed limit instead of throwing on bytes beyond its admitted size, and caps retained regions, not runs; - DfuSe caps elements with bytes only, names empty elements and element-less targets, drops the constant bVersion and bcdDFU fields, and says whether the CRC matches when DFUImageSize fits neither convention; - classification evidence carries a suffix only when the probe saw the end, and root classification reads the tail only for a DfuSe prefix; - the UF2 domain region owns family_name, and the root records the family registry revision; the table's MIT notice is preserved; - docs pin the producer revisions (dfu-util ee6b393d and f9a537c9, MicroPython cb99ca98), give the MD5 structure's real meaning, and describe container contradictions in the MCP contract guide; - tests and the esptool lane add the equal-count inputs and a block lost from one of them, accepted boundaries beside each refusal, a DfuSe root larger than the classification prefix, and CLI refusals. * feat(artifacts): name the UF2 families that LibreTiny builds for LibreTiny writes UF2 for Realtek, Beken and Bouffalo Lab modules. The Microsoft registry revision REA already cites (cf3bd7f9) lists eight of those families: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U, BK7231N, BK7251 and BL602. Their regions now carry those names. LN882H, which that revision omits, keeps its familyID with no name. * fix(artifacts): count DfuSe declarations without bytes instead of listing them Empty DfuSe elements have no entry, so DFUSE_MAX_ELEMENTS does not cap them, and the target walk kept a description string for each one. At 8 bytes per element, a file under the 128 MiB ceiling can declare about 16.7 million of them. The walk now counts every declaration without bytes and builds only the first NAMED_ELEMENTS descriptions, which is all that the limitation shows. The limitation text and empty_element_count do not change. The guide now says that both DfuSe limitations name the first four. On this host, decoding 4,000,000 empty elements (a 32 MB file) ran out of memory under a 128 MB V8 heap before this change, and it peaked at 1.59 GB RSS with the default heap. It now decodes in 114 ms with a 220 MB peak. --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
This commit is contained in:
Entrpiandmorluto authored and GitHub committed
1 parent
22e5aa97af
commit
b43c329df7
55 files changed
+4667
-390
No files matched your search
@@ -623,6 +623,13 @@ semantics are unsupported. `m68k` requires an even entry and decodes the 68040
|
||||
superset of the 68000 family; Amiga/Atari containers and vector-table entry
|
||||
discovery are unsupported. Windows Ghidra P0 admits raw images only for `6502`. See [raw binary guide](https://github.com/morluto/rea/blob/main/docs/raw-binary.md).
|
||||
|
||||
Intel HEX, S-record, UF2 and DfuSe files are containers, not raw images:
|
||||
`inspect_artifact` lists address regions (UF2 per familyID, DfuSe per element
|
||||
with its alternate setting), and `extract_artifact` writes each region. Open an
|
||||
extracted region as `raw-binary` with its start address as `entry_point`. UF2
|
||||
numbering gaps leave coverage unknown; a DfuSe suffix CRC mismatch follows
|
||||
`integrity_policy`. See [UF2 and DfuSe](https://github.com/morluto/rea/blob/main/docs/uf2-dfuse.md).
|
||||
|
||||
### Function annotations in Ghidra
|
||||
|
||||
Fresh-import sessions admit annotations; existing-project reuse is read-only.
|
||||
|
||||
@@ -17,6 +17,7 @@ notices supplied with the software you actually obtain or redistribute.
|
||||
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| npm dependencies | [Complete inventory](licenses/npm-dependencies.txt) of every entry in `package-lock.json`, including transitive, optional platform, development, and website dependencies. [Runtime notices](licenses/npm-runtime-notices.txt) preserve the copyright, license, and NOTICE documents from integrity-verified release archives, plus the MIT notice for Rolldown's generated runtime helpers. Development dependencies are inventoried but are not all distributed in REA's npm package. |
|
||||
| pefile ordinal tables | Import-hash ordinal names are derived from [pefile 2024.8.26](https://github.com/erocarrera/pefile/tree/4b3b1e2e568a88d4f1897d694d684f23d9e270c4), copyright 2004–2024 Ero Carrera. Preserve the [MIT license](licenses/pefile-2024.8.26-LICENSE.txt). The Python package is not a runtime dependency. |
|
||||
| UF2 family names | UF2 family IDs and short names are copied from [microsoft/uf2 `utils/uf2families.json`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json), copyright Microsoft Corporation. Preserve the [MIT license](licenses/microsoft-uf2-MIT.txt). |
|
||||
| NativeAOT parser | REA's derived parser uses the MIT-licensed [ghidra-nativeaot](https://github.com/Washi1337/ghidra-nativeaot/tree/effeb734fc570c32650f88b159608979dc7b423e) work of Washi and the NativeAOT layout definitions in [dotnet/runtime](https://github.com/dotnet/runtime/tree/a3b2d40328be0be3f8dd43f0e5cc925b8827b044). Preserve [Washi's MIT notice](licenses/nativeaot-parser-MIT.txt) and the [.NET Foundation MIT notice](licenses/dotnet-runtime-MIT.txt). |
|
||||
| EVM analysis | `evmole@0.9.3`, MIT, copyright 2023 Maxim Andreev. The published npm archive omits its license document; REA retains the [original upstream license](third_party/evmole/LICENSE) and [provenance](third_party/evmole/README.md). It is also included in the runtime notices. |
|
||||
| Windows native addon | The release addon is compiled with MinGW-w64 and statically links GCC runtime libraries. Preserve the [MinGW-w64 runtime notices](licenses/MinGW-w64-runtime.txt), [GPLv3 text](licenses/GPL-3.0.txt), and [GCC Runtime Library Exception 3.1](licenses/GCC-Runtime-Library-Exception-3.1.txt). The exception governs eligible compiled combinations; it does not relicense the runtime itself under MIT. The Node-API build headers are from Node 22.19.0; its [original license and component notices](licenses/Node-22.19.0-LICENSE.txt) are retained. Node itself is supplied by the user. Snapshot provenance and release-toolchain review requirements are in [licenses/README.md](licenses/README.md). |
|
||||
|
||||
+10
@@ -329,6 +329,16 @@ overlap and archive expansion checks. The native streaming decoder keeps the
|
||||
existing 128 MiB Zstandard window cap; frames requiring a larger window are
|
||||
refused. Deprecated method 20 and other unregistered ZIP codecs remain unsupported.
|
||||
|
||||
Inventory and extract UF2 or DfuSe regions:
|
||||
|
||||
```sh
|
||||
rea inspect-artifact ./firmware.uf2 --json
|
||||
rea extract-artifact ./firmware.dfu --json
|
||||
```
|
||||
|
||||
See [UF2 and DfuSe images](uf2-dfuse.md) for recognition, families, numbering,
|
||||
the DfuSe suffix CRC and limits.
|
||||
|
||||
Inventory and extract U-Boot FIT images:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -728,6 +728,32 @@
|
||||
"unpacked"
|
||||
],
|
||||
"additionalProperties": {}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"logical_path": { "type": "string" },
|
||||
"algorithm": { "type": "string", "const": "dfu-crc32" },
|
||||
"declared_crc32": {
|
||||
"type": "string",
|
||||
"pattern": "^[a-f0-9]{8}$"
|
||||
},
|
||||
"calculated_crc32": {
|
||||
"anyOf": [
|
||||
{ "type": "string", "pattern": "^[a-f0-9]{8}$" },
|
||||
{ "type": "null" }
|
||||
]
|
||||
},
|
||||
"unpacked": { "type": "boolean" }
|
||||
},
|
||||
"required": [
|
||||
"logical_path",
|
||||
"algorithm",
|
||||
"declared_crc32",
|
||||
"calculated_crc32",
|
||||
"unpacked"
|
||||
],
|
||||
"additionalProperties": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -340,6 +340,15 @@ payloads stay as stored. Intel HEX/S-record and FIT images are separate
|
||||
readers and are not covered here. See [U-Boot legacy uImage](uimage.md) for
|
||||
component padding, preserved names, resource policies and verification scope.
|
||||
|
||||
## Inspect UF2 and DfuSe images
|
||||
|
||||
Use `inspect-artifact` / `inspect_artifact` on a UF2 file for each familyID's
|
||||
address regions and its block numbering, or on a DfuSe file for its targets,
|
||||
elements and DFU suffix, then `extract-artifact` / `extract_artifact` for one
|
||||
raw file per region or element. A DfuSe suffix CRC mismatch follows
|
||||
`integrity_policy`. Decoding is portable and requires no external tool. See
|
||||
[UF2 and DfuSe images](uf2-dfuse.md).
|
||||
|
||||
## Inspect U-Boot FIT images
|
||||
|
||||
Use `inspect-artifact` / `inspect_artifact` on a U-Boot FIT (`.itb`) for its
|
||||
|
||||
@@ -1317,12 +1317,20 @@ bytes and returns the `integrity_contradictions` records for the extracted
|
||||
files. An active ASAR unpacked entry whose companion file is absent cannot be
|
||||
materialized; extraction fails as `unavailable` with that logical path.
|
||||
|
||||
A container can also declare a checksum over its own bytes; today that is the
|
||||
DFU suffix CRC of a DfuSe file. A mismatch is a contradiction of the root
|
||||
occurrence (logical path `.`), recorded as a `dfu-crc32` pair. Under
|
||||
`record-and-continue` the root becomes `mismatched` while its elements stay
|
||||
listed and extractable as observed-untrusted bytes, and `extract_artifact`
|
||||
returns the root record with the extracted files' records.
|
||||
|
||||
If native DMG child traversal is unavailable after carrier identification,
|
||||
the typed failure retains a revalidated root-only inventory under
|
||||
`details.partial_observation.inventory`, with unknown child coverage and the
|
||||
producer's diagnostics/provenance. It does not borrow or detach an existing
|
||||
attachment. Native checksum verification failures still refuse traversal;
|
||||
`record-and-continue` recovery applies to structured entry contradictions.
|
||||
`record-and-continue` recovery applies to structured entry and container
|
||||
contradictions.
|
||||
|
||||
Occurrences use normalized logical paths for identity and lookup. Optional
|
||||
`source_path` retains the original path within the owning reader when Unicode
|
||||
|
||||
@@ -2038,6 +2038,13 @@ tool before building and warns when esptool is not the pinned v5.6.0.
|
||||
`REA_XTENSA_ESP32_GCC` adds an ESP32 Xtensa build. See
|
||||
[ESP images](esp-image.md).
|
||||
|
||||
`npm run verify:uf2` needs only a Python that runs esptool, named by
|
||||
`REA_ESPTOOL_PYTHON` (default `python3`). It writes inputs at known addresses,
|
||||
builds UF2 files with esptool's `merge-bin --format uf2` (v4 `merge_bin`) with
|
||||
and without MD5 and with 256-byte payloads, and concatenates two chips'
|
||||
outputs. Inspection and extraction through the built CLI must return each input
|
||||
at its address under the familyID esptool itself reports. It is not part of CI.
|
||||
See [UF2 and DfuSe images](uf2-dfuse.md).
|
||||
`npm run verify:esp-flash` needs the same tools and an ESP-IDF checkout
|
||||
(`REA_ESP_IDF_PATH`) for `gen_esp32part.py` and `otatool.py`. It builds
|
||||
partition tables, assembles dumps with `esptool merge-bin`, writes OTA records
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
# UF2 and DfuSe images
|
||||
|
||||
REA inventories UF2 (Microsoft's USB Flashing Format) and DfuSe (ST's DFU
|
||||
update container, `.dfu`) files through `inspect-artifact` /
|
||||
`inspect_artifact`, and materializes their decoded bytes through
|
||||
`extract-artifact` / `extract_artifact`. UF2 is the drag-and-drop format of the
|
||||
RP2040 and RP2350 bootloaders, of TinyUF2 on ESP32-S2 and ESP32-S3, and of
|
||||
`esptool merge-bin --format uf2`. Decoding is static and needs no external tool.
|
||||
|
||||
```sh
|
||||
rea inspect-artifact ./firmware.uf2 --json
|
||||
rea inspect-artifact ./firmware.dfu --detail summary --evidence_output evidence.json --json
|
||||
rea extract-artifact ./firmware.uf2 --json
|
||||
```
|
||||
|
||||
For MCP, open the file with `open_binary`, then call `inspect_artifact`. Summary
|
||||
delivery retains the complete Evidence; use `parent_evidence_id` with
|
||||
`inspect_analysis_view` to read selected regions and fields. `extract_artifact`
|
||||
returns a fresh `output_root` containing one raw file per region.
|
||||
|
||||
## Recognition
|
||||
|
||||
A file is UF2 only when its first 512 bytes are a complete block with all three
|
||||
magic numbers: `0x0A324655` and `0x9E5D5157` at the start and `0x0AB16F30` at
|
||||
the end. A file is DfuSe only when it begins with `DfuSe` and format version 1
|
||||
and ends with a 16-byte DFU 1.1a suffix: `bcdDFU` `0x011A`, the `UFD` signature
|
||||
and `bLength` 16. The suffix CRC is not part of recognition. The file name does
|
||||
not matter, and a file with only one of the DfuSe markers stays unrecognized.
|
||||
The container formats are `uf2` and `dfuse`.
|
||||
|
||||
## UF2 regions and metadata
|
||||
|
||||
Each entry is one address-contiguous region of one family's main-flash
|
||||
payloads, named `family-0x<familyID>/0x<start>-0x<end>.bin` with an inclusive
|
||||
end address, for example `family-0xe48bff56/0x10000000-0x1000ffff.bin`. Blocks
|
||||
without the familyID flag (`0x2000`) go under `no-family/`. Families are
|
||||
separate address spaces: a multi-family file, such as a Pico SDK image for both
|
||||
RP2040 and RP2350, has a region per family even at the same addresses. Gaps are
|
||||
never filled.
|
||||
|
||||
Region `format_metadata` (`uf2-region`) records `family_id` (or `null`),
|
||||
`family_name`, `start_address`, `end_address`, `length`, `block_count`, and
|
||||
`first_file_block`/`last_file_block`: the zero-based positions in the file of
|
||||
the contributing blocks, not their UF2 block numbers. `family_name` is REA's
|
||||
label, not file content. It comes from a small table copied from Microsoft's
|
||||
`uf2families.json` at commit
|
||||
[`cf3bd7f9`](https://github.com/microsoft/uf2/blob/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/utils/uf2families.json),
|
||||
which the root's `family_name_registry` names. The table covers the ESP32
|
||||
family, ESP8266, RP2040, the RP2350 and RP2XXX download families, STM32F4,
|
||||
nRF52840, SAMD21 and SAMD51. It also covers the families that LibreTiny builds
|
||||
for and the registry lists: RTL8710A, RTL8710B, RTL8720C, RTL8720D, BK7231U,
|
||||
BK7231N, BK7251 and BL602. Other IDs, LibreTiny's LN882H among them, keep their
|
||||
value with `family_name: null`. A familyID names a target family; it does not
|
||||
establish the CPU state, memory map or entry point.
|
||||
|
||||
Root `format_metadata` (`uf2`) records:
|
||||
|
||||
| Field | Meaning |
|
||||
| ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| `block_count`, `trailing_bytes` | Complete 512-byte blocks, and bytes after the last complete block |
|
||||
| `invalid_block_count` | Blocks with a wrong magic number, a payload over 476 bytes, or a payload that wraps past 4 GiB |
|
||||
| `flash_block_count` | Valid main-flash blocks, the only source of region bytes |
|
||||
| `not_main_flash_block_count`, `file_container_block_count` | Blocks with flag `0x1`, and blocks with flag `0x1000` |
|
||||
| `md5_block_count`, `extension_tag_block_count` | Blocks that declare an MD5 flash region (`0x4000`) or extension tags (`0x8000`) |
|
||||
| `family_count`, `family_name_registry` | Distinct familyIDs among main-flash blocks, and the registry revision of `family_name` |
|
||||
| `numbering_group_count` | Groups of valid blocks that share a familyID (or none) and a declared block count |
|
||||
| `missing_block_count` | Block numbers below a group's block count that no valid block carries |
|
||||
| `ambiguous_numbering_group_count` | Groups whose numbers appear at unequal counts of target addresses |
|
||||
| `repeated_block_count`, `out_of_order_block_count` | Blocks whose number already appeared in their group, and blocks numbered at or below the previous block of their group |
|
||||
| `block_number_out_of_range_count` | Blocks numbered at or above their block count, or with a block count of 0 |
|
||||
| `region_count`, `data_bytes`, `duplicate_bytes` | Regions, unique decoded bytes, and bytes rewritten with identical values |
|
||||
|
||||
Numbering is grouped by familyID because a bootloader ignores every field of
|
||||
another family's block, including its numbering. Within a group, a block number
|
||||
seen at several target addresses belongs to several copies of a sequence:
|
||||
esptool numbers each merged input file from 0, so inputs with equal block
|
||||
counts share a group, and each number then appears once per input. A block sent
|
||||
twice to the same address is one block. A group is complete when every number
|
||||
below its block count is present at the same count of addresses. If some numbers
|
||||
appear at more addresses than others, a copy may have lost a block, so coverage
|
||||
is unknown and a limitation names the group.
|
||||
|
||||
## UF2 facts, limitations and failures
|
||||
|
||||
| Situation | Outcome |
|
||||
| ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
|
||||
| Blocks out of address order | Sorted into regions |
|
||||
| Blocks numbered at or below the previous block of their group | Counted in `out_of_order_block_count`; the spec lets blocks arrive in any order |
|
||||
| An address written twice with the same value in one family | Accepted and counted in `duplicate_bytes` |
|
||||
| An address written twice with different values in one family | Fails, naming the family, the address and the block spans of both writes |
|
||||
| The same addresses in different families | Separate regions; never a conflict |
|
||||
| Numbered blocks absent from a group | Regions are returned; coverage is unknown and a limitation names the groups |
|
||||
| Reused numbers at unequal address counts | Regions are returned; coverage is unknown and a limitation names the groups |
|
||||
| An invalid block | Excluded; coverage is unknown and a limitation names the first one and its offset |
|
||||
| A partial final block | Excluded; coverage is unknown and the file may be truncated |
|
||||
| Block numbers at or above the block count | Payloads are kept; coverage is unknown |
|
||||
| Not-main-flash blocks (`0x1`) | Excluded from regions; their count and payload byte count are reported; coverage is unaffected |
|
||||
| File-container blocks (`0x1000`) | Excluded from regions; their files are not reconstructed, so coverage is unknown |
|
||||
| MD5 (`0x4000`) and extension-tag (`0x8000`) blocks | Counted and reported as a limitation; neither is decoded or verified |
|
||||
|
||||
Unlike a malformed Intel HEX record, which fails the whole file, an invalid UF2
|
||||
block is excluded and reported: each block carries its own framing and address,
|
||||
so its neighbours stay readable.
|
||||
|
||||
The MD5 structure is not checked because it is not a checksum of the block's
|
||||
bytes in the file. It names a region of device flash, usually aligned to erase
|
||||
pages, whose MD5 a flashing tool compares to skip writing. Producers also
|
||||
disagree on its place: the specification puts it in the last 24 bytes of the
|
||||
data area, while esptool v4.12.0 and v5.6.0 write it right after a shorter final
|
||||
payload, so those 24 bytes hold zeros or part of that structure.
|
||||
|
||||
## DfuSe regions and metadata
|
||||
|
||||
Each element is one entry at its own file range, named
|
||||
`target-<t>/element-<e>-0x<start>-0x<end>.bin` with zero-based target and
|
||||
element indexes, for example `target-0/element-0-0x08000000-0x0800ffff.bin`.
|
||||
Elements are never merged. Each occurrence's `source_location` is the byte range
|
||||
of the element data in the file.
|
||||
|
||||
Element `format_metadata` (`dfuse-element`) records `target_index`,
|
||||
`alternate_setting`, `target_name`, `element_index`, `start_address`,
|
||||
`end_address` and `length`. `target_name` is `null` when `bTargetNamed` is 0;
|
||||
otherwise it is `szTargetName` up to its first NUL, one character per byte.
|
||||
`compare_artifacts` treats an element's alternate setting and target name as
|
||||
its declaration, so the same bytes moved to another device memory are
|
||||
`changed` in `metadata`.
|
||||
|
||||
Root `format_metadata` (`dfuse`) records:
|
||||
|
||||
| Field | Meaning |
|
||||
| ---------------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||
| `image_size`, `target_count` | `DFUImageSize` and `bTargets` from the prefix |
|
||||
| `element_count`, `empty_element_count`, `data_bytes` | All elements, elements of 0 bytes (which have no entry), and element data bytes |
|
||||
| `overlapping_element_count` | Elements that overlap an earlier element of the same alternate setting |
|
||||
| `device_version`, `product_id`, `vendor_id` | `bcdDevice`, `idProduct` and `idVendor` from the suffix |
|
||||
| `suffix_crc`, `computed_crc` | The stored `dwCRC`, and the same CRC computed over every byte before it |
|
||||
|
||||
Recognition already requires `bVersion` 1 and `bcdDFU` `0x011A`, so they are
|
||||
not repeated. The DFU suffix CRC is CRC-32 without the final inversion, as
|
||||
dfu-util stores and checks it.
|
||||
|
||||
Producers disagree on `DFUImageSize`. UM0391 calls it the file length, and
|
||||
dfu-util's `dfuse-pack.py` writes that value since commit `ee6b393d`
|
||||
(2020-10-08). Earlier `dfuse-pack.py` and MicroPython's `tools/dfu.py`
|
||||
(checked at `cb99ca98`) write the length before the suffix, and an ST community
|
||||
report describes ST's DfuFileMgr the same way; REA has not run DfuFileMgr.
|
||||
dfu-util's own parser ignores the field. REA accepts either value and keeps the
|
||||
declared one in `image_size`.
|
||||
|
||||
## DfuSe facts, limitations and failures
|
||||
|
||||
| Situation | Outcome |
|
||||
| ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
| `dwCRC` differs from the computed CRC | Follows `integrity_policy`, below |
|
||||
| A target without the `Target` signature | Fails with its byte offset |
|
||||
| `DFUImageSize` that equals neither the file length nor the bytes before the suffix | Fails with the declared size, the suffix offset, the file length and whether the suffix CRC matches |
|
||||
| A target size or an element size that disagrees with the bytes that hold it | Fails with the byte offset and the declared and available sizes |
|
||||
| An element whose address range wraps past `0xFFFFFFFF` | Fails with its byte offset |
|
||||
| Overlapping elements of one alternate setting | Kept as separate entries; a limitation names the first four; coverage is unaffected |
|
||||
| The same addresses in different alternate settings | Separate entries; an alternate setting selects a device memory |
|
||||
| Elements of 0 bytes, and targets with no elements | No entry; counted; a limitation names the first four, with address and alternate setting |
|
||||
|
||||
A CRC mismatch is an integrity contradiction, never proof of a cause. With the
|
||||
default `integrity_policy` `fail`, inspection and extraction fail with
|
||||
`artifact_integrity_mismatch`, whose details carry `algorithm: "dfu-crc32"`,
|
||||
`declared_crc32` and `calculated_crc32`. With `record-and-continue`
|
||||
(CLI `--integrity-policy record-and-continue`), the root occurrence becomes
|
||||
`mismatched`, `integrity_contradictions` records the pair for logical path `.`,
|
||||
and extraction carries that contradiction, because every element came from the
|
||||
contradicted container. The root occurrence has no `declared_integrity`: that
|
||||
field holds entry digests, so the declared CRC appears in the contradiction and,
|
||||
as a number, in `suffix_crc`. Under `fail`, the elements are hashed before the
|
||||
mismatch is raised.
|
||||
|
||||
## Limits
|
||||
|
||||
Each file must be at most 128 MiB. The UF2 decoder reserves at most 476 bytes
|
||||
per complete block before reading, merges payloads into regions of at most the
|
||||
same total size, and keeps one numbering entry per valid block, so its peak is
|
||||
about twice the decoded bytes. It retains at most 65,536 regions as entries; the
|
||||
order of blocks does not count against that bound. The DfuSe decoder holds the
|
||||
file once, serves elements as views of it, and retains at most 65,536 elements
|
||||
with bytes as entries; empty elements are only counted. These are fixed
|
||||
resource bounds, not format limits, and fail as limits (`file-size` and
|
||||
`capacity`) with no runtime override. Configurable ceilings are tracked in
|
||||
[#6176](https://github.com/morluto/rea/issues/6176).
|
||||
|
||||
Decoded regions are cached only for the reader's lifetime, and each open checks
|
||||
the admitted carrier identity again. These readers share the Intel HEX
|
||||
reader's region streams, cancellation and close behavior.
|
||||
|
||||
## Analyzing a region
|
||||
|
||||
Inventory does not establish the CPU, byte order or memory map, or an entry
|
||||
point. To disassemble a region, extract it and open the region file with
|
||||
`open_binary` using format `raw-binary`, an architecture you choose, and
|
||||
`entry_point` set to the region start address; raw-binary maps file offset 0 at
|
||||
`entry_point`. Each region is opened separately, so references between regions
|
||||
are not resolved.
|
||||
|
||||
## Verification
|
||||
|
||||
Fixtures written from the UF2 specification and from ST UM0391 cover block and
|
||||
element layouts, numbering groups and reused numbers, families, rewrites and
|
||||
conflicts, invalid and partial blocks, not-main-flash and file-container blocks,
|
||||
structural faults, overlaps, both integrity policies, accepted boundaries beside
|
||||
each refusal, resource refusals, inventory, extraction, comparison and reader
|
||||
identity checks. The reader lifecycle (cancellation, close and stream
|
||||
ownership) is covered by the Intel HEX reader tests, which exercise the shared
|
||||
reader. Two DfuSe fixtures are byte-identical, CRC included, to what the
|
||||
`build()` functions of dfu-util's `dfuse-pack.py` (at `f9a537c9`) and
|
||||
MicroPython's `tools/dfu.py` (at `cb99ca98`) produce for the same image, one for
|
||||
each `DFUImageSize` convention. Those functions were transcribed and run under
|
||||
Python 3; the scripts themselves were not run. CLI and stdio MCP tests exercise
|
||||
the same artifact provider and advertised schemas.
|
||||
|
||||
`npm run verify:uf2` runs esptool's `merge-bin --format uf2` on generated
|
||||
inputs at known addresses, with and without MD5, with 256-byte payloads, with
|
||||
two inputs of equal block count, and on a concatenation of two chips' outputs.
|
||||
Inspection and extraction must return every input at its address under
|
||||
esptool's own familyID. It passed with esptool v4.12.0 and v5.6.0 on macOS. Set
|
||||
`REA_ESPTOOL_PYTHON` to a Python that runs `-m esptool`. Pico SDK, picotool,
|
||||
`uf2conv.py`, TinyUF2 and real DfuSe producers remain unverified, and no device
|
||||
accepted these files in REA's tests.
|
||||
@@ -9,6 +9,7 @@ upstream texts; they do not change [REA's MIT license](../LICENSE).
|
||||
| `npm-dependencies.txt` | Every package path in the root lockfile, with its exact version, declared license, scope, download URL, and integrity. Includes development and optional platform entries. |
|
||||
| `npm-runtime-notices.txt` | Original license, copyright, and NOTICE texts collected from integrity-verified locked runtime npm archives across all platforms; also the license of Rolldown's generated runtime helpers. The archive manifest's name and version are checked. |
|
||||
| `protobuf-es-v2.16.0-LICENSE.txt` | Original Apache license omitted from the protobuf-es npm archive, preserved from its v2.16.0 source commit. Its embedded Google BSD notice is retained in `npm-runtime-notices.txt`. |
|
||||
| `microsoft-uf2-MIT.txt` | Unchanged `LICENSE.txt` from microsoft/uf2 revision `cf3bd7f9c274b949528f8cd6a19eef2502a687d2`, for the UF2 family names copied from its `utils/uf2families.json`. |
|
||||
| `nativeaot-parser-MIT.txt` | Original Washi notice for derived NativeAOT parser code; source revision is recorded in the parser and the root notices. |
|
||||
| `dotnet-runtime-MIT.txt` | Original .NET Foundation MIT text from the NativeAOT reference revision `a3b2d40328be0be3f8dd43f0e5cc925b8827b044`. |
|
||||
| `ghidra-12.1.4-{LICENSE,NOTICE,DISCLAIMER}.txt` | Unchanged top-level documents from `Ghidra_12.1.4_build`. Ghidra is not bundled; these reference copies do not replace its component and GPL support-program notices. The US-government-specific disclaimer applies to Ghidra, not to REA. |
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
Microsoft UF2
|
||||
|
||||
The MIT License (MIT)
|
||||
|
||||
Copyright (c) Microsoft Corporation
|
||||
|
||||
All rights reserved.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -78,5 +78,10 @@
|
||||
"file": "licenses/pefile-2024.8.26-LICENSE.txt",
|
||||
"source": "https://raw.githubusercontent.com/erocarrera/pefile/4b3b1e2e568a88d4f1897d694d684f23d9e270c4/LICENSE",
|
||||
"sha256": "b44409c067c6da52bfb54bb6624fb11a7b52157c3a13ae1400232f8196e86ad3"
|
||||
},
|
||||
{
|
||||
"file": "licenses/microsoft-uf2-MIT.txt",
|
||||
"source": "https://raw.githubusercontent.com/microsoft/uf2/cf3bd7f9c274b949528f8cd6a19eef2502a687d2/LICENSE.txt",
|
||||
"sha256": "c581accf1e5449482b70ac5fcdfb171599317ccff559cf0794d771671486f800"
|
||||
}
|
||||
]
|
||||
@@ -239,6 +239,7 @@
|
||||
"verify:javascript:retained-facts": "npm run build:cached && node scripts/verify/javascript/retained-facts.mjs",
|
||||
"verify:application-workflows": "npm run build:cached && node scripts/verify/javascript/application-workflows.mjs",
|
||||
"verify:managed": "npm run build:cached && node scripts/verify/managed/conformance.mjs",
|
||||
"verify:uf2": "npm run build:cached && node scripts/verify/firmware/uf2-esptool.mjs",
|
||||
"verify:firmware": "npm run build:cached && node scripts/verify/firmware/analysis.mjs",
|
||||
"verify:devicetree": "npm run build:cached && node scripts/verify/firmware/devicetree.mjs",
|
||||
"verify:esp-images": "npm run build:cached && node scripts/verify/firmware/esp-image.mjs",
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
#!/usr/bin/env node
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { createVerifierCommands } from "../../lib/verify-commands.mjs";
|
||||
import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs";
|
||||
|
||||
// Optional verification tool only; the UF2 reader starts no tool.
|
||||
const python = process.env.REA_ESPTOOL_PYTHON ?? "python3";
|
||||
const entrypoint = fileURLToPath(new URL("../../rea.mjs", import.meta.url));
|
||||
const options = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 };
|
||||
/** Every tool runs as an owned process tree that close() stops. */
|
||||
const commands = await createVerifierCommands();
|
||||
|
||||
let workspace;
|
||||
const path = (name) => join(workspace, name);
|
||||
|
||||
/** Deterministic input bytes; lengths include one that is not word-aligned. */
|
||||
const input = (length, seed) =>
|
||||
Buffer.from(Array.from({ length }, (_, index) => (index * 13 + seed) & 0xff));
|
||||
|
||||
// `otadata` matches the table's block count, so esptool's per-input numbering
|
||||
// from 0 puts both inputs in one numbering group.
|
||||
const FILES = {
|
||||
bootloader: input(1000, 1),
|
||||
table: input(3072, 2),
|
||||
otadata: input(3072, 4),
|
||||
app: input(501, 3),
|
||||
};
|
||||
|
||||
const CASES = [
|
||||
{
|
||||
name: "esp32s3.uf2",
|
||||
chip: "esp32s3",
|
||||
layout: [
|
||||
[0x0, "bootloader"],
|
||||
[0x8000, "table"],
|
||||
[0xd000, "otadata"],
|
||||
[0x10000, "app"],
|
||||
],
|
||||
extra: [],
|
||||
md5: true,
|
||||
payload: 452,
|
||||
},
|
||||
{
|
||||
name: "esp32s2-tinyuf2.uf2",
|
||||
chip: "esp32s2",
|
||||
layout: [
|
||||
[0x1000, "bootloader"],
|
||||
[0x8000, "table"],
|
||||
],
|
||||
extra: ["--chunk-size", "256", "--md5-disable"],
|
||||
md5: false,
|
||||
payload: 256,
|
||||
},
|
||||
];
|
||||
|
||||
const tool = async (args) => {
|
||||
const { stdout, stderr } = await commands.exec(
|
||||
python,
|
||||
["-m", "esptool", ...args],
|
||||
options,
|
||||
);
|
||||
return `${stdout}${stderr}`;
|
||||
};
|
||||
|
||||
/** esptool's own family ID for a chip: the oracle REA's table must match. */
|
||||
const familyOf = async (chip) => {
|
||||
const { stdout } = await commands.exec(
|
||||
python,
|
||||
[
|
||||
"-c",
|
||||
"import sys, esptool.targets as t; print(t.CHIP_DEFS[sys.argv[1]].UF2_FAMILY_ID)",
|
||||
chip,
|
||||
],
|
||||
options,
|
||||
);
|
||||
return Number(stdout.trim());
|
||||
};
|
||||
|
||||
const rea = async (command, file) => {
|
||||
const { stdout } = await commands.exec(
|
||||
process.execPath,
|
||||
[entrypoint, command, file, "--json"],
|
||||
{
|
||||
...options,
|
||||
env: {
|
||||
...process.env,
|
||||
REA_LOG_LEVEL: "silent",
|
||||
TMPDIR: workspace,
|
||||
TMP: workspace,
|
||||
TEMP: workspace,
|
||||
},
|
||||
},
|
||||
);
|
||||
return JSON.parse(stdout).normalized_result;
|
||||
};
|
||||
|
||||
const familyDirectory = (familyId) =>
|
||||
`family-0x${familyId.toString(16).padStart(8, "0")}`;
|
||||
const hex32 = (value) => `0x${value.toString(16).padStart(8, "0")}`;
|
||||
const blocksOf = (entry, name) => Math.ceil(FILES[name].length / entry.payload);
|
||||
const inventoryOf = async (file) =>
|
||||
(await rea("inspect-artifact", path(file))).substeps[0].evidence
|
||||
.normalized_result;
|
||||
const rootOf = (inventory) =>
|
||||
inventory.occurrences.find((row) => row.logical_path === ".").format_metadata;
|
||||
|
||||
/** Write the inputs, merge each case with esptool, and concatenate the cases. */
|
||||
const produce = async (v5) => {
|
||||
for (const [name, bytes] of Object.entries(FILES))
|
||||
await writeFile(path(`${name}.bin`), bytes);
|
||||
for (const entry of CASES) {
|
||||
await tool([
|
||||
"--chip",
|
||||
entry.chip,
|
||||
v5 ? "merge-bin" : "merge_bin",
|
||||
"--format",
|
||||
"uf2",
|
||||
...entry.extra,
|
||||
"-o",
|
||||
path(entry.name),
|
||||
...entry.layout.flatMap(([address, name]) => [
|
||||
hex32(address),
|
||||
path(`${name}.bin`),
|
||||
]),
|
||||
]);
|
||||
entry.familyId = await familyOf(entry.chip);
|
||||
}
|
||||
// The UF2 specification makes a multi-family file by concatenation.
|
||||
await writeFile(
|
||||
path("concatenated.uf2"),
|
||||
Buffer.concat(
|
||||
await Promise.all(CASES.map((entry) => readFile(path(entry.name)))),
|
||||
),
|
||||
);
|
||||
};
|
||||
|
||||
/** Root facts a complete file from these cases must report. */
|
||||
const expectedRoot = (included) => {
|
||||
const blocks = included.flatMap((entry) =>
|
||||
entry.layout.map(([, name]) => ({
|
||||
group: `${entry.chip}:${String(blocksOf(entry, name))}`,
|
||||
count: blocksOf(entry, name),
|
||||
md5: entry.md5,
|
||||
})),
|
||||
);
|
||||
const total = blocks.reduce((sum, { count }) => sum + count, 0);
|
||||
// Inputs of one chip with equal block counts form one numbering group,
|
||||
// and every input after the first in a group repeats its numbers.
|
||||
const groups = new Map();
|
||||
for (const { group, count } of blocks)
|
||||
groups.set(group, [...(groups.get(group) ?? []), count]);
|
||||
return {
|
||||
block_count: total,
|
||||
invalid_block_count: 0,
|
||||
flash_block_count: total,
|
||||
md5_block_count: blocks
|
||||
.filter(({ md5 }) => md5)
|
||||
.reduce((sum, { count }) => sum + count, 0),
|
||||
family_count: included.length,
|
||||
numbering_group_count: groups.size,
|
||||
missing_block_count: 0,
|
||||
ambiguous_numbering_group_count: 0,
|
||||
repeated_block_count: [...groups.values()].reduce(
|
||||
(sum, counts) => sum + (counts.length - 1) * (counts[0] ?? 0),
|
||||
0,
|
||||
),
|
||||
duplicate_bytes: 0,
|
||||
};
|
||||
};
|
||||
|
||||
/** Every input must come back at its address under esptool's familyID. */
|
||||
const check = async (file, included) => {
|
||||
const inventory = await inventoryOf(file);
|
||||
assert.equal(inventory.manifest.root_format, "uf2", file);
|
||||
assert.equal(inventory.coverage.status, "complete", file);
|
||||
const root = rootOf(inventory);
|
||||
const expected = expectedRoot(included);
|
||||
assert.deepEqual(
|
||||
Object.fromEntries(Object.keys(expected).map((key) => [key, root[key]])),
|
||||
expected,
|
||||
file,
|
||||
);
|
||||
const inputs = included.flatMap((entry) =>
|
||||
entry.layout.map(([address, name]) => ({
|
||||
path: `${familyDirectory(entry.familyId)}/${hex32(address)}-${hex32(address + FILES[name].length - 1)}.bin`,
|
||||
familyId: entry.familyId,
|
||||
bytes: FILES[name],
|
||||
})),
|
||||
);
|
||||
const regions = inventory.occurrences.filter(
|
||||
(row) => row.format_metadata?.kind === "uf2-region",
|
||||
);
|
||||
assert.deepEqual(
|
||||
regions.map((row) => row.logical_path).sort(),
|
||||
inputs.map((region) => region.path).sort(),
|
||||
file,
|
||||
);
|
||||
for (const region of regions)
|
||||
assert.equal(
|
||||
region.format_metadata.family_id,
|
||||
inputs.find((row) => row.path === region.logical_path).familyId,
|
||||
`${file} ${region.logical_path}`,
|
||||
);
|
||||
const extracted = await rea("extract-artifact", path(file));
|
||||
assert.equal(extracted.containment_verified, true, file);
|
||||
for (const region of inputs)
|
||||
assert.equal(
|
||||
Buffer.compare(
|
||||
await readFile(join(extracted.output_root, region.path)),
|
||||
region.bytes,
|
||||
),
|
||||
0,
|
||||
`${file} ${region.path}`,
|
||||
);
|
||||
return {
|
||||
file,
|
||||
families: included.map((entry) => ({
|
||||
chip: entry.chip,
|
||||
family_id: hex32(entry.familyId),
|
||||
family_name: regions.find(
|
||||
(row) => row.format_metadata.family_id === entry.familyId,
|
||||
).format_metadata.family_name,
|
||||
})),
|
||||
blocks: root.block_count,
|
||||
regions: regions.length,
|
||||
md5_blocks: root.md5_block_count,
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Losing one block of one of two equal-count inputs must not read as
|
||||
* complete: the other input still carries every block number.
|
||||
*/
|
||||
const checkLostBlock = async () => {
|
||||
const [s3] = CASES;
|
||||
const position = s3.layout.findIndex(([, name]) => name === "otadata");
|
||||
const lostIndex =
|
||||
s3.layout
|
||||
.slice(0, position)
|
||||
.reduce((sum, [, name]) => sum + blocksOf(s3, name), 0) + 1;
|
||||
const whole = await readFile(path(s3.name));
|
||||
await writeFile(
|
||||
path("lost-block.uf2"),
|
||||
Buffer.concat([
|
||||
whole.subarray(0, lostIndex * 512),
|
||||
whole.subarray((lostIndex + 1) * 512),
|
||||
]),
|
||||
);
|
||||
const inventory = await inventoryOf("lost-block.uf2");
|
||||
const root = rootOf(inventory);
|
||||
assert.equal(inventory.coverage.status, "unknown", "lost-block.uf2");
|
||||
assert.equal(root.ambiguous_numbering_group_count, 1, "lost-block.uf2");
|
||||
assert.equal(root.missing_block_count, 0, "lost-block.uf2");
|
||||
return {
|
||||
file: "lost-block.uf2",
|
||||
removed_block: lostIndex,
|
||||
coverage: inventory.coverage.status,
|
||||
ambiguous_numbering_groups: root.ambiguous_numbering_group_count,
|
||||
};
|
||||
};
|
||||
|
||||
await withVerifierCleanup(async () => {
|
||||
const versionText = await tool(["version"]);
|
||||
const version = /\bv?(\d+)\.(\d+)\.(\d+)\b/u.exec(versionText);
|
||||
assert.ok(version, `Unrecognized esptool version: ${versionText}`);
|
||||
workspace = await mkdtemp(join(tmpdir(), "rea-uf2-esptool-"));
|
||||
await produce(Number(version[1]) >= 5);
|
||||
const report = { esptool: version[0], cases: [] };
|
||||
for (const entry of CASES)
|
||||
report.cases.push(await check(entry.name, [entry]));
|
||||
report.cases.push(await check("concatenated.uf2", CASES));
|
||||
report.cases.push(await checkLostBlock());
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
}, [
|
||||
() => commands.close(),
|
||||
() =>
|
||||
workspace === undefined
|
||||
? undefined
|
||||
: rm(workspace, { recursive: true, force: true }),
|
||||
]);
|
||||
@@ -86,6 +86,12 @@ import { hasGodotPckHeader } from "../domain/godotPckFormat.js";
|
||||
import { hasUImageMagic } from "../domain/uimageFormat.js";
|
||||
import { hasUnityFsHeader } from "../domain/unityfsFormat.js";
|
||||
import { detectHexRecordEncoding } from "../domain/firmware/hexRecords.js";
|
||||
import { hasUf2Block } from "../domain/firmware/uf2.js";
|
||||
import {
|
||||
DFUSE_SUFFIX_BYTES,
|
||||
hasDfuSePrefix,
|
||||
isDfuSeFile,
|
||||
} from "../domain/firmware/dfuse.js";
|
||||
import { hasFitImagesNode } from "../domain/firmware/fitImage.js";
|
||||
import {
|
||||
locateEspFlashLayout,
|
||||
@@ -777,6 +783,22 @@ const detectArtifactFormat = async (
|
||||
if (hasUImageMagic(magic)) return "uimage";
|
||||
const hexRecords = detectHexRecordEncoding(magic);
|
||||
if (hexRecords !== undefined) return hexRecords;
|
||||
if (hasUf2Block(magic)) return "uf2";
|
||||
if (
|
||||
hasDfuSePrefix(magic) &&
|
||||
initial.size >= DFUSE_SUFFIX_BYTES &&
|
||||
isDfuSeFile(
|
||||
magic,
|
||||
await readHandleRange(
|
||||
handle,
|
||||
initial.size - DFUSE_SUFFIX_BYTES,
|
||||
DFUSE_SUFFIX_BYTES,
|
||||
signal,
|
||||
),
|
||||
initial.size,
|
||||
)
|
||||
)
|
||||
return "dfuse";
|
||||
// A C3 or S3 dump begins with its bootloader, so a layout comes first.
|
||||
const flash = await locateEspFlashLayout(
|
||||
initial.size,
|
||||
@@ -870,6 +892,8 @@ const isArchiveFormat = (
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"esp-flash-image",
|
||||
"fit",
|
||||
].includes(format);
|
||||
|
||||
@@ -121,6 +121,8 @@ const TARGETS: TargetFixtures = {
|
||||
"godot-pck": { ...archive, format: "godot-pck" },
|
||||
unityfs: { ...archive, format: "unityfs" },
|
||||
uimage: { ...archive, format: "uimage" },
|
||||
uf2: { ...archive, format: "uf2" },
|
||||
dfuse: { ...archive, format: "dfuse" },
|
||||
dmg: { ...archive, format: "dmg" },
|
||||
pkg: { ...archive, format: "pkg" },
|
||||
"intel-hex": { ...archive, format: "intel-hex" },
|
||||
|
||||
@@ -19,13 +19,18 @@ import {
|
||||
observeMzWindowsHeader,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
} from "../domain/dosMz.js";
|
||||
import { DFUSE_SUFFIX_BYTES } from "../domain/firmware/dfuse.js";
|
||||
|
||||
/** Initial bytes required by the other bounded format classifiers. */
|
||||
export const ARTIFACT_CLASSIFICATION_PREFIX_BYTES = 8_192;
|
||||
/** Final bytes kept for formats identified by a trailer: the DFU suffix. */
|
||||
const ARTIFACT_CLASSIFICATION_SUFFIX_BYTES = DFUSE_SUFFIX_BYTES;
|
||||
|
||||
/** Retained prefix plus bounded format-specific observations from the stream. */
|
||||
export interface ArtifactClassificationEvidence {
|
||||
readonly prefix: Buffer;
|
||||
/** The last bytes of the stream, when the producer observed its end. */
|
||||
readonly suffix?: Buffer;
|
||||
readonly asarHeader: FinalAsarHeaderProbeObservation;
|
||||
readonly fatSliceHeaders?: FatSliceHeaderEvidence;
|
||||
readonly mzWindowsHeaderSignature?: MzWindowsHeaderSignatureEvidence;
|
||||
@@ -76,6 +81,7 @@ const observeFatSliceHeaders = (input: {
|
||||
|
||||
const finishClassificationEvidence = (
|
||||
prefix: Buffer,
|
||||
suffix: Buffer | undefined,
|
||||
archiveBytes: number,
|
||||
finishAsar: (archiveBytes: number) => FinalAsarHeaderProbeObservation,
|
||||
sliceHeadersByOffset: Map<number, MachoHeaderPrefixFacts>,
|
||||
@@ -106,6 +112,7 @@ const finishClassificationEvidence = (
|
||||
: undefined;
|
||||
return {
|
||||
prefix,
|
||||
...(suffix === undefined ? {} : { suffix }),
|
||||
asarHeader: finishAsar(archiveBytes),
|
||||
...(fatSliceHeaders === undefined ? {} : { fatSliceHeaders }),
|
||||
...(mzWindowsHeaderSignature === undefined
|
||||
@@ -132,6 +139,18 @@ export const createArtifactClassificationProbe = (
|
||||
let mzWindowsHeaderSignature: MzWindowsHeaderSignatureEvidence | undefined;
|
||||
let mzWindowsHeaderOffset: number | undefined;
|
||||
let mzHeaderChecked = false;
|
||||
let suffix = Buffer.alloc(0);
|
||||
const observeSuffix = (bytes: Uint8Array): void => {
|
||||
const joined =
|
||||
bytes.length >= ARTIFACT_CLASSIFICATION_SUFFIX_BYTES
|
||||
? bytes
|
||||
: Buffer.concat([suffix, bytes]);
|
||||
suffix = Buffer.from(
|
||||
joined.subarray(
|
||||
Math.max(0, joined.length - ARTIFACT_CLASSIFICATION_SUFFIX_BYTES),
|
||||
),
|
||||
);
|
||||
};
|
||||
const observeSignatureWindow = (
|
||||
offset: number,
|
||||
sourceOffset: number,
|
||||
@@ -258,11 +277,14 @@ export const createArtifactClassificationProbe = (
|
||||
}
|
||||
if (mzWindowsHeaderOffset !== undefined)
|
||||
observeSignatureWindow(mzWindowsHeaderOffset, streamOffset, bytes);
|
||||
observeSuffix(bytes);
|
||||
observedBytes += bytes.length;
|
||||
},
|
||||
finish(archiveBytes: number): ArtifactClassificationEvidence {
|
||||
return finishClassificationEvidence(
|
||||
retainedPrefix(),
|
||||
// Bytes the probe did not observe could still follow its last write.
|
||||
observedBytes === archiveBytes ? suffix : undefined,
|
||||
archiveBytes,
|
||||
(bytes) => asar.finish(bytes),
|
||||
fatSliceHeadersByOffset,
|
||||
|
||||
@@ -4,6 +4,7 @@ import type {
|
||||
ArtifactLimitBoundary,
|
||||
} from "../domain/artifactOperationError.js";
|
||||
import type {
|
||||
ArtifactContainerIntegrityPair,
|
||||
ArtifactIntegrityClaim,
|
||||
ArtifactIntegrityDetails,
|
||||
} from "../domain/artifactIntegrity.js";
|
||||
@@ -204,6 +205,8 @@ export interface ArtifactReader {
|
||||
| "intel-hex"
|
||||
| "srec"
|
||||
| "esp-image"
|
||||
| "uf2"
|
||||
| "dfuse"
|
||||
| "esp-flash-image"
|
||||
| "fit"
|
||||
| "file";
|
||||
@@ -214,6 +217,8 @@ export interface ArtifactReader {
|
||||
open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable>;
|
||||
provenance(): readonly ArtifactCommand[];
|
||||
formatMetadata?(): ArtifactFormatMetadata | undefined;
|
||||
/** A checksum the container declares over its own bytes, with the observed value. */
|
||||
containerIntegrity?(): ArtifactContainerIntegrityPair | undefined;
|
||||
/** Bind materialization to the exact inventoried carrier before creating outputs. */
|
||||
prepareContainer?(
|
||||
expectedSha256: string,
|
||||
|
||||
@@ -0,0 +1,297 @@
|
||||
import { Readable } from "node:stream";
|
||||
import { setImmediate } from "node:timers/promises";
|
||||
|
||||
import type { ArtifactFormatMetadata } from "../domain/artifactFormatMetadata.js";
|
||||
import type { ArtifactContainerIntegrityPair } from "../domain/artifactIntegrity.js";
|
||||
import type { Result } from "../domain/result.js";
|
||||
import type {
|
||||
RegionDecodeFailure,
|
||||
RegionDecoder,
|
||||
} from "../domain/firmware/regionDecoder.js";
|
||||
import {
|
||||
ArtifactReaderFailure,
|
||||
copyArtifactEntry,
|
||||
sameArtifactEntry,
|
||||
type ArtifactEntry,
|
||||
type ArtifactReader,
|
||||
type ArtifactReaderTraversal,
|
||||
type ArtifactTraversalOptions,
|
||||
} from "./ArtifactReader.js";
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
|
||||
import { ArtifactFileSource } from "./ArtifactFileSource.js";
|
||||
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
|
||||
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
|
||||
|
||||
/** One extractable region: its inventory facts and the decoded bytes behind them. */
|
||||
export interface DecodedRegion {
|
||||
readonly path: string;
|
||||
/** The file offset of the bytes when they are one contiguous file range. */
|
||||
readonly byteOffset: number | null;
|
||||
readonly formatMetadata: ArtifactFormatMetadata;
|
||||
readonly bytes: Uint8Array;
|
||||
}
|
||||
|
||||
/** What one format's decoder established about the whole file. */
|
||||
export interface DecodedRegions {
|
||||
readonly metadata: ArtifactFormatMetadata;
|
||||
readonly regions: readonly DecodedRegion[];
|
||||
/** Omitted when every byte was examined and no limitation applies. */
|
||||
readonly coverage?: ArtifactReaderTraversal;
|
||||
readonly containerIntegrity?: ArtifactContainerIntegrityPair;
|
||||
}
|
||||
|
||||
/** How one firmware format turns admitted file bytes into regions. */
|
||||
export interface RegionDecoding<Image> {
|
||||
readonly label: string;
|
||||
readonly keyPrefix: string;
|
||||
createDecoder(
|
||||
fileBytes: number,
|
||||
): Result<RegionDecoder<Image>, RegionDecodeFailure>;
|
||||
describe(image: Image): DecodedRegions;
|
||||
}
|
||||
|
||||
const addressText = (address: number): string =>
|
||||
`0x${address.toString(16).padStart(8, "0")}`;
|
||||
|
||||
/** The shared region file name: inclusive start and end as eight hex digits. */
|
||||
export const regionFileName = (start: number, end: number): string =>
|
||||
`${addressText(start)}-${addressText(end)}.bin`;
|
||||
|
||||
/**
|
||||
* Decode a whole firmware file once, retain its region bytes for the reader's
|
||||
* lifetime, and serve each region as an inventory entry.
|
||||
*/
|
||||
export class DecodedRegionArtifactReader<Image> implements ArtifactReader {
|
||||
readonly #scope = new ArtifactResourceScope();
|
||||
readonly #entries = new Map<string, ArtifactEntry>();
|
||||
readonly #regions = new Map<string, Uint8Array>();
|
||||
readonly #streams = new Set<Readable>();
|
||||
readonly #source: ArtifactFileSource;
|
||||
readonly #decoding: RegionDecoding<Image>;
|
||||
#prepared: Promise<void> | undefined;
|
||||
/** Whole-file facts only; the region map is the sole owner of region bytes. */
|
||||
#described: Omit<DecodedRegions, "regions"> | undefined;
|
||||
#closing = false;
|
||||
|
||||
constructor(
|
||||
path: string,
|
||||
readonly format: "intel-hex" | "srec" | "uf2" | "dfuse",
|
||||
decoding: RegionDecoding<Image>,
|
||||
admitted?: StableRegularFileDescriptor,
|
||||
) {
|
||||
this.#decoding = decoding;
|
||||
this.#source = new ArtifactFileSource(path, decoding.label, admitted);
|
||||
}
|
||||
|
||||
formatMetadata(): ArtifactFormatMetadata | undefined {
|
||||
const metadata = this.#described?.metadata;
|
||||
return metadata === undefined ? undefined : { ...metadata };
|
||||
}
|
||||
|
||||
containerIntegrity(): ArtifactContainerIntegrityPair | undefined {
|
||||
const integrity = this.#described?.containerIntegrity;
|
||||
return integrity === undefined ? undefined : { ...integrity };
|
||||
}
|
||||
|
||||
async prepareContainer(
|
||||
expectedSha256: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
await this.#scope.run(async () => {
|
||||
this.#assertOpen(signal);
|
||||
await this.#source.bindSha256(expectedSha256, signal);
|
||||
});
|
||||
}
|
||||
|
||||
async *entries(
|
||||
signal?: AbortSignal,
|
||||
options?: ArtifactTraversalOptions,
|
||||
): AsyncIterable<ArtifactEntry> {
|
||||
await this.#scope.run(async () => {
|
||||
this.#assertOpen(signal);
|
||||
this.#prepared ??= this.#prepare(signal);
|
||||
await this.#prepared;
|
||||
await this.#source.read(signal);
|
||||
});
|
||||
const coverage = this.#described?.coverage;
|
||||
if (coverage !== undefined)
|
||||
options?.onCoverage?.({
|
||||
...coverage,
|
||||
limitations: [...coverage.limitations],
|
||||
});
|
||||
let index = 0;
|
||||
for (const entry of this.#entries.values()) {
|
||||
if (index % 256 === 0) await setImmediate();
|
||||
index += 1;
|
||||
this.#assertOpen(signal);
|
||||
yield copyArtifactEntry(entry);
|
||||
}
|
||||
}
|
||||
|
||||
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
|
||||
return this.#scope.run(async () => {
|
||||
this.#assertOpen(signal);
|
||||
const stored = this.#entries.get(entry.adapterKey);
|
||||
const bytes = this.#regions.get(entry.adapterKey);
|
||||
if (
|
||||
stored === undefined ||
|
||||
bytes === undefined ||
|
||||
!sameArtifactEntry(entry, stored)
|
||||
)
|
||||
throw new ArtifactReaderFailure(
|
||||
"integrity",
|
||||
`${this.#decoding.label} entry does not match its owning inventory`,
|
||||
);
|
||||
// Decoded bytes are retained, so only the carrier identity is rechecked.
|
||||
await this.#source.read(signal);
|
||||
this.#assertOpen(signal);
|
||||
const output = Readable.from(this.#regionChunks(bytes, signal), {
|
||||
objectMode: false,
|
||||
});
|
||||
const onAbort = (): void => {
|
||||
output.destroy(
|
||||
new ArtifactReaderFailure(
|
||||
"cancelled",
|
||||
`${this.#decoding.label} region read cancelled`,
|
||||
{ cause: signal?.reason },
|
||||
),
|
||||
);
|
||||
};
|
||||
signal?.addEventListener("abort", onAbort, { once: true });
|
||||
output.once("close", () => signal?.removeEventListener("abort", onAbort));
|
||||
output.on("error", () => undefined);
|
||||
this.#streams.add(output);
|
||||
output.once("close", () => this.#streams.delete(output));
|
||||
// Draining the scope must wait for the readable side, including an
|
||||
// abandoned consumer and an in-flight asynchronous chunk producer.
|
||||
void this.#scope
|
||||
.run(
|
||||
() =>
|
||||
new Promise<void>((resolve) => {
|
||||
output.once("close", () => resolve());
|
||||
}),
|
||||
)
|
||||
.catch((cause: unknown) =>
|
||||
output.destroy(
|
||||
cause instanceof Error
|
||||
? cause
|
||||
: new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`${this.#decoding.label} stream ownership failed`,
|
||||
{ cause },
|
||||
),
|
||||
),
|
||||
);
|
||||
return output;
|
||||
});
|
||||
}
|
||||
|
||||
async *#regionChunks(
|
||||
bytes: Uint8Array,
|
||||
signal?: AbortSignal,
|
||||
): AsyncIterable<Buffer> {
|
||||
// Keep cached bytes isolated from consumers without copying a whole
|
||||
// region for every open. Match descriptor reads' 64 KiB chunk size.
|
||||
for (let offset = 0; offset < bytes.length; offset += 64 * 1024) {
|
||||
await setImmediate();
|
||||
this.#assertOpen(signal);
|
||||
yield Buffer.from(bytes.subarray(offset, offset + 64 * 1024));
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.#closing = true;
|
||||
for (const stream of this.#streams)
|
||||
stream.destroy(
|
||||
new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`${this.#decoding.label} reader closed`,
|
||||
),
|
||||
);
|
||||
await this.#scope.close();
|
||||
await this.#source.close();
|
||||
this.#entries.clear();
|
||||
this.#regions.clear();
|
||||
}
|
||||
|
||||
provenance(): readonly [] {
|
||||
return [];
|
||||
}
|
||||
|
||||
#assertOpen(signal?: AbortSignal): void {
|
||||
assertArtifactReadActive(signal);
|
||||
if (this.#closing)
|
||||
throw new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`${this.#decoding.label} reader is closed`,
|
||||
);
|
||||
}
|
||||
|
||||
async #prepare(signal?: AbortSignal): Promise<void> {
|
||||
const described = this.#decoding.describe(await this.#decode(signal));
|
||||
let index = 0;
|
||||
for (const region of described.regions) {
|
||||
if (index % 256 === 0) await setImmediate();
|
||||
this.#assertOpen(signal);
|
||||
const entry = {
|
||||
path: region.path,
|
||||
kind: "file" as const,
|
||||
declaredSize: region.bytes.length,
|
||||
compressedSize: null,
|
||||
executable: false,
|
||||
encrypted: false,
|
||||
byteOffset: region.byteOffset,
|
||||
declaredIntegrity: null,
|
||||
unpacked: false,
|
||||
limitations: [],
|
||||
adapterKey: `${this.#decoding.keyPrefix}:${String(index)}`,
|
||||
formatMetadata: region.formatMetadata,
|
||||
} satisfies ArtifactEntry;
|
||||
this.#entries.set(entry.adapterKey, entry);
|
||||
this.#regions.set(entry.adapterKey, region.bytes);
|
||||
index += 1;
|
||||
}
|
||||
const { regions: _regions, ...facts } = described;
|
||||
this.#described = facts;
|
||||
await this.#source.read(signal);
|
||||
}
|
||||
|
||||
async #decode(signal?: AbortSignal): Promise<Image> {
|
||||
const { handle, initial } = await this.#source.read(signal);
|
||||
const decoder = this.#decoding.createDecoder(initial.size);
|
||||
if (!decoder.ok) throw this.#failure(decoder.error);
|
||||
// Reading only the admitted extent keeps decoded bytes within the arena.
|
||||
if (initial.size > 0)
|
||||
for await (const chunk of readFileHandleChunks(handle, {
|
||||
end: initial.size - 1,
|
||||
signal,
|
||||
cancellationFailure: (reason) =>
|
||||
new ArtifactReaderFailure(
|
||||
"cancelled",
|
||||
`${this.#decoding.label} read cancelled`,
|
||||
{ cause: reason },
|
||||
),
|
||||
}) as AsyncIterable<Buffer>) {
|
||||
this.#assertOpen(signal);
|
||||
const written = decoder.value.write(chunk);
|
||||
if (!written.ok) throw this.#failure(written.error);
|
||||
}
|
||||
const image = decoder.value.finish();
|
||||
if (!image.ok) throw this.#failure(image.error);
|
||||
return image.value;
|
||||
}
|
||||
|
||||
#failure(failure: RegionDecodeFailure): ArtifactReaderFailure {
|
||||
return failure.kind === "format"
|
||||
? new ArtifactReaderFailure(
|
||||
"format",
|
||||
`${this.#decoding.label} is malformed: ${failure.message}`,
|
||||
)
|
||||
: new ArtifactReaderFailure(
|
||||
"limit",
|
||||
`${this.#decoding.label} exceeds a fixed limit: ${failure.message}`,
|
||||
failure.boundary === undefined ? {} : { boundary: failure.boundary },
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
|
||||
import {
|
||||
dfuseFile,
|
||||
type DfuSeFixture,
|
||||
} from "../domain/firmware/dfuse.fixture.js";
|
||||
import { ARTIFACT_COMPARISON_EXAMPLE } from "../contracts/artifactComparisonExample.js";
|
||||
import { compareArtifacts } from "../domain/artifactComparison.js";
|
||||
import { artifactInventoryResultSchema } from "../domain/artifactGraph.js";
|
||||
import { createEvidence } from "../domain/evidence.js";
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js";
|
||||
|
||||
const FIRMWARE: DfuSeFixture = {
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
name: "Internal Flash",
|
||||
elements: [
|
||||
{ address: 0x0800_0000, data: [1, 2, 3, 4] },
|
||||
{ address: 0x0800_4000, data: [5, 6] },
|
||||
],
|
||||
},
|
||||
{
|
||||
alternateSetting: 1,
|
||||
elements: [{ address: 0x1fff_c000, data: [0xaa, 0x55] }],
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const writeImage = async (name: string, bytes: Uint8Array): Promise<string> => {
|
||||
const directory = await createTestTempDirectory("rea-dfuse-");
|
||||
const path = join(directory, name);
|
||||
await writeFile(path, bytes);
|
||||
return path;
|
||||
};
|
||||
|
||||
it("inventories one region per element with its file range and target facts", async () => {
|
||||
const snapshot = await scanCanonicalArtifactInventory(
|
||||
await writeImage("firmware.dfu", dfuseFile(FIRMWARE)),
|
||||
);
|
||||
expect(snapshot.manifest.root_format).toBe("dfuse");
|
||||
expect(snapshot.coverage.status).toBe("complete");
|
||||
expect(snapshot.integrity_contradictions).toEqual([]);
|
||||
expect(
|
||||
snapshot.occurrences.find((row) => row.logical_path === "."),
|
||||
).toMatchObject({
|
||||
artifact_kind: "container",
|
||||
hash_status: "verified",
|
||||
format_metadata: { kind: "dfuse", target_count: 2, element_count: 3 },
|
||||
});
|
||||
expect(
|
||||
snapshot.occurrences
|
||||
.filter((row) => row.logical_path !== ".")
|
||||
.map((row) => [
|
||||
row.logical_path,
|
||||
row.source_location,
|
||||
row.format_metadata,
|
||||
]),
|
||||
).toEqual([
|
||||
[
|
||||
"target-0/element-0-0x08000000-0x08000003.bin",
|
||||
{ offset: 293, length: 4 },
|
||||
{
|
||||
kind: "dfuse-element",
|
||||
target_index: 0,
|
||||
alternate_setting: 0,
|
||||
target_name: "Internal Flash",
|
||||
element_index: 0,
|
||||
start_address: 0x0800_0000,
|
||||
end_address: 0x0800_0003,
|
||||
length: 4,
|
||||
},
|
||||
],
|
||||
[
|
||||
"target-0/element-1-0x08004000-0x08004001.bin",
|
||||
{ offset: 305, length: 2 },
|
||||
{
|
||||
kind: "dfuse-element",
|
||||
target_index: 0,
|
||||
alternate_setting: 0,
|
||||
target_name: "Internal Flash",
|
||||
element_index: 1,
|
||||
start_address: 0x0800_4000,
|
||||
end_address: 0x0800_4001,
|
||||
length: 2,
|
||||
},
|
||||
],
|
||||
[
|
||||
"target-1/element-0-0x1fffc000-0x1fffc001.bin",
|
||||
{ offset: 589, length: 2 },
|
||||
{
|
||||
kind: "dfuse-element",
|
||||
target_index: 1,
|
||||
alternate_setting: 1,
|
||||
target_name: null,
|
||||
element_index: 0,
|
||||
start_address: 0x1fff_c000,
|
||||
end_address: 0x1fff_c001,
|
||||
length: 2,
|
||||
},
|
||||
],
|
||||
]);
|
||||
});
|
||||
|
||||
it("refuses a suffix CRC mismatch under the default integrity policy", async () => {
|
||||
const path = await writeImage(
|
||||
"bad-crc.dfu",
|
||||
dfuseFile({ ...FIRMWARE, crc: 0x1234_5678 }),
|
||||
);
|
||||
const expected = (
|
||||
await scanCanonicalArtifactInventory(
|
||||
await writeImage("good.dfu", dfuseFile(FIRMWARE)),
|
||||
)
|
||||
).occurrences.find((row) => row.logical_path === ".")?.format_metadata;
|
||||
if (expected?.kind !== "dfuse") throw new Error("Missing DfuSe metadata");
|
||||
const computed = expected.computed_crc.toString(16).padStart(8, "0");
|
||||
await expect(scanCanonicalArtifactInventory(path)).rejects.toMatchObject({
|
||||
reason: "integrity",
|
||||
details: {
|
||||
logicalPath: ".",
|
||||
unpacked: false,
|
||||
algorithm: "dfu-crc32",
|
||||
declaredCrc32: "12345678",
|
||||
calculatedCrc32: computed,
|
||||
},
|
||||
});
|
||||
const recorded = await scanCanonicalArtifactInventory(path, {
|
||||
integrity: "record-and-continue",
|
||||
});
|
||||
const root = recorded.occurrences.find((row) => row.logical_path === ".");
|
||||
expect(root).toMatchObject({
|
||||
hash_status: "mismatched",
|
||||
limitations: ["Declared integrity metadata contradicts observed bytes."],
|
||||
format_metadata: {
|
||||
suffix_crc: 0x1234_5678,
|
||||
computed_crc: expected.computed_crc,
|
||||
},
|
||||
});
|
||||
expect(recorded.integrity_contradictions).toEqual([
|
||||
expect.objectContaining({
|
||||
occurrence_id: root?.occurrence_id,
|
||||
logical_path: ".",
|
||||
algorithm: "dfu-crc32",
|
||||
declared_crc32: "12345678",
|
||||
observed_crc32: computed,
|
||||
trust: "observed-untrusted",
|
||||
}),
|
||||
]);
|
||||
expect(recorded.limitations).toContain(
|
||||
"1 integrity contradiction(s) were recorded; mismatched content is observed-untrusted.",
|
||||
);
|
||||
expect(recorded.occurrences).toHaveLength(4);
|
||||
});
|
||||
|
||||
it("extracts elements and carries a root contradiction into the extraction", async () => {
|
||||
const directory = await createTestTempDirectory("rea-dfuse-extract-");
|
||||
const path = join(directory, "firmware.dfu");
|
||||
await writeFile(path, dfuseFile({ ...FIRMWARE, crc: 0 }));
|
||||
const resourceScope = new ArtifactResourceScope();
|
||||
try {
|
||||
const extracted = await extractArtifact({
|
||||
inputPath: path,
|
||||
inputFormat: "dfuse",
|
||||
outputRoot: join(directory, "out"),
|
||||
environment: process.env,
|
||||
integrityPolicy: "record-and-continue",
|
||||
resourceScope,
|
||||
});
|
||||
expect([
|
||||
...(await readFile(
|
||||
join(
|
||||
extracted.output_root,
|
||||
"target-1/element-0-0x1fffc000-0x1fffc001.bin",
|
||||
),
|
||||
)),
|
||||
]).toEqual([0xaa, 0x55]);
|
||||
expect(extracted.artifacts).toHaveLength(3);
|
||||
expect(extracted.integrity_contradictions).toEqual([
|
||||
expect.objectContaining({ logical_path: ".", algorithm: "dfu-crc32" }),
|
||||
]);
|
||||
expect(extracted.limitations).toContain(
|
||||
"The container contradicts its declared integrity metadata; every extracted file is observed-untrusted.",
|
||||
);
|
||||
expect(extracted.limitations.join("\n")).not.toContain(
|
||||
"extracted file(s) contradict",
|
||||
);
|
||||
} finally {
|
||||
await resourceScope.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("recognizes DfuSe only with its suffix, including inside a directory", async () => {
|
||||
const directory = await createTestTempDirectory("rea-dfuse-nested-");
|
||||
await mkdir(join(directory, "updates"));
|
||||
const bytes = dfuseFile(FIRMWARE);
|
||||
await writeFile(join(directory, "updates", "image.bin"), bytes);
|
||||
await writeFile(
|
||||
join(directory, "updates", "no-suffix.dfu"),
|
||||
bytes.subarray(0, bytes.length - 16),
|
||||
);
|
||||
const snapshot = await scanCanonicalArtifactInventory(directory);
|
||||
expect(
|
||||
snapshot.occurrences
|
||||
.filter((row) => row.entry_kind === "file")
|
||||
.map((row) => [row.logical_path, row.artifact_kind, row.artifact_format]),
|
||||
).toEqual([
|
||||
["updates/image.bin", "container", "dfuse"],
|
||||
["updates/no-suffix.dfu", "resource", "file"],
|
||||
]);
|
||||
const root = await scanCanonicalArtifactInventory(
|
||||
await writeImage("no-suffix.dfu", bytes.subarray(0, bytes.length - 16)),
|
||||
);
|
||||
expect(root.manifest.root_format).toBe("file");
|
||||
});
|
||||
|
||||
it("reads the suffix of a DfuSe root longer than the classification prefix", async () => {
|
||||
const large = dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
elements: [
|
||||
{ address: 0x0800_0000, data: new Uint8Array(20_000).fill(0x5a) },
|
||||
],
|
||||
},
|
||||
],
|
||||
});
|
||||
const snapshot = await scanCanonicalArtifactInventory(
|
||||
await writeImage("large.dfu", large),
|
||||
);
|
||||
expect(snapshot.manifest.root_format).toBe("dfuse");
|
||||
expect(
|
||||
snapshot.occurrences.find((row) => row.logical_path !== ".")
|
||||
?.source_location,
|
||||
).toEqual({ offset: 293, length: 20_000 });
|
||||
// The same prefix without the suffix stays a plain file.
|
||||
const unsuffixed = await scanCanonicalArtifactInventory(
|
||||
await writeImage(
|
||||
"large-no-suffix.dfu",
|
||||
large.subarray(0, large.length - 1),
|
||||
),
|
||||
);
|
||||
expect(unsuffixed.manifest.root_format).toBe("file");
|
||||
});
|
||||
|
||||
it("compares an element's target memory, not only its bytes and path", async () => {
|
||||
const evidence = async (alternateSetting: number, name: string) => {
|
||||
const inventory = artifactInventoryResultSchema.parse(
|
||||
await scanCanonicalArtifactInventory(
|
||||
await writeImage(
|
||||
"firmware.dfu",
|
||||
dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting,
|
||||
name,
|
||||
elements: [{ address: 0x0800_0000, data: [1, 2, 3, 4] }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
),
|
||||
),
|
||||
);
|
||||
return createEvidence(
|
||||
{
|
||||
path: "firmware.dfu",
|
||||
sha256: inventory.manifest.root_sha256,
|
||||
format: "dfuse",
|
||||
},
|
||||
ARTIFACT_COMPARISON_EXAMPLE.left.provider,
|
||||
{
|
||||
operation: "inventory_artifact",
|
||||
parameters: {},
|
||||
result: inventory,
|
||||
confidence: "observed",
|
||||
authority: "shipped-artifact",
|
||||
},
|
||||
);
|
||||
};
|
||||
const left = await evidence(0, "Internal Flash");
|
||||
const path = "target-0/element-0-0x08000000-0x08000003.bin";
|
||||
for (const right of [
|
||||
await evidence(1, "Internal Flash"),
|
||||
await evidence(0, "Option Bytes"),
|
||||
])
|
||||
expect(
|
||||
compareArtifacts(left, right).changes.find(
|
||||
(change) => change.logical_path === path,
|
||||
),
|
||||
).toMatchObject({ classification: "changed", dimensions: ["metadata"] });
|
||||
expect(
|
||||
compareArtifacts(left, await evidence(0, "Internal Flash")).changes,
|
||||
).toEqual([]);
|
||||
});
|
||||
@@ -0,0 +1,55 @@
|
||||
import {
|
||||
createDfuSeDecoder,
|
||||
type DfuSeImage,
|
||||
} from "../domain/firmware/dfuse.js";
|
||||
import {
|
||||
DecodedRegionArtifactReader,
|
||||
regionFileName,
|
||||
type RegionDecoding,
|
||||
} from "./DecodedRegionArtifactReader.js";
|
||||
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
|
||||
|
||||
const crcText = (value: number): string => value.toString(16).padStart(8, "0");
|
||||
|
||||
const dfuSeRegions: RegionDecoding<DfuSeImage> = {
|
||||
label: "DfuSe",
|
||||
keyPrefix: "dfuse",
|
||||
createDecoder: createDfuSeDecoder,
|
||||
describe: (image) => ({
|
||||
metadata: image.metadata,
|
||||
// Each element stays one region at its own file range, even beside others.
|
||||
regions: image.elements.map((element) => {
|
||||
const end = element.address + element.bytes.length - 1;
|
||||
return {
|
||||
path: `target-${String(element.targetIndex)}/element-${String(element.elementIndex)}-${regionFileName(element.address, end)}`,
|
||||
byteOffset: element.dataOffset,
|
||||
formatMetadata: {
|
||||
kind: "dfuse-element",
|
||||
target_index: element.targetIndex,
|
||||
alternate_setting: element.alternateSetting,
|
||||
target_name: element.targetName,
|
||||
element_index: element.elementIndex,
|
||||
start_address: element.address,
|
||||
end_address: end,
|
||||
length: element.bytes.length,
|
||||
},
|
||||
bytes: element.bytes,
|
||||
};
|
||||
}),
|
||||
...(image.limitations.length === 0
|
||||
? {}
|
||||
: { coverage: { status: "complete", limitations: image.limitations } }),
|
||||
containerIntegrity: {
|
||||
algorithm: "dfu-crc32",
|
||||
declared_crc32: crcText(image.metadata.suffix_crc),
|
||||
observed_crc32: crcText(image.metadata.computed_crc),
|
||||
},
|
||||
}),
|
||||
};
|
||||
|
||||
/** Decode DfuSe targets into one entry per element, verifying the suffix CRC. */
|
||||
export class DfuSeArtifactReader extends DecodedRegionArtifactReader<DfuSeImage> {
|
||||
constructor(path: string, admitted?: StableRegularFileDescriptor) {
|
||||
super(path, "dfuse", dfuSeRegions, admitted);
|
||||
}
|
||||
}
|
||||
@@ -1,267 +1,56 @@
|
||||
import { Readable } from "node:stream";
|
||||
import { setImmediate } from "node:timers/promises";
|
||||
|
||||
import type { HexRecordsMetadata } from "../domain/artifactFormatMetadata.js";
|
||||
import {
|
||||
HEX_RECORD_ENCODING_LABELS,
|
||||
createHexRecordDecoder,
|
||||
type HexRecordFailure,
|
||||
type HexRecordImage,
|
||||
} from "../domain/firmware/hexRecords.js";
|
||||
import type { HexRecordEncoding } from "../domain/firmware/hexRecordSyntax.js";
|
||||
import {
|
||||
ArtifactReaderFailure,
|
||||
copyArtifactEntry,
|
||||
sameArtifactEntry,
|
||||
type ArtifactEntry,
|
||||
type ArtifactReader,
|
||||
type ArtifactReaderTraversal,
|
||||
type ArtifactTraversalOptions,
|
||||
} from "./ArtifactReader.js";
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
|
||||
import { ArtifactFileSource } from "./ArtifactFileSource.js";
|
||||
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
|
||||
DecodedRegionArtifactReader,
|
||||
regionFileName,
|
||||
type RegionDecoding,
|
||||
} from "./DecodedRegionArtifactReader.js";
|
||||
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
|
||||
|
||||
const addressText = (address: number): string =>
|
||||
`0x${address.toString(16).padStart(8, "0")}`;
|
||||
const hexRecordRegions = (
|
||||
format: HexRecordEncoding,
|
||||
): RegionDecoding<HexRecordImage> => ({
|
||||
label: HEX_RECORD_ENCODING_LABELS[format],
|
||||
keyPrefix: "hex-records",
|
||||
createDecoder: (fileBytes) => createHexRecordDecoder(format, fileBytes),
|
||||
describe: (image) => ({
|
||||
metadata: image.metadata,
|
||||
regions: image.regions.map((region) => ({
|
||||
path: regionFileName(region.start, region.end),
|
||||
byteOffset: null,
|
||||
formatMetadata: {
|
||||
kind: "hex-record-region",
|
||||
start_address: region.start,
|
||||
end_address: region.end,
|
||||
length: region.bytes.length,
|
||||
record_count: region.recordCount,
|
||||
first_line: region.firstLine,
|
||||
last_line: region.lastLine,
|
||||
},
|
||||
bytes: region.bytes,
|
||||
})),
|
||||
...(image.terminated && image.limitations.length === 0
|
||||
? {}
|
||||
: {
|
||||
coverage: {
|
||||
status: image.terminated ? "complete" : "unknown",
|
||||
limitations: image.limitations,
|
||||
},
|
||||
}),
|
||||
}),
|
||||
});
|
||||
|
||||
/** Decode Intel HEX or S-record text into address-contiguous region entries. */
|
||||
export class HexRecordArtifactReader implements ArtifactReader {
|
||||
readonly #scope = new ArtifactResourceScope();
|
||||
readonly #entries = new Map<string, ArtifactEntry>();
|
||||
readonly #regions = new Map<string, Uint8Array>();
|
||||
readonly #streams = new Set<Readable>();
|
||||
readonly #source: ArtifactFileSource;
|
||||
readonly #label: string;
|
||||
#prepared: Promise<void> | undefined;
|
||||
#metadata: HexRecordsMetadata | undefined;
|
||||
#coverage: ArtifactReaderTraversal | undefined;
|
||||
#closing = false;
|
||||
|
||||
export class HexRecordArtifactReader extends DecodedRegionArtifactReader<HexRecordImage> {
|
||||
constructor(
|
||||
path: string,
|
||||
readonly format: HexRecordEncoding,
|
||||
format: HexRecordEncoding,
|
||||
admitted?: StableRegularFileDescriptor,
|
||||
) {
|
||||
this.#label = HEX_RECORD_ENCODING_LABELS[format];
|
||||
this.#source = new ArtifactFileSource(path, this.#label, admitted);
|
||||
}
|
||||
|
||||
formatMetadata(): HexRecordsMetadata | undefined {
|
||||
return this.#metadata === undefined ? undefined : { ...this.#metadata };
|
||||
}
|
||||
|
||||
async prepareContainer(
|
||||
expectedSha256: string,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
await this.#scope.run(async () => {
|
||||
this.#assertOpen(signal);
|
||||
await this.#source.bindSha256(expectedSha256, signal);
|
||||
});
|
||||
}
|
||||
|
||||
async *entries(
|
||||
signal?: AbortSignal,
|
||||
options?: ArtifactTraversalOptions,
|
||||
): AsyncIterable<ArtifactEntry> {
|
||||
await this.#scope.run(async () => {
|
||||
this.#assertOpen(signal);
|
||||
this.#prepared ??= this.#prepare(signal);
|
||||
await this.#prepared;
|
||||
await this.#source.read(signal);
|
||||
});
|
||||
if (this.#coverage !== undefined)
|
||||
options?.onCoverage?.({
|
||||
...this.#coverage,
|
||||
limitations: [...this.#coverage.limitations],
|
||||
});
|
||||
let index = 0;
|
||||
for (const entry of this.#entries.values()) {
|
||||
if (index % 256 === 0) await setImmediate();
|
||||
index += 1;
|
||||
this.#assertOpen(signal);
|
||||
yield copyArtifactEntry(entry);
|
||||
}
|
||||
}
|
||||
|
||||
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
|
||||
return this.#scope.run(async () => {
|
||||
this.#assertOpen(signal);
|
||||
const stored = this.#entries.get(entry.adapterKey);
|
||||
const bytes = this.#regions.get(entry.adapterKey);
|
||||
if (
|
||||
stored === undefined ||
|
||||
bytes === undefined ||
|
||||
!sameArtifactEntry(entry, stored)
|
||||
)
|
||||
throw new ArtifactReaderFailure(
|
||||
"integrity",
|
||||
`${this.#label} entry does not match its owning inventory`,
|
||||
);
|
||||
// Decoded bytes are retained, so only the carrier identity is rechecked.
|
||||
await this.#source.read(signal);
|
||||
this.#assertOpen(signal);
|
||||
const output = Readable.from(this.#regionChunks(bytes, signal), {
|
||||
objectMode: false,
|
||||
});
|
||||
const onAbort = (): void => {
|
||||
output.destroy(
|
||||
new ArtifactReaderFailure(
|
||||
"cancelled",
|
||||
`${this.#label} region read cancelled`,
|
||||
{ cause: signal?.reason },
|
||||
),
|
||||
);
|
||||
};
|
||||
signal?.addEventListener("abort", onAbort, { once: true });
|
||||
output.once("close", () => signal?.removeEventListener("abort", onAbort));
|
||||
output.on("error", () => undefined);
|
||||
this.#streams.add(output);
|
||||
output.once("close", () => this.#streams.delete(output));
|
||||
// Draining the scope must wait for the readable side, including an
|
||||
// abandoned consumer and an in-flight asynchronous chunk producer.
|
||||
void this.#scope
|
||||
.run(
|
||||
() =>
|
||||
new Promise<void>((resolve) => {
|
||||
output.once("close", () => resolve());
|
||||
}),
|
||||
)
|
||||
.catch((cause: unknown) =>
|
||||
output.destroy(
|
||||
cause instanceof Error
|
||||
? cause
|
||||
: new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`${this.#label} stream ownership failed`,
|
||||
{ cause },
|
||||
),
|
||||
),
|
||||
);
|
||||
return output;
|
||||
});
|
||||
}
|
||||
|
||||
async *#regionChunks(
|
||||
bytes: Uint8Array,
|
||||
signal?: AbortSignal,
|
||||
): AsyncIterable<Buffer> {
|
||||
// Keep cached bytes isolated from consumers without copying a whole
|
||||
// region for every open. Match descriptor reads' 64 KiB chunk size.
|
||||
for (let offset = 0; offset < bytes.length; offset += 64 * 1024) {
|
||||
await setImmediate();
|
||||
this.#assertOpen(signal);
|
||||
yield Buffer.from(bytes.subarray(offset, offset + 64 * 1024));
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.#closing = true;
|
||||
for (const stream of this.#streams)
|
||||
stream.destroy(
|
||||
new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`${this.#label} reader closed`,
|
||||
),
|
||||
);
|
||||
await this.#scope.close();
|
||||
await this.#source.close();
|
||||
this.#entries.clear();
|
||||
this.#regions.clear();
|
||||
}
|
||||
|
||||
provenance(): readonly [] {
|
||||
return [];
|
||||
}
|
||||
|
||||
#assertOpen(signal?: AbortSignal): void {
|
||||
assertArtifactReadActive(signal);
|
||||
if (this.#closing)
|
||||
throw new ArtifactReaderFailure(
|
||||
"unavailable",
|
||||
`${this.#label} reader is closed`,
|
||||
);
|
||||
}
|
||||
|
||||
async #prepare(signal?: AbortSignal): Promise<void> {
|
||||
const image = await this.#decode(signal);
|
||||
let index = 0;
|
||||
for (const region of image.regions) {
|
||||
if (index % 256 === 0) await setImmediate();
|
||||
this.#assertOpen(signal);
|
||||
const entry = {
|
||||
path: `${addressText(region.start)}-${addressText(region.end)}.bin`,
|
||||
kind: "file" as const,
|
||||
declaredSize: region.bytes.length,
|
||||
compressedSize: null,
|
||||
executable: false,
|
||||
encrypted: false,
|
||||
byteOffset: null,
|
||||
declaredIntegrity: null,
|
||||
unpacked: false,
|
||||
limitations: [],
|
||||
adapterKey: `hex-records:${String(index)}`,
|
||||
formatMetadata: {
|
||||
kind: "hex-record-region" as const,
|
||||
start_address: region.start,
|
||||
end_address: region.end,
|
||||
length: region.bytes.length,
|
||||
record_count: region.recordCount,
|
||||
first_line: region.firstLine,
|
||||
last_line: region.lastLine,
|
||||
},
|
||||
} satisfies ArtifactEntry;
|
||||
this.#entries.set(entry.adapterKey, entry);
|
||||
this.#regions.set(entry.adapterKey, region.bytes);
|
||||
index += 1;
|
||||
}
|
||||
this.#metadata = image.metadata;
|
||||
if (!image.terminated || image.limitations.length > 0)
|
||||
this.#coverage = {
|
||||
status: image.terminated ? "complete" : "unknown",
|
||||
limitations: image.limitations,
|
||||
};
|
||||
await this.#source.read(signal);
|
||||
}
|
||||
|
||||
async #decode(signal?: AbortSignal): Promise<HexRecordImage> {
|
||||
const { handle, initial } = await this.#source.read(signal);
|
||||
const decoder = createHexRecordDecoder(this.format, initial.size);
|
||||
if (!decoder.ok) throw this.#failure(decoder.error);
|
||||
// Reading only the admitted extent keeps decoded bytes within the arena.
|
||||
if (initial.size > 0)
|
||||
for await (const chunk of readFileHandleChunks(handle, {
|
||||
end: initial.size - 1,
|
||||
signal,
|
||||
cancellationFailure: (reason) =>
|
||||
new ArtifactReaderFailure(
|
||||
"cancelled",
|
||||
`${this.#label} read cancelled`,
|
||||
{ cause: reason },
|
||||
),
|
||||
}) as AsyncIterable<Buffer>) {
|
||||
this.#assertOpen(signal);
|
||||
const written = decoder.value.write(chunk);
|
||||
if (!written.ok) throw this.#failure(written.error);
|
||||
}
|
||||
const image = decoder.value.finish();
|
||||
if (!image.ok) throw this.#failure(image.error);
|
||||
return image.value;
|
||||
}
|
||||
|
||||
#failure(failure: HexRecordFailure): ArtifactReaderFailure {
|
||||
return failure.kind === "format"
|
||||
? new ArtifactReaderFailure(
|
||||
"format",
|
||||
`${this.#label} is malformed: ${failure.message}`,
|
||||
)
|
||||
: new ArtifactReaderFailure(
|
||||
"limit",
|
||||
`${this.#label} exceeds a fixed limit: ${failure.message}`,
|
||||
failure.boundary === undefined ? {} : { boundary: failure.boundary },
|
||||
);
|
||||
super(path, format, hexRecordRegions(format), admitted);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,231 @@
|
||||
import { readFile, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import { Uf2ArtifactReader } from "./Uf2ArtifactReader.js";
|
||||
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
|
||||
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
|
||||
import {
|
||||
RP2040_FAMILY,
|
||||
RP2XXX_ABSOLUTE_FAMILY,
|
||||
patternBytes,
|
||||
uf2Block,
|
||||
uf2File,
|
||||
uf2Image,
|
||||
} from "../domain/firmware/uf2.fixture.js";
|
||||
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
|
||||
import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js";
|
||||
|
||||
const INVENTORY_LIMITATION =
|
||||
"UF2 inventory decodes main-flash block payloads into address-contiguous regions per familyID. It does not establish CPU, byte order, memory map, or an entry point; a familyID names a target family, and gaps are unwritten, not zero.";
|
||||
|
||||
const writeImage = async (name: string, bytes: Uint8Array): Promise<string> => {
|
||||
const directory = await createTestTempDirectory("rea-uf2-");
|
||||
const path = join(directory, name);
|
||||
await writeFile(path, bytes);
|
||||
return path;
|
||||
};
|
||||
|
||||
const APP = patternBytes(600, 5);
|
||||
const ABSOLUTE = uf2Block({
|
||||
address: 0x10ff_ff00,
|
||||
payload: new Uint8Array(256).fill(0xef),
|
||||
blockNo: 0,
|
||||
numBlocks: 2,
|
||||
familyId: RP2XXX_ABSOLUTE_FAMILY,
|
||||
});
|
||||
const UNNAMED = uf2Block({
|
||||
address: 0x2000_0000,
|
||||
payload: [1, 2, 3],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
familyId: 0x1234_5678,
|
||||
});
|
||||
|
||||
it("inventories each family's regions with family and block facts", async () => {
|
||||
const snapshot = await scanCanonicalArtifactInventory(
|
||||
await writeImage(
|
||||
"firmware.bin",
|
||||
uf2File([...uf2Image(RP2040_FAMILY, 0x1000_0000, APP), UNNAMED]),
|
||||
),
|
||||
);
|
||||
expect(snapshot.manifest.root_format).toBe("uf2");
|
||||
expect(snapshot.coverage).toEqual({
|
||||
scope: "container-entries",
|
||||
status: "complete",
|
||||
});
|
||||
expect(snapshot.limitations).toContain(INVENTORY_LIMITATION);
|
||||
expect(
|
||||
snapshot.occurrences.find((row) => row.logical_path === "."),
|
||||
).toMatchObject({
|
||||
artifact_kind: "container",
|
||||
artifact_format: "uf2",
|
||||
format_metadata: {
|
||||
kind: "uf2",
|
||||
block_count: 4,
|
||||
family_count: 2,
|
||||
region_count: 2,
|
||||
data_bytes: 603,
|
||||
},
|
||||
});
|
||||
expect(
|
||||
snapshot.occurrences
|
||||
.filter((row) => row.logical_path !== ".")
|
||||
.map((row) => [
|
||||
row.logical_path,
|
||||
row.declared_size,
|
||||
row.source_location,
|
||||
row.format_metadata,
|
||||
]),
|
||||
).toEqual([
|
||||
[
|
||||
"family-0x12345678/0x20000000-0x20000002.bin",
|
||||
3,
|
||||
null,
|
||||
{
|
||||
kind: "uf2-region",
|
||||
family_id: 0x1234_5678,
|
||||
family_name: null,
|
||||
start_address: 0x2000_0000,
|
||||
end_address: 0x2000_0002,
|
||||
length: 3,
|
||||
block_count: 1,
|
||||
first_file_block: 3,
|
||||
last_file_block: 3,
|
||||
},
|
||||
],
|
||||
[
|
||||
"family-0xe48bff56/0x10000000-0x10000257.bin",
|
||||
600,
|
||||
null,
|
||||
{
|
||||
kind: "uf2-region",
|
||||
family_id: RP2040_FAMILY,
|
||||
family_name: "RP2040",
|
||||
start_address: 0x1000_0000,
|
||||
end_address: 0x1000_0257,
|
||||
length: 600,
|
||||
block_count: 3,
|
||||
first_file_block: 0,
|
||||
last_file_block: 2,
|
||||
},
|
||||
],
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves coverage unknown when numbered blocks are absent", async () => {
|
||||
const snapshot = await scanCanonicalArtifactInventory(
|
||||
await writeImage(
|
||||
"universal.uf2",
|
||||
uf2File([ABSOLUTE, ...uf2Image(RP2040_FAMILY, 0x1000_0000, APP)]),
|
||||
),
|
||||
);
|
||||
expect(snapshot.coverage.status).toBe("unknown");
|
||||
expect(snapshot.limitations).toContain(
|
||||
"UF2 numbering declares 1 block(s) that are absent: family 0xe48bff57 (RP2XXX_ABSOLUTE) with 2 numbered blocks has 1 of them.",
|
||||
);
|
||||
expect(
|
||||
snapshot.occurrences
|
||||
.filter((row) => row.logical_path !== ".")
|
||||
.map((row) => row.logical_path),
|
||||
).toEqual([
|
||||
"family-0xe48bff56/0x10000000-0x10000257.bin",
|
||||
"family-0xe48bff57/0x10ffff00-0x10ffffff.bin",
|
||||
]);
|
||||
});
|
||||
|
||||
it("fails closed on a conflicting rewrite and leaves short files unrecognized", async () => {
|
||||
const rewrite = uf2Block({
|
||||
address: 0x1000_0010,
|
||||
payload: [0xff],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
familyId: RP2040_FAMILY,
|
||||
});
|
||||
await expect(
|
||||
scanCanonicalArtifactInventory(
|
||||
await writeImage(
|
||||
"conflict.uf2",
|
||||
uf2File([...uf2Image(RP2040_FAMILY, 0x1000_0000, APP), rewrite]),
|
||||
),
|
||||
),
|
||||
).rejects.toMatchObject({
|
||||
reason: "format",
|
||||
message:
|
||||
"UF2 is malformed: family 0xe48bff56 (RP2040) address 0x10000010 is written with different values by blocks 0-2 (0x75) and block 3 (0xff)",
|
||||
});
|
||||
const truncated = await scanCanonicalArtifactInventory(
|
||||
await writeImage("short.uf2", ABSOLUTE.subarray(0, 511)),
|
||||
);
|
||||
expect(truncated.manifest.root_format).toBe("file");
|
||||
});
|
||||
|
||||
it("extracts byte-exact regions into family directories", async () => {
|
||||
const directory = await createTestTempDirectory("rea-uf2-extract-");
|
||||
const path = join(directory, "firmware.uf2");
|
||||
await writeFile(
|
||||
path,
|
||||
uf2File([ABSOLUTE, ...uf2Image(RP2040_FAMILY, 0x1000_0000, APP)]),
|
||||
);
|
||||
const resourceScope = new ArtifactResourceScope();
|
||||
try {
|
||||
const extracted = await extractArtifact({
|
||||
inputPath: path,
|
||||
inputFormat: "uf2",
|
||||
outputRoot: join(directory, "out"),
|
||||
environment: process.env,
|
||||
integrityPolicy: "fail",
|
||||
resourceScope,
|
||||
});
|
||||
expect(extracted.containment_verified).toBe(true);
|
||||
expect(extracted.coverage.status).toBe("unknown");
|
||||
expect(
|
||||
Buffer.compare(
|
||||
await readFile(
|
||||
join(
|
||||
extracted.output_root,
|
||||
"family-0xe48bff56/0x10000000-0x10000257.bin",
|
||||
),
|
||||
),
|
||||
APP,
|
||||
),
|
||||
).toBe(0);
|
||||
expect(
|
||||
Buffer.compare(
|
||||
await readFile(
|
||||
join(
|
||||
extracted.output_root,
|
||||
"family-0xe48bff57/0x10ffff00-0x10ffffff.bin",
|
||||
),
|
||||
),
|
||||
Buffer.alloc(256, 0xef),
|
||||
),
|
||||
).toBe(0);
|
||||
} finally {
|
||||
await resourceScope.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects a forged region entry and a changed carrier", async () => {
|
||||
const path = await writeImage(
|
||||
"identity.uf2",
|
||||
uf2File(uf2Image(RP2040_FAMILY, 0x1000_0000, APP)),
|
||||
);
|
||||
const reader = new Uf2ArtifactReader(path);
|
||||
try {
|
||||
const entries = [];
|
||||
for await (const entry of reader.entries()) entries.push(entry);
|
||||
const entry = entries[0];
|
||||
if (entry === undefined) throw new Error("Missing region");
|
||||
await expect(
|
||||
reader.open({ ...entry, adapterKey: "uf2:1" }),
|
||||
).rejects.toMatchObject({ reason: "integrity" });
|
||||
await writeFile(path, uf2File(uf2Image(RP2040_FAMILY, 0, APP)));
|
||||
await expect(reader.open(entry)).rejects.toMatchObject({
|
||||
reason: "integrity",
|
||||
});
|
||||
} finally {
|
||||
await reader.close();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
import { createUf2Decoder, type Uf2Image } from "../domain/firmware/uf2.js";
|
||||
import {
|
||||
DecodedRegionArtifactReader,
|
||||
regionFileName,
|
||||
type RegionDecoding,
|
||||
} from "./DecodedRegionArtifactReader.js";
|
||||
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
|
||||
|
||||
/** Families are separate address spaces, so each names its own directory. */
|
||||
const familyDirectory = (familyId: number | null): string =>
|
||||
familyId === null
|
||||
? "no-family"
|
||||
: `family-0x${familyId.toString(16).padStart(8, "0")}`;
|
||||
|
||||
const uf2Regions: RegionDecoding<Uf2Image> = {
|
||||
label: "UF2",
|
||||
keyPrefix: "uf2",
|
||||
createDecoder: createUf2Decoder,
|
||||
describe: (image) => ({
|
||||
metadata: image.metadata,
|
||||
regions: image.regions.map((region) => ({
|
||||
path: `${familyDirectory(region.familyId)}/${regionFileName(region.start, region.end)}`,
|
||||
byteOffset: null,
|
||||
formatMetadata: {
|
||||
kind: "uf2-region",
|
||||
family_id: region.familyId,
|
||||
family_name: region.familyName,
|
||||
start_address: region.start,
|
||||
end_address: region.end,
|
||||
length: region.bytes.length,
|
||||
block_count: region.blockCount,
|
||||
first_file_block: region.firstFileBlock,
|
||||
last_file_block: region.lastFileBlock,
|
||||
},
|
||||
bytes: region.bytes,
|
||||
})),
|
||||
...(image.complete && image.limitations.length === 0
|
||||
? {}
|
||||
: {
|
||||
coverage: {
|
||||
status: image.complete ? "complete" : "unknown",
|
||||
limitations: image.limitations,
|
||||
},
|
||||
}),
|
||||
}),
|
||||
};
|
||||
|
||||
/** Decode UF2 blocks into each family's address-contiguous region entries. */
|
||||
export class Uf2ArtifactReader extends DecodedRegionArtifactReader<Uf2Image> {
|
||||
constructor(path: string, admitted?: StableRegularFileDescriptor) {
|
||||
super(path, "uf2", uf2Regions, admitted);
|
||||
}
|
||||
}
|
||||
@@ -38,6 +38,8 @@ type FormatsWithReaders =
|
||||
| "intel-hex"
|
||||
| "srec"
|
||||
| "esp-image"
|
||||
| "uf2"
|
||||
| "dfuse"
|
||||
| "esp-flash-image"
|
||||
| "fit";
|
||||
|
||||
|
||||
@@ -17,6 +17,7 @@ import type { ArtifactReaderAdmission } from "./ArtifactReaderAdmission.js";
|
||||
import { AsarArtifactReader } from "./AsarArtifactReader.js";
|
||||
import type { BunStandaloneAdmission } from "./bun/BunStandaloneAdmission.js";
|
||||
import { BunStandaloneArtifactReader } from "./BunStandaloneArtifactReader.js";
|
||||
import { DfuSeArtifactReader } from "./DfuSeArtifactReader.js";
|
||||
import { DirectoryArtifactReader } from "./DirectoryArtifactReader.js";
|
||||
import type { DotnetBundleAdmission } from "./dotnet/DotnetBundleAdmission.js";
|
||||
import { DotnetBundleArtifactReader } from "./DotnetBundleArtifactReader.js";
|
||||
@@ -36,6 +37,7 @@ import type { NodeSeaAdmission } from "./nodeSea/NodeSeaAdmission.js";
|
||||
import { NodeSeaArtifactReader } from "./NodeSeaArtifactReader.js";
|
||||
import type { PyInstallerAdmission } from "./pyinstaller/PyInstallerAdmission.js";
|
||||
import { PyInstallerArtifactReader } from "./PyInstallerArtifactReader.js";
|
||||
import { Uf2ArtifactReader } from "./Uf2ArtifactReader.js";
|
||||
import { UImageArtifactReader } from "./UImageArtifactReader.js";
|
||||
import { UnityFsArtifactReader } from "./UnityFsArtifactReader.js";
|
||||
import { ZipArtifactReader } from "./ZipArtifactReader.js";
|
||||
@@ -369,6 +371,26 @@ const NON_ZIP_FORMATS = {
|
||||
createReader: (path, options) =>
|
||||
new EspImageArtifactReader(path, sourceOf(options)),
|
||||
}),
|
||||
uf2: container("uf2", {
|
||||
roleContainer: true,
|
||||
retainClassifiedSource: true,
|
||||
extractable: true,
|
||||
absentInventory: "unavailable",
|
||||
readerLimitation:
|
||||
"UF2 inventory decodes main-flash block payloads into address-contiguous regions per familyID. It does not establish CPU, byte order, memory map, or an entry point; a familyID names a target family, and gaps are unwritten, not zero.",
|
||||
createReader: (path, options) =>
|
||||
new Uf2ArtifactReader(path, sourceOf(options)),
|
||||
}),
|
||||
dfuse: container("dfuse", {
|
||||
roleContainer: true,
|
||||
retainClassifiedSource: true,
|
||||
extractable: true,
|
||||
absentInventory: "unavailable",
|
||||
readerLimitation:
|
||||
"DfuSe inventory lists each target element as one region at its declared address. It does not establish CPU, byte order, memory map, or an entry point; an alternate setting selects a device memory, not an address space REA can interpret.",
|
||||
createReader: (path, options) =>
|
||||
new DfuSeArtifactReader(path, sourceOf(options)),
|
||||
}),
|
||||
"esp-flash-image": container("esp-flash-image", {
|
||||
roleContainer: true,
|
||||
retainClassifiedSource: true,
|
||||
|
||||
@@ -534,9 +534,18 @@ const createExtractionResult = (
|
||||
const selectedIds = new Set(
|
||||
selected.map(({ occurrence }) => occurrence.occurrence_id),
|
||||
);
|
||||
const integrityContradictions = inventory.integrity_contradictions.filter(
|
||||
({ occurrence_id: id }) => selectedIds.has(id),
|
||||
// A root contradiction covers the container bytes every extracted file came from.
|
||||
const rootId = inventory.occurrences.find(
|
||||
(row) => row.logical_path === ".",
|
||||
)?.occurrence_id;
|
||||
const rootContradicted = inventory.integrity_contradictions.some(
|
||||
({ occurrence_id: id }) => id === rootId,
|
||||
);
|
||||
const integrityContradictions = inventory.integrity_contradictions.filter(
|
||||
({ occurrence_id: id }) => selectedIds.has(id) || id === rootId,
|
||||
);
|
||||
const extractedContradictions =
|
||||
integrityContradictions.length - (rootContradicted ? 1 : 0);
|
||||
const extractionSemantic = {
|
||||
source_manifest_id: inventory.manifest.manifest_id,
|
||||
selected_occurrence_ids: selected
|
||||
@@ -572,11 +581,16 @@ const createExtractionResult = (
|
||||
? "All regular files exposed by the active artifact reader were materialized."
|
||||
: "All inventoried regular files exposed by the active reader were materialized; source traversal coverage is unknown, so the output is not established as a complete source copy.",
|
||||
"Publication verifies the selected inventoried regular files; it does not establish complete content decoding.",
|
||||
...(integrityContradictions.length === 0
|
||||
...(extractedContradictions === 0
|
||||
? []
|
||||
: [
|
||||
`${String(integrityContradictions.length)} extracted file(s) contradict declared integrity metadata; their bytes are observed-untrusted.`,
|
||||
`${String(extractedContradictions)} extracted file(s) contradict declared integrity metadata; their bytes are observed-untrusted.`,
|
||||
]),
|
||||
...(rootContradicted
|
||||
? [
|
||||
"The container contradicts its declared integrity metadata; every extracted file is observed-untrusted.",
|
||||
]
|
||||
: []),
|
||||
],
|
||||
});
|
||||
};
|
||||
|
||||
@@ -17,6 +17,13 @@ import { createArtifactClassificationProbe } from "../ArtifactClassificationProb
|
||||
import { artifactIdForContent } from "../../domain/artifactIdentity.js";
|
||||
import { pe } from "../../domain/binaryTarget.fixture.js";
|
||||
import type { ArtifactEntry } from "../ArtifactReader.js";
|
||||
import { dfuseFile } from "../../domain/firmware/dfuse.fixture.js";
|
||||
import {
|
||||
RP2040_FAMILY,
|
||||
patternBytes,
|
||||
uf2File,
|
||||
uf2Image,
|
||||
} from "../../domain/firmware/uf2.fixture.js";
|
||||
|
||||
import { classifyArtifactBytesAtPath } from "../../../tests/fixtures/artifactClassification.js";
|
||||
import {
|
||||
@@ -493,6 +500,41 @@ it("classifies nested ESP images as containers by bytes", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("classifies nested UF2 by block magics and DfuSe by prefix and suffix", () => {
|
||||
const uf2 = uf2File(uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(300)));
|
||||
expect(classifyArtifactBytesAtPath("firmware/update.bin", uf2)).toEqual({
|
||||
kind: "container",
|
||||
format: "uf2",
|
||||
});
|
||||
expect(
|
||||
classifyArtifactBytesAtPath("firmware/short.uf2", uf2.subarray(0, 511)),
|
||||
).toEqual({ kind: "resource", format: "file" });
|
||||
const dfu = dfuseFile({
|
||||
targets: [{ alternateSetting: 0, elements: [{ address: 0, data: [1] }] }],
|
||||
});
|
||||
// The prefix alone is not DfuSe; the suffix is observed at the stream end.
|
||||
expect(classifyArtifactBytesAtPath("firmware/update.dfu", dfu)).toEqual({
|
||||
kind: "resource",
|
||||
format: "file",
|
||||
});
|
||||
const probe = createArtifactClassificationProbe();
|
||||
for (let offset = 0; offset < dfu.length; offset += 5)
|
||||
probe.write(dfu.subarray(offset, offset + 5));
|
||||
expect(
|
||||
classifyArtifactContent(
|
||||
"firmware/update.bin",
|
||||
probe.finish(dfu.length),
|
||||
dfu.length,
|
||||
),
|
||||
).toMatchObject({ kind: "container", format: "dfuse" });
|
||||
// Bytes the probe never saw may follow, so it reports no suffix at all.
|
||||
const partial = probe.finish(dfu.length + 1);
|
||||
expect(partial.suffix).toBeUndefined();
|
||||
expect(
|
||||
classifyArtifactContent("firmware/update.bin", partial, dfu.length + 1),
|
||||
).toMatchObject({ format: "file" });
|
||||
});
|
||||
|
||||
it("classifies a FIT by a /images node within the first 4 KiB", () => {
|
||||
const images = [{ name: "kernel", data: Buffer.from("kernel") }];
|
||||
const late = devicetreeFixture({
|
||||
|
||||
@@ -4,6 +4,8 @@ import { hasUImageMagic } from "../../domain/uimageFormat.js";
|
||||
import { hasUnityFsHeader } from "../../domain/unityfsFormat.js";
|
||||
import { detectHexRecordEncoding } from "../../domain/firmware/hexRecords.js";
|
||||
import { recognizeEspImage } from "../../domain/firmware/espImage.js";
|
||||
import { hasUf2Block } from "../../domain/firmware/uf2.js";
|
||||
import { isDfuSeFile } from "../../domain/firmware/dfuse.js";
|
||||
import { hasFitImagesNode } from "../../domain/firmware/fitImage.js";
|
||||
import type { ArtifactClassificationEvidence } from "../ArtifactClassificationProbe.js";
|
||||
import { asarHeaderAdmissionFailure } from "../AsarHeader.js";
|
||||
@@ -591,7 +593,7 @@ export const classifyArtifactBytes = (
|
||||
evidence: ArtifactClassificationEvidence,
|
||||
fileSize = evidence.prefix.length,
|
||||
): ArtifactNode["format"] => {
|
||||
const { prefix, mzWindowsHeaderSignature } = evidence;
|
||||
const { prefix, suffix, mzWindowsHeaderSignature } = evidence;
|
||||
// A header-only region has the signature but not the declared cartridge
|
||||
// extent. Keep it as a file instead of inventing another container.
|
||||
if (
|
||||
@@ -605,6 +607,9 @@ export const classifyArtifactBytes = (
|
||||
if (hasUImageMagic(prefix)) return "uimage";
|
||||
const hexRecords = detectHexRecordEncoding(prefix);
|
||||
if (hexRecords !== undefined) return hexRecords;
|
||||
if (hasUf2Block(prefix)) return "uf2";
|
||||
if (suffix !== undefined && isDfuSeFile(prefix, suffix, fileSize))
|
||||
return "dfuse";
|
||||
if (recognizeEspImage(prefix, fileSize) === "esp-image") return "esp-image";
|
||||
if (hasFitImagesNode(prefix)) return "fit";
|
||||
if (hasBinaryPlistSignature(prefix)) return "plist";
|
||||
|
||||
@@ -52,6 +52,10 @@ import {
|
||||
import type { ArtifactInventoryOptions } from "./types.js";
|
||||
import { readMachoHeaderPrefixFacts } from "../../domain/apple/machoContainer.js";
|
||||
import { hasDiskImageSignature } from "../../domain/apple/diskImageFormat.js";
|
||||
import {
|
||||
DFUSE_SUFFIX_BYTES,
|
||||
hasDfuSePrefix,
|
||||
} from "../../domain/firmware/dfuse.js";
|
||||
import {
|
||||
observeMzWindowsHeader,
|
||||
type MzWindowsHeaderSignatureEvidence,
|
||||
@@ -317,10 +321,21 @@ const classifyRootFormat = async (
|
||||
archiveBytes,
|
||||
signal,
|
||||
);
|
||||
const classificationEvidence =
|
||||
mzWindowsHeaderSignature === undefined
|
||||
? fatEvidence
|
||||
: { ...fatEvidence, mzWindowsHeaderSignature };
|
||||
// The probe stops at its bounded prefix, so a DfuSe trailer is read from
|
||||
// the file's own end unless the probe already observed every byte.
|
||||
const suffix =
|
||||
fatEvidence.suffix === undefined &&
|
||||
archiveBytes >= DFUSE_SUFFIX_BYTES &&
|
||||
hasDfuSePrefix(evidence.prefix)
|
||||
? await read(archiveBytes - DFUSE_SUFFIX_BYTES, DFUSE_SUFFIX_BYTES)
|
||||
: fatEvidence.suffix;
|
||||
const classificationEvidence = {
|
||||
...fatEvidence,
|
||||
...(suffix === undefined ? {} : { suffix }),
|
||||
...(mzWindowsHeaderSignature === undefined
|
||||
? {}
|
||||
: { mzWindowsHeaderSignature }),
|
||||
};
|
||||
const format = classifyArtifactContent(
|
||||
classificationPath,
|
||||
classificationEvidence,
|
||||
|
||||
@@ -53,6 +53,7 @@ import {
|
||||
type PendingIntegrityContradiction,
|
||||
} from "./scanReader.js";
|
||||
import type { ArtifactInventoryOptions } from "./types.js";
|
||||
import type { ArtifactContainerIntegrityPair } from "../../domain/artifactIntegrity.js";
|
||||
|
||||
export interface TransferredArtifactInventorySource {
|
||||
readonly owner: ArtifactResourceOwner;
|
||||
@@ -147,6 +148,7 @@ const scanCanonical = async (
|
||||
inesRegionScan,
|
||||
unknownParentOccurrences,
|
||||
traversal,
|
||||
rootContradiction,
|
||||
} = await scanReader(
|
||||
reader,
|
||||
ownedReaders,
|
||||
@@ -172,6 +174,7 @@ const scanCanonical = async (
|
||||
inesRegionScan,
|
||||
unknownParentOccurrences,
|
||||
traversal,
|
||||
rootContradiction,
|
||||
}),
|
||||
};
|
||||
} catch (cause: unknown) {
|
||||
@@ -203,6 +206,7 @@ const scanCanonical = async (
|
||||
caseCollisionLimitation: undefined,
|
||||
inesRegionScan: undefined,
|
||||
unknownParentOccurrences: new Set(),
|
||||
rootContradiction: undefined,
|
||||
traversal: {
|
||||
status: "unknown",
|
||||
limitations: [
|
||||
@@ -405,6 +409,7 @@ interface SnapshotBuildInput {
|
||||
readonly inesRegionScan: InesRegionScan | undefined;
|
||||
readonly unknownParentOccurrences: ReadonlySet<MutableOccurrence>;
|
||||
readonly traversal: ArtifactReaderTraversal;
|
||||
readonly rootContradiction: ArtifactContainerIntegrityPair | undefined;
|
||||
}
|
||||
|
||||
const buildInventorySnapshot = async (
|
||||
@@ -433,6 +438,18 @@ const buildInventorySnapshot = async (
|
||||
)
|
||||
occurrence.parent_occurrence_id = rootOccurrence.occurrence_id;
|
||||
input.occurrences.unshift(rootOccurrence);
|
||||
if (input.rootContradiction !== undefined) {
|
||||
rootOccurrence.hash_status = "mismatched";
|
||||
rootOccurrence.limitations.push(
|
||||
"Declared integrity metadata contradicts observed bytes.",
|
||||
);
|
||||
input.pendingContradictions.push({
|
||||
occurrence: rootOccurrence,
|
||||
integrity: input.rootContradiction,
|
||||
entryKind: "file",
|
||||
unpacked: false,
|
||||
});
|
||||
}
|
||||
|
||||
const occurrenceById = new Map(
|
||||
input.occurrences.map((occurrence) => [
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
|
||||
import {
|
||||
artifactContainerIntegrityDetails,
|
||||
artifactIntegrityPair,
|
||||
artifactIntegrityDetails,
|
||||
type ArtifactContainerIntegrityPair,
|
||||
type ArtifactIntegrityPair,
|
||||
} from "../../domain/artifactIntegrity.js";
|
||||
import { isAbsolute } from "node:path";
|
||||
@@ -89,6 +91,8 @@ export const scanReader = async (
|
||||
readonly inesRegionScan: InesRegionScan | undefined;
|
||||
readonly unknownParentOccurrences: ReadonlySet<MutableOccurrence>;
|
||||
readonly traversal: ArtifactReaderTraversal;
|
||||
/** A recorded disagreement between the root's own checksum and its bytes. */
|
||||
readonly rootContradiction: ArtifactContainerIntegrityPair | undefined;
|
||||
}> => {
|
||||
const { nodes, occurrences, pendingContradictions } = emptyScan();
|
||||
if (reader === undefined)
|
||||
@@ -100,6 +104,7 @@ export const scanReader = async (
|
||||
inesRegionScan: undefined,
|
||||
unknownParentOccurrences: new Set(),
|
||||
traversal: { status: "complete", limitations: [] },
|
||||
rootContradiction: undefined,
|
||||
};
|
||||
const context: ScanContext = {
|
||||
reader,
|
||||
@@ -120,6 +125,7 @@ export const scanReader = async (
|
||||
traversal: new ArtifactTraversalObservation(),
|
||||
};
|
||||
await visitArtifactEntries(context, reader, "");
|
||||
const rootContradiction = containerContradiction(reader, integrity);
|
||||
// Archive directories may appear after their children. Resolve containment
|
||||
// against the complete index before directory identities are materialized.
|
||||
const byPath = indexDestinationOccurrencesByPath(occurrences);
|
||||
@@ -148,9 +154,28 @@ export const scanReader = async (
|
||||
},
|
||||
unknownParentOccurrences: parentResolution.unknownParentOccurrences,
|
||||
traversal: context.traversal.snapshot(),
|
||||
rootContradiction,
|
||||
};
|
||||
};
|
||||
|
||||
/** Apply the integrity policy to a checksum the root declares over its own bytes. */
|
||||
const containerContradiction = (
|
||||
reader: ArtifactReader,
|
||||
integrity: ArtifactIntegrityPolicyName,
|
||||
): ArtifactContainerIntegrityPair | undefined => {
|
||||
const pair = reader.containerIntegrity?.();
|
||||
if (pair === undefined || pair.declared_crc32 === pair.observed_crc32)
|
||||
return undefined;
|
||||
if (integrity === "fail")
|
||||
throw new ArtifactReaderFailure(
|
||||
"integrity",
|
||||
"Artifact integrity metadata disagrees with content: .",
|
||||
undefined,
|
||||
artifactContainerIntegrityDetails(".", pair),
|
||||
);
|
||||
return pair;
|
||||
};
|
||||
|
||||
const inventoryEntries = (
|
||||
context: ScanContext,
|
||||
reader: ArtifactReader,
|
||||
|
||||
@@ -71,6 +71,18 @@ export const analysisErrorProjectionSchema = z.discriminatedUnion("code", [
|
||||
unpacked: z.boolean(),
|
||||
})
|
||||
.catchall(jsonValueSchema),
|
||||
z
|
||||
.object({
|
||||
logical_path: z.string(),
|
||||
algorithm: z.literal("dfu-crc32"),
|
||||
declared_crc32: z.string().regex(/^[a-f0-9]{8}$/u),
|
||||
calculated_crc32: z
|
||||
.string()
|
||||
.regex(/^[a-f0-9]{8}$/u)
|
||||
.nullable(),
|
||||
unpacked: z.boolean(),
|
||||
})
|
||||
.catchall(jsonValueSchema),
|
||||
]),
|
||||
})
|
||||
.strict(),
|
||||
|
||||
@@ -81,6 +81,8 @@ const formatSchema = z.enum([
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"esp-flash-image",
|
||||
"fit",
|
||||
"plist",
|
||||
|
||||
@@ -282,6 +282,8 @@ export const projectArtifactInventoryView = (
|
||||
inventory.manifest.root_format === "intel-hex" ||
|
||||
inventory.manifest.root_format === "srec" ||
|
||||
inventory.manifest.root_format === "esp-image" ||
|
||||
inventory.manifest.root_format === "uf2" ||
|
||||
inventory.manifest.root_format === "dfuse" ||
|
||||
inventory.manifest.root_format === "esp-flash-image" ||
|
||||
inventory.manifest.root_format === "fit";
|
||||
const counts = inventoryCounts(source);
|
||||
|
||||
@@ -14,6 +14,7 @@ import {
|
||||
type InventorySet,
|
||||
} from "./artifactInventoryEvidence.js";
|
||||
import {
|
||||
dfuSeElementDeclaration,
|
||||
espFlashPartitionDeclaration,
|
||||
espImageSegmentDeclaration,
|
||||
fitImageDeclaration,
|
||||
@@ -469,6 +470,7 @@ interface OccurrenceMetadata {
|
||||
readonly espFlashPartition: MetadataComparison;
|
||||
readonly fitImage: MetadataComparison;
|
||||
readonly fitImageAvailability: MetadataComparison;
|
||||
readonly dfuSeElement: MetadataComparison;
|
||||
readonly pyInstaller: MetadataComparison;
|
||||
}
|
||||
|
||||
@@ -478,7 +480,8 @@ const declarationChanged = (metadata: OccurrenceMetadata): boolean =>
|
||||
metadata.nodeSea === "changed" ||
|
||||
metadata.espImageSegment === "changed" ||
|
||||
metadata.espFlashPartition === "changed" ||
|
||||
metadata.fitImage === "changed";
|
||||
metadata.fitImage === "changed" ||
|
||||
metadata.dfuSeElement === "changed";
|
||||
|
||||
/** Unknown classifications and absent node metadata cannot establish a difference. */
|
||||
const differsWithUnknown = (
|
||||
@@ -584,6 +587,15 @@ const occurrenceMetadata = (input: OccurrencePair): OccurrenceMetadata => {
|
||||
: undefined,
|
||||
espImageSegmentDeclaration,
|
||||
),
|
||||
dfuSeElement: compareOptionalMetadata(
|
||||
left?.format_metadata?.kind === "dfuse-element"
|
||||
? left.format_metadata
|
||||
: undefined,
|
||||
right?.format_metadata?.kind === "dfuse-element"
|
||||
? right.format_metadata
|
||||
: undefined,
|
||||
dfuSeElementDeclaration,
|
||||
),
|
||||
espFlashPartition: compareOptionalMetadata(
|
||||
left?.format_metadata?.kind === "esp-flash-partition"
|
||||
? left.format_metadata
|
||||
|
||||
@@ -163,6 +163,86 @@ export const nodeSeaEntryMetadataSchema = z.strictObject({
|
||||
index: uint32,
|
||||
});
|
||||
|
||||
/**
|
||||
* UF2 facts derived from every 512-byte block. Regions hold only valid
|
||||
* main-flash payloads; block numbering is grouped by familyID and block count.
|
||||
*/
|
||||
export const uf2MetadataSchema = z.strictObject({
|
||||
kind: z.literal("uf2"),
|
||||
block_count: offset,
|
||||
trailing_bytes: offset,
|
||||
invalid_block_count: offset,
|
||||
flash_block_count: offset,
|
||||
not_main_flash_block_count: offset,
|
||||
file_container_block_count: offset,
|
||||
md5_block_count: offset,
|
||||
extension_tag_block_count: offset,
|
||||
family_count: offset,
|
||||
/** The registry revision REA's region `family_name` values come from. */
|
||||
family_name_registry: z.string(),
|
||||
numbering_group_count: offset,
|
||||
missing_block_count: offset,
|
||||
ambiguous_numbering_group_count: offset,
|
||||
repeated_block_count: offset,
|
||||
out_of_order_block_count: offset,
|
||||
block_number_out_of_range_count: offset,
|
||||
region_count: offset,
|
||||
data_bytes: offset,
|
||||
duplicate_bytes: offset,
|
||||
});
|
||||
export type Uf2Metadata = z.infer<typeof uf2MetadataSchema>;
|
||||
|
||||
/** One family's address-contiguous region; file blocks are zero-based positions. */
|
||||
export const uf2RegionMetadataSchema = z.strictObject({
|
||||
kind: z.literal("uf2-region"),
|
||||
family_id: uint32.nullable(),
|
||||
/** REA's label for `family_id`, from `family_name_registry`; not file content. */
|
||||
family_name: z.string().nullable(),
|
||||
start_address: uint32,
|
||||
end_address: uint32,
|
||||
length: offset,
|
||||
block_count: offset,
|
||||
first_file_block: offset,
|
||||
last_file_block: offset,
|
||||
});
|
||||
|
||||
/** DfuSe prefix and DFU suffix facts; both CRCs use the suffix's stored form. */
|
||||
export const dfuSeMetadataSchema = z.strictObject({
|
||||
kind: z.literal("dfuse"),
|
||||
image_size: uint32,
|
||||
target_count: uint32,
|
||||
element_count: offset,
|
||||
empty_element_count: offset,
|
||||
overlapping_element_count: offset,
|
||||
data_bytes: offset,
|
||||
device_version: uint32,
|
||||
product_id: uint32,
|
||||
vendor_id: uint32,
|
||||
suffix_crc: uint32,
|
||||
computed_crc: uint32,
|
||||
});
|
||||
export type DfuSeMetadata = z.infer<typeof dfuSeMetadataSchema>;
|
||||
|
||||
/** One DfuSe element with its target's alternate setting and name. */
|
||||
export const dfuSeElementMetadataSchema = z.strictObject({
|
||||
kind: z.literal("dfuse-element"),
|
||||
target_index: uint32,
|
||||
alternate_setting: uint32,
|
||||
target_name: z.string().nullable(),
|
||||
element_index: uint32,
|
||||
start_address: uint32,
|
||||
end_address: uint32,
|
||||
length: offset,
|
||||
});
|
||||
|
||||
/** The device memory an element targets, which its path and bytes do not show. */
|
||||
export const dfuSeElementDeclaration = (
|
||||
metadata: z.infer<typeof dfuSeElementMetadataSchema>,
|
||||
) => ({
|
||||
alternate_setting: metadata.alternate_setting,
|
||||
target_name: metadata.target_name,
|
||||
});
|
||||
|
||||
/**
|
||||
* Intel HEX or Motorola S-record facts derived from every record. Start
|
||||
* addresses are declared values, not proven entry points.
|
||||
@@ -763,6 +843,10 @@ export const artifactFormatMetadataSchema = z.discriminatedUnion("kind", [
|
||||
espImageMetadataSchema,
|
||||
espImageSegmentMetadataSchema,
|
||||
espImageSignatureBlockMetadataSchema,
|
||||
uf2MetadataSchema,
|
||||
uf2RegionMetadataSchema,
|
||||
dfuSeMetadataSchema,
|
||||
dfuSeElementMetadataSchema,
|
||||
espFlashImageMetadataSchema,
|
||||
espFlashPartitionMetadataSchema,
|
||||
fitMetadataSchema,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import {
|
||||
artifactDfuCrc32PairSchema,
|
||||
artifactIntegrityClaimSchema,
|
||||
artifactSha256PairSchema,
|
||||
artifactMd5PairSchema,
|
||||
@@ -95,6 +96,8 @@ export const ARTIFACT_FORMAT_IDS = [
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"esp-flash-image",
|
||||
"fit",
|
||||
"unknown",
|
||||
@@ -114,6 +117,8 @@ const artifactContentFormatSchema = artifactFormatSchema.extract([
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"fit",
|
||||
"mach-o-universal",
|
||||
"mach-o",
|
||||
@@ -198,6 +203,7 @@ const integrityContradictionBaseSchema = z.object({
|
||||
export const integrityContradictionSchema = z.union([
|
||||
integrityContradictionBaseSchema.extend(artifactSha256PairSchema.shape),
|
||||
integrityContradictionBaseSchema.extend(artifactMd5PairSchema.shape),
|
||||
integrityContradictionBaseSchema.extend(artifactDfuCrc32PairSchema.shape),
|
||||
]);
|
||||
|
||||
/**
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
immutableJsonSnapshotMember,
|
||||
} from "./immutableJson.js";
|
||||
import {
|
||||
artifactDfuCrc32PairSchema,
|
||||
artifactIntegrityPairOf,
|
||||
artifactSha256PairSchema,
|
||||
artifactMd5PairSchema,
|
||||
@@ -61,6 +62,7 @@ const inspectionContradictionBaseSchema = z.strictObject({
|
||||
const inspectionContradictionSchema = z.union([
|
||||
inspectionContradictionBaseSchema.extend(artifactSha256PairSchema.shape),
|
||||
inspectionContradictionBaseSchema.extend(artifactMd5PairSchema.shape),
|
||||
inspectionContradictionBaseSchema.extend(artifactDfuCrc32PairSchema.shape),
|
||||
]);
|
||||
|
||||
const nextProbeSchema = z.strictObject({
|
||||
@@ -307,7 +309,10 @@ const contradictions = (
|
||||
): ArtifactInspectionResult["contradictions"] =>
|
||||
inventory.integrity_contradictions.map((value) => {
|
||||
const semantic = {
|
||||
statement: `Declared integrity for ${value.logical_path} contradicts observed bytes.`,
|
||||
statement:
|
||||
value.logical_path === "."
|
||||
? "The container's declared integrity contradicts its observed bytes."
|
||||
: `Declared integrity for ${value.logical_path} contradicts observed bytes.`,
|
||||
occurrence_id: value.occurrence_id,
|
||||
...artifactIntegrityPairOf(value),
|
||||
evidence_id: evidenceId,
|
||||
@@ -340,9 +345,17 @@ const probesFor = (
|
||||
},
|
||||
]
|
||||
: [];
|
||||
if (format === "intel-hex" || format === "srec")
|
||||
if (
|
||||
format === "intel-hex" ||
|
||||
format === "srec" ||
|
||||
format === "uf2" ||
|
||||
format === "dfuse"
|
||||
)
|
||||
return inventory.occurrences.some(
|
||||
(row) => row.format_metadata?.kind === "hex-record-region",
|
||||
(row) =>
|
||||
row.format_metadata?.kind === "hex-record-region" ||
|
||||
row.format_metadata?.kind === "uf2-region" ||
|
||||
row.format_metadata?.kind === "dfuse-element",
|
||||
)
|
||||
? [
|
||||
{
|
||||
|
||||
@@ -2,6 +2,7 @@ import { z } from "zod";
|
||||
|
||||
const sha256 = z.string().regex(/^[a-f0-9]{64}$/u);
|
||||
const md5 = z.string().regex(/^[a-f0-9]{32}$/u);
|
||||
const crc32 = z.string().regex(/^[a-f0-9]{8}$/u);
|
||||
|
||||
/** A container's claim is distinct from the SHA-256 identity of observed bytes. */
|
||||
export const artifactIntegrityClaimSchema = z.discriminatedUnion("algorithm", [
|
||||
@@ -22,9 +23,23 @@ export const artifactMd5PairSchema = z.object({
|
||||
declared_md5: md5,
|
||||
observed_md5: md5,
|
||||
});
|
||||
/**
|
||||
* A container's DFU suffix checksum over its own bytes before `dwCRC`: CRC-32
|
||||
* without the final inversion, as the suffix stores it.
|
||||
*/
|
||||
export const artifactDfuCrc32PairSchema = z.object({
|
||||
algorithm: z.literal("dfu-crc32"),
|
||||
declared_crc32: crc32,
|
||||
observed_crc32: crc32,
|
||||
});
|
||||
/** A checksum a container declares over its own bytes, with the observed value. */
|
||||
export type ArtifactContainerIntegrityPair = z.infer<
|
||||
typeof artifactDfuCrc32PairSchema
|
||||
>;
|
||||
export type ArtifactIntegrityPair =
|
||||
| z.infer<typeof artifactSha256PairSchema>
|
||||
| z.infer<typeof artifactMd5PairSchema>;
|
||||
| z.infer<typeof artifactMd5PairSchema>
|
||||
| ArtifactContainerIntegrityPair;
|
||||
|
||||
/** The canonical portable fields shared by inventory and inspection findings. */
|
||||
export const artifactIntegrityPair = (
|
||||
@@ -38,17 +53,24 @@ export const artifactIntegrityPair = (
|
||||
/** Select only digest facts from an authenticated inventory contradiction. */
|
||||
export const artifactIntegrityPairOf = (
|
||||
pair: ArtifactIntegrityPair,
|
||||
): ArtifactIntegrityPair =>
|
||||
"declared_sha256" in pair
|
||||
): ArtifactIntegrityPair => {
|
||||
if ("declared_sha256" in pair)
|
||||
return {
|
||||
declared_sha256: pair.declared_sha256,
|
||||
observed_sha256: pair.observed_sha256,
|
||||
};
|
||||
return pair.algorithm === "md5"
|
||||
? {
|
||||
declared_sha256: pair.declared_sha256,
|
||||
observed_sha256: pair.observed_sha256,
|
||||
}
|
||||
: {
|
||||
algorithm: pair.algorithm,
|
||||
declared_md5: pair.declared_md5,
|
||||
observed_md5: pair.observed_md5,
|
||||
}
|
||||
: {
|
||||
algorithm: pair.algorithm,
|
||||
declared_crc32: pair.declared_crc32,
|
||||
observed_crc32: pair.observed_crc32,
|
||||
};
|
||||
};
|
||||
|
||||
export type ArtifactIntegrityDetails = {
|
||||
readonly logicalPath: string;
|
||||
@@ -63,6 +85,11 @@ export type ArtifactIntegrityDetails = {
|
||||
readonly declaredMd5: string;
|
||||
readonly calculatedMd5: string | null;
|
||||
}
|
||||
| {
|
||||
readonly algorithm: "dfu-crc32";
|
||||
readonly declaredCrc32: string;
|
||||
readonly calculatedCrc32: string | null;
|
||||
}
|
||||
);
|
||||
|
||||
/** Preserve the existing error carrier while naming non-SHA digest algorithms. */
|
||||
@@ -83,19 +110,41 @@ export const artifactIntegrityDetails = (
|
||||
: { declaredSha256: claim?.digest ?? null, calculatedSha256: observed }),
|
||||
});
|
||||
|
||||
/** Error details for a container checksum that disagrees with its own bytes. */
|
||||
export const artifactContainerIntegrityDetails = (
|
||||
logicalPath: string,
|
||||
pair: ArtifactContainerIntegrityPair,
|
||||
): ArtifactIntegrityDetails => ({
|
||||
logicalPath,
|
||||
unpacked: false,
|
||||
algorithm: pair.algorithm,
|
||||
declaredCrc32: pair.declared_crc32,
|
||||
calculatedCrc32: pair.observed_crc32,
|
||||
});
|
||||
|
||||
const projectedDigests = (details: ArtifactIntegrityDetails) => {
|
||||
if ("declaredSha256" in details)
|
||||
return {
|
||||
declared_sha256: details.declaredSha256,
|
||||
calculated_sha256: details.calculatedSha256,
|
||||
};
|
||||
return details.algorithm === "md5"
|
||||
? {
|
||||
algorithm: details.algorithm,
|
||||
declared_md5: details.declaredMd5,
|
||||
calculated_md5: details.calculatedMd5,
|
||||
}
|
||||
: {
|
||||
algorithm: details.algorithm,
|
||||
declared_crc32: details.declaredCrc32,
|
||||
calculated_crc32: details.calculatedCrc32,
|
||||
};
|
||||
};
|
||||
|
||||
export const projectArtifactIntegrityDetails = (
|
||||
details: ArtifactIntegrityDetails,
|
||||
) => ({
|
||||
logical_path: details.logicalPath,
|
||||
...("declaredSha256" in details
|
||||
? {
|
||||
declared_sha256: details.declaredSha256,
|
||||
calculated_sha256: details.calculatedSha256,
|
||||
}
|
||||
: {
|
||||
algorithm: details.algorithm,
|
||||
declared_md5: details.declaredMd5,
|
||||
calculated_md5: details.calculatedMd5,
|
||||
}),
|
||||
...projectedDigests(details),
|
||||
unpacked: details.unpacked,
|
||||
});
|
||||
@@ -299,6 +299,8 @@ export type BinaryTarget =
|
||||
| "intel-hex"
|
||||
| "srec"
|
||||
| "esp-image"
|
||||
| "uf2"
|
||||
| "dfuse"
|
||||
| "fit";
|
||||
})
|
||||
| (BinaryTargetIdentity &
|
||||
|
||||
@@ -46,6 +46,8 @@ export const evidenceSubjectFormatSchema = z.enum([
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"esp-flash-image",
|
||||
"fit",
|
||||
]);
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
|
||||
/** Bytes written at consecutive addresses, held at `offset` in a decoder arena. */
|
||||
export interface ArenaRun {
|
||||
readonly address: number;
|
||||
readonly offset: number;
|
||||
readonly length: number;
|
||||
}
|
||||
|
||||
/** One address-contiguous range and the runs that wrote it, in address order. */
|
||||
export interface MergedRuns<Run extends ArenaRun> {
|
||||
readonly start: number;
|
||||
/** The range's bytes; its inclusive end is `start + bytes.length - 1`. */
|
||||
readonly bytes: Uint8Array;
|
||||
readonly runs: readonly Run[];
|
||||
}
|
||||
|
||||
/** The first address two overlapping runs write with different values. */
|
||||
export interface RunConflict<Run extends ArenaRun> {
|
||||
readonly address: number;
|
||||
/** The earlier covering run in address order, which supplied the kept byte. */
|
||||
readonly writer: Run;
|
||||
readonly run: Run;
|
||||
readonly existing: number;
|
||||
readonly incoming: number;
|
||||
}
|
||||
|
||||
const runEnd = (run: ArenaRun | undefined): number =>
|
||||
run === undefined ? 0 : run.address + run.length;
|
||||
|
||||
/**
|
||||
* Merge runs into address-contiguous ranges, never filling gaps. Runs in
|
||||
* different address spaces never merge; `space` orders them before address.
|
||||
* Overlapping runs must repeat the same bytes, which count as duplicates.
|
||||
*/
|
||||
export const mergeArenaRuns = <Run extends ArenaRun>(
|
||||
runs: readonly Run[],
|
||||
arena: Uint8Array,
|
||||
space: (run: Run) => number = () => 0,
|
||||
): Result<
|
||||
{ readonly ranges: MergedRuns<Run>[]; readonly duplicateBytes: number },
|
||||
RunConflict<Run>
|
||||
> => {
|
||||
const ordered = [...runs].sort(
|
||||
(left, right) => space(left) - space(right) || left.address - right.address,
|
||||
);
|
||||
const ranges: MergedRuns<Run>[] = [];
|
||||
let duplicateBytes = 0;
|
||||
let first = 0;
|
||||
while (first < ordered.length) {
|
||||
const head = ordered[first];
|
||||
if (head === undefined) break;
|
||||
let next = first + 1;
|
||||
let end = runEnd(head);
|
||||
while (next < ordered.length) {
|
||||
const candidate = ordered[next];
|
||||
if (
|
||||
candidate === undefined ||
|
||||
space(candidate) !== space(head) ||
|
||||
candidate.address > end
|
||||
)
|
||||
break;
|
||||
end = Math.max(end, runEnd(candidate));
|
||||
next += 1;
|
||||
}
|
||||
const range = mergeRange(ordered.slice(first, next), end, arena);
|
||||
if (!range.ok) return range;
|
||||
ranges.push(range.value.range);
|
||||
duplicateBytes += range.value.duplicateBytes;
|
||||
first = next;
|
||||
}
|
||||
return ok({ ranges, duplicateBytes });
|
||||
};
|
||||
|
||||
/** Copy one contiguous group; written addresses always form [start, written). */
|
||||
const mergeRange = <Run extends ArenaRun>(
|
||||
group: readonly Run[],
|
||||
end: number,
|
||||
arena: Uint8Array,
|
||||
): Result<
|
||||
{ readonly range: MergedRuns<Run>; readonly duplicateBytes: number },
|
||||
RunConflict<Run>
|
||||
> => {
|
||||
const start = group[0]?.address ?? 0;
|
||||
const bytes = new Uint8Array(end - start);
|
||||
let written = start;
|
||||
let duplicateBytes = 0;
|
||||
for (const run of group) {
|
||||
const overlapEnd = Math.min(written, runEnd(run));
|
||||
for (let address = run.address; address < overlapEnd; address += 1) {
|
||||
const existing = bytes[address - start] ?? 0;
|
||||
const incoming = arena[run.offset + address - run.address] ?? 0;
|
||||
if (existing !== incoming)
|
||||
return err({
|
||||
address,
|
||||
writer:
|
||||
group.find(
|
||||
(candidate) =>
|
||||
candidate !== run &&
|
||||
candidate.address <= address &&
|
||||
address < runEnd(candidate),
|
||||
) ?? run,
|
||||
run,
|
||||
existing,
|
||||
incoming,
|
||||
});
|
||||
}
|
||||
duplicateBytes += Math.max(overlapEnd - run.address, 0);
|
||||
if (runEnd(run) > written) {
|
||||
bytes.set(
|
||||
arena.subarray(
|
||||
run.offset + written - run.address,
|
||||
run.offset + run.length,
|
||||
),
|
||||
written - start,
|
||||
);
|
||||
written = runEnd(run);
|
||||
}
|
||||
}
|
||||
return ok({ range: { start, bytes, runs: group }, duplicateBytes });
|
||||
};
|
||||
@@ -0,0 +1,93 @@
|
||||
import { crc32 } from "node:zlib";
|
||||
|
||||
export interface DfuSeFixtureElement {
|
||||
readonly address: number;
|
||||
readonly data: Uint8Array | readonly number[];
|
||||
/** A dwElementSize that differs from the data, for malformed elements. */
|
||||
readonly declaredSize?: number;
|
||||
}
|
||||
|
||||
export interface DfuSeFixtureTarget {
|
||||
readonly alternateSetting: number;
|
||||
/** Omitted or null writes bTargetNamed 0; raw bytes cover non-text names. */
|
||||
readonly name?: string | Uint8Array | null;
|
||||
readonly elements: readonly DfuSeFixtureElement[];
|
||||
readonly declaredSize?: number;
|
||||
readonly declaredElements?: number;
|
||||
readonly signature?: string;
|
||||
}
|
||||
|
||||
export interface DfuSeFixture {
|
||||
readonly targets: readonly DfuSeFixtureTarget[];
|
||||
readonly vendorId?: number;
|
||||
readonly productId?: number;
|
||||
readonly deviceVersion?: number;
|
||||
readonly declaredTargets?: number;
|
||||
/** Replaces DFUImageSize, which otherwise counts the prefix and targets. */
|
||||
readonly imageSize?: number;
|
||||
/** Count the suffix in DFUImageSize too, as dfu-util's dfuse-pack.py does. */
|
||||
readonly imageSizeIncludesSuffix?: boolean;
|
||||
/** Bytes between the last target and the suffix. */
|
||||
readonly gap?: Uint8Array;
|
||||
readonly dfuVersion?: number;
|
||||
/** Replaces the computed dwCRC. */
|
||||
readonly crc?: number;
|
||||
}
|
||||
|
||||
const targetBytes = (target: DfuSeFixtureTarget): Buffer => {
|
||||
const elements = Buffer.concat(
|
||||
target.elements.map((element) => {
|
||||
const data = Uint8Array.from(element.data);
|
||||
const header = Buffer.alloc(8);
|
||||
header.writeUInt32LE(element.address, 0);
|
||||
header.writeUInt32LE(element.declaredSize ?? data.length, 4);
|
||||
return Buffer.concat([header, data]);
|
||||
}),
|
||||
);
|
||||
const prefix = Buffer.alloc(274);
|
||||
prefix.write(target.signature ?? "Target", 0, "latin1");
|
||||
prefix.writeUInt8(target.alternateSetting, 6);
|
||||
if (target.name !== undefined && target.name !== null) {
|
||||
prefix.writeUInt32LE(1, 7);
|
||||
const name =
|
||||
typeof target.name === "string"
|
||||
? Buffer.from(target.name, "latin1")
|
||||
: Buffer.from(target.name);
|
||||
name.copy(prefix, 11, 0, Math.min(name.length, 255));
|
||||
}
|
||||
prefix.writeUInt32LE(target.declaredSize ?? elements.length, 266);
|
||||
prefix.writeUInt32LE(target.declaredElements ?? target.elements.length, 270);
|
||||
return Buffer.concat([prefix, elements]);
|
||||
};
|
||||
|
||||
/**
|
||||
* A DfuSe file written from ST UM0391 and the DFU 1.1 suffix, with the suffix
|
||||
* CRC computed as dfu-util's dfuse-pack.py does: the inverted zlib CRC-32.
|
||||
* DFUImageSize excludes the suffix unless `imageSizeIncludesSuffix` is set.
|
||||
*/
|
||||
export const dfuseFile = (fixture: DfuSeFixture): Buffer => {
|
||||
const targets = Buffer.concat(fixture.targets.map(targetBytes));
|
||||
const gap = Buffer.from(fixture.gap ?? []);
|
||||
const prefix = Buffer.alloc(11);
|
||||
prefix.write("DfuSe", 0, "latin1");
|
||||
prefix.writeUInt8(1, 5);
|
||||
prefix.writeUInt32LE(
|
||||
fixture.imageSize ??
|
||||
prefix.length +
|
||||
targets.length +
|
||||
gap.length +
|
||||
(fixture.imageSizeIncludesSuffix === true ? 16 : 0),
|
||||
6,
|
||||
);
|
||||
prefix.writeUInt8(fixture.declaredTargets ?? fixture.targets.length, 10);
|
||||
const suffix = Buffer.alloc(16);
|
||||
suffix.writeUInt16LE(fixture.deviceVersion ?? 0xffff, 0);
|
||||
suffix.writeUInt16LE(fixture.productId ?? 0xdf11, 2);
|
||||
suffix.writeUInt16LE(fixture.vendorId ?? 0x0483, 4);
|
||||
suffix.writeUInt16LE(fixture.dfuVersion ?? 0x011a, 6);
|
||||
suffix.write("UFD", 8, "latin1");
|
||||
suffix.writeUInt8(16, 11);
|
||||
const body = Buffer.concat([prefix, targets, gap, suffix.subarray(0, 12)]);
|
||||
suffix.writeUInt32LE(fixture.crc ?? ~crc32(body) >>> 0, 12);
|
||||
return Buffer.concat([body, suffix.subarray(12)]);
|
||||
};
|
||||
@@ -0,0 +1,406 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
DFUSE_MAX_FILE_BYTES,
|
||||
createDfuSeDecoder,
|
||||
isDfuSeFile,
|
||||
type DfuSeImage,
|
||||
} from "./dfuse.js";
|
||||
import { dfuseFile, type DfuSeFixture } from "./dfuse.fixture.js";
|
||||
|
||||
const decode = (bytes: Uint8Array, chunkSize = 64) => {
|
||||
const decoder = createDfuSeDecoder(bytes.length);
|
||||
if (!decoder.ok) throw new Error(decoder.error.message);
|
||||
for (let offset = 0; offset < bytes.length; offset += chunkSize) {
|
||||
const written = decoder.value.write(
|
||||
bytes.subarray(offset, offset + chunkSize),
|
||||
);
|
||||
if (!written.ok) return written;
|
||||
}
|
||||
return decoder.value.finish();
|
||||
};
|
||||
|
||||
const image = (bytes: Uint8Array): DfuSeImage => {
|
||||
const decoded = decode(bytes);
|
||||
if (!decoded.ok) throw new Error(decoded.error.message);
|
||||
return decoded.value;
|
||||
};
|
||||
|
||||
const recognized = (bytes: Uint8Array): boolean =>
|
||||
isDfuSeFile(bytes.subarray(0, 4096), bytes.subarray(-16), bytes.length);
|
||||
|
||||
const elementFacts = ({ elements }: DfuSeImage) =>
|
||||
elements.map(({ bytes, ...facts }) => ({ ...facts, length: bytes.length }));
|
||||
|
||||
/** One 16-byte image at 0x08000000 in target "ST...", as the packers below write it. */
|
||||
const DFUSE_PACK_EQUIVALENT: DfuSeFixture = {
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
name: "ST...",
|
||||
elements: [{ address: 0x0800_0000, data: [...Array(16).keys()] }],
|
||||
},
|
||||
],
|
||||
deviceVersion: 0,
|
||||
};
|
||||
|
||||
it("decodes files packed with and without the suffix in DFUImageSize", () => {
|
||||
// Oracle: build() and compute_crc() of dfu-util's dfuse-pack.py at f9a537c9,
|
||||
// transcribed and run under Python 3. Since ee6b393d it writes DFUImageSize
|
||||
// as PREFIX_SIZE + len(data) + SUFFIX_SIZE, the whole file.
|
||||
const packed = dfuseFile({
|
||||
...DFUSE_PACK_EQUIVALENT,
|
||||
imageSizeIncludesSuffix: true,
|
||||
});
|
||||
expect(createHash("sha256").update(packed).digest("hex")).toBe(
|
||||
"4d9f8c231291504484745f0b0586031c8f804d141e964cb33a9b82965f8a69d0",
|
||||
);
|
||||
expect(image(packed).metadata).toMatchObject({
|
||||
image_size: 325,
|
||||
suffix_crc: 0x2c4c_4f44,
|
||||
computed_crc: 0x2c4c_4f44,
|
||||
});
|
||||
const bytes = dfuseFile(DFUSE_PACK_EQUIVALENT);
|
||||
// Oracle: build() and compute_crc() of MicroPython's tools/dfu.py at
|
||||
// cb99ca98, transcribed and run under Python 3. It writes len(data) + 11,
|
||||
// leaving the suffix out, as dfuse-pack.py did before ee6b393d.
|
||||
expect(bytes.length).toBe(325);
|
||||
expect(createHash("sha256").update(bytes).digest("hex")).toBe(
|
||||
"65ad96452f1f82cc8b44d30dfe94f268f4fd230a32b30a8826a639b269f7034d",
|
||||
);
|
||||
const decoded = image(bytes);
|
||||
expect(decoded.metadata).toEqual({
|
||||
kind: "dfuse",
|
||||
image_size: 309,
|
||||
target_count: 1,
|
||||
element_count: 1,
|
||||
empty_element_count: 0,
|
||||
overlapping_element_count: 0,
|
||||
data_bytes: 16,
|
||||
device_version: 0,
|
||||
product_id: 0xdf11,
|
||||
vendor_id: 0x0483,
|
||||
suffix_crc: 0x7a37_11da,
|
||||
computed_crc: 0x7a37_11da,
|
||||
});
|
||||
expect(elementFacts(decoded)).toEqual([
|
||||
{
|
||||
targetIndex: 0,
|
||||
alternateSetting: 0,
|
||||
targetName: "ST...",
|
||||
elementIndex: 0,
|
||||
address: 0x0800_0000,
|
||||
dataOffset: 293,
|
||||
length: 16,
|
||||
},
|
||||
]);
|
||||
expect(decoded.limitations).toEqual([]);
|
||||
});
|
||||
|
||||
it("keeps each target's alternate setting and name on its elements", () => {
|
||||
const decoded = image(
|
||||
dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
name: "Internal Flash",
|
||||
elements: [
|
||||
{ address: 0x0800_0000, data: [1, 2, 3, 4] },
|
||||
{ address: 0x0800_4000, data: [5, 6] },
|
||||
],
|
||||
},
|
||||
{
|
||||
alternateSetting: 1,
|
||||
elements: [{ address: 0x1fff_c000, data: [0xaa, 0x55] }],
|
||||
},
|
||||
{
|
||||
alternateSetting: 2,
|
||||
name: Uint8Array.from([0x4f, 0xe9, 0, 0x58]),
|
||||
elements: [{ address: 0x1fff_7800, data: [] }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(
|
||||
decoded.elements.map((element) => [
|
||||
element.targetIndex,
|
||||
element.alternateSetting,
|
||||
element.targetName,
|
||||
element.elementIndex,
|
||||
element.address,
|
||||
[...element.bytes],
|
||||
]),
|
||||
).toEqual([
|
||||
[0, 0, "Internal Flash", 0, 0x0800_0000, [1, 2, 3, 4]],
|
||||
[0, 0, "Internal Flash", 1, 0x0800_4000, [5, 6]],
|
||||
[1, 1, null, 0, 0x1fff_c000, [0xaa, 0x55]],
|
||||
]);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
target_count: 3,
|
||||
element_count: 4,
|
||||
empty_element_count: 1,
|
||||
data_bytes: 8,
|
||||
});
|
||||
// A named target's field is read to its first NUL, one character per byte.
|
||||
const named = image(
|
||||
dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 3,
|
||||
name: Uint8Array.from([0x4f, 0xe9, 0, 0x58]),
|
||||
elements: [{ address: 0, data: [1] }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(named.elements[0]?.targetName).toBe("Oé");
|
||||
});
|
||||
|
||||
it("reports a suffix CRC mismatch as facts without refusing the file", () => {
|
||||
const decoded = image(dfuseFile({ ...DFUSE_PACK_EQUIVALENT, crc: 0 }));
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
suffix_crc: 0,
|
||||
computed_crc: 0x7a37_11da,
|
||||
});
|
||||
expect(decoded.elements).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("names overlapping elements within one alternate setting only", () => {
|
||||
const decoded = image(
|
||||
dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
elements: [
|
||||
{ address: 0x0800_0000, data: [1, 2, 3, 4] },
|
||||
{ address: 0x0800_0002, data: [3, 4] },
|
||||
],
|
||||
},
|
||||
{
|
||||
alternateSetting: 1,
|
||||
elements: [{ address: 0x0800_0000, data: [9] }],
|
||||
},
|
||||
{
|
||||
alternateSetting: 0,
|
||||
elements: [{ address: 0x0800_0003, data: [4, 5] }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(decoded.metadata.overlapping_element_count).toBe(2);
|
||||
expect(decoded.limitations).toEqual([
|
||||
"2 DfuSe element(s) overlap an earlier element of the same alternate setting; each stays a separate entry and no bytes were merged: target 0 element 1 (0x08000002-0x08000003) overlaps target 0 element 0 (0x08000000-0x08000003) in alternate setting 0; target 2 element 0 (0x08000003-0x08000004) overlaps target 0 element 0 (0x08000000-0x08000003) in alternate setting 0.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("accepts boundary declarations beside the malformed ones", () => {
|
||||
// A valid file may hold no targets at all.
|
||||
expect(image(dfuseFile({ targets: [] })).metadata).toMatchObject({
|
||||
image_size: 11,
|
||||
target_count: 0,
|
||||
element_count: 0,
|
||||
});
|
||||
const name = Uint8Array.from({ length: 255 }, () => 0x41);
|
||||
const decoded = image(
|
||||
dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 4,
|
||||
// A name filling all 255 bytes has no NUL.
|
||||
name,
|
||||
elements: [
|
||||
// An element may end exactly at the top of the address space.
|
||||
{ address: 0xffff_fffe, data: [1, 2] },
|
||||
{ address: 0x2000_0000, data: [] },
|
||||
],
|
||||
},
|
||||
{ alternateSetting: 5, elements: [] },
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(decoded.elements.map((element) => element.targetName)).toEqual([
|
||||
"A".repeat(255),
|
||||
]);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
target_count: 2,
|
||||
element_count: 2,
|
||||
empty_element_count: 1,
|
||||
});
|
||||
expect(decoded.limitations).toEqual([
|
||||
"DfuSe declarations without bytes have no entry: target 0 element 1 at 0x20000000 (alternate setting 4) holds no bytes; target 1 (alternate setting 5) declares no elements.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("recognizes a file only by the DfuSe prefix together with the DFU suffix", () => {
|
||||
const bytes = dfuseFile(DFUSE_PACK_EQUIVALENT);
|
||||
expect(recognized(bytes)).toBe(true);
|
||||
const changed = (offset: number, value: number): Buffer => {
|
||||
const copy = Buffer.from(bytes);
|
||||
copy[offset] = value;
|
||||
return copy;
|
||||
};
|
||||
expect(recognized(changed(5, 2))).toBe(false);
|
||||
expect(recognized(changed(0, 0x64))).toBe(false);
|
||||
expect(recognized(changed(bytes.length - 10, 0x00))).toBe(false);
|
||||
expect(recognized(changed(bytes.length - 8, 0x58))).toBe(false);
|
||||
expect(recognized(changed(bytes.length - 5, 15))).toBe(false);
|
||||
expect(recognized(bytes.subarray(0, bytes.length - 1))).toBe(false);
|
||||
// The CRC is checked after recognition, so a mismatch still recognizes.
|
||||
expect(recognized(dfuseFile({ ...DFUSE_PACK_EQUIVALENT, crc: 1 }))).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[
|
||||
"a target without its signature",
|
||||
{
|
||||
targets: [{ alternateSetting: 0, signature: "Tarxet", elements: [] }],
|
||||
},
|
||||
'byte offset 11: target 0 does not begin with the "Target" signature',
|
||||
],
|
||||
[
|
||||
"an element larger than its target",
|
||||
{
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
elements: [{ address: 0, data: [1, 2], declaredSize: 3 }],
|
||||
},
|
||||
],
|
||||
},
|
||||
"byte offset 289: target 0 element 0 declares 3 bytes, but its target ends at byte 295",
|
||||
],
|
||||
[
|
||||
"a target larger than the image",
|
||||
{
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
declaredSize: 100,
|
||||
elements: [{ address: 0, data: [1] }],
|
||||
},
|
||||
],
|
||||
},
|
||||
"byte offset 277: target 0 declares 100 bytes, but the image ends at byte 294",
|
||||
],
|
||||
[
|
||||
"fewer elements than the target size",
|
||||
{
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
declaredElements: 0,
|
||||
elements: [{ address: 0, data: [1] }],
|
||||
},
|
||||
],
|
||||
},
|
||||
"byte offset 285: target 0 declares 9 bytes, but its 0 element(s) end 9 bytes earlier",
|
||||
],
|
||||
[
|
||||
"an image size that matches neither packer convention",
|
||||
{ targets: [], imageSize: 12 },
|
||||
"byte offset 6: DFUImageSize declares 12 bytes, but the DFU suffix begins at byte 11 and the file holds 27 bytes; the suffix CRC matches the file",
|
||||
],
|
||||
[
|
||||
"bytes between the last target and the suffix",
|
||||
{ targets: [], gap: Uint8Array.from([0]) },
|
||||
"byte offset 11: 0 target(s) end at byte 11, but the DFU suffix begins at byte 12",
|
||||
],
|
||||
[
|
||||
"more targets than the image holds",
|
||||
{ targets: [], declaredTargets: 1 },
|
||||
"byte offset 11: target 0 prefix needs 274 bytes, but the image ends at byte 11",
|
||||
],
|
||||
[
|
||||
"more elements than the target holds",
|
||||
{
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
declaredElements: 2,
|
||||
elements: [{ address: 0, data: [1] }],
|
||||
},
|
||||
],
|
||||
},
|
||||
"byte offset 294: target 0 element 1 header lies beyond the target's end at byte 294",
|
||||
],
|
||||
[
|
||||
"an element that wraps the address space",
|
||||
{
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
elements: [{ address: 0xffff_ffff, data: [1, 2] }],
|
||||
},
|
||||
],
|
||||
},
|
||||
"byte offset 285: target 0 element 0 at 0xffffffff with 2 bytes wraps past 0xffffffff",
|
||||
],
|
||||
] as const)("refuses %s with its byte offset", (_name, fixture, message) => {
|
||||
expect(decode(dfuseFile(fixture))).toEqual({
|
||||
ok: false,
|
||||
error: { kind: "format", message },
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses files above the fixed ceiling and more elements than retained", () => {
|
||||
expect(createDfuSeDecoder(DFUSE_MAX_FILE_BYTES + 1)).toMatchObject({
|
||||
ok: false,
|
||||
error: { kind: "limit", boundary: "file-size" },
|
||||
});
|
||||
const elements = (count: number, data: readonly number[]) =>
|
||||
Array.from({ length: count }, (_, index) => ({
|
||||
address: index * 2,
|
||||
data,
|
||||
}));
|
||||
const accepted = decode(
|
||||
dfuseFile({
|
||||
targets: [{ alternateSetting: 0, elements: elements(65_536, [1]) }],
|
||||
}),
|
||||
1 << 20,
|
||||
);
|
||||
if (!accepted.ok) throw new Error(accepted.error.message);
|
||||
expect(accepted.value.elements).toHaveLength(65_536);
|
||||
expect(
|
||||
decode(
|
||||
dfuseFile({
|
||||
targets: [{ alternateSetting: 0, elements: elements(65_537, [1]) }],
|
||||
}),
|
||||
1 << 20,
|
||||
),
|
||||
).toMatchObject({
|
||||
ok: false,
|
||||
error: { kind: "limit", boundary: "capacity" },
|
||||
});
|
||||
});
|
||||
|
||||
it("counts empty elements past the entry cap and names only the first four", () => {
|
||||
// Empty elements have no entry, so they are not capped; 8 header bytes each
|
||||
// let a file hold millions, and only a count and four descriptions are kept.
|
||||
const empty = Array.from({ length: 65_537 }, (_, index) => ({
|
||||
address: index * 2,
|
||||
data: [],
|
||||
}));
|
||||
const decoded = decode(
|
||||
dfuseFile({
|
||||
targets: [
|
||||
{ alternateSetting: 0, elements: empty },
|
||||
{ alternateSetting: 1, elements: [] },
|
||||
],
|
||||
}),
|
||||
1 << 20,
|
||||
);
|
||||
if (!decoded.ok) throw new Error(decoded.error.message);
|
||||
expect(decoded.value.elements).toEqual([]);
|
||||
expect(decoded.value.metadata).toMatchObject({
|
||||
element_count: 65_537,
|
||||
empty_element_count: 65_537,
|
||||
data_bytes: 0,
|
||||
});
|
||||
expect(decoded.value.limitations).toEqual([
|
||||
"DfuSe declarations without bytes have no entry: target 0 element 0 at 0x00000000 (alternate setting 0) holds no bytes; target 0 element 1 at 0x00000002 (alternate setting 0) holds no bytes; target 0 element 2 at 0x00000004 (alternate setting 0) holds no bytes; target 0 element 3 at 0x00000006 (alternate setting 0) holds no bytes; and 65534 more.",
|
||||
]);
|
||||
});
|
||||
@@ -0,0 +1,412 @@
|
||||
import { crc32 } from "node:zlib";
|
||||
|
||||
import type { DfuSeMetadata } from "../artifactFormatMetadata.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import type { RegionDecodeFailure, RegionDecoder } from "./regionDecoder.js";
|
||||
|
||||
/** "DfuSe", bVersion, DFUImageSize and bTargets (ST UM0391). */
|
||||
const PREFIX_BYTES = 11;
|
||||
/** "Target", bAlternateSetting, bTargetNamed, szTargetName, dwTargetSize, dwNbElements. */
|
||||
const TARGET_PREFIX_BYTES = 274;
|
||||
const TARGET_NAME_OFFSET = 11;
|
||||
const TARGET_NAME_BYTES = 255;
|
||||
/** dwElementAddress and dwElementSize. */
|
||||
const ELEMENT_HEADER_BYTES = 8;
|
||||
/** bcdDevice, idProduct, idVendor, bcdDFU, "UFD", bLength and dwCRC (DFU 1.1). */
|
||||
export const DFUSE_SUFFIX_BYTES = 16;
|
||||
const DFUSE_VERSION = 1;
|
||||
const DFU_SUFFIX_VERSION = 0x011a;
|
||||
const ADDRESS_SPACE = 0x1_0000_0000;
|
||||
/** Elements named in one limitation before the rest are counted. */
|
||||
const NAMED_ELEMENTS = 4;
|
||||
const SIGNATURE = [0x44, 0x66, 0x75, 0x53, 0x65];
|
||||
const TARGET_SIGNATURE = [0x54, 0x61, 0x72, 0x67, 0x65, 0x74];
|
||||
const SUFFIX_SIGNATURE = [0x55, 0x46, 0x44];
|
||||
|
||||
/**
|
||||
* Fixed whole-file ceiling, as for Intel HEX; configurable ceilings are
|
||||
* tracked in #6176. The decoder holds the file once and slices elements from it.
|
||||
*/
|
||||
export const DFUSE_MAX_FILE_BYTES = 128 * 1024 * 1024;
|
||||
/** Non-empty elements retained as inventory entries, as for UF2 regions. */
|
||||
export const DFUSE_MAX_ELEMENTS = 65_536;
|
||||
|
||||
/** One element: bytes the file declares for an address in one alternate setting. */
|
||||
export interface DfuSeElement {
|
||||
readonly targetIndex: number;
|
||||
readonly alternateSetting: number;
|
||||
/** szTargetName up to its first NUL, one character per byte; null when unnamed. */
|
||||
readonly targetName: string | null;
|
||||
readonly elementIndex: number;
|
||||
readonly address: number;
|
||||
/** The file offset of the element data. */
|
||||
readonly dataOffset: number;
|
||||
readonly bytes: Uint8Array;
|
||||
}
|
||||
|
||||
export interface DfuSeImage {
|
||||
readonly metadata: DfuSeMetadata;
|
||||
/** In file order; empty elements are counted in metadata but not listed. */
|
||||
readonly elements: readonly DfuSeElement[];
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
const readUint16 = (bytes: Uint8Array, offset: number): number =>
|
||||
(bytes[offset] ?? 0) | ((bytes[offset + 1] ?? 0) << 8);
|
||||
|
||||
const readUint32 = (bytes: Uint8Array, offset: number): number =>
|
||||
(readUint16(bytes, offset) | (readUint16(bytes, offset + 2) << 16)) >>> 0;
|
||||
|
||||
const startsWith = (
|
||||
bytes: Uint8Array,
|
||||
offset: number,
|
||||
signature: readonly number[],
|
||||
): boolean => signature.every((byte, index) => bytes[offset + index] === byte);
|
||||
|
||||
const hex32 = (value: number): string =>
|
||||
`0x${value.toString(16).padStart(8, "0")}`;
|
||||
|
||||
/** Whether the leading bytes begin a DfuSe version 1 image. */
|
||||
export const hasDfuSePrefix = (prefix: Uint8Array): boolean =>
|
||||
startsWith(prefix, 0, SIGNATURE) && prefix[5] === DFUSE_VERSION;
|
||||
|
||||
/** Whether the final 16 bytes are a DFU 1.1a suffix, as DfuSe requires. */
|
||||
const hasDfuSeSuffix = (suffix: Uint8Array): boolean =>
|
||||
suffix.length === DFUSE_SUFFIX_BYTES &&
|
||||
readUint16(suffix, 6) === DFU_SUFFIX_VERSION &&
|
||||
startsWith(suffix, 8, SUFFIX_SIGNATURE) &&
|
||||
suffix[11] === DFUSE_SUFFIX_BYTES;
|
||||
|
||||
/**
|
||||
* Recognize a file only by the DfuSe prefix and the DFU suffix together. The
|
||||
* suffix CRC is not part of recognition: a mismatch is an integrity fact.
|
||||
*/
|
||||
export const isDfuSeFile = (
|
||||
prefix: Uint8Array,
|
||||
suffix: Uint8Array,
|
||||
fileBytes: number,
|
||||
): boolean =>
|
||||
fileBytes >= PREFIX_BYTES + DFUSE_SUFFIX_BYTES &&
|
||||
hasDfuSePrefix(prefix) &&
|
||||
hasDfuSeSuffix(suffix);
|
||||
|
||||
/** Start a decoder that holds at most the admitted file size. */
|
||||
export const createDfuSeDecoder = (
|
||||
fileBytes: number,
|
||||
): Result<RegionDecoder<DfuSeImage>, RegionDecodeFailure> =>
|
||||
Number.isSafeInteger(fileBytes) &&
|
||||
fileBytes >= 0 &&
|
||||
fileBytes <= DFUSE_MAX_FILE_BYTES
|
||||
? ok(new RetainingDfuSeDecoder(fileBytes))
|
||||
: err({
|
||||
kind: "limit",
|
||||
boundary: "file-size",
|
||||
message: `file is ${String(fileBytes)} bytes, exceeding the fixed ${String(DFUSE_MAX_FILE_BYTES)}-byte (128 MiB) ceiling`,
|
||||
});
|
||||
|
||||
/** The DFU suffix CRC: CRC-32 without the final inversion, over every byte before dwCRC. */
|
||||
const dfuCrc = (bytes: Uint8Array): number => ~crc32(bytes) >>> 0;
|
||||
|
||||
const formatFailure = (
|
||||
offset: number,
|
||||
detail: string,
|
||||
): RegionDecodeFailure => ({
|
||||
kind: "format",
|
||||
message: `byte offset ${String(offset)}: ${detail}`,
|
||||
});
|
||||
|
||||
/** At most NAMED_ELEMENTS descriptions, then a count of the rest of `total`. */
|
||||
const named = (items: readonly string[], total = items.length): string =>
|
||||
`${items.slice(0, NAMED_ELEMENTS).join("; ")}${total > NAMED_ELEMENTS ? `; and ${String(total - NAMED_ELEMENTS)} more` : ""}`;
|
||||
|
||||
class RetainingDfuSeDecoder implements RegionDecoder<DfuSeImage> {
|
||||
readonly #bytes: Uint8Array;
|
||||
#length = 0;
|
||||
|
||||
constructor(fileBytes: number) {
|
||||
this.#bytes = new Uint8Array(fileBytes);
|
||||
}
|
||||
|
||||
write(chunk: Uint8Array): Result<void, RegionDecodeFailure> {
|
||||
if (this.#length + chunk.length > this.#bytes.length)
|
||||
return err({
|
||||
kind: "limit",
|
||||
boundary: "file-size",
|
||||
message: "file grew beyond its admitted size while being read",
|
||||
});
|
||||
this.#bytes.set(chunk, this.#length);
|
||||
this.#length += chunk.length;
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
finish(): Result<DfuSeImage, RegionDecodeFailure> {
|
||||
return parseDfuSe(this.#bytes.subarray(0, this.#length));
|
||||
}
|
||||
}
|
||||
|
||||
const parseDfuSe = (
|
||||
bytes: Uint8Array,
|
||||
): Result<DfuSeImage, RegionDecodeFailure> => {
|
||||
const suffixStart = bytes.length - DFUSE_SUFFIX_BYTES;
|
||||
if (
|
||||
bytes.length < PREFIX_BYTES + DFUSE_SUFFIX_BYTES ||
|
||||
!hasDfuSePrefix(bytes) ||
|
||||
!hasDfuSeSuffix(bytes.subarray(suffixStart))
|
||||
)
|
||||
return err(
|
||||
formatFailure(
|
||||
0,
|
||||
"the file does not begin with a DfuSe prefix and end with a DFU 1.1a suffix",
|
||||
),
|
||||
);
|
||||
// UM0391 calls DFUImageSize the file length, which dfu-util's dfuse-pack.py
|
||||
// writes since ee6b393d (2020-10-08). Earlier dfuse-pack.py and MicroPython's
|
||||
// tools/dfu.py write the length before the suffix.
|
||||
const imageSize = readUint32(bytes, 6);
|
||||
const suffixCrc = readUint32(bytes, suffixStart + 12);
|
||||
const computedCrc = dfuCrc(bytes.subarray(0, bytes.length - 4));
|
||||
if (imageSize !== suffixStart && imageSize !== bytes.length)
|
||||
return err(
|
||||
formatFailure(
|
||||
6,
|
||||
`DFUImageSize declares ${String(imageSize)} bytes, but the DFU suffix begins at byte ${String(suffixStart)} and the file holds ${String(bytes.length)} bytes; the suffix CRC ${suffixCrc === computedCrc ? "matches" : "does not match"} the file`,
|
||||
),
|
||||
);
|
||||
const targetCount = bytes[10] ?? 0;
|
||||
const walk = new TargetWalk(bytes, suffixStart);
|
||||
let offset = PREFIX_BYTES;
|
||||
for (let targetIndex = 0; targetIndex < targetCount; targetIndex += 1) {
|
||||
const next = walk.target(targetIndex, offset);
|
||||
if (!next.ok) return next;
|
||||
offset = next.value;
|
||||
}
|
||||
if (offset !== suffixStart)
|
||||
return err(
|
||||
formatFailure(
|
||||
offset,
|
||||
`${String(targetCount)} target(s) end at byte ${String(offset)}, but the DFU suffix begins at byte ${String(suffixStart)}`,
|
||||
),
|
||||
);
|
||||
const { elements } = walk;
|
||||
const overlaps = overlappingElements(elements);
|
||||
return ok({
|
||||
metadata: {
|
||||
kind: "dfuse",
|
||||
image_size: imageSize,
|
||||
target_count: targetCount,
|
||||
element_count: walk.elementCount,
|
||||
empty_element_count: walk.emptyElements,
|
||||
overlapping_element_count: overlaps.length,
|
||||
data_bytes: elements.reduce(
|
||||
(sum, element) => sum + element.bytes.length,
|
||||
0,
|
||||
),
|
||||
device_version: readUint16(bytes, suffixStart),
|
||||
product_id: readUint16(bytes, suffixStart + 2),
|
||||
vendor_id: readUint16(bytes, suffixStart + 4),
|
||||
suffix_crc: suffixCrc,
|
||||
computed_crc: computedCrc,
|
||||
},
|
||||
elements,
|
||||
limitations: [
|
||||
...(overlaps.length === 0
|
||||
? []
|
||||
: [
|
||||
`${String(overlaps.length)} DfuSe element(s) overlap an earlier element of the same alternate setting; each stays a separate entry and no bytes were merged: ${named(overlaps)}.`,
|
||||
]),
|
||||
...(walk.unlistedCount === 0
|
||||
? []
|
||||
: [
|
||||
`DfuSe declarations without bytes have no entry: ${named(walk.unlisted, walk.unlistedCount)}.`,
|
||||
]),
|
||||
],
|
||||
});
|
||||
};
|
||||
|
||||
/** Targets and elements in file order; every size is checked against its container. */
|
||||
class TargetWalk {
|
||||
readonly elements: DfuSeElement[] = [];
|
||||
/**
|
||||
* The first NAMED_ELEMENTS empty elements and element-less targets, which no
|
||||
* entry can show. A file can hold millions of empty elements, so the rest
|
||||
* are only counted in `unlistedCount`.
|
||||
*/
|
||||
readonly unlisted: string[] = [];
|
||||
unlistedCount = 0;
|
||||
elementCount = 0;
|
||||
emptyElements = 0;
|
||||
|
||||
constructor(
|
||||
private readonly bytes: Uint8Array,
|
||||
private readonly suffixStart: number,
|
||||
) {}
|
||||
|
||||
/** Parse one target at `offset`; the result is the offset after its elements. */
|
||||
target(
|
||||
targetIndex: number,
|
||||
offset: number,
|
||||
): Result<number, RegionDecodeFailure> {
|
||||
const { bytes } = this;
|
||||
if (offset + TARGET_PREFIX_BYTES > this.suffixStart)
|
||||
return err(
|
||||
formatFailure(
|
||||
offset,
|
||||
`target ${String(targetIndex)} prefix needs ${String(TARGET_PREFIX_BYTES)} bytes, but the image ends at byte ${String(this.suffixStart)}`,
|
||||
),
|
||||
);
|
||||
if (!startsWith(bytes, offset, TARGET_SIGNATURE))
|
||||
return err(
|
||||
formatFailure(
|
||||
offset,
|
||||
`target ${String(targetIndex)} does not begin with the "Target" signature`,
|
||||
),
|
||||
);
|
||||
const targetSize = readUint32(bytes, offset + 266);
|
||||
const declaredElements = readUint32(bytes, offset + 270);
|
||||
const bodyStart = offset + TARGET_PREFIX_BYTES;
|
||||
const bodyEnd = bodyStart + targetSize;
|
||||
if (bodyEnd > this.suffixStart)
|
||||
return err(
|
||||
formatFailure(
|
||||
offset + 266,
|
||||
`target ${String(targetIndex)} declares ${String(targetSize)} bytes, but the image ends at byte ${String(this.suffixStart)}`,
|
||||
),
|
||||
);
|
||||
const target = {
|
||||
targetIndex,
|
||||
alternateSetting: bytes[offset + 6] ?? 0,
|
||||
targetName:
|
||||
readUint32(bytes, offset + 7) === 0
|
||||
? null
|
||||
: latin1UntilNul(
|
||||
bytes.subarray(
|
||||
offset + TARGET_NAME_OFFSET,
|
||||
offset + TARGET_NAME_OFFSET + TARGET_NAME_BYTES,
|
||||
),
|
||||
),
|
||||
};
|
||||
if (declaredElements === 0)
|
||||
this.#unlisted(
|
||||
() =>
|
||||
`target ${String(targetIndex)} (alternate setting ${String(target.alternateSetting)}) declares no elements`,
|
||||
);
|
||||
let position = bodyStart;
|
||||
for (let index = 0; index < declaredElements; index += 1) {
|
||||
const next = this.#element(target, index, position, bodyEnd);
|
||||
if (!next.ok) return next;
|
||||
position = next.value;
|
||||
}
|
||||
return position === bodyEnd
|
||||
? ok(bodyEnd)
|
||||
: err(
|
||||
formatFailure(
|
||||
position,
|
||||
`target ${String(targetIndex)} declares ${String(targetSize)} bytes, but its ${String(declaredElements)} element(s) end ${String(bodyEnd - position)} bytes earlier`,
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
#element(
|
||||
target: Pick<
|
||||
DfuSeElement,
|
||||
"targetIndex" | "alternateSetting" | "targetName"
|
||||
>,
|
||||
elementIndex: number,
|
||||
position: number,
|
||||
bodyEnd: number,
|
||||
): Result<number, RegionDecodeFailure> {
|
||||
const name = `target ${String(target.targetIndex)} element ${String(elementIndex)}`;
|
||||
if (position + ELEMENT_HEADER_BYTES > bodyEnd)
|
||||
return err(
|
||||
formatFailure(
|
||||
position,
|
||||
`${name} header lies beyond the target's end at byte ${String(bodyEnd)}`,
|
||||
),
|
||||
);
|
||||
const address = readUint32(this.bytes, position);
|
||||
const size = readUint32(this.bytes, position + 4);
|
||||
const dataOffset = position + ELEMENT_HEADER_BYTES;
|
||||
if (dataOffset + size > bodyEnd)
|
||||
return err(
|
||||
formatFailure(
|
||||
position + 4,
|
||||
`${name} declares ${String(size)} bytes, but its target ends at byte ${String(bodyEnd)}`,
|
||||
),
|
||||
);
|
||||
if (address + size > ADDRESS_SPACE)
|
||||
return err(
|
||||
formatFailure(
|
||||
position,
|
||||
`${name} at ${hex32(address)} with ${String(size)} bytes wraps past 0xffffffff`,
|
||||
),
|
||||
);
|
||||
if (size > 0 && this.elements.length >= DFUSE_MAX_ELEMENTS)
|
||||
return err({
|
||||
kind: "limit",
|
||||
boundary: "capacity",
|
||||
message: `byte offset ${String(position)}: the file holds more than ${String(DFUSE_MAX_ELEMENTS)} elements with bytes, the most retained as entries`,
|
||||
});
|
||||
this.elementCount += 1;
|
||||
if (size === 0) {
|
||||
this.emptyElements += 1;
|
||||
this.#unlisted(
|
||||
() =>
|
||||
`${name} at ${hex32(address)} (alternate setting ${String(target.alternateSetting)}) holds no bytes`,
|
||||
);
|
||||
} else
|
||||
this.elements.push({
|
||||
...target,
|
||||
elementIndex,
|
||||
address,
|
||||
dataOffset,
|
||||
bytes: this.bytes.subarray(dataOffset, dataOffset + size),
|
||||
});
|
||||
return ok(dataOffset + size);
|
||||
}
|
||||
|
||||
/** Count a declaration without bytes; describe only the first few. */
|
||||
#unlisted(describe: () => string): void {
|
||||
this.unlistedCount += 1;
|
||||
if (this.unlisted.length < NAMED_ELEMENTS) this.unlisted.push(describe());
|
||||
}
|
||||
}
|
||||
|
||||
/** szTargetName is a fixed field; bytes after the first NUL are padding. */
|
||||
const latin1UntilNul = (field: Uint8Array): string => {
|
||||
const end = field.indexOf(0);
|
||||
return String.fromCharCode(
|
||||
...field.subarray(0, end < 0 ? field.length : end),
|
||||
);
|
||||
};
|
||||
|
||||
const elementText = (element: DfuSeElement): string =>
|
||||
`target ${String(element.targetIndex)} element ${String(element.elementIndex)} (${hex32(element.address)}-${hex32(element.address + element.bytes.length - 1)})`;
|
||||
|
||||
/** Elements that overlap an earlier one in address order within one alternate setting. */
|
||||
const overlappingElements = (elements: readonly DfuSeElement[]): string[] => {
|
||||
const ordered = [...elements].sort(
|
||||
(left, right) =>
|
||||
left.alternateSetting - right.alternateSetting ||
|
||||
left.address - right.address,
|
||||
);
|
||||
const overlaps: string[] = [];
|
||||
let reach: DfuSeElement | undefined;
|
||||
for (const element of ordered) {
|
||||
if (
|
||||
reach !== undefined &&
|
||||
reach.alternateSetting === element.alternateSetting &&
|
||||
element.address < reach.address + reach.bytes.length
|
||||
)
|
||||
overlaps.push(
|
||||
`${elementText(element)} overlaps ${elementText(reach)} in alternate setting ${String(element.alternateSetting)}`,
|
||||
);
|
||||
if (
|
||||
reach === undefined ||
|
||||
reach.alternateSetting !== element.alternateSetting ||
|
||||
element.address + element.bytes.length >
|
||||
reach.address + reach.bytes.length
|
||||
)
|
||||
reach = element;
|
||||
}
|
||||
return overlaps;
|
||||
};
|
||||
@@ -1,5 +1,7 @@
|
||||
import type { HexRecordsMetadata } from "../artifactFormatMetadata.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import { mergeArenaRuns, type RunConflict } from "./addressRuns.js";
|
||||
import type { RegionDecodeFailure, RegionDecoder } from "./regionDecoder.js";
|
||||
import {
|
||||
COLON,
|
||||
CR,
|
||||
@@ -28,13 +30,6 @@ export const HEX_RECORD_ENCODING_LABELS: Readonly<
|
||||
Record<HexRecordEncoding, string>
|
||||
> = { "intel-hex": "Intel HEX", srec: "S-record" };
|
||||
|
||||
/** A whole-file refusal; no partial regions accompany it. */
|
||||
export interface HexRecordFailure {
|
||||
readonly kind: "format" | "limit";
|
||||
readonly message: string;
|
||||
readonly boundary?: "file-size" | "capacity";
|
||||
}
|
||||
|
||||
/** Decoded bytes for one address-contiguous range; `end` is inclusive. */
|
||||
export interface HexRecordRegion {
|
||||
readonly start: number;
|
||||
@@ -53,13 +48,6 @@ export interface HexRecordImage {
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
export interface HexRecordDecoder {
|
||||
/** Decode the next file bytes; a line may span chunks. */
|
||||
write(chunk: Uint8Array): Result<void, HexRecordFailure>;
|
||||
/** Decode a final unterminated line, then merge runs into regions. */
|
||||
finish(): Result<HexRecordImage, HexRecordFailure>;
|
||||
}
|
||||
|
||||
/** Consecutive records whose addresses continue each other. */
|
||||
interface RecordRun {
|
||||
readonly address: number;
|
||||
@@ -99,7 +87,7 @@ export const detectHexRecordEncoding = (
|
||||
export const createHexRecordDecoder = (
|
||||
encoding: HexRecordEncoding,
|
||||
fileBytes: number,
|
||||
): Result<HexRecordDecoder, HexRecordFailure> =>
|
||||
): Result<RegionDecoder<HexRecordImage>, RegionDecodeFailure> =>
|
||||
Number.isSafeInteger(fileBytes) &&
|
||||
fileBytes >= 0 &&
|
||||
fileBytes <= HEX_RECORD_MAX_FILE_BYTES
|
||||
@@ -110,12 +98,12 @@ export const createHexRecordDecoder = (
|
||||
message: `file is ${String(fileBytes)} bytes, exceeding the fixed ${String(HEX_RECORD_MAX_FILE_BYTES)}-byte (128 MiB) ceiling`,
|
||||
});
|
||||
|
||||
const formatFailure = (line: number, detail: string): HexRecordFailure => ({
|
||||
const formatFailure = (line: number, detail: string): RegionDecodeFailure => ({
|
||||
kind: "format",
|
||||
message: `line ${String(line)}: ${detail}`,
|
||||
});
|
||||
|
||||
class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
class StreamingHexRecordDecoder implements RegionDecoder<HexRecordImage> {
|
||||
readonly #encoding: HexRecordEncoding;
|
||||
readonly #line: Uint8Array;
|
||||
readonly #runs: RecordRun[] = [];
|
||||
@@ -143,7 +131,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
this.#arena = new Uint8Array(Math.floor(fileBytes / 2));
|
||||
}
|
||||
|
||||
write(chunk: Uint8Array): Result<void, HexRecordFailure> {
|
||||
write(chunk: Uint8Array): Result<void, RegionDecodeFailure> {
|
||||
let from = 0;
|
||||
while (from < chunk.length) {
|
||||
const lf = chunk.indexOf(LF, from);
|
||||
@@ -160,7 +148,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
finish(): Result<HexRecordImage, HexRecordFailure> {
|
||||
finish(): Result<HexRecordImage, RegionDecodeFailure> {
|
||||
if (this.#lineLength > 0) {
|
||||
const ended = this.#endLine(false);
|
||||
if (!ended.ok) return ended;
|
||||
@@ -177,14 +165,14 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
});
|
||||
}
|
||||
|
||||
#overlong(line: number): HexRecordFailure {
|
||||
#overlong(line: number): RegionDecodeFailure {
|
||||
return formatFailure(
|
||||
line,
|
||||
`line exceeds the ${String(maxLineChars(this.#encoding))}-character ${HEX_RECORD_ENCODING_LABELS[this.#encoding]} maximum`,
|
||||
);
|
||||
}
|
||||
|
||||
#endLine(lfTerminated: boolean): Result<void, HexRecordFailure> {
|
||||
#endLine(lfTerminated: boolean): Result<void, RegionDecodeFailure> {
|
||||
this.#lineNumber += 1;
|
||||
let length = this.#lineLength;
|
||||
this.#lineLength = 0;
|
||||
@@ -208,11 +196,11 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
: this.#applySrec(parsed.value);
|
||||
}
|
||||
|
||||
#failure(detail: string): HexRecordFailure {
|
||||
#failure(detail: string): RegionDecodeFailure {
|
||||
return formatFailure(this.#lineNumber, detail);
|
||||
}
|
||||
|
||||
#applyIntel(record: HexRecord): Result<void, HexRecordFailure> {
|
||||
#applyIntel(record: HexRecord): Result<void, RegionDecodeFailure> {
|
||||
const value = readBigEndian(record.data);
|
||||
switch (record.type) {
|
||||
case "00":
|
||||
@@ -236,7 +224,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
}
|
||||
}
|
||||
|
||||
#intelData({ address, data }: HexRecord): Result<void, HexRecordFailure> {
|
||||
#intelData({ address, data }: HexRecord): Result<void, RegionDecodeFailure> {
|
||||
this.#dataRecordCount += 1;
|
||||
// The specification wraps within 64 KiB, but some producers continue
|
||||
// linearly, so the address of the wrapped bytes is ambiguous.
|
||||
@@ -256,7 +244,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
return this.#addData(start, data);
|
||||
}
|
||||
|
||||
#applySrec(record: HexRecord): Result<void, HexRecordFailure> {
|
||||
#applySrec(record: HexRecord): Result<void, RegionDecodeFailure> {
|
||||
switch (record.type) {
|
||||
case "S0":
|
||||
if (this.#header !== undefined)
|
||||
@@ -291,7 +279,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
type,
|
||||
address,
|
||||
data,
|
||||
}: HexRecord): Result<void, HexRecordFailure> {
|
||||
}: HexRecord): Result<void, RegionDecodeFailure> {
|
||||
this.#dataRecordCount += 1;
|
||||
this.#dataTypes.add(type);
|
||||
const limit = SREC_DATA_LIMITS[type] ?? 0;
|
||||
@@ -304,7 +292,10 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
return this.#addData(address, data);
|
||||
}
|
||||
|
||||
#declareStart(value: number, record: string): Result<void, HexRecordFailure> {
|
||||
#declareStart(
|
||||
value: number,
|
||||
record: string,
|
||||
): Result<void, RegionDecodeFailure> {
|
||||
const existing = this.#start;
|
||||
if (existing !== undefined && existing.value !== value)
|
||||
return err(
|
||||
@@ -318,7 +309,7 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
|
||||
#declareStartSegment(
|
||||
value: readonly [number, number],
|
||||
): Result<void, HexRecordFailure> {
|
||||
): Result<void, RegionDecodeFailure> {
|
||||
this.#segmentRecords = true;
|
||||
const existing = this.#startSegment;
|
||||
if (
|
||||
@@ -334,7 +325,10 @@ class StreamingHexRecordDecoder implements HexRecordDecoder {
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
#addData(address: number, data: Uint8Array): Result<void, HexRecordFailure> {
|
||||
#addData(
|
||||
address: number,
|
||||
data: Uint8Array,
|
||||
): Result<void, RegionDecodeFailure> {
|
||||
if (data.length === 0) return ok(undefined);
|
||||
if (this.#arenaUsed + data.length > this.#arena.length)
|
||||
return err({
|
||||
@@ -455,68 +449,14 @@ const mergeRuns = (
|
||||
arena: Uint8Array,
|
||||
): Result<
|
||||
{ regions: HexRecordRegion[]; duplicateBytes: number },
|
||||
HexRecordFailure
|
||||
RegionDecodeFailure
|
||||
> => {
|
||||
const ordered = [...runs].sort((left, right) => left.address - right.address);
|
||||
const regions: HexRecordRegion[] = [];
|
||||
let duplicateBytes = 0;
|
||||
let first = 0;
|
||||
while (first < ordered.length) {
|
||||
let next = first + 1;
|
||||
let end = runEnd(ordered[first]);
|
||||
while (next < ordered.length && (ordered[next]?.address ?? 0) <= end) {
|
||||
end = Math.max(end, runEnd(ordered[next]));
|
||||
next += 1;
|
||||
}
|
||||
const region = mergeRegion(ordered.slice(first, next), end, arena);
|
||||
if (!region.ok) return region;
|
||||
regions.push(region.value.region);
|
||||
duplicateBytes += region.value.duplicateBytes;
|
||||
first = next;
|
||||
}
|
||||
return ok({ regions, duplicateBytes });
|
||||
};
|
||||
|
||||
const runEnd = (run: RecordRun | undefined): number =>
|
||||
run === undefined ? 0 : run.address + run.length;
|
||||
|
||||
/** Copy one contiguous group; written addresses always form [start, written). */
|
||||
const mergeRegion = (
|
||||
group: readonly RecordRun[],
|
||||
end: number,
|
||||
arena: Uint8Array,
|
||||
): Result<
|
||||
{ region: HexRecordRegion; duplicateBytes: number },
|
||||
HexRecordFailure
|
||||
> => {
|
||||
const start = group[0]?.address ?? 0;
|
||||
const bytes = new Uint8Array(end - start);
|
||||
let written = start;
|
||||
let duplicateBytes = 0;
|
||||
for (const run of group) {
|
||||
const overlapEnd = Math.min(written, runEnd(run));
|
||||
for (let address = run.address; address < overlapEnd; address += 1) {
|
||||
const existing = bytes[address - start] ?? 0;
|
||||
const incoming = arena[run.offset + address - run.address] ?? 0;
|
||||
if (existing !== incoming)
|
||||
return err(conflict(group, run, address, [existing, incoming]));
|
||||
}
|
||||
duplicateBytes += Math.max(overlapEnd - run.address, 0);
|
||||
if (runEnd(run) > written) {
|
||||
bytes.set(
|
||||
arena.subarray(
|
||||
run.offset + written - run.address,
|
||||
run.offset + run.length,
|
||||
),
|
||||
written - start,
|
||||
);
|
||||
written = runEnd(run);
|
||||
}
|
||||
}
|
||||
const merged = mergeArenaRuns(runs, arena);
|
||||
if (!merged.ok) return err(conflict(merged.error));
|
||||
return ok({
|
||||
region: {
|
||||
regions: merged.value.ranges.map(({ start, bytes, runs: group }) => ({
|
||||
start,
|
||||
end: end - 1,
|
||||
end: start + bytes.length - 1,
|
||||
bytes,
|
||||
recordCount: group.reduce((sum, run) => sum + run.recordCount, 0),
|
||||
firstLine: group.reduce(
|
||||
@@ -524,27 +464,18 @@ const mergeRegion = (
|
||||
Number.MAX_SAFE_INTEGER,
|
||||
),
|
||||
lastLine: group.reduce((line, run) => Math.max(line, run.lastLine), 0),
|
||||
},
|
||||
duplicateBytes,
|
||||
})),
|
||||
duplicateBytes: merged.value.duplicateBytes,
|
||||
});
|
||||
};
|
||||
|
||||
const conflict = (
|
||||
group: readonly RecordRun[],
|
||||
run: RecordRun,
|
||||
address: number,
|
||||
[existing, incoming]: readonly [number, number],
|
||||
): HexRecordFailure => {
|
||||
// The first covering run in address order supplied the retained byte.
|
||||
const writer =
|
||||
group.find(
|
||||
(candidate) =>
|
||||
candidate !== run &&
|
||||
candidate.address <= address &&
|
||||
address < runEnd(candidate),
|
||||
) ?? run;
|
||||
return {
|
||||
kind: "format",
|
||||
message: `address 0x${hex(address, 8)} written with different values at ${lineSpan(writer)} (0x${hex(existing, 2)}) and ${lineSpan(run)} (0x${hex(incoming, 2)})`,
|
||||
};
|
||||
};
|
||||
const conflict = ({
|
||||
address,
|
||||
writer,
|
||||
run,
|
||||
existing,
|
||||
incoming,
|
||||
}: RunConflict<RecordRun>): RegionDecodeFailure => ({
|
||||
kind: "format",
|
||||
message: `address 0x${hex(address, 8)} written with different values at ${lineSpan(writer)} (0x${hex(existing, 2)}) and ${lineSpan(run)} (0x${hex(incoming, 2)})`,
|
||||
});
|
||||
@@ -0,0 +1,14 @@
|
||||
import type { Result } from "../result.js";
|
||||
|
||||
/** A whole-file refusal from a firmware decoder; no partial regions accompany it. */
|
||||
export interface RegionDecodeFailure {
|
||||
readonly kind: "format" | "limit";
|
||||
readonly message: string;
|
||||
readonly boundary?: "file-size" | "capacity";
|
||||
}
|
||||
|
||||
/** A streaming firmware decoder, fed the admitted file bytes in order. */
|
||||
export interface RegionDecoder<Image> {
|
||||
write(chunk: Uint8Array): Result<void, RegionDecodeFailure>;
|
||||
finish(): Result<Image, RegionDecodeFailure>;
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
/** UF2 flag values from Microsoft's UF2 specification. */
|
||||
export const UF2_FLAGS = {
|
||||
notMainFlash: 0x0000_0001,
|
||||
fileContainer: 0x0000_1000,
|
||||
familyId: 0x0000_2000,
|
||||
md5: 0x0000_4000,
|
||||
extensionTags: 0x0000_8000,
|
||||
} as const;
|
||||
|
||||
export const RP2040_FAMILY = 0xe48b_ff56;
|
||||
export const RP2XXX_ABSOLUTE_FAMILY = 0xe48b_ff57;
|
||||
export const RP2350_ARM_S_FAMILY = 0xe48b_ff59;
|
||||
|
||||
export interface Uf2BlockFields {
|
||||
readonly address: number;
|
||||
readonly payload: Uint8Array | readonly number[];
|
||||
readonly blockNo: number;
|
||||
readonly numBlocks: number;
|
||||
/** Sets flag 0x2000 and the familyID word unless `fileSize` is given. */
|
||||
readonly familyId?: number;
|
||||
readonly fileSize?: number;
|
||||
readonly flags?: number;
|
||||
/** A payload size field that differs from the payload, for malformed blocks. */
|
||||
readonly declaredSize?: number;
|
||||
readonly magics?: readonly [number, number, number];
|
||||
}
|
||||
|
||||
/** One 512-byte block written from the specification's field layout. */
|
||||
export const uf2Block = (fields: Uf2BlockFields): Uint8Array => {
|
||||
const block = Buffer.alloc(512);
|
||||
const [start0, start1, end] = fields.magics ?? [
|
||||
0x0a32_4655, 0x9e5d_5157, 0x0ab1_6f30,
|
||||
];
|
||||
const payload = Uint8Array.from(fields.payload);
|
||||
const flags =
|
||||
(fields.flags ?? 0) |
|
||||
(fields.familyId === undefined ? 0 : UF2_FLAGS.familyId);
|
||||
block.writeUInt32LE(start0, 0);
|
||||
block.writeUInt32LE(start1, 4);
|
||||
block.writeUInt32LE(flags >>> 0, 8);
|
||||
block.writeUInt32LE(fields.address, 12);
|
||||
block.writeUInt32LE(fields.declaredSize ?? payload.length, 16);
|
||||
block.writeUInt32LE(fields.blockNo, 20);
|
||||
block.writeUInt32LE(fields.numBlocks, 24);
|
||||
block.writeUInt32LE(fields.familyId ?? fields.fileSize ?? 0, 28);
|
||||
block.set(payload.subarray(0, 476), 32);
|
||||
block.writeUInt32LE(end, 508);
|
||||
return block;
|
||||
};
|
||||
|
||||
/** Numbered 256-byte blocks for one family, as RP2040 producers write them. */
|
||||
export const uf2Image = (
|
||||
familyId: number | undefined,
|
||||
address: number,
|
||||
bytes: Uint8Array,
|
||||
payloadSize = 256,
|
||||
): Uint8Array[] => {
|
||||
const numBlocks = Math.ceil(bytes.length / payloadSize);
|
||||
return Array.from({ length: numBlocks }, (_, blockNo) =>
|
||||
uf2Block({
|
||||
address: address + blockNo * payloadSize,
|
||||
payload: bytes.subarray(
|
||||
blockNo * payloadSize,
|
||||
(blockNo + 1) * payloadSize,
|
||||
),
|
||||
blockNo,
|
||||
numBlocks,
|
||||
...(familyId === undefined ? {} : { familyId }),
|
||||
}),
|
||||
);
|
||||
};
|
||||
|
||||
/** Deterministic, address-dependent bytes so misplaced payloads are visible. */
|
||||
export const patternBytes = (length: number, seed = 0): Uint8Array =>
|
||||
Uint8Array.from({ length }, (_, index) => (index * 7 + seed) & 0xff);
|
||||
|
||||
export const uf2File = (blocks: readonly Uint8Array[]): Buffer =>
|
||||
Buffer.concat(blocks);
|
||||
@@ -0,0 +1,567 @@
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
createUf2Decoder,
|
||||
hasUf2Block,
|
||||
UF2_MAX_FILE_BYTES,
|
||||
type Uf2Image,
|
||||
} from "./uf2.js";
|
||||
import {
|
||||
RP2040_FAMILY,
|
||||
RP2350_ARM_S_FAMILY,
|
||||
RP2XXX_ABSOLUTE_FAMILY,
|
||||
UF2_FLAGS,
|
||||
patternBytes,
|
||||
uf2Block,
|
||||
uf2File,
|
||||
uf2Image,
|
||||
} from "./uf2.fixture.js";
|
||||
|
||||
const decode = (bytes: Uint8Array, chunkSize = bytes.length || 1) => {
|
||||
const decoder = createUf2Decoder(bytes.length);
|
||||
if (!decoder.ok) throw new Error(decoder.error.message);
|
||||
for (let offset = 0; offset < bytes.length; offset += chunkSize) {
|
||||
const written = decoder.value.write(
|
||||
bytes.subarray(offset, offset + chunkSize),
|
||||
);
|
||||
if (!written.ok) return written;
|
||||
}
|
||||
return decoder.value.finish();
|
||||
};
|
||||
|
||||
const image = (bytes: Uint8Array, chunkSize?: number): Uf2Image => {
|
||||
const decoded = decode(bytes, chunkSize);
|
||||
if (!decoded.ok) throw new Error(decoded.error.message);
|
||||
return decoded.value;
|
||||
};
|
||||
|
||||
const regionFacts = ({ regions }: Uf2Image) =>
|
||||
regions.map(({ bytes, ...facts }) => ({ ...facts, length: bytes.length }));
|
||||
|
||||
it("recognizes only a complete first block with all three magic numbers", () => {
|
||||
const block = uf2Block({
|
||||
address: 0x1000_0000,
|
||||
payload: [1],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
});
|
||||
expect(hasUf2Block(block)).toBe(true);
|
||||
expect(hasUf2Block(block.subarray(0, 511))).toBe(false);
|
||||
for (const magics of [
|
||||
[0x0a32_4656, 0x9e5d_5157, 0x0ab1_6f30],
|
||||
[0x0a32_4655, 0x9e5d_5156, 0x0ab1_6f30],
|
||||
[0x0a32_4655, 0x9e5d_5157, 0x0ab1_6f31],
|
||||
] as const)
|
||||
expect(
|
||||
hasUf2Block(
|
||||
uf2Block({
|
||||
address: 0,
|
||||
payload: [1],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
magics,
|
||||
}),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("joins one family's numbered blocks into a region across chunk boundaries", () => {
|
||||
const bytes = patternBytes(1000);
|
||||
const file = uf2File(uf2Image(RP2040_FAMILY, 0x1000_0000, bytes));
|
||||
for (const chunkSize of [file.length, 512, 100, 7]) {
|
||||
const decoded = image(file, chunkSize);
|
||||
expect(decoded.complete).toBe(true);
|
||||
expect(decoded.limitations).toEqual([]);
|
||||
expect(regionFacts(decoded)).toEqual([
|
||||
{
|
||||
familyId: RP2040_FAMILY,
|
||||
familyName: "RP2040",
|
||||
start: 0x1000_0000,
|
||||
end: 0x1000_03e7,
|
||||
length: 1000,
|
||||
blockCount: 4,
|
||||
firstFileBlock: 0,
|
||||
lastFileBlock: 3,
|
||||
},
|
||||
]);
|
||||
expect(Buffer.compare(decoded.regions[0]!.bytes, bytes)).toBe(0);
|
||||
}
|
||||
expect(image(file).metadata).toEqual({
|
||||
kind: "uf2",
|
||||
block_count: 4,
|
||||
trailing_bytes: 0,
|
||||
invalid_block_count: 0,
|
||||
flash_block_count: 4,
|
||||
not_main_flash_block_count: 0,
|
||||
file_container_block_count: 0,
|
||||
md5_block_count: 0,
|
||||
extension_tag_block_count: 0,
|
||||
family_count: 1,
|
||||
family_name_registry: "microsoft/uf2 cf3bd7f9 utils/uf2families.json",
|
||||
numbering_group_count: 1,
|
||||
missing_block_count: 0,
|
||||
ambiguous_numbering_group_count: 0,
|
||||
repeated_block_count: 0,
|
||||
out_of_order_block_count: 0,
|
||||
block_number_out_of_range_count: 0,
|
||||
region_count: 1,
|
||||
data_bytes: 1000,
|
||||
duplicate_bytes: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps gaps between regions and sorts blocks that arrive out of order", () => {
|
||||
const payload = patternBytes(768);
|
||||
const [first, second, third, late] = [0, 1, 2, 3].map((blockNo) =>
|
||||
uf2Block({
|
||||
address: blockNo === 3 ? 0x1000_1000 : 0x1000_0000 + blockNo * 256,
|
||||
payload:
|
||||
blockNo === 3
|
||||
? [0xaa, 0xbb]
|
||||
: payload.subarray(blockNo * 256, (blockNo + 1) * 256),
|
||||
blockNo,
|
||||
numBlocks: 4,
|
||||
familyId: RP2040_FAMILY,
|
||||
}),
|
||||
);
|
||||
const decoded = image(uf2File([late!, third!, first!, second!]));
|
||||
expect(Buffer.compare(decoded.regions[0]!.bytes, payload)).toBe(0);
|
||||
expect(regionFacts(decoded)).toEqual([
|
||||
expect.objectContaining({
|
||||
start: 0x1000_0000,
|
||||
end: 0x1000_02ff,
|
||||
blockCount: 3,
|
||||
firstFileBlock: 1,
|
||||
lastFileBlock: 3,
|
||||
}),
|
||||
expect.objectContaining({
|
||||
start: 0x1000_1000,
|
||||
end: 0x1000_1001,
|
||||
blockCount: 1,
|
||||
firstFileBlock: 0,
|
||||
lastFileBlock: 0,
|
||||
}),
|
||||
]);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
out_of_order_block_count: 2,
|
||||
missing_block_count: 0,
|
||||
region_count: 2,
|
||||
data_bytes: 770,
|
||||
});
|
||||
expect(decoded.complete).toBe(true);
|
||||
});
|
||||
|
||||
it("counts identical rewrites and refuses a conflicting rewrite of one address", () => {
|
||||
const first = uf2Block({
|
||||
address: 0x2000_0000,
|
||||
payload: [1, 2, 3, 4],
|
||||
blockNo: 0,
|
||||
numBlocks: 2,
|
||||
familyId: RP2040_FAMILY,
|
||||
});
|
||||
const same = uf2Block({
|
||||
address: 0x2000_0002,
|
||||
payload: [3, 4, 5],
|
||||
blockNo: 1,
|
||||
numBlocks: 2,
|
||||
familyId: RP2040_FAMILY,
|
||||
});
|
||||
const decoded = image(uf2File([first, same]));
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
duplicate_bytes: 2,
|
||||
data_bytes: 5,
|
||||
});
|
||||
expect(
|
||||
Buffer.compare(decoded.regions[0]!.bytes, Uint8Array.from([1, 2, 3, 4, 5])),
|
||||
).toBe(0);
|
||||
const different = uf2Block({
|
||||
address: 0x2000_0002,
|
||||
payload: [3, 9],
|
||||
blockNo: 1,
|
||||
numBlocks: 2,
|
||||
familyId: RP2040_FAMILY,
|
||||
});
|
||||
expect(decode(uf2File([first, different]))).toEqual({
|
||||
ok: false,
|
||||
error: {
|
||||
kind: "format",
|
||||
message:
|
||||
"family 0xe48bff56 (RP2040) address 0x20000003 is written with different values by block 0 (0x04) and block 1 (0x09)",
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps families apart even when they target the same addresses", () => {
|
||||
// A Pico SDK universal image: an RP2350 absolute block, then RP2040 and RP2350 images.
|
||||
const absolute = uf2Block({
|
||||
address: 0x10ff_ff00,
|
||||
payload: new Uint8Array(256).fill(0xef),
|
||||
blockNo: 0,
|
||||
numBlocks: 2,
|
||||
familyId: RP2XXX_ABSOLUTE_FAMILY,
|
||||
});
|
||||
const decoded = image(
|
||||
uf2File([
|
||||
absolute,
|
||||
...uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(512, 1)),
|
||||
...uf2Image(RP2350_ARM_S_FAMILY, 0x1000_0000, patternBytes(512, 2)),
|
||||
]),
|
||||
);
|
||||
expect(
|
||||
decoded.regions.map(({ familyId, start, end }) => [familyId, start, end]),
|
||||
).toEqual([
|
||||
[RP2040_FAMILY, 0x1000_0000, 0x1000_01ff],
|
||||
[RP2XXX_ABSOLUTE_FAMILY, 0x10ff_ff00, 0x10ff_ffff],
|
||||
[RP2350_ARM_S_FAMILY, 0x1000_0000, 0x1000_01ff],
|
||||
]);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
family_count: 3,
|
||||
numbering_group_count: 3,
|
||||
missing_block_count: 1,
|
||||
duplicate_bytes: 0,
|
||||
});
|
||||
expect(decoded.complete).toBe(false);
|
||||
expect(decoded.limitations).toEqual([
|
||||
"UF2 numbering declares 1 block(s) that are absent: family 0xe48bff57 (RP2XXX_ABSOLUTE) with 2 numbered blocks has 1 of them.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("names the LibreTiny families that the cited registry lists", () => {
|
||||
// LibreTiny also builds LN882H (0xa38090a8), which that registry omits.
|
||||
const families = new Map<number, string | null>([
|
||||
[0x9fff_d543, "RTL8710A"],
|
||||
[0x22e0_d6fc, "RTL8710B"],
|
||||
[0xe08f_7564, "RTL8720C"],
|
||||
[0x3379_cfe2, "RTL8720D"],
|
||||
[0x675a_40b0, "BK7231U"],
|
||||
[0x7b3e_f230, "BK7231N"],
|
||||
[0x6a82_cc42, "BK7251"],
|
||||
[0xde12_70b7, "BL602"],
|
||||
[0xa380_90a8, null],
|
||||
]);
|
||||
const decoded = image(
|
||||
uf2File(
|
||||
[...families.keys()].flatMap((familyId) =>
|
||||
uf2Image(familyId, 0x0, patternBytes(256)),
|
||||
),
|
||||
),
|
||||
);
|
||||
expect(
|
||||
new Map(
|
||||
decoded.regions.map(({ familyId, familyName }) => [familyId, familyName]),
|
||||
),
|
||||
).toEqual(families);
|
||||
});
|
||||
|
||||
it("reports numbered blocks that are absent and leaves coverage unknown", () => {
|
||||
const blocks = uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(1024));
|
||||
const decoded = image(uf2File([blocks[0]!, blocks[1]!, blocks[3]!]));
|
||||
expect(decoded.complete).toBe(false);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
missing_block_count: 1,
|
||||
region_count: 2,
|
||||
});
|
||||
expect(decoded.limitations).toEqual([
|
||||
"UF2 numbering declares 1 block(s) that are absent: family 0xe48bff56 (RP2040) with 4 numbered blocks has 3 of them.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("keeps equal-sized esptool copies complete and an identical resend complete", () => {
|
||||
// esptool numbers each merged file from 0, so equal-sized files reuse numbers.
|
||||
const decoded = image(
|
||||
uf2File([
|
||||
...uf2Image(0xc47e_5767, 0x0, patternBytes(512, 3)),
|
||||
...uf2Image(0xc47e_5767, 0x8000, patternBytes(512, 4)),
|
||||
]),
|
||||
);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
numbering_group_count: 1,
|
||||
repeated_block_count: 2,
|
||||
out_of_order_block_count: 1,
|
||||
missing_block_count: 0,
|
||||
ambiguous_numbering_group_count: 0,
|
||||
family_count: 1,
|
||||
region_count: 2,
|
||||
});
|
||||
expect(decoded.complete).toBe(true);
|
||||
expect(decoded.regions[1]?.familyName).toBe("ESP32S3");
|
||||
// A block sent twice to its own address is one block, not a second copy.
|
||||
const blocks = uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(512));
|
||||
const resent = image(uf2File([blocks[0]!, blocks[1]!, blocks[1]!]));
|
||||
expect(resent.complete).toBe(true);
|
||||
expect(resent.metadata).toMatchObject({
|
||||
repeated_block_count: 1,
|
||||
duplicate_bytes: 256,
|
||||
ambiguous_numbering_group_count: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it("leaves coverage unknown when one copy of reused numbering lost a block", () => {
|
||||
const first = uf2Image(0xc47e_5767, 0x0, patternBytes(768, 5));
|
||||
const second = uf2Image(0xc47e_5767, 0x8000, patternBytes(768, 6));
|
||||
for (const lost of [first[1]!, second[2]!]) {
|
||||
const decoded = image(
|
||||
uf2File([...first, ...second].filter((block) => block !== lost)),
|
||||
);
|
||||
expect(decoded.complete).toBe(false);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
missing_block_count: 0,
|
||||
ambiguous_numbering_group_count: 1,
|
||||
});
|
||||
expect(decoded.limitations).toEqual([
|
||||
"UF2 numbering cannot show that every copy of a block sequence is complete: family 0xc47e5767 (ESP32S3) with 3 numbered blocks has some numbers at 1 address(es) and others at 2.",
|
||||
]);
|
||||
}
|
||||
});
|
||||
|
||||
it("excludes not-main-flash and file-container blocks from regions", () => {
|
||||
const flash = uf2Block({
|
||||
address: 0x0,
|
||||
payload: [1, 2],
|
||||
blockNo: 0,
|
||||
numBlocks: 3,
|
||||
});
|
||||
const comment = uf2Block({
|
||||
address: 0x0,
|
||||
payload: [9, 9, 9],
|
||||
blockNo: 1,
|
||||
numBlocks: 3,
|
||||
flags: UF2_FLAGS.notMainFlash,
|
||||
});
|
||||
const decoded = image(uf2File([flash, comment]));
|
||||
expect(regionFacts(decoded)).toEqual([
|
||||
{
|
||||
familyId: null,
|
||||
familyName: null,
|
||||
start: 0,
|
||||
end: 1,
|
||||
length: 2,
|
||||
blockCount: 1,
|
||||
firstFileBlock: 0,
|
||||
lastFileBlock: 0,
|
||||
},
|
||||
]);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
not_main_flash_block_count: 1,
|
||||
flash_block_count: 1,
|
||||
family_count: 0,
|
||||
missing_block_count: 1,
|
||||
});
|
||||
expect(decoded.limitations).toContain(
|
||||
"1 UF2 block(s) are flagged not main flash (flag 0x1); their 3 payload bytes are in no region.",
|
||||
);
|
||||
const file = uf2Block({
|
||||
address: 0,
|
||||
payload: [0x41],
|
||||
blockNo: 2,
|
||||
numBlocks: 3,
|
||||
fileSize: 1,
|
||||
flags: UF2_FLAGS.fileContainer | UF2_FLAGS.notMainFlash,
|
||||
});
|
||||
const contained = image(uf2File([flash, comment, file]));
|
||||
expect(contained.complete).toBe(false);
|
||||
expect(contained.metadata).toMatchObject({
|
||||
file_container_block_count: 1,
|
||||
not_main_flash_block_count: 1,
|
||||
missing_block_count: 0,
|
||||
region_count: 1,
|
||||
});
|
||||
expect(contained.limitations).toContain(
|
||||
"1 UF2 block(s) belong to file containers (flag 0x1000); their files were not reconstructed.",
|
||||
);
|
||||
});
|
||||
|
||||
it("excludes invalid blocks and a truncated final block, leaving coverage unknown", () => {
|
||||
const blocks = uf2Image(RP2040_FAMILY, 0x1000_0000, patternBytes(768));
|
||||
const badEnd = uf2Block({
|
||||
address: 0x1000_0100,
|
||||
payload: patternBytes(256),
|
||||
blockNo: 1,
|
||||
numBlocks: 3,
|
||||
familyId: RP2040_FAMILY,
|
||||
magics: [0x0a32_4655, 0x9e5d_5157, 0],
|
||||
});
|
||||
const oversized = uf2Block({
|
||||
address: 0x1000_0300,
|
||||
payload: [1],
|
||||
declaredSize: 477,
|
||||
blockNo: 2,
|
||||
numBlocks: 3,
|
||||
familyId: RP2040_FAMILY,
|
||||
});
|
||||
const wrapping = uf2Block({
|
||||
address: 0xffff_ffff,
|
||||
payload: [1, 2],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
});
|
||||
const decoded = image(
|
||||
uf2File([
|
||||
blocks[0]!,
|
||||
badEnd,
|
||||
blocks[2]!,
|
||||
oversized,
|
||||
wrapping,
|
||||
blocks[1]!.subarray(0, 100),
|
||||
]),
|
||||
);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
block_count: 5,
|
||||
trailing_bytes: 100,
|
||||
invalid_block_count: 3,
|
||||
flash_block_count: 2,
|
||||
missing_block_count: 1,
|
||||
region_count: 2,
|
||||
});
|
||||
expect(decoded.complete).toBe(false);
|
||||
expect(decoded.limitations).toEqual([
|
||||
"UF2 file ends with 100 bytes after its last complete 512-byte block; the file may be truncated.",
|
||||
"3 UF2 block(s) are invalid and were excluded; the first is block 1 at byte offset 512: end magic number does not match.",
|
||||
"UF2 numbering declares 1 block(s) that are absent: family 0xe48bff56 (RP2040) with 3 numbered blocks has 2 of them.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports block numbers outside their declared count", () => {
|
||||
const decoded = image(
|
||||
uf2File([
|
||||
uf2Block({ address: 0, payload: [1], blockNo: 1, numBlocks: 1 }),
|
||||
uf2Block({ address: 1, payload: [2], blockNo: 0, numBlocks: 0 }),
|
||||
]),
|
||||
);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
block_number_out_of_range_count: 2,
|
||||
numbering_group_count: 0,
|
||||
region_count: 1,
|
||||
data_bytes: 2,
|
||||
});
|
||||
expect(decoded.complete).toBe(false);
|
||||
});
|
||||
|
||||
it("counts MD5 and extension-tag flags as declared, unverified facts", () => {
|
||||
const decoded = image(
|
||||
uf2File([
|
||||
uf2Block({
|
||||
address: 0,
|
||||
payload: [1],
|
||||
blockNo: 0,
|
||||
numBlocks: 2,
|
||||
flags: UF2_FLAGS.md5,
|
||||
}),
|
||||
uf2Block({
|
||||
address: 1,
|
||||
payload: [2],
|
||||
blockNo: 1,
|
||||
numBlocks: 2,
|
||||
flags: UF2_FLAGS.extensionTags,
|
||||
}),
|
||||
]),
|
||||
);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
md5_block_count: 1,
|
||||
extension_tag_block_count: 1,
|
||||
region_count: 1,
|
||||
});
|
||||
expect(decoded.complete).toBe(true);
|
||||
expect(decoded.limitations).toEqual([
|
||||
"1 UF2 block(s) declare an MD5 flash region (flag 0x4000) and 1 declare extension tags (flag 0x8000); both are counted, not decoded or verified.",
|
||||
]);
|
||||
});
|
||||
|
||||
it("accepts boundary blocks beside the invalid ones", () => {
|
||||
const decoded = image(
|
||||
uf2File([
|
||||
// A full 476-byte payload ends exactly at the top of the address space.
|
||||
uf2Block({
|
||||
address: 0xffff_ffff - 475,
|
||||
payload: patternBytes(476),
|
||||
blockNo: 0,
|
||||
numBlocks: 3,
|
||||
familyId: RP2040_FAMILY,
|
||||
}),
|
||||
// A main-flash block may carry no payload.
|
||||
uf2Block({
|
||||
address: 0x1000_0000,
|
||||
payload: [],
|
||||
blockNo: 1,
|
||||
numBlocks: 3,
|
||||
familyId: RP2040_FAMILY,
|
||||
}),
|
||||
// Blocks without a familyID form their own address space.
|
||||
uf2Block({
|
||||
address: 0xffff_ffff - 475,
|
||||
payload: patternBytes(476, 1),
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
}),
|
||||
uf2Block({
|
||||
address: 0x1000_0000,
|
||||
payload: [7],
|
||||
blockNo: 2,
|
||||
numBlocks: 3,
|
||||
familyId: RP2040_FAMILY,
|
||||
}),
|
||||
]),
|
||||
);
|
||||
expect(decoded.complete).toBe(true);
|
||||
expect(
|
||||
decoded.regions.map(({ familyId, start, end }) => [familyId, start, end]),
|
||||
).toEqual([
|
||||
[null, 0xffff_fe24, 0xffff_ffff],
|
||||
[RP2040_FAMILY, 0x1000_0000, 0x1000_0000],
|
||||
[RP2040_FAMILY, 0xffff_fe24, 0xffff_ffff],
|
||||
]);
|
||||
expect(decoded.metadata).toMatchObject({
|
||||
invalid_block_count: 0,
|
||||
flash_block_count: 4,
|
||||
family_count: 1,
|
||||
numbering_group_count: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses a larger file than its decoder admitted with a typed limit", () => {
|
||||
const block = uf2Block({
|
||||
address: 0,
|
||||
payload: [1],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
});
|
||||
const decoder = createUf2Decoder(511);
|
||||
if (!decoder.ok) throw new Error(decoder.error.message);
|
||||
expect(decoder.value.write(block)).toMatchObject({
|
||||
ok: false,
|
||||
error: { kind: "limit", boundary: "file-size" },
|
||||
});
|
||||
});
|
||||
|
||||
it("refuses files above the fixed ceiling and more regions than retained", () => {
|
||||
expect(createUf2Decoder(UF2_MAX_FILE_BYTES + 1)).toMatchObject({
|
||||
ok: false,
|
||||
error: { kind: "limit", boundary: "file-size" },
|
||||
});
|
||||
const blocks = (count: number) =>
|
||||
Array.from({ length: count }, (_, index) =>
|
||||
uf2Block({ address: index * 2, payload: [1], blockNo: 0, numBlocks: 1 }),
|
||||
);
|
||||
const accepted = decode(uf2File(blocks(65_536)), 512 * 1024);
|
||||
if (!accepted.ok) throw new Error(accepted.error.message);
|
||||
expect(accepted.value.metadata.region_count).toBe(65_536);
|
||||
expect(decode(uf2File(blocks(65_537)), 512 * 1024)).toMatchObject({
|
||||
ok: false,
|
||||
error: { kind: "limit", boundary: "capacity" },
|
||||
});
|
||||
// Runs are not capped: blocks in reverse order still merge into one region.
|
||||
const reversed = Array.from({ length: 70_000 }, (_, index) =>
|
||||
uf2Block({
|
||||
address: (69_999 - index) * 4,
|
||||
payload: [1, 2, 3, 4],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
}),
|
||||
);
|
||||
const merged = decode(uf2File(reversed), 512 * 1024);
|
||||
if (!merged.ok) throw new Error(merged.error.message);
|
||||
expect(merged.value.metadata).toMatchObject({
|
||||
region_count: 1,
|
||||
data_bytes: 280_000,
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,475 @@
|
||||
import type { Uf2Metadata } from "../artifactFormatMetadata.js";
|
||||
import { err, ok, type Result } from "../result.js";
|
||||
import { mergeArenaRuns, type RunConflict } from "./addressRuns.js";
|
||||
import type { RegionDecodeFailure, RegionDecoder } from "./regionDecoder.js";
|
||||
import { UF2_FAMILY_REGISTRY, uf2FamilyName } from "./uf2Families.js";
|
||||
|
||||
/** Every UF2 block is 512 bytes: a 32-byte header, 476 data bytes, an end magic. */
|
||||
export const UF2_BLOCK_BYTES = 512;
|
||||
const UF2_DATA_OFFSET = 32;
|
||||
const UF2_DATA_BYTES = 476;
|
||||
const UF2_MAGIC_START0 = 0x0a32_4655;
|
||||
const UF2_MAGIC_START1 = 0x9e5d_5157;
|
||||
const UF2_MAGIC_END = 0x0ab1_6f30;
|
||||
const UF2_FLAG_NOT_MAIN_FLASH = 0x0000_0001;
|
||||
const UF2_FLAG_FILE_CONTAINER = 0x0000_1000;
|
||||
const UF2_FLAG_FAMILY_ID = 0x0000_2000;
|
||||
const UF2_FLAG_MD5 = 0x0000_4000;
|
||||
const UF2_FLAG_EXTENSION_TAGS = 0x0000_8000;
|
||||
const ADDRESS_SPACE = 0x1_0000_0000;
|
||||
/** Numbering groups named in one limitation before the rest are counted. */
|
||||
const NAMED_GROUPS = 4;
|
||||
|
||||
/**
|
||||
* Fixed whole-file ceiling, as for Intel HEX; configurable ceilings are
|
||||
* tracked in #6176. Decoded bytes stay below 476/512 of it.
|
||||
*/
|
||||
export const UF2_MAX_FILE_BYTES = 128 * 1024 * 1024;
|
||||
/** Regions retained as inventory entries, as for Intel HEX record runs. */
|
||||
export const UF2_MAX_REGIONS = 65_536;
|
||||
|
||||
/** Decoded bytes for one family's address-contiguous range; `end` is inclusive. */
|
||||
export interface Uf2Region {
|
||||
/** The declared familyID, or null when the block did not set flag 0x2000. */
|
||||
readonly familyId: number | null;
|
||||
/** The registry short name REA knows for `familyId`, or null. */
|
||||
readonly familyName: string | null;
|
||||
readonly start: number;
|
||||
readonly end: number;
|
||||
readonly bytes: Uint8Array;
|
||||
readonly blockCount: number;
|
||||
/** Zero-based positions of the contributing blocks in the file. */
|
||||
readonly firstFileBlock: number;
|
||||
readonly lastFileBlock: number;
|
||||
}
|
||||
|
||||
export interface Uf2Image {
|
||||
readonly metadata: Uf2Metadata;
|
||||
/** Ordered by family, with undeclared families first, then by address. */
|
||||
readonly regions: readonly Uf2Region[];
|
||||
readonly complete: boolean;
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
/** Consecutive main-flash blocks of one family whose addresses continue each other. */
|
||||
interface BlockRun {
|
||||
readonly familyId: number | null;
|
||||
readonly address: number;
|
||||
readonly offset: number;
|
||||
length: number;
|
||||
blockCount: number;
|
||||
readonly firstFileBlock: number;
|
||||
lastFileBlock: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Valid blocks sharing a familyID (or none) and a declared block count. A
|
||||
* number seen at several addresses belongs to several copies, as when esptool
|
||||
* numbers each merged file from 0; a repeat at one address is the same block.
|
||||
*/
|
||||
interface NumberingGroup {
|
||||
readonly familyId: number | null;
|
||||
readonly declared: number;
|
||||
/** The first target address seen for each block number. */
|
||||
readonly first: Map<number, number>;
|
||||
/** Further distinct target addresses, only for numbers that have them. */
|
||||
readonly further: Map<number, Set<number>>;
|
||||
previous: number;
|
||||
}
|
||||
|
||||
/** What block numbering establishes about completeness. */
|
||||
interface NumberingFacts {
|
||||
readonly missing: number;
|
||||
readonly absent: readonly string[];
|
||||
readonly ambiguous: readonly string[];
|
||||
}
|
||||
|
||||
const readUint32 = (bytes: Uint8Array, offset: number): number =>
|
||||
((bytes[offset] ?? 0) |
|
||||
((bytes[offset + 1] ?? 0) << 8) |
|
||||
((bytes[offset + 2] ?? 0) << 16) |
|
||||
((bytes[offset + 3] ?? 0) << 24)) >>>
|
||||
0;
|
||||
|
||||
const hex32 = (value: number): string =>
|
||||
`0x${value.toString(16).padStart(8, "0")}`;
|
||||
|
||||
const familyText = (familyId: number | null): string => {
|
||||
if (familyId === null) return "no familyID";
|
||||
const name = uf2FamilyName(familyId);
|
||||
return name === null
|
||||
? `family ${hex32(familyId)}`
|
||||
: `family ${hex32(familyId)} (${name})`;
|
||||
};
|
||||
|
||||
/** At most NAMED_GROUPS descriptions, then a count of the rest. */
|
||||
const named = (items: readonly string[]): string =>
|
||||
`${items.slice(0, NAMED_GROUPS).join("; ")}${items.length > NAMED_GROUPS ? `; and ${String(items.length - NAMED_GROUPS)} more` : ""}`;
|
||||
|
||||
/** Recognize a file only by a complete first block with all three magic numbers. */
|
||||
export const hasUf2Block = (prefix: Uint8Array): boolean =>
|
||||
prefix.length >= UF2_BLOCK_BYTES &&
|
||||
readUint32(prefix, 0) === UF2_MAGIC_START0 &&
|
||||
readUint32(prefix, 4) === UF2_MAGIC_START1 &&
|
||||
readUint32(prefix, UF2_BLOCK_BYTES - 4) === UF2_MAGIC_END;
|
||||
|
||||
/** Start a block decoder whose arena is bounded by the admitted file size. */
|
||||
export const createUf2Decoder = (
|
||||
fileBytes: number,
|
||||
): Result<RegionDecoder<Uf2Image>, RegionDecodeFailure> =>
|
||||
Number.isSafeInteger(fileBytes) &&
|
||||
fileBytes >= 0 &&
|
||||
fileBytes <= UF2_MAX_FILE_BYTES
|
||||
? ok(new StreamingUf2Decoder(fileBytes))
|
||||
: err({
|
||||
kind: "limit",
|
||||
boundary: "file-size",
|
||||
message: `file is ${String(fileBytes)} bytes, exceeding the fixed ${String(UF2_MAX_FILE_BYTES)}-byte (128 MiB) ceiling`,
|
||||
});
|
||||
|
||||
/** Absent numbers, and groups whose numbers appear at unequal address counts. */
|
||||
const numberingFacts = (groups: Iterable<NumberingGroup>): NumberingFacts => {
|
||||
let missing = 0;
|
||||
const absent: string[] = [];
|
||||
const ambiguous: string[] = [];
|
||||
for (const group of groups) {
|
||||
const seen = group.first.size;
|
||||
const name = `${familyText(group.familyId)} with ${String(group.declared)} numbered blocks`;
|
||||
if (seen < group.declared) {
|
||||
missing += group.declared - seen;
|
||||
absent.push(`${name} has ${String(seen)} of them`);
|
||||
}
|
||||
// Every seen number holds one address unless another copy was seen.
|
||||
let most = 1;
|
||||
let fewest = group.further.size < seen ? 1 : Number.MAX_SAFE_INTEGER;
|
||||
for (const further of group.further.values()) {
|
||||
most = Math.max(most, further.size + 1);
|
||||
fewest = Math.min(fewest, further.size + 1);
|
||||
}
|
||||
if (most !== fewest)
|
||||
ambiguous.push(
|
||||
`${name} has some numbers at ${String(fewest)} address(es) and others at ${String(most)}`,
|
||||
);
|
||||
}
|
||||
return { missing, absent, ambiguous };
|
||||
};
|
||||
|
||||
class StreamingUf2Decoder implements RegionDecoder<Uf2Image> {
|
||||
readonly #block = new Uint8Array(UF2_BLOCK_BYTES);
|
||||
readonly #runs: BlockRun[] = [];
|
||||
readonly #groups = new Map<string, NumberingGroup>();
|
||||
readonly #families = new Set<number>();
|
||||
#arena: Uint8Array;
|
||||
#arenaUsed = 0;
|
||||
#blockLength = 0;
|
||||
#blockCount = 0;
|
||||
#invalid = 0;
|
||||
#firstInvalid:
|
||||
| { readonly block: number; readonly reason: string }
|
||||
| undefined;
|
||||
#flash = 0;
|
||||
#notMainFlash = 0;
|
||||
#notMainFlashBytes = 0;
|
||||
#fileContainer = 0;
|
||||
#md5 = 0;
|
||||
#extensionTags = 0;
|
||||
#repeated = 0;
|
||||
#outOfOrder = 0;
|
||||
#outOfRange = 0;
|
||||
|
||||
constructor(fileBytes: number) {
|
||||
this.#arena = new Uint8Array(
|
||||
Math.floor(fileBytes / UF2_BLOCK_BYTES) * UF2_DATA_BYTES,
|
||||
);
|
||||
}
|
||||
|
||||
write(chunk: Uint8Array): Result<void, RegionDecodeFailure> {
|
||||
let from = 0;
|
||||
while (from < chunk.length) {
|
||||
const count = Math.min(
|
||||
UF2_BLOCK_BYTES - this.#blockLength,
|
||||
chunk.length - from,
|
||||
);
|
||||
this.#block.set(chunk.subarray(from, from + count), this.#blockLength);
|
||||
this.#blockLength += count;
|
||||
from += count;
|
||||
if (this.#blockLength < UF2_BLOCK_BYTES) break;
|
||||
this.#blockLength = 0;
|
||||
const decoded = this.#decodeBlock(this.#block);
|
||||
if (!decoded.ok) return decoded;
|
||||
}
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
finish(): Result<Uf2Image, RegionDecodeFailure> {
|
||||
const merged = mergeRuns(this.#runs, this.#arena);
|
||||
this.#arena = new Uint8Array(0);
|
||||
if (!merged.ok) return merged;
|
||||
const { regions, duplicateBytes } = merged.value;
|
||||
if (regions.length > UF2_MAX_REGIONS)
|
||||
return err({
|
||||
kind: "limit",
|
||||
boundary: "capacity",
|
||||
message: `payloads form ${String(regions.length)} address-discontiguous regions, more than the ${String(UF2_MAX_REGIONS)} retained as entries`,
|
||||
});
|
||||
const numbering = numberingFacts(this.#groups.values());
|
||||
const limitations = this.#limitations(numbering);
|
||||
return ok({
|
||||
metadata: this.#metadata(regions, duplicateBytes, numbering),
|
||||
regions,
|
||||
complete: limitations.unknown.length === 0,
|
||||
limitations: [...limitations.unknown, ...limitations.known],
|
||||
});
|
||||
}
|
||||
|
||||
#decodeBlock(block: Uint8Array): Result<void, RegionDecodeFailure> {
|
||||
const index = this.#blockCount;
|
||||
this.#blockCount += 1;
|
||||
const flags = readUint32(block, 8);
|
||||
const address = readUint32(block, 12);
|
||||
const size = readUint32(block, 16);
|
||||
const fault =
|
||||
readUint32(block, 0) !== UF2_MAGIC_START0 ||
|
||||
readUint32(block, 4) !== UF2_MAGIC_START1
|
||||
? "start magic numbers do not match"
|
||||
: readUint32(block, UF2_BLOCK_BYTES - 4) !== UF2_MAGIC_END
|
||||
? "end magic number does not match"
|
||||
: size > UF2_DATA_BYTES
|
||||
? `payload size ${String(size)} exceeds the ${String(UF2_DATA_BYTES)}-byte data area`
|
||||
: address + size > ADDRESS_SPACE
|
||||
? `payload at ${hex32(address)} with ${String(size)} bytes wraps past 0xffffffff`
|
||||
: undefined;
|
||||
if (fault !== undefined) {
|
||||
this.#invalid += 1;
|
||||
this.#firstInvalid ??= { block: index, reason: fault };
|
||||
return ok(undefined);
|
||||
}
|
||||
const familyId =
|
||||
(flags & UF2_FLAG_FAMILY_ID) !== 0 ? readUint32(block, 28) : null;
|
||||
if ((flags & UF2_FLAG_MD5) !== 0) this.#md5 += 1;
|
||||
if ((flags & UF2_FLAG_EXTENSION_TAGS) !== 0) this.#extensionTags += 1;
|
||||
this.#number(
|
||||
familyId,
|
||||
readUint32(block, 24),
|
||||
readUint32(block, 20),
|
||||
address,
|
||||
);
|
||||
// The file-container flag overrides not-main-flash: such blocks hold files.
|
||||
if ((flags & UF2_FLAG_FILE_CONTAINER) !== 0) {
|
||||
this.#fileContainer += 1;
|
||||
return ok(undefined);
|
||||
}
|
||||
if ((flags & UF2_FLAG_NOT_MAIN_FLASH) !== 0) {
|
||||
this.#notMainFlash += 1;
|
||||
this.#notMainFlashBytes += size;
|
||||
return ok(undefined);
|
||||
}
|
||||
this.#flash += 1;
|
||||
if (familyId !== null) this.#families.add(familyId);
|
||||
return this.#addData(
|
||||
index,
|
||||
familyId,
|
||||
address,
|
||||
block.subarray(UF2_DATA_OFFSET, UF2_DATA_OFFSET + size),
|
||||
);
|
||||
}
|
||||
|
||||
/** Numbering is per familyID and block count, as bootloaders ignore other families. */
|
||||
#number(
|
||||
familyId: number | null,
|
||||
declared: number,
|
||||
blockNo: number,
|
||||
address: number,
|
||||
): void {
|
||||
if (declared === 0 || blockNo >= declared) {
|
||||
this.#outOfRange += 1;
|
||||
return;
|
||||
}
|
||||
const key = `${familyId === null ? "none" : String(familyId)}:${String(declared)}`;
|
||||
let group = this.#groups.get(key);
|
||||
if (group === undefined) {
|
||||
group = {
|
||||
familyId,
|
||||
declared,
|
||||
first: new Map(),
|
||||
further: new Map(),
|
||||
previous: -1,
|
||||
};
|
||||
this.#groups.set(key, group);
|
||||
}
|
||||
const first = group.first.get(blockNo);
|
||||
if (first === undefined) group.first.set(blockNo, address);
|
||||
else {
|
||||
this.#repeated += 1;
|
||||
if (first !== address) {
|
||||
let further = group.further.get(blockNo);
|
||||
if (further === undefined) {
|
||||
further = new Set();
|
||||
group.further.set(blockNo, further);
|
||||
}
|
||||
further.add(address);
|
||||
}
|
||||
}
|
||||
if (blockNo <= group.previous) this.#outOfOrder += 1;
|
||||
group.previous = blockNo;
|
||||
}
|
||||
|
||||
#addData(
|
||||
block: number,
|
||||
familyId: number | null,
|
||||
address: number,
|
||||
data: Uint8Array,
|
||||
): Result<void, RegionDecodeFailure> {
|
||||
if (data.length === 0) return ok(undefined);
|
||||
if (this.#arenaUsed + data.length > this.#arena.length)
|
||||
return err({
|
||||
kind: "limit",
|
||||
boundary: "file-size",
|
||||
message: `block ${String(block)}: payload bytes exceed ${String(UF2_DATA_BYTES)} per complete block of the admitted file size`,
|
||||
});
|
||||
const run = this.#runs.at(-1);
|
||||
if (
|
||||
run !== undefined &&
|
||||
run.familyId === familyId &&
|
||||
run.address + run.length === address
|
||||
) {
|
||||
run.length += data.length;
|
||||
run.blockCount += 1;
|
||||
run.lastFileBlock = block;
|
||||
} else
|
||||
this.#runs.push({
|
||||
familyId,
|
||||
address,
|
||||
offset: this.#arenaUsed,
|
||||
length: data.length,
|
||||
blockCount: 1,
|
||||
firstFileBlock: block,
|
||||
lastFileBlock: block,
|
||||
});
|
||||
this.#arena.set(data, this.#arenaUsed);
|
||||
this.#arenaUsed += data.length;
|
||||
return ok(undefined);
|
||||
}
|
||||
|
||||
/** Limitations that leave coverage unknown, then facts that do not. */
|
||||
#limitations(numbering: NumberingFacts): {
|
||||
readonly unknown: string[];
|
||||
readonly known: string[];
|
||||
} {
|
||||
const unknown: string[] = [];
|
||||
const known: string[] = [];
|
||||
if (this.#blockLength > 0)
|
||||
unknown.push(
|
||||
`UF2 file ends with ${String(this.#blockLength)} bytes after its last complete 512-byte block; the file may be truncated.`,
|
||||
);
|
||||
if (this.#firstInvalid !== undefined)
|
||||
unknown.push(
|
||||
`${String(this.#invalid)} UF2 block(s) are invalid and were excluded; the first is block ${String(this.#firstInvalid.block)} at byte offset ${String(this.#firstInvalid.block * UF2_BLOCK_BYTES)}: ${this.#firstInvalid.reason}.`,
|
||||
);
|
||||
if (numbering.missing > 0)
|
||||
unknown.push(
|
||||
`UF2 numbering declares ${String(numbering.missing)} block(s) that are absent: ${named(numbering.absent)}.`,
|
||||
);
|
||||
if (numbering.ambiguous.length > 0)
|
||||
unknown.push(
|
||||
`UF2 numbering cannot show that every copy of a block sequence is complete: ${named(numbering.ambiguous)}.`,
|
||||
);
|
||||
if (this.#outOfRange > 0)
|
||||
unknown.push(
|
||||
`${String(this.#outOfRange)} UF2 block(s) have a block number at or above their block count, so their numbering cannot establish completeness.`,
|
||||
);
|
||||
if (this.#fileContainer > 0)
|
||||
unknown.push(
|
||||
`${String(this.#fileContainer)} UF2 block(s) belong to file containers (flag 0x1000); their files were not reconstructed.`,
|
||||
);
|
||||
if (this.#notMainFlash > 0)
|
||||
known.push(
|
||||
`${String(this.#notMainFlash)} UF2 block(s) are flagged not main flash (flag 0x1); their ${String(this.#notMainFlashBytes)} payload bytes are in no region.`,
|
||||
);
|
||||
if (this.#md5 > 0 || this.#extensionTags > 0)
|
||||
known.push(
|
||||
`${String(this.#md5)} UF2 block(s) declare an MD5 flash region (flag 0x4000) and ${String(this.#extensionTags)} declare extension tags (flag 0x8000); both are counted, not decoded or verified.`,
|
||||
);
|
||||
return { unknown, known };
|
||||
}
|
||||
|
||||
#metadata(
|
||||
regions: readonly Uf2Region[],
|
||||
duplicateBytes: number,
|
||||
numbering: NumberingFacts,
|
||||
): Uf2Metadata {
|
||||
return {
|
||||
kind: "uf2",
|
||||
block_count: this.#blockCount,
|
||||
trailing_bytes: this.#blockLength,
|
||||
invalid_block_count: this.#invalid,
|
||||
flash_block_count: this.#flash,
|
||||
not_main_flash_block_count: this.#notMainFlash,
|
||||
file_container_block_count: this.#fileContainer,
|
||||
md5_block_count: this.#md5,
|
||||
extension_tag_block_count: this.#extensionTags,
|
||||
family_count: this.#families.size,
|
||||
family_name_registry: UF2_FAMILY_REGISTRY,
|
||||
numbering_group_count: this.#groups.size,
|
||||
missing_block_count: numbering.missing,
|
||||
ambiguous_numbering_group_count: numbering.ambiguous.length,
|
||||
repeated_block_count: this.#repeated,
|
||||
out_of_order_block_count: this.#outOfOrder,
|
||||
block_number_out_of_range_count: this.#outOfRange,
|
||||
region_count: regions.length,
|
||||
data_bytes: regions.reduce((sum, region) => sum + region.bytes.length, 0),
|
||||
duplicate_bytes: duplicateBytes,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
const blockSpan = (run: BlockRun): string =>
|
||||
run.firstFileBlock === run.lastFileBlock
|
||||
? `block ${String(run.firstFileBlock)}`
|
||||
: `blocks ${String(run.firstFileBlock)}-${String(run.lastFileBlock)}`;
|
||||
|
||||
/** Merge each family's runs by address; overlaps must repeat the same bytes. */
|
||||
const mergeRuns = (
|
||||
runs: readonly BlockRun[],
|
||||
arena: Uint8Array,
|
||||
): Result<
|
||||
{ readonly regions: Uf2Region[]; readonly duplicateBytes: number },
|
||||
RegionDecodeFailure
|
||||
> => {
|
||||
// Undeclared families sort first; other families never merge with them.
|
||||
const merged = mergeArenaRuns(runs, arena, (run) => run.familyId ?? -1);
|
||||
if (!merged.ok) return err(conflict(merged.error));
|
||||
return ok({
|
||||
regions: merged.value.ranges.map(({ start, bytes, runs: group }) => {
|
||||
const familyId = group[0]?.familyId ?? null;
|
||||
return {
|
||||
familyId,
|
||||
familyName: familyId === null ? null : uf2FamilyName(familyId),
|
||||
start,
|
||||
end: start + bytes.length - 1,
|
||||
bytes,
|
||||
blockCount: group.reduce((sum, run) => sum + run.blockCount, 0),
|
||||
firstFileBlock: group.reduce(
|
||||
(block, run) => Math.min(block, run.firstFileBlock),
|
||||
Number.MAX_SAFE_INTEGER,
|
||||
),
|
||||
lastFileBlock: group.reduce(
|
||||
(block, run) => Math.max(block, run.lastFileBlock),
|
||||
0,
|
||||
),
|
||||
};
|
||||
}),
|
||||
duplicateBytes: merged.value.duplicateBytes,
|
||||
});
|
||||
};
|
||||
|
||||
const conflict = ({
|
||||
address,
|
||||
writer,
|
||||
run,
|
||||
existing,
|
||||
incoming,
|
||||
}: RunConflict<BlockRun>): RegionDecodeFailure => ({
|
||||
kind: "format",
|
||||
message: `${familyText(run.familyId)} address ${hex32(address)} is written with different values by ${blockSpan(writer)} (0x${existing.toString(16).padStart(2, "0")}) and ${blockSpan(run)} (0x${incoming.toString(16).padStart(2, "0")})`,
|
||||
});
|
||||
@@ -0,0 +1,53 @@
|
||||
/**
|
||||
* UF2 family IDs that REA names, copied from Microsoft's registry
|
||||
* `utils/uf2families.json` at microsoft/uf2
|
||||
* `cf3bd7f9c274b949528f8cd6a19eef2502a687d2` (2026-01-26, MIT licence; file
|
||||
* SHA-256 `9d8f561e050507c5909063005cd6f5b157e6fe6db1e8d25b5d9027d57064c194`).
|
||||
*
|
||||
* The subset covers the families named by REA's embedded work. The Espressif
|
||||
* IDs equal esptool v4.12.0 and v5.6.0 `UF2_FAMILY_ID` values. Any other ID is
|
||||
* reported by value only; a missing name never means an unknown family is
|
||||
* invalid.
|
||||
*/
|
||||
/** The registry revision the names below come from, recorded with each inventory. */
|
||||
export const UF2_FAMILY_REGISTRY =
|
||||
"microsoft/uf2 cf3bd7f9 utils/uf2families.json";
|
||||
|
||||
const UF2_FAMILY_NAMES: ReadonlyMap<number, string> = new Map([
|
||||
[0x1c5f_21b0, "ESP32"],
|
||||
[0xbfdd_4eee, "ESP32S2"],
|
||||
[0xc47e_5767, "ESP32S3"],
|
||||
[0x3101_f7c1, "ESP32S31"],
|
||||
[0x2b88_d29c, "ESP32C2"],
|
||||
[0xd42b_a06c, "ESP32C3"],
|
||||
[0xf71c_0343, "ESP32C5"],
|
||||
[0x540d_df62, "ESP32C6"],
|
||||
[0x77d8_50c4, "ESP32C61"],
|
||||
[0x3327_26f6, "ESP32H2"],
|
||||
[0xb6dd_00af, "ESP32H21"],
|
||||
[0x9e0b_aa8a, "ESP32H4"],
|
||||
[0x3d30_8e94, "ESP32P4"],
|
||||
[0x7eab_61ed, "ESP8266"],
|
||||
[0xe48b_ff56, "RP2040"],
|
||||
[0xe48b_ff57, "RP2XXX_ABSOLUTE"],
|
||||
[0xe48b_ff58, "RP2XXX_DATA"],
|
||||
[0xe48b_ff59, "RP2350_ARM_S"],
|
||||
[0xe48b_ff5a, "RP2350_RISCV"],
|
||||
[0xe48b_ff5b, "RP2350_ARM_NS"],
|
||||
[0x5775_5a57, "STM32F4"],
|
||||
[0xada5_2840, "NRF52840"],
|
||||
[0x68ed_2b88, "SAMD21"],
|
||||
[0x5511_4460, "SAMD51"],
|
||||
[0x9fff_d543, "RTL8710A"],
|
||||
[0x22e0_d6fc, "RTL8710B"],
|
||||
[0xe08f_7564, "RTL8720C"],
|
||||
[0x3379_cfe2, "RTL8720D"],
|
||||
[0x675a_40b0, "BK7231U"],
|
||||
[0x7b3e_f230, "BK7231N"],
|
||||
[0x6a82_cc42, "BK7251"],
|
||||
[0xde12_70b7, "BL602"],
|
||||
]);
|
||||
|
||||
/** The registry short name for a family ID, or null when REA does not name it. */
|
||||
export const uf2FamilyName = (familyId: number): string | null =>
|
||||
UF2_FAMILY_NAMES.get(familyId) ?? null;
|
||||
@@ -79,6 +79,8 @@ export const liveBinaryTargetSummaryVariants = [
|
||||
"intel-hex",
|
||||
"srec",
|
||||
"esp-image",
|
||||
"uf2",
|
||||
"dfuse",
|
||||
"fit",
|
||||
]),
|
||||
architecture: z.null(),
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
import { readFile, readdir, rm, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect, onTestFinished } from "vitest";
|
||||
import { cliTest } from "../support/cli/cliFixture.js";
|
||||
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
|
||||
import { parseEvidence } from "../../src/domain/evidence.js";
|
||||
import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js";
|
||||
import {
|
||||
artifactExtractionResultSchema,
|
||||
artifactInventoryResultSchema,
|
||||
} from "../../src/domain/artifactGraph.js";
|
||||
import { dfuseFile } from "../../src/domain/firmware/dfuse.fixture.js";
|
||||
import {
|
||||
RP2040_FAMILY,
|
||||
RP2350_ARM_S_FAMILY,
|
||||
patternBytes,
|
||||
uf2Block,
|
||||
uf2File,
|
||||
uf2Image,
|
||||
} from "../../src/domain/firmware/uf2.fixture.js";
|
||||
|
||||
const RP2040_APP = patternBytes(300, 1);
|
||||
const RP2350_APP = patternBytes(300, 2);
|
||||
const FLASH = Uint8Array.from([1, 2, 3, 4]);
|
||||
const OPTION_BYTES = Uint8Array.from([0xaa, 0x55]);
|
||||
|
||||
const cases = [
|
||||
{
|
||||
format: "uf2",
|
||||
name: "universal.uf2",
|
||||
bytes: uf2File([
|
||||
...uf2Image(RP2040_FAMILY, 0x1000_0000, RP2040_APP),
|
||||
...uf2Image(RP2350_ARM_S_FAMILY, 0x1000_0000, RP2350_APP),
|
||||
]),
|
||||
root: { kind: "uf2", family_count: 2, region_count: 2 },
|
||||
regions: {
|
||||
"family-0xe48bff56/0x10000000-0x1000012b.bin": RP2040_APP,
|
||||
"family-0xe48bff59/0x10000000-0x1000012b.bin": RP2350_APP,
|
||||
},
|
||||
},
|
||||
{
|
||||
format: "dfuse",
|
||||
name: "firmware.dfu",
|
||||
bytes: dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
name: "Internal Flash",
|
||||
elements: [{ address: 0x0800_0000, data: FLASH }],
|
||||
},
|
||||
{
|
||||
alternateSetting: 1,
|
||||
name: "Option Bytes",
|
||||
elements: [{ address: 0x1fff_c000, data: OPTION_BYTES }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
root: { kind: "dfuse", target_count: 2, element_count: 2 },
|
||||
regions: {
|
||||
"target-0/element-0-0x08000000-0x08000003.bin": FLASH,
|
||||
"target-1/element-0-0x1fffc000-0x1fffc001.bin": OPTION_BYTES,
|
||||
},
|
||||
},
|
||||
] as const;
|
||||
|
||||
const graphFrom = (json: unknown) =>
|
||||
artifactInventoryResultSchema.parse(
|
||||
artifactInspectionResultSchema.parse(parseEvidence(json).normalized_result)
|
||||
.substeps[0]?.evidence.normalized_result,
|
||||
);
|
||||
|
||||
cliTest.for(cases)(
|
||||
"inventories and extracts $format regions through the compiled CLI",
|
||||
async ({ format, name, bytes, root, regions }, { cli }) => {
|
||||
const directory = await createTestTempDirectory("rea-firmware-cli-");
|
||||
const source = join(directory, name);
|
||||
await writeFile(source, bytes);
|
||||
const inspected = await cli.run({
|
||||
arguments: ["inspect-artifact", source, "--json"],
|
||||
});
|
||||
expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0);
|
||||
const inspection = artifactInspectionResultSchema.parse(
|
||||
parseEvidence(inspected.json).normalized_result,
|
||||
);
|
||||
const graph = graphFrom(inspected.json);
|
||||
expect(graph.manifest.root_format).toBe(format);
|
||||
expect(graph.coverage.status).toBe("complete");
|
||||
expect(
|
||||
graph.occurrences.find((row) => row.logical_path === ".")
|
||||
?.format_metadata,
|
||||
).toMatchObject(root);
|
||||
expect(
|
||||
graph.occurrences
|
||||
.filter((row) => row.logical_path !== ".")
|
||||
.map((row) => row.logical_path),
|
||||
).toEqual(Object.keys(regions));
|
||||
expect(inspection.next_probes.map((probe) => probe.operation)).toEqual([
|
||||
"open_binary",
|
||||
]);
|
||||
const extracted = await cli.run({
|
||||
arguments: ["extract-artifact", source, "--json"],
|
||||
environment: { TMPDIR: directory, TMP: directory, TEMP: directory },
|
||||
});
|
||||
expect(extracted.exitCode, extracted.stdout + extracted.stderr).toBe(0);
|
||||
const result = artifactExtractionResultSchema.parse(
|
||||
parseEvidence(extracted.json).normalized_result,
|
||||
);
|
||||
onTestFinished(() =>
|
||||
rm(result.output_root, { recursive: true, force: true }),
|
||||
);
|
||||
expect(result.containment_verified).toBe(true);
|
||||
for (const [path, expected] of Object.entries(regions))
|
||||
expect(
|
||||
Buffer.compare(
|
||||
await readFile(join(result.output_root, path)),
|
||||
expected,
|
||||
),
|
||||
).toBe(0);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"applies strict and record-and-continue policies to a DfuSe suffix CRC",
|
||||
async ({ cli }) => {
|
||||
const directory = await createTestTempDirectory("rea-dfuse-crc-cli-");
|
||||
const source = join(directory, "firmware.dfu");
|
||||
const valid = dfuseFile({
|
||||
targets: [
|
||||
{ alternateSetting: 0, elements: [{ address: 0, data: FLASH }] },
|
||||
],
|
||||
});
|
||||
const observed = valid.readUInt32LE(valid.length - 4);
|
||||
const corrupt = Buffer.from(valid);
|
||||
corrupt.writeUInt32LE(0x1234_5678, corrupt.length - 4);
|
||||
await writeFile(source, corrupt);
|
||||
const rejected = await cli.run({
|
||||
arguments: ["inspect-artifact", source, "--json"],
|
||||
});
|
||||
expect(rejected.exitCode).toBe(1);
|
||||
expect(rejected.json).toMatchObject({
|
||||
code: "artifact_integrity_mismatch",
|
||||
details: {
|
||||
logical_path: ".",
|
||||
algorithm: "dfu-crc32",
|
||||
declared_crc32: "12345678",
|
||||
calculated_crc32: observed.toString(16).padStart(8, "0"),
|
||||
unpacked: false,
|
||||
},
|
||||
});
|
||||
const inspected = await cli.run({
|
||||
arguments: [
|
||||
"inspect-artifact",
|
||||
source,
|
||||
"--integrity-policy",
|
||||
"record-and-continue",
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0);
|
||||
expect(graphFrom(inspected.json).integrity_contradictions).toEqual([
|
||||
expect.objectContaining({
|
||||
logical_path: ".",
|
||||
algorithm: "dfu-crc32",
|
||||
declared_crc32: "12345678",
|
||||
observed_crc32: observed.toString(16).padStart(8, "0"),
|
||||
trust: "observed-untrusted",
|
||||
}),
|
||||
]);
|
||||
// Extraction applies the same default policy and leaves no output behind.
|
||||
const refused = await cli.run({
|
||||
arguments: ["extract-artifact", source, "--json"],
|
||||
environment: { TMPDIR: directory, TMP: directory, TEMP: directory },
|
||||
});
|
||||
expect(refused.exitCode).toBe(1);
|
||||
expect(refused.json).toMatchObject({
|
||||
code: "artifact_integrity_mismatch",
|
||||
details: { logical_path: ".", algorithm: "dfu-crc32" },
|
||||
});
|
||||
expect(await readdir(directory)).toEqual(["firmware.dfu"]);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"refuses a conflicting UF2 rewrite through the CLI",
|
||||
async ({ cli }) => {
|
||||
const directory = await createTestTempDirectory("rea-uf2-conflict-cli-");
|
||||
const source = join(directory, "firmware.uf2");
|
||||
await writeFile(
|
||||
source,
|
||||
uf2File([
|
||||
...uf2Image(RP2040_FAMILY, 0x1000_0000, RP2040_APP),
|
||||
uf2Block({
|
||||
address: 0x1000_0000,
|
||||
payload: [0xff],
|
||||
blockNo: 0,
|
||||
numBlocks: 1,
|
||||
familyId: RP2040_FAMILY,
|
||||
}),
|
||||
]),
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-artifact", source, "--json"],
|
||||
});
|
||||
expect(result.exitCode).toBe(1);
|
||||
expect(result.stdout + result.stderr).toContain(
|
||||
"UF2 is malformed: family 0xe48bff56 (RP2040) address 0x10000000 is written with different values by blocks 0-1 (0x01) and block 2 (0xff)",
|
||||
);
|
||||
},
|
||||
);
|
||||
@@ -41,6 +41,12 @@ import {
|
||||
} from "../../fixtures/espImage.js";
|
||||
import { espFlashDump } from "../../../src/domain/firmware/espFlash.fixture.js";
|
||||
import { javaClassFile } from "../../fixtures/javaClassFile.js";
|
||||
import { dfuseFile } from "../../../src/domain/firmware/dfuse.fixture.js";
|
||||
import {
|
||||
patternBytes,
|
||||
uf2File,
|
||||
uf2Image,
|
||||
} from "../../../src/domain/firmware/uf2.fixture.js";
|
||||
import {
|
||||
MAX_JVM_CLASS_IDENTITY_BYTES,
|
||||
JVM_CLASS_IDENTITY_LIMIT_REASON,
|
||||
@@ -603,6 +609,47 @@ describe("binary target I/O: image classification", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("binary target I/O: firmware containers", () => {
|
||||
it("classifies UF2 by block magics and DfuSe by its prefix and suffix", async () => {
|
||||
const directory = await createTestTempDirectory("rea-firmware-target-");
|
||||
const uf2 = uf2File(uf2Image(0xe48b_ff56, 0x1000_0000, patternBytes(300)));
|
||||
const dfu = dfuseFile({
|
||||
targets: [{ alternateSetting: 0, elements: [{ address: 0, data: [1] }] }],
|
||||
});
|
||||
const paths = {
|
||||
uf2: join(directory, "firmware.bin"),
|
||||
dfuse: join(directory, "firmware.img"),
|
||||
badCrc: join(directory, "bad-crc.dfu"),
|
||||
shortUf2: join(directory, "short.uf2"),
|
||||
noSuffix: join(directory, "no-suffix.dfu"),
|
||||
};
|
||||
// The suffix CRC is checked by inventory, not by recognition.
|
||||
const badCrc = Buffer.from(dfu);
|
||||
badCrc.writeUInt32LE(0x1234_5678, badCrc.length - 4);
|
||||
await Promise.all([
|
||||
writeFile(paths.uf2, uf2),
|
||||
writeFile(paths.dfuse, dfu),
|
||||
writeFile(paths.badCrc, badCrc),
|
||||
writeFile(paths.shortUf2, uf2.subarray(0, 511)),
|
||||
writeFile(paths.noSuffix, dfu.subarray(0, dfu.length - 16)),
|
||||
]);
|
||||
for (const [path, format] of [
|
||||
[paths.uf2, "uf2"],
|
||||
[paths.dfuse, "dfuse"],
|
||||
[paths.badCrc, "dfuse"],
|
||||
] as const)
|
||||
expect(await parseBinaryTarget(path)).toMatchObject({
|
||||
ok: true,
|
||||
value: { kind: "archive", format },
|
||||
});
|
||||
for (const path of [paths.shortUf2, paths.noSuffix])
|
||||
expect(await parseBinaryTarget(path)).toMatchObject({
|
||||
ok: false,
|
||||
error: { _tag: "BinaryTargetError" },
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("binary target I/O: ESP flash layouts", () => {
|
||||
it("detects a flash layout before the bootloader it begins with", async () => {
|
||||
const directory = await createTestTempDirectory("rea-esp-flash-target-");
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
import { readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import { z } from "zod";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
import { parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import {
|
||||
artifactExtractionResultSchema,
|
||||
artifactInventoryResultSchema,
|
||||
} from "../../../src/domain/artifactGraph.js";
|
||||
import { artifactInspectionResultSchema } from "../../../src/domain/artifactInspection.js";
|
||||
import { dfuseFile } from "../../../src/domain/firmware/dfuse.fixture.js";
|
||||
import {
|
||||
RP2040_FAMILY,
|
||||
RP2XXX_ABSOLUTE_FAMILY,
|
||||
patternBytes,
|
||||
uf2Block,
|
||||
uf2File,
|
||||
uf2Image,
|
||||
} from "../../../src/domain/firmware/uf2.fixture.js";
|
||||
|
||||
const APP = patternBytes(300, 9);
|
||||
const REGION = "family-0xe48bff56/0x10000000-0x1000012b.bin";
|
||||
|
||||
// The compiled stdio fixture checks advertised schemas against their dialect and
|
||||
// validates actual tool responses, avoiding a second source server module graph.
|
||||
it.each(["full", "compact"] as const)(
|
||||
"exposes UF2 facts, retained regions and extraction through MCP (%s)",
|
||||
async (profile) => {
|
||||
const root = await createTestTempDirectory("rea-uf2-mcp-");
|
||||
const path = join(root, "firmware.uf2");
|
||||
await writeFile(
|
||||
path,
|
||||
uf2File([
|
||||
uf2Block({
|
||||
address: 0x10ff_ff00,
|
||||
payload: new Uint8Array(256).fill(0xef),
|
||||
blockNo: 0,
|
||||
numBlocks: 2,
|
||||
familyId: RP2XXX_ABSOLUTE_FAMILY,
|
||||
}),
|
||||
...uf2Image(RP2040_FAMILY, 0x1000_0000, APP),
|
||||
]),
|
||||
);
|
||||
const { call } = await connectLocalToolsMcp({
|
||||
REA_MCP_INPUT_SCHEMA_PROFILE: profile,
|
||||
});
|
||||
let outputRoot: string | undefined;
|
||||
try {
|
||||
const opened = await call("open_binary", { path });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const reply = await call("inspect_artifact", { detail: "summary" });
|
||||
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
|
||||
const summary = z
|
||||
.object({
|
||||
parent_evidence_id: z.string(),
|
||||
summary: z.object({
|
||||
root_format_metadata: z.unknown(),
|
||||
source_coverage: z.unknown(),
|
||||
}),
|
||||
})
|
||||
.parse(parseEvidence(reply.structuredContent).normalized_result);
|
||||
expect(summary.summary.root_format_metadata).toMatchObject({
|
||||
kind: "uf2",
|
||||
family_count: 2,
|
||||
missing_block_count: 1,
|
||||
region_count: 2,
|
||||
});
|
||||
expect(summary.summary.source_coverage).toMatchObject({
|
||||
status: "unknown",
|
||||
});
|
||||
const retained = await call("inspect_analysis_view", {
|
||||
source: {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: summary.parent_evidence_id,
|
||||
},
|
||||
view: {
|
||||
kind: "inventory",
|
||||
collection: "occurrences",
|
||||
logical_paths: [REGION],
|
||||
fields: ["format_metadata"],
|
||||
},
|
||||
});
|
||||
expect(retained.isError, JSON.stringify(retained)).not.toBe(true);
|
||||
expect(JSON.stringify(retained.structuredContent)).toContain(
|
||||
'"family_name":"RP2040"',
|
||||
);
|
||||
const extracted = await call("extract_artifact", {});
|
||||
expect(extracted.isError, JSON.stringify(extracted)).not.toBe(true);
|
||||
const result = artifactExtractionResultSchema.parse(
|
||||
parseEvidence(extracted.structuredContent).normalized_result,
|
||||
);
|
||||
outputRoot = result.output_root;
|
||||
expect(
|
||||
Buffer.compare(await readFile(join(result.output_root, REGION)), APP),
|
||||
).toBe(0);
|
||||
await call("close_binary", {});
|
||||
} finally {
|
||||
if (outputRoot !== undefined)
|
||||
await rm(outputRoot, { recursive: true, force: true });
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("delivers a DfuSe CRC contradiction under both integrity policies through MCP", async () => {
|
||||
const root = await createTestTempDirectory("rea-dfuse-mcp-");
|
||||
const path = join(root, "firmware.dfu");
|
||||
const valid = dfuseFile({
|
||||
targets: [
|
||||
{
|
||||
alternateSetting: 0,
|
||||
name: "Internal Flash",
|
||||
elements: [{ address: 0x0800_0000, data: [1, 2, 3, 4] }],
|
||||
},
|
||||
],
|
||||
});
|
||||
const corrupt = Buffer.from(valid);
|
||||
corrupt.writeUInt32LE(0, corrupt.length - 4);
|
||||
await writeFile(path, corrupt);
|
||||
const { call } = await connectLocalToolsMcp({});
|
||||
const opened = await call("open_binary", { path });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const rejected = parseMcpToolError(await call("inspect_artifact", {}));
|
||||
expect(rejected.error).toMatchObject({
|
||||
code: "artifact_integrity_mismatch",
|
||||
details: {
|
||||
logical_path: ".",
|
||||
algorithm: "dfu-crc32",
|
||||
declared_crc32: "00000000",
|
||||
calculated_crc32: valid
|
||||
.readUInt32LE(valid.length - 4)
|
||||
.toString(16)
|
||||
.padStart(8, "0"),
|
||||
},
|
||||
});
|
||||
const recorded = await call("inspect_artifact", {
|
||||
integrity_policy: "record-and-continue",
|
||||
});
|
||||
expect(recorded.isError, JSON.stringify(recorded)).not.toBe(true);
|
||||
const inspection = artifactInspectionResultSchema.parse(
|
||||
parseEvidence(recorded.structuredContent).normalized_result,
|
||||
);
|
||||
expect(inspection.contradictions).toEqual([
|
||||
expect.objectContaining({
|
||||
algorithm: "dfu-crc32",
|
||||
declared_crc32: "00000000",
|
||||
}),
|
||||
]);
|
||||
const graph = artifactInventoryResultSchema.parse(
|
||||
inspection.substeps[0]?.evidence.normalized_result,
|
||||
);
|
||||
expect(
|
||||
graph.occurrences.find((row) => row.logical_path === ".")?.hash_status,
|
||||
).toBe("mismatched");
|
||||
await call("close_binary", {});
|
||||
});
|
||||
Reference in new issue
Block a user