dilim/rea
1
0 Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-06 05:17:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(artifacts): inventory and extract ESP flash layouts (#8228)

Browse Source 
* feat(artifacts): inventory and extract ESP32 firmware images

ESP32-family app and bootloader images (magic 0xE9) had no reader, so
open_binary refused OTA downloads, app partitions and bootloaders.

Add the esp-image container format behind inspect_artifact and
extract_artifact. A pure decoder reads each byte once and keeps only
fixed-size fields. Derived facts cover the chip, from a table cited to
ESP-IDF v6.1; segment classification as ESP-IDF's should_map and
should_load; checksum and appended SHA-256 results; app and bootloader
descriptors; Secure Boot v2 signature blocks, observed and unverified;
and undescribed trailing byte ranges. Truncation and undescribed bytes
make coverage unknown. ESP8266 images are refused with a specific
reason, and ciphertext stays unrecognized.

Add the format to every target, Evidence, snapshot, live-summary and
artifact vocabulary, document it, and add an optional verify:esp-images
lane that compares esptool-built images with esptool image-info and
espsecure signature-info-v2.

Refs #3615

* test(artifacts): run the ESP image lane on esptool v5.6.0

The lane compared descriptor fields only when esptool image-info printed
one, so a descriptor that REA reports and esptool does not print passed
without comparison. Compare the descriptor kind in both directions.

Record that the lane passed for all 13 images with esptool and espsecure
v5.6.0, the version the layout facts cite, as well as with 4.12.0.

* refactor(artifacts): define ESP image layout sizes once

The decoder and the derived facts each defined the segment-header,
appended-digest and flash-sector sizes, the erased-byte scan and the
offset formatter. Define them once with the other fixed-layout structures
and import them in both places. The signature slot count is the one
constant that drives the slot scan.

Correct the observedEnd comment: it is the end of the last chunk read.

* fix(artifacts): report ESP-IDF's segment length limit by chip

ESP-IDF's verify_segment_header rejects a segment of 16 MiB or more
unless the chip maps flash through 32-bit cache addresses
(SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP: ESP32-S3, P4, C5, C61, H4 and
S31 in v6.1). Record that capability in the chip table and add a
limitation for such segments; unknown chips get none.

`loaded` repeats should_load. It does not mean the bootloader accepts the
segment: verify_load_addresses still rejects one outside the chip's RAM,
which REA does not model. Say so in the docs, the skill reference and the
inventory limitation.

* test(artifacts): check ESP lane tools first and verify extraction

The lane started work before checking its tools, so a missing RISC-V
target, ld.lld, esptool or espsecure surfaced as a raw spawn or command
error. Check each one before building and name the variable to set. Warn
when esptool is not v5.6.0, the release the layout facts cite.

Assert each segment's class for the ESP32-C3 app, its variants and the
bootloader, as the Xtensa image already did. Extract the app and the
signed image through the CLI and compare every file with the bytes at
esptool's segment offsets or in the appended signature sector; the
source's SHA-256 must be unchanged and equal the reported root_sha256.

* docs(artifacts): state what the ESP image lane does not establish

The Verification section listed the lane's checks but not its gaps: only
ESP32-C3 and ESP32 images come from a real producer, no real project
build, OTA download or flash dump is inspected, no signature is verified,
and the lane does not run in CI.

Record the timing of the final lane run on esptool v5.6.0.

* fix(artifacts): compare ESP segment declarations

Segment entries hash only their data, so compare_artifacts reported a
segment that moved to another load address with the same bytes as
unchanged; only the image itself changed. Compare each segment's
load_addr and data_len as its declaration, as Node SEA and PyInstaller
members are compared, and count a changed declaration as a metadata
change, also when content is unavailable.

Header offsets are layout and the classification is derived, so neither
is compared.

* test(artifacts): run ESP lane commands as owned process trees

The lane ran its tools through execFile. A timeout killed only the direct
child, so a linker or assembler that Clang or GCC had started could keep
running while the lane removed its workspace. Run every command through
createVerifierCommands, check the tools inside withVerifierCleanup, and
remove the workspace only after commands.close() has stopped every tree;
a failed close keeps the workspace.

Only a nonzero exit now counts as a refusal or an expected esptool
failure; timeouts and spawn failures propagate. Supervising each command
roughly doubles the lane's run time.

* test(artifacts): cancel ESP decoding after its first decoded chunk

The cancellation test requested cancel or close on setImmediate, which
can land while the file is still being opened and checked, so it never
showed that decoding had started. It also passed against a reader that
ignored both requests while decoding. Acknowledge the decoder's first
chunk, then cancel or close, join the operation, and assert that no
later chunk reaches the decoder and that the reader ends closed.

* docs(artifacts): record which ESP metadata comparison ignores

Segments are compared by their declaration. The image's and the
signature blocks' format_metadata are not: every field comes from their
own bytes, which content comparison already covers, and a block's
image_digest_matches follows the image's content.

* feat(artifacts): inventory and extract ESP flash layouts

A whole-flash dump of an ESP32-family chip was read as its bootloader
image, with the partition table, apps and data left as undescribed bytes,
or refused when it began with erased flash. Read it as an esp-flash-image
container instead.

The partition table is decoded as esp_partition_table_verify accepts it
and is found at the caller's partition_table_offset, at the default
0x8000, or by scanning 0x1000-aligned sectors for a table whose MD5
verifies, which the inventory reports as derived. The bootloader is
located at its chip's ROM bootloader offset. Each partition is an entry
with its declaration in flat format_metadata; the bootloader and app
partitions carry ESP image facts from the existing streaming decoder; the
otadata records are reported with a selection derived without rollback or
anti-rollback, and the running partition stays unknown. One unreadable
partition does not hide the others, and extraction writes every partition,
nvs_keys included, as ordinary data.

Flash layouts are detected before single ESP images. The scan runs only
when an ESP image sits at a ROM bootloader offset, and is refused when a
bootloader at another chip's offset shows that the dump does not start at
flash address 0. open_binary gains flash_offset and partition_table_offset,
which require format esp-flash-image and are part of the snapshot target
identity; the CLI takes them with --target-format esp-flash-image.
Comparison treats a partition's label, type, subtype, range and flags as
its declaration.

* test(artifacts): verify ESP flash layouts with gen_esp32part, merge-bin and otatool

Add the optional verify:esp-flash lane. It builds default and relocated
partition tables with ESP-IDF's gen_esp32part.py, assembles dumps with
esptool merge-bin, including one written from flash 0x1000, and writes
OTA records with otatool through flash_file_tool.py, which replaces
parttool's esptool calls with reads and writes of the dump file.

It compares REA's bootloader and app facts with esptool image-info, every
partition with gen_esp32part.py's decoding of the extracted table, and the
OTA records and derived slot with otatool read_otadata after each switch.
It also checks extracted bytes and source integrity, the scan and caller
table offsets, flash_offset on the 0x1000 dump, a corrupted app beside
valid partitions and a truncated dump. ESP32 linker scripts place the
conformance bootloader and app in ESP32 address ranges.

* fix(artifacts): read ESP flash layouts as the bootloader does

Read the partition table as bootloader_utility_load_partition_table
does: the last factory, otadata and same-subtype OTA declaration wins,
and a limitation names repeats. Plan each otadata record within its
partition and the dump. An otadata partition smaller than two sectors
selects none, the bootloader's INVALID_INDEX; a record the dump lacks
leaves the selection unknown while the other record stays reported. The
fallbacks follow the bootloader's indexes: ota_0-fallback and
factory-fallback name no partition when the table declares none, and the
assumptions add GPIO holds and deep-sleep wake.

Refuse a dump whose bootloader contradicts its origin before the default
table offset is tried, whether flash_offset was omitted or given, and keep
the ESP image reader's ESP8266 refusal. A table never lies inside the
located bootloader image, so single app images are no longer scanned. An
explicit esp-flash-image selection is decided before other container
signatures, and the native commands' --target-format no longer takes it.
Say when no bootloader confirms the default origin.

Exempt the primary bootloader and partition_table entries of ESP-IDF 5.3
from placement limits, report every region inside a longer one, and name
partitions by index when labels repeat ignoring case. Keep an undecodable
label's bytes in label_hex and compare them. Word undescribed ranges as
starting at a byte that is not erased, and treat a short erased region as
erased.

Add an MCP compare_artifacts test for partition declarations, reader
identity and close tests, scan edge cases, the 0x2000 ROM offset and a
literal otatool CRC.

* test(artifacts): verify ESP primary layout entries and the otadata fallback

Build a table with the primary bootloader and partition_table entries
that gen_esp32part.py fills from --primary-bootloader-offset, and check
that REA reads it with complete coverage, no overlap or misplacement
limitation, and a table that agrees with gen_esp32part's decoding. Erase
the otadata with otatool after the slot switches and check the records
against read_otadata and the factory fallback.

* fix(artifacts): keep ESP flash coverage complete for a dump cut short

An esptool merge-bin dump ends after the last part it wrote, usually inside
its last partition. Such a partition made the layout's coverage unknown, so
artifact comparison of two merge-bin dumps reported every unchanged
partition as unknown.

The partition table names every partition whatever the file's length, so a
partition cut short or left out by the end of the file hides no entry. As
the ESP image reader does for a cut-short segment, its entry holds the bytes
the file has and an entry limitation gives the declared size; a partition
with no bytes stays in partitions_outside_dump. Coverage stays complete
unless undescribed bytes remain or no bootloader is located. An MCP
compare_artifacts test of two dumps that end inside ota_0 now classifies
every partition, and fails on the old rule.

* fix(artifacts): report written bytes in ESP partitions, erased slots included

An app slot is erased for ESP-IDF's bootloader when its image header bytes
are erased flash, and REA reported only that status. A slot can still hold
data further in: a dump read from a device held a complete older app image
1 MiB into an OTA slot reported as erased, with no limitation.

Each partition entry now records where the bytes the dump holds are
written, relative to the partition: written_offset, the first byte that is
not 0xFF, written_end, just past the last, and written_bytes, how many are
not 0xFF. They are null for a partition outside the dump. The status stays
erased, which is the bootloader's view, and a limitation names an erased
app slot that holds written bytes and says they are not read as an image.
The reader reads every partition byte once more to find them.

* fix(artifacts): derive the ESP boot selection for every partition table

The boot selection lived inside otadata, so a table without an otadata
partition, the common factory-only layout, had none. ESP-IDF's
bootloader_utility_get_selected_boot_partition returns the factory index
when the table declares no otadata partition.

Move it to a top-level boot_selection that every layout has. Its basis
names the bootloader's branch: otadata, factory-fallback and ota_0-fallback
from the records, no-otadata for a table without otadata, none for the
bootloader's INVALID_INDEX, and unknown when the dump lacks a record the
bootloader reads, which replaces the null selection. running_partition
moves with it, and otadata keeps its partition index and records. The
assumptions name what each branch leaves out: no records assumption
without otadata, and none for none or unknown. The real-tool lane checks
no-otadata on a table gen_esp32part built without otadata.

* fix(artifacts): say when an ESP image is a flash dump read from 0x1000

An ESP32 dump read from flash 0x1000, as read-flash 0x1000 or merge-bin
--target-offset 0x1000 writes it, opened automatically as its bootloader
image with a megabyte of undescribed bytes and no next probe. The flash
reader refuses that origin, but nothing told an agent which flash_offset to
pass.

espFlashDumpOrigin owns the inference: an image of a chip whose ROM loads
the bootloader from flash R, at file offset F, is the bootloader of a dump
read from R - F when a partition table sits where that origin puts 0x8000.
The ESP image reader asks it for a non-app image at file offset 0, so the
result does not depend on how the reader was created, and records
flash_dump {flash_offset, table_file_offset}, null otherwise, with a
limitation that says to reopen the file as esp-flash-image with that
flash_offset. inspect_artifact proposes the same open_binary call as a next
probe, and the flash reader's misplaced-origin refusal names the offset.

* perf(artifacts): summarize nested ESP images in the flash layout summary view

The summary view returned an ESP flash layout's root facts whole, so each
app partition and the bootloader carried every segment and signature
block, about half of a dump's summary.

The summary view now gives each nested image a summary instead of its
facts: status, chip, entry point, segment count, load ranges (each mapped,
loaded or unclassified segment's address and declared length), the
checksum and appended SHA-256 statuses, coverage and limitations. Complete
detail keeps the full facts, every segment included, and the retained
Evidence is unchanged. On a 16 MiB device dump the summary's root falls
from 5,141 to 2,657 bytes, and the summary Evidence from 10,038 to 7,554.

* test(artifacts): expect the misplaced image the ESP bootloader search reports

8a748d3cc added misplacedImage to the bootloader search, which the
locator uses to name the origin a misplaced bootloader implies, but one
test still compared the search result without it. Expect the image's
chip and file offset, and no image for an unknown chip.

* fix(artifacts): give every ESP partition from the dump's start an entry

After ecbc1a772 kept coverage complete for a dump cut short, a partition
wholly past the end of the file still had no entry, only a
partitions_outside_dump row. Comparing a merge-bin dump with a full device
read then reported that partition as added or removed, as if one table
did not declare it.

Every partition from the dump's first byte on is now an entry holding the
bytes the file has: none for one past the end, with a limitation saying
the file holds none of its bytes, no image facts and null written fields.
partitions_outside_dump keeps only partitions that begin before the dump's
first byte, and coverage is unknown when one does, since it has no entry.
Two dumps that lack the same partition now compare it as unchanged, and a
dump against a longer one reports the partition as changed; an MCP test
of a merge-bin dump against a full one checks that no path is added or
removed.

Unlike the ESP image reader, which marks a truncated image's coverage
unknown because later segments and signature blocks may be hidden, the
flash reader can list every partition from the table whatever the file's
length; only the entry limitation follows the image reader.

* fix(artifacts): select no ESP app from a partition table the bootloader rejects

boot_selection followed bootloader_utility_get_selected_boot_partition but
not the table check before it. For a table whose MD5 does not match, or
that has a second MD5 entry, esp_partition_table_verify fails, the
bootloader selects no app and resets. REA still derived a slot from the
otadata records, although the table's own limitation said the bootloader
rejects it.

The selection now has basis none, with no assumptions, for a table the
decoder marks invalid. It also follows two more branches of the
bootloader: an otadata partition at offset 0, which the bootloader takes
for none, selects the factory index as no-otadata, and a table without an
OTA app selects its fallback even when the dump lacks a record, since the
bootloader then ignores the records.

The bootloader also rejects a table whose partitions end past the flash
size in its header. REA does not model that check; the format guide says
so.

* fix(artifacts): require the ESP dump origin's table to follow the image

espFlashDumpOrigin named a later flash origin whenever a partition table
decoded where that origin puts 0x8000. Bytes inside the image's own
segments could be read that way, so an ESP32 bootloader whose segment data
held table-shaped bytes at file offset 0x7000 was reported as a dump read
from 0x1000.

The table must now start at or after the image's end, the rule the locator
already applies to the default table. The ESP image reader passes the
image_end it decoded, or the file size for a truncated image; the flash
reader's refusal passes the end of the misplaced image's segments. The
bootloader search drops its misplaced flag, which only restated
misplacedImage, and the next probe formats offsets with offsetText.

Reader tests cover the 0x1000 and 0x2000 ROM offsets, a standalone
bootloader, bytes that begin like a table, an app with a table at 0x7000
and a table inside a segment; without the boundary check, three of them
fail.

* fix(artifacts): give the ESP flash summary its own kind and each image's identity

The summary view's ESP flash root kept kind esp-flash-image, so a consumer
bound to kind could not tell an image's summary from its full facts, and
the advertised schema held two objects with the same kind const. It is now
esp-flash-image-summary.

The per-image summary also dropped the descriptor, which is what tells two
app slots apart, and named each image only by its partition index, which
the summary view, having no entry rows, could not resolve. Each nested
image now carries its entry path from the layout plan, in both views. The
summary adds the descriptor kind with its identifying fields (an app's
project_name, version, idf_ver, secure_version and app_elf_sha256; a
bootloader's version, idf_ver and secure_version), the signature block
count and each load range's classification. Its enums and fields derive
from the schemas that own them.

An MCP test reads a dump with an app in ota_0 and an erased ota_1 that
holds older bytes: it checks both summaries, that the summary's inventory
Evidence is the one the complete view returns with full facts, and the
erased slot's written range.

* test(artifacts): cover cancel and close during the ESP written-bytes scan

The written-bytes pass reads every byte of each partition the dump holds,
checking that the reader is open before each chunk, but only app decoding
had a cancellation and close test. A data partition, which only that pass
reads, now holds 9 MiB; once the scan has examined its first chunk, a
cancel rejects preparation with the caller's reason and a close with
unavailable, and the scan examines no further chunk.

The erased-slot test now asserts the whole set of erased-slot limitations,
so factory, ota_0 and nvs, which the bootloader reads or which are not app
slots, are shown to have none.

* docs(artifacts): re-time the ESP flash lane and state only observed causes

The lane ran again after its assertions changed; both esptool runs passed,
and the timed run's figures replace the earlier ones. The erased-slot
paragraph no longer names a cause for the older bytes, which REA cannot
observe, and the skill reference states that every table has a
boot_selection in its own sentence, with the none basis for a table the
bootloader rejects.

* fix(artifacts): retire ESP flash members as the ESP image reader does

#7498's maintainer commits made the ESP image reader keep a member owned
until its consumer closes it, retire fully buffered bytes when
cancellation arrives after production, and refuse to finish an inventory
paused across close. The ESP flash reader was copied from the earlier
reader. Main's shared stored-member lifecycle cases failed two of eight
against it: a paused inventory completed after close, and buffered member
bytes stayed readable after cancellation.

Give the flash reader the same lifecycle and run the shared cases against
it.

* docs(artifacts): re-time the ESP flash lane after merging main

The lane passed again with esptool v5.6.0 and 4.12.0 on the merge of upstream main. Record the v5.6.0 run's time and peak resident set.

* fix(artifacts): read ESP image Evidence recorded before flash_dump

The flash layout work made `flash_dump` a required field of the
`esp-image` root metadata. Evidence recorded before it has no such
field, so newer REA rejected records that were valid when they were
made: `compare_artifacts` and `inspect_analysis_view` reported
`evidence_integrity_mismatch` for them.

The field is now optional when read. A record without it parses as it
is, so the graph digest it commits to still matches; nothing fills the
field in. The image reader still writes it for every image, and the
inspection probe reads a missing field as no hint.

A regression test recommits an image inventory without the field and
passes it inline to `compare_artifacts` and `inspect_analysis_view`.
Against a server built with the required field, `compare_artifacts`
rejects it.

* fix(artifacts): stop the ESP flash gap scan when the reader closes

The scan for the first written byte in each undescribed gap checked
only the caller's signal. `close()` marks the reader closed and waits
for preparation, so it waited while the rest of a long erased gap was
read.

The gap scan now takes the reader's open check, as the written-bytes
scan and the image decoder do, and checks it before each chunk. Tests
cancel or close the reader after the scan's first chunk of a 9 MiB
erased gap and confirm that no later chunk is examined.

* fix(artifacts): name ESP partitions with Windows device labels by index

The safe-label rule accepted labels such as `NUL`, `con` or `COM1`, so
extraction wrote `partitions/NUL.bin`. Windows reserves those names
before any extension, and ordinary Windows tools cannot use such a
file.

A label that is a Windows device name (`CON`, `PRN`, `AUX`, `NUL`,
`COM0`-`COM9` and `LPT0`-`LPT9`, any case, alone or before a dot) now
takes the existing index fallback. The partition metadata and the
entry's source path keep the label.

* fix(artifacts): read ESP flash partitions' written bytes in one pass

Each partition got its own scan of the bytes the dump holds, so
partitions declared over the same bytes read them again and again.
Overlaps only add limitations, so a crafted table reached this normally:
94 overlapping partitions over a 32 MiB dump would examine about 3 GiB
before the first entry.

The reader now reads the bytes the partitions cover once. Partition
boundaries cut them into segments; one pass counts each segment's
written bytes and bounds, and each partition's extent is the sum of the
segments inside it. Every partition keeps its own declaration, extent
and limitations. The pass checks the reader's open state before each
chunk, as the per-partition scan did.

A test declares 60 partitions over one MiB, 40 over all of it and 20
over its first half, and checks each partition's extent and the
overlap limitations. The reader reads 1,113,256 bytes of the
1,114,112-byte dump; one scan per partition read 52,493,480.

* test(artifacts): fail the ESP lanes on failed commands and refusals

`otatool` turned a nonzero exit into returned text, which its callers
discarded, and `inspect()` returned `{ refused }`, which the flash lane
never checked. A failed command then surfaced later as a slot
assertion, an output-parser error or a TypeError, hiding its cause.

Commands the lanes need to succeed now reject with the output they
exited with: `otatool` and `inspect()` do so through a shared
`failedCommand`. An expected refusal is opt-in: `refusal()` returns
REA's output and fails the lane when REA opens the file. The ESP image
lane's two ESP8266 cases use it.

Checked by calling `otatool` on a file without a table, `inspect()` on a
file REA refuses and `refusal()` on a file REA opens; each rejects with
its cause.

* fix(artifacts): require a partition entry in an ESP partition table

The decoder counted the slot, not the partitions, so a lone MD5 entry
over the empty prefix, followed by the terminator, decoded as a table
with no partitions. Default and caller-selected detection then accepted
that empty layout, although the guide requires one partition.

A table now needs one partition entry before its terminator. An MD5
entry alone is refused as `entry 1 at 0x20 ends the table before any
partition entry`. The cases cover the decoder and both automatic and
caller-selected location.

* fix(artifacts): decode an image shared by ESP app partitions once

Each app partition got its own decode, so app partitions declared over
the same bytes decoded and hashed the same image again and again, the
same pattern the written-bytes scan had. With 60 app partitions over
one 960 KiB image the reader read 64,029,736 bytes of a 1,114,112-byte
dump.

Regions with the same file range now share one decode; each keeps its
own copy of the facts. Partitions with different ranges still decode
separately, because truncation can change the facts: at most one decode
per partition the table holds, each within its partition.

A test declares 60 app partitions over one image, 40 holding all of it
and 20 cutting it short, and expects three decodes: the bootloader, the
whole image and the cut one. The 60-partition dump now reads
2,161,864 bytes, as a dump with one app partition does.
This commit is contained in:
Entrpi authored and GitHub committed 2026-10-31 15:24:32 +08:00
1 parent 091892dfed
commit e6d0618979
69 files changed
+7447 -131
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md
+26
View File
@@ -713,6 +713,32 @@ recognized. Segments are not disassembled, and RISC-V or Xtensa segments
cannot be opened as raw-binary. See
[ESP images](https://github.com/morluto/rea/blob/main/docs/esp-image.md).
## ESP flash images
A whole-flash dump opens as `esp-flash-image`. `root_format_metadata` holds
`partition_table` (its `method`: `default`, `caller` or a derived
`scan`), `bootloader` and `apps` ESP image facts, the `otadata` records,
`partitions_outside_dump` and `boot_selection`. Every table has a
`boot_selection`: its `basis` names the bootloader's branch (`no-otadata`
without otadata, `none` when the bootloader rejects the table, `unknown`
when a record is missing), and it lists its `assumptions`;
`running_partition` is always `unknown`. Each image names its entry
`path`. With `detail: "summary"` the root kind is `esp-flash-image-summary`
and each image is a compact `summary` (chip, entry point, segment count,
classified load ranges, signature block count, descriptor identity,
integrity); complete detail keeps every segment. Partition rows
(`partitions/<label>.bin`) carry the declared `type`, `subtype`, names,
`flash_address`, `size` and flags; `encrypted` is a storage flag, not
proof of ciphertext. `written_offset`/`written_end` bound the bytes that
are not erased: an app slot whose facts say `erased` can still hold an older
image, and a limitation says so. For a dump read from another flash address, call
`open_binary` with `format: "esp-flash-image"` and `flash_offset`; an
`esp-image` whose `flash_dump` is set names that offset in a next probe; a
relocated table can be named with `partition_table_offset`. No table found
means unknown, often flash encryption. Extraction writes `nvs_keys` as
ordinary data. See
[ESP flash images](https://github.com/morluto/rea/blob/main/docs/esp-flash-image.md).
## Bun standalone executables
A `bun build --compile` executable stores its module graph in Mach-O
docs/cli.md
+12
View File
@@ -281,6 +281,18 @@ The inventory lists each segment with its ESP-IDF load classification, the
checksum and SHA-256 results, the descriptor and any Secure Boot v2 signature
blocks. See [ESP images](esp-image.md) for recognition, coverage and refusals.
Inspect an ESP flash dump, or one read from a flash address other than 0:
```sh
rea inspect-artifact flash.bin --json
rea inspect-artifact dump.bin --target-format esp-flash-image --flash-offset 0x1000 --json
rea extract-artifact flash.bin --json
```
The inventory lists the bootloader, the partition table and each partition,
with ESP image facts for app partitions and the OTA records. See
[ESP flash images](esp-flash-image.md) for table location, inputs and limits.
Inspect a Node.js SEA carrier without running it:
```sh
docs/esp-flash-image.md
+348
View File
@@ -0,0 +1,348 @@
# ESP flash images
REA reads a dump of an ESP32-family SPI flash, such as the output of
`esptool read-flash 0 ALL dump.bin` or `esptool merge-bin`, through
`inspect_artifact` and `extract_artifact`. The reader is static: no device,
esptool or ESP-IDF is needed, and no code is run.
The container format is `esp-flash-image`. Entries are the bootloader region,
the partition table sector and one file per partition. The bootloader and
each app partition are also decoded as [ESP images](esp-image.md), so their
headers, segments, integrity and descriptors appear in the root
`format_metadata`. Layout facts are cited to ESP-IDF v6.1
(`esp_flash_partitions.h`, `flash_partitions.c`, `gen_esp32part.py`,
`bootloader_common_loader.c`, `bootloader_utility.c` and each chip's
`esp_rom_caps.h`).
## Recognition
A file is a flash image when REA finds a partition table in the entry
structure ESP-IDF's bootloader verifies (`esp_partition_table_verify`):
32-byte `0x50AA` entries, at most one `0xEBEB` MD5 entry over the entries
before it, and a terminating entry within `0xC00` bytes. ESP-IDF accepts a
table with no partition entries, but REA requires one, so that neither
erased flash nor a lone MD5 entry is read as a table. A table whose MD5 does not match, or that has a second MD5 entry, is
still read, with `md5.status` `invalid` and a limitation, because the
bootloader would refuse it. The table is located in this order:
1. at the caller's `partition_table_offset`, when given; no scan follows;
2. at the default `0x8000`;
3. by scanning every `0x1000`-aligned flash sector for a table whose MD5
verifies. This location is derived, and the inventory says so.
A table never lies inside the located bootloader image, so the default
offset and the scan are tried only after the image's last segment. Without
a caller selection, the scan runs only when an ESP image sits at a ROM
bootloader offset, and a single app or bootloader image ends at its last
segment, so neither ordinary files nor single images are scanned. A C3 or
S3 dump begins with its bootloader, so flash layouts are detected before a
file is read as one ESP image.
Unless the caller names the table with `partition_table_offset`, REA
refuses two kinds of dump before looking for a table:
- an image of a known chip at another chip's ROM offset, such as an ESP32
bootloader at the start of a dump read from `0x1000`. The dump's origin
is not the flash address REA assumed, given or default, and a table found
there would place every partition at the wrong file offset; pass the
`flash_offset` where the dump starts. When a partition table sits where
the image's own ROM offset puts `0x8000`, after the image's segments, the
refusal names that offset.
Without a selected format such a dump opens as its bootloader image, whose
[ESP image](esp-image.md) facts say how to reopen it.
- an ESP8266 image at a ROM offset. ESP8266 dumps can hold the same table
format, and they keep the [ESP image](esp-image.md) reader's ESP8266
refusal.
When no table is found the layout is not recognized. This is an unknown
outcome, not a malformed file: flash encryption, which encrypts the
bootloader and the table, is one cause. When a table is found without a
located bootloader and `flash_offset` is omitted, a limitation says that
each partition's file offset assumes the dump starts at flash `0x0`.
## Caller inputs
| Input | Meaning |
| ------------------------- | ----------------------------------------------------------------------------- |
| `format: esp-flash-image` | Read the file as a flash dump; with no table found, refuse it |
| `flash_offset` | Flash address (`0x…`) of the dump's first byte; default 0 |
| `partition_table_offset` | Flash address (`0x…`, a multiple of `0x1000`) of the table; disables the scan |
MCP takes these on `open_binary`; the CLI takes `--target-format
esp-flash-image`, `--flash-offset` and `--partition-table-offset` on
`inspect-artifact` and `extract-artifact`. The offsets require the format, and
the format takes no `architecture` or `entry_point`. The inputs are part of
the target's identity, so reopening the same bytes with other inputs reads
them again instead of reusing the earlier reading. The `open_binary` summary
reports them as `flash_offset` and `partition_table_offset`, null when
omitted.
## Layout facts
The root `format_metadata` (`esp-flash-image`) records:
- `flash_offset`: its `value` and whether the `caller` gave it or it is the
`default`;
- `chip`: from the bootloader's `chip_id`, or null without a located
bootloader;
- `bootloader`: the region from the chip's ROM bootloader offset to the
table, with its ESP image facts;
- `partition_table`: its flash address, the `method` that found it
(`default`, `caller` or `scan`), the partition count and the MD5 `status`
(`valid`, `invalid` or `absent`) with the stored and computed digests;
- `apps`: ESP image facts for each app partition the dump holds; app
partitions declared over the same bytes share one decode;
- `otadata`: the otadata partition's records, or null without one;
- `boot_selection`: the app the bootloader would select, for every table;
- `partitions_outside_dump`: declared partitions that begin before the
dump's first byte, which have no entry;
- `undescribed`: ranges outside every region, each from a byte that is not
erased flash to the end of its gap.
The ROM loads the bootloader from `ESP_ROM_BOOTLOADER_OFFSET_FLASH`:
`0x1000` for ESP32 and ESP32-S2; `0x0` for ESP32-S3, C2, C3, C6, C61, H2 and
H21; and `0x2000` for ESP32-C5, H4, P4 and S31. A bootloader is located only
where its own `chip_id` says the ROM loads it.
### Partitions
Each partition entry is `partitions/<label>.bin` when the label is a safe
name of up to 16 letters, digits, `.`, `_` and `-` that starts with a
letter, digit or underscore and has at least one letter or underscore, is
not a Windows device name such as `NUL` or `COM1`, alone or before a dot,
and no other label repeats it, ignoring case as some file systems do; it is
`partitions/<index>.bin` otherwise. `source_path` keeps the label, the
table's own name for the partition. Its `format_metadata`
(`esp-flash-partition`) records the table `index`, `label` (null when not
UTF-8), `label_hex` (the label's bytes when it is not UTF-8, else null),
`type`, `subtype`, ESP-IDF's `type_name` and `subtype_name`,
`flash_address`, `size`, the `encrypted` and `readonly` flags and any
`unknown_flags`. The `encrypted` flag says how a device with flash
encryption stores the partition; it does not establish that the dump's
bytes are ciphertext.
Each partition entry also records where the bytes the dump holds are
written, relative to the partition's start: `written_offset`, the first
byte that is not erased flash (`0xFF`), `written_end`, just past the last
such byte, and `written_bytes`, how many bytes are not `0xFF`. A partition
erased throughout has null offsets and `written_bytes` `0`; a partition the
file holds none of has all three null. The range bounds the written
bytes and does not say whether erased flash lies between them; extract the
partition to read them. REA reads the bytes the partitions cover once, so
overlapping partitions share that read however many the table declares.
ESP-IDF 5.3 and later may declare primary `bootloader` and `partition_table`
entries over the bootloader region and the table sector. They are
partitions like any other, whose bytes are also `bootloader.bin` and
`partition-table.bin`, and they are exempt from the rule that partitions
follow the table, as in `gen_esp32part.py`.
Every partition from the dump's first byte on is an entry holding the bytes
the file has. One that the file holds only in part, as when an `esptool
merge-bin` dump ends inside its last partition, has an entry limitation that
gives the size the table declares; one that lies past the end of the file is
an entry with no bytes and a limitation, and has no image facts. A partition
that begins before the dump's first byte has no entry and is listed in
`partitions_outside_dump`. One unreadable partition does not hide the
others: partitions outside the dump or overlapping another are each reported
while the rest stay readable.
### App partitions and the bootloader
Each app partition and the bootloader region carry their entry's `path`
and ESP image `facts`:
`status` is `esp-image`, `erased` (the image header bytes are erased flash,
as in an empty OTA slot) or `unrecognized` (ciphertext or other data). For an
image they hold the [ESP image](esp-image.md) root facts, `segments`,
`signature_blocks`, `coverage` and `limitations`, with offsets relative to
the region. Extract a partition and inspect it as an ESP image for its
segments as entries.
The summary view (`inspect_artifact` with `detail: "summary"`, or the
inventory summary of `inspect_analysis_view`) reports the root facts under
their own kind, `esp-flash-image-summary`, and gives each of these images a
`summary` instead of its `facts`: `status`, `chip`, `entry_addr`,
`segment_count`, `load_ranges` (the load address, declared length and
classification of each segment that is mapped, loaded or unclassified),
`signature_block_count`, the `descriptor` kind with the fields that tell
builds apart (an app's `project_name`, `version`, `idf_ver`,
`secure_version` and `app_elf_sha256`; a bootloader's `version`, `idf_ver`
and `secure_version`), the `checksum` and `appended_sha256` statuses,
`coverage` and `limitations`. The default complete detail keeps the full
facts, every segment included, and the summary's `inventory_evidence_ids`
name the Evidence that holds them.
`erased` is the bootloader's view: with no image header at the start of the
slot, it finds no app there. An erased slot can still hold data further in,
even a complete app image. When it does, its entry's `written_offset` and
`written_end` say where, and a limitation says that the slot holds bytes the
bootloader does not read; REA does not look for images in them.
### OTA data
The otadata partition holds two `esp_ota_select_entry_t` records, one at the
start of each 4 KiB sector. Like the bootloader, REA reads the last otadata
partition the table declares, and the last factory app and OTA app of each
subtype; a limitation names any repeated declaration. REA reports each
record the partition and the dump hold: its `ota_seq`, `seq_label`,
`ota_state` and its name, `crc`, whether the CRC matches (CRC-32 of
`ota_seq` seeded with `0xFFFFFFFF`) and whether the bootloader would treat
it as valid.
### Boot selection
`boot_selection` is present for every table. It is derived as
`bootloader_utility_get_selected_boot_partition` does without app rollback
or anti-rollback, and `basis` names its branch:
- `otadata`: the valid record with the higher `ota_seq` (record 0 on a tie)
selects slot `(ota_seq - 1) % ota_app_count`, indexed by OTA subtype;
- `factory-fallback`: no record is usable, or the table has no OTA app, and
a factory app is declared; or records are set but none is valid, as after
a CRC mismatch, which selects the factory index whether or not a factory
app is declared;
- `ota_0-fallback`: no record is usable, or the table has no OTA app, and no
factory app is declared, so the bootloader selects boot index 0;
- `no-otadata`: the table declares no otadata partition, or one at offset 0,
which the bootloader takes for none, so it selects the factory index, as in
the common factory-only layout;
- `none`: the bootloader rejects the table, for an MD5 mismatch or a second
MD5 entry, or cannot read an otadata partition smaller than two 4 KiB
sectors, and resets without selecting an app;
- `unknown`: the dump lacks an otadata record the bootloader reads.
Without an OTA app the records do not matter, so a dump that lacks them
still selects a fallback. REA does not compare the partitions with the
flash size in the bootloader header, which the bootloader also checks.
`active_record` and `slot` are set for `otadata` and `slot` for
`ota_0-fallback`. `partition_index` is null when the table declares no
partition for the selected slot or fallback; the bootloader then tries the
other apps. `assumptions` list what the derivation leaves out: records used
as written (for the branches that read them), anti-rollback, load fallback,
and factory-reset and test-app GPIO holds and a wake from deep sleep; they
are empty for `none` and `unknown`. `running_partition` is always
`unknown`: at boot the bootloader may first mark a pending-verify record
aborted, check the app's secure version against eFuse, and fall back to
another partition when an app fails to load.
## Coverage and extraction
Erased flash (`0xFF`) between regions is described. In a gap between
regions, the first byte that is not erased starts an `undescribed` range to
the end of the gap, and makes coverage `unknown`; the rest of that range is
not examined. Coverage is also `unknown` when no bootloader is located, or
when a declared partition begins before the dump's first byte, since it has
no entry. A partition cut short or left out by the end of the file does not
make it `unknown`: its entry holds the bytes the file has, none for one past
the end, and its limitation says what the file lacks. Coverage describes the
inventory of the file, not the observation of every declared flash byte.
So artifact comparison classifies every partition entry of two dumps. Where
both lack the same bytes, as two `merge-bin` dumps of the same layout do,
the entries are equal; a dump against a longer one reports each partition
it cuts short or lacks as `changed`, and that entry's limitation says the
file did not hold the bytes.
`extract_artifact` writes every entry as an ordinary file, `nvs_keys`
partitions included. On a device without flash encryption an `nvs_keys`
partition holds the NVS encryption keys in plain text, and a dump of it
contains them. REA decrypts nothing, recovers no keys and does not interpret
NVS.
## Comparison
Artifact comparison uses each partition's `label`, `label_hex`, `type`,
`subtype`, `flash_address`, `size` and flags as its declaration, so a
partition that moves, changes type or flags, or is renamed with unchanged
bytes is `changed` in `metadata`. Partitions pair by path. The root facts
are not compared as a declaration: they derive from the dump's bytes, which
content comparison covers, and from the caller's `flash_offset` and
`partition_table_offset`, which are part of the target's identity and
decide each partition's byte range.
## Limits
- A flash dump inside another container, such as a ZIP, is classified from
its first bytes and so reads as an ESP image, not as a flash layout; extract
it first.
- ESP8266 dumps are refused; Secure Boot v1 bootloader digests and
flash-encrypted layouts are not read.
- Partitions are not checked against the flash chip size, which the
bootloader also verifies.
- NVS, FAT, SPIFFS and other data partitions are extracted, not decoded.
- An image nested in a partition is decoded for its facts; its segments are
not inventory entries.
## Verification
Synthetic fixtures with computed MD5s and CRCs, and one CRC otatool wrote,
cover the table rules, every way the table is located, the scan's chunk and
sector edges, the misplaced-origin guard with the origin omitted or given,
the ESP8266 refusal, the `0x0`, `0x1000` and `0x2000` ROM bootloader
offsets, single images that are not scanned, overlaps, primary entries,
truncation and partitions outside the dump, index names for unsafe labels,
repeated declarations, the otadata states and records, and each branch of
the bootloader's selection. CLI and stdio MCP tests exercise the inputs and
their identity and extraction. MCP `compare_artifacts` tests compare
partition declarations, labels that are not UTF-8 included, two dumps that
end inside their last partition, and such a dump against a full one.
The optional real lane builds its dumps with esptool and ESP-IDF's own tools:
```sh
REA_RISCV_CLANG=/path/to/clang REA_RISCV_LLD=/path/to/ld.lld \
REA_ESPTOOL_PYTHON=/path/to/python-with-esptool \
REA_ESP_IDF_PATH=/path/to/esp-idf \
npm run verify:esp-flash
```
It converts the conformance ELFs with esptool, builds a default, a
relocated and a primary-entry partition table with `gen_esp32part.py`,
assembles dumps with `esptool merge-bin`, one of them written from flash
`0x1000` with `--target-offset`, and writes OTA records with `otatool`
through `tests/conformance/esp/flash_file_tool.py`. That script replaces
parttool's esptool calls with reads and writes of the dump file, so
otatool's own OTA logic runs unchanged. The lane then checks:
- the bootloader and app facts against `esptool image-info`;
- every partition against `gen_esp32part.py`'s decoding of the table REA
extracted;
- both OTA records against `otatool read_otadata` after switching to slot 0,
then to slot 1, then erasing the otadata, and the derived selection after
each: slot 0, slot 1, then the factory fallback;
- that each extracted file is the dump's bytes at its range and that the
source is unchanged;
- a table with the primary `bootloader` and `partition_table` entries that
`--primary-bootloader-offset` fills, read with complete coverage and no
overlap or misplacement limitation; it has no otadata, so its boot
selection is `no-otadata` with the factory app;
- the relocated table, found by scanning and named with
`partition_table_offset`, and the dump written from `0x1000`, read with
`flash_offset` and, without it, left an ESP image whose `flash_dump`
names `flash_offset` `0x1000`;
- an app with a flipped byte beside valid partitions, and a truncated dump,
read with complete coverage and its missing partitions as entries with no
bytes.
The lane ran on macOS arm64 with Node 24.22.1, Homebrew Clang and LLD 21.1.8
and ESP-IDF v5.5.2's `gen_esp32part.py`, `otatool.py` and `parttool.py`,
once with esptool v5.6.0 and once with 4.12.0. Its `gen_esp32part.py` is
identical to v6.1's, and its otatool and parttool differ only in import style
and esptool command spelling. All checks passed in both runs. With v5.6.0 and
the build already cached, the run took 24.1 s elapsed, 16.9 s user CPU and
6.3 s system CPU under `/usr/bin/time -l`; the largest single process peaked
at a 278 MiB resident set, and the combined peak of all processes was not
measured.
The lane does not establish:
- dumps read from a device: its dumps come from `merge-bin`;
- chips other than ESP32-C3 and ESP32;
- `ota_state` and record validity, which `read_otadata` does not print, or
OTA states the bootloader writes itself, app rollback or anti-rollback;
- selection with gaps or repeated OTA subtypes, which otatool does not
write;
- flash-encrypted dumps, which are not recognized by design.
It does not run in CI.
docs/esp-image.md
+20 -3
View File
@@ -223,9 +223,26 @@ first byte that is not erased starts a range to the end of that region. A
slot that is neither a block nor erased is reported whole. Adjacent ranges
are merged.
A whole-flash dump of a chip whose bootloader is at offset 0, such as
ESP32-C3 or ESP32-S3, is recognized as that bootloader image; the partition
table, apps and data follow as undescribed bytes.
A whole-flash dump whose partition table REA finds is read as an
[ESP flash image](esp-flash-image.md). Without one, a dump of a chip whose
bootloader is at offset 0, such as ESP32-C3 or ESP32-S3, is recognized as that
bootloader image, and the partition table, apps and data follow as
undescribed bytes.
A dump read from a later flash address, such as an ESP32 dump read from
`0x1000` with `read-flash 0x1000 …` or `merge-bin --target-offset 0x1000`,
opens as its bootloader image. REA says so when the file shows it: the image
at file offset 0 is not an app, its chip's ROM loads the bootloader from a
later address (`0x1000` for ESP32 and S2, `0x2000` for C5, H4, P4 and S31),
and a partition table sits where that origin puts `0x8000` (file offset
`0x7000` for `0x1000`), after the image ends. Then `flash_dump` records
`flash_offset` and
`table_file_offset`, a limitation says to reopen the file with
`format: "esp-flash-image"` and that `flash_offset`, and `inspect_artifact`
lists the same `open_binary` call as a next probe. `flash_dump` is null for
every other image. Evidence that REA recorded before it reported the field
omits it; REA still reads and compares such Evidence, and the omission says
nothing about where the file was read from.
## Limits
docs/firmware-analysis.md
+11
View File
@@ -300,6 +300,17 @@ v2 signature blocks. `extract-artifact` / `extract_artifact` writes one file
per segment and signature block. Reading is portable and needs no external
tool. See [ESP images](esp-image.md).
## Inspect ESP flash dumps
Use the same tools on a whole-flash dump, such as `esptool read-flash 0 ALL`
output. The inventory reports the bootloader, the partition table and how it
was found, each partition with its declared type, flags and digest, ESP image
facts for the bootloader and each app partition, and the OTA records with a
derived selection; the running partition stays unknown. Extraction writes
each partition, `nvs_keys` included, as an ordinary file. For a dump that
does not start at flash address 0, select `esp-flash-image` with
`flash_offset`. See [ESP flash images](esp-flash-image.md).
## Inspect extracted devicetrees
Use [`inspect_devicetree`](devicetree.md) / `rea inspect-devicetree` on an
docs/index.md
+14 -14
View File
@@ -47,20 +47,20 @@ For runtime instrumentation, see the [Frida guide](./frida-instrumentation.md).
## Choose a guide
| Task | Guides |
| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [UnityFS](unityfs.md), [U-Boot uImage](uimage.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
| Capture and compare a command | [Process capture](process-capture.md) |
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
| Task | Guides |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Identify a desktop installation | [Runtime candidates, evidence and follow-up files](desktop-applications.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Linux kernel modules](kernel-modules.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Inspect native functions, references, or types | [Native binaries](native-investigation.md), [Go metadata](go-binaries.md), [Rust dependencies](rust-binaries.md), [CPython pyc files](python-pyc.md), [PE certificates](pe-certificates.md), [PE imports](pe-imports.md), [PE TLS/load config](pe-load-config.md), [ELF notes](elf-notes.md), [Apple IPS](apple-ips.md), [Erlang BEAM](erlang-beam.md), [offline ELF layout](binary-diagnostics.md), [recorded crashes](recorded-crashes.md) |
| Select a native provider or import format | [Ghidra semantics](ghidra-provider.md), [Windows](windows-ghidra-p0.md), [DOS](ghidra-dos.md), [raw bytes](raw-binary.md), [GameCube DOL](ghidra-dol.md), [MIPS](ghidra-mips.md), [PSP](ghidra-psp.md), [NativeAOT](ghidra-nativeaot.md), [IDA](ida-provider.md), [Kuna](kuna-provider.md), [JEB](jeb-analysis.md) |
| Analyze JavaScript or Electron artifacts | [Static analysis](javascript-artifact-reconstruction.md), [tracing and comparisons](javascript-application-workflows.md), [application graph](javascript-application-graph.md), [source recovery](javascript-recovery.md) |
| Compare static JavaScript with runtime observations | [Runtime attribution](javascript-runtime-observation.md), [reconciliation](javascript-runtime-reconciliation.md), [Electron observation](electron-observation.md), [Electron fuses](electron-fuses.md) |
| Inspect assemblies, packages, or firmware | [Godot PCK](godot-pck.md), [UnityFS](unityfs.md), [U-Boot uImage](uimage.md), [.NET](managed-code-analysis.md), [Python packages](python-packaged-applications.md), [Android](android-analysis.md), [APK signatures](apk-signature.md), [Apktool resources](apktool-resource-analysis.md), [ADB acquisition](adb-device-analysis.md), [Apple](apple-application-analysis.md), [Harmony](harmony-analysis.md), [firmware](firmware-analysis.md), [ESP images](esp-image.md), [ESP flash images](esp-flash-image.md), [WASM](wasm-artifacts.md), [EVM](evm-bytecode.md) |
| Observe or compare a browser scenario | [CDP observation](browser-observation.md), [scenarios](browser-scenario-contract.md), [runtime evidence](web-runtime.md), [historical captures](web-network-captures.md) |
| Serve recorded behavior as an agent environment | [Agent environments](agent-environments.md) |
| Trace captured website source | [Source locations](web-source-location.md), [module traces](website-module-trace.md), [script export](website-script-export.md) |
| Capture and compare a command | [Process capture](process-capture.md) |
| Use CLI/MCP and retain evidence | [CLI](cli.md), [MCP contracts](mcp-contracts.md), [agent prompts](mcp-prompts.md), [obligation ledgers](reconstruction-obligation-ledgers.md) |
For development, use [testing](testing.md), [client verification](client-verification.md),
[tool design](tool-design.md), [admission limits and recovery](design/limits-and-recovery.md),
docs/testing.md
+7
View File
@@ -2038,6 +2038,13 @@ tool before building and warns when esptool is not the pinned v5.6.0.
`REA_XTENSA_ESP32_GCC` adds an ESP32 Xtensa build. See
[ESP images](esp-image.md).
`npm run verify:esp-flash` needs the same tools and an ESP-IDF checkout
(`REA_ESP_IDF_PATH`) for `gen_esp32part.py` and `otatool.py`. It builds
partition tables, assembles dumps with `esptool merge-bin`, writes OTA records
with otatool through `tests/conformance/esp/flash_file_tool.py`, and compares
REA's flash facts with `image-info`, the decoded table and
`otatool read_otadata`. See [ESP flash images](esp-flash-image.md).
### JavaScript source recovery
Build once with `npm run build:cached`, then run `npm run verify:javascript:recovery`.
package.json
+1
View File
@@ -241,6 +241,7 @@
"verify:firmware": "npm run build:cached && node scripts/verify/firmware/analysis.mjs",
"verify:devicetree": "npm run build:cached && node scripts/verify/firmware/devicetree.mjs",
"verify:esp-images": "npm run build:cached && node scripts/verify/firmware/esp-image.mjs",
"verify:esp-flash": "npm run build:cached && node scripts/verify/firmware/esp-flash.mjs",
"fixtures:firmware": "node scripts/fixtures/firmware/generate.mjs",
"verify:android": "npm run build:cached && node scripts/verify/android/analysis.mjs",
"verify:jeb": "npm run build:cached && node scripts/verify-jeb.mjs",
scripts/verify/firmware/esp-flash-tools.mjs
+136
View File
@@ -0,0 +1,136 @@
import assert from "node:assert/strict";
import { access, readFile } from "node:fs/promises";
import { join } from "node:path";
import {
execute,
failedCommand,
options,
source,
spell,
tool,
} from "./esp-image-tools.mjs";
// Optional verification tools only; the runtime reader starts no tool.
const idf = process.env.REA_ESP_IDF_PATH;
const python = process.env.REA_ESPTOOL_PYTHON ?? "python3";
const generator = () =>
join(idf, "components", "partition_table", "gen_esp32part.py");
const environment = () => ({ ...process.env, IDF_PATH: idf });
/** Check the ESP-IDF checkout before any work and report its version. */
export const checkEspIdf = async () => {
if (idf === undefined)
throw new Error(
"verify:esp-flash needs an ESP-IDF checkout for gen_esp32part.py and otatool.py; set REA_ESP_IDF_PATH",
);
try {
await access(generator());
await access(join(idf, "components", "app_update", "otatool.py"));
const version = await readFile(
join(idf, "tools", "cmake", "version.cmake"),
"utf8",
);
const part = (name) =>
new RegExp(`IDF_VERSION_${name} (\\d+)`, "u").exec(version)?.[1];
return `v${part("MAJOR")}.${part("MINOR")}.${part("PATCH")}`;
} catch (failure) {
throw new Error(
`verify:esp-flash needs gen_esp32part.py, otatool.py and tools/cmake/version.cmake under REA_ESP_IDF_PATH (${idf}). ${failure.message}`,
{ cause: failure },
);
}
};
/** Build a binary table from CSV with ESP-IDF's generator. */
export const buildTable = (csv, output, flags = []) =>
execute(python, ["-I", generator(), "--quiet", ...flags, csv, output], {
...options,
env: environment(),
});
const bytesOf = (text) => {
const value = text.trim();
if (value.endsWith("M")) return Number(value.slice(0, -1)) * 0x10_0000;
if (value.endsWith("K")) return Number(value.slice(0, -1)) * 0x400;
return Number(value);
};
/** ESP-IDF's own reading of a binary table, as the CSV it prints. */
export const decodeTable = async (binary) => {
const { stdout } = await execute(python, ["-I", generator(), binary], {
...options,
env: environment(),
});
return stdout
.split("\n")
.filter((line) => line.trim() !== "" && !line.startsWith("#"))
.map((line) => {
const [name, type, subtype, offset, size, flags = ""] = line
.split(",")
.map((field) => field.trim());
return {
name,
type,
subtype,
offset: bytesOf(offset),
size: bytesOf(size),
encrypted: flags.split(":").includes("encrypted"),
readonly: flags.split(":").includes("readonly"),
};
});
};
/** Run otatool against a dump file through the conformance shim. */
export const otatool = async (image, flashOffset, args) => {
const run = await execute(
python,
[
"-I",
source("flash_file_tool.py"),
image,
`0x${flashOffset.toString(16)}`,
"otatool",
...args,
],
{ ...options, env: environment() },
).then(
({ stdout, stderr }) => `${stdout}${stderr}`,
failedCommand(`otatool ${args.join(" ")}`),
);
return run;
};
/** The two records otatool read_otadata prints. */
export const readOtadata = async (image, flashOffset, tableOffset) => {
const text = await otatool(image, flashOffset, [
"--partition-table-offset",
`0x${tableOffset.toString(16)}`,
"read_otadata",
]);
const row =
/Firmware:\s+0x([0-9a-f]{8})\s+0x([0-9a-f]{8})\s*\|\s*0x([0-9a-f]{8})\s+0x([0-9a-f]{8})/u.exec(
text,
);
assert.ok(row, `Unrecognized read_otadata output: ${text}`);
return [
{ seq: Number.parseInt(row[1], 16), crc: Number.parseInt(row[2], 16) },
{ seq: Number.parseInt(row[3], 16), crc: Number.parseInt(row[4], 16) },
];
};
/** Assemble a flash image with esptool merge-bin. */
export const mergeBin = (chip, output, parts, flags = []) =>
tool("esptool", [
"--chip",
chip,
spell("merge_bin", "merge-bin"),
"-o",
output,
...flags,
...parts.flatMap(({ at, file }) => [`0x${at.toString(16)}`, file]),
]);
export const padTo = (size) => [
spell("--fill-flash-size", "--pad-to-size"),
size,
];
scripts/verify/firmware/esp-flash.mjs
+419
View File
@@ -0,0 +1,419 @@
#!/usr/bin/env node
import assert from "node:assert/strict";
import { createHash } from "node:crypto";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { withVerifierCleanup } from "../../lib/verifier-cleanup.mjs";
import {
assertAgrees,
buildElf,
checkTools,
commands,
elf2image,
extract,
imageInfo,
inspect,
} from "./esp-image-tools.mjs";
import {
buildTable,
checkEspIdf,
decodeTable,
mergeBin,
otatool,
padTo,
readOtadata,
} from "./esp-flash-tools.mjs";
let workspace;
const path = (name) => join(workspace, name);
const sha256 = (bytes) => createHash("sha256").update(bytes).digest("hex");
const DEFAULT_CSV = `# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,0x4000,
otadata,data,ota,0xd000,0x2000,
phy_init,data,phy,0xf000,0x1000,
factory,app,factory,0x10000,0x20000,
ota_0,app,ota_0,0x30000,0x20000,
ota_1,app,ota_1,0x50000,0x20000,
keys,data,nvs_keys,0x70000,0x1000,encrypted
`;
// Blank offsets: gen_esp32part places each partition after the moved table.
const RELOCATED_CSV = `# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,,0x4000,
otadata,data,ota,,0x2000,
factory,app,factory,,0x20000,
`;
// ESP-IDF 5.3 entries over the bootloader region and the table sector;
// gen_esp32part fills their ranges from --primary-bootloader-offset.
const PRIMARY_CSV = `# Name, Type, SubType, Offset, Size, Flags
PrimaryBTLDR,bootloader,primary,N/A,N/A,
PrimaryPrtTable,partition_table,primary,N/A,N/A,
nvs,data,nvs,0x9000,0x4000,
factory,app,factory,0x10000,0x20000,
`;
/** Nested ESP image facts, shaped like a top-level inventory for assertAgrees. */
const nested = (facts) => ({
inventory: { manifest: { root_format: "esp-image" } },
root: facts.image,
segments: facts.segments.map((segment) => ({ format_metadata: segment })),
});
const appFacts = (rea, label) => {
const row = rea.inventory.occurrences.find(
({ format_metadata: metadata }) =>
metadata?.kind === "esp-flash-partition" && metadata.label === label,
);
return rea.root.apps.find(
({ partition_index: index }) => index === row?.format_metadata.index,
)?.facts;
};
/** Every declared partition agrees with ESP-IDF's decoding of the table REA extracted. */
const assertTableAgrees = async (name, rea, extraction) => {
const decoded = await decodeTable(
join(extraction.output_root, "partition-table.bin"),
);
const declared = [
...rea.inventory.occurrences
.filter(({ format_metadata: m }) => m?.kind === "esp-flash-partition")
.map(({ format_metadata: m }) => m),
...rea.root.partitions_outside_dump,
].sort((left, right) => left.index - right.index);
assert.deepEqual(
declared.map((p) => ({
name: p.label,
type: p.type_name,
subtype: p.subtype_name,
offset: p.flash_address,
size: p.size,
encrypted: p.encrypted,
readonly: p.readonly,
})),
decoded,
`${name} table`,
);
};
/** Each extracted file is the dump's bytes at its range; the source is unchanged. */
const assertExtraction = async (name, file, rea, flags = []) => {
const bytes = await readFile(file);
const extraction = await extract(file, workspace, flags);
assert.equal(extraction.containment_verified, true, name);
assert.equal(
extraction.manifest.root_sha256,
sha256(bytes),
`${name} source`,
);
const rows = rea.inventory.occurrences.filter(
({ logical_path }) => logical_path !== ".",
);
assert.equal(extraction.artifacts.length, rows.length, `${name} entries`);
for (const { logical_path: entry, source_location: range } of rows) {
const written = await readFile(join(extraction.output_root, entry));
assert.ok(
written.equals(bytes.subarray(range.offset, range.offset + range.length)),
`${name} ${entry}`,
);
}
assert.equal(sha256(await readFile(file)), sha256(bytes), `${name} source`);
return extraction;
};
const build = async () => {
const app = await buildElf("app.c", "app.ld", path("app.elf"));
const boot = await buildElf(
"bootloader.c",
"bootloader.ld",
path("boot.elf"),
);
const descriptor = ["--elf-sha256-offset", "0xb0"];
await elf2image("esp32c3", app, path("app.bin"), descriptor);
await elf2image("esp32c3", boot, path("boot.bin"));
// esptool converts the RISC-V ELF for any chip; these use ESP32 addresses.
const app32 = await buildElf(
"app.c",
"esp32-app-riscv.ld",
path("app32.elf"),
);
const boot32 = await buildElf(
"bootloader.c",
"esp32-bootloader.ld",
path("boot32.elf"),
);
await elf2image("esp32", app32, path("app32.bin"), descriptor);
await elf2image("esp32", boot32, path("boot32.bin"));
await writeFile(path("default.csv"), DEFAULT_CSV);
await writeFile(path("relocated.csv"), RELOCATED_CSV);
await writeFile(path("primary.csv"), PRIMARY_CSV);
await buildTable(path("default.csv"), path("default-table.bin"));
await buildTable(path("relocated.csv"), path("relocated-table.bin"), [
"--offset",
"0xa000",
]);
await buildTable(path("primary.csv"), path("primary-table.bin"), [
"--primary-bootloader-offset",
"0x0",
]);
};
const assertOtaAgrees = async (name, file, rea) => {
const records = await readOtadata(file, 0, 0x8000);
assert.deepEqual(
rea.root.otadata.records.map(({ ota_seq: seq, crc }) => ({ seq, crc })),
records,
`${name} otadata`,
);
};
const defaultLayout = async (report) => {
const file = path("c3.bin");
await mergeBin(
"esp32c3",
file,
[
{ at: 0x0, file: path("boot.bin") },
{ at: 0x8000, file: path("default-table.bin") },
{ at: 0x1_0000, file: path("app.bin") },
{ at: 0x3_0000, file: path("app.bin") },
],
padTo("1MB"),
);
await otatool(file, 0, ["switch_ota_partition", "--slot", "0"]);
const rea = await inspect(file);
assert.equal(rea.inventory.manifest.root_format, "esp-flash-image");
assert.equal(rea.inventory.coverage.status, "complete");
assert.deepEqual(rea.root.undescribed, []);
assert.equal(rea.root.partition_table.method, "default");
assert.equal(rea.root.partition_table.md5.status, "valid");
assertAgrees(
"c3 bootloader",
await imageInfo(path("boot.bin")),
nested(rea.root.bootloader.facts),
);
const appOracle = await imageInfo(path("app.bin"));
for (const label of ["factory", "ota_0"])
assertAgrees(`c3 ${label}`, appOracle, nested(appFacts(rea, label)));
assert.equal(appFacts(rea, "ota_1").status, "erased");
await assertOtaAgrees("c3 slot 0", file, rea);
assert.deepEqual(
[rea.root.boot_selection.slot, rea.root.boot_selection.basis],
[0, "otadata"],
);
const extraction = await assertExtraction("c3", file, rea);
await assertTableAgrees("c3", rea, extraction);
// A second switch writes the other record; the higher sequence selects ota_1.
await otatool(file, 0, ["switch_ota_partition", "--slot", "1"]);
const switched = await inspect(file);
await assertOtaAgrees("c3 slot 1", file, switched);
assert.equal(switched.root.boot_selection.slot, 1);
assert.equal(switched.root.boot_selection.running_partition, "unknown");
// Erased records leave the bootloader's factory fallback.
await otatool(file, 0, ["erase_otadata"]);
const erased = await inspect(file);
await assertOtaAgrees("c3 erased", file, erased);
const factory = erased.inventory.occurrences.find(
({ format_metadata: m }) => m?.label === "factory",
);
assert.deepEqual(
[
erased.root.boot_selection.basis,
erased.root.boot_selection.partition_index,
],
["factory-fallback", factory?.format_metadata.index],
);
report.cases.push({
name: "c3.bin",
partitions: extraction.artifacts.length - 2,
selected: [0, 1, "factory-fallback"],
});
};
const primaryEntries = async (report) => {
const file = path("primary.bin");
await mergeBin(
"esp32c3",
file,
[
{ at: 0x0, file: path("boot.bin") },
{ at: 0x8000, file: path("primary-table.bin") },
{ at: 0x1_0000, file: path("app.bin") },
],
padTo("1MB"),
);
const rea = await inspect(file);
assert.equal(rea.inventory.coverage.status, "complete");
assert.deepEqual(
rea.inventory.limitations.filter((text) =>
/overlaps|before the end of the partition table/u.test(text),
),
[],
"primary entries are not misplaced",
);
const extraction = await assertExtraction("primary", file, rea);
await assertTableAgrees("primary", rea, extraction);
// Without otadata the bootloader selects the factory index.
const factory = rea.inventory.occurrences.find(
({ format_metadata: m }) => m?.label === "factory",
);
assert.deepEqual(
[
rea.root.otadata,
rea.root.boot_selection.basis,
rea.root.boot_selection.partition_index,
],
[null, "no-otadata", factory?.format_metadata.index],
);
report.cases.push({
name: "primary.bin",
partitions: extraction.artifacts.length - 2,
});
};
const relocated = async (report) => {
const [, , factory] = await decodeTable(path("relocated-table.bin"));
const file = path("relocated.bin");
await mergeBin(
"esp32c3",
file,
[
{ at: 0x0, file: path("boot.bin") },
{ at: 0xa000, file: path("relocated-table.bin") },
{ at: factory.offset, file: path("app.bin") },
],
padTo("1MB"),
);
const scanned = await inspect(file);
assert.deepEqual(
[
scanned.root.partition_table.flash_address,
scanned.root.partition_table.method,
],
[0xa000, "scan"],
);
const caller = [
"--target-format",
"esp-flash-image",
"--partition-table-offset",
"0xa000",
];
const selected = await inspect(file, caller);
assert.equal(selected.root.partition_table.method, "caller");
await assertTableAgrees(
"relocated",
selected,
await assertExtraction("relocated", file, selected, caller),
);
report.cases.push({
name: "relocated.bin",
table: "0xa000",
methods: ["scan", "caller"],
});
};
const esp32Origin = async (report) => {
const parts = [
{ at: 0x1000, file: path("boot32.bin") },
{ at: 0x8000, file: path("default-table.bin") },
{ at: 0x1_0000, file: path("app32.bin") },
];
const whole = path("esp32.bin");
await mergeBin("esp32", whole, parts, padTo("1MB"));
const rea = await inspect(whole);
assert.deepEqual(
[rea.root.chip.target, rea.root.bootloader.flash_address],
["esp32", 0x1000],
);
assertAgrees(
"esp32 factory",
await imageInfo(path("app32.bin")),
nested(appFacts(rea, "factory")),
);
// esptool writes the same parts as a dump that starts at flash 0x1000.
const partial = path("esp32-from-0x1000.bin");
await mergeBin("esp32", partial, parts, ["--target-offset", "0x1000"]);
const guessed = await inspect(partial);
assert.equal(
guessed.inventory.manifest.root_format,
"esp-image",
"no origin, no layout",
);
// The table at file offset 0x7000 shows where the dump was read from.
assert.deepEqual(guessed.root.flash_dump, {
flash_offset: 0x1000,
table_file_offset: 0x7000,
});
const flags = [
"--target-format",
"esp-flash-image",
"--flash-offset",
"0x1000",
];
const origin = await inspect(partial, flags);
assert.deepEqual(origin.root.flash_offset, {
value: 0x1000,
source: "caller",
});
await assertExtraction("esp32 from 0x1000", partial, origin, flags);
report.cases.push({
name: "esp32-from-0x1000.bin",
automatic: "esp-image",
flash_offset: "0x1000",
});
};
const damaged = async (report) => {
const bytes = await readFile(path("c3.bin"));
const corrupt = Buffer.from(bytes);
// Flip a byte of the factory app's first segment data.
corrupt[0x1_0000 + 0x18 + 8] ^= 0x01;
await writeFile(path("corrupt.bin"), corrupt);
const rea = await inspect(path("corrupt.bin"));
assert.equal(appFacts(rea, "factory").image.checksum.status, "invalid");
assert.equal(appFacts(rea, "ota_0").image.checksum.status, "valid");
await writeFile(path("truncated.bin"), bytes.subarray(0, 0x4_0000));
const cut = await inspect(path("truncated.bin"));
// Partitions past the end of the file are entries holding no bytes.
assert.equal(cut.inventory.coverage.status, "complete");
assert.deepEqual(cut.root.partitions_outside_dump, []);
const empty = cut.inventory.occurrences
.filter(({ declared_size: size }) => size === 0)
.map(({ format_metadata: m }) => m?.label);
assert.deepEqual(empty, ["keys", "ota_1"]);
assert.ok(
cut.inventory.occurrences.some(
({ logical_path }) => logical_path === "partitions/factory.bin",
),
);
report.cases.push(
{ name: "corrupt.bin", factory: "invalid" },
{ name: "truncated.bin", without_bytes: empty },
);
};
await withVerifierCleanup(async () => {
const tools = await checkTools("verify:esp-flash");
const espIdf = await checkEspIdf();
workspace = await mkdtemp(join(tmpdir(), "rea-esp-flash-proof-"));
const report = {
esptool: tools.esptoolVersion,
esp_idf: espIdf,
compiler: tools.compilerVersion,
cases: [],
};
await build();
await defaultLayout(report);
await primaryEntries(report);
await relocated(report);
await esp32Origin(report);
await damaged(report);
console.log(JSON.stringify(report, null, 2));
}, [
// Remove outputs only after every owned command tree has stopped.
async () => {
await commands.close();
if (workspace !== undefined)
await rm(workspace, { recursive: true, force: true });
},
]);
scripts/verify/firmware/esp-image-tools.mjs
+33 -15
View File
@@ -22,6 +22,13 @@ export const exitOutput = (failure) => {
if (typeof failure?.code !== "number" || failure.code === 0) throw failure;
return `${failure.snapshot.stdout.text}${failure.snapshot.stderr.text}`;
};
/** Reject a command that had to succeed, with the output it exited with. */
export const failedCommand = (label) => (failure) => {
throw new Error(`${label} failed: ${exitOutput(failure)}`, {
cause: failure,
});
};
export const source = (name) =>
fileURLToPath(
new URL(`../../../tests/conformance/esp/${name}`, import.meta.url),
@@ -41,13 +48,16 @@ export const tool = async (module, args) => {
return `${stdout}${stderr}`;
};
/** The npm script whose tools are checked, named in each failure. */
let lane = "verify:esp-images";
/** Check one tool before any work; a failure names the variable to set. */
const prerequisite = async (needed, variable, check) => {
try {
return await check();
} catch (failure) {
throw new Error(
`verify:esp-images needs ${needed}; set ${variable}. ${failure.message}`,
`${lane} needs ${needed}; set ${variable}. ${failure.message}`,
{ cause: failure },
);
}
@@ -63,7 +73,8 @@ export const spell = (v4, later) => {
};
/** Check every tool before any work and report their versions. */
export const checkTools = async () => {
export const checkTools = async (script = "verify:esp-images") => {
lane = script;
const compilerVersion = await prerequisite(
`a Clang with the RISC-V target (tried ${compiler})`,
"REA_RISCV_CLANG",
@@ -98,7 +109,7 @@ export const checkTools = async () => {
);
if (version[0] !== PINNED_ESPTOOL)
console.warn(
`verify:esp-images: esptool ${version[0]} is not the pinned ${PINNED_ESPTOOL}, whose sources the reader's layout facts cite.`,
`${lane}: esptool ${version[0]} is not the pinned ${PINNED_ESPTOOL}, whose sources the reader's layout facts cite.`,
);
const xtensaVersion =
xtensa === undefined
@@ -201,19 +212,20 @@ export const imageInfo = async (file, chip) => {
};
};
/** REA's complete inline inventory for one file through the built CLI. */
export const inspect = async (file) => {
const result = await execute(
const inspectArtifact = (file, flags) =>
execute(
process.execPath,
[entrypoint, "inspect-artifact", file, "--json"],
[entrypoint, "inspect-artifact", file, ...flags, "--json"],
{ ...options, env: { ...process.env, REA_LOG_LEVEL: "silent" } },
).then(
({ stdout }) => ({ ok: true, stdout }),
(failure) => ({ ok: false, stdout: exitOutput(failure) }),
);
if (!result.ok) return { refused: result.stdout };
const inventory = JSON.parse(result.stdout).normalized_result.substeps[0]
.evidence.normalized_result;
/** REA's complete inline inventory for one file through the built CLI. */
export const inspect = async (file, flags = []) => {
const { stdout } = await inspectArtifact(file, flags).catch(
failedCommand(`inspect-artifact ${file}`),
);
const inventory =
JSON.parse(stdout).normalized_result.substeps[0].evidence.normalized_result;
const rows = inventory.occurrences.filter((row) => row.logical_path !== ".");
return {
inventory,
@@ -228,6 +240,12 @@ export const inspect = async (file) => {
};
};
/** REA's output for a file it must refuse; opening the file fails the lane. */
export const refusal = (file, flags = []) =>
inspectArtifact(file, flags).then(() => {
throw new Error(`inspect-artifact opened ${file}, which it must refuse`);
}, exitOutput);
const oracleDescriptorKind = (oracle) => {
if (oracle.project !== undefined) return "app";
if (oracle.bootloaderVersion !== undefined) return "bootloader";
@@ -235,10 +253,10 @@ const oracleDescriptorKind = (oracle) => {
};
/** Extract every entry through the built CLI into a new directory in `root`. */
export const extract = async (file, root) => {
export const extract = async (file, root, flags = []) => {
const { stdout } = await execute(
process.execPath,
[entrypoint, "extract-artifact", file, "--json"],
[entrypoint, "extract-artifact", file, ...flags, "--json"],
{
...options,
env: {
scripts/verify/firmware/esp-image.mjs
+3 -3
View File
@@ -17,6 +17,7 @@ import {
imageInfo,
inspect,
options,
refusal,
source,
spell,
tool,
@@ -392,7 +393,7 @@ await withVerifierCleanup(async () => {
const v3Oracle = await imageInfo(path("esp8266-v3.bin"), "esp8266");
assert.ok((v3Oracle.segments[0]?.address ?? 0) >= 0x4020_0000);
assert.match(
(await inspect(path("esp8266-v3.bin"))).refused ?? "",
await refusal(path("esp8266-v3.bin")),
/ESP8266 firmware images are not supported/u,
);
report.images.push({ name: "esp8266-v3.bin", refused: true });
@@ -400,9 +401,8 @@ await withVerifierCleanup(async () => {
// ESP8266 v1: esptool falls back to ESP8266; REA refuses it by name.
const esp8266Image = path("esp8266-0x00000.bin");
assert.equal((await imageInfo(esp8266Image)).detected, "ESP8266");
const refused = await inspect(esp8266Image);
assert.match(
refused.refused ?? "",
await refusal(esp8266Image),
/ESP8266 firmware images are not supported/u,
);
report.images.push({ name: "esp8266-0x00000.bin", refused: true });
src/application/BinaryTargetResolver.ts
+78
View File
@@ -84,6 +84,11 @@ import { hasUImageMagic } from "../domain/uimageFormat.js";
import { hasUnityFsHeader } from "../domain/unityfsFormat.js";
import { detectHexRecordEncoding } from "../domain/firmware/hexRecords.js";
import { hasFitImagesNode } from "../domain/firmware/fitImage.js";
import {
locateEspFlashLayout,
type EspFlashSelection,
} from "../domain/firmware/espFlashLocation.js";
import { espFlashSelectionError } from "../domain/firmware/espFlashSelection.js";
import {
ESP8266_REFUSAL,
recognizeEspImage,
@@ -136,6 +141,8 @@ export interface BinaryTargetResolverOptions {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
/** Inputs for format=esp-flash-image; rejected with any other format. */
readonly espFlash?: EspFlashSelection;
readonly signal?: AbortSignal;
readonly resources?: ArtifactResourceScope;
}
@@ -273,6 +280,18 @@ const readOpenedBinaryTarget = async (
const hostArchitecture = options.hostArchitecture ?? process.arch;
const { targetKind, formatHint, signal, architecture, entryPoint } = options;
throwIfTargetResolutionCancelled(signal);
// Flash inputs pair with one format; check before any hint branch.
const flashPairing = espFlashSelectionError({
format: formatHint,
flashOffset: options.espFlash?.flashOffset,
partitionTableOffset: options.espFlash?.partitionTableOffset,
});
if (flashPairing !== undefined)
return err(
new BinaryTargetError(path, flashPairing, {
constraint: "load_selection_rejected",
}),
);
if (formatHint === "raw-binary") {
if (targetKind !== undefined && targetKind !== "executable")
return err(
@@ -360,6 +379,45 @@ const readOpenedBinaryTarget = async (
dol: admitted.value.header,
});
}
if (formatHint === "esp-flash-image") {
if (targetKind !== undefined && targetKind !== "archive")
return err(
new BinaryTargetError(
path,
"ESP flash image format requires an archive target kind",
{ constraint: "load_selection_rejected" },
),
);
const selection = options.espFlash ?? {};
// An encrypted dump has no readable table: unknown, not malformed.
const located = await locateEspFlashLayout(
metadata.size,
(position, length) => readHandleRange(handle, position, length, signal),
selection,
{ explicit: true, ...(signal === undefined ? {} : { signal }) },
);
if (!located.ok)
return err(
new BinaryTargetError(
path,
`ESP flash image layout is not recognized: ${located.error}`,
{ constraint: "file_format_rejected" },
),
);
return ok({
path,
sourcePath,
artifactSelectedPath: selectedPath,
sha256: await sha256Handle(handle, path, metadata, signal),
kind: "archive",
format: "esp-flash-image",
espFlash: {
selected: true,
flashOffset: selection.flashOffset ?? null,
partitionTableOffset: selection.partitionTableOffset ?? null,
},
});
}
if (formatHint === "dos-com") {
if (targetKind !== undefined && targetKind !== "executable")
return err(
@@ -412,6 +470,17 @@ const readOpenedBinaryTarget = async (
artifactSelectedPath: selectedPath,
sha256: await sha256Handle(handle, path, metadata, signal),
};
if (artifactFormat === "esp-flash-image")
return ok({
...identity,
kind: "archive",
format: artifactFormat,
espFlash: {
selected: false,
flashOffset: null,
partitionTableOffset: null,
},
});
return isArchiveFormat(artifactFormat)
? ok({ ...identity, kind: "archive", format: artifactFormat })
: ok({ ...identity, kind: "artifact", format: artifactFormat });
@@ -682,6 +751,14 @@ const detectArtifactFormat = async (
if (hasUImageMagic(magic)) return "uimage";
const hexRecords = detectHexRecordEncoding(magic);
if (hexRecords !== undefined) return hexRecords;
// A C3 or S3 dump begins with its bootloader, so a layout comes first.
const flash = await locateEspFlashLayout(
initial.size,
(position, length) => readHandleRange(handle, position, length, signal),
{},
{ explicit: false, ...(signal === undefined ? {} : { signal }) },
);
if (flash.ok) return "esp-flash-image";
const esp = recognizeEspImage(magic, initial.size);
if (esp === "esp8266")
throw new BinaryTargetError(path, ESP8266_REFUSAL, {
@@ -766,6 +843,7 @@ const isArchiveFormat = (
"intel-hex",
"srec",
"esp-image",
"esp-flash-image",
"fit",
].includes(format);
src/application/DirectAnalysis.ts
+53 -46
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import type { ExecutableFormatHint } from "../domain/dosCom.js";
import { dispatchQueuedInterrupts } from "../process/queuedInterrupts.js";
import type {
@@ -133,6 +134,7 @@ export const runDirectAnalysis = async (
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
} = {},
): Promise<Result<Evidence, JsonValue>> =>
withProcessCancellation(options.signal, (signal) =>
@@ -140,25 +142,50 @@ export const runDirectAnalysis = async (
logger: options.logger ?? silentLogger,
snapshotPath: options.snapshotPath,
signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
...(options.providerId === undefined
? {}
: { providerId: options.providerId }),
}),
);
/** How the caller asks an artifact target to be read. */
export interface ArtifactTargetSelection {
readonly formatHint?: ExecutableFormatHint;
readonly espFlash?: EspFlashSelection;
}
/** The caller's target interpretation, forwarded unchanged to every open. */
const targetSelection = (options: {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
}) => ({
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...(options.espFlash === undefined ? {} : { espFlash: options.espFlash }),
});
/** Execute one provider-native semantic operation with atomic provenance. */
export const runProviderAnalysis = async (
dependencies: DirectAnalysisDependencies,
...[path, tool, arguments_, logger = silentLogger, signal, delivery]:
...[
path,
tool,
arguments_,
logger = silentLogger,
signal,
delivery,
selection,
]:
| readonly [
path: string,
tool: NativeToolName | ArtifactAnalysisOperation | ManagedToolName,
@@ -177,6 +204,15 @@ export const runProviderAnalysis = async (
readonly evidenceBundlePath?: string;
},
]
| readonly [
path: string,
tool: ArtifactAnalysisOperation,
arguments_: Readonly<Record<string, JsonValue>>,
logger: Logger | undefined,
signal: AbortSignal | undefined,
delivery: undefined,
selection: ArtifactTargetSelection,
]
): Promise<Result<Evidence, JsonValue>> => {
if (delivery !== undefined && tool !== "inspect_managed_members")
return err(
@@ -248,6 +284,7 @@ export const runProviderAnalysis = async (
snapshotPath: undefined,
signal: operationSignal,
evidenceParameters,
...targetSelection(selection ?? {}),
...(ARTIFACT_ANALYSIS_OPERATION_SET.has(tool)
? { createSession: dependencies.createArtifactBinarySession }
: NATIVE_INSPECTION_OPERATIONS.has(tool)
@@ -328,6 +365,7 @@ const runAnalysis = async (
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
},
): Promise<Result<Evidence, JsonValue>> => {
const { logger, signal, snapshotPath } = options;
@@ -374,15 +412,7 @@ const runAnalysis = async (
path,
snapshotPath,
signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
});
if (!prepared.ok) return err(prepared.error);
const { snapshot } = prepared.value;
@@ -391,15 +421,7 @@ const runAnalysis = async (
const preview = await session.previewTarget(prepared.value.target, {
signal,
snapshot,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
...(options.providerId === undefined
? {}
: { providerId: options.providerId }),
@@ -438,15 +460,7 @@ const runAnalysis = async (
}
const openOptions = {
signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
...(snapshot === undefined ? {} : { snapshot }),
...(options.providerId === undefined
? {}
@@ -706,6 +720,7 @@ const prepareSnapshot = async (options: {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
readonly snapshotPath: string | undefined;
readonly signal: AbortSignal;
}): Promise<
@@ -724,15 +739,7 @@ const prepareSnapshot = async (options: {
if (!loaded.ok) return loaded;
const target = await session.resolveTarget(path, {
signal: options.signal,
...(options.formatHint === undefined
? {}
: { formatHint: options.formatHint }),
...(options.architecture === undefined
? {}
: { architecture: options.architecture }),
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...targetSelection(options),
});
if (!target.ok) return target;
if (!snapshotMatchesTarget(loaded.value.target, target.value))
src/application/binary/BinarySessionOpen.ts
+4
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import type { ExecutableFormatHint } from "../../domain/dosCom.js";
import type {
BinaryArchitecture,
@@ -38,6 +39,8 @@ export interface BinarySessionOpenOptions {
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
/** Inputs for format=esp-flash-image. */
readonly espFlash?: EspFlashSelection;
readonly snapshot?: AnalysisSnapshot;
/**
* Load the snapshot inside the open's turn instead of before it, so calls
@@ -101,6 +104,7 @@ export const resolveSessionOpen = async (
...(options.entryPoint === undefined
? {}
: { entryPoint: options.entryPoint }),
...(options.espFlash === undefined ? {} : { espFlash: options.espFlash }),
...(options.signal === undefined ? {} : { signal: options.signal }),
});
if (!parsed.ok) return parsed;
src/application/binary/BinarySessionPort.ts
+2
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import type { InvestigationRecordPort } from "../investigation/InvestigationRecordPort.js";
import type { ExecutableFormatHint } from "../../domain/dosCom.js";
import type {
@@ -64,6 +65,7 @@ export interface BinarySessionPort
readonly formatHint?: ExecutableFormatHint;
readonly architecture?: BinaryArchitecture;
readonly entryPoint?: string;
readonly espFlash?: EspFlashSelection;
readonly snapshot?: AnalysisSnapshot;
readonly loadSnapshot?: AnalysisSnapshotLoader;
readonly providerId?: AnalysisProviderSelector;
src/application/binary/BinaryTargetSummary.ts
+6
View File
@@ -13,4 +13,10 @@ export const binaryTargetSummary = (target: BinaryTarget) =>
...(target.format === "raw-binary" || target.format === "gamecube-dol"
? { entry_point: target.entryPoint }
: {}),
...(target.format === "esp-flash-image"
? {
flash_offset: target.espFlash.flashOffset,
partition_table_offset: target.espFlash.partitionTableOffset,
}
: {}),
});
src/application/binary/TargetFormatVocabularies.test.ts
+11 -2
View File
@@ -125,6 +125,11 @@ const TARGETS: TargetFixtures = {
"intel-hex": { ...archive, format: "intel-hex" },
srec: { ...archive, format: "srec" },
"esp-image": { ...archive, format: "esp-image" },
"esp-flash-image": {
...archive,
format: "esp-flash-image",
espFlash: { selected: true, flashOffset: null, partitionTableOffset: null },
},
fit: { ...archive, format: "fit" },
plist: { ...artifact, format: "plist" },
javascript: { ...artifact, format: "javascript" },
@@ -169,9 +174,13 @@ describe("target format vocabularies", () => {
);
it.each(executableFormatHintSchema.options)(
"admits the caller-selected %s hint only as an executable target format",
"admits the caller-selected %s hint only as its own target format",
(hint) => {
expect(TARGETS[hint]).toMatchObject({ format: hint, kind: "executable" });
// An ESP flash dump is the one caller-selected format read as an archive.
expect(TARGETS[hint]).toMatchObject({
format: hint,
kind: hint === "esp-flash-image" ? "archive" : "executable",
});
},
);
});
src/artifacts/ArtifactProvider.ts
+14
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import type { AnalysisSettingsReader } from "../domain/analysisSettings.js";
import { runtimeAsarHeaderAdmission } from "../config/runtimeAnalysisSettings.js";
import { artifactTargetRefusal } from "../domain/apple/artifactTargetApplicability.js";
@@ -332,6 +333,7 @@ class ArtifactClient implements AnalysisClient {
: undefined,
selectedPath: this.target.artifactSelectedPath,
inputFormat: this.target.format,
espFlashSelection: espFlashSelection(this.target),
outputRoot: parsed.output_root,
environment: this.environment,
asarHeaderAdmission: runtimeAsarHeaderAdmission(
@@ -542,12 +544,24 @@ class ArtifactClient implements AnalysisClient {
this.environment,
this.settings,
),
espFlashSelection: espFlashSelection(this.target),
...(options?.signal === undefined ? {} : { signal: options.signal }),
integrity: parsed.integrity_policy,
});
}
}
/** Caller inputs of an explicitly selected ESP flash image, for its readers. */
const espFlashSelection = (
target: BinaryTarget,
): EspFlashSelection | undefined =>
target.format === "esp-flash-image" && target.espFlash.selected
? {
flashOffset: target.espFlash.flashOffset ?? undefined,
partitionTableOffset: target.espFlash.partitionTableOffset ?? undefined,
}
: undefined;
const isArtifactOperation = (
operation: AnalysisOperation,
): operation is ArtifactAnalysisOperation =>
src/artifacts/ArtifactReader.ts
+1
View File
@@ -203,6 +203,7 @@ export interface ArtifactReader {
| "intel-hex"
| "srec"
| "esp-image"
| "esp-flash-image"
| "fit"
| "file";
entries(
src/artifacts/ArtifactReaderAdmission.ts
+3 -1
View File
@@ -4,6 +4,7 @@ import type { NodeSeaAdmission } from "./nodeSea/NodeSeaAdmission.js";
import type { PyInstallerAdmission } from "./pyinstaller/PyInstallerAdmission.js";
import type { BunStandaloneAdmission } from "./bun/BunStandaloneAdmission.js";
import type { DotnetBundleAdmission } from "./dotnet/DotnetBundleAdmission.js";
import type { EspFlashAdmission } from "./EspFlashImageArtifactReader.js";
/** Parsed format facts carried with the exact source that established them. */
export type ArtifactFormatAdmission =
@@ -11,7 +12,8 @@ export type ArtifactFormatAdmission =
| NodeSeaAdmission
| PyInstallerAdmission
| DotnetBundleAdmission
| BunStandaloneAdmission;
| BunStandaloneAdmission
| EspFlashAdmission;
export type ArtifactReaderAdmission =
| StableRegularFileDescriptor
src/artifacts/EspFlashImageArtifactReader.boundaries.test.ts
+19
View File
@@ -0,0 +1,19 @@
import { artifactReaderLifecycleCases } from "../../tests/fixtures/artifactReaderLifecycle.js";
import {
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
} from "../../tests/fixtures/espImage.js";
import { espFlashDump } from "../domain/firmware/espFlash.fixture.js";
import { EspFlashImageArtifactReader } from "./EspFlashImageArtifactReader.js";
artifactReaderLifecycleCases({
image: () =>
espFlashDump({
bootloader: {
at: 0,
bytes: espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS }),
},
}),
createReader: (path, admitted) =>
new EspFlashImageArtifactReader(path, admitted),
});
src/artifacts/EspFlashImageArtifactReader.test.ts
+815
View File
@@ -0,0 +1,815 @@
import { readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { describe, expect, it, vi } from "vitest";
import { EspFlashImageArtifactReader } from "./EspFlashImageArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import type {
ArtifactEntry,
ArtifactReaderTraversal,
} from "./ArtifactReader.js";
import { openRegularFile } from "../filesystem/RegularFile.js";
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
import * as regionReads from "./EspFlashRegionReads.js";
import * as espImageDecoding from "../domain/firmware/espImage.js";
import {
TWO_OTA_PARTITIONS,
espFlashDump,
otaData,
otaRecord,
partitionTable,
} from "../domain/firmware/espFlash.fixture.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
pattern,
} from "../../tests/fixtures/espImage.js";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
import { scanCanonicalArtifactInventory } from "../../tests/fixtures/artifactInventory.js";
const INVENTORY_LIMITATION =
"ESP flash layout inventory reads a dump's bootloader region, partition table and partitions and decodes the bootloader and app partitions as ESP images. The OTA selection is derived without app rollback or anti-rollback, and the running partition is unknown. Partitions, nvs_keys included, are extracted as ordinary data; nothing is decrypted.";
const BOOTLOADER = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
const APP = espImage({ segments: ESP32C3_APP_SEGMENTS });
/** A C3 dump: factory and ota_0 hold the app, ota_1 is erased, otadata selects ota_0. */
const DUMP = espFlashDump({
bootloader: { at: 0, bytes: BOOTLOADER },
parts: [
{ at: 0xd000, bytes: otaData(otaRecord({ seq: 1 })) },
{ at: 0x1_0000, bytes: APP },
{ at: 0x2_0000, bytes: APP },
],
});
const writeDump = async (bytes: Uint8Array, name = "flash.bin") => {
const path = join(await createTestTempDirectory("rea-esp-flash-"), name);
await writeFile(path, bytes);
return path;
};
const root = (
snapshot: Awaited<ReturnType<typeof scanCanonicalArtifactInventory>>,
) =>
snapshot.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata;
const list = async (reader: EspFlashImageArtifactReader) => {
const entries: ArtifactEntry[] = [];
for await (const entry of reader.entries()) entries.push(entry);
return entries;
};
describe("ESP flash layout inventory", () => {
it("inventories the bootloader, table and partitions with app and otadata facts", async () => {
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(DUMP),
);
expect(snapshot.manifest.root_format).toBe("esp-flash-image");
expect(snapshot.coverage).toEqual({
scope: "container-entries",
status: "complete",
});
expect(snapshot.limitations).toEqual([INVENTORY_LIMITATION]);
expect(
snapshot.occurrences.map(({ logical_path }) => logical_path),
).toEqual([
".",
"bootloader.bin",
"partition-table.bin",
"partitions/factory.bin",
"partitions/nvs.bin",
"partitions/ota_0.bin",
"partitions/ota_1.bin",
"partitions/otadata.bin",
"partitions/phy_init.bin",
]);
expect(root(snapshot)).toMatchObject({
kind: "esp-flash-image",
flash_offset: { value: 0, source: "default" },
chip: { target: "esp32c3", isa: "riscv" },
bootloader: {
path: "bootloader.bin",
flash_address: 0,
region_size: 0x8000,
facts: {
status: "esp-image",
image: { descriptor: { kind: "bootloader" } },
coverage: "complete",
},
},
partition_table: {
flash_address: 0x8000,
method: "default",
partition_count: 6,
md5: { status: "valid" },
},
partitions_outside_dump: [],
otadata: { partition_index: 1 },
boot_selection: {
basis: "otadata",
active_record: 0,
slot: 0,
partition_index: 4,
running_partition: "unknown",
},
undescribed: [],
});
const apps = (
root(snapshot) as {
apps: {
path: string;
partition_index: number;
facts: { status: string };
}[];
}
).apps;
// Each image names its entry, so no lookup by index is needed.
expect(
apps.map(({ path, partition_index, facts }) => [
path,
partition_index,
facts.status,
]),
).toEqual([
["partitions/factory.bin", 3, "esp-image"],
["partitions/ota_0.bin", 4, "esp-image"],
["partitions/ota_1.bin", 5, "erased"],
]);
const factory = snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/factory.bin",
);
expect(factory).toMatchObject({
source_path: "factory",
artifact_format: "esp-image",
format_metadata: {
kind: "esp-flash-partition",
index: 3,
label: "factory",
type_name: "app",
subtype_name: "factory",
flash_address: 0x1_0000,
size: 0x1_0000,
},
});
});
it("finds a relocated table by scanning and says the location is derived", async () => {
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(
espFlashDump({
bootloader: { at: 0, bytes: BOOTLOADER },
tableAt: 0xa000,
}),
),
);
expect(root(snapshot)).toMatchObject({
partition_table: { flash_address: 0xa000, method: "scan" },
});
expect(snapshot.limitations).toContain(
"The partition table at 0xa000 was found by scanning 0x1000-aligned flash sectors for a table whose MD5 verifies; the dump does not record the bootloader's configured table offset.",
);
});
});
describe("ESP flash layout coverage and selection", () => {
it("keeps usable partitions beside a truncated and a missing one", async () => {
const cut = DUMP.subarray(0, 0x2_8000);
const snapshot = await scanCanonicalArtifactInventory(await writeDump(cut));
// The table names every partition, so the end of the file hides no entry.
expect(snapshot.coverage.status).toBe("complete");
const ota0 = snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/ota_0.bin",
);
expect(ota0).toMatchObject({
declared_size: 0x8000,
limitations: [
"partitions/ota_0.bin is truncated: the dump holds 32768 of its 65536 bytes.",
],
});
// ota_1 lies past the end: an entry holding none of its bytes.
expect(root(snapshot)).toMatchObject({ partitions_outside_dump: [] });
expect(
snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/ota_1.bin",
),
).toMatchObject({
declared_size: 0,
limitations: [
"partitions/ota_1.bin lies outside the dump: the file holds none of its 65536 bytes.",
],
format_metadata: {
index: 5,
written_offset: null,
written_end: null,
written_bytes: null,
},
});
const { apps } = root(snapshot) as {
apps: { partition_index: number }[];
};
// An app partition the file holds none of has no image facts.
expect(apps.map(({ partition_index: index }) => index)).toEqual([3, 4]);
expect(
snapshot.occurrences.some(
({ logical_path }) => logical_path === "partitions/factory.bin",
),
).toBe(true);
});
it("reports written bytes in an app slot the bootloader finds empty", async () => {
// ota_1's header is erased, but an older image sits 0x8000 into the slot.
const dump = espFlashDump({
bootloader: { at: 0, bytes: BOOTLOADER },
parts: [
{ at: 0xd000, bytes: otaData(otaRecord({ seq: 1 })) },
{ at: 0x1_0000, bytes: APP },
{ at: 0x2_0000, bytes: APP },
{ at: 0x3_8000, bytes: APP },
],
});
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(dump),
);
const indices = [...APP.keys()].filter((index) => APP[index] !== 0xff);
const start = 0x8000 + indices[0]!;
const end = 0x8000 + indices.at(-1)! + 1;
const row = (path: string) =>
snapshot.occurrences.find(({ logical_path }) => logical_path === path)
?.format_metadata;
expect(row("partitions/ota_1.bin")).toMatchObject({
written_offset: start,
written_end: end,
written_bytes: indices.length,
});
const { apps } = root(snapshot) as {
apps: { partition_index: number; facts: { status: string } }[];
};
expect(
apps.find(({ partition_index: index }) => index === 5),
).toMatchObject({
facts: { status: "erased" },
});
// Only the erased app slot holds bytes the bootloader does not read.
expect(
snapshot.limitations.filter((text) => text.includes("finds no app")),
).toEqual([
`partitions/ota_1.bin begins with erased flash, so ESP-IDF's bootloader finds no app in it, but ${String(indices.length)} of its bytes from offset 0x${start.toString(16)} to 0x${end.toString(16)} are not erased flash; they are not read as an image.`,
]);
// A partition erased throughout has no written bytes.
expect(row("partitions/nvs.bin")).toMatchObject({
written_offset: null,
written_end: null,
written_bytes: 0,
});
});
it("reads a dump that starts at the caller's flash offset", async () => {
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
const whole = espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } });
const snapshot = await scanCanonicalArtifactInventory(
await writeDump(whole.subarray(0x1000)),
{
espFlashSelection: { flashOffset: 0x1000 },
},
);
expect(root(snapshot)).toMatchObject({
flash_offset: { value: 0x1000, source: "caller" },
chip: { target: "esp32" },
bootloader: { flash_address: 0x1000, region_size: 0x7000 },
partition_table: { flash_address: 0x8000, method: "default" },
});
const nvs = snapshot.occurrences.find(
({ logical_path }) => logical_path === "partitions/nvs.bin",
);
expect(nvs?.source_location).toEqual({ offset: 0x8000, length: 0x4000 });
});
it("refuses a selected dump without a readable table as not recognized", async () => {
const reader = new EspFlashImageArtifactReader(
await writeDump(pattern(0x2_0000, 7)),
);
try {
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({
reason: "unavailable",
message: expect.stringContaining(
"ESP flash image layout is not recognized",
),
});
} finally {
await reader.close();
}
// The caller's selection decides before another container's signature.
const zipLike = await writeDump(
Buffer.concat([Buffer.from("PK\x03\x04"), pattern(0x1_0000, 3)]),
"dump.zip",
);
await expect(
scanCanonicalArtifactInventory(zipLike, { espFlashSelection: {} }),
).rejects.toThrow(/ESP flash image layout is not recognized/u);
});
});
describe("ESP flash layout reader identity", () => {
it("rejects a changed carrier during extraction preparation", async () => {
const reader = new EspFlashImageArtifactReader(await writeDump(DUMP));
try {
await expect(
reader.prepareContainer("0".repeat(64)),
).rejects.toMatchObject({ reason: "integrity" });
} finally {
await reader.close();
}
});
it("rejects forged entries and changed carrier identity, then closes idempotently", async () => {
const path = await writeDump(DUMP);
const reader = new EspFlashImageArtifactReader(path);
try {
const entry = (await list(reader)).find(
({ path: name }) => name === "partitions/nvs.bin",
);
if (entry === undefined) throw new Error("Missing nvs partition");
await expect(
reader.open({ ...entry, declaredSize: 1 }),
).rejects.toMatchObject({ reason: "integrity" });
await expect(
reader.open({ ...entry, byteOffset: 0 }),
).rejects.toMatchObject({ reason: "integrity" });
await expect(
reader.open({
...entry,
adapterKey: "esp-flash-image:partitions/x.bin",
}),
).rejects.toMatchObject({ reason: "integrity" });
await writeFile(path, DUMP.subarray(0, 0x1_0000));
await expect(reader.open(entry)).rejects.toMatchObject({
reason: "integrity",
});
} finally {
await reader.close();
await reader.close();
}
});
it("settles an active entry stream when closed", async () => {
const reader = new EspFlashImageArtifactReader(await writeDump(DUMP));
try {
const [entry] = await list(reader);
if (entry === undefined) throw new Error("Missing entry");
const stream = await reader.open(entry);
await reader.close();
expect(stream.destroyed).toBe(true);
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({ reason: "unavailable" });
} finally {
await reader.close();
}
});
it("honors cancellation before admitting a file with the exact reason", async () => {
const reader = new EspFlashImageArtifactReader(await writeDump(DUMP));
const controller = new AbortController();
const reason = { stage: "before admission" };
controller.abort(reason);
try {
await expect(
reader.entries(controller.signal)[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({ reason: "cancelled", cause: reason });
} finally {
await reader.close();
}
});
});
describe("ESP flash layout extraction and lifecycle", () => {
it("extracts every partition byte-exactly, nvs_keys included", async () => {
const parts = [
...TWO_OTA_PARTITIONS.slice(0, 3),
{
label: "keys",
type: 0x01,
subtype: 0x04,
offset: 0x1_0000,
size: 0x1000,
flags: 1,
},
];
const keys = pattern(0x1000, 3);
const dump = espFlashDump({
size: 0x1_1000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [{ at: 0x1_0000, bytes: keys }],
});
const directory = await createTestTempDirectory("rea-esp-flash-extract-");
const path = join(directory, "flash.bin");
await writeFile(path, dump);
const resourceScope = new ArtifactResourceScope();
try {
const extracted = await extractArtifact({
inputPath: path,
inputFormat: "esp-flash-image",
outputRoot: join(directory, "out"),
environment: process.env,
integrityPolicy: "fail",
resourceScope,
});
expect(extracted.containment_verified).toBe(true);
expect(
extracted.artifacts.map(({ relative_path }) => relative_path).sort(),
).toEqual([
"bootloader.bin",
"partition-table.bin",
"partitions/keys.bin",
"partitions/nvs.bin",
"partitions/otadata.bin",
"partitions/phy_init.bin",
]);
for (const [name, offset, length] of [
["bootloader.bin", 0, 0x8000],
["partition-table.bin", 0x8000, 0x1000],
["partitions/keys.bin", 0x1_0000, 0x1000],
] as const)
expect(
Buffer.compare(
await readFile(join(extracted.output_root, name)),
dump.subarray(offset, offset + length),
),
).toBe(0);
} finally {
await resourceScope.close();
}
});
it.each(["cancel", "close"] as const)(
"settles app decoding when the caller requests %s after the first decoded chunk",
async (action) => {
const large = espImage({
segments: [{ address: 0x4038_0000, data: Buffer.alloc(8 << 20, 0x5a) }],
});
const parts = [
{
label: "factory",
type: 0,
subtype: 0,
offset: 0x1_0000,
size: 0x90_0000,
},
];
const reader = new EspFlashImageArtifactReader(
await writeDump(
espFlashDump({
size: 0x91_0000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [{ at: 0x1_0000, bytes: large }],
}),
),
);
const controller = new AbortController();
const reason = { stage: "decoding" };
let completion: Promise<void> | undefined;
let decoded = 0;
let acknowledge = (): void => undefined;
const decoding = new Promise<void>((resolve) => {
acknowledge = resolve;
});
const create = espImageDecoding.createEspImageDecoder;
// The bootloader decodes in one chunk; acknowledge the app's second chunk.
const observer = vi
.spyOn(espImageDecoding, "createEspImageDecoder")
.mockImplementation((size) => {
const decoder = create(size);
let writes = 0;
return {
write: (chunk) => {
decoder.write(chunk);
decoded += 1;
writes += 1;
if (size > 0x10_0000 && writes === 1) acknowledge();
},
get satisfied() {
return decoder.satisfied;
},
finish: () => decoder.finish(),
};
});
try {
const pending = reader
.entries(controller.signal)
[Symbol.asyncIterator]()
.next();
await Promise.race([
decoding,
pending.then(() => {
throw new Error("Preparation settled before app decoding began");
}),
]);
const requestedAt = decoded;
if (action === "cancel") controller.abort(reason);
else completion = reader.close();
await expect(pending).rejects.toMatchObject(
action === "cancel"
? { reason: "cancelled", cause: reason }
: { reason: "unavailable" },
);
await completion;
expect(decoded).toBe(requestedAt);
await reader.close();
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({
reason: "unavailable",
});
} finally {
observer.mockRestore();
await reader.close();
}
},
);
});
describe("ESP flash layout written-bytes scan lifecycle", () => {
it.each(["cancel", "close"] as const)(
"settles the written-bytes scan when the caller requests %s after its first chunk",
async (action) => {
// Only the written-bytes scan reads a data partition's bytes.
const parts = [
{
label: "storage",
type: 1,
subtype: 0x81,
offset: 0x1_0000,
size: 0x90_0000,
},
];
const reader = new EspFlashImageArtifactReader(
await writeDump(
espFlashDump({
size: 0x91_0000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [{ at: 0x1_0000, bytes: Buffer.alloc(0x90_0000, 0x5a) }],
}),
),
);
const controller = new AbortController();
const reason = { stage: "written-bytes scan" };
let completion: Promise<void> | undefined;
let scanned = 0;
let acknowledge = (): void => undefined;
const scanning = new Promise<void>((resolve) => {
acknowledge = resolve;
});
const scan = regionReads.writtenExtents;
// Count the chunks the scan examines once the reader admits them.
const observer = vi
.spyOn(regionReads, "writtenExtents")
.mockImplementation((context, regions, assertOpen) =>
scan(context, regions, () => {
assertOpen();
if (!regions.some(({ length }) => length >= 0x10_0000)) return;
scanned += 1;
if (scanned === 1) acknowledge();
}),
);
try {
const pending = reader
.entries(controller.signal)
[Symbol.asyncIterator]()
.next();
await Promise.race([
scanning,
pending.then(() => {
throw new Error("Preparation settled before the scan began");
}),
]);
const requestedAt = scanned;
if (action === "cancel") controller.abort(reason);
else completion = reader.close();
await expect(pending).rejects.toMatchObject(
action === "cancel"
? { reason: "cancelled", cause: reason }
: { reason: "unavailable" },
);
await completion;
expect(scanned).toBe(requestedAt);
await reader.close();
await expect(
reader.entries()[Symbol.asyncIterator]().next(),
).rejects.toMatchObject({
reason: "unavailable",
});
} finally {
observer.mockRestore();
await reader.close();
}
},
);
});
describe("ESP flash layout erased-gap scan lifecycle", () => {
it.each(["cancel", "close"] as const)(
"settles the erased-gap scan when the caller requests %s after its first chunk",
async (action) => {
// One small partition leaves the rest of a 9 MiB dump as one erased gap.
const parts = [
{ label: "nvs", type: 1, subtype: 2, offset: 0x9000, size: 0x1000 },
];
const reader = new EspFlashImageArtifactReader(
await writeDump(
espFlashDump({
size: 0x90_0000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
}),
),
);
const controller = new AbortController();
const reason = { stage: "erased-gap scan" };
let completion: Promise<void> | undefined;
let scanned = 0;
let acknowledge = (): void => undefined;
const scanning = new Promise<void>((resolve) => {
acknowledge = resolve;
});
const scan = regionReads.firstUnerasedInRange;
// Count the chunks the scan examines once the reader admits them.
const observer = vi
.spyOn(regionReads, "firstUnerasedInRange")
.mockImplementation((context, range, assertOpen) =>
scan(context, range, () => {
assertOpen();
if (range.length < 0x10_0000) return;
scanned += 1;
if (scanned === 1) acknowledge();
}),
);
try {
const pending = reader
.entries(controller.signal)
[Symbol.asyncIterator]()
.next();
await Promise.race([
scanning,
pending.then(() => {
throw new Error("Preparation settled before the scan began");
}),
]);
const requestedAt = scanned;
if (action === "cancel") controller.abort(reason);
else completion = reader.close();
await expect(pending).rejects.toMatchObject(
action === "cancel"
? { reason: "cancelled", cause: reason }
: { reason: "unavailable" },
);
await completion;
expect(scanned).toBe(requestedAt);
} finally {
observer.mockRestore();
await reader.close();
}
},
);
});
describe("ESP flash layout written bytes of overlapping partitions", () => {
it("reads the bytes they cover once and keeps each partition's own facts", async () => {
// 60 data partitions over one MiB: 40 cover all of it, 20 its first half.
const parts = Array.from({ length: 60 }, (_, index) => ({
label: `p${String(index)}`,
type: 1,
subtype: 0x81,
offset: 0x1_0000,
size: index < 40 ? 0x10_0000 : 0x8_0000,
}));
const bytes = espFlashDump({
size: 0x11_0000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [
{ at: 0x1_0100, bytes: Buffer.alloc(0x100, 0x5a) },
{ at: 0x9_0000, bytes: Buffer.alloc(0x100, 0x5a) },
],
});
const path = await writeDump(bytes);
const admitted = await openRegularFile(path, { symlinks: "reject" });
const nativeRead = admitted.handle.read.bind(admitted.handle);
let read = 0;
const reads = vi.spyOn(admitted.handle, "read").mockImplementation((async (
...args: unknown[]
) => {
const result = (await Reflect.apply(
nativeRead,
admitted.handle,
args,
)) as {
bytesRead: number;
};
read += result.bytesRead;
return result;
}) as typeof admitted.handle.read);
const reader = new EspFlashImageArtifactReader(path, admitted);
try {
let coverage: ArtifactReaderTraversal | undefined;
const entries: ArtifactEntry[] = [];
for await (const entry of reader.entries(undefined, {
onCoverage: (value) => {
coverage = value;
},
}))
entries.push(entry);
// Each covered byte is read once, not once per partition over it.
expect(read).toBeLessThan(2 * bytes.length);
expect(
entries.flatMap(({ path: entryPath, formatMetadata }) =>
formatMetadata?.kind === "esp-flash-partition"
? [[entryPath, formatMetadata]]
: [],
),
).toEqual(
parts.map((part, index) => [
`partitions/${part.label}.bin`,
expect.objectContaining({
index,
label: part.label,
flash_address: 0x1_0000,
size: part.size,
written_offset: 0x100,
written_end: index < 40 ? 0x8_0100 : 0x200,
written_bytes: index < 40 ? 0x200 : 0x100,
}),
]),
);
expect(
coverage?.limitations.filter((text) => text.includes(" overlaps ")),
).toHaveLength(59);
} finally {
await reader.close();
reads.mockRestore();
await admitted.handle.close();
}
});
});
describe("ESP flash layout app images of overlapping partitions", () => {
it("decodes each declared range once and keeps each partition's facts", async () => {
// 60 app partitions over one image: 40 hold all of it, 20 cut it short.
const image = espImage({
segments: [{ address: 0x3c00_0020, data: Buffer.alloc(0xf_0000, 0x5a) }],
});
const parts = Array.from({ length: 60 }, (_, index) => ({
label: `a${String(index)}`,
type: 0,
subtype: 0x10 + (index % 16),
offset: 0x1_0000,
size: index < 40 ? 0x10_0000 : 0x8_0000,
}));
const reader = new EspFlashImageArtifactReader(
await writeDump(
espFlashDump({
size: 0x11_0000,
bootloader: { at: 0, bytes: BOOTLOADER },
table: partitionTable(parts),
parts: [{ at: 0x1_0000, bytes: image }],
}),
),
);
const decode = vi.spyOn(regionReads, "decodeEspFlashImage");
try {
await list(reader);
// The bootloader, the whole image and the image cut at 0x80000.
expect(decode).toHaveBeenCalledTimes(3);
const apps = reader.formatMetadata()?.apps ?? [];
expect(
apps.map(({ path, partition_index: index, facts }) => [
path,
index,
facts.status,
facts.coverage,
]),
).toEqual(
parts.map((part, index) => [
`partitions/${part.label}.bin`,
index,
"esp-image",
index < 40 ? "complete" : "unknown",
]),
);
} finally {
decode.mockRestore();
await reader.close();
}
});
});
src/artifacts/EspFlashImageArtifactReader.ts
+537
View File
@@ -0,0 +1,537 @@
import { Readable, Transform, pipeline } from "node:stream";
import type { FileHandle } from "node:fs/promises";
import type { EspFlashImageMetadata } from "../domain/artifactFormatMetadata.js";
import { ESP_CHIP_TABLE_SOURCE } from "../domain/firmware/espChips.js";
import {
espErasedSlotLimitation,
espFlashPartitionFacts,
finishEspFlashLayout,
planEspFlashLayout,
type EspFlashLayoutFacts,
type EspFlashPlan,
type EspFlashRegion,
type EspFlashWrittenExtent,
} from "../domain/firmware/espFlashLayout.js";
import {
locateEspFlashLayout,
type EspFlashLocation,
type EspFlashSelection,
} from "../domain/firmware/espFlashLocation.js";
import { offsetText } from "../domain/firmware/espImageStructures.js";
import { ESP_OTA_SELECT_ENTRY_BYTES } from "../domain/firmware/espOtaData.js";
import { isEspAppPartition } from "../domain/firmware/espPartitionTable.js";
import {
ArtifactReaderFailure,
copyArtifactEntry,
sameArtifactEntry,
type ArtifactEntry,
type ArtifactReader,
type ArtifactReaderTraversal,
type ArtifactTraversalOptions,
} from "./ArtifactReader.js";
import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import { assertArtifactReadActive } from "./ArtifactFileOperation.js";
import { ArtifactFileSource } from "./ArtifactFileSource.js";
import {
decodeEspFlashImage,
firstUnerasedInRange,
readEspFlashRange,
writtenExtents,
type EspFlashReadContext,
} from "./EspFlashRegionReads.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import type { StableRegularFileDescriptor } from "../filesystem/RegularFile.js";
const LABEL = "ESP flash image";
/** A located layout borrowed with the descriptor that produced it. */
export interface EspFlashAdmission {
readonly kind: "esp-flash-image";
readonly source: StableRegularFileDescriptor;
readonly location: EspFlashLocation;
}
type EspFlashImageFacts = NonNullable<
EspFlashImageMetadata["bootloader"]
>["facts"];
/**
* Inventory an ESP flash dump as its bootloader region, partition table and
* partitions. The bootloader and app partitions are decoded as ESP images;
* each entry is a positional read of the carrier, rechecked against its
* identity.
*/
export class EspFlashImageArtifactReader implements ArtifactReader {
readonly format = "esp-flash-image" as const;
readonly #scope = new ArtifactResourceScope();
readonly #entries = new Map<string, ArtifactEntry>();
readonly #streams = new Set<Readable>();
readonly #source: ArtifactFileSource;
readonly #located: EspFlashLocation | undefined;
readonly #selection: EspFlashSelection;
#prepared: Promise<void> | undefined;
#metadata: EspFlashImageMetadata | undefined;
#coverage: ArtifactReaderTraversal | undefined;
#closing = false;
constructor(
path: string,
admitted?: EspFlashAdmission | StableRegularFileDescriptor,
selection: EspFlashSelection = {},
) {
const admission =
admitted !== undefined && "kind" in admitted ? admitted : undefined;
this.#source = new ArtifactFileSource(
path,
LABEL,
admission?.source ??
(admitted as StableRegularFileDescriptor | undefined),
);
this.#located = admission?.location;
this.#selection = selection;
}
formatMetadata(): EspFlashImageMetadata | undefined {
return this.#metadata === undefined
? undefined
: structuredClone(this.#metadata);
}
async prepareContainer(
expectedSha256: string,
signal?: AbortSignal,
): Promise<void> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
await this.#source.bindSha256(expectedSha256, signal);
});
}
async *entries(
signal?: AbortSignal,
options?: ArtifactTraversalOptions,
): AsyncIterable<ArtifactEntry> {
await this.#scope.run(async () => {
this.#assertOpen(signal);
this.#prepared ??= this.#prepare(signal);
await this.#prepared;
await this.#source.read(signal);
});
this.#assertOpen(signal);
if (this.#coverage !== undefined)
options?.onCoverage?.({
...this.#coverage,
limitations: [...this.#coverage.limitations],
});
for (const entry of this.#entries.values()) {
this.#assertOpen(signal);
yield copyArtifactEntry(entry);
}
this.#assertOpen(signal);
}
async open(entry: ArtifactEntry, signal?: AbortSignal): Promise<Readable> {
return this.#scope.run(async () => {
this.#assertOpen(signal);
const stored = this.#entries.get(entry.adapterKey);
if (
stored === undefined ||
!sameArtifactEntry(entry, stored) ||
stored.byteOffset === null ||
stored.declaredSize === null
)
throw new ArtifactReaderFailure(
"integrity",
`${LABEL} entry does not match its owning inventory`,
);
const { handle } = await this.#source.read(signal);
this.#assertOpen(signal);
const source =
stored.declaredSize === 0
? Readable.from([])
: readFileHandleChunks(handle, {
start: stored.byteOffset,
end: stored.byteOffset + stored.declaredSize - 1,
signal,
cancellationFailure: (reason) =>
new ArtifactReaderFailure(
"cancelled",
`${LABEL} read cancelled`,
{ cause: reason },
),
});
const output = this.#entryStream(stored, signal);
// Retirement can destroy the stream before its consumer listens.
output.on("error", () => undefined);
this.#streams.add(output);
let producerComplete = false;
const onAbort = (): void => {
// Preserve active read/validation errors; after production, the caller
// still owns buffered bytes and cancellation must retire that result.
if (!producerComplete || output.destroyed) return;
output.destroy(
new ArtifactReaderFailure("cancelled", `${LABEL} read cancelled`, {
cause: signal?.reason,
}),
);
};
const retired = new Promise<void>((resolve) => {
output.once("close", () => {
this.#streams.delete(output);
signal?.removeEventListener("abort", onAbort);
resolve();
});
});
signal?.addEventListener("abort", onAbort, { once: true });
if (signal?.aborted) onAbort();
void this.#scope
.run(async () => {
await Promise.all([
new Promise<void>((resolve) => {
pipeline(source, output, () => {
producerComplete = true;
if (signal?.aborted) onAbort();
resolve();
});
}),
retired,
]);
})
.catch((cause: unknown) =>
output.destroy(
cause instanceof Error
? cause
: new Error(`${LABEL} stream ownership failed`, { cause }),
),
);
return output;
});
}
async close(): Promise<void> {
this.#closing = true;
for (const stream of this.#streams)
stream.destroy(
new ArtifactReaderFailure("unavailable", `${LABEL} reader closed`),
);
await this.#scope.close();
await this.#source.close();
this.#entries.clear();
}
provenance(): readonly [] {
return [];
}
#assertOpen(signal?: AbortSignal): void {
assertArtifactReadActive(signal);
if (this.#closing)
throw new ArtifactReaderFailure(
"unavailable",
`${LABEL} reader is closed`,
);
}
async #prepare(signal?: AbortSignal): Promise<void> {
const { handle, initial } = await this.#source.read(signal);
const fileSize = initial.size;
const location =
this.#located ?? (await this.#locate(handle, fileSize, signal));
const plan = planEspFlashLayout({
fileSize,
origin: location.origin,
bootloader: location.bootloader,
tableAddress: location.tableAddress,
tableMethod: location.tableMethod,
table: location.table,
});
const context: EspFlashReadContext = { handle, fileSize, signal };
const facts = await this.#readFacts(context, plan);
const images = await this.#decodeImages(context, plan);
const written = await this.#readWritten(context, plan);
// A changed carrier is an identity failure, not a different layout.
await this.#source.read(signal);
for (const region of plan.regions) {
const entry = regionEntry(region, written.get(region), fileSize);
this.#entries.set(entry.adapterKey, entry);
}
this.#metadata = layoutMetadata(location, plan, facts, images);
const limitations = [
...location.limitations,
...(location.tableMethod === "scan"
? [
`The partition table at ${offsetText(location.tableAddress)} was found by scanning 0x1000-aligned flash sectors for a table whose MD5 verifies; the dump does not record the bootloader's configured table offset.`,
]
: []),
...facts.limitations,
...erasedSlotLimitations(images, written),
];
if (facts.coverage === "unknown" || limitations.length > 0)
this.#coverage = { status: facts.coverage, limitations };
}
async #locate(
handle: FileHandle,
fileSize: number,
signal?: AbortSignal,
): Promise<EspFlashLocation> {
const located = await locateEspFlashLayout(
fileSize,
(position, length) =>
readEspFlashRange({ handle, fileSize, signal }, position, length),
this.#selection,
{ explicit: true, ...(signal === undefined ? {} : { signal }) },
);
if (!located.ok)
throw new ArtifactReaderFailure(
"unavailable",
`${LABEL} layout is not recognized: ${located.error}`,
);
return located.value;
}
async #readFacts(
context: EspFlashReadContext,
plan: EspFlashPlan,
): Promise<EspFlashLayoutFacts> {
const otadataRecords: (Uint8Array | undefined)[] = [];
for (const { fileOffset, available } of plan.otadata?.records ?? [])
otadataRecords.push(
available
? await readEspFlashRange(
context,
fileOffset,
ESP_OTA_SELECT_ENTRY_BYTES,
)
: undefined,
);
const gapFirstUnerased: (number | undefined)[] = [];
for (const gap of plan.gaps) {
this.#assertOpen(context.signal);
gapFirstUnerased.push(
await firstUnerasedInRange(
context,
{ start: gap.offset, length: gap.length },
() => this.#assertOpen(context.signal),
),
);
}
return finishEspFlashLayout(plan, { otadataRecords, gapFirstUnerased });
}
/**
* Where each partition the dump holds is not erased flash, from one read
* of the bytes the partitions cover, however many overlap.
*/
async #readWritten(
context: EspFlashReadContext,
plan: EspFlashPlan,
): Promise<ReadonlyMap<EspFlashRegion, EspFlashWrittenExtent>> {
// A partition the dump holds none of has nothing to observe.
const held = plan.regions.filter(
({ partition, available }) => partition !== undefined && available > 0,
);
this.#assertOpen(context.signal);
const extents = await writtenExtents(
context,
held.map(({ fileOffset, available }) => ({
start: fileOffset,
length: available,
})),
() => this.#assertOpen(context.signal),
);
return new Map(held.map((region, index) => [region, extents[index]!]));
}
/**
* ESP image facts for the bootloader region and each app partition.
* Regions over the same bytes share one decode, however many the table
* declares.
*/
async #decodeImages(
context: EspFlashReadContext,
plan: EspFlashPlan,
): Promise<ReadonlyMap<EspFlashRegion, EspFlashImageFacts>> {
const images = new Map<EspFlashRegion, EspFlashImageFacts>();
const decoded = new Map<string, EspFlashImageFacts>();
for (const region of plan.regions) {
const image =
region.role === "bootloader" ||
(region.partition !== undefined && isEspAppPartition(region.partition));
if (!image || region.available === 0) continue;
this.#assertOpen(context.signal);
const range = `${String(region.fileOffset)}+${String(region.available)}`;
let facts = decoded.get(range);
if (facts === undefined) {
facts = await decodeEspFlashImage(
context,
{ start: region.fileOffset, length: region.available },
() => this.#assertOpen(context.signal),
);
decoded.set(range, facts);
}
// Each region owns its facts, so no metadata aliases another's.
images.set(region, structuredClone(facts));
}
return images;
}
#entryStream(entry: ArtifactEntry, signal?: AbortSignal): Transform {
let bytes = 0;
const check = async (): Promise<void> => {
if (bytes !== entry.declaredSize)
throw new ArtifactReaderFailure(
"integrity",
`${LABEL} entry ended early: ${entry.path}`,
);
await this.#source.read(signal);
};
return new Transform({
transform(chunk: Buffer, _encoding, callback) {
bytes += chunk.length;
callback(null, chunk);
},
flush(callback) {
void check().then(
() => callback(),
(cause: unknown) =>
callback(
cause instanceof Error
? cause
: new Error(`${LABEL} read failed`, { cause }),
),
);
},
});
}
}
/** App slots the bootloader finds empty that still hold written bytes. */
const erasedSlotLimitations = (
images: ReadonlyMap<EspFlashRegion, EspFlashImageFacts>,
written: ReadonlyMap<EspFlashRegion, EspFlashWrittenExtent>,
): string[] =>
[...images].flatMap(([region, image]) => {
const extent = written.get(region);
const limitation =
region.partition === undefined ||
image.status !== "erased" ||
extent === undefined
? undefined
: espErasedSlotLimitation(region.path, extent);
return limitation === undefined ? [] : [limitation];
});
/** One positional entry; a partition's declared flags stay metadata. */
const regionEntry = (
region: EspFlashRegion,
written: EspFlashWrittenExtent | undefined,
fileSize: number,
): ArtifactEntry => ({
path: region.path,
...(region.partition?.label === null || region.partition === undefined
? {}
: { sourcePath: region.partition.label }),
kind: "file",
declaredSize: region.available,
compressedSize: null,
executable: false,
// The table's encrypted flag says how a device stores the partition, not
// that these bytes are ciphertext.
encrypted: false,
// A partition past the end holds no bytes; its entry sits at the end.
byteOffset: Math.min(region.fileOffset, fileSize),
declaredIntegrity: null,
unpacked: false,
limitations:
region.available === 0
? [
`${region.path} lies outside the dump: the file holds none of its ${String(region.length)} bytes.`,
]
: region.available < region.length
? [
`${region.path} is truncated: the dump holds ${String(region.available)} of its ${String(region.length)} bytes.`,
]
: [],
adapterKey: `esp-flash-image:${region.path}`,
...(region.partition === undefined
? {}
: { formatMetadata: espFlashPartitionFacts(region.partition, written) }),
});
const layoutMetadata = (
location: EspFlashLocation,
plan: EspFlashPlan,
facts: EspFlashLayoutFacts,
images: ReadonlyMap<EspFlashRegion, EspFlashImageFacts>,
): EspFlashImageMetadata => {
const bootloaderRegion = plan.regions.find(
({ role }) => role === "bootloader",
);
const bootloaderFacts =
bootloaderRegion === undefined ? undefined : images.get(bootloaderRegion);
const placed = new Set(
plan.regions.flatMap(({ partition }) =>
partition === undefined ? [] : [partition.index],
),
);
const { bootloader, table } = location;
return {
kind: "esp-flash-image",
flash_offset: location.origin,
chip:
bootloader === undefined
? null
: {
target: bootloader.chip.target,
isa: bootloader.chip.isa,
source: ESP_CHIP_TABLE_SOURCE,
},
bootloader:
bootloaderRegion === undefined || bootloaderFacts === undefined
? null
: {
path: bootloaderRegion.path,
flash_address: bootloaderRegion.flashAddress,
region_size: bootloaderRegion.length,
facts: bootloaderFacts,
},
partition_table: {
flash_address: location.tableAddress,
method: location.tableMethod,
partition_count: table.partitions.length,
md5: {
status: table.md5.status,
stored: table.md5.stored,
computed: table.md5.computed,
},
},
partitions_outside_dump: table.partitions
.filter(({ index }) => !placed.has(index))
.map((part) => espFlashPartitionFacts(part)),
apps: [...images].flatMap(([region, image]) =>
region.partition === undefined
? []
: [
{
path: region.path,
partition_index: region.partition.index,
facts: image,
},
],
),
otadata:
facts.otadata === null
? null
: {
partition_index: facts.otadata.partition_index,
records: facts.otadata.records.map((record) => ({ ...record })),
},
boot_selection: {
...facts.boot_selection,
assumptions: [...facts.boot_selection.assumptions],
},
undescribed: [...facts.undescribed],
};
};
src/artifacts/EspFlashRegionReads.ts
+243
View File
@@ -0,0 +1,243 @@
import type { FileHandle } from "node:fs/promises";
import type { EspFlashImageMetadata } from "../domain/artifactFormatMetadata.js";
import type { EspFlashWrittenExtent } from "../domain/firmware/espFlashLayout.js";
import {
ESP8266_REFUSAL,
createEspImageDecoder,
recognizeEspImage,
} from "../domain/firmware/espImage.js";
import { firstUnerased } from "../domain/firmware/espImageStructures.js";
import { readFileHandleChunks } from "../filesystem/readFileHandleChunks.js";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
const LABEL = "ESP flash image";
const HEADER_BYTES = 24;
const PROBE_BYTES = 32;
/** The carrier being read, its size and the caller's cancellation. */
export interface EspFlashReadContext {
readonly handle: FileHandle;
readonly fileSize: number;
readonly signal?: AbortSignal | undefined;
}
type EspFlashImageFacts = NonNullable<
EspFlashImageMetadata["bootloader"]
>["facts"];
const chunks = (
{ handle, signal }: EspFlashReadContext,
start: number,
length: number,
): AsyncIterable<Buffer> =>
readFileHandleChunks(handle, {
start,
end: start + length - 1,
signal,
cancellationFailure: (reason) =>
new ArtifactReaderFailure("cancelled", `${LABEL} read cancelled`, {
cause: reason,
}),
}) as AsyncIterable<Buffer>;
/** Up to `length` bytes at a file offset; fewer at the end of the file. */
export const readEspFlashRange = async (
context: EspFlashReadContext,
position: number,
length: number,
): Promise<Buffer> => {
const available = Math.min(length, context.fileSize - position);
if (position < 0 || available <= 0) return Buffer.alloc(0);
const parts: Buffer[] = [];
for await (const chunk of chunks(context, position, available))
parts.push(chunk);
return Buffer.concat(parts);
};
/** The file offset of the first byte in a range that is not erased flash. */
export const firstUnerasedInRange = async (
context: EspFlashReadContext,
range: { readonly start: number; readonly length: number },
assertOpen: () => void,
): Promise<number | undefined> => {
if (range.length <= 0) return undefined;
let position = range.start;
for await (const chunk of chunks(context, range.start, range.length)) {
assertOpen();
const found = firstUnerased(chunk, position);
if (found !== undefined) return found;
position += chunk.length;
}
return undefined;
};
type FileRange = { readonly start: number; readonly length: number };
/** Bytes between two adjacent region boundaries that are not erased flash. */
interface WrittenSegment {
readonly start: number;
readonly end: number;
/** File offsets of the first and last byte that is not `0xFF`, or -1. */
first: number;
last: number;
bytes: number;
}
/** The segments region boundaries cut, kept where some region covers them. */
const coveredSegments = (ranges: readonly FileRange[]): WrittenSegment[] => {
const cuts = [
...new Set(ranges.flatMap(({ start, length }) => [start, start + length])),
].sort((left, right) => left - right);
return cuts.slice(1).flatMap((end, index) => {
const start = cuts[index]!;
const covered = ranges.some(
(range) => range.start <= start && end <= range.start + range.length,
);
return covered ? [{ start, end, first: -1, last: -1, bytes: 0 }] : [];
});
};
const tally = (
segment: WrittenSegment,
bytes: Uint8Array,
position: number,
): void => {
for (let index = 0; index < bytes.length; index += 1)
if (bytes[index] !== 0xff) {
segment.bytes += 1;
segment.last = position + index;
if (segment.first < 0) segment.first = segment.last;
}
};
/** Read adjacent segments as one range, tallying each byte in its segment. */
const readSegments = async (
context: EspFlashReadContext,
segments: readonly WrittenSegment[],
assertOpen: () => void,
): Promise<void> => {
const start = segments[0]!.start;
let position = start;
let current = 0;
for await (const chunk of chunks(
context,
start,
segments.at(-1)!.end - start,
)) {
assertOpen();
let offset = 0;
while (offset < chunk.length) {
const segment = segments[current]!;
const taken = Math.min(chunk.length - offset, segment.end - position);
tally(segment, chunk.subarray(offset, offset + taken), position);
offset += taken;
position += taken;
if (position === segment.end) current += 1;
}
}
};
/** One region's extent from the segments inside it, relative to its start. */
const regionExtent = (
{ start, length }: FileRange,
segments: readonly WrittenSegment[],
): EspFlashWrittenExtent => {
let first = -1;
let last = -1;
let bytes = 0;
for (const segment of segments) {
if (segment.start < start || segment.end > start + length) continue;
bytes += segment.bytes;
if (first < 0) first = segment.first;
last = Math.max(last, segment.last);
}
return first < 0
? { offset: null, end: null, bytes: 0 }
: { offset: first - start, end: last + 1 - start, bytes };
};
/**
* Where each region is not erased, reading the bytes the regions cover once.
* Region boundaries cut those bytes into segments, so overlapping regions
* share their reads however many the table declares.
*/
export const writtenExtents = async (
context: EspFlashReadContext,
regions: readonly FileRange[],
assertOpen: () => void,
): Promise<EspFlashWrittenExtent[]> => {
const segments = coveredSegments(regions.filter(({ length }) => length > 0));
let run = 0;
for (let index = 1; index <= segments.length; index += 1)
if (
index === segments.length ||
segments[index]!.start !== segments[index - 1]!.end
) {
await readSegments(context, segments.slice(run, index), assertOpen);
run = index;
}
return regions.map((region) => regionExtent(region, segments));
};
const noImage = (
status: "erased" | "unrecognized",
limitations: readonly string[],
): EspFlashImageFacts => ({
status,
image: null,
segments: [],
signature_blocks: [],
coverage: null,
limitations: [...limitations],
});
/**
* Decode an ESP image at the start of a region, as the bootloader reads a
* partition: one streaming pass with the ESP image decoder. Offsets in the
* facts are relative to the region; `erased` means the header bytes are
* erased flash.
*/
export const decodeEspFlashImage = async (
context: EspFlashReadContext,
region: { readonly start: number; readonly length: number },
assertOpen: () => void,
): Promise<EspFlashImageFacts> => {
// The region's end bounds the read, so a short region stays short.
const prefix = await readEspFlashRange(
{ ...context, fileSize: region.start + region.length },
region.start,
PROBE_BYTES,
);
// A region shorter than a header is erased when all of its bytes are.
if (
prefix.length > 0 &&
prefix.subarray(0, HEADER_BYTES).every((byte) => byte === 0xff)
)
return noImage("erased", []);
const recognized = recognizeEspImage(prefix, region.length);
if (recognized === "esp8266")
return noImage("unrecognized", [ESP8266_REFUSAL]);
if (recognized !== "esp-image")
return noImage("unrecognized", [
"The region does not begin with a recognized ESP image header; it may be encrypted or hold other data.",
]);
const decoder = createEspImageDecoder(region.length);
for await (const chunk of chunks(context, region.start, region.length)) {
assertOpen();
decoder.write(chunk);
if (decoder.satisfied) break;
}
const image = decoder.finish();
if (!image.ok) return noImage("unrecognized", [image.error.message]);
return {
status: "esp-image",
image: image.value.metadata,
segments: image.value.segments.map(({ metadata }) => ({ ...metadata })),
signature_blocks: image.value.signatureBlocks.map(({ metadata }) => ({
...metadata,
})),
coverage: image.value.coverage,
limitations: [...image.value.limitations],
};
};
src/artifacts/EspImageArtifactReader.test.ts
+68
View File
@@ -9,10 +9,17 @@ import { ArtifactResourceScope } from "./ArtifactResourceScope.js";
import type { ArtifactEntry } from "./ArtifactReader.js";
import { extractArtifact } from "./extraction/ArtifactExtraction.js";
import * as espImageDecoding from "../domain/firmware/espImage.js";
import {
TWO_OTA_PARTITIONS,
flashImage,
partitionTable,
} from "../domain/firmware/espFlash.fixture.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
pattern,
signedEspImage,
} from "../../tests/fixtures/espImage.js";
import { createTestTempDirectory } from "../../tests/fixtures/temporaryDirectory.js";
@@ -214,6 +221,67 @@ it("extracts byte-exact segments that re-inventory as raw files", async () => {
}
});
const rootMetadata = async (bytes: Uint8Array) =>
(
await scanCanonicalArtifactInventory(await writeImage(bytes))
).occurrences.find((row) => row.logical_path === ".")?.format_metadata;
const bootloaderOf = (chipId: number) =>
espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS, chipId });
const TABLE = partitionTable(TWO_OTA_PARTITIONS);
it("says when an image is the bootloader of a flash dump read from a later address", async () => {
// ESP32 and S2 ROMs load the bootloader from 0x1000, so a dump read from
// there has the table at file 0x7000; a C5's ROM loads from 0x2000.
for (const [chipId, flashOffset] of [
[0x0000, 0x1000],
[0x0017, 0x2000],
] as const) {
const tableFileOffset = 0x8000 - flashOffset;
expect(
await rootMetadata(
flashImage(0x8000, [
{ at: 0, bytes: bootloaderOf(chipId) },
{ at: tableFileOffset, bytes: TABLE },
]),
),
).toMatchObject({
flash_dump: {
flash_offset: flashOffset,
table_file_offset: tableFileOffset,
},
});
}
});
it("reads no flash dump without a table after a bootloader", async () => {
const esp32 = bootloaderOf(0x0000);
// A table inside the image's segment is segment data.
const data = Buffer.alloc(0x7400);
TABLE.copy(data, 0x7000 - 0x20);
const spanning = espImage({
segments: [{ address: 0x3ffe_0000, data }],
chipId: 0x0000,
});
for (const bytes of [
esp32,
// Bytes that begin like a table but are not one.
flashImage(0x8000, [
{ at: 0, bytes: esp32 },
{
at: 0x7000,
bytes: Buffer.concat([TABLE.subarray(0, 32), pattern(32, 9)]),
},
]),
// An app carries esp_app_desc_t, so it is no bootloader.
flashImage(0x8000, [
{ at: 0, bytes: espImage({ segments: ESP32C3_APP_SEGMENTS, chipId: 0 }) },
{ at: 0x7000, bytes: TABLE },
]),
spanning,
])
expect(await rootMetadata(bytes)).toMatchObject({ flash_dump: null });
});
it("refuses an ESP8266 image with its specific reason", async () => {
const reader = new EspImageArtifactReader(
await writeImage(esp8266Image(), "esp8266.bin"),
src/artifacts/EspImageArtifactReader.ts
+58 -7
View File
@@ -5,6 +5,11 @@ import type {
EspImageSegmentMetadata,
EspImageSignatureBlockMetadata,
} from "../domain/artifactFormatMetadata.js";
import { espChipForId } from "../domain/firmware/espChips.js";
import {
espFlashDumpOrigin,
type EspFlashDumpOrigin,
} from "../domain/firmware/espFlashLocation.js";
import {
createEspImageDecoder,
type EspImageFailure,
@@ -13,7 +18,11 @@ import {
ESP_IMAGE_LIMITATIONS,
type EspImage,
} from "../domain/firmware/espImageFacts.js";
import { ESP_SIGNATURE_BLOCK_BYTES } from "../domain/firmware/espImageStructures.js";
import {
ESP_SIGNATURE_BLOCK_BYTES,
offsetText,
} from "../domain/firmware/espImageStructures.js";
import { readEspFlashRange } from "./EspFlashRegionReads.js";
import {
ArtifactReaderFailure,
copyArtifactEntry,
@@ -240,12 +249,48 @@ export class EspImageArtifactReader implements ArtifactReader {
} satisfies EspImageEntry;
this.#entries.set(entry.adapterKey, entry);
}
this.#metadata = image.metadata;
if (image.coverage === "unknown" || image.limitations.length > 0)
this.#coverage = {
status: image.coverage,
limitations: [...image.limitations],
};
const dump = await this.#flashDump(image, signal);
this.#metadata = {
...image.metadata,
flash_dump:
dump === undefined
? null
: {
flash_offset: dump.flashOffset,
table_file_offset: dump.tableFileOffset,
},
};
const limitations = [
...image.limitations,
...(dump === undefined ? [] : [flashDumpLimitation(image, dump)]),
];
if (image.coverage === "unknown" || limitations.length > 0)
this.#coverage = { status: image.coverage, limitations };
}
/**
* Whether this image is the bootloader at the start of a flash dump read
* from a later flash address. An app carries `esp_app_desc_t` and is not.
*/
async #flashDump(
image: EspImage,
signal?: AbortSignal,
): Promise<EspFlashDumpOrigin | undefined> {
const chip = espChipForId(image.metadata.header.chip_id);
if (
chip === undefined ||
chip.romBootloaderOffset === 0 ||
image.metadata.descriptor?.kind === "app"
)
return undefined;
const { handle, initial } = await this.#source.read(signal);
const context = { handle, fileSize: initial.size, signal };
return espFlashDumpOrigin(
(position, length) => readEspFlashRange(context, position, length),
initial.size,
// A truncated image has no end, so no table can follow it.
{ chip, fileOffset: 0, end: image.metadata.image_end ?? initial.size },
);
}
/** Stream the carrier once, stopping when later bytes cannot matter. */
@@ -302,6 +347,12 @@ export class EspImageArtifactReader implements ArtifactReader {
}
}
const flashDumpLimitation = (
image: EspImage,
dump: EspFlashDumpOrigin,
): string =>
`This file looks like a flash dump read from flash ${offsetText(dump.flashOffset)}, not one ESP image: its ${image.metadata.chip?.target ?? "ESP"} image at file offset 0 is where the ROM loads the bootloader from ${offsetText(dump.flashOffset)}, and a partition table sits at file offset ${offsetText(dump.tableFileOffset)}, which is flash 0x8000 from that origin. Reopen it with format esp-flash-image and flash_offset ${offsetText(dump.flashOffset)}.`;
const decodeFailure = (failure: EspImageFailure): ArtifactReaderFailure =>
failure.kind === "unsupported"
? new ArtifactReaderFailure("unavailable", failure.message)
src/artifacts/artifactFormats.test.ts
+1
View File
@@ -37,6 +37,7 @@ type FormatsWithReaders =
| "intel-hex"
| "srec"
| "esp-image"
| "esp-flash-image"
| "fit";
const readerFormatsMatch: Equal<ArtifactReaderFormat, FormatsWithReaders> =
src/artifacts/artifactFormats.ts
+25
View File
@@ -3,6 +3,7 @@ import {
ARTIFACT_FORMAT_IDS,
type ArtifactFormatId,
} from "../domain/artifactGraph.js";
import type { EspFlashSelection } from "../domain/firmware/espFlashLocation.js";
import { ESP_IMAGE_LIMITATIONS } from "../domain/firmware/espImageFacts.js";
import {
ZIP_PACKAGE_FORMATS,
@@ -16,6 +17,10 @@ import { BunStandaloneArtifactReader } from "./BunStandaloneArtifactReader.js";
import { DirectoryArtifactReader } from "./DirectoryArtifactReader.js";
import type { DotnetBundleAdmission } from "./dotnet/DotnetBundleAdmission.js";
import { DotnetBundleArtifactReader } from "./DotnetBundleArtifactReader.js";
import {
EspFlashImageArtifactReader,
type EspFlashAdmission,
} from "./EspFlashImageArtifactReader.js";
import { EspImageArtifactReader } from "./EspImageArtifactReader.js";
import { FitArtifactReader } from "./FitArtifactReader.js";
import type { GodotPckAdmission } from "./GodotPckLocation.js";
@@ -39,6 +44,8 @@ export interface ArtifactReaderConstruction {
readonly unreadableDirectories: "fail" | "omit";
readonly admission?: ArtifactReaderAdmission | undefined;
readonly asarHeaderAdmission?: AsarHeaderAdmission | undefined;
/** Caller inputs for an explicitly selected ESP flash image. */
readonly espFlashSelection?: EspFlashSelection | undefined;
/** Caller-selected root spelling; only ASAR companion lookup consumes this. */
readonly selectedPath?: string | undefined;
}
@@ -93,6 +100,10 @@ function admittedAs(
kind: "node-sea",
options: ArtifactReaderConstruction,
): NodeSeaAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: "esp-flash-image",
options: ArtifactReaderConstruction,
): EspFlashAdmission | ReturnType<typeof sourceOf>;
function admittedAs(
kind: string,
options: ArtifactReaderConstruction,
@@ -338,6 +349,20 @@ const NON_ZIP_FORMATS = {
createReader: (path, options) =>
new EspImageArtifactReader(path, sourceOf(options)),
}),
"esp-flash-image": container("esp-flash-image", {
roleContainer: true,
retainClassifiedSource: true,
extractable: true,
absentInventory: "unavailable",
readerLimitation:
"ESP flash layout inventory reads a dump's bootloader region, partition table and partitions and decodes the bootloader and app partitions as ESP images. The OTA selection is derived without app rollback or anti-rollback, and the running partition is unknown. Partitions, nvs_keys included, are extracted as ordinary data; nothing is decrypted.",
createReader: (path, options) =>
new EspFlashImageArtifactReader(
path,
admittedAs("esp-flash-image", options),
options.espFlashSelection,
),
}),
fit: container("fit", {
roleContainer: true,
retainClassifiedSource: true,
src/artifacts/extraction/ArtifactExtraction.ts
+10 -1
View File
@@ -7,6 +7,7 @@ import {
import { hashStableRootArtifactHandle } from "../inventory/hashStableRootArtifact.js";
import { absoluteCallerPath } from "../../filesystem/callerPath.js";
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import { artifactFormatDescriptor } from "../artifactFormats.js";
import { createArtifactReader } from "../createArtifactReader.js";
import { resolveAsarHeaderAdmission } from "../AsarHeader.js";
@@ -59,6 +60,8 @@ import type { ArtifactIntegrityPolicyName } from "../../domain/artifactIntegrity
/** Local extraction input with the output root chosen by the adapter. */
export interface ArtifactExtractionInput {
readonly asarHeaderAdmission?: AsarHeaderAdmission;
/** Caller inputs when the target was explicitly selected as an ESP flash image. */
readonly espFlashSelection?: EspFlashSelection | undefined;
readonly inputPath: string;
readonly expectedRootSha256?: string | undefined;
/** Selected artifact spelling; packed I/O remains on inputPath. */
@@ -99,6 +102,7 @@ const extractArtifactInScope = async (
asarHeaderAdmission:
input.asarHeaderAdmission ??
resolveAsarHeaderAdmission(input.environment),
espFlashSelection: input.espFlashSelection,
integrity: input.integrityPolicy,
},
probeFacts,
@@ -628,7 +632,11 @@ const createReader = (
path: string,
input: Pick<
ArtifactExtractionInput,
"inputPath" | "selectedPath" | "environment" | "asarHeaderAdmission"
| "inputPath"
| "selectedPath"
| "environment"
| "asarHeaderAdmission"
| "espFlashSelection"
>,
containerFormat: ArtifactGraphManifest["root_format"],
transferredSource?: ActiveArtifactInventorySource,
@@ -647,6 +655,7 @@ const createReader = (
input.selectedPath ?? input.inputPath,
),
admission: transferredSource?.admission,
espFlashSelection: input.espFlashSelection,
asarHeaderAdmission:
containerFormat === "asar"
? (input.asarHeaderAdmission ??
src/artifacts/inventory/classify.ts
+38 -11
View File
@@ -2,6 +2,10 @@ import type {
ArtifactReaderAdmission,
ArtifactFormatAdmission,
} from "../ArtifactReaderAdmission.js";
import {
locateEspFlashLayout,
type EspFlashSelection,
} from "../../domain/firmware/espFlashLocation.js";
import { assertArtifactSelectedSource } from "../ArtifactSelectedSource.js";
import { artifactNameForClassification } from "../../domain/artifactNameSelection.js";
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
@@ -71,6 +75,7 @@ export const classifyAndHashRootForInventory = async (
| "asarHeaderAdmission"
| "selectedPath"
| "expectedRootSha256"
| "espFlashSelection"
>,
probeFacts?: ArtifactEmbeddedFormatProbeFacts,
): Promise<RootInventoryClassification> => {
@@ -146,6 +151,7 @@ export const classifyAndHashRootForInventory = async (
options.signal,
options.asarHeaderAdmission,
options.selectedPath,
options.espFlashSelection,
);
const format =
typeof classified === "string" ? classified : classified.kind;
@@ -198,6 +204,7 @@ const classifyRootFormat = async (
signal?: AbortSignal,
asarHeaderAdmission?: AsarHeaderAdmission,
selectedPath?: string,
espFlashSelection?: EspFlashSelection,
): Promise<ArtifactOccurrence["artifact_format"] | ArtifactFormatAdmission> => {
const { handle } = source;
const probe = createArtifactClassificationProbe(asarHeaderAdmission);
@@ -217,17 +224,6 @@ const classifyRootFormat = async (
const archiveBytes = (await handle.stat()).size;
const evidence = probe.finish(archiveBytes);
abortIfNeeded(signal);
const classificationPath = artifactNameForClassification(
selectedPath ?? path,
path,
);
const lower = classificationPath.toLowerCase();
if (hasZipSignature(evidence.prefix)) return zipRootPackageFormat(lower);
if (
lower.endsWith(".pkg") &&
evidence.prefix.subarray(0, 4).equals(Buffer.from("xar!"))
)
return "pkg";
// Fill each range: a short result is end of file, not a partial host read.
const read = async (position: number, length: number): Promise<Buffer> => {
const bytes = Buffer.alloc(length);
@@ -246,6 +242,32 @@ const classifyRootFormat = async (
}
return bytes.subarray(0, filled);
};
const locateFlash = (selection: EspFlashSelection, explicit: boolean) =>
locateEspFlashLayout(archiveBytes, read, selection, {
explicit,
...(signal === undefined ? {} : { signal }),
});
// The caller's format selection decides before any other signature.
if (espFlashSelection !== undefined) {
const selected = await locateFlash(espFlashSelection, true);
if (!selected.ok)
throw new ArtifactReaderFailure(
"unavailable",
`ESP flash image layout is not recognized: ${selected.error}`,
);
return { kind: "esp-flash-image", source, location: selected.value };
}
const classificationPath = artifactNameForClassification(
selectedPath ?? path,
path,
);
const lower = classificationPath.toLowerCase();
if (hasZipSignature(evidence.prefix)) return zipRootPackageFormat(lower);
if (
lower.endsWith(".pkg") &&
evidence.prefix.subarray(0, 4).equals(Buffer.from("xar!"))
)
return "pkg";
if (
lower.endsWith(".dmg") &&
(await hasDiskImageSignature(archiveBytes, read))
@@ -253,6 +275,11 @@ const classifyRootFormat = async (
return "dmg";
const pck = await embeddedProbes.godotPck(evidence.prefix);
if (pck !== undefined) return { kind: "godot-pck", source, location: pck };
// A C3 or S3 dump begins with its bootloader, so the layout is located
// before the bytes are read as one ESP image.
const flash = await locateFlash({}, false);
if (flash.ok)
return { kind: "esp-flash-image", source, location: flash.value };
const fatEvidence = await readRootFatSliceHeaders(handle, evidence, signal);
const mzWindowsHeaderSignature = await readRootMzWindowsHeaderSignature(
handle,
src/artifacts/inventory/types.ts
+6
View File
@@ -1,3 +1,4 @@
import type { EspFlashSelection } from "../../domain/firmware/espFlashLocation.js";
import type { ArtifactIntegrityPolicyName } from "../../domain/artifactIntegrityPolicy.js";
import type { AsarHeaderAdmission } from "../../config/asarHeaderAdmission.js";
import type { ArtifactResourceScope } from "../ArtifactResourceScope.js";
@@ -14,4 +15,9 @@ export interface ArtifactInventoryOptions {
readonly environment?: Readonly<NodeJS.ProcessEnv> | undefined;
/** ASAR header/index admission resolved from the owner's environment; default when absent. */
readonly asarHeaderAdmission?: AsarHeaderAdmission | undefined;
/**
* Caller inputs when the target was explicitly selected as an ESP flash
* image; detection with defaults applies otherwise.
*/
readonly espFlashSelection?: EspFlashSelection | undefined;
}
src/cli/artifactCommands.ts
+57
View File
@@ -1,3 +1,8 @@
import {
espFlashAddressSchema,
parseEspFlashAddress,
} from "../domain/firmware/espFlashSelection.js";
import type { ArtifactTargetSelection } from "../application/DirectAnalysis.js";
import { projectCliAnalysisResult } from "./projectCliAnalysisResult.js";
import {
analysisEvidenceDeliveryOptions,
@@ -198,6 +203,51 @@ const registerInterfaceBuilderCommand = (
});
};
/** Options that read a dump as an ESP flash image with caller inputs. */
const espFlashSelectionOptions = {
targetFormat: z
.enum(["esp-flash-image"])
.optional()
.describe(
"Read the target as an ESP flash dump; required with --flash-offset or --partition-table-offset",
),
flashOffset: espFlashAddressSchema
.optional()
.describe("Flash address (0xhex) of the dump's first byte"),
partitionTableOffset: espFlashAddressSchema
.optional()
.describe(
"Flash address (0xhex, a multiple of 0x1000) of the partition table; disables the table scan",
),
};
const espFlashTargetSelection = (options: {
readonly targetFormat?: "esp-flash-image" | undefined;
readonly flashOffset?: string | undefined;
readonly partitionTableOffset?: string | undefined;
}): ArtifactTargetSelection => ({
...(options.targetFormat === undefined
? {}
: { formatHint: options.targetFormat }),
...(options.flashOffset === undefined &&
options.partitionTableOffset === undefined
? {}
: {
espFlash: {
...(options.flashOffset === undefined
? {}
: { flashOffset: parseEspFlashAddress(options.flashOffset) }),
...(options.partitionTableOffset === undefined
? {}
: {
partitionTableOffset: parseEspFlashAddress(
options.partitionTableOffset,
),
}),
},
}),
});
const registerExtractionCommand = (
cli: CliInstance,
logger: Logger,
@@ -214,6 +264,7 @@ const registerExtractionCommand = (
.enum(["fail", "record-and-continue"])
.default("fail")
.describe("Behavior when declared artifact integrity does not match"),
...espFlashSelectionOptions,
}),
run: ({ args, options }) =>
logCliCommand(logger, "extract-artifact", () =>
@@ -225,6 +276,9 @@ const registerExtractionCommand = (
output_root: createArtifactExtractionDestination(),
},
logger,
undefined,
undefined,
espFlashTargetSelection(options),
).then(projectCliAnalysisResult),
),
});
@@ -247,6 +301,7 @@ const registerInspectionCommand = (
.enum(["fail", "record-and-continue"])
.default("fail")
.describe("Behavior when declared artifact integrity does not match"),
...espFlashSelectionOptions,
}),
run: ({ args, options }) =>
logCliCommand(logger, "inspect-artifact", () =>
@@ -266,6 +321,8 @@ const registerInspectionCommand = (
{ integrity_policy: options.integrityPolicy },
logger,
signal,
undefined,
espFlashTargetSelection(options),
);
return publishCliAnalysisEvidence(complete, delivery);
}),
src/cli/options.ts
+5 -1
View File
@@ -33,8 +33,12 @@ export const snapshotSelectionOption = z
.optional()
.describe("Load and update a local analysis snapshot");
/** Explicit interpretation for headerless native executable formats. */
/**
* Explicit interpretation for headerless native executable formats. Flash
* dumps take their offsets on the artifact commands instead.
*/
export const formatSelectionOption = executableFormatHintSchema
.exclude(["esp-flash-image"])
.optional()
.describe(
"Explicit interpretation: dos-com or raw-binary (headerless), or gamecube-dol (GameCube/Wii DOL, never auto-detected); omission preserves header-based detection",
src/contracts/sessionLifecycleInputs.ts
+37 -1
View File
@@ -6,6 +6,11 @@ import {
rawBinarySelectionError,
} from "../domain/rawBinary.js";
import { z } from "zod";
import {
espFlashAddressSchema,
espFlashSelectionError,
parseEspFlashAddress,
} from "../domain/firmware/espFlashSelection.js";
import { analysisProviderSelectorSchema } from "../domain/providerSelection.js";
/** Caller-supplied analysis snapshot path; must name its destination absolutely. */
@@ -25,7 +30,7 @@ export const openBinaryInputSchema = z
format: executableFormatHintSchema
.optional()
.describe(
"Explicit interpretation: dos-com or raw-binary (headerless), or gamecube-dol (GameCube/Wii DOL, never auto-detected). Omission preserves header-based detection.",
"Explicit interpretation: dos-com or raw-binary (headerless), gamecube-dol (GameCube/Wii DOL, never auto-detected), or esp-flash-image (an ESP flash dump read with flash_offset and partition_table_offset). Omission preserves header-based detection.",
),
architecture: binaryArchitectureSchema
.optional()
@@ -37,10 +42,41 @@ export const openBinaryInputSchema = z
.describe(
"Caller-supplied load and entry address (0xhex). File offset 0 is mapped here. Required with format=raw-binary; rejected otherwise. Optional with format=gamecube-dol; must equal the DOL header entry.",
),
flash_offset: espFlashAddressSchema
.optional()
.describe(
"Flash address (0xhex) of the dump's first byte, when it does not start at flash address 0. Only with format=esp-flash-image.",
),
partition_table_offset: espFlashAddressSchema
.optional()
.describe(
"Flash address (0xhex, a multiple of 0x1000) of the partition table, which disables the table scan. Only with format=esp-flash-image.",
),
provider_id: analysisProviderSelectorSchema.optional(),
snapshot_path: snapshotPathSchema.optional(),
})
.superRefine((value, context) => {
const flash = espFlashSelectionError({
format: value.format,
flashOffset:
value.flash_offset === undefined
? undefined
: parseEspFlashAddress(value.flash_offset),
partitionTableOffset:
value.partition_table_offset === undefined
? undefined
: parseEspFlashAddress(value.partition_table_offset),
});
if (flash !== undefined)
context.addIssue({
code: "custom",
path: [
value.partition_table_offset === undefined
? "flash_offset"
: "partition_table_offset",
],
message: flash,
});
const pairing = rawBinarySelectionError({
...(value.format === undefined ? {} : { format: value.format }),
...(value.architecture === undefined
src/domain/analysisSnapshot.ts
+34 -8
View File
@@ -80,6 +80,7 @@ const formatSchema = z.enum([
"intel-hex",
"srec",
"esp-image",
"esp-flash-image",
"fit",
"plist",
"javascript",
@@ -101,7 +102,10 @@ const targetSchema = z.discriminatedUnion("format", [
targetIdentitySchema.extend({
format: z.enum(
formatSchema.options.filter(
(format) => format !== "raw-binary" && format !== "gamecube-dol",
(format) =>
format !== "raw-binary" &&
format !== "gamecube-dol" &&
format !== "esp-flash-image",
),
),
architecture: architectureSchema
@@ -120,6 +124,14 @@ const targetSchema = z.discriminatedUnion("format", [
architecture: z.literal("powerpc"),
entry_point: rawBinaryEntryPointSchema,
}),
// The caller's inputs select how the dump is read, so they are identity.
targetIdentitySchema.extend({
format: z.literal("esp-flash-image"),
kind: z.literal("archive"),
architecture: z.null(),
flash_offset: z.number().int().min(0).max(0xffff_ffff).nullable(),
partition_table_offset: z.number().int().min(0).max(0xffff_ffff).nullable(),
}),
]);
const bindingSchema = z
.object({
@@ -388,12 +400,21 @@ export const snapshotTarget = (target: BinaryTarget): AnalysisSnapshotTarget =>
format: target.format,
entry_point: target.entryPoint,
}
: {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture ?? null,
format: target.format,
};
: target.format === "esp-flash-image"
? {
sha256: target.sha256,
kind: target.kind,
architecture: null,
format: target.format,
flash_offset: target.espFlash.flashOffset,
partition_table_offset: target.espFlash.partitionTableOffset,
}
: {
sha256: target.sha256,
kind: target.kind,
architecture: target.architecture ?? null,
format: target.format,
};
/** Build the immutable provider/profile binding used by a cached snapshot. */
export const snapshotBinding = (
@@ -418,7 +439,12 @@ export const snapshotMatchesTarget = (
snapshot.entry_point === target.entryPoint)) &&
(snapshot.format !== "gamecube-dol" ||
(target.format === "gamecube-dol" &&
snapshot.entry_point === target.entryPoint));
snapshot.entry_point === target.entryPoint)) &&
(snapshot.format !== "esp-flash-image" ||
(target.format === "esp-flash-image" &&
snapshot.flash_offset === target.espFlash.flashOffset &&
snapshot.partition_table_offset ===
target.espFlash.partitionTableOffset));
/** Compare a snapshot binding with the selected concrete analysis profile. */
export const snapshotMatchesProfile = (
src/domain/analysisView/artifactInventoryView.ts
+10 -2
View File
@@ -39,6 +39,10 @@ import {
} from "./analysisViewCoverage.js";
import { analysisViewResultInputError } from "./analysisViewParent.js";
import { ARTIFACT_INVENTORY_VIEW_CAVEATS } from "./artifactInventoryViewCaveats.js";
import {
espFlashImageSummaryMetadataSchema,
summaryRootMetadata,
} from "./espFlashImageSummary.js";
const page = {
kind: z.literal("inventory"),
@@ -99,7 +103,10 @@ export const artifactInventorySummarySchema = z.strictObject({
})
.exactOptional(),
manifest: artifactGraphManifestSchema,
root_format_metadata: artifactFormatMetadataSchema.exactOptional(),
/** An ESP flash layout's nested images appear as summaries here. */
root_format_metadata: z
.union([artifactFormatMetadataSchema, espFlashImageSummaryMetadataSchema])
.exactOptional(),
available_counts: z.strictObject({
nodes: z.number().int().nonnegative(),
occurrences: z.number().int().nonnegative(),
@@ -275,6 +282,7 @@ export const projectArtifactInventoryView = (
inventory.manifest.root_format === "intel-hex" ||
inventory.manifest.root_format === "srec" ||
inventory.manifest.root_format === "esp-image" ||
inventory.manifest.root_format === "esp-flash-image" ||
inventory.manifest.root_format === "fit";
const counts = inventoryCounts(source);
const limitations = uniqueSorted([
@@ -346,7 +354,7 @@ export const projectArtifactInventoryView = (
inventory_evidence_ids: [...evidenceIds],
...(rootMetadata === null
? {}
: { root_format_metadata: rootMetadata }),
: { root_format_metadata: summaryRootMetadata(rootMetadata) }),
inspection:
inspection === null
? null
src/domain/analysisView/espFlashImageSummary.ts
+151
View File
@@ -0,0 +1,151 @@
import { z } from "zod";
import {
espAppDescriptorSchema,
espBootloaderDescriptorSchema,
espFlashImageFactsSchema,
espFlashImageMetadataSchema,
espImageMetadataSchema,
espImageSegmentMetadataSchema,
uint8,
uint32,
type ArtifactFormatMetadata,
} from "../artifactFormatMetadata.js";
type EspFlashImageFacts = z.infer<typeof espFlashImageFactsSchema>;
/**
* One ESP image in a flash layout's summary view: what tells two app slots
* apart, where its segments go and its integrity. The complete Evidence
* keeps its full facts, segments and signature blocks included.
*/
const espImageSummarySchema = z.strictObject({
status: espFlashImageFactsSchema.shape.status,
/** The chip target, from the header's `chip_id`; null when unknown. */
chip: espImageMetadataSchema.shape.chip.unwrap().shape.target.nullable(),
entry_addr: uint32.nullable(),
/** The header's segment count. */
segment_count: uint8.nullable(),
/** Where each segment that is mapped, loaded or unclassified goes. */
load_ranges: z.array(
z.strictObject({
address: uint32,
length: uint32,
classification:
espImageSegmentMetadataSchema.shape.classification.exclude([
"not-loaded",
]),
}),
),
signature_block_count: uint8,
/** The descriptor's identifying fields; null without one. */
descriptor: z
.discriminatedUnion("kind", [
espAppDescriptorSchema.pick({
kind: true,
project_name: true,
version: true,
idf_ver: true,
secure_version: true,
app_elf_sha256: true,
}),
espBootloaderDescriptorSchema.pick({
kind: true,
version: true,
idf_ver: true,
secure_version: true,
}),
])
.nullable(),
checksum: espImageMetadataSchema.shape.checksum.shape.status.nullable(),
appended_sha256:
espImageMetadataSchema.shape.appended_sha256.shape.status.nullable(),
coverage: espFlashImageFactsSchema.shape.coverage,
limitations: z.array(z.string()),
});
/**
* An ESP flash layout's root facts in the summary view, under their own
* kind: each nested image is a `summary` instead of its full `facts`.
*/
export const espFlashImageSummaryMetadataSchema =
espFlashImageMetadataSchema.extend({
kind: z.literal("esp-flash-image-summary"),
bootloader: espFlashImageMetadataSchema.shape.bootloader
.unwrap()
.omit({ facts: true })
.extend({ summary: espImageSummarySchema })
.nullable(),
apps: z.array(
espFlashImageMetadataSchema.shape.apps.element
.omit({ facts: true })
.extend({ summary: espImageSummarySchema }),
),
});
export type EspFlashImageSummaryMetadata = z.infer<
typeof espFlashImageSummaryMetadataSchema
>;
const summarizeDescriptor = (
descriptor: NonNullable<EspFlashImageFacts["image"]>["descriptor"],
): z.infer<typeof espImageSummarySchema>["descriptor"] => {
if (descriptor === null) return null;
const { kind, version, idf_ver, secure_version } = descriptor;
return kind === "app"
? {
kind,
project_name: descriptor.project_name,
version,
idf_ver,
secure_version,
app_elf_sha256: descriptor.app_elf_sha256,
}
: { kind, version, idf_ver, secure_version };
};
const summarizeImage = (
facts: EspFlashImageFacts,
): z.infer<typeof espImageSummarySchema> => ({
status: facts.status,
chip: facts.image?.chip?.target ?? null,
entry_addr: facts.image?.header.entry_addr ?? null,
segment_count: facts.image?.header.segment_count ?? null,
load_ranges: facts.segments.flatMap(
({ load_addr: address, data_len: length, classification }) =>
classification === "not-loaded"
? []
: [{ address, length, classification }],
),
signature_block_count: facts.signature_blocks.length,
descriptor: summarizeDescriptor(facts.image?.descriptor ?? null),
checksum: facts.image?.checksum.status ?? null,
appended_sha256: facts.image?.appended_sha256.status ?? null,
coverage: facts.coverage,
limitations: [...facts.limitations],
});
/** Root facts for the summary view: compact for an ESP flash layout. */
export const summaryRootMetadata = (
metadata: ArtifactFormatMetadata,
): ArtifactFormatMetadata | EspFlashImageSummaryMetadata => {
if (metadata.kind !== "esp-flash-image") return metadata;
const { bootloader, apps, ...rest } = metadata;
return {
...rest,
kind: "esp-flash-image-summary",
bootloader:
bootloader === null
? null
: {
path: bootloader.path,
flash_address: bootloader.flash_address,
region_size: bootloader.region_size,
summary: summarizeImage(bootloader.facts),
},
apps: apps.map(({ path, partition_index, facts }) => ({
path,
partition_index,
summary: summarizeImage(facts),
})),
};
};
src/domain/artifactComparison.ts
+12
View File
@@ -14,6 +14,7 @@ import {
type InventorySet,
} from "./artifactInventoryEvidence.js";
import {
espFlashPartitionDeclaration,
espImageSegmentDeclaration,
fitImageDeclaration,
nodeSeaEntryDeclaration,
@@ -465,6 +466,7 @@ interface OccurrenceMetadata {
readonly asarPacking: MetadataComparison;
readonly nodeSea: MetadataComparison;
readonly espImageSegment: MetadataComparison;
readonly espFlashPartition: MetadataComparison;
readonly fitImage: MetadataComparison;
readonly fitImageAvailability: MetadataComparison;
readonly pyInstaller: MetadataComparison;
@@ -475,6 +477,7 @@ const declarationChanged = (metadata: OccurrenceMetadata): boolean =>
metadata.asarPacking === "changed" ||
metadata.nodeSea === "changed" ||
metadata.espImageSegment === "changed" ||
metadata.espFlashPartition === "changed" ||
metadata.fitImage === "changed";
/** Unknown classifications and absent node metadata cannot establish a difference. */
@@ -581,6 +584,15 @@ const occurrenceMetadata = (input: OccurrencePair): OccurrenceMetadata => {
: undefined,
espImageSegmentDeclaration,
),
espFlashPartition: compareOptionalMetadata(
left?.format_metadata?.kind === "esp-flash-partition"
? left.format_metadata
: undefined,
right?.format_metadata?.kind === "esp-flash-partition"
? right.format_metadata
: undefined,
espFlashPartitionDeclaration,
),
pyInstaller: compareOptionalMetadata(
isPyInstallerEntry(left) ? left.format_metadata : undefined,
isPyInstallerEntry(right) ? right.format_metadata : undefined,
src/domain/artifactFormatMetadata.ts
+205 -11
View File
@@ -1,7 +1,7 @@
import { z } from "zod";
const offset = z.number().int().min(0).max(Number.MAX_SAFE_INTEGER);
const uint32 = z.number().int().min(0).max(0xffff_ffff);
export const uint32 = z.number().int().min(0).max(0xffff_ffff);
/** Parsed source fields owned by the PCK adapter, independent of content hashes. */
export const godotPckMetadataSchema = z.strictObject({
@@ -297,16 +297,23 @@ export const nodeSeaEntryDeclaration = (
metadata: z.infer<typeof nodeSeaEntryMetadataSchema>,
) => ({ role: metadata.role, source_path: metadata.source_path });
const uint8 = z.number().int().min(0).max(0xff);
export const uint8 = z.number().int().min(0).max(0xff);
const uint16 = z.number().int().min(0).max(0xffff);
const sha256Hex = z.string().regex(/^[0-9a-f]{64}$/u);
const byteRange = z.strictObject({ offset, length: offset });
/** Derived from `chip_id` through the cited table. */
const espChipFactsSchema = z.strictObject({
target: z.string(),
isa: z.enum(["xtensa", "riscv"]),
source: z.string(),
});
/**
* `esp_app_desc_t` fields, as ESP-IDF names them; strings stop at NUL. A field
* the file cuts off is null; `observed_length` says where the descriptor ends.
*/
const espAppDescriptorSchema = z.strictObject({
export const espAppDescriptorSchema = z.strictObject({
kind: z.literal("app"),
offset,
/** Descriptor bytes the file holds, at most the 256 of `esp_app_desc_t`. */
@@ -333,7 +340,7 @@ const espAppDescriptorSchema = z.strictObject({
});
/** `esp_bootloader_desc_t` fields, as ESP-IDF names them. */
const espBootloaderDescriptorSchema = z.strictObject({
export const espBootloaderDescriptorSchema = z.strictObject({
kind: z.literal("bootloader"),
offset,
secure_version: uint8,
@@ -366,13 +373,7 @@ export const espImageMetadataSchema = z.strictObject({
hash_appended: z.boolean(),
}),
/** Derived from `chip_id` through the cited table; null for unknown IDs. */
chip: z
.strictObject({
target: z.string(),
isa: z.enum(["xtensa", "riscv"]),
source: z.string(),
})
.nullable(),
chip: espChipFactsSchema.nullable(),
checksum: z.strictObject({
offset: offset.nullable(),
stored: uint8.nullable(),
@@ -398,6 +399,19 @@ export const espImageMetadataSchema = z.strictObject({
espBootloaderDescriptorSchema,
])
.nullable(),
/**
* Set when the file is a flash dump read from a later flash address: this
* image is not an app, its chip's ROM loads the bootloader from
* `flash_offset`, and a partition table sits at `table_file_offset`, where
* that origin puts the default `0x8000`. Reopen the file as
* `esp-flash-image` with that `flash_offset`. Null otherwise, and in the
* nested facts of a flash layout. Absent from Evidence recorded before REA
* reported it; its graph digest covers the record without the field.
*/
flash_dump: z
.strictObject({ flash_offset: uint32, table_file_offset: offset })
.nullable()
.exactOptional(),
});
export type EspImageMetadata = z.infer<typeof espImageMetadataSchema>;
@@ -449,6 +463,184 @@ export type EspImageSignatureBlockMetadata = z.infer<
typeof espImageSignatureBlockMetadataSchema
>;
/** One partition table entry as declared, with ESP-IDF's type names. */
export const espFlashPartitionMetadataSchema = z.strictObject({
kind: z.literal("esp-flash-partition"),
/** Position among the table's partition entries. */
index: uint8,
/** The label up to its first NUL; null when it is not UTF-8. */
label: z.string().nullable(),
/** The label's bytes up to its first NUL in hex, when it is not UTF-8. */
label_hex: z
.string()
.regex(/^(?:[0-9a-f]{2}){1,16}$/u)
.nullable(),
type: uint8,
subtype: uint8,
/** `gen_esp32part.py` names; null for values it does not name. */
type_name: z.string().nullable(),
subtype_name: z.string().nullable(),
flash_address: uint32,
size: uint32,
encrypted: z.boolean(),
readonly: z.boolean(),
/** Flag bits other than `encrypted` and `readonly`, which ESP-IDF ignores. */
unknown_flags: uint32,
/**
* Observed in the bytes the dump holds, from the partition's start: the
* first byte that is not 0xFF and just past the last, null when every byte
* is erased, and how many bytes are not 0xFF. All null for a partition the
* dump holds none of.
*/
written_offset: uint32.nullable(),
written_end: uint32.nullable(),
written_bytes: uint32.nullable(),
});
export type EspFlashPartitionMetadata = z.infer<
typeof espFlashPartitionMetadataSchema
>;
/**
* What a partition entry declares; its index is layout, and the written
* fields are observed from its bytes, which content comparison covers.
*/
export const espFlashPartitionDeclaration = (
metadata: EspFlashPartitionMetadata,
) => ({
label: metadata.label,
label_hex: metadata.label_hex,
type: metadata.type,
subtype: metadata.subtype,
flash_address: metadata.flash_address,
size: metadata.size,
encrypted: metadata.encrypted,
readonly: metadata.readonly,
unknown_flags: metadata.unknown_flags,
});
/** ESP image facts for a region of a dump; offsets are relative to the region. */
export const espFlashImageFactsSchema = z.strictObject({
/** An ESP image, erased flash, or bytes that are not one, such as ciphertext. */
status: z.enum(["esp-image", "erased", "unrecognized"]),
image: espImageMetadataSchema.nullable(),
segments: z.array(espImageSegmentMetadataSchema),
signature_blocks: z.array(espImageSignatureBlockMetadataSchema),
coverage: z.enum(["complete", "unknown"]).nullable(),
limitations: z.array(z.string()),
});
const espOtaRecordSchema = z.strictObject({
index: z.union([z.literal(0), z.literal(1)]),
ota_seq: uint32,
seq_label: z.string().regex(/^[0-9a-f]{40}$/u),
ota_state: uint32,
state_name: z
.enum(["new", "pending-verify", "valid", "invalid", "aborted", "undefined"])
.nullable(),
crc: uint32,
/** CRC-32 of `ota_seq` seeded with 0xFFFFFFFF, as the bootloader checks it. */
crc_matches: z.boolean(),
valid: z.boolean(),
});
/**
* An ESP flash dump: where it sits in flash, its bootloader, partition table,
* partitions and OTA data. Entries are the table's partitions; app and
* bootloader regions carry their ESP image facts here.
*/
export const espFlashImageMetadataSchema = z.strictObject({
kind: z.literal("esp-flash-image"),
/** Flash address of the file's first byte: 0 unless the caller gave one. */
flash_offset: z.strictObject({
value: uint32,
source: z.enum(["default", "caller"]),
}),
/** From the bootloader's `chip_id`; null without a located bootloader. */
chip: espChipFactsSchema.nullable(),
/**
* The image at its chip's ROM offset, through the partition table; `path`
* is its entry.
*/
bootloader: z
.strictObject({
path: z.string(),
flash_address: uint32,
region_size: offset,
facts: espFlashImageFactsSchema,
})
.nullable(),
partition_table: z.strictObject({
flash_address: uint32,
/** 0x8000, the caller's offset, or a scan for a verified MD5 (derived). */
method: z.enum(["default", "caller", "scan"]),
partition_count: uint8,
md5: z.strictObject({
status: z.enum(["valid", "invalid", "absent"]),
stored: z
.string()
.regex(/^[0-9a-f]{32}$/u)
.nullable(),
computed: z
.string()
.regex(/^[0-9a-f]{32}$/u)
.nullable(),
}),
}),
/**
* Declared partitions that begin before the dump's first byte, which have
* no entry. A partition past the dump's end is an entry with no bytes.
*/
partitions_outside_dump: z.array(espFlashPartitionMetadataSchema),
/** ESP image facts for each app partition the dump holds, by its entry. */
apps: z.array(
z.strictObject({
path: z.string(),
partition_index: uint8,
facts: espFlashImageFactsSchema,
}),
),
/** The otadata partition the bootloader uses: the last one declared. */
otadata: z
.strictObject({
partition_index: uint8,
/** The records the dump holds, at most one per sector. */
records: z.array(espOtaRecordSchema).max(2),
})
.nullable(),
/**
* The app the bootloader would select, derived for every table under the
* stated assumptions. `basis` names the branch: `otadata`,
* `factory-fallback` and `ota_0-fallback` from the otadata records,
* `no-otadata` for a table without otadata (the factory index), `none`
* for the bootloader's INVALID_INDEX, after which it resets, and
* `unknown` when the dump lacks a record the bootloader reads.
* `partition_index` is null when the table declares no partition for the
* selected slot or fallback.
*/
boot_selection: z.strictObject({
basis: z.enum([
"otadata",
"factory-fallback",
"ota_0-fallback",
"no-otadata",
"none",
"unknown",
]),
active_record: z.union([z.literal(0), z.literal(1)]).nullable(),
slot: uint8.nullable(),
partition_index: uint8.nullable(),
assumptions: z.array(z.string()),
/** Rollback, anti-rollback and load fallback decide it at boot. */
running_partition: z.literal("unknown"),
}),
/**
* Ranges outside every described region, each from a byte that is not
* erased flash to the end of its gap.
*/
undescribed: z.array(byteRange),
});
export type EspFlashImageMetadata = z.infer<typeof espFlashImageMetadataSchema>;
/** Declaration identity that is stable across archive layout changes. */
export const pyInstallerEntryDeclaration = (
metadata: PyInstallerEntryMetadata,
@@ -571,6 +763,8 @@ export const artifactFormatMetadataSchema = z.discriminatedUnion("kind", [
espImageMetadataSchema,
espImageSegmentMetadataSchema,
espImageSignatureBlockMetadataSchema,
espFlashImageMetadataSchema,
espFlashPartitionMetadataSchema,
fitMetadataSchema,
fitImageMetadataSchema,
]);
src/domain/artifactGraph.ts
+1
View File
@@ -93,6 +93,7 @@ export const ARTIFACT_FORMAT_IDS = [
"intel-hex",
"srec",
"esp-image",
"esp-flash-image",
"fit",
"unknown",
] as const;
src/domain/artifactInspection.ts
+24
View File
@@ -21,6 +21,7 @@ import { isJavaScriptSourcePath } from "./javascript/javascriptArtifactFiles.js"
import { admitRawBinary } from "./rawBinary.js";
import { uniqueSorted } from "./canonicalOrdering.js";
import { evidenceSchema, parseEvidence, type Evidence } from "./evidence.js";
import { offsetText } from "./firmware/espImageStructures.js";
import { jsonValueSchema } from "./jsonValue.js";
import { prefixedDigestSchema } from "./../domain/digests.js";
@@ -331,12 +332,35 @@ const probesFor = (
},
]
: [];
if (format === "esp-image") return espFlashDumpProbes(inventory);
if (format === "fit") return fitProbes(inventory);
return format === "directory"
? directoryProbes(inventory)
: probesForFormat(format);
};
// An ESP image that its reader found at the start of a flash dump read from
// a later address is better read as that dump.
const espFlashDumpProbes = (
inventory: ArtifactInventoryResult,
): ArtifactInspectionResult["next_probes"] => {
const root = inventory.occurrences.find(
({ logical_path }) => logical_path === ".",
)?.format_metadata;
if (root?.kind !== "esp-image") return [];
// Evidence recorded before REA reported the origin has no hint to offer.
const dump = root.flash_dump;
if (dump === null || dump === undefined) return [];
const offset = offsetText(dump.flash_offset);
return [
{
operation: "open_binary",
rationale: `Reopen the file with format esp-flash-image and flash_offset ${offset}, then inspect_artifact: it is a flash dump read from flash ${offset}, with a partition table at file offset ${offsetText(dump.table_file_offset)}.`,
authority: "artifact-bytes",
},
];
};
const fitProbes = (
inventory: ArtifactInventoryResult,
): ArtifactInspectionResult["next_probes"] => {
src/domain/binaryTargetTypes.ts
+17
View File
@@ -275,12 +275,29 @@ export type BinaryTarget =
| "esp-image"
| "fit";
})
| (BinaryTargetIdentity &
NonExecutableMetadata & {
readonly kind: "archive";
readonly format: "esp-flash-image";
/** How the dump is read; null inputs take their defaults. */
readonly espFlash: EspFlashTargetSelection;
})
| (BinaryTargetIdentity &
NonExecutableMetadata & {
readonly kind: "artifact";
readonly format: "plist" | "javascript" | "source-map";
});
/** Caller inputs that select how an ESP flash dump is read. */
export interface EspFlashTargetSelection {
/** The caller selected the format, so the table scan runs without a bootloader. */
readonly selected: boolean;
/** Flash address of the dump's first byte. */
readonly flashOffset: number | null;
/** Flash address of the partition table. */
readonly partitionTableOffset: number | null;
}
/** Whether an executable target contains a thin or universal Mach-O image. */
export const isMachOFormat = (
format: unknown,
src/domain/dosCom.ts
+5 -1
View File
@@ -9,8 +9,12 @@ export const executableFormatHintSchema = z.enum([
"dos-com",
"raw-binary",
"gamecube-dol",
"esp-flash-image",
]);
/** Explicit executable interpretation for formats with no identifying header. */
/**
* Explicit interpretation for formats with no identifying header, or, for
* esp-flash-image, a dump read with caller inputs.
*/
export type ExecutableFormatHint = z.infer<typeof executableFormatHintSchema>;
/** Validate the single 64 KiB COM segment, excluding its 256-byte PSP prefix. */
src/domain/evidenceSubject.ts
+1
View File
@@ -45,6 +45,7 @@ export const evidenceSubjectFormatSchema = z.enum([
"intel-hex",
"srec",
"esp-image",
"esp-flash-image",
"fit",
]);
src/domain/firmware/espChips.ts
+22
View File
@@ -9,6 +9,8 @@
* `components/soc/<target>/include/soc/soc.h`;
* - page sizes: `SOC_MMU_PAGE_SIZE_8KB_SUPPORTED` in `soc_caps.h` and
* `components/soc/<target>/Kconfig.mmu`;
* - bootloader flash offset: `ESP_ROM_BOOTLOADER_OFFSET_FLASH` in
* `components/esp_rom/<target>/esp_rom_caps.h`, which the ROM fixes;
* - segment length limit: `SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP` in
* `soc_caps.h`, which sets `ESP_IMAGE_MAX_FLASH_ADDR_SIZE` in
* `components/bootloader_support/src/esp_image_format.c`.
@@ -31,6 +33,8 @@ export interface EspChip {
/** ESP-IDF `IDF_TARGET` name, as `esptool --chip` also spells it. */
readonly target: string;
readonly isa: EspChipIsa;
/** Where the ROM loads the second-stage bootloader from flash. */
readonly romBootloaderOffset: number;
/** `SOC_SPI_MEM_SUPPORT_CACHE_32BIT_ADDR_MAP`, which lifts the 16 MiB segment limit. */
readonly cache32BitAddressMap: boolean;
/** `should_map` ranges with fixed bounds. */
@@ -56,6 +60,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0000,
target: "esp32",
isa: "xtensa",
romBootloaderOffset: 0x1000,
cache32BitAddressMap: false,
mapped: [range(0x400d_0000, 0x4040_0000), range(0x3f40_0000, 0x3f80_0000)],
},
@@ -63,6 +68,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0002,
target: "esp32s2",
isa: "xtensa",
romBootloaderOffset: 0x1000,
cache32BitAddressMap: false,
mapped: [range(0x4008_0000, 0x4080_0000), range(0x3f00_0000, 0x3ff8_0000)],
},
@@ -70,6 +76,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0005,
target: "esp32c3",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [range(0x4200_0000, 0x4280_0000), range(0x3c00_0000, 0x3c80_0000)],
},
@@ -77,6 +84,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0009,
target: "esp32s3",
isa: "xtensa",
romBootloaderOffset: 0x0,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000), range(0x3c00_0000, 0x3e00_0000)],
},
@@ -84,6 +92,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x000c,
target: "esp32c2",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [range(0x4200_0000, 0x4240_0000), range(0x3c00_0000, 0x3c40_0000)],
},
@@ -91,6 +100,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x000d,
target: "esp32c6",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [],
pageSizedMapping: { low: 0x4200_0000, pageSizes: C6_H2_PAGE_SIZES },
@@ -99,6 +109,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0010,
target: "esp32h2",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [],
pageSizedMapping: { low: 0x4200_0000, pageSizes: C6_H2_PAGE_SIZES },
@@ -107,6 +118,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0012,
target: "esp32p4",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4000_0000, 0x4400_0000), range(0x4800_0000, 0x4c00_0000)],
},
@@ -114,6 +126,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0014,
target: "esp32c61",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000)],
},
@@ -121,6 +134,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0017,
target: "esp32c5",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000)],
},
@@ -128,6 +142,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0019,
target: "esp32h21",
isa: "riscv",
romBootloaderOffset: 0x0,
cache32BitAddressMap: false,
mapped: [range(0x4200_0000, 0x4300_0000)],
},
@@ -135,6 +150,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x001c,
target: "esp32h4",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4200_0000, 0x4400_0000)],
},
@@ -142,6 +158,7 @@ const CHIPS: readonly EspChip[] = [
chipId: 0x0020,
target: "esp32s31",
isa: "riscv",
romBootloaderOffset: 0x2000,
cache32BitAddressMap: true,
mapped: [range(0x4000_0000, 0x4400_0000), range(0x5000_0000, 0x5400_0000)],
},
@@ -152,6 +169,11 @@ const BY_ID = new Map(CHIPS.map((chip) => [chip.chipId, chip]));
export const espChipForId = (chipId: number): EspChip | undefined =>
BY_ID.get(chipId);
/** Every flash offset a ROM loads a bootloader from, lowest first. */
export const ESP_ROM_BOOTLOADER_OFFSETS: readonly number[] = [
...new Set(CHIPS.map(({ romBootloaderOffset }) => romBootloaderOffset)),
].sort((left, right) => left - right);
const SIXTEEN_MIB = 0x100_0000;
/**
src/domain/firmware/espFlash.fixture.ts
+140
View File
@@ -0,0 +1,140 @@
import { createHash } from "node:crypto";
import { crc32 } from "node:zlib";
/** One `esp_partition_info_t` as `gen_esp32part.py` writes it. */
export interface EspPartitionFixture {
readonly label: string;
readonly type: number;
readonly subtype: number;
readonly offset: number;
readonly size: number;
readonly flags?: number;
}
export const partitionEntry = (part: EspPartitionFixture): Buffer => {
const entry = Buffer.alloc(32);
entry.writeUInt16LE(0x50aa, 0);
entry.writeUInt8(part.type, 2);
entry.writeUInt8(part.subtype, 3);
entry.writeUInt32LE(part.offset, 4);
entry.writeUInt32LE(part.size, 8);
entry.write(part.label, 12, 16, "utf8");
entry.writeUInt32LE(part.flags ?? 0, 28);
return entry;
};
/** The entries, an MD5 entry unless omitted, the terminator and 0xFF fill. */
export const partitionTable = (
parts: readonly EspPartitionFixture[],
{ md5 = "valid" }: { readonly md5?: "valid" | "corrupt" | "absent" } = {},
): Buffer => {
const entries = Buffer.concat(parts.map(partitionEntry));
const digest = createHash("md5").update(entries).digest();
if (md5 === "corrupt") digest.writeUInt8(digest.readUInt8(0) ^ 0xff, 0);
const md5Entry =
md5 === "absent"
? Buffer.alloc(0)
: Buffer.concat([
Buffer.from([0xeb, 0xeb]),
Buffer.alloc(14, 0xff),
digest,
]);
const table = Buffer.concat([entries, md5Entry]);
return Buffer.concat([table, Buffer.alloc(0xc00 - table.length, 0xff)]);
};
/** The layout `idf.py` builds for two OTA apps, from `partitions_two_ota.csv`. */
export const TWO_OTA_PARTITIONS: readonly EspPartitionFixture[] = [
{ label: "nvs", type: 0x01, subtype: 0x02, offset: 0x9000, size: 0x4000 },
{ label: "otadata", type: 0x01, subtype: 0x00, offset: 0xd000, size: 0x2000 },
{
label: "phy_init",
type: 0x01,
subtype: 0x01,
offset: 0xf000,
size: 0x1000,
},
{
label: "factory",
type: 0x00,
subtype: 0x00,
offset: 0x1_0000,
size: 0x1_0000,
},
{
label: "ota_0",
type: 0x00,
subtype: 0x10,
offset: 0x2_0000,
size: 0x1_0000,
},
{
label: "ota_1",
type: 0x00,
subtype: 0x11,
offset: 0x3_0000,
size: 0x1_0000,
},
];
/** One `esp_ota_select_entry_t`; the CRC is computed unless given. */
export const otaRecord = ({
seq,
state = 0xffff_ffff,
crc,
}: {
readonly seq: number;
readonly state?: number;
readonly crc?: number;
}): Buffer => {
const record = Buffer.alloc(32, 0xff);
record.writeUInt32LE(seq >>> 0, 0);
record.writeUInt32LE(state >>> 0, 24);
const computed = crc32(record.subarray(0, 4), 0xffff_ffff) >>> 0;
record.writeUInt32LE((crc ?? computed) >>> 0, 28);
return record;
};
/** Two 4 KiB sectors with one record at the start of each; omitted is erased. */
export const otaData = (first?: Buffer, second?: Buffer): Buffer => {
const data = Buffer.alloc(0x2000, 0xff);
first?.copy(data, 0);
second?.copy(data, 0x1000);
return data;
};
/** A flash dump of `size` bytes of erased flash with the given contents. */
export const flashImage = (
size: number,
contents: readonly { readonly at: number; readonly bytes: Uint8Array }[],
): Buffer => {
const image = Buffer.alloc(size, 0xff);
for (const { at, bytes } of contents) Buffer.from(bytes).copy(image, at);
return image;
};
/** Where a synthetic dump places each part, by flash address. */
export interface EspFlashDumpOptions {
readonly size?: number;
readonly bootloader?: { readonly at: number; readonly bytes: Uint8Array };
readonly tableAt?: number;
readonly table?: Uint8Array;
readonly parts?: readonly {
readonly at: number;
readonly bytes: Uint8Array;
}[];
}
/** A whole-flash dump: erased flash with a bootloader, a table and contents. */
export const espFlashDump = ({
size = 0x4_0000,
bootloader,
tableAt = 0x8000,
table = partitionTable(TWO_OTA_PARTITIONS),
parts = [],
}: EspFlashDumpOptions = {}): Buffer =>
flashImage(size, [
...(bootloader === undefined ? [] : [bootloader]),
{ at: tableAt, bytes: table },
...parts,
]);
src/domain/firmware/espFlashBootloader.ts
+125
View File
@@ -0,0 +1,125 @@
import {
ESP_ROM_BOOTLOADER_OFFSETS,
espChipForId,
type EspChip,
} from "./espChips.js";
import { ESP_IMAGE_HEADER_BYTES, recognizeEspImage } from "./espImage.js";
import {
ESP_IMAGE_SEGMENT_HEADER_BYTES,
offsetText,
u16,
u32,
} from "./espImageStructures.js";
/** Bytes that recognition needs at a candidate bootloader offset. */
export const ESP_BOOTLOADER_PROBE_BYTES = 32;
/** Where the dump holds bytes for each ROM bootloader offset. */
export interface EspBootloaderProbe {
/** Flash address the ROM loads from. */
readonly flashAddress: number;
/** The same address in the file, given the dump's flash offset. */
readonly fileOffset: number;
/** Up to `ESP_BOOTLOADER_PROBE_BYTES` from that file offset. */
readonly prefix: Uint8Array;
/** Bytes from that file offset to the end of the file. */
readonly available: number;
}
export interface EspBootloaderLocation {
readonly flashAddress: number;
readonly fileOffset: number;
readonly chip: EspChip;
}
export interface EspBootloaderSearch {
readonly location: EspBootloaderLocation | undefined;
readonly limitations: readonly string[];
/**
* The first image of a known chip at another chip's ROM offset, which says
* the dump does not start where its origin says and may be the bootloader
* of a later origin.
*/
readonly misplacedImage:
| { readonly chip: EspChip; readonly fileOffset: number }
| undefined;
/** An ESP8266 image sits at a ROM offset. */
readonly esp8266: boolean;
}
/**
* The bootloader is the ESP image at the flash address its chip's ROM loads
* from (`ESP_ROM_BOOTLOADER_OFFSET_FLASH`). Without one, an image of a known
* chip at another chip's ROM offset says the dump may not start where its
* flash offset says, and an ESP8266 image says it is an ESP8266 dump.
*/
export const locateEspBootloader = (
probes: readonly EspBootloaderProbe[],
): EspBootloaderSearch => {
const limitations: string[] = [];
let misplacedImage: EspBootloaderSearch["misplacedImage"];
let esp8266 = false;
for (const probe of probes) {
const recognized = recognizeEspImage(probe.prefix, probe.available);
if (recognized === "esp8266") esp8266 = true;
if (recognized !== "esp-image") continue;
const chipId = u16(probe.prefix, 12);
const chip = espChipForId(chipId);
if (chip?.romBootloaderOffset === probe.flashAddress)
return {
location: {
flashAddress: probe.flashAddress,
fileOffset: probe.fileOffset,
chip,
},
limitations: [],
misplacedImage: undefined,
esp8266: false,
};
if (chip !== undefined)
misplacedImage ??= { chip, fileOffset: probe.fileOffset };
limitations.push(
chip === undefined
? `An ESP image at flash ${offsetText(probe.flashAddress)} has chip_id 0x${chipId.toString(16).padStart(4, "0")}, which is not in the chip table; it is not read as the bootloader.`
: `An ${chip.target} image is at flash ${offsetText(probe.flashAddress)}, but its ROM loads the bootloader from ${offsetText(chip.romBootloaderOffset)}; the dump may not start at flash ${offsetText(probe.flashAddress - probe.fileOffset)}.`,
);
}
return {
location: undefined,
limitations,
misplacedImage,
esp8266,
};
};
/**
* The file offset just past the last segment of the image at `fileOffset`,
* found by following its segment headers, or the file size when they run
* past the end. No partition table lies before it.
*/
export const espImageSegmentsEnd = async (
read: (position: number, length: number) => Promise<Uint8Array>,
fileOffset: number,
fileSize: number,
): Promise<number> => {
const header = await read(fileOffset, ESP_IMAGE_HEADER_BYTES);
if (header.length < ESP_IMAGE_HEADER_BYTES) return fileSize;
let cursor = fileOffset + ESP_IMAGE_HEADER_BYTES;
for (let index = 0; index < (header[1] ?? 0); index += 1) {
const segment = await read(cursor, ESP_IMAGE_SEGMENT_HEADER_BYTES);
if (segment.length < ESP_IMAGE_SEGMENT_HEADER_BYTES) return fileSize;
cursor += ESP_IMAGE_SEGMENT_HEADER_BYTES + u32(segment, 4);
if (cursor >= fileSize) return fileSize;
}
return cursor;
};
/** The file offsets that hold each ROM bootloader offset, given the dump's origin. */
export const espBootloaderProbeOffsets = (
flashOffset: number,
fileSize: number,
): readonly { readonly flashAddress: number; readonly fileOffset: number }[] =>
ESP_ROM_BOOTLOADER_OFFSETS.map((flashAddress) => ({
flashAddress,
fileOffset: flashAddress - flashOffset,
})).filter(({ fileOffset }) => fileOffset >= 0 && fileOffset < fileSize);
src/domain/firmware/espFlashLayout.test.ts
+493
View File
@@ -0,0 +1,493 @@
import { describe, expect, it } from "vitest";
import {
decodeEspPartitionTable,
espBootPartitions,
espPartitionSubtypeName,
espPartitionTypeName,
type EspPartition,
} from "./espPartitionTable.js";
import { decodeEspOtaRecord, selectEspBootPartition } from "./espOtaData.js";
import {
espBootloaderProbeOffsets,
espImageSegmentsEnd,
locateEspBootloader,
} from "./espFlashBootloader.js";
import {
TWO_OTA_PARTITIONS,
otaRecord,
partitionEntry,
partitionTable,
} from "./espFlash.fixture.js";
import {
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
} from "../../../tests/fixtures/espImage.js";
const table = (bytes: Uint8Array) => {
const decoded = decodeEspPartitionTable(bytes);
if (!decoded.ok) throw new Error(decoded.error);
return decoded.value;
};
describe("ESP partition tables", () => {
it("decodes entries, flags and a verified MD5 as gen_esp32part writes them", () => {
const decoded = table(
partitionTable([
...TWO_OTA_PARTITIONS,
{
label: "keys",
type: 0x01,
subtype: 0x04,
offset: 0x4_0000,
size: 0x1000,
flags: 0b11,
},
]),
);
expect(decoded.md5.status).toBe("valid");
expect(decoded.md5.entryOffset).toBe(7 * 32);
expect(decoded.length).toBe(9 * 32);
expect(decoded.limitations).toEqual([]);
expect(decoded.partitions.map(({ label }) => label)).toEqual([
"nvs",
"otadata",
"phy_init",
"factory",
"ota_0",
"ota_1",
"keys",
]);
expect(decoded.partitions.at(-1)).toMatchObject({
index: 6,
entryOffset: 6 * 32,
type: 0x01,
subtype: 0x04,
offset: 0x4_0000,
size: 0x1000,
encrypted: true,
readonly: true,
unknownFlags: 0,
});
});
it("accepts a table without an MD5 entry, as the bootloader does", () => {
const decoded = table(
partitionTable(TWO_OTA_PARTITIONS, { md5: "absent" }),
);
expect(decoded.md5).toEqual({
status: "absent",
entryOffset: null,
stored: null,
computed: null,
});
expect(decoded.partitions).toHaveLength(6);
});
it("keeps the entries of a table whose MD5 does not match", () => {
const decoded = table(
partitionTable(TWO_OTA_PARTITIONS, { md5: "corrupt" }),
);
expect(decoded.md5.status).toBe("invalid");
expect(decoded.partitions).toHaveLength(6);
expect(decoded.limitations).toEqual([
"The partition table MD5 at 0xc0 does not match the entries before it; ESP-IDF's bootloader rejects the table.",
]);
});
it.each([
["erased flash", Buffer.alloc(0xc00, 0xff), "the first entry is empty"],
[
"an MD5 entry alone",
partitionTable([]),
"entry 1 at 0x20 ends the table before any partition entry",
],
[
"a foreign entry",
Buffer.concat([
partitionEntry(TWO_OTA_PARTITIONS[0]!),
Buffer.alloc(32, 0),
Buffer.alloc(0xbc0, 0xff),
]),
"entry 1 at 0x20 has magic 0x0, not a partition, MD5 or terminating entry",
],
[
"no terminator",
Buffer.concat(
Array.from({ length: 96 }, () =>
partitionEntry(TWO_OTA_PARTITIONS[0]!),
),
),
"no terminating entry within 96 entries (0xc00 bytes)",
],
[
"short bytes",
partitionEntry(TWO_OTA_PARTITIONS[0]!),
"the bytes end at 0x20, inside entry 1",
],
])("rejects %s as not a table", (_name, bytes, reason) => {
expect(decodeEspPartitionTable(bytes)).toEqual({
ok: false,
error: reason,
});
});
it("names types and subtypes as ESP-IDF does, and keeps unknown flags and labels", () => {
expect(espPartitionTypeName(0x00)).toBe("app");
expect(espPartitionSubtypeName(0x00, 0x1f)).toBe("ota_15");
expect(espPartitionSubtypeName(0x01, 0x04)).toBe("nvs_keys");
expect(espPartitionTypeName(0x40)).toBeNull();
expect(espPartitionSubtypeName(0x40, 0x00)).toBeNull();
const entry = partitionEntry({
label: "x",
type: 0x40,
subtype: 0,
offset: 0x1_0000,
size: 0x1000,
flags: 0x8000_0001,
});
entry.writeUInt8(0xff, 13);
const [decoded] = table(
Buffer.concat([entry, Buffer.alloc(0xbe0, 0xff)]),
).partitions;
expect(decoded).toMatchObject({
label: null,
// The source bytes stay comparable when the label is not UTF-8.
labelHex: "78ff",
encrypted: true,
unknownFlags: 0x8000_0000,
});
});
it("keeps the partitions the bootloader keeps: the last of each role", () => {
const [nvs, otadata, , factory, ota0] = TWO_OTA_PARTITIONS;
const boot = espBootPartitions(
table(
partitionTable([
...TWO_OTA_PARTITIONS,
{ ...factory!, label: "factory2", offset: 0x4_0000 },
{ ...otadata!, label: "otadata2", offset: 0x5_0000 },
{ ...ota0!, label: "again", offset: 0x6_0000 },
{ ...nvs!, label: "nvs2", offset: 0x7_0000 },
]),
).partitions,
);
expect(boot.factory?.label).toBe("factory2");
expect(boot.otadata?.label).toBe("otadata2");
// A same-subtype OTA entry replaces the slot but still counts.
expect(boot.otaSlots.get(0)?.label).toBe("again");
expect(boot.otaAppCount).toBe(3);
expect(boot.limitations).toEqual([
"The table declares otadata 2 times (partitions #1, #7); ESP-IDF's bootloader uses the last.",
"The table declares factory 2 times (partitions #3, #6); ESP-IDF's bootloader uses the last.",
"The table declares ota_0 2 times (partitions #4, #8); ESP-IDF's bootloader uses the last.",
]);
});
});
const partitions = (): readonly EspPartition[] =>
table(partitionTable(TWO_OTA_PARTITIONS)).partitions;
const select = (
first: Buffer,
second: Buffer,
parts = partitions(),
otadataSize = 0x2000,
) =>
selectEspBootPartition(espBootPartitions(parts), {
offset: 0xd000,
size: otadataSize,
records: [decodeEspOtaRecord(first, 0), decodeEspOtaRecord(second, 1)],
});
const erased = Buffer.alloc(32, 0xff);
describe("ESP otadata selection", () => {
it("decodes records and checks the CRC the bootloader computes", () => {
expect(
decodeEspOtaRecord(otaRecord({ seq: 3, state: 2 }), 0),
).toMatchObject({
index: 0,
ota_seq: 3,
ota_state: 2,
state_name: "valid",
crc_matches: true,
valid: true,
});
expect(decodeEspOtaRecord(otaRecord({ seq: 3, crc: 0 }), 1)).toMatchObject({
crc_matches: false,
valid: false,
});
// The CRC otatool v5.5.2 wrote for ota_seq 1 in the real-tool lane.
expect(
decodeEspOtaRecord(otaRecord({ seq: 1, crc: 0x4743_989a }), 0),
).toMatchObject({ crc_matches: true, valid: true });
expect(decodeEspOtaRecord(Buffer.alloc(32, 0xff), 1)).toMatchObject({
ota_seq: 0xffff_ffff,
state_name: "undefined",
valid: false,
});
});
it("selects the higher valid sequence and maps it onto the OTA slots", () => {
expect(select(otaRecord({ seq: 4 }), otaRecord({ seq: 5 }))).toMatchObject({
active_record: 1,
slot: 0,
partition_index: 4,
basis: "otadata",
});
// An invalid state removes a record; a tie keeps record 0.
expect(
select(otaRecord({ seq: 6, state: 3 }), otaRecord({ seq: 5 })),
).toMatchObject({ active_record: 1, slot: 0 });
expect(select(otaRecord({ seq: 2 }), otaRecord({ seq: 2 }))).toMatchObject({
active_record: 0,
slot: 1,
partition_index: 5,
});
// ota_seq 0 is set; (0 - 1) wraps to 0xFFFFFFFF, and 0xFFFFFFFF % 2 is 1.
expect(select(otaRecord({ seq: 0 }), erased)).toMatchObject({
active_record: 0,
slot: 1,
partition_index: 5,
});
});
it("falls back as the bootloader does without usable records", () => {
expect(select(erased, erased)).toMatchObject({
basis: "factory-fallback",
partition_index: 3,
slot: null,
});
const noFactory = partitions().filter(({ label }) => label !== "factory");
expect(select(erased, erased, noFactory)).toMatchObject({
basis: "ota_0-fallback",
slot: 0,
partition_index: 4,
});
// Boot index 0 stands without an ota_0 partition.
const onlyOta1 = noFactory.filter(({ label }) => label !== "ota_0");
expect(select(erased, erased, onlyOta1)).toMatchObject({
basis: "ota_0-fallback",
slot: 0,
partition_index: null,
});
// A set sequence with a bad CRC is not unusable, but no record is valid:
// the factory index, with no factory partition to name.
expect(
select(otaRecord({ seq: 2, crc: 1 }), erased, noFactory),
).toMatchObject({ basis: "factory-fallback", partition_index: null });
expect(select(otaRecord({ seq: 2 }), erased)).toMatchObject({
running_partition: "unknown",
});
expect(select(otaRecord({ seq: 2 }), erased).assumptions).toHaveLength(4);
});
it("indexes OTA slots by subtype, so a missing subtype selects no partition", () => {
const gapped = partitions().filter(({ label }) => label !== "ota_0");
// Two OTA apps would be counted, but only ota_1 exists: seq 1 selects slot 0.
const withSecond = [
...gapped,
{ ...gapped.at(-1)!, index: 9, subtype: 0x12, label: "ota_2" },
];
expect(select(otaRecord({ seq: 1 }), erased, withSecond)).toMatchObject({
slot: 0,
partition_index: null,
basis: "otadata",
});
});
});
describe("ESP boot selection without a derivable slot", () => {
it("selects the factory index for a table without otadata", () => {
const withoutOtadata = partitions().filter(
({ label }) => label !== "otadata",
);
const selection = selectEspBootPartition(
espBootPartitions(withoutOtadata),
undefined,
);
expect(selection).toMatchObject({
basis: "no-otadata",
active_record: null,
slot: null,
partition_index: 3,
running_partition: "unknown",
});
// No record is read, so the records assumption does not apply.
expect(selection.assumptions).toHaveLength(3);
const factoryOnly = withoutOtadata.filter(
({ label }) => !label?.startsWith("ota_"),
);
expect(
selectEspBootPartition(espBootPartitions(factoryOnly), undefined),
).toMatchObject({ basis: "no-otadata", partition_index: 3 });
// Without a factory app the bootloader still selects the factory index.
expect(
selectEspBootPartition(
espBootPartitions(
withoutOtadata.filter(({ label }) => label !== "factory"),
),
undefined,
),
).toMatchObject({ basis: "no-otadata", partition_index: null });
});
it("selects nothing from a short otadata partition and nothing known from one record", () => {
expect(
select(
otaRecord({ seq: 1 }),
otaRecord({ seq: 2 }),
partitions(),
0x1000,
),
).toEqual({
basis: "none",
active_record: null,
slot: null,
partition_index: null,
assumptions: [],
running_partition: "unknown",
});
const oneRecord = {
offset: 0xd000,
size: 0x2000,
records: [decodeEspOtaRecord(otaRecord({ seq: 1 }), 0)],
};
expect(
selectEspBootPartition(espBootPartitions(partitions()), oneRecord),
).toMatchObject({
basis: "unknown",
partition_index: null,
assumptions: [],
});
// Without an OTA app the bootloader does not need the records.
const noOtaApps = partitions().filter(
({ label }) => !label?.startsWith("ota_"),
);
expect(
selectEspBootPartition(espBootPartitions(noOtaApps), oneRecord),
).toMatchObject({ basis: "factory-fallback", partition_index: 3 });
});
it("selects nothing from a rejected table and treats an otadata at offset 0 as absent", () => {
// An MD5 mismatch fails esp_partition_table_verify before any selection.
expect(
select(otaRecord({ seq: 1 }), erased, partitions(), 0x2000),
).toMatchObject({
basis: "otadata",
});
expect(
selectEspBootPartition(
espBootPartitions(partitions()),
{
offset: 0xd000,
size: 0x2000,
records: [
decodeEspOtaRecord(otaRecord({ seq: 1 }), 0),
decodeEspOtaRecord(erased, 1),
],
},
true,
),
).toEqual({
basis: "none",
active_record: null,
slot: null,
partition_index: null,
assumptions: [],
running_partition: "unknown",
});
// bootloader_utility_get_selected_boot_partition takes offset 0 for no otadata.
expect(
selectEspBootPartition(espBootPartitions(partitions()), {
offset: 0,
size: 0x2000,
records: [],
}),
).toMatchObject({ basis: "no-otadata", partition_index: 3 });
});
});
describe("ESP bootloader location", () => {
const bootloader = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0005,
});
const probe = (flashAddress: number, bytes: Uint8Array) => ({
flashAddress,
fileOffset: flashAddress,
prefix: bytes.subarray(0, 32),
available: bytes.length,
});
it("locates the image at its chip's ROM offset", () => {
expect(locateEspBootloader([probe(0, bootloader)])).toMatchObject({
location: { flashAddress: 0, fileOffset: 0, chip: { target: "esp32c3" } },
limitations: [],
});
});
it("reports an image at another chip's ROM offset instead of locating it", () => {
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
expect(locateEspBootloader([probe(0, esp32)])).toEqual({
location: undefined,
limitations: [
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at flash 0x0.",
],
// Its chip and offset let the caller look for the origin it implies.
misplacedImage: {
chip: expect.objectContaining({ target: "esp32" }),
fileOffset: 0,
},
esp8266: false,
});
});
it("tells an unknown chip and an ESP8266 image from a misplaced origin", () => {
const unknown = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x7777,
});
expect(locateEspBootloader([probe(0, unknown)])).toMatchObject({
location: undefined,
misplacedImage: undefined,
esp8266: false,
});
expect(locateEspBootloader([probe(0, esp8266Image())])).toMatchObject({
location: undefined,
limitations: [],
misplacedImage: undefined,
esp8266: true,
});
});
it("finds where a located image's segments end by their headers", async () => {
const read = async (position: number, length: number) =>
bootloader.subarray(position, position + length);
const declared = ESP32C3_BOOTLOADER_SEGMENTS.reduce(
(end, { data }) => end + 8 + data.length,
24,
);
expect(await espImageSegmentsEnd(read, 0, bootloader.length)).toBe(
declared,
);
// Segments that run past the file end at the file.
expect(await espImageSegmentsEnd(read, 0, 40)).toBe(40);
});
it("probes only the ROM offsets the file holds after its flash offset", () => {
expect(espBootloaderProbeOffsets(0, 0x10_0000)).toEqual([
{ flashAddress: 0, fileOffset: 0 },
{ flashAddress: 0x1000, fileOffset: 0x1000 },
{ flashAddress: 0x2000, fileOffset: 0x2000 },
]);
expect(espBootloaderProbeOffsets(0x1000, 0x10_0000)).toEqual([
{ flashAddress: 0x1000, fileOffset: 0 },
{ flashAddress: 0x2000, fileOffset: 0x1000 },
]);
});
});
src/domain/firmware/espFlashLayout.ts
+449
View File
@@ -0,0 +1,449 @@
import { offsetText } from "./espImageStructures.js";
import type { EspBootloaderLocation } from "./espFlashBootloader.js";
import {
ESP_PARTITION_TABLE_SECTOR_BYTES,
espBootPartitions,
espPartitionSubtypeName,
espPartitionTypeName,
espPrimaryLayoutRole,
type EspBootPartitions,
type EspPartition,
type EspPartitionTable,
} from "./espPartitionTable.js";
import {
ESP_OTA_DATA_MIN_BYTES,
ESP_OTA_DATA_SECTOR_BYTES,
ESP_OTA_SELECT_ENTRY_BYTES,
decodeEspOtaRecord,
selectEspBootPartition,
type EspBootSelection,
type EspOtaRecord,
} from "./espOtaData.js";
/** Where the file's first byte sits in flash, and who said so. */
export interface EspFlashOrigin {
readonly value: number;
readonly source: "default" | "caller";
}
export type EspPartitionTableMethod = "default" | "caller" | "scan";
/** One described byte range of the dump, in flash and in the file. */
export interface EspFlashRegion {
readonly path: string;
readonly role: "bootloader" | "partition-table" | "partition";
readonly flashAddress: number;
/** Length the layout gives the region. */
readonly length: number;
readonly fileOffset: number;
/** Bytes of the region the file holds, from `fileOffset`. */
readonly available: number;
readonly partition: EspPartition | undefined;
}
export interface EspFlashPlanInput {
readonly fileSize: number;
readonly origin: EspFlashOrigin;
readonly bootloader: EspBootloaderLocation | undefined;
readonly tableAddress: number;
readonly tableMethod: EspPartitionTableMethod;
readonly table: EspPartitionTable;
}
/** One otadata record within its partition, and whether the file holds it. */
export interface EspOtaRecordPlan {
readonly index: 0 | 1;
readonly fileOffset: number;
readonly available: boolean;
}
/** Regions to inventory and the byte ranges the adapter must still read. */
export interface EspFlashPlan {
readonly input: EspFlashPlanInput;
readonly regions: readonly EspFlashRegion[];
/** File ranges no region describes; erased flash there is described. */
readonly gaps: readonly {
readonly offset: number;
readonly length: number;
}[];
/** The partitions the bootloader keeps from the table. */
readonly boot: EspBootPartitions;
/** The otadata partition the bootloader uses and its records. */
readonly otadata:
| {
readonly partition: EspPartition;
readonly records: readonly EspOtaRecordPlan[];
}
| undefined;
readonly limitations: readonly string[];
}
const SAFE_LABEL = /^(?=.*[A-Za-z_])[A-Za-z0-9_][A-Za-z0-9._-]{0,15}$/u;
/** Windows reserves these device names, alone or before any extension. */
const WINDOWS_DEVICE_NAME =
/^(?:con|prn|aux|nul|com[0-9¹²³]|lpt[0-9¹²³])(?:\.|$)/iu;
/**
* Partition paths use the label when it is a safe path segment with a
* letter or underscore that is not a Windows device name and that no other
* label repeats, ignoring case as some file systems do; else the table
* index, which cannot collide with a label.
*/
const partitionPaths = (
partitions: readonly EspPartition[],
): ReadonlyMap<number, string> => {
const counts = new Map<string, number>();
for (const { label } of partitions)
if (label !== null) {
const folded = label.toLowerCase();
counts.set(folded, (counts.get(folded) ?? 0) + 1);
}
return new Map(
partitions.map((part) => {
const label = part.label;
const named =
label !== null &&
SAFE_LABEL.test(label) &&
!WINDOWS_DEVICE_NAME.test(label) &&
counts.get(label.toLowerCase()) === 1;
return [
part.index,
`partitions/${named ? label : String(part.index)}.bin`,
];
}),
);
};
const region = (
input: EspFlashPlanInput,
fields: Omit<EspFlashRegion, "fileOffset" | "available">,
): EspFlashRegion => {
const fileOffset = fields.flashAddress - input.origin.value;
const start = Math.max(0, fileOffset);
const end = Math.min(input.fileSize, fileOffset + fields.length);
return { ...fields, fileOffset, available: Math.max(0, end - start) };
};
const rangeText = (address: number, length: number): string =>
`${offsetText(address)}-${offsetText(address + length)}`;
const partitionName = (part: EspPartition): string =>
part.label ?? `#${String(part.index)}`;
/** The records the otadata partition holds, each where the file holds it. */
const otadataPlan = (
input: EspFlashPlanInput,
partition: EspPartition,
limitations: string[],
): NonNullable<EspFlashPlan["otadata"]> => {
const name = partitionName(partition);
const small = partition.size < ESP_OTA_DATA_MIN_BYTES;
if (small)
limitations.push(
`The otadata partition ${name} is ${offsetText(partition.size)} bytes, smaller than the two 4 KiB sectors ESP-IDF's bootloader reads; the bootloader resets without selecting an app.`,
);
const records: EspOtaRecordPlan[] = [];
for (const index of [0, 1] as const) {
const within = index * ESP_OTA_DATA_SECTOR_BYTES;
// A record past the declared size belongs to whatever follows.
if (within + ESP_OTA_SELECT_ENTRY_BYTES > partition.size) continue;
const fileOffset = partition.offset + within - input.origin.value;
const available =
fileOffset >= 0 &&
fileOffset + ESP_OTA_SELECT_ENTRY_BYTES <= input.fileSize;
if (!available)
limitations.push(
`otadata record ${String(index)} of partition ${name} at flash ${offsetText(partition.offset + within)} is outside the dump${small ? "." : "; the OTA selection is unknown."}`,
);
records.push({ index, fileOffset, available });
}
return { partition, records };
};
/** Lay the bootloader, table and partitions over the dump. */
export const planEspFlashLayout = (input: EspFlashPlanInput): EspFlashPlan => {
const limitations: string[] = [...input.table.limitations];
const regions: EspFlashRegion[] = [];
const tableEnd = input.tableAddress + ESP_PARTITION_TABLE_SECTOR_BYTES;
const { bootloader } = input;
if (bootloader !== undefined) {
if (bootloader.flashAddress < input.tableAddress)
regions.push(
region(input, {
path: "bootloader.bin",
role: "bootloader",
// ESP_BOOTLOADER_SIZE: from the bootloader to the partition table.
flashAddress: bootloader.flashAddress,
length: input.tableAddress - bootloader.flashAddress,
partition: undefined,
}),
);
else
limitations.push(
`The partition table at ${offsetText(input.tableAddress)} does not follow the bootloader at ${offsetText(bootloader.flashAddress)}; the bootloader region is not described.`,
);
}
regions.push(
region(input, {
path: "partition-table.bin",
role: "partition-table",
flashAddress: input.tableAddress,
length: ESP_PARTITION_TABLE_SECTOR_BYTES,
partition: undefined,
}),
);
const paths = partitionPaths(input.table.partitions);
for (const part of input.table.partitions) {
const placed = region(input, {
path: paths.get(part.index) ?? `partitions/${String(part.index)}.bin`,
role: "partition",
flashAddress: part.offset,
length: part.size,
partition: part,
});
const name = partitionName(part);
if (part.offset < tableEnd && espPrimaryLayoutRole(part) === undefined)
limitations.push(
`Partition ${name} at ${offsetText(part.offset)} starts before the end of the partition table at ${offsetText(tableEnd)}; gen_esp32part.py rejects such a table.`,
);
if (placed.fileOffset < 0)
limitations.push(
`Partition ${name} at ${offsetText(part.offset)} begins before the dump's first byte at flash ${offsetText(input.origin.value)}; it is not read.`,
);
else if (placed.available === 0)
limitations.push(
`Partition ${name} at ${rangeText(part.offset, part.size)} lies outside the dump, which ends at flash ${offsetText(input.origin.value + input.fileSize)}.`,
);
else if (placed.available < part.size)
limitations.push(
`Partition ${name} at ${rangeText(part.offset, part.size)} is truncated: the dump holds ${String(placed.available)} of its ${String(part.size)} bytes.`,
);
// Every partition from the dump's first byte on is an entry holding the
// bytes the file has, none when it lies past the end, so comparison sees
// what the file lacks instead of a missing path.
if (placed.fileOffset >= 0) regions.push(placed);
}
limitations.push(...overlaps(regions));
const boot = espBootPartitions(input.table.partitions);
limitations.push(...boot.limitations);
return {
input,
regions,
gaps: gapsBetween(regions, input.fileSize),
boot,
otadata:
boot.otadata === undefined
? undefined
: otadataPlan(input, boot.otadata, limitations),
limitations,
};
};
const regionEnd = ({ flashAddress, length }: EspFlashRegion): number =>
flashAddress + length;
/** A primary bootloader or table entry declared over exactly its region. */
const declaresRegion = (
first: EspFlashRegion,
second: EspFlashRegion,
): boolean => {
const [region, entry] =
first.partition === undefined ? [first, second] : [second, first];
return (
region.partition === undefined &&
entry.partition !== undefined &&
espPrimaryLayoutRole(entry.partition) === region.role &&
region.flashAddress === entry.flashAddress &&
region.length === entry.length
);
};
/** Regions that share bytes, which gen_esp32part.py rejects for partitions. */
const overlaps = (regions: readonly EspFlashRegion[]): string[] => {
const ordered = [...regions].sort(
(left, right) => left.flashAddress - right.flashAddress,
);
const found: string[] = [];
// The region reaching furthest so far, so a long one shows every overlap.
let furthest: EspFlashRegion | undefined;
for (const current of ordered) {
if (
furthest !== undefined &&
current.flashAddress < regionEnd(furthest) &&
!declaresRegion(furthest, current)
)
found.push(
`${current.path} at ${offsetText(current.flashAddress)} overlaps ${furthest.path}, which ends at ${offsetText(regionEnd(furthest))}.`,
);
if (furthest === undefined || regionEnd(current) > regionEnd(furthest))
furthest = current;
}
return found;
};
/** File ranges outside every region, in order. */
const gapsBetween = (
regions: readonly EspFlashRegion[],
fileSize: number,
): { offset: number; length: number }[] => {
const covered = regions
.map(({ fileOffset, available }) => ({
start: Math.max(0, fileOffset),
end: Math.max(0, fileOffset) + available,
}))
.filter(({ end, start }) => end > start)
.sort((left, right) => left.start - right.start);
const gaps: { offset: number; length: number }[] = [];
let cursor = 0;
for (const { start, end } of covered) {
if (start > cursor) gaps.push({ offset: cursor, length: start - cursor });
cursor = Math.max(cursor, end);
}
if (fileSize > cursor)
gaps.push({ offset: cursor, length: fileSize - cursor });
return gaps;
};
export interface EspFlashOtaFacts {
readonly partition_index: number;
/** The records the dump holds, each with its index. */
readonly records: readonly EspOtaRecord[];
}
/** Facts the adapter read for the plan: the otadata records and the gaps. */
export interface EspFlashReads {
/** The bytes of each planned otadata record the file holds, in plan order. */
readonly otadataRecords: readonly (Uint8Array | undefined)[];
/** The first non-erased file offset in each gap, in plan order. */
readonly gapFirstUnerased: readonly (number | undefined)[];
}
export interface EspFlashLayoutFacts {
readonly otadata: EspFlashOtaFacts | null;
/** Derived for every table, with or without otadata; see its basis. */
readonly boot_selection: EspBootSelection;
readonly undescribed: readonly { offset: number; length: number }[];
readonly coverage: "complete" | "unknown";
readonly limitations: readonly string[];
}
/** Complete the plan with the bytes the adapter read. */
export const finishEspFlashLayout = (
plan: EspFlashPlan,
reads: EspFlashReads,
): EspFlashLayoutFacts => {
const limitations = [...plan.limitations];
const undescribed: { offset: number; length: number }[] = [];
plan.gaps.forEach((gap, index) => {
const first = reads.gapFirstUnerased[index];
if (first !== undefined)
undescribed.push({
offset: first,
length: gap.offset + gap.length - first,
});
});
const total = undescribed.reduce((sum, { length }) => sum + length, 0);
if (undescribed.length > 0)
limitations.push(
`${String(undescribed.length)} range(s) of the dump, ${String(total)} bytes in all, lie outside the bootloader region, the partition table and every partition and are not described; each runs from a byte that is not erased flash to the end of its gap.`,
);
const otadata =
plan.otadata === undefined
? null
: otaFacts(plan.otadata, reads.otadataRecords);
// Every partition from the dump's first byte on has an entry holding the
// bytes the file has; only one that begins before it has none.
const unplaced = plan.input.table.partitions.some(
(part) => !plan.regions.some(({ partition }) => partition === part),
);
return {
otadata,
boot_selection: selectEspBootPartition(
plan.boot,
plan.otadata === undefined || otadata === null
? undefined
: {
offset: plan.otadata.partition.offset,
size: plan.otadata.partition.size,
records: otadata.records,
},
plan.input.table.md5.status === "invalid",
),
undescribed,
coverage:
undescribed.length > 0 || plan.input.bootloader === undefined || unplaced
? "unknown"
: "complete",
limitations,
};
};
const otaFacts = (
otadata: NonNullable<EspFlashPlan["otadata"]>,
bytes: readonly (Uint8Array | undefined)[],
): EspFlashOtaFacts => ({
partition_index: otadata.partition.index,
records: otadata.records.flatMap(({ index, available }, position) => {
const read = bytes[position];
return available &&
read !== undefined &&
read.length >= ESP_OTA_SELECT_ENTRY_BYTES
? [decodeEspOtaRecord(read, index)]
: [];
}),
});
/**
* Where the bytes of a partition the dump holds are not erased flash,
* relative to the partition's start: one range from the first such byte to
* just past the last, and how many such bytes there are.
*/
export interface EspFlashWrittenExtent {
/** The first byte that is not 0xFF, or null when every byte is. */
readonly offset: number | null;
/** Just past the last byte that is not 0xFF, or null. */
readonly end: number | null;
/** How many of the bytes are not 0xFF. */
readonly bytes: number;
}
/**
* Flat entry metadata for one partition: what the table declares, and where
* the bytes the dump holds are written. `written` is undefined for a
* partition the dump holds none of.
*/
export const espFlashPartitionFacts = (
part: EspPartition,
written?: EspFlashWrittenExtent,
) => ({
kind: "esp-flash-partition" as const,
index: part.index,
label: part.label,
label_hex: part.labelHex,
type: part.type,
subtype: part.subtype,
type_name: espPartitionTypeName(part.type),
subtype_name: espPartitionSubtypeName(part.type, part.subtype),
flash_address: part.offset,
size: part.size,
encrypted: part.encrypted,
readonly: part.readonly,
unknown_flags: part.unknownFlags,
written_offset: written?.offset ?? null,
written_end: written?.end ?? null,
written_bytes: written?.bytes ?? null,
});
/**
* An app slot whose header bytes are erased holds no image for the
* bootloader, which then treats the slot as empty; bytes written later in
* it, such as an older image, are reported but not read as an image.
*/
export const espErasedSlotLimitation = (
path: string,
written: EspFlashWrittenExtent,
): string | undefined =>
written.offset === null || written.end === null
? undefined
: `${path} begins with erased flash, so ESP-IDF's bootloader finds no app in it, but ${String(written.bytes)} of its bytes from offset ${offsetText(written.offset)} to ${offsetText(written.end)} are not erased flash; they are not read as an image.`;
src/domain/firmware/espFlashLocation.test.ts
+337
View File
@@ -0,0 +1,337 @@
import { describe, expect, it } from "vitest";
import { espChipForId } from "./espChips.js";
import {
espFlashDumpOrigin,
locateEspFlashLayout,
type EspFlashSelection,
} from "./espFlashLocation.js";
import { ESP8266_REFUSAL } from "./espImage.js";
import {
TWO_OTA_PARTITIONS,
flashImage,
partitionTable,
} from "./espFlash.fixture.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
} from "../../../tests/fixtures/espImage.js";
const c3Bootloader = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
const esp32Bootloader = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
const locate = (
bytes: Uint8Array,
selection: EspFlashSelection = {},
explicit = false,
reads: [number, number][] = [],
) =>
locateEspFlashLayout(
bytes.length,
async (position, length) => {
reads.push([position, length]);
return bytes.subarray(
Math.max(0, position),
Math.max(0, position) + length,
);
},
selection,
{ explicit },
);
const dump = (
tableAt: number,
{
bootloader = c3Bootloader,
bootloaderAt = 0,
md5 = "valid",
}: {
readonly bootloader?: Uint8Array;
readonly bootloaderAt?: number;
readonly md5?: "valid" | "corrupt" | "absent";
} = {},
) =>
flashImage(0x2_0000, [
{ at: bootloaderAt, bytes: bootloader },
{ at: tableAt, bytes: partitionTable(TWO_OTA_PARTITIONS, { md5 }) },
]);
/** An ESP32 dump read from flash 0x1000: its bootloader is the file's first byte. */
const fromSecondSector = (tableAt: number) =>
dump(tableAt, { bootloader: esp32Bootloader, bootloaderAt: 0x1000 }).subarray(
0x1000,
);
const UNCONFIRMED =
"No bootloader at a ROM offset confirms that the dump starts at flash 0x0; each partition's file offset assumes it. Pass flash_offset if the dump starts elsewhere.";
describe("ESP flash layout location", () => {
it("finds the bootloader and the table at their default offsets", async () => {
const located = await locate(dump(0x8000));
expect(located.ok && located.value).toMatchObject({
origin: { value: 0, source: "default" },
bootloader: { flashAddress: 0, chip: { target: "esp32c3" } },
tableAddress: 0x8000,
tableMethod: "default",
table: { md5: { status: "valid" } },
limitations: [],
});
});
it("locates a bootloader at the 0x2000 ROM offset of the chips that use it", async () => {
const c5 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x17,
});
const located = await locate(
dump(0x8000, { bootloader: c5, bootloaderAt: 0x2000 }),
);
expect(located.ok && located.value).toMatchObject({
bootloader: { flashAddress: 0x2000, chip: { target: "esp32c5" } },
tableMethod: "default",
});
});
it("scans for a relocated table only when its MD5 verifies", async () => {
const located = await locate(dump(0xa000));
expect(located.ok && located.value).toMatchObject({
tableAddress: 0xa000,
tableMethod: "scan",
});
expect(await locate(dump(0xa000, { md5: "absent" }))).toEqual({
ok: false,
error:
"no partition table at 0x8000 or in any 0x1000-aligned flash sector after the bootloader image with a verified MD5; flash encryption is one cause",
});
});
it.each([0x1_0000, 0x1_1000])(
"scans across read chunks to a table at %s",
async (tableAt) => {
const located = await locate(dump(tableAt));
expect(located.ok && located.value).toMatchObject({
tableAddress: tableAt,
tableMethod: "scan",
});
},
);
it("scans the sectors of a dump whose origin is not sector-aligned", async () => {
// Flash 0x9000 is file offset 0x8800 when the dump starts at flash 0x800.
const bytes = flashImage(0x2_0000, [
{ at: 0x8800, bytes: partitionTable(TWO_OTA_PARTITIONS) },
]);
const located = await locate(bytes, { flashOffset: 0x800 }, true);
expect(located.ok && located.value).toMatchObject({
origin: { value: 0x800, source: "caller" },
tableAddress: 0x9000,
tableMethod: "scan",
limitations: [],
});
});
it("does not look for a table inside a single app image", async () => {
const app = espImage({
segments: [
...ESP32C3_APP_SEGMENTS,
{ address: 0x4038_0000, data: Buffer.alloc(0x2_0000, 0x5a) },
],
});
const reads: [number, number][] = [];
expect((await locate(app, {}, false, reads)).ok).toBe(false);
// Probes and segment headers only: no table read, no scan chunk.
expect(reads.every(([, length]) => length <= 32)).toBe(true);
});
it("uses the caller's table offset and does not scan", async () => {
const located = await locate(dump(0xa000), {
partitionTableOffset: 0xa000,
});
expect(located.ok && located.value).toMatchObject({
tableAddress: 0xa000,
tableMethod: "caller",
});
expect(
await locate(dump(0xa000), { partitionTableOffset: 0x9000 }),
).toEqual({
ok: false,
error:
"no partition table at the given offset 0x9000: the first entry is empty",
});
});
it("does not read a lone MD5 entry as a table", async () => {
const md5Only = flashImage(0x2_0000, [
{ at: 0, bytes: c3Bootloader },
{ at: 0x8000, bytes: partitionTable([]) },
]);
expect((await locate(md5Only)).ok).toBe(false);
expect(await locate(md5Only, { partitionTableOffset: 0x8000 })).toEqual({
ok: false,
error:
"no partition table at the given offset 0x8000: entry 1 at 0x20 ends the table before any partition entry",
});
});
it("rejects a table offset outside the dump", async () => {
expect(
await locate(dump(0x8000), { partitionTableOffset: 0x10_0000 }),
).toEqual({
ok: false,
error:
"the partition table offset 0x100000 is outside the dump, which holds flash 0x0-0x20000",
});
});
});
describe("ESP flash layout origins", () => {
it("reads a dump that starts after flash address 0 at the caller's origin", async () => {
const located = await locate(fromSecondSector(0x8000), {
flashOffset: 0x1000,
});
expect(located.ok && located.value).toMatchObject({
origin: { value: 0x1000, source: "caller" },
bootloader: {
flashAddress: 0x1000,
fileOffset: 0,
chip: { target: "esp32" },
},
tableAddress: 0x8000,
tableMethod: "default",
limitations: [],
});
});
it.each([
["omitted", {}],
["given as 0x0", { flashOffset: 0 }],
] as const)(
"refuses an origin the bootloader contradicts when it is %s",
async (_name, selection) => {
// The table sits at flash 0x9000, so the default 0x8000 would find it
// at file 0x8000 and misplace every partition by 0x1000.
const refused = await locate(fromSecondSector(0x9000), selection);
expect(refused).toEqual({
ok: false,
error:
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at flash 0x0. Pass the flash_offset where the dump starts, or partition_table_offset",
});
},
);
it("names the origin a misplaced bootloader and the table at its 0x8000 show", async () => {
// Read from 0x1000: the bootloader at file 0, the table at file 0x7000.
const dump = fromSecondSector(0x8000);
const reader = (bytes: Uint8Array) => async (at: number, length: number) =>
bytes.subarray(at, at + length);
const esp32 = espChipForId(0x0000)!;
const image = { chip: esp32, fileOffset: 0, end: esp32Bootloader.length };
expect(await espFlashDumpOrigin(reader(dump), dump.length, image)).toEqual({
flashOffset: 0x1000,
tableFileOffset: 0x7000,
});
expect(await locate(dump)).toEqual({
ok: false,
error:
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at flash 0x0. A partition table at file offset 0x7000 shows the dump was read from flash 0x1000; pass flash_offset 0x1000",
});
// No table there, a table the image holds, or a chip whose ROM loads
// from 0x0 shows no origin.
const relocated = fromSecondSector(0x9000);
expect(
await espFlashDumpOrigin(reader(relocated), relocated.length, image),
).toBeUndefined();
expect(
await espFlashDumpOrigin(reader(dump), dump.length, {
...image,
end: 0x7001,
}),
).toBeUndefined();
expect(
await espFlashDumpOrigin(reader(dump), dump.length, {
...image,
chip: espChipForId(0x0005)!,
}),
).toBeUndefined();
});
it("names no origin from a table inside the misplaced image's segments", async () => {
// The table bytes at file 0x7000 are segment data, not flash 0x8000.
const data = Buffer.alloc(0x7400);
partitionTable(TWO_OTA_PARTITIONS).copy(data, 0x7000 - 0x20);
const spanning = espImage({
segments: [{ address: 0x3ffe_0000, data }],
chipId: 0x0000,
});
expect(
await locate(flashImage(0x1_0000, [{ at: 0, bytes: spanning }])),
).toEqual({
ok: false,
error:
"An esp32 image is at flash 0x0, but its ROM loads the bootloader from 0x1000; the dump may not start at flash 0x0. Pass the flash_offset where the dump starts, or partition_table_offset",
});
});
it("finds the table after the bootloader once the origin is right", async () => {
const located = await locate(fromSecondSector(0x9000), {
flashOffset: 0x1000,
});
expect(located.ok && located.value).toMatchObject({
tableAddress: 0x9000,
tableMethod: "scan",
});
});
it("does not scan arbitrary files without a bootloader unless the format was selected", async () => {
const bare = flashImage(0x2_0000, [
{ at: 0xa000, bytes: partitionTable(TWO_OTA_PARTITIONS) },
]);
expect(await locate(bare)).toEqual({
ok: false,
error: "no partition table at 0x8000 and no bootloader at a ROM offset",
});
const explicit = await locate(bare, {}, true);
expect(explicit.ok && explicit.value).toMatchObject({
bootloader: undefined,
tableAddress: 0xa000,
tableMethod: "scan",
limitations: [UNCONFIRMED],
});
});
it("reads an unknown chip's dump at its default table without calling it misplaced", async () => {
const unknown = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x7777,
});
const located = await locate(dump(0x8000, { bootloader: unknown }));
expect(located.ok && located.value).toMatchObject({
bootloader: undefined,
tableMethod: "default",
limitations: [
"An ESP image at flash 0x0 has chip_id 0x7777, which is not in the chip table; it is not read as the bootloader.",
UNCONFIRMED,
],
});
});
it("refuses an ESP8266 dump as the ESP image reader does", async () => {
const esp8266 = flashImage(0x2_0000, [
{ at: 0, bytes: esp8266Image() },
{ at: 0x8000, bytes: partitionTable(TWO_OTA_PARTITIONS) },
]);
expect(await locate(esp8266)).toEqual({
ok: false,
error: ESP8266_REFUSAL,
});
expect(await locate(esp8266, {}, true)).toEqual({
ok: false,
error: ESP8266_REFUSAL,
});
});
});
src/domain/firmware/espFlashLocation.ts
+264
View File
@@ -0,0 +1,264 @@
import { err, ok, type Result } from "../result.js";
import {
ESP_BOOTLOADER_PROBE_BYTES,
espBootloaderProbeOffsets,
espImageSegmentsEnd,
locateEspBootloader,
type EspBootloaderLocation,
} from "./espFlashBootloader.js";
import type {
EspFlashOrigin,
EspPartitionTableMethod,
} from "./espFlashLayout.js";
import type { EspChip } from "./espChips.js";
import { ESP8266_REFUSAL } from "./espImage.js";
import { offsetText } from "./espImageStructures.js";
import {
ESP_PARTITION_TABLE_DEFAULT_OFFSET,
ESP_PARTITION_TABLE_MAX_BYTES,
ESP_PARTITION_TABLE_SECTOR_BYTES,
decodeEspPartitionTable,
type EspPartitionTable,
} from "./espPartitionTable.js";
/** Caller inputs that select how a dump is read; omitted means the default. */
export interface EspFlashSelection {
/** Flash address of the dump's first byte. */
readonly flashOffset?: number | undefined;
/** Flash address of the partition table; it disables the scan. */
readonly partitionTableOffset?: number | undefined;
}
/** Where the layout sits in the dump, found once and shared by its readers. */
export interface EspFlashLocation {
readonly origin: EspFlashOrigin;
readonly bootloader: EspBootloaderLocation | undefined;
readonly tableAddress: number;
readonly tableMethod: EspPartitionTableMethod;
readonly table: EspPartitionTable;
readonly limitations: readonly string[];
}
/** Read up to `length` bytes at a file offset; fewer means end of file. */
export type EspFlashRead = (
position: number,
length: number,
) => Promise<Uint8Array>;
const SCAN_CHUNK_BYTES = 0x1_0000;
const MAGIC_LOW = 0xaa;
const MAGIC_HIGH = 0x50;
const tableAt = async (
read: EspFlashRead,
fileOffset: number,
): Promise<Result<EspPartitionTable, string>> =>
decodeEspPartitionTable(
await read(fileOffset, ESP_PARTITION_TABLE_MAX_BYTES),
);
/** Where a dump read from a later flash address starts, as its bytes show. */
export interface EspFlashDumpOrigin {
/** The flash address of the file's first byte. */
readonly flashOffset: number;
/** The file offset of the partition table at that origin's `0x8000`. */
readonly tableFileOffset: number;
}
/**
* An image of a chip whose ROM loads the bootloader from flash `R`, at file
* offset `F`, is the bootloader of a dump read from flash `R - F` when a
* partition table sits where that origin puts the default table, at file
* offset `0x8000 - (R - F)`, at or after the image's `end`. An ESP32 dump
* read from `0x1000`, as `read-flash 0x1000 …` or
* `merge-bin --target-offset 0x1000` writes it, has its bootloader at file
* offset 0 and its table at `0x7000`.
*/
export const espFlashDumpOrigin = async (
read: EspFlashRead,
fileSize: number,
image: {
readonly chip: EspChip;
readonly fileOffset: number;
/** The file offset where the image ends; no table lies before it. */
readonly end: number;
},
): Promise<EspFlashDumpOrigin | undefined> => {
const flashOffset = image.chip.romBootloaderOffset - image.fileOffset;
const tableFileOffset = ESP_PARTITION_TABLE_DEFAULT_OFFSET - flashOffset;
if (
flashOffset <= 0 ||
tableFileOffset < image.end ||
tableFileOffset >= fileSize
)
return undefined;
// As at the default offset, the MD5 need not verify: the layout reports a
// mismatch, and the bootloader's rejection, as facts of the table.
return (await tableAt(read, tableFileOffset)).ok
? { flashOffset, tableFileOffset }
: undefined;
};
/**
* The first table whose MD5 verifies at a flash sector boundary from the
* file offset `start`; requiring the MD5 keeps arbitrary bytes that begin
* with the magic from passing. Chunks start at sector boundaries, so each
* holds whole sector heads.
*/
const scanForTable = async (
read: EspFlashRead,
{
fileSize,
origin,
start,
}: { fileSize: number; origin: number; start: number },
signal?: AbortSignal,
): Promise<{ address: number; table: EspPartitionTable } | undefined> => {
const sector = ESP_PARTITION_TABLE_SECTOR_BYTES;
// The file offset of the first sector boundary at or after `start`.
const boundary = (sector - (origin % sector)) % sector;
let position =
start <= boundary
? boundary
: boundary + Math.ceil((start - boundary) / sector) * sector;
let chunkStart = position;
let bytes: Uint8Array = new Uint8Array(0);
for (; position < fileSize; position += sector) {
if (position - chunkStart + 2 > bytes.length) {
signal?.throwIfAborted();
chunkStart = position;
bytes = await read(position, SCAN_CHUNK_BYTES);
}
const at = position - chunkStart;
if (bytes[at] !== MAGIC_LOW || bytes[at + 1] !== MAGIC_HIGH) continue;
const decoded = await tableAt(read, position);
if (decoded.ok && decoded.value.md5.status === "valid")
return { address: origin + position, table: decoded.value };
}
return undefined;
};
/**
* Locate the bootloader and partition table: the caller's table offset
* when given, else the default `0x8000`, else a scan of 0x1000-aligned
* flash sectors for an MD5-verified table (a derived location). A table
* never lies inside the located bootloader image. The scan runs only when a
* bootloader sits at its chip's ROM offset or the caller selected this
* format. Without a caller table offset, an ESP8266 image at a ROM offset
* refuses the dump, and a known chip's image at another chip's ROM offset
* refuses it as read from the wrong flash offset, given or default.
*/
export const locateEspFlashLayout = async (
fileSize: number,
read: EspFlashRead,
selection: EspFlashSelection,
{
explicit,
signal,
}: { readonly explicit: boolean; readonly signal?: AbortSignal },
): Promise<Result<EspFlashLocation, string>> => {
const origin: EspFlashOrigin =
selection.flashOffset === undefined
? { value: 0, source: "default" }
: { value: selection.flashOffset, source: "caller" };
const probes = await Promise.all(
espBootloaderProbeOffsets(origin.value, fileSize).map(
async ({ flashAddress, fileOffset }) => ({
flashAddress,
fileOffset,
prefix: await read(fileOffset, ESP_BOOTLOADER_PROBE_BYTES),
available: fileSize - fileOffset,
}),
),
);
const bootloader = locateEspBootloader(probes);
// Without a bootloader, nothing in the dump confirms a default origin.
const unconfirmed =
bootloader.location === undefined &&
bootloader.misplacedImage === undefined &&
origin.source === "default"
? [
`No bootloader at a ROM offset confirms that the dump starts at flash ${offsetText(origin.value)}; each partition's file offset assumes it. Pass flash_offset if the dump starts elsewhere.`,
]
: [];
const located = (
tableAddress: number,
tableMethod: EspPartitionTableMethod,
table: EspPartitionTable,
): Result<EspFlashLocation, string> =>
ok({
origin,
bootloader: bootloader.location,
tableAddress,
tableMethod,
table,
limitations: [...bootloader.limitations, ...unconfirmed],
});
const fileOffsetOf = (address: number): number => address - origin.value;
if (selection.partitionTableOffset !== undefined) {
const address = selection.partitionTableOffset;
if (fileOffsetOf(address) < 0 || fileOffsetOf(address) >= fileSize)
return err(
`the partition table offset ${offsetText(address)} is outside the dump, which holds flash ${offsetText(origin.value)}-${offsetText(origin.value + fileSize)}`,
);
const decoded = await tableAt(read, fileOffsetOf(address));
return decoded.ok
? located(address, "caller", decoded.value)
: err(
`no partition table at the given offset ${offsetText(address)}: ${decoded.error}`,
);
}
if (bootloader.location === undefined && bootloader.esp8266)
return err(ESP8266_REFUSAL);
// A table found at an origin the bootloader contradicts would place every
// partition at the wrong file offset.
const misplaced = bootloader.misplacedImage;
if (bootloader.location === undefined && misplaced !== undefined) {
const implied = await espFlashDumpOrigin(read, fileSize, {
...misplaced,
end: await espImageSegmentsEnd(read, misplaced.fileOffset, fileSize),
});
return err(
`${bootloader.limitations.join(" ")} ${
implied === undefined
? "Pass the flash_offset where the dump starts, or partition_table_offset"
: `A partition table at file offset ${offsetText(implied.tableFileOffset)} shows the dump was read from flash ${offsetText(implied.flashOffset)}; pass flash_offset ${offsetText(implied.flashOffset)}`
}`,
);
}
const imageEnd =
bootloader.location === undefined
? 0
: await espImageSegmentsEnd(
read,
bootloader.location.fileOffset,
fileSize,
);
const defaultOffset = fileOffsetOf(ESP_PARTITION_TABLE_DEFAULT_OFFSET);
if (defaultOffset >= imageEnd && defaultOffset < fileSize) {
const decoded = await tableAt(read, defaultOffset);
if (decoded.ok)
return located(
ESP_PARTITION_TABLE_DEFAULT_OFFSET,
"default",
decoded.value,
);
}
if (bootloader.location === undefined && !explicit)
return err(
[
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} and no bootloader at a ROM offset`,
...bootloader.limitations,
].join("; "),
);
const scanned = await scanForTable(
read,
{ fileSize, origin: origin.value, start: imageEnd },
signal,
);
return scanned === undefined
? err(
`no partition table at ${offsetText(ESP_PARTITION_TABLE_DEFAULT_OFFSET)} or in any 0x1000-aligned flash sector${bootloader.location === undefined ? "" : " after the bootloader image"} with a verified MD5; flash encryption is one cause`,
)
: located(scanned.address, "scan", scanned.table);
};
src/domain/firmware/espFlashPlan.test.ts
+460
View File
@@ -0,0 +1,460 @@
import { describe, expect, it } from "vitest";
import { espChipForId } from "./espChips.js";
import {
espFlashPartitionFacts,
finishEspFlashLayout,
planEspFlashLayout,
type EspFlashPlanInput,
} from "./espFlashLayout.js";
import {
decodeEspPartitionTable,
type EspPartitionTable,
} from "./espPartitionTable.js";
import {
TWO_OTA_PARTITIONS,
otaRecord,
partitionTable,
type EspPartitionFixture,
} from "./espFlash.fixture.js";
const table = (parts: readonly EspPartitionFixture[]): EspPartitionTable => {
const decoded = decodeEspPartitionTable(partitionTable(parts));
if (!decoded.ok) throw new Error(decoded.error);
return decoded.value;
};
const c3 = espChipForId(0x0005)!;
const input = (fields: Partial<EspFlashPlanInput> = {}): EspFlashPlanInput => ({
fileSize: 0x4_0000,
origin: { value: 0, source: "default" },
bootloader: { flashAddress: 0, fileOffset: 0, chip: c3 },
tableAddress: 0x8000,
tableMethod: "default",
table: table(TWO_OTA_PARTITIONS),
...fields,
});
const erasedGaps = (count: number) =>
Array.from({ length: count }, () => undefined);
describe("ESP flash layout plans", () => {
it("describes the bootloader region, the table sector and every partition", () => {
const plan = planEspFlashLayout(input());
expect(
plan.regions.map(({ path, flashAddress, length, available }) => [
path,
flashAddress,
length,
available,
]),
).toEqual([
["bootloader.bin", 0, 0x8000, 0x8000],
["partition-table.bin", 0x8000, 0x1000, 0x1000],
["partitions/nvs.bin", 0x9000, 0x4000, 0x4000],
["partitions/otadata.bin", 0xd000, 0x2000, 0x2000],
["partitions/phy_init.bin", 0xf000, 0x1000, 0x1000],
["partitions/factory.bin", 0x1_0000, 0x1_0000, 0x1_0000],
["partitions/ota_0.bin", 0x2_0000, 0x1_0000, 0x1_0000],
["partitions/ota_1.bin", 0x3_0000, 0x1_0000, 0x1_0000],
]);
expect(plan.gaps).toEqual([]);
expect(plan.otadata?.records).toEqual([
{ index: 0, fileOffset: 0xd000, available: true },
{ index: 1, fileOffset: 0xe000, available: true },
]);
expect(plan.limitations).toEqual([]);
});
it("falls back to the index for unsafe or repeated labels", () => {
const parts: EspPartitionFixture[] = [
{ label: "../x", type: 1, subtype: 2, offset: 0x9000, size: 0x1000 },
{ label: "dup", type: 1, subtype: 2, offset: 0xa000, size: 0x1000 },
{ label: "dup", type: 1, subtype: 2, offset: 0xb000, size: 0x1000 },
{ label: "7", type: 1, subtype: 2, offset: 0xc000, size: 0x1000 },
// Case-insensitive file systems would extract these to one file.
{ label: "Data", type: 1, subtype: 2, offset: 0xd000, size: 0x1000 },
{ label: "data", type: 1, subtype: 2, offset: 0xe000, size: 0x1000 },
];
expect(
planEspFlashLayout(input({ table: table(parts) }))
.regions.map(({ path }) => path)
.slice(2),
).toEqual([
"partitions/0.bin",
"partitions/1.bin",
"partitions/2.bin",
"partitions/3.bin",
"partitions/4.bin",
"partitions/5.bin",
]);
});
it("accepts the primary bootloader and table entries ESP-IDF 5.3 declares", () => {
const plan = planEspFlashLayout(
input({
table: table([
{
label: "PrimaryBTLDR",
type: 2,
subtype: 0,
offset: 0,
size: 0x8000,
},
{
label: "PrimaryPrtTable",
type: 3,
subtype: 0,
offset: 0x8000,
size: 0x1000,
},
...TWO_OTA_PARTITIONS,
]),
}),
);
expect(plan.limitations).toEqual([]);
expect(plan.regions.map(({ path }) => path).slice(0, 4)).toEqual([
"bootloader.bin",
"partition-table.bin",
"partitions/PrimaryBTLDR.bin",
"partitions/PrimaryPrtTable.bin",
]);
});
it("keeps a usable partition beside truncated, missing and early ones", () => {
const plan = planEspFlashLayout(
input({
fileSize: 0x3_8000,
origin: { value: 0x1000, source: "caller" },
bootloader: undefined,
table: table([
{ label: "early", type: 1, subtype: 2, offset: 0x0, size: 0x1000 },
{
label: "app",
type: 0,
subtype: 0,
offset: 0x1_0000,
size: 0x1_0000,
},
{
label: "cut",
type: 0,
subtype: 0x10,
offset: 0x3_0000,
size: 0x1_0000,
},
{
label: "gone",
type: 1,
subtype: 0x82,
offset: 0x10_0000,
size: 0x1000,
},
]),
}),
);
expect(
plan.regions.map(({ path, fileOffset, available }) => [
path,
fileOffset,
available,
]),
).toEqual([
["partition-table.bin", 0x7000, 0x1000],
["partitions/app.bin", 0xf000, 0x1_0000],
["partitions/cut.bin", 0x2_f000, 0x9000],
// Past the end of the file: an entry holding none of its bytes.
["partitions/gone.bin", 0xf_f000, 0],
]);
expect(plan.limitations).toEqual([
"Partition early at 0x0 starts before the end of the partition table at 0x9000; gen_esp32part.py rejects such a table.",
"Partition early at 0x0 begins before the dump's first byte at flash 0x1000; it is not read.",
"Partition cut at 0x30000-0x40000 is truncated: the dump holds 36864 of its 65536 bytes.",
"Partition gone at 0x100000-0x101000 lies outside the dump, which ends at flash 0x39000.",
]);
});
it("reports every region inside a longer one", () => {
const plan = planEspFlashLayout(
input({
table: table([
{ label: "a", type: 1, subtype: 2, offset: 0x9000, size: 0x8000 },
{ label: "b", type: 1, subtype: 2, offset: 0xa000, size: 0x1000 },
{ label: "c", type: 1, subtype: 2, offset: 0xc000, size: 0x1000 },
]),
}),
);
expect(plan.limitations).toEqual([
"partitions/b.bin at 0xa000 overlaps partitions/a.bin, which ends at 0x11000.",
"partitions/c.bin at 0xc000 overlaps partitions/a.bin, which ends at 0x11000.",
]);
});
});
describe("ESP flash layout partition paths", () => {
it("falls back to the index for Windows device names and keeps the label", () => {
const labels = [
"NUL",
"con",
"Com1",
"lpt9",
"aux.cfg",
"prn.",
// Names that only begin like a device name stay names.
"console",
"com10",
"nul_data",
];
const parts = labels.map((label, index): EspPartitionFixture => ({
label,
type: 1,
subtype: 2,
offset: 0x9000 + index * 0x1000,
size: 0x1000,
}));
const regions = planEspFlashLayout(
input({ table: table(parts) }),
).regions.slice(2);
expect(regions.map(({ path }) => path)).toEqual([
"partitions/0.bin",
"partitions/1.bin",
"partitions/2.bin",
"partitions/3.bin",
"partitions/4.bin",
"partitions/5.bin",
"partitions/console.bin",
"partitions/com10.bin",
"partitions/nul_data.bin",
]);
expect(regions.map(({ partition }) => partition?.label)).toEqual(labels);
});
});
describe("ESP flash layout coverage", () => {
it("keeps coverage complete for partitions the end of the file cuts short or leaves out", () => {
const cut = planEspFlashLayout(input({ fileSize: 0x2_8000 }));
const reads = (plan: typeof cut) => ({
otadataRecords: [],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
// ota_0 is cut short and ota_1 lies past the end; both are entries.
expect(
cut.regions.slice(-2).map(({ path, available }) => [path, available]),
).toEqual([
["partitions/ota_0.bin", 0x8000],
["partitions/ota_1.bin", 0],
]);
expect(finishEspFlashLayout(cut, reads(cut)).coverage).toBe("complete");
// A partition before the dump's first byte has no entry, so coverage is
// unknown even with the bootloader located.
const later = planEspFlashLayout(
input({
fileSize: 0x3_f000,
origin: { value: 0x1000, source: "caller" },
bootloader: {
flashAddress: 0x1000,
fileOffset: 0,
chip: espChipForId(0x0000)!,
},
table: table([
{ label: "early", type: 1, subtype: 2, offset: 0x0, size: 0x1000 },
...TWO_OTA_PARTITIONS,
]),
}),
);
expect(finishEspFlashLayout(later, reads(later)).coverage).toBe("unknown");
});
});
describe("ESP flash layout otadata", () => {
const [, otadata] = TWO_OTA_PARTITIONS;
it("reads the otadata partition the bootloader uses: the last declared", () => {
const plan = planEspFlashLayout(
input({
table: table([
...TWO_OTA_PARTITIONS,
{ ...otadata!, label: "otadata2", offset: 0x3_0000 + 0x1_0000 },
]),
fileSize: 0x5_0000,
}),
);
expect(plan.otadata?.partition.label).toBe("otadata2");
expect(plan.otadata?.records.map(({ fileOffset }) => fileOffset)).toEqual([
0x4_0000, 0x4_1000,
]);
expect(plan.limitations).toContain(
"The table declares otadata 2 times (partitions #1, #6); ESP-IDF's bootloader uses the last.",
);
});
it("does not read a record past a short otadata partition and selects none", () => {
const short = TWO_OTA_PARTITIONS.map((part) =>
part.label === "otadata" ? { ...part, size: 0x1000 } : part,
);
const plan = planEspFlashLayout(input({ table: table(short) }));
expect(plan.otadata?.records).toEqual([
{ index: 0, fileOffset: 0xd000, available: true },
]);
expect(plan.limitations).toContain(
"The otadata partition otadata is 0x1000 bytes, smaller than the two 4 KiB sectors ESP-IDF's bootloader reads; the bootloader resets without selecting an app.",
);
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 1 })],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.otadata).toMatchObject({
records: [{ index: 0, ota_seq: 1 }],
});
expect(facts.boot_selection).toMatchObject({
basis: "none",
partition_index: null,
});
});
it("selects nothing from a table whose MD5 the bootloader rejects", () => {
const decoded = decodeEspPartitionTable(
partitionTable(TWO_OTA_PARTITIONS, { md5: "corrupt" }),
);
if (!decoded.ok) throw new Error(decoded.error);
const plan = planEspFlashLayout(input({ table: decoded.value }));
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 1 }), Buffer.alloc(32, 0xff)],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.boot_selection).toMatchObject({
basis: "none",
partition_index: null,
assumptions: [],
});
expect(facts.limitations).toContain(
"The partition table MD5 at 0xc0 does not match the entries before it; ESP-IDF's bootloader rejects the table.",
);
});
it("keeps the record a dump holds when it ends between the two", () => {
const plan = planEspFlashLayout(input({ fileSize: 0xe000 }));
expect(plan.otadata?.records).toEqual([
{ index: 0, fileOffset: 0xd000, available: true },
{ index: 1, fileOffset: 0xe000, available: false },
]);
expect(plan.limitations).toContain(
"otadata record 1 of partition otadata at flash 0xe000 is outside the dump; the OTA selection is unknown.",
);
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 3 }), undefined],
gapFirstUnerased: erasedGaps(plan.gaps.length),
});
expect(facts.otadata).toEqual({
partition_index: 1,
records: [expect.objectContaining({ index: 0, ota_seq: 3, valid: true })],
});
expect(facts.boot_selection).toMatchObject({
basis: "unknown",
running_partition: "unknown",
});
});
});
describe("ESP flash layout facts", () => {
it("describes erased gaps and reports the rest as undescribed", () => {
const plan = planEspFlashLayout(
input({
table: table(TWO_OTA_PARTITIONS.slice(0, 3)),
fileSize: 0x2_0000,
}),
);
expect(plan.gaps).toEqual([{ offset: 0x1_0000, length: 0x1_0000 }]);
const erased = finishEspFlashLayout(plan, {
otadataRecords: [],
gapFirstUnerased: [undefined],
});
expect(erased).toMatchObject({ coverage: "complete", undescribed: [] });
const written = finishEspFlashLayout(plan, {
otadataRecords: [],
gapFirstUnerased: [0x1_8000],
});
expect(written.undescribed).toEqual([{ offset: 0x1_8000, length: 0x8000 }]);
expect(written.coverage).toBe("unknown");
expect(written.limitations.at(-1)).toBe(
"1 range(s) of the dump, 32768 bytes in all, lie outside the bootloader region, the partition table and every partition and are not described; each runs from a byte that is not erased flash to the end of its gap.",
);
});
it("derives the selected app from the otadata records and leaves the running one unknown", () => {
const plan = planEspFlashLayout(input());
const facts = finishEspFlashLayout(plan, {
otadataRecords: [otaRecord({ seq: 2 }), Buffer.alloc(32, 0xff)],
gapFirstUnerased: [],
});
expect(facts.otadata).toMatchObject({
partition_index: 1,
records: [{ ota_seq: 2, valid: true }, { valid: false }],
});
expect(facts.boot_selection).toMatchObject({
basis: "otadata",
active_record: 0,
slot: 1,
partition_index: 5,
running_partition: "unknown",
});
expect(facts.coverage).toBe("complete");
// A factory-only table selects its factory app without otadata.
const withoutOtadata = planEspFlashLayout(
input({
table: table(
TWO_OTA_PARTITIONS.filter(({ label }) =>
["nvs", "phy_init", "factory"].includes(label),
),
),
}),
);
expect(
finishEspFlashLayout(withoutOtadata, {
otadataRecords: [],
gapFirstUnerased: erasedGaps(withoutOtadata.gaps.length),
}),
).toMatchObject({
otadata: null,
boot_selection: { basis: "no-otadata", partition_index: 2 },
});
});
it("projects a partition's declaration with ESP-IDF names and where it is written", () => {
const [nvs] = table(TWO_OTA_PARTITIONS).partitions;
// A partition the dump holds none of has no observed bytes.
expect(espFlashPartitionFacts(nvs!)).toEqual({
kind: "esp-flash-partition",
index: 0,
label: "nvs",
label_hex: null,
type: 1,
subtype: 2,
type_name: "data",
subtype_name: "nvs",
flash_address: 0x9000,
size: 0x4000,
encrypted: false,
readonly: false,
unknown_flags: 0,
written_offset: null,
written_end: null,
written_bytes: null,
});
expect(
espFlashPartitionFacts(nvs!, { offset: 0x40, end: 0x1000, bytes: 9 }),
).toMatchObject({
written_offset: 0x40,
written_end: 0x1000,
written_bytes: 9,
});
expect(
espFlashPartitionFacts(nvs!, { offset: null, end: null, bytes: 0 }),
).toMatchObject({
written_offset: null,
written_end: null,
written_bytes: 0,
});
});
});
src/domain/firmware/espFlashSelection.ts
+32
View File
@@ -0,0 +1,32 @@
import { z } from "zod";
import type { ExecutableFormatHint } from "../dosCom.js";
import { ESP_PARTITION_TABLE_SECTOR_BYTES } from "./espPartitionTable.js";
/** A 32-bit flash address written as `0x`-prefixed hexadecimal. */
export const espFlashAddressSchema = z.string().regex(/^0x[0-9a-fA-F]{1,8}$/u, {
message: "must be a hexadecimal flash address of the form 0x…",
});
export const parseEspFlashAddress = (text: string): number =>
Number.parseInt(text.slice(2), 16);
/** Describe the format and flash-input pairing without performing I/O. */
export const espFlashSelectionError = (input: {
readonly format?: ExecutableFormatHint | undefined;
readonly flashOffset?: number | undefined;
readonly partitionTableOffset?: number | undefined;
}): string | undefined => {
if (
(input.flashOffset !== undefined ||
input.partitionTableOffset !== undefined) &&
input.format !== "esp-flash-image"
)
return "flash_offset and partition_table_offset apply only with format=esp-flash-image";
if (
input.partitionTableOffset !== undefined &&
input.partitionTableOffset % ESP_PARTITION_TABLE_SECTOR_BYTES !== 0
)
return "partition_table_offset must be a multiple of 0x1000, as ESP-IDF requires";
return undefined;
};
src/domain/firmware/espImage.test.ts
+1
View File
@@ -160,6 +160,7 @@ describe("ESP image decoding", () => {
mmu_page_size: 16,
spi_flash_mode: 2,
},
flash_dump: null,
});
expect(
decoded.segments.map(({ dataOffset, availableLength, metadata }) => [
src/domain/firmware/espImageFacts.ts
+2
View File
@@ -214,6 +214,8 @@ export const deriveEspImage = (observed: EspImageObservations): EspImage => {
trailing.blocks.length > 0 ? (observed.sectorOffset ?? null) : null,
undescribed_trailing: trailing.undescribed,
descriptor,
// The image's bytes alone cannot place it in a dump; the reader can.
flash_dump: null,
},
segments,
signatureBlocks: trailing.blocks,
src/domain/firmware/espImageStructures.ts
+8 -3
View File
@@ -56,11 +56,16 @@ const allZero = (bytes: Uint8Array): boolean =>
const UTF8 = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
/** A NUL-terminated C string in a fixed array, or null when not UTF-8. */
const cString = (bytes: Uint8Array): string | null => {
/** The bytes of a NUL-terminated C string in a fixed array. */
export const cStringBytes = (bytes: Uint8Array): Uint8Array => {
const end = bytes.indexOf(0);
return end < 0 ? bytes : bytes.subarray(0, end);
};
/** A NUL-terminated C string in a fixed array, or null when not UTF-8. */
export const cString = (bytes: Uint8Array): string | null => {
try {
return UTF8.decode(end < 0 ? bytes : bytes.subarray(0, end));
return UTF8.decode(cStringBytes(bytes));
} catch {
return null;
}
src/domain/firmware/espOtaData.ts
+196
View File
@@ -0,0 +1,196 @@
import { crc32 } from "node:zlib";
import { hex, u32 } from "./espImageStructures.js";
import type { EspBootPartitions } from "./espPartitionTable.js";
/*
* OTA data, from ESP-IDF v6.1 `esp_flash_partitions.h`
* (`esp_ota_select_entry_t`, `esp_ota_img_states_t`),
* `bootloader_common_loader.c` and `bootloader_utility.c`
* (`bootloader_utility_get_selected_boot_partition`), and `otatool.py`.
*/
/** The otadata partition holds one record at the start of each of two sectors. */
export const ESP_OTA_DATA_SECTOR_BYTES = 0x1000;
/** `read_otadata` refuses a smaller otadata partition. */
export const ESP_OTA_DATA_MIN_BYTES = 2 * ESP_OTA_DATA_SECTOR_BYTES;
export const ESP_OTA_SELECT_ENTRY_BYTES = 32;
const UNSET = 0xffff_ffff;
export type EspOtaStateName =
| "new"
| "pending-verify"
| "valid"
| "invalid"
| "aborted"
| "undefined";
const STATE_NAMES: Readonly<Record<number, EspOtaStateName>> = {
0x0: "new",
0x1: "pending-verify",
0x2: "valid",
0x3: "invalid",
0x4: "aborted",
[UNSET]: "undefined",
};
/** One `esp_ota_select_entry_t`, observed as written. */
export interface EspOtaRecord {
readonly index: 0 | 1;
readonly ota_seq: number;
readonly seq_label: string;
readonly ota_state: number;
readonly state_name: EspOtaStateName | null;
readonly crc: number;
/** CRC-32 of `ota_seq`, seeded with `0xFFFFFFFF`, as the bootloader computes it. */
readonly crc_matches: boolean;
/** `bootloader_common_ota_select_valid`: set sequence, usable state, matching CRC. */
readonly valid: boolean;
}
export const decodeEspOtaRecord = (
bytes: Uint8Array,
index: 0 | 1,
): EspOtaRecord => {
const sequence = bytes.subarray(0, 4);
const otaSeq = u32(bytes, 0);
const state = u32(bytes, 24);
const crc = u32(bytes, 28);
const crcMatches = crc32(sequence, UNSET) >>> 0 === crc;
return {
index,
ota_seq: otaSeq,
seq_label: hex(bytes.subarray(4, 24)),
ota_state: state,
state_name: STATE_NAMES[state] ?? null,
crc,
crc_matches: crcMatches,
valid: otaSeq !== UNSET && state !== 0x3 && state !== 0x4 && crcMatches,
};
};
const RECORDS_ASSUMPTION =
"Records are used as written: the bootloader may first mark a pending-verify record aborted when app rollback is enabled.";
/** What a derived selection leaves out, whatever selected it. */
const BOOT_ASSUMPTIONS = [
"Anti-rollback, which compares the app's secure_version with eFuse, is not applied.",
"The selected app is not checked for a loadable image; the bootloader falls back to other partitions when it fails to load or is not declared.",
"Factory-reset and test-app GPIO holds and a wake from deep sleep, which can override the selection, are not applied.",
] as const;
/**
* The branch of `bootloader_utility_get_selected_boot_partition` that
* selects the app: `no-otadata` for a table without an otadata partition,
* `none` for its INVALID_INDEX, after which the bootloader resets, and
* `unknown` when the dump lacks a record the bootloader reads.
*/
export type EspBootSelectionBasis =
| "otadata"
| "factory-fallback"
| "ota_0-fallback"
| "no-otadata"
| "none"
| "unknown";
export interface EspBootSelection {
readonly basis: EspBootSelectionBasis;
/** The record the bootloader reads as active: the higher valid `ota_seq`. */
readonly active_record: 0 | 1 | null;
/** `(ota_seq - 1) % ota_app_count`, the bootloader's slot arithmetic. */
readonly slot: number | null;
/** The table index of the selected app; null when none is declared. */
readonly partition_index: number | null;
/** What the derivation leaves out; empty when nothing is selected. */
readonly assumptions: readonly string[];
/** Rollback, anti-rollback and load fallback decide it at boot. */
readonly running_partition: "unknown";
}
/** `bootloader_common_ota_select_invalid`, which does not check the CRC. */
const unusable = ({ ota_seq, ota_state }: EspOtaRecord): boolean =>
ota_seq === UNSET || ota_state === 0x3 || ota_state === 0x4;
/**
* Derive the app partition the bootloader would select, as
* `select_partition_number` and `bootloader_utility_get_selected_boot_partition`
* do without app rollback or anti-rollback:
* - a table `esp_partition_table_verify` rejects (an MD5 mismatch or a
* second MD5 entry): none, because loading it fails and the bootloader
* resets;
* - no otadata partition, or one at offset 0: the factory index
* (`no-otadata`), whether or not a factory app is declared;
* - an otadata partition smaller than two sectors: none, because
* `read_otadata` fails;
* - no OTA app, or both records unusable: the factory app when one is
* declared, else boot index 0 (`ota_0`); without an OTA app the records
* do not matter;
* - otherwise the valid record with the higher `ota_seq` (record 0 on a
* tie), whose slot is `(ota_seq - 1) % ota_app_count`;
* - no valid record, as after a CRC mismatch: the factory index, whether
* or not a factory app is declared.
* `unknown` when the dump lacks a record the bootloader would read.
*/
export const selectEspBootPartition = (
boot: EspBootPartitions,
otadata:
| {
readonly offset: number;
readonly size: number;
readonly records: readonly EspOtaRecord[];
}
| undefined,
tableRejected = false,
): EspBootSelection => {
const selection = (
basis: EspBootSelectionBasis,
fields: Partial<
Pick<EspBootSelection, "active_record" | "slot" | "partition_index">
> = {},
): EspBootSelection => ({
basis,
active_record: null,
slot: null,
partition_index: null,
...fields,
assumptions:
basis === "none" || basis === "unknown"
? []
: basis === "no-otadata"
? [...BOOT_ASSUMPTIONS]
: [RECORDS_ASSUMPTION, ...BOOT_ASSUMPTIONS],
running_partition: "unknown",
});
const factoryIndex = { partition_index: boot.factory?.index ?? null };
if (tableRejected) return selection("none");
// The bootloader takes an otadata partition at offset 0 for none.
if (otadata === undefined || otadata.offset === 0)
return selection("no-otadata", factoryIndex);
const { records, size } = otadata;
if (size < ESP_OTA_DATA_MIN_BYTES) return selection("none");
const factory = selection("factory-fallback", factoryIndex);
const fallback =
boot.factory !== undefined
? factory
: selection("ota_0-fallback", {
slot: 0,
partition_index: boot.otaSlots.get(0)?.index ?? null,
});
if (boot.otaAppCount === 0) return fallback;
if (records.length < 2) return selection("unknown");
if (records.every(unusable)) return fallback;
const valid = records.filter(({ valid }) => valid);
const [first, second] = valid;
const active =
first !== undefined &&
second !== undefined &&
second.ota_seq > first.ota_seq
? second
: first;
if (active === undefined) return factory;
const slot = ((active.ota_seq - 1) >>> 0) % boot.otaAppCount;
return selection("otadata", {
active_record: active.index,
slot,
// A gap in the OTA subtypes selects a slot with no partition.
partition_index: boot.otaSlots.get(slot)?.index ?? null,
});
};
src/domain/firmware/espPartitionTable.ts
+307
View File
@@ -0,0 +1,307 @@
import { createHash } from "node:crypto";
import { err, ok, type Result } from "../result.js";
import {
cString,
cStringBytes,
hex,
offsetText,
u16,
u32,
} from "./espImageStructures.js";
/*
* The partition table, from ESP-IDF v6.1 `esp_flash_partitions.h`,
* `bootloader_support/src/flash_partitions.c` (`esp_partition_table_verify`,
* which the bootloader applies) and `partition_table/gen_esp32part.py`.
*/
export const ESP_PARTITION_TABLE_DEFAULT_OFFSET = 0x8000;
/** The table occupies one flash sector (`ESP_PARTITION_TABLE_SIZE`). */
export const ESP_PARTITION_TABLE_SECTOR_BYTES = 0x1000;
/** `ESP_PARTITION_TABLE_MAX_LEN`: 96 entries, the MD5 and the terminator included. */
export const ESP_PARTITION_TABLE_MAX_BYTES = 0xc00;
const ENTRY_BYTES = 32;
const MAX_ENTRIES = ESP_PARTITION_TABLE_MAX_BYTES / ENTRY_BYTES;
const PARTITION_MAGIC = 0x50aa;
const MD5_MAGIC = 0xebeb;
const END_MAGIC = 0xffff;
const END_TYPE = 0xff;
/** `ESP_PARTITION_MD5_OFFSET`: the digest follows 16 bytes into its entry. */
const MD5_DIGEST_OFFSET = 16;
const KNOWN_FLAGS = 0b11;
/** One `esp_partition_info_t`, as the table declares it. */
export interface EspPartition {
/** Position among the partition entries; the MD5 entry is not counted. */
readonly index: number;
/** Byte offset of the entry from the start of the table. */
readonly entryOffset: number;
readonly type: number;
readonly subtype: number;
/** Flash address. */
readonly offset: number;
readonly size: number;
/** The label up to its first NUL, or null when it is not UTF-8. */
readonly label: string | null;
/** The label's bytes up to its first NUL, in hex, when it is not UTF-8. */
readonly labelHex: string | null;
readonly encrypted: boolean;
readonly readonly: boolean;
/** Flag bits other than `encrypted` and `readonly`, which ESP-IDF ignores. */
readonly unknownFlags: number;
}
export interface EspPartitionTableMd5 {
readonly status: "valid" | "invalid" | "absent";
readonly entryOffset: number | null;
readonly stored: string | null;
readonly computed: string | null;
}
export interface EspPartitionTable {
readonly partitions: readonly EspPartition[];
readonly md5: EspPartitionTableMd5;
/** Bytes through the terminating entry. */
readonly length: number;
readonly limitations: readonly string[];
}
const partition = (
entry: Uint8Array,
index: number,
entryOffset: number,
): EspPartition => {
const flags = u32(entry, 28);
const labelBytes = entry.subarray(12, 28);
const label = cString(labelBytes);
return {
index,
entryOffset,
type: entry[2] ?? 0,
subtype: entry[3] ?? 0,
offset: u32(entry, 4),
size: u32(entry, 8),
label,
labelHex: label === null ? hex(cStringBytes(labelBytes)) : null,
encrypted: (flags & 1) !== 0,
readonly: (flags & 2) !== 0,
unknownFlags: (flags & ~KNOWN_FLAGS) >>> 0,
};
};
/**
* Decode a table in the entry structure ESP-IDF's bootloader verifies:
* 32-byte entries with the `0x50AA` magic, at most one `0xEBEB` MD5 entry
* over the entries before it, and a terminator whose magic, type and subtype
* are all `0xFF`. A structural fault means the bytes are not a table. An MD5
* mismatch or a second MD5 entry, which the bootloader rejects, is reported
* as an integrity fact so that the declared entries stay visible. ESP-IDF
* accepts a table with no partition entries, but REA requires one before
* the terminator, so that neither erased flash nor a lone MD5 entry is read
* as a table.
*/
export const decodeEspPartitionTable = (
bytes: Uint8Array,
): Result<EspPartitionTable, string> => {
const partitions: EspPartition[] = [];
const limitations: string[] = [];
let md5: EspPartitionTableMd5 = {
status: "absent",
entryOffset: null,
stored: null,
computed: null,
};
for (let slot = 0; slot < MAX_ENTRIES; slot += 1) {
const at = slot * ENTRY_BYTES;
const entry = bytes.subarray(at, at + ENTRY_BYTES);
if (entry.length < ENTRY_BYTES)
return err(
`the bytes end at ${offsetText(at + entry.length)}, inside entry ${String(slot)}`,
);
const magic = u16(entry, 0);
if (magic === PARTITION_MAGIC) {
partitions.push(partition(entry, partitions.length, at));
continue;
}
if (magic === MD5_MAGIC) {
const computed = createHash("md5")
.update(bytes.subarray(0, at))
.digest("hex");
const stored = hex(entry.subarray(MD5_DIGEST_OFFSET, ENTRY_BYTES));
if (md5.status !== "absent") {
limitations.push(
`The partition table has a second MD5 entry at ${offsetText(at)}; ESP-IDF allows one and rejects the table.`,
);
md5 = { ...md5, status: "invalid" };
continue;
}
md5 = {
status: stored === computed ? "valid" : "invalid",
entryOffset: at,
stored,
computed,
};
if (stored !== computed)
limitations.push(
`The partition table MD5 at ${offsetText(at)} does not match the entries before it; ESP-IDF's bootloader rejects the table.`,
);
continue;
}
const terminator =
magic === END_MAGIC && entry[2] === END_TYPE && entry[3] === END_TYPE;
// An MD5 entry is not a partition: a table declares at least one.
if (terminator && partitions.length > 0)
return ok({ partitions, md5, length: at + ENTRY_BYTES, limitations });
return err(
slot === 0 && magic === END_MAGIC
? "the first entry is empty"
: terminator
? `entry ${String(slot)} at ${offsetText(at)} ends the table before any partition entry`
: `entry ${String(slot)} at ${offsetText(at)} has magic ${offsetText(magic)}, not a partition, MD5 or terminating entry`,
);
}
return err(
`no terminating entry within ${String(MAX_ENTRIES)} entries (${offsetText(ESP_PARTITION_TABLE_MAX_BYTES)} bytes)`,
);
};
/** ESP-IDF's names for partition types (`gen_esp32part.py` `TYPES`). */
const TYPE_NAMES: Readonly<Record<number, string>> = {
0x00: "app",
0x01: "data",
0x02: "bootloader",
0x03: "partition_table",
};
/** ESP-IDF's subtype names per type (`gen_esp32part.py` `SUBTYPES`). */
const SUBTYPE_NAMES: Readonly<
Record<number, Readonly<Record<number, string>>>
> = {
0x00: {
0x00: "factory",
0x20: "test",
...Object.fromEntries(
Array.from({ length: 16 }, (_, slot) => [
0x10 + slot,
`ota_${String(slot)}`,
]),
),
0x30: "tee_0",
0x31: "tee_1",
},
0x01: {
0x00: "ota",
0x01: "phy",
0x02: "nvs",
0x03: "coredump",
0x04: "nvs_keys",
0x05: "efuse",
0x06: "undefined",
0x80: "esphttpd",
0x81: "fat",
0x82: "spiffs",
0x83: "littlefs",
0x90: "tee_ota",
},
0x02: { 0x00: "primary", 0x01: "ota", 0x02: "recovery" },
0x03: { 0x00: "primary", 0x01: "ota" },
};
export const espPartitionTypeName = (type: number): string | null =>
TYPE_NAMES[type] ?? null;
export const espPartitionSubtypeName = (
type: number,
subtype: number,
): string | null => SUBTYPE_NAMES[type]?.[subtype] ?? null;
const APP_TYPE = 0x00;
const OTA_DATA_SUBTYPE = 0x00;
const DATA_TYPE = 0x01;
const BOOTLOADER_TYPE = 0x02;
const PARTITION_TABLE_TYPE = 0x03;
const PRIMARY_SUBTYPE = 0x00;
const OTA_APP_SUBTYPE_FLAG = 0x10;
const OTA_APP_SUBTYPE_MASK = 0x0f;
/** The OTA slot of an app partition (`ota_0` to `ota_15`), if it has one. */
export const espOtaAppSlot = (part: EspPartition): number | null =>
part.type === APP_TYPE &&
(part.subtype & ~OTA_APP_SUBTYPE_MASK) === OTA_APP_SUBTYPE_FLAG
? part.subtype & OTA_APP_SUBTYPE_MASK
: null;
const isEspFactoryApp = (part: EspPartition): boolean =>
part.type === APP_TYPE && part.subtype === 0x00;
const isEspOtaData = (part: EspPartition): boolean =>
part.type === DATA_TYPE && part.subtype === OTA_DATA_SUBTYPE;
export const isEspAppPartition = (part: EspPartition): boolean =>
part.type === APP_TYPE;
/**
* The region a primary `bootloader` or `partition_table` entry declares.
* ESP-IDF 5.3 and later may declare them over the bootloader region and the
* table sector, and `gen_esp32part.py` exempts them from its rule that
* partitions follow the table.
*/
export const espPrimaryLayoutRole = (
part: EspPartition,
): "bootloader" | "partition-table" | undefined => {
if (part.subtype !== PRIMARY_SUBTYPE) return undefined;
if (part.type === BOOTLOADER_TYPE) return "bootloader";
return part.type === PARTITION_TABLE_TYPE ? "partition-table" : undefined;
};
/** The partitions the bootloader keeps from a table, and the OTA app count. */
export interface EspBootPartitions {
readonly factory: EspPartition | undefined;
readonly otadata: EspPartition | undefined;
/** Each OTA slot's partition, indexed by subtype. */
readonly otaSlots: ReadonlyMap<number, EspPartition>;
/** Every OTA app entry, duplicates included, as `bs->app_count` counts. */
readonly otaAppCount: number;
readonly limitations: readonly string[];
}
/**
* Read the table as `bootloader_utility_load_partition_table` does: a later
* factory, otadata or same-subtype OTA entry replaces an earlier one, and
* every OTA app entry adds to the app count.
*/
export const espBootPartitions = (
partitions: readonly EspPartition[],
): EspBootPartitions => {
let factory: EspPartition | undefined;
let otadata: EspPartition | undefined;
let otaAppCount = 0;
const otaSlots = new Map<number, EspPartition>();
const roles = new Map<string, EspPartition[]>();
for (const part of partitions) {
const slot = espOtaAppSlot(part);
const role = isEspFactoryApp(part)
? "factory"
: isEspOtaData(part)
? "otadata"
: slot === null
? undefined
: `ota_${String(slot)}`;
if (role === undefined) continue;
roles.set(role, [...(roles.get(role) ?? []), part]);
if (role === "factory") factory = part;
else if (role === "otadata") otadata = part;
else if (slot !== null) {
otaSlots.set(slot, part);
otaAppCount += 1;
}
}
const limitations = [...roles].flatMap(([role, parts]) =>
parts.length < 2
? []
: [
`The table declares ${role} ${String(parts.length)} times (partitions ${parts.map(({ index }) => `#${String(index)}`).join(", ")}); ESP-IDF's bootloader uses the last.`,
],
);
return { factory, otadata, otaSlots, otaAppCount, limitations };
};
src/domain/liveBinaryTargetSummary.ts
+24
View File
@@ -82,6 +82,30 @@ export const liveBinaryTargetSummaryVariants = [
]),
architecture: z.null(),
}),
z.object({
...identity,
kind: z.literal("archive"),
format: z.literal("esp-flash-image"),
architecture: z.null(),
flash_offset: z
.number()
.int()
.min(0)
.max(0xffff_ffff)
.nullable()
.describe(
"Caller-selected flash address of the dump's first byte; null reads it as flash address 0.",
),
partition_table_offset: z
.number()
.int()
.min(0)
.max(0xffff_ffff)
.nullable()
.describe(
"Caller-selected partition table address; null tries 0x8000, then a scan.",
),
}),
z.object({
...identity,
kind: z.literal("artifact"),
src/server/registerSessionTools.ts
+20
View File
@@ -1,3 +1,4 @@
import { parseEspFlashAddress } from "../domain/firmware/espFlashSelection.js";
import { registerNativeFunctionMatchingTool } from "./registerNativeFunctionMatchingTool.js";
import type { EvidenceMcpServer } from "./EvidenceMcpServer.js";
import type { ArtifactResourceScope } from "../artifacts/ArtifactResourceScope.js";
@@ -192,6 +193,25 @@ const registerOpenLifecycleTool = ({
...(input.entry_point === undefined
? {}
: { entryPoint: input.entry_point }),
...(input.flash_offset === undefined &&
input.partition_table_offset === undefined
? {}
: {
espFlash: {
...(input.flash_offset === undefined
? {}
: {
flashOffset: parseEspFlashAddress(input.flash_offset),
}),
...(input.partition_table_offset === undefined
? {}
: {
partitionTableOffset: parseEspFlashAddress(
input.partition_table_offset,
),
}),
},
}),
...(input.provider_id === undefined
? {}
: { providerId: input.provider_id }),
tests/acceptance/artifactComparisonEspFlash.test.ts
+208
View File
@@ -0,0 +1,208 @@
import { writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
} from "../fixtures/espImage.js";
import {
TWO_OTA_PARTITIONS,
espFlashDump,
flashImage,
partitionTable,
} from "../../src/domain/firmware/espFlash.fixture.js";
import { artifactComparisonResultSchema } from "../../src/domain/artifactComparison.js";
import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js";
import { artifactInventoryResultSchema } from "../../src/domain/artifactGraph.js";
import { parseEvidence, type Evidence } from "../../src/domain/evidence.js";
import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
const BOOTLOADER = {
at: 0,
bytes: espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS }),
};
/** A table without an MD5 whose first entry has these label bytes. */
const labelBytes = (bytes: readonly number[]): Buffer => {
const table = Buffer.from(
partitionTable(TWO_OTA_PARTITIONS, { md5: "absent" }),
);
table.fill(0, 12, 28);
Buffer.from(bytes).copy(table, 12);
return table;
};
/** Inspect files and compare their inventories through one MCP session. */
const comparisonSession = async () => {
const root = await createTestTempDirectory("rea-esp-flash-comparison-");
const { call } = await connectLocalToolsMcp();
const inspect = async (name: string, bytes: Buffer): Promise<Evidence> => {
const path = join(root, name);
await writeFile(path, bytes);
const opened = await call("open_binary", { path });
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
const response = await call("inspect_artifact", {});
expect(response.isError, JSON.stringify(response)).not.toBe(true);
const inventory = artifactInspectionResultSchema
.parse(parseEvidence(response.structuredContent).normalized_result)
.substeps.find(
({ operation }) => operation === "inventory_artifact",
)?.evidence;
if (inventory === undefined) throw new Error("Missing inventory Evidence");
return parseEvidence(inventory);
};
// Inline flash-dump inventories exceed the default MCP input element
// allowance, so compare the Evidence the session retained.
const compare = async (left: Evidence, right: Evidence) => {
const response = await call("compare_artifacts", {
left: { kind: "retained-evidence", evidence_id: left.evidence_id },
right: { kind: "retained-evidence", evidence_id: right.evidence_id },
});
expect(response.isError, JSON.stringify(response)).not.toBe(true);
return artifactComparisonResultSchema.parse(
parseEvidence(response.structuredContent).normalized_result,
);
};
return { inspect, compare };
};
const occurrence = (evidence: Evidence, path: string) => {
const found = artifactInventoryResultSchema
.parse(evidence.normalized_result)
.occurrences.find(({ logical_path }) => logical_path === path);
if (found === undefined) throw new Error(`Missing ${path}`);
return found;
};
const changedPaths = (result: { changes: { logical_path: string }[] }) =>
result.changes
.map(({ logical_path }) => logical_path)
.filter((path) => path !== ".")
.sort();
it("reports an ESP partition whose declaration alone changed", async () => {
const { inspect, compare } = await comparisonSession();
const dump = (table: Buffer) =>
espFlashDump({ bootloader: BOOTLOADER, table });
// The same partition bytes, with nvs declared read-only.
const readonly = TWO_OTA_PARTITIONS.map((part) =>
part.label === "nvs" ? { ...part, flags: 0x2 } : part,
);
const left = await inspect(
"flash.bin",
dump(partitionTable(TWO_OTA_PARTITIONS)),
);
const right = await inspect("readonly.bin", dump(partitionTable(readonly)));
const leftNvs = occurrence(left, "partitions/nvs.bin");
const rightNvs = occurrence(right, "partitions/nvs.bin");
expect(rightNvs.artifact_id).toBe(leftNvs.artifact_id);
expect(leftNvs.format_metadata).toMatchObject({ readonly: false });
expect(rightNvs.format_metadata).toMatchObject({ readonly: true });
const changed = await compare(left, right);
expect(
changed.changes.find(
({ logical_path }) => logical_path === "partitions/nvs.bin",
),
).toEqual({
classification: "changed",
logical_path: "partitions/nvs.bin",
dimensions: ["metadata"],
left_occurrence_id: leftNvs.occurrence_id,
right_occurrence_id: rightNvs.occurrence_id,
left_artifact_id: leftNvs.artifact_id,
right_artifact_id: rightNvs.artifact_id,
evidence_links: [left.evidence_id, right.evidence_id],
});
// Only the table's bytes and the declaration it gives nvs changed.
expect(changedPaths(changed)).toEqual([
"partition-table.bin",
"partitions/nvs.bin",
]);
// Labels that are not UTF-8 compare by their bytes.
const first = await inspect("label-a.bin", dump(labelBytes([0xff, 0x01])));
const second = await inspect("label-b.bin", dump(labelBytes([0xfe, 0x01])));
expect(occurrence(first, "partitions/0.bin").format_metadata).toMatchObject({
label: null,
label_hex: "ff01",
});
expect(changedPaths(await compare(first, second))).toEqual([
"partition-table.bin",
"partitions/0.bin",
]);
// Equal declarations leave every partition unchanged.
const same = await compare(
left,
await inspect("copy.bin", dump(partitionTable(TWO_OTA_PARTITIONS))),
);
expect(same.changes).toEqual([]);
});
it("classifies every partition of two dumps that end inside their last partition", async () => {
const { inspect, compare } = await comparisonSession();
const app = espImage({ segments: ESP32C3_APP_SEGMENTS });
// The same app with its DRAM segment loading 8 KiB higher.
const moved = espImage({
segments: ESP32C3_APP_SEGMENTS.map((segment, index) =>
index === 1 ? { ...segment, address: segment.address + 0x2000 } : segment,
),
});
// Like esptool merge-bin: the file ends after the last part written, here
// inside ota_0, and ota_1 lies past its end.
const mergeBin = (factory: Buffer) =>
flashImage(0x2_0000 + app.length, [
BOOTLOADER,
{ at: 0x8000, bytes: partitionTable(TWO_OTA_PARTITIONS) },
{ at: 0x1_0000, bytes: factory },
{ at: 0x2_0000, bytes: app },
]);
const left = await inspect("left.bin", mergeBin(app));
const right = await inspect("right.bin", mergeBin(moved));
const inventory = artifactInventoryResultSchema.parse(left.normalized_result);
expect(inventory.coverage.status).toBe("complete");
expect(occurrence(left, "partitions/ota_0.bin").limitations).toEqual([
`partitions/ota_0.bin is truncated: the dump holds ${String(app.length)} of its 65536 bytes.`,
]);
const result = await compare(left, right);
expect(result.status).toBe("changed");
expect(changedPaths(result)).toEqual(["partitions/factory.bin"]);
// The bootloader, the table, nvs, otadata, phy_init, the truncated ota_0
// and ota_1, which neither file holds, are the same on both sides.
expect(result.summary).toMatchObject({
unchanged: 7,
changed: 2,
unknown: 0,
});
// A full dump of the same parts holds ota_0 whole and ota_1 erased: those
// entries change, and no partition is added or removed.
const full = await inspect(
"full.bin",
flashImage(0x4_0000, [
BOOTLOADER,
{ at: 0x8000, bytes: partitionTable(TWO_OTA_PARTITIONS) },
{ at: 0x1_0000, bytes: app },
{ at: 0x2_0000, bytes: app },
]),
);
const longer = await compare(left, full);
expect(longer.status).toBe("changed");
expect(
longer.changes
.filter(({ logical_path }) => logical_path !== ".")
.map(({ logical_path, classification }) => [logical_path, classification])
.sort(),
).toEqual([
["partitions/ota_0.bin", "changed"],
["partitions/ota_1.bin", "changed"],
]);
expect(occurrence(left, "partitions/ota_1.bin").limitations).toEqual([
"partitions/ota_1.bin lies outside the dump: the file holds none of its 65536 bytes.",
]);
});
tests/acceptance/artifactComparisonEspImage.test.ts
+108 -1
View File
@@ -3,10 +3,19 @@ import { join } from "node:path";
import { expect, it } from "vitest";
import { ESP32C3_APP_SEGMENTS, espImage } from "../fixtures/espImage.js";
import { artifactInventorySummaryViewResultSchema } from "../../src/domain/analysisView/analysisView.js";
import { artifactComparisonResultSchema } from "../../src/domain/artifactComparison.js";
import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js";
import { artifactInventoryResultSchema } from "../../src/domain/artifactGraph.js";
import { parseEvidence, type Evidence } from "../../src/domain/evidence.js";
import {
artifactGraphDigest,
artifactManifestId,
} from "../../src/domain/artifactIdentity.js";
import {
createEvidence,
parseEvidence,
type Evidence,
} from "../../src/domain/evidence.js";
import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
@@ -83,3 +92,101 @@ it("reports an ESP segment whose load address alone changed", async () => {
);
expect(same.changes).toEqual([]);
});
/** The inventory as REA recorded it before ESP image roots had `flash_dump`. */
const withoutFlashDump = (source: Evidence): Evidence => {
const inventory = artifactInventoryResultSchema.parse(
source.normalized_result,
);
const occurrences = inventory.occurrences.map((occurrence) => {
if (occurrence.format_metadata?.kind !== "esp-image") return occurrence;
const { flash_dump: _dump, ...historical } = occurrence.format_metadata;
return { ...occurrence, format_metadata: historical };
});
// Recommit the reduced graph, as that REA committed it.
const graphSha256 = artifactGraphDigest({
nodes: inventory.nodes,
occurrences,
edges: inventory.edges,
contradictions: inventory.integrity_contradictions,
});
return createEvidence(
{
path: source.subject!.local_path,
sha256: inventory.manifest.root_sha256,
format: inventory.manifest.root_format,
},
source.provider,
{
operation: "inventory_artifact",
parameters: {},
result: {
...inventory,
occurrences,
manifest: {
...inventory.manifest,
graph_sha256: graphSha256,
manifest_id: artifactManifestId(
inventory.manifest.root_artifact_id,
graphSha256,
),
},
},
confidence: "observed",
authority: "shipped-artifact",
},
);
};
it("reads and compares ESP image Evidence recorded before flash_dump", async () => {
const root = await createTestTempDirectory("rea-esp-historical-");
const path = join(root, "app.bin");
await writeFile(path, espImage({ segments: ESP32C3_APP_SEGMENTS }));
const { call } = await connectLocalToolsMcp();
const opened = await call("open_binary", { path });
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
const inspected = await call("inspect_artifact", {});
expect(inspected.isError, JSON.stringify(inspected)).not.toBe(true);
const current = parseEvidence(
artifactInspectionResultSchema
.parse(parseEvidence(inspected.structuredContent).normalized_result)
.substeps.find(({ operation }) => operation === "inventory_artifact")
?.evidence,
);
const historical = withoutFlashDump(current);
const recorded = JSON.stringify(historical);
const rootMetadata = (evidence: Evidence) =>
artifactInventoryResultSchema
.parse(evidence.normalized_result)
.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata;
expect(rootMetadata(current)).toMatchObject({
kind: "esp-image",
flash_dump: null,
});
expect(rootMetadata(historical)).not.toHaveProperty("flash_dump");
const compared = await call("compare_artifacts", {
left: historical,
right: current,
});
expect(compared.isError, JSON.stringify(compared)).not.toBe(true);
expect(
artifactComparisonResultSchema.parse(
parseEvidence(compared.structuredContent).normalized_result,
),
).toMatchObject({ status: "unchanged", changes: [] });
const viewed = await call("inspect_analysis_view", {
source: { kind: "inline", evidence: historical },
view: { kind: "summary" },
});
expect(viewed.isError, JSON.stringify(viewed)).not.toBe(true);
const summary = artifactInventorySummaryViewResultSchema.parse(
parseEvidence(viewed.structuredContent).normalized_result,
).summary;
expect(summary.root_format_metadata).toMatchObject({ kind: "esp-image" });
expect(summary.root_format_metadata).not.toHaveProperty("flash_dump");
// Reading the record leaves the graph its digest covers unchanged.
expect(JSON.stringify(historical)).toBe(recorded);
});
tests/acceptance/espFlashImageCli.test.ts
+164
View File
@@ -0,0 +1,164 @@
import { readFile, rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, onTestFinished } from "vitest";
import { cliTest } from "../support/cli/cliFixture.js";
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
} from "../fixtures/espImage.js";
import {
espFlashDump,
otaData,
otaRecord,
} from "../../src/domain/firmware/espFlash.fixture.js";
import { parseEvidence } from "../../src/domain/evidence.js";
import { artifactInspectionResultSchema } from "../../src/domain/artifactInspection.js";
import {
artifactExtractionResultSchema,
artifactInventoryResultSchema,
} from "../../src/domain/artifactGraph.js";
const rootMetadata = (json: unknown) => {
const graph = artifactInventoryResultSchema.parse(
artifactInspectionResultSchema.parse(parseEvidence(json).normalized_result)
.substeps[0]?.evidence.normalized_result,
);
return {
graph,
root: graph.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata,
};
};
cliTest(
"inventories and extracts a whole-flash dump through the compiled CLI",
async ({ cli }) => {
const directory = await createTestTempDirectory("rea-esp-flash-cli-");
const source = join(directory, "flash.bin");
const app = espImage({ segments: ESP32C3_APP_SEGMENTS });
const dump = espFlashDump({
bootloader: {
at: 0,
bytes: espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS }),
},
parts: [
{ at: 0xd000, bytes: otaData(otaRecord({ seq: 2 })) },
{ at: 0x2_0000, bytes: app },
{ at: 0x3_0000, bytes: app },
],
});
await writeFile(source, dump);
const inspected = await cli.run({
arguments: ["inspect-artifact", source, "--json"],
});
expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0);
const { graph, root } = rootMetadata(inspected.json);
expect(graph.manifest.root_format).toBe("esp-flash-image");
expect(root).toMatchObject({
kind: "esp-flash-image",
chip: { target: "esp32c3" },
partition_table: { method: "default", md5: { status: "valid" } },
boot_selection: {
basis: "otadata",
slot: 1,
partition_index: 5,
running_partition: "unknown",
},
});
const extracted = await cli.run({
arguments: ["extract-artifact", source, "--json"],
environment: { TMPDIR: directory, TMP: directory, TEMP: directory },
});
expect(extracted.exitCode, extracted.stdout + extracted.stderr).toBe(0);
const result = artifactExtractionResultSchema.parse(
parseEvidence(extracted.json).normalized_result,
);
onTestFinished(() =>
rm(result.output_root, { recursive: true, force: true }),
);
expect(
Buffer.compare(
await readFile(join(result.output_root, "partitions/ota_1.bin")),
dump.subarray(0x3_0000, 0x4_0000),
),
).toBe(0);
},
);
cliTest(
"reads a dump from a caller-selected flash offset and refuses unpaired inputs",
async ({ cli }) => {
const directory = await createTestTempDirectory(
"rea-esp-flash-cli-origin-",
);
const source = join(directory, "from-0x1000.bin");
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
await writeFile(
source,
espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } }).subarray(
0x1000,
),
);
const unpaired = await cli.run({
arguments: [
"inspect-artifact",
source,
"--flash-offset",
"0x1000",
"--json",
],
});
expect(unpaired.exitCode).not.toBe(0);
expect(unpaired.stdout + unpaired.stderr).toContain(
"flash_offset and partition_table_offset apply only with format=esp-flash-image",
);
// Without the format, the bootloader is read as one ESP image, and the
// table at file offset 0x7000 says how to reopen the dump.
const automatic = await cli.run({
arguments: ["inspect-artifact", source, "--json"],
});
expect(automatic.exitCode, automatic.stdout + automatic.stderr).toBe(0);
const guessed = rootMetadata(automatic.json);
expect(guessed.graph.manifest.root_format).toBe("esp-image");
expect(guessed.root).toMatchObject({
flash_dump: { flash_offset: 0x1000, table_file_offset: 0x7000 },
});
expect(
artifactInspectionResultSchema.parse(
parseEvidence(automatic.json).normalized_result,
).next_probes,
).toEqual([
{
operation: "open_binary",
rationale:
"Reopen the file with format esp-flash-image and flash_offset 0x1000, then inspect_artifact: it is a flash dump read from flash 0x1000, with a partition table at file offset 0x7000.",
authority: "artifact-bytes",
},
]);
expect(guessed.graph.limitations).toContain(
"This file looks like a flash dump read from flash 0x1000, not one ESP image: its esp32 image at file offset 0 is where the ROM loads the bootloader from 0x1000, and a partition table sits at file offset 0x7000, which is flash 0x8000 from that origin. Reopen it with format esp-flash-image and flash_offset 0x1000.",
);
const inspected = await cli.run({
arguments: [
"inspect-artifact",
source,
"--target-format",
"esp-flash-image",
"--flash-offset",
"0x1000",
"--json",
],
});
expect(inspected.exitCode, inspected.stdout + inspected.stderr).toBe(0);
expect(rootMetadata(inspected.json).root).toMatchObject({
flash_offset: { value: 0x1000, source: "caller" },
chip: { target: "esp32" },
bootloader: { flash_address: 0x1000 },
});
},
);
tests/boundary/filesystem/binaryTarget.test.ts
+116
View File
@@ -35,9 +35,11 @@ import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { ESP8266_REFUSAL } from "../../../src/domain/firmware/espImage.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
esp8266Image,
espImage,
} from "../../fixtures/espImage.js";
import { espFlashDump } from "../../../src/domain/firmware/espFlash.fixture.js";
import { javaClassFile } from "../../fixtures/javaClassFile.js";
import {
MAX_JVM_CLASS_IDENTITY_BYTES,
@@ -596,6 +598,120 @@ describe("binary target I/O: package classification", () => {
});
});
describe("binary target I/O: ESP flash layouts", () => {
it("detects a flash layout before the bootloader it begins with", async () => {
const directory = await createTestTempDirectory("rea-esp-flash-target-");
const dump = join(directory, "flash.bin");
const bootloader = join(directory, "bootloader.bin");
const image = espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS });
await Promise.all([
writeFile(dump, espFlashDump({ bootloader: { at: 0, bytes: image } })),
writeFile(bootloader, image),
]);
expect(await parseBinaryTarget(dump)).toMatchObject({
ok: true,
value: {
kind: "archive",
format: "esp-flash-image",
espFlash: {
selected: false,
flashOffset: null,
partitionTableOffset: null,
},
},
});
expect(await parseBinaryTarget(bootloader)).toMatchObject({
ok: true,
value: { kind: "archive", format: "esp-image" },
});
});
it("keeps refusing an ESP8266 dump that has an ESP-IDF-style table", async () => {
const directory = await createTestTempDirectory("rea-esp8266-flash-");
const dump = join(directory, "esp8266-flash.bin");
await writeFile(
dump,
espFlashDump({ bootloader: { at: 0, bytes: esp8266Image() } }),
);
expect(await parseBinaryTarget(dump)).toMatchObject({
ok: false,
error: {
constraint: "file_format_rejected",
reason: ESP8266_REFUSAL,
},
});
});
it("reads a selected flash dump with caller inputs and refuses bad pairings", async () => {
const directory = await createTestTempDirectory("rea-esp-flash-select-");
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0,
});
const partial = join(directory, "from-0x1000.bin");
const noise = join(directory, "noise.bin");
await Promise.all([
writeFile(
partial,
espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } }).subarray(
0x1000,
),
),
writeFile(noise, Buffer.alloc(0x1_0000, 0x5a)),
]);
expect(
await parseBinaryTarget(partial, {
formatHint: "esp-flash-image",
espFlash: { flashOffset: 0x1000 },
}),
).toMatchObject({
ok: true,
value: {
kind: "archive",
format: "esp-flash-image",
espFlash: {
selected: true,
flashOffset: 0x1000,
partitionTableOffset: null,
},
},
});
for (const [options, reason] of [
[
{ espFlash: { flashOffset: 0x1000 } },
"flash_offset and partition_table_offset apply only with format=esp-flash-image",
],
[
{
formatHint: "esp-flash-image",
espFlash: { partitionTableOffset: 0x8800 },
},
"partition_table_offset must be a multiple of 0x1000, as ESP-IDF requires",
],
] as const)
expect(await parseBinaryTarget(partial, options)).toMatchObject({
ok: false,
error: {
_tag: "BinaryTargetError",
constraint: "load_selection_rejected",
reason,
},
});
// Ciphertext is not recognized as a layout: unknown, not malformed.
expect(
await parseBinaryTarget(noise, { formatHint: "esp-flash-image" }),
).toMatchObject({
ok: false,
error: {
constraint: "file_format_rejected",
reason: expect.stringContaining(
"ESP flash image layout is not recognized",
),
},
});
});
});
it("classifies a FIT by its /images node and leaves a plain DTB rejected", async () => {
const directory = await createTestTempDirectory("rea-fit-target-");
const itb = join(directory, "boot.itb");
tests/boundary/mcp/espFlashImageMcp.test.ts
+287
View File
@@ -0,0 +1,287 @@
import { readFile, rm, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { expect, it } from "vitest";
import { z } from "zod";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
import {
ESP32C3_APP_SEGMENTS,
ESP32C3_BOOTLOADER_SEGMENTS,
espImage,
pattern,
} from "../../fixtures/espImage.js";
import {
espFlashDump,
otaData,
otaRecord,
} from "../../../src/domain/firmware/espFlash.fixture.js";
import { parseEvidence } from "../../../src/domain/evidence.js";
import {
artifactExtractionResultSchema,
artifactInventoryResultSchema,
} from "../../../src/domain/artifactGraph.js";
import { artifactInspectionResultSchema } from "../../../src/domain/artifactInspection.js";
const summarySchema = z.object({
summary: z.object({ root_format_metadata: z.unknown() }),
});
const appSummariesSchema = z.object({
summary: z.object({
inventory_evidence_ids: z.array(z.string()),
root_format_metadata: z.object({
apps: z.array(
z.object({
summary: z.object({
descriptor: z.unknown(),
load_ranges: z.array(z.object({ classification: z.string() })),
}),
}),
),
}),
}),
});
const completeAppsSchema = z.object({
apps: z.array(
z.object({
path: z.string(),
facts: z.object({ image: z.unknown(), segments: z.array(z.unknown()) }),
}),
),
});
const completeBootloaderSchema = z.object({
bootloader: z.object({
facts: z.object({
segments: z.array(
z.object({
load_addr: z.number(),
data_len: z.number(),
classification: z.string(),
}),
),
}),
}),
});
// The compiled stdio fixture checks advertised schemas against their dialect
// and validates actual tool responses.
it.each(["full", "compact"] as const)(
"reads a flash dump at the caller's origin and keeps inputs in its identity (%s)",
async (profile) => {
const root = await createTestTempDirectory("rea-esp-flash-mcp-");
const path = join(root, "from-0x1000.bin");
const esp32 = espImage({
segments: ESP32C3_BOOTLOADER_SEGMENTS,
chipId: 0x0000,
});
const dump = espFlashDump({ bootloader: { at: 0x1000, bytes: esp32 } });
await writeFile(path, dump.subarray(0x1000));
const { call } = await connectLocalToolsMcp({
REA_MCP_INPUT_SCHEMA_PROFILE: profile,
});
let outputRoot: string | undefined;
try {
const rejected = await call("open_binary", {
path,
flash_offset: "0x1000",
});
expect(rejected.isError).toBe(true);
expect(JSON.stringify(rejected)).toContain(
"flash_offset and partition_table_offset apply only with format=esp-flash-image",
);
const opened = await call("open_binary", {
path,
format: "esp-flash-image",
flash_offset: "0x1000",
});
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
expect(JSON.stringify(opened.structuredContent)).toContain(
'"partition_table_offset":null',
);
const reply = await call("inspect_artifact", { detail: "summary" });
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
const summaryRoot = summarySchema.parse(
parseEvidence(reply.structuredContent).normalized_result,
).summary.root_format_metadata;
expect(summaryRoot).toMatchObject({
kind: "esp-flash-image-summary",
flash_offset: { value: 0x1000, source: "caller" },
chip: { target: "esp32" },
partition_table: { flash_address: 0x8000, method: "default" },
});
// The summary view gives each image's summary; complete keeps segments.
const complete = await call("inspect_artifact", {});
expect(complete.isError, JSON.stringify(complete)).not.toBe(true);
const { facts } = completeBootloaderSchema.parse(
artifactInventoryResultSchema
.parse(
artifactInspectionResultSchema.parse(
parseEvidence(complete.structuredContent).normalized_result,
).substeps[0]?.evidence.normalized_result,
)
.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata,
).bootloader;
expect(facts.segments).toHaveLength(ESP32C3_BOOTLOADER_SEGMENTS.length);
expect(summaryRoot).toMatchObject({
bootloader: {
path: "bootloader.bin",
flash_address: 0x1000,
summary: {
status: "esp-image",
chip: "esp32",
segment_count: ESP32C3_BOOTLOADER_SEGMENTS.length,
load_ranges: facts.segments
.filter(({ classification }) => classification !== "not-loaded")
.map(({ load_addr, data_len, classification }) => ({
address: load_addr,
length: data_len,
classification,
})),
signature_block_count: 0,
descriptor: { kind: "bootloader" },
checksum: "valid",
},
},
});
expect(JSON.stringify(summaryRoot)).not.toContain('"segments"');
const extracted = await call("extract_artifact", {});
expect(extracted.isError, JSON.stringify(extracted)).not.toBe(true);
const result = artifactExtractionResultSchema.parse(
parseEvidence(extracted.structuredContent).normalized_result,
);
outputRoot = result.output_root;
// nvs at flash 0x9000 is file offset 0x8000 in a dump read from 0x1000.
expect(
Buffer.compare(
await readFile(join(result.output_root, "partitions/nvs.bin")),
dump.subarray(0x9000, 0xd000),
),
).toBe(0);
// Reopening the same bytes with other inputs is a different target.
const reopened = await call("open_binary", {
path,
format: "esp-flash-image",
flash_offset: "0x1000",
partition_table_offset: "0x8000",
});
expect(reopened.isError, JSON.stringify(reopened)).not.toBe(true);
const again = await call("inspect_artifact", { detail: "summary" });
expect(
summarySchema.parse(
parseEvidence(again.structuredContent).normalized_result,
).summary.root_format_metadata,
).toMatchObject({ partition_table: { method: "caller" } });
} finally {
if (outputRoot !== undefined)
await rm(outputRoot, { recursive: true, force: true });
}
},
);
it("summarizes a populated app slot and an erased one, and retains their full facts", async () => {
const root = await createTestTempDirectory("rea-esp-flash-mcp-apps-");
const path = join(root, "device.bin");
const app = espImage({ segments: ESP32C3_APP_SEGMENTS });
// ota_1's first sector is erased; an older image's tail lies further in.
const leftover = { at: 0x3_2000, bytes: pattern(0x200, 3) };
await writeFile(
path,
espFlashDump({
bootloader: {
at: 0,
bytes: espImage({ segments: ESP32C3_BOOTLOADER_SEGMENTS }),
},
parts: [
{ at: 0xd000, bytes: otaData(otaRecord({ seq: 1 })) },
{ at: 0x2_0000, bytes: app },
leftover,
],
}),
);
const { call } = await connectLocalToolsMcp({});
const opened = await call("open_binary", { path });
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
const reply = await call("inspect_artifact", { detail: "summary" });
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
const result = parseEvidence(reply.structuredContent).normalized_result;
const { summary } = appSummariesSchema.parse(result);
expect(
summarySchema.parse(result).summary.root_format_metadata,
).toMatchObject({
kind: "esp-flash-image-summary",
boot_selection: { basis: "otadata", partition_index: 4 },
apps: [
{ path: "partitions/factory.bin", summary: { status: "erased" } },
{
path: "partitions/ota_0.bin",
partition_index: 4,
summary: {
status: "esp-image",
chip: "esp32c3",
segment_count: ESP32C3_APP_SEGMENTS.length,
signature_block_count: 0,
checksum: "valid",
coverage: "complete",
},
},
{
path: "partitions/ota_1.bin",
partition_index: 5,
summary: {
status: "erased",
chip: null,
load_ranges: [],
descriptor: null,
checksum: null,
},
},
],
});
// The descriptor keeps what tells two builds apart, and no more.
const ota0 = summary.root_format_metadata.apps[1]?.summary;
expect(ota0?.descriptor).toEqual({
kind: "app",
project_name: "rea_esp_fixture",
version: "1.2.3",
idf_ver: "v6.1",
secure_version: 0,
app_elf_sha256: "ab".repeat(32),
});
// Every load range names whether it is mapped from flash or loaded to RAM.
expect(
new Set(ota0?.load_ranges.map(({ classification }) => classification)),
).toEqual(new Set(["mapped", "loaded"]));
// The summary points at the inventory Evidence, which keeps full facts.
const complete = await call("inspect_artifact", {});
expect(complete.isError, JSON.stringify(complete)).not.toBe(true);
const inventory = artifactInspectionResultSchema.parse(
parseEvidence(complete.structuredContent).normalized_result,
).substeps[0]?.evidence;
expect(summary.inventory_evidence_ids).toContain(inventory?.evidence_id);
const full = completeAppsSchema.parse(
artifactInventoryResultSchema
.parse(inventory?.normalized_result)
.occurrences.find(({ logical_path }) => logical_path === ".")
?.format_metadata,
).apps;
expect(full[1]).toMatchObject({
path: "partitions/ota_0.bin",
facts: { image: { descriptor: { kind: "app", date: expect.any(String) } } },
});
expect(full[1]?.facts.segments).toHaveLength(ESP32C3_APP_SEGMENTS.length);
// The erased slot's written bytes stay on its entry, with the limitation.
const entries = artifactInventoryResultSchema.parse(
inventory?.normalized_result,
).occurrences;
expect(
entries.find(({ logical_path }) => logical_path === "partitions/ota_1.bin")
?.format_metadata,
).toMatchObject({
written_offset: 0x2000,
written_end: 0x2200,
written_bytes: expect.any(Number),
});
});
tests/conformance/esp/esp32-app-riscv.ld
+23
View File
@@ -0,0 +1,23 @@
ENTRY(rea_entry)
/*
* ESP32 address ranges from ESP-IDF v6.1
* components/soc/esp32/include/soc/soc.h for the RISC-V conformance app,
* which esptool converts for any chip and REA never runs. One program
* header per memory keeps the ELF small.
*/
PHDRS
{
drom PT_LOAD;
dram PT_LOAD;
iram PT_LOAD;
irom PT_LOAD;
}
SECTIONS
{
.flash.appdesc 0x3F400020 : { KEEP(*(.flash.appdesc)) } :drom
.flash.rodata : { *(.flash.rodata) *(.rodata .rodata.* .srodata .srodata.*) } :drom
.dram0.data 0x3FFB0000 : { *(.dram0.data) *(.data .data.* .sdata .sdata.*) } :dram
.iram0.text 0x40080000 : { *(.iram0.text) *(.text .text.*) } :iram
.flash.text 0x400D0020 : { *(.flash.text) } :irom
/DISCARD/ : { *(.comment) *(.note.*) }
}
tests/conformance/esp/esp32-bootloader.ld
+19
View File
@@ -0,0 +1,19 @@
ENTRY(rea_entry)
/*
* RAM-only ESP32 layout for the RISC-V conformance bootloader, which esptool
* converts for any chip and REA never runs. The addresses are in the DRAM
* and IRAM ranges of ESP-IDF v6.1 components/soc/esp32/include/soc/soc.h,
* where an ESP32 second-stage bootloader loads.
*/
PHDRS
{
dram PT_LOAD;
iram PT_LOAD;
}
SECTIONS
{
.dram0.bootdesc 0x3FFF0030 : { KEEP(*(.dram0.bootdesc)) } :dram
.dram0.data : { *(.dram0.data) *(.data .data.* .sdata .sdata.*) *(.rodata .rodata.* .srodata .srodata.*) } :dram
.iram_loader.text 0x40080400 : { *(.iram_loader.text) *(.text .text.*) } :iram
/DISCARD/ : { *(.comment) *(.note.*) }
}
tests/conformance/esp/flash_file_tool.py
+61
View File
@@ -0,0 +1,61 @@
"""Run ESP-IDF's otatool against a flash dump file instead of a device.
Usage: python flash_file_tool.py IMAGE FLASH_OFFSET otatool [ARGS...]
ESP-IDF's parttool, which otatool uses, reaches flash only through
ParttoolTarget._call_esptool with esptool's read-flash, erase-region and
write-flash commands (v4 spells them with underscores). This script replaces
that one method with reads and writes of IMAGE, whose first byte is flash
address FLASH_OFFSET; otatool's own otadata logic runs unchanged. IDF_PATH
names the ESP-IDF checkout to import from.
"""
import os
import sys
IDF = os.environ['IDF_PATH']
sys.path[:0] = [
os.path.join(IDF, 'components', 'partition_table'),
os.path.join(IDF, 'components', 'app_update'),
]
import otatool # noqa: E402
import parttool # noqa: E402
IMAGE, FLASH_OFFSET = sys.argv[1], int(sys.argv[2], 0)
def _file_offset(address):
offset = int(address, 0) - FLASH_OFFSET
if offset < 0:
raise SystemExit(f'flash address {address} precedes the dump')
return offset
def _call_esptool(self, args, out=None):
command, *rest = args
name = command.replace('_', '-')
with open(IMAGE, 'r+b') as flash:
if name == 'read-flash':
offset, size = _file_offset(rest[0]), int(rest[1], 0)
flash.seek(offset)
data = flash.read(size)
with open(rest[2], 'wb') as output:
output.write(data + b'\xff' * (size - len(data)))
elif name == 'erase-region':
flash.seek(_file_offset(rest[0]))
flash.write(b'\xff' * int(rest[1], 0))
elif name == 'write-flash':
flash.seek(_file_offset(rest[0]))
with open(rest[1], 'rb') as data:
flash.write(data.read())
else:
raise SystemExit(f'unsupported esptool command {command}')
parttool.ParttoolTarget._call_esptool = _call_esptool
if sys.argv[3] != 'otatool':
raise SystemExit('usage: flash_file_tool.py IMAGE FLASH_OFFSET otatool ARGS...')
sys.argv = ['otatool.py', *sys.argv[4:]]
otatool.main()