Thirteen guard workflows (fifteen jobs) ran on the same 4-vCPU Ubuntu
runner, all started right after detect, each paying a checkout and a full
billed minute for 0-13s of work: infographics, profile archives, case
collisions, blob size, lazy_deps imports, stale icons, unrelated
histories, contributor attribution, uv.lock, prompt-surface and
package-lock diffs, the supply-chain scan and dependency bounds, and the
Docker lint.
repo-guards.yml runs them as steps of one job. Each step keeps its old
lane gate in its `if:`, and runs whatever an earlier step did
(`!cancelled()`), so one red guard never hides another; any failed step
fails the job, which all-checks-pass requires. The review-status-*
artifacts the live PR comment reads keep their names. Files the steps
write go under $RUNNER_TEMP, and the guards that need no toolchain run
before setup-pm, whose dependency phase also checks the lock.
The four PR-only jobs the strict release gate tolerated as skips are now
steps of a job that runs on every event, so the only tolerated release
skip left is the deferred Desktop E2E.
Publishing v0.23.0 committed the Microsoft Store submission, and Partner
Center answered POST .../commit with 202 Accepted and status CommitStarted.
The submission API helper only accepted 200, so the publication pass logged
"commit failed: 202 {'status': 'CommitStarted'}" and told the release owner
to check Partner Center by hand, even though the commit had started.
Any 2xx is now success. The fake Partner Center in the tests answers a commit
with 202, as the live API did, so the six tests that commit a submission fail
on the old check.
With MS_STORE_HOLD set after a build was committed Immediate, the API
refuses the switch to Manual and the pass told the operator to select
Publish now, though the build goes live by itself once certified. A
refused switch now reads the committed mode: Immediate reports that it
publishes on its own and that the hold cannot apply.
Committing the Store draft at publication makes every certified build go
live by itself, with no way left to keep one back for a chosen release time.
Setting the repository variable MS_STORE_HOLD=true makes the publication
pass commit the draft Manual instead: it is certified and then waits for
Publish now in Partner Center, with a warning that says so. A rerun of a
held submission is a no-op and never flips it to Immediate. Unset keeps the
auto-publish default.
Also quote the real v0.22.7 refusal (state 'PreProcessing') and update the
stable-releases developer guide, which still described the held commit.
The green run committed the Store submission held (Manual) and the
publication pass tried to switch it to Immediate. Partner Center refuses
to update a committed submission (v0.22.7: 409 InvalidState, "Cannot
update the submission because it is in certification"), so every
release needed Publish now by hand.
stable-store now leaves an uncommitted draft set to Immediate, and the
publication pass commits it. Nothing reaches certification or customers
before the release is published, and the build goes live as soon as it
passes certification.
scripts/releases/desktop_targets.py holds the (electron platform, arch)
triple and derives the manifest spelling and the native handoff legs from it.
channel_artifacts, channel_publish, channel_releases, stable and the builds
table read it instead of five hand-written copies, and NATIVE_LEGS has one
definition instead of two.
read_native_receipts always reads every native leg, so its legs parameter
and _leg_targets go. A claim with includeMacosIntel: true now says why it is
refused. Tests iterate the targets directly instead of skipping a 2x2 case.
Remove the darwin-x64 build, receipt and smoke jobs from Desktop Bundled
Release and the candidates/transitions/packaged-x64 jobs from Stable Release,
with the release scripts that required them: the macos-intel lane, the
darwin-x64 job group, the x64 macOS target in candidate/channel manifests and
the two-leg macOS feed merge. The macOS updater feed is arm64-only.
Pre-lanes claims still read: the published ones all carry
includeMacosIntel:false; a claim with true is now refused.
stable-store commits the Store submission held (targetPublishMode Manual)
instead of leaving an uncommitted draft, so certification runs before the
release gate and a run whose stable-publish fails still shows customers
nothing. The publication pass then PUTs the held submission with
Immediate and commits it again, so it goes live without a Partner Center
click.
Microsoft documents no call that releases a held submission and does not
say whether update accepts a committed one (409 is documented only as
"the current state of the app"). A 409 from the switch leaves the
submission held and prints the Publish now instruction; the release
stays green. The first real release shows which answer the API gives.
Reconcile receipt-bound Store work independently of channel advancement, including
already-current published releases, so an interruption cannot strand a held
submission. Record each upload's claim in certification notes and replace a held
Manual predecessor only after its exact remote abandon marker matches the claim
commit. Unknown owners and publishing submissions remain protected.
candidate-manifest spent ~4 of its 5.2 min downloading 8.7 GB (stable run
38491090442: both MSIX bundles, including the 4.3 GB Store bundle) only to
re-hash them, and each stage-receipt job re-downloaded its group's ZIPs and
bundle (60-256 s). The same bytes are hashed by the build job that uploaded
them, bound by digest in its handoff receipt, and downloaded and digest-
checked again by release_artifacts.publish (materialize) before anything
goes live.
Candidate staging and receipts now download only the metadata JSONs; package
rows still carry the receipt digests. The one check that read package
contents, the universal bundle identity against its per-arch rows, moves to
publish, after materialize has verified the bytes. A Store-only claim has no
metadata to fetch, so candidate-manifest reads its receipts alone.
The Store build used to be committed at green with targetPublishMode
Manual, and the publication pass could only print "click Publish now":
the submission API has no call that releases a held submission (update
and commit return 409 once it is committed). Every stable release needed a
person in Partner Center.
stable-store now uploads the verified bundle as an uncommitted draft
(msstore publish --noCommit) and stops. The publication pass, which runs
only when the release really goes out, reads the receipt-bound candidate
manifest, checks the pending draft carries that release's Store bundle,
sets targetPublishMode Immediate and commits it. Certification starts
then, and the build goes live when it passes.
- A draft from a newer green run is left for its own release; a committed
or live submission is a no-op, so reruns are safe.
- submit refuses to replace a committed submission still on its way live
(an earlier release); msstore publish would delete it.
- complete's Reconcile step now receives the MS_STORE_* credentials; it had
none, so the success path silently skipped the Store.
- A failed or unreadable Store step stays a warning: the release publishes.
Stable builds each Windows arch in its own desktop-bundled-release call and
assembles the universal bundle in a third call that needs both. Each per-arch
call also ran that arch's MSIX install-and-chat smoke, so assembly waited for
the slower arch's smoke after its build: on rc.2-v0.22.2 the bundle sat behind
the arm64 smoke from 07:16 to 07:28.
A new windows-smokes-with-bundle input moves both MSIX smokes into the
win32-bundle call. The smokes already fetch the staged package from the R2
handoff, so they run there beside assembly. The per-arch calls then only
build, the bundle call's phase result requires both smokes, and the candidate
manifest records the Windows smoke results from the bundle call, so a failed
smoke still blocks the release. Callers that build everything in one call
(canary, channel, commit builds) leave the input off and are unchanged.
The release entrypoint and the stable gate both required the claimed commit
to be an ancestor of main. Both now accept an ancestor of main or of any
release/* branch, via one shared git query (stable.release_line_contains).
The entrypoint prunes its mirror of release/* so a deleted branch stops
admitting commits. The termux previous-package ancestry check follows the
same rule. Workflows still dispatch from main.
- The Windows build step passes AZURE_* empty when skip-signed is set.
after-pack and electron-builder sign whenever the vars exist, and the
Azure login is skipped with the signed lane, so a Store-only build with
the vars configured would sign without credentials.
- candidate_manifest records a smoke result only for calls whose package is
a target. `--without windows` runs the Windows candidate calls for the
Store bundle; their success is not a smoke, and counting it made assemble
refuse the manifest.
`windows` used to build the signed MSIX, the universal bundle and the Store
bundle together, so skipping the signed pipeline meant skipping the Store too.
Now `windows` is the signed pipeline and `windows-store` the unsigned Store
bundle; both default on and the shared per-arch builds only make what the
lanes ask for.
- lanes: new `windows-store` lane and derived `windows-build` (either lane),
which gates the shared candidates-win32-* jobs and handoffs.
- desktop-bundled-release: `skip-signed` builds only the Store variant and
needs no Azure login, signing, metadata, smoke or receipt; `skip-store`
builds and submits no Store bundle (replaces the old `store` input).
- manifest: a Store-only release has no package rows, only the Store bundle
file; assemble() now takes the claim's lanes and checks each Windows bundle
against its lane.
- claims record `lanesRevision: 2`. A lanes claim without it was cut when
`windows` also meant the Store bundle (rc.2-v0.22.1), so it reads as
windows + windows-store.
- release_artifacts: drop the unused `assemble` CLI command, which could not
name the lanes.
Two review findings on the lanes change:
- candidate-manifest judged only the desktop calls, and when a claim built
none (--only termux) it called require_success with an empty list and could
never finish. It now requires every candidate call the claim built, Termux
included, and judges smoke results only for the desktop calls that have one.
- `complete` rendered the shared releases/stable/index.html for any release
with bundles, so a windows-only release replaced the previous full release's
macOS links. admit now emits a derived `desktop-channel` flag and the
renderer takes --no-channel-page, leaving the tag page and release table
alone while the shared page waits for a release that serves the channel.
The skip and opt-in flags (--skip-bundles, --skip-tests, --include-macos-intel,
--include-termux) were four booleans in the claim, and every consumer
re-derived what they meant. A release is now a set of lanes, defined once in
scripts/releases/lanes.py: five build lanes (docker, windows, macos-arm64,
macos-intel, termux) and ten test lanes (ci, nix, docker-tests, pm-bundle,
windows-live, install-e2e, bootstrap-version, termux-checks, smoke, upgrade).
The default is every lane except macos-intel and termux.
release.py release --only windows,docker # the Store bundle + image, untested
release.py release --with macos-intel --without install-e2e
- The claim records `lanes`. A claim cut before lanes existed still parses:
skipBundles/skipTests/include* translate to the lane set that gated the
same jobs, because the sequencer reads every claim tag ever cut.
- `admit` emits `lanes` and `jobs` (a run flag per lane-controlled job). Each
job's `if` reads its own `jobs` entry and the gates derive their expected
skips from the same table, so the workflow and the gates cannot drift.
tests/ci/test_stable_release_graph.py evaluates every `if` against 37 lane
sets.
- A release may build any subset: the candidate manifest carries exactly the
claim's targets, and the final tag records null for the docker digest or the
manifest digest when the claim built neither.
- The protected desktop channel, feeds, APT index and upgrade baseline move
only when the claim built windows and macos-arm64. A partial set is staged,
verified and (with windows) submitted to the Store, and leaves the channels
on the previous release. A release that moves no channel is finished once
public.
- publish-bundles passes `store` so a claim without windows submits nothing
to the Microsoft Store.
Not exercised end to end: no real release was cut with a partial set.
The channel advance wrote releases/stable/release-candidates.json with no
ETag, which the store sends as If-None-Match: *. After the first bundled
release the head always exists, so every later publish failed with R2 412
(ChannelConflict) after making the GitHub release public: v0.22.0 went
public while the stable head, desktop feeds and Docker aliases stayed on
v0.21.7. Read the head and write under its current ETag.
Stable ran on its claim tag, so its Actions caches lived in a scope no
later claim could read. Every attempt rebuilt wheels and re-signed every
Windows payload file. Canary already runs from the default branch with the
tag as an input. Stable now does the same.
- `check_claim` reads the commit from the claim tag. It requires the
default-branch dispatch of stable-release.yml instead of GITHUB_REF and
GITHUB_SHA.
- The protected publisher accepts stable-release.yml on the default branch.
- `release.py` and the release entrypoint dispatch on the default branch.
- Runs are matched to a claim by their run name. Every run now has the same
head branch. Runs that started on the tag still match.
- `release_artifacts` reads RELEASE_COMMIT before GITHUB_SHA. The publish
and Store jobs now set it to the admitted commit.
After #135054 the images are pushed under the attempt ref (rc.N-vX.Y.Z),
and promote_stable only moved stable/latest, so :vX.Y.Z and
:vX.Y.Z-desktop were never created. The publication pass now tags both
variants with the final release tag alongside the stable/latest aliases,
from the attempt's digest, and reads each back. The final tag still only
appears at publish, so an earlier attempt cannot collide with it.
The Microsoft Store variant forced HERMES_GUEST_ONBOARDING=1 into its bundle env,
and commit/channel builds could bake it with --bundle-env. Nothing reads it now.
- scripts/bundles/desktop_inputs.py: the Store bundle env is just the request's.
- bundle_env.py / bundle-env.mjs: the key is no longer allowed for new builds.
- channel-protocol.ts: the key moves to RETIRED_BUNDLE_ENV_KEYS, so preview
manifests already published with it still decode (the HERMES_SKIP_INTRO
precedent).
- BUILDING.md example uses HERMES_PREVIEW_FULL_CONNECTORS.
- Build tests use another allowed key as their sample; both validators reject the
retired key and the decoder drops it.
- msix-virtualization.test.mjs: every generated MSIX manifest (store, release,
canary, channel) keeps filesystem and registry write virtualisation disabled,
and the builder config declares unvirtualizedResources. The machine marker and
the shared %LOCALAPPDATA%\hermes root depend on that.
The 0.21.7 publication pass downloaded every package twice (~13 GB of
native handoffs, then all 20.5 GB of candidate files including the Store
packages the feeds never reference) to recompute digests the receipts
already bind, then died in the macOS feed merge:
ModuleNotFoundError: No module named 'ruamel'
- The sequencer steps run under python_packages with ruamel.yaml (the
node toolchain has no project deps), as the canary finalize already does.
- Stable advance fetches only metadata-*.json and the per-arch *-mac.yml
legs. assemble(local_bytes=False) binds each artifact by its receipt
sha256 and takes the macOS sha512s from the receipt-bound feed legs,
cross-checked against receipt sizes.
- promote() materializes only non-package files (feed legs, APT indexes).
- finalize(verify_bytes=False) checks remote sizes with HEAD instead of
streaming the zip/dmg back down.
Canary (no accepted candidate) keeps the full download path unchanged.
A publication run sat silent for 8+ minutes while it downloaded ~20 GB,
with no way to tell which phase it was in or why a claim was held.
scripts/releases/progress.py logs to stderr (stdout stays the JSON steps)
with GitHub ::group::/::notice::/::error:: markers:
- every git/gh/docker command the sequencer runs, with its time and result
- discovery counts, and each claim's state with the reason (run attempt
and conclusion, draft location, abandon marker, receipt/release match)
- the plan: why each version advances, is skipped, or is held
- each step as a timed group: publish (receipt digests) and advance
- the channel advance's phases, plus per-file progress for the handoff
fetch and candidate materialize downloads (the slow part)
- the head before and after, and a top-level ::error:: on failure
Publishing 0.21.7 made the release public, then failed the channel
advance in the same pass with
ChannelError: Stable candidate manifest differs from the final release receipt
discover() builds each record before the vX.Y.Z receipt tag exists, so a
green claim's record has candidate_manifest_sha256 = None.
publish_attempt() writes the receipt but returned only the Docker digest,
so the advance compared the archive's real digest against None. Every
first-pass publish of a bundle release failed here; only a rerun (which
rediscovers the receipt) could advance the head.
publish_attempt() now returns both receipt digests and the sequencer
applies them to the record before the advance. The sequencer fixture can
now stage a fresh green claim (no receipt tag, draft on the attempt ref,
one successful run); the new test fails on main with the production error.
The publication reconcile checks out with persist-credentials: false, so
when the sequencer reached ensure_final_tag() its `git push origin
refs/tags/vX.Y.Z` died with
fatal: could not read Username for 'https://github.com'
GH_TOKEN authenticated gh but never git. The push now carries its own
auth: when GH_TOKEN is set it resets inherited credential helpers and
uses `gh auth git-credential`, so the token stays out of argv and out of
.git/config, and the checkout keeps persisting nothing.
Stable Release Publication for 0.21.7 failed in advance_stable with
ChannelError: Native metadata release identity mismatch
publish_release() passes the attempt ref (rc.15-v0.21.7) to
read_native_receipts(), which is right for the handoff receipts: they are
staged under releases/tag/<attempt>/. But record() stamps each
metadata-<platform>-<arch>.json with the payload tag (v0.21.7), so the
row check compared v0.21.7 against rc.15-v0.21.7 and refused every
stable release. Canary never noticed because its tag == payload_tag.
read_native_receipts() now takes payload_tag for the metadata row check
while receipts keep binding the attempt ref. The existing fixture used a
plain vX.Y.Z tag for both, which is how this stayed hidden; the new test
stages attempt-ref receipts with payload-tag rows.
The Microsoft Store step is no longer a release gate:
- stable-store runs with continue-on-error, and stable-phase-result no
longer requires it, so a failed submission leaves the run green and
'release.py publish' available.
- On failure it prints a warning and a run-summary section with the exact
Store .msixbundle URL to upload in Partner Center by hand.
- The publication pass turns a failed or unreadable Store check into a
warning instead of failing after the release has moved.
Also covers a camelCase (REST-shaped) submission (review).
rc.13-v0.21.7 failed 'Submit the verified Store package' with
'unreadable submission: {"Id": ..., "ApplicationCategory": ...'.
msstore submission get serializes the .NET DevCenterSubmission property names
(Status, TargetPublishMode), not the REST API's camelCase, so the in-flight
check never found 'status' and the Manual-publish write added a second
'targetPublishMode' key the CLI would ignore. Both now use the key as the CLI
spells it. The test fake emits the CLI's casing; with it main fails 5 tests
with the production error.
rc.12-v0.21.7 failed 'Submit the verified Store package' with
StoreError: unreadable submission status: ''.
store.submit parsed 'msstore submission status' as JSON. For a packaged
(MSIX) app that command prints a Spectre markup line ('Submission Status =
...') plus status tables, never JSON, and prints nothing when no submission
is pending (msstore-cli Commands/Submission/StatusCommand.cs). The first real
Store submission therefore always failed. 'msstore submission get' serializes
the submission as JSON (GetCommand.cs) and is already what submit reads after
the upload, so the in-flight check reads that instead.
The test fake now answers 'status' the way the CLI does; with it, main fails
5 tests with the production error.
Review findings on the abandon prompt:
- The prompt approved one ref but `abandon()` re-read the remote and cleared
every outstanding attempt of that version, so a concurrent operator's
replacement attempt could be cancelled and its draft deleted. The remote
is now read again after the answer, and anything other than the same single
attempt is refused; only that attempt is abandoned.
- Abandonment ran before the ancestry, final-tag and draft-body checks, so a
yes could destroy the old attempt and then refuse the replacement. Those
checks now run first; the prompt is the last step before the claim. The
attempt number ignores markers, so abandoning does not change the tag.
- The prompt test now varies stdin and stdout separately and asserts `input`
is never called unless both are terminals.
`release.py release` refuses while any attempt is outstanding and prints
the abandon command for the operator to copy. When stdin and stdout are
both terminals it now asks `Abandon <ref> and cut a new attempt? [y/N]`
right after that guidance. Yes goes through the same `abandon` as the
subcommand (cancel runs, delete the draft, write the marker) and the cut
continues as the next attempt; the default is no, and anything
non-interactive still refuses as before.
The confirmer and `inspect` are parameters of `release()`, not globals,
so only the CLI entrypoint opts in.
- Retiring a preview channel now requires the stable head to carry every
package the preview ships. An Intel preview could otherwise be pinned to an
arm64-only stable head it has no package in, and its client would fail to
resolve the retirement for good.
- The docs and release.py's next-steps text said Intel Macs stay on the last
Intel release's feed. They do not: promote rewrites the one shared macOS
stable feed with the arm64 leg, so Intel Macs get no update until a release
opts Intel back in. Only the Termux APT channel stays on its previous release.
- New scripts/releases/optional_platforms.py holds the claim flag, jobs,
handoff, target and smoke of each opt-in platform; stable.py, channel
releases, release artifacts and the entrypoint read from it. stable.py is
back under 1,000 lines. No behavior change.
- Test: a macOS Intel receipt against a baseline without Intel plans no
upgrade arm (baseline=none); against a baseline with Intel it plans one.
Stable releases no longer build the macOS Intel (darwin-x64) desktop bundle
or the Termux APT package unless the release is cut with an opt-in flag.
Intel macOS is not supported on main, and its macos-15-intel bundle leg
takes 76-88 minutes, which is most of the release's wall time.
python scripts/release.py release --commit SHA --include-macos-intel
python scripts/release.py release --commit SHA --include-termux
The flags are written into the signed claim as includeMacosIntel and
includeTermux, like skipBundles/skipTests. admit emits them, and the
candidates-darwin-x64, transitions-darwin-x64 and macos-packaged-x64 jobs
(Intel) and candidates-termux and termux-checks (Termux) run only when
their flag is on. The gate requires those jobs to report skipped when the
flag is off; the skip flags keep their meaning.
Claims cut before these keys existed (every rc.* tag so far) have neither
key. validate_claim treats a missing key as false and still refuses a
non-boolean value or any unknown key, so sequencer discover/reconcile keep
working over the historical tags.
Without the opt-in platforms the candidate manifest, receipts, mac feed,
protected R2 head, APT promotion and downloads table all work with those
rows absent: the macOS feed carries the arm64 leg alone, the APT indexes
are left on the previous Termux release, and the unbuilt smoke is recorded
as skipped (a passed smoke for an unbuilt target is refused). A release
that opts Intel back in after one that left it out has no Intel baseline,
so its Intel upgrade arm is skipped like a first release.
Preview channels, canary and the PM bundle artifacts are unchanged.
rc.11-v0.21.7's Windows x64 MSIX smoke failed with 'Artifact checksum
mismatch' on the 2.6 GB package. The stored object was correct (re-downloaded:
size and SHA256 match the receipt exactly), so the runner received bad bytes.
_download_url retried HTTP and connection errors but raised on a completed
body that failed its hash, so one bad transfer failed the job and its
dependents until a whole-run rerun.
A size/SHA256 mismatch now counts as a bad transfer and is fetched again
within the same three attempts. A stored object that really differs from its
receipt fails all three the same way, and the verified destination is never
replaced.
Review on #136434 (@jquesnelle): discover() sorted records by version only,
so attempts inside a version kept git ls-remote's NAME order (rc.10 between
rc.1 and rc.2). reconcile indexes records by version with the last one
winning, so once a version reached rc.10 it handed rc.9 (burned, no release)
to publish_attempt, which refuses with "Final tag release database ID is
invalid". v0.21.7 is at rc.9, so its next cut is exactly rc.10.
- Sort by (version, attempt).
- The test fakes list refs sorted by name, as git does; insertion order hid this.
- rc.10 test (fails without the sort), and a same-commit retry test (rc.1
abandoned, rc.2 re-cut at the same commit wins).
Every Stable Release Publication run since v0.21.6 has crashed in
sequencer.discover():
ValueError: v0.21.6 points at a different commit than rc.1-v0.21.6
v0.21.6 took four attempts: rc.1-3 were abandoned (each has its
abandoned-rc.<N> marker) and rc.4 won and became v0.21.6. discover() tested
"does the final tag exist" before "is this attempt abandoned", so for rc.1 it
compared v0.21.6's commit (rc.4's) with rc.1's and raised; the abandoned
branch was only reachable for versions that never got a final tag. Any
version whose winner was not rc.1 broke every later reconcile, including the
next release's publish.
The abandoned marker is now checked first: an abandoned attempt is burned
whatever its version became. A NON-abandoned attempt whose final tag points
elsewhere still raises.
A release that skipped bundles never writes the stable baseline pointer,
so its absence is normal at any point, not only before the first bundle
release. Drop the final-tag scan. A 404 on the default pointer now always
plans no upgrade arm. A supplied baseline-manifest that does not exist
still blocks, because that is a bad input.
No stable release has shipped signed packages yet. Every final tag so far
skipped bundles or predates the claim pipeline. So
releases/stable/release-candidates.json does not exist, and the three
"Pin actual OLD and NEW signed packages" jobs refused with "No published
stable package baseline".
transitions now treats a 404 as a first release only when no final stable
tag records a candidate manifest and no baseline-manifest input was given.
It still validates the receipt, emits empty matrices and baseline=none.
The packaged upgrade arms skip on that output. The gate accepts their skip
only when their own planner succeeded with baseline=none, so a missing
baseline after a real bundle release still blocks.
Maciej Lisiewski