Symptom: LSPClient starts every language server in its own session (start_new_session=True) so a
killpg aimed at the server can never reach Hermes. That also took the server's descendants out of
Hermes's kill domain. The server itself exits on stdin EOF when Hermes dies, but a descendant that
neither holds nor watches that pipe (a wrapper's background child, a SIGTERM-ignoring worker)
outlived a SIGKILLed / OOM-killed Hermes forever. The atexit hook and the idle reaper both live
inside the owner, so neither runs on that path.
Change: _spawn hands the server's process group to the per-process owner-death supervisor
(own_process_groups) and records it in the spawn ledger (register_child "lsp-server").
_cleanup_process hands the group back only once nothing in it is left, so a member that survived
teardown stays covered and still dies with Hermes. No new process: the supervisor is shared with
every other owned child. The atexit hook stays: it runs the LSP shutdown/exit handshake on a
clean exit, which the supervisor's TERM/KILL does not replace. Windows path unchanged.
Measured on Linux (Linux), kill -9 of a real owner running LSPClient, survivors of the server
tree after 30 s:
mock server with a SIGTERM-ignoring child (slow_tree), 3 trials: main 1/2 alive every trial,
branch 0/2 every trial (gone by +5 s).
basedpyright, pylsp, typescript-language-server (4 procs), gopls, rust-analyzer: 0 on main and
branch (they exit on stdin EOF, and their helpers die with them).
Tests (tests/agent/lsp/test_lsp_owned_server.py): owned on spawn + released on shutdown, a
surviving member keeps coverage, a real kill -9-the-owner end to end, and a live-owner control.
Removing the own_process_groups call turns 3 of them red; making the release unconditional turns
the coverage test red.
Each `hermes update` now spends at most 60 s on the on-demand packs a
partial clone piles up (#129712), and every finished unit is kept, so a
large checkout converges over a few updates instead of blocking one
update for 20+ minutes:
1. Erase on-demand packs (empty .promisor marker, older than an hour)
that hold no commits. Their objects are copies of what the promisor
remote serves; git fetches what it needs again. This frees disk,
which the old gc fold never did. A pack holding commits is kept
because a ref may point into it; a pack git cannot classify in time
counts as holding one.
2. While more than 50 promisor packs remain, merge the smallest few
with `git pack-objects --stdin-packs` (cost proportional to those
packs, no reachability walk), marker first and index last so the
merged pack is a promisor pack from the moment git sees it.
Verdicts and a merge batch size that halves after a timed-out merge are
remembered in .git/hermes-pack-tidy.json, so no update repeats work an
earlier one could not finish. bounded_probe_run gains `input=` for
stdin-fed git commands.
A console-less parent (pythonw backend, detached daemons) allocates a visible
console per bare spawn, so Windows startup flashes ~7-8 console windows
(#117781) and the GPU statusbar flashes two more every 5s (#101895). Threads
creationflags=windows_hide_flags() through every hermes_cli spawn site the
desktop reaches: gitlock (tasklist probe + git plumbing), the scheduled-task
PowerShell probe, the updater's _git_run/_no_prompt_git_kwargs/_GIT_TEXT_KW
runners, the safe-directory config probe, and consolidates the statusbar's two
nvidia-smi reads per poll into one cached hidden query. The statusbar also
stops polling while the document is hidden and resumes on visibilitychange
(#120262).
Supersedes #117805, #117833, #102047, #120285 (thanks @JoaoMarcos44, @Finn763,
@monerostar, @szzhoujiarui).
scripts/ci/list_os_marked_tests.py matched the literal lane word inside a
platforms() string, so 80 files gated platforms("posix") never reached the
macOS lane ("posix = linux or macOS" was false for CI), and "any" files
reached none. The selector now resolves specs the way the conftest gate
does (posix ⊇ linux+macos, any ⊇ all, "not X" admits the rest) and only
looks inside mark.platforms(...) calls, dropping a false positive whose
only "macos" was inside a generated-file string.
scripts/run_tests_parallel.py still grepped the retired linux_only/
macos_only/windows_only names, so its "N files SKIPPED on this host" note
had been silent since the migration. It now shares the selector's
resolver and names the spec and the lane(s) it runs on.
Restore the platforms("windows") mark that
test_suppress_platform_ver_console_stubs_syscmd_ver lost in the
windows_only migration (its docstring still declared it); it passed
vacuously on Linux and was deselected on the Windows lane.
Resolved toward the branch: PM provisions uv/python (main's install.ps1 uv-shim
salvage + its test and workflow steps dropped), the shim re-exec stays retired,
package.json carries no electron-builder block (afterExtract identity stamp wired
into electron-builder.config.cjs instead; after-pack.mjs keeps signing only),
Desktop workspace-deps helpers stay retired. Main's scratch-dir bootstrap
(export_scratch_tmp_env) is taken and re-run after profile resolution.
bounded_probe_run now spawns through local_runtime.processes.spawn_server on
Windows (CREATE_SUSPENDED + AssignProcessToJobObject on the real handle), which
a fake Popen cannot satisfy, so the probe fell to its fail-open "" and the
contract assertions never ran (windows-only job on 0eae148f0d). Fake the
container instead; the kwargs the probe hands it are the contract under test.
Competing installers and checkout-local venv assumptions bypassed PM
selection, install consent, and generation lifetimes. Route consumers
through PM and installation-bound launchers. Refresh source launchers
before obsolete Python entries can be collected.
Remove Node, browser, and CUA acquisition engines, obsolete venv-holder
handling, detached sync, and unused PM APIs. Keep historical updater
exports inert and preserve external tool ownership and native integration.
Share product freshness and prepared inputs across builders. Align plugin
admission, Docker provisioning, setup instructions, and behavioral tests.
Verified targeted Python and JavaScript tests, desktop and web typechecks,
scoped lint, real product builds, and the Docker frontend smoke test.
The missed post-setup test cleanup is included and verified.
Native Windows/macOS execution, full Rust compilation, and the complete
repository suite remain unverified. Historical compatibility requirements
were preserved and extended, not fully rescanned.
Merge upstream b1f003e186 while preserving PM runtime ownership and
Python 3.14 worker startup, Windows signing, and macOS wait recovery.
Keep retired runtime modules deleted. Port upstream updater preflight
checks into the checkout strategy and preserve live build logging.
Carry checkpoint filename handling and process recovery into the current
module layout. Regenerate locks and adapt incoming platform test markers.
Focused Python and JavaScript tests, desktop and root-test typechecks,
conflict-path lint checks, lock validation, and retired-import checks pass.
The full test suite and packaged release builds were not run.
`hermes_cli/banner.py::_git_run` is the shared spawn path for every banner and
passive update-check git probe (rev-parse, rev-list, remote get-url, ls-remote).
It carried the UTF-8 text contract but not `creationflags=windows_hide_flags()`,
so on Windows each probe run from a GUI-hosted backend (desktop-spawned
`hermes serve`, `tui_gateway` import kicking off `prefetch_update_check`) flashes
a console window. Every other short-lived helper in `hermes_cli/` already passes
the flag; this brings the banner path in line.
Surfaced by CI on this PR: the stray `git ... origin` spawn from the prefetch
daemon landed in `test_env_probe_run_hides_console_window`'s process-wide
`subprocess.run` capture and tripped its call-count assertion. The test now
scopes its assertion through the module's `_spawns` helper like its siblings,
so an unrelated daemon spawn cannot fail it (the production fix is what makes
that stray spawn carry the flag in the first place).
A/B: base `_git_run` -> no `creationflags` kwarg; fixed -> 0x08000000.
Introduce the pm store: a unified, hash-verified package store that
replaces lazy_deps and the old installer's ad-hoc tool downloads.
Store tools are provisioned on PATH (ffmpeg, node/npm via pinned uv),
with a resumable 8-way downloader, verify() returning failure reasons,
and adopt() made EPERM-safe. chromium ships in the payload for every
target. The 3600-line install.sh is replaced by a staged bootstrapper
(heavy deps are pm's job after this); setup-hermes.sh, Dockerfile and
nix pin tables are rewired onto the store. Old install-script tests,
lazy_deps/managed_uv/build_info, and the ps1/bash installer test
batteries are removed with the machinery they tested.
Rebuilt from ethie/pm onto upstream/main (ac6c8028e0) after the
utf-8-sig sweep. 16 hot files (main also churned them) hand-merged:
platform adapters, main.py, electron/main.ts, tui_gateway/server.py,
cua_backend, installer-tests workflow, install.sh (full rewrite),
setup-hermes.sh, plugins doc.
No shims, one marker system. The full mechanical sweep:
- 165 test files converted: every @pytest.mark.<os>_only decorator and
pytestmark assignment is now platforms("<os>"). Docstring/comment
prose mentioning the trio rewritten to the platforms() vocabulary.
- conftest: _OS_MARKS, the legacy skip loop, the double-mark reject for
the trio, and the selector-tagging shim are deleted. The single
platforms() gate does all host gating; _reject_contradictory_platform_
marks replaces the old reject (one platforms() marker per test — a
module-level pytestmark stacked on a per-test marker is the historic
skipped-everywhere-green-everywhere failure and stays a hard error).
- pyproject: only the platforms marker is registered.
- list_os_marked_tests.py: rewritten for the single vocabulary — takes
a platform name (linux/macos/windows), matches quoted platforms(...)
specs including negated and any-of forms, exits nonzero on empty
selection. Its test file rewritten to match (bare identifiers must
not match; the spec must appear inside the string literal).
- tests.yml macOS lane: marker: macos feeds the lister; the pytest
selection is -m "platforms and not integration" — the marker name is
the selector, the conftest's per-test host skips are the gate.
- test_os_marker_gating.py rewritten for the new reject (the old file
tested deleted machinery).
- AGENTS.md / CONTRIBUTING.md updated to the single vocabulary.
Verified: full-tree compile; marker unit tests (36); lister tests;
macOS-lane simulation (21 files, 180 collected / 429 deselected on a
windows host); broad xdist slices through scripts/run_tests.sh
(367 + 1674 passed, zero refactor-attributable failures — the 6 red
tests in the second slice fail identically with the changes stashed).
Full rewrite of test_browser_npx_warmup.py for the Popen-based
credential-scrubbing, PATH-propagation, and process-tree-kill rework:
argv shape, env scrubbing, PATH merge for managed-only npx, POSIX
process-group creation, Windows CREATE_NEW_PROCESS_GROUP, whole-tree
kill (not just the PID) on timeout with a bounded post-kill drain, and
_kill_process_tree's own POSIX/Windows/failure paths directly.
Also fixes test_windows_subprocess_no_window_flags.py's matching
regression test, which still mocked subprocess.run and a shutil.which
signature that didn't accept the path= kwarg _resolve_npx_bin's
extended-path rung now passes; its creationflags assertion becomes a
bitwise check since Windows now ORs CREATE_NEW_PROCESS_GROUP in
alongside the console-hiding flag.
warm_agent_browser_npx_cache() spawns a resolved npx.cmd via
subprocess.run with a list arg and no shell=True, which Windows still
routes through cmd.exe. Without creationflags=windows_hide_flags(),
that can flash a console window during hermes update/doctor --fix,
same as the existing agent-browser subprocess spawn elsewhere in this
file already guards against.
Adds a regression test to the cross-cutting Windows no-window-flags
audit suite so a future refactor can't silently drop the flag again.
_spawn() now uses subprocess.Popen + communicate() instead of
subprocess.run(); the windows_only creationflags assertion mocks Popen
accordingly and additionally pins that the POSIX-only process_group
kwarg never reaches a Windows spawn.
many tests patched sys.platform or a module's _IS_WINDOWS flag, then
ran on linux ci. the patch selects the branch under test, but the host
does not have the behavior the branch exists for. the test proves the
patch, not the platform. some gated assertions never ran on any host.
this commit adds three markers: linux_only, macos_only, windows_only.
a conftest hook skips a marked test on the other hosts, with a clear
reason. no test fakes a host now. two documented fakes remain
(android/termux, freebsd) because no ci runner exists for them.
each fake site got one of four treatments:
- gate it: the real host supplies the platform; mocks cover real
dependencies only, never host identity
- patch the module's own probe when the subject is the probe's consumer
- assert against the real host when the fake stood in for any non-x host
- delete the patch when it set the value the host already has
bare skipif(sys.platform != ...) guards became markers too. the lane
model skips these on linux and never imports them on windows, so they
ran on no host. platform parametrize tables are now one marked test
per os.
running on real hosts found real errors: a chrome-sandbox failure in
test_gui_command that main hides, and two windows failures fixed here.
the agents.md testing section now documents the policy.
cron/scheduler.py deliberately applies utf-8/replace only on Windows via
popen_kwargs (non-Windows keeps locale default per its test contract) —
drop the sweep's unconditional inline kwargs there. Update the gateway
force-kill kwarg snapshot for the new guard.
Follow-up to the #67690 salvage (@m4r13y). The PR's tools/env_probe.py
hunk was written against the old capture_output=True _run(); #67964/#67999
rewrote _run to temp-file capture on July 20, so that hunk no longer
applied — but the rewritten _run still lacked creationflags and kept
flashing one console per probe (~5 per kanban worker start) from
windowless parents. Re-implement the one-line fix against the current
shape: creationflags=windows_hide_flags() on the temp-file subprocess.run,
preserving the #67964 grandchild-can't-wedge-the-pipe contract.
Also add the tests the PR didn't ship, in
tests/test_windows_subprocess_no_window_flags.py:
- env_probe._run passes CREATE_NO_WINDOW and keeps temp-file (non-PIPE)
stdout/stderr + DEVNULL stdin
- lazy_deps uv install / pip --version probe / pip install fallback /
ensurepip bootstrap all pass CREATE_NO_WINDOW
- suppress_platform_ver_console: POSIX no-op (platform._syscmd_ver
untouched, win32_ver() still returns), and simulated-Windows stubbing
(echo stub installed, idempotent, never raises)
Regression tests for the #47971 salvage: the LSP language-server spawn
must pass windows_hide_flags() creationflags while keeping PIPE stdio
and start_new_session, and the npm/go LSP auto-installer subprocess.run
calls must carry the same hide flags with DEVNULL stdin and
capture_output intact.
Mocked-subprocess tests asserting creationflags == CREATE_NO_WINDOW for
each path salvaged from PR #56877: tui_gateway cli.exec / shell.exec /
quick-command dispatch, the CLI quick-command exec handler, and the
Copilot ACP + Codex app-server Popen transports (pipes asserted intact).
Verified: all 6 fail with the fix reverted, pass with it applied.
subprocess.run(["git", ...], timeout=...) deadlocks on Windows: run()'s
post-timeout cleanup calls an unbounded communicate() after killing git.
Killing the PATH-resolved launcher can leave a suspended descendant git.exe
holding duplicates of the captured stdout/stderr handles, so the pipes never
reach EOF and the reader-thread join blocks forever — leaking a process +
two reader threads per fired timeout (the accumulating git.exe load behind
Windows Defender CPU spikes).
Two fail-open probe call sites had this identical flaw:
- tui_gateway/git_probe.py::run_git — on the Desktop agent-build path
(_start_agent_build -> _session_info -> branch() -> run_git), where the
hang turned an optional branch label into "agent initialization timed
out" (#68609).
- agent/coding_context.py::_git — hangs the agent turn inside
build_coding_workspace_block under an ACP host (#66037).
Consolidate both onto one shared bounded_git_probe() in
hermes_cli/_subprocess_compat.py (both files already import from there, so
no new import surface):
- explicit communicate(timeout), then on ANY failure a tree-kill —
proc.kill() AND, on Windows, best-effort taskkill /T /F so the suspended
descendant that holds the pipe writers dies too — plus a bounded 1s
post-kill drain; if the pipes are still held they're abandoned (the
orphaned reader threads are daemonic and cost nothing).
- fail open to "" on every path: spawn error, timeout, kill() raising
(access denied / already reaped — a raise inside the except handler
previously escaped the contract), and non-timeout communicate() failures
now also terminate the child instead of leaving it running.
- the taskkill spawn can't re-enter the deadlock class: it captures no
pipes (DEVNULL), so its own timeout cleanup has no reader threads to join.
Normal-path spawn contract is preserved byte-for-byte: PIPE/PIPE/DEVNULL,
text + utf-8 errors="replace", hidden-window creationflags on Windows only,
nonzero returncode -> "". Each call site keeps its own timeout (1.5s / 2.5s).
Supersedes #68622 (Sora-bluesky — git_probe fix + tree-kill) and #66038
(iamwongeeeee — coding_context fix), folding both into one shared helper so
the two sites can't drift and every timeout tree-kills the descendant. Tests
consolidated onto the helper, incl. the previously-missing assertion that a
Windows timeout escalates to taskkill /T /F.
Co-authored-by: Sora-bluesky <sora.bluesky.dev@gmail.com>
Co-authored-by: iamwongeeeee <wykim777@naver.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
These tests patch `<module>.subprocess.run`, which is the shared `subprocess`
module singleton, so the patch is process-wide. Importing `tui_gateway.server`
runs `prefetch_update_check()` at import time, spawning an unnamed daemon thread
(`Thread-N (_run)`) that shells out to `git ... origin` (`text=True, timeout=5`).
That call races the test and lands in the captured list, intermittently failing
`test_tui_gateway_fuzzy_file_listing_hides_git_windows` with either
`KeyError: 'creationflags'` (the daemon's git call has no creationflags) or a
call-count mismatch (3 git calls captured, not 2). It only reproduced under the
parallel test harness because of the extra concurrency/timing.
Filter captured calls to the distinctive argv tokens of the call under test
(`--show-toplevel`, `ls-files`, `branch --show-current`, `diff`, `rg`,
`taskkill`) and read `creationflags` via `.get`, mirroring the existing
hardening on `test_gateway_pid_scan_hides_wmic_and_powershell_windows`. The
production code is unchanged; this is a test-isolation fix.
test_gateway_pid_scan_hides_wmic_and_powershell_windows flaked once in CI
(slice 7/8) with 'KeyError: creationflags' while passing 15/15 under exact
CI-parity locally. The positional 'kwargs["creationflags"]' indexing raises
a bare KeyError the moment any stray subprocess.run call is captured, masking
the real contract. Filter captured calls to the two intended Windows console
spawns (wmic + PowerShell fallback) and assert each is windowless via
.get('creationflags'); a leaked/extra call now surfaces as a readable
len-mismatch with the full captured list, not a cryptic KeyError.
windows_hide_flags() already returns 0 on POSIX (and creationflags=0 is
the no-op default there, exactly how server.py::_list_repo_files does it),
so drop the IS_WINDOWS import + ternary/one-use-dict gating and just pass
creationflags=windows_hide_flags() directly. Tests lose the now-pointless
IS_WINDOWS monkeypatch.
The #54236/#54417 backend git/gh sweep routed git_probe, the repo-file
picker, coding_context, context_references, copilot_auth, and the gateway
process scans through CREATE_NO_WINDOW, but two sibling spawn legs that
also run inside the console-less desktop/gateway backend were missed:
- tools/checkpoint_manager.py `_run_git` (and the one-shot `git init
--bare` in `_init_store`) — when checkpoints are enabled, every
file-mutating turn fires multiple bare `git` calls (status, add,
write-tree/commit-tree, update-ref). Spawned from a parent with no
console (Electron spawns the backend with windowsHide → CREATE_NO_WINDOW),
each one allocates its own conhost window → a flurry of terminal popups.
- tools/skills_hub.py `GitHubAuth._try_gh_cli` — `gh auth token`, the same
bug class as the already-fixed copilot_auth gh probe.
Route both through `windows_hide_flags()` (no-op on POSIX), matching the
established per-site pattern. Tests added to
tests/test_windows_subprocess_no_window_flags.py.
main (cb982ad99) wired windows_hide_flags() into the auxiliary git/gh/wmic/
bash/powershell/taskkill legs but left two it didn't reach, plus the Electron
backend-launch leg it explicitly deferred. Cover them the same way:
- apps/desktop/electron/main.cjs: getNoConsoleVenvPython resolves the BASE
pythonw.exe instead of the venv Scripts\pythonw.exe shim, which re-execs a
console python.exe and flashes a conhost the desktop backend can't suppress.
Both backend creators put the venv site-packages on PYTHONPATH so imports
still resolve under the base interpreter. (main's commit said this Electron
leg "needs a Windows-tested change of its own".)
- tools/tts_tool.py, tools/transcription_tools.py, plugins/platforms/discord:
ffmpeg conversions (voice notes / TTS / STT) via windows_hide_flags().
- plugins/platforms/whatsapp: netstat + taskkill bridge-port cleanup via
windows_hide_flags().
All no-ops on POSIX. Tests assert the base-pythonw preference and the ffmpeg
legs pass CREATE_NO_WINDOW.
kshitijk4poor