comm is cut at 15 bytes and can end mid-UTF-8 sequence (multibyte exec
name, prctl/libuv titles); read_text raised UnicodeDecodeError, the stamp
fell back to a wall-clock birth and was compared exactly against ticks,
so after a clock step the launcher shell was signalled and its entry
removed. Parse /proc/<pid>/stat as bytes; on a /proc host a read failure
now fails closed (capture: no lineage, spare everything; check: spare
the pid) instead of mixing sources.
Reviewer probe with a non-UTF-8 comm, btime shifted +5/-5/0: before,
shell TERMed and entry removed at +5 and -5; after, kept in all three.
The detached pass got only the launcher's pid and ppid and walked their ancestry itself, ~40 ms
later (more under load). When both had already exited, the user's shell sitting in an idle
entry was no longer anyone's ancestor: it was TERMed and the entry deleted, where the inline
prune on main keeps both.
- The launcher computes its ancestry as (pid, create_time) pairs before it spawns the pass and
hands them over; the pass spares a passed pid only while it still has that create time.
- A launcher that cannot compute it (no psutil, a process refusing inspection) hands over
nothing: the pass signals no process and keeps every entry a live process works in.
- A process check that raises deletes nothing that pass (logged), instead of deleting with no
ownership check; prune_scratch_dir's only production caller is now the detached pass.
- A failing pass leaves one line in scratch-prune.log (its stderr is /dev/null).
- tests: an autouse conftest fixture stubs spawn_prune per test, so a test that reloads
hermes_constants no longer starts a real detached pass.
On a daemon thread the hourly prune died with every short-lived `hermes` command: the
stamp was already spent, so a run of quick commands could eat every hourly pass and
reclaim nothing, and a pass cut inside rmtree left a remainder the idle rule kept for
another 24 h.
The process that is due a pass now keeps the once-per-process flag and the hourly stamp
and hands the pass to a detached child (start_new_session on POSIX, the repo's Windows
detach flags with the no-breakaway retry). The child runs `python -I` on the launcher's
own sys.path with the scratch path the caller resolved, and calls prune_scratch_dir
directly: no hermes_bootstrap, no boot hook, no second prune, no re-resolved home. The
pass is the unchanged origin/main pass: same idle rule, re-check, reap and audit log.
The launcher's process tree stays the pass's own lineage after the child is reparented
(captured at child start): a shell working inside an idle entry that ran the Hermes
command still keeps that entry, as it did when the prune ran inline. A failed spawn is
logged at debug and dropped; the next process past the hourly mark retries.
The test conftest marks the once-per-process prune as done so no test starts a real
child (or feeds one to its Popen mock); the prune tests reset it.
Tests: the two daemon-thread tests become (1) the boot hook and the prompt build hand the
pass to exactly one child for the resolved (and routed) scratch dir, red on main; (2) a
real short-lived process exits at once and its detached pass still removes the routed
home's idle entry, leaving the launch home alone, red on the daemon-thread head.
get_scratch_dir(prune=True) ran the once-per-process idle scan and rmtree inline. Its first
caller in nearly every process is the boot hook (hermes_bootstrap -> export_scratch_tmp_env ->
apply_scratch_tmp_env), so `hermes`, the TUI backend, serve, gateway and every Hermes child sat
on the prune before doing anything: a 34 GB idle entry held one CLI start for 13 s.
The prune=True path now schedules the same pass on the scratch-prune daemon thread (the
background helper from the prompt-builder fix), so the boot hook, the post --profile re-export,
the child-env builders (apply_subprocess_home_env) and the system prompt all share one
non-blocking path; the prompt builder goes back to a plain get_scratch_dir(). The directory is
still created and permissioned before the call returns, so TMPDIR is valid at once. Stamp,
once-per-process flag, 24 h idle rule, reap and audit log are unchanged; the thread gets the
path the caller resolved.
A process that exits mid-prune kills the thread with the hourly stamp already touched: entries
it never reached go on the next hourly pass, and an entry cut inside rmtree waits 24 h (the
unlinks freshen its directory mtimes), the same as a Ctrl-C during the old inline prune.
Tests: the 3 prompt-builder tests become 2 invariants (boot hook + prompt never wait and prune
once on the resolved dir; the thread prunes the routed home's dir), both red on main.
The host-hints section only needs the scratch path. It called get_scratch_dir() with the default prune=True,
so the first prompt build in a process ran the once-per-process scratch prune synchronously. On macOS under
launchd and on Windows the boot hook returns early (the OS already sets TMPDIR/TEMP), so that first build
pays the whole prune, sometimes inside an automatic compaction boundary while a user turn waits.
Ask for the path only and run the same once-per-process prune on one daemon thread, against the path the
caller resolved (threads do not inherit the profile override contextvar). The prompt text is unchanged.
git clone --local hardlinks pack files; git freshens a pack's mtime in the
source repo whenever it re-writes an object the pack already holds, and the
shared inode carries that mtime into the scratch clone, so the 24h-idle
pruner saw the clone as active forever. On one host this kept 4 entries
(100 GB of their own data) alive for 5 days. Hardlinked regular files no
longer count as recent activity, the same reasoning that already excludes
symlinks.
Independent review on #140267:
- The owning home was derived from the resolved scratch path, so a
cache/scratch symlinked onto another disk read the wrong registries and
reaped a registered worker. Derive it from the configured path.
- A ledger mcp-helper whose recorded spawner is provably dead counted as
owned forever, keeping its entry and skipping the reap main used to do.
Apply the same spawner_is_dead rule reap_orphaned_mcp_helpers uses;
long-lived entry points that outlive their launcher stay owned.
Review (Greptile): _own_lineage() was skipped before the cwd check, so a
live session inside an idle entry that launched another Hermes command
never reached the keep guard and its entry could be deleted under it. The
lineage is still never signalled, but its cwd now keeps the entry. Test
uses the posix platform marker (macOS lane selects it) and wait_until
instead of sleep loops.
The prune TERM/KILLed every same-user process whose cwd sat in an idle
scratch entry, including a background terminal process or CLI session
Hermes itself started that was computing, not writing.
Before signalling, each candidate is checked against the registries
Hermes already writes: the machine spawn ledger (register_self /
register_child) and the process registry checkpoint (processes.json),
with the recorded start time guarding against pid reuse. A registered
live process, or a descendant still parented under one, is never
signalled; every idle entry it works inside is dropped from the doomed
list and recorded as "kept ... live Hermes process pid=N (name) works
inside". Orphans (a dead lane's headless browsers, the grandchild of an
exited shell, now reparented away) are reaped exactly as before.
A full-tree size walk between the last activity re-check and the delete let a
write landing during that walk be destroyed. Collect size first; nothing sits
between the re-check and rmtree now. Docs: only entries kept by the re-check
are audited.
The original rescue test writes a fresh file directly in the entry, which also
bumps the entry directory's own mtime — so a shallow re-validation that never
looked below the top level would pass it. This regression holds every directory
mtime ancient and overwrites only an existing file three levels down (an
overwrite moves the file's timestamp alone), so a top-level-only re-check sees
idle and deletes real work while the descending walk rescues it.
Red against a shallow (lstat-only) subtree_touch_state mutant: removed == 1 and
the deep payload is destroyed. Green on the real walk; scratch battery 29
passed, 2 platform skips; ruff clean.
After rmtree(ignore_errors=True), the removal-confirmation gate used
os.path.lexists(entry), which swallows OSError from lstat and returns
False: an entry that stayed (rmtree left residue) but became unreadable
was counted and recorded as removed — removed_return=1 with the payload
still on disk, a false 'removed …' record in scratch-prune.log, and the
'count actually removed' contract broken (@andrexibiza, P2 on #134173 —
reproduced live with his exact interleaving before the fix).
Only an explicit lstat proves absence now: FileNotFoundError confirms
gone (counted), a successful stat is residue ('could not fully remove',
not counted), any other OSError means the removal state is unknown
('could not confirm removal', not counted) — the same invented-absence
rule the vanish branch already enforces, carried through to the
completion gate.
Regression: test_postremoval_unreadable_entry_is_not_counted_as_removed
(red at the previous head) leaves residue, flips only the post-removal
stat to PermissionError, and asserts count 0, payload preservation,
and no 'removed' record. The reviewer's public repro passes at this
head. Scratch battery: 28 passed, 2 skipped; the 8 local.py caller
tests stay green; ruff and check-windows-footguns clean.
subtree_touched_since() is a fail-closed keep predicate: its OSError paths
return True, so the mid-prune re-validation recorded a kept entry as
'touched since selection' even when nothing was written and the re-scan
merely failed to read the entry — a misleading claim operators cannot
distinguish from verified recent activity (@andrexibiza, P3 on #134173).
- subtree_touch_state(): tri-state walk (touched / scan-failed / idle);
subtree_touched_since() stays as a thin fail-closed wrapper (!= idle),
so selection and the local.py temp-cache caller keep their exact contract.
- The kept-entry record names what happened: a genuine touch still logs
'rescued … touched since selection, kept'; a failed re-scan now logs
'kept … activity re-scan failed, left untouched'.
- Only FileNotFoundError counts as vanished: lexists swallowed
PermissionError, so an unreadable-but-present entry was reported as
'vanished since selection' — the same invented reason, fixed with it.
- Regressions: test_post_selection_scan_failure_keeps_entry_with_honest_audit
(3 probes, red at the previous head) checks payload preservation and honest
wording both, and reclaim-once-readable; a vanished-state test pins the
third state. 27 passed, 2 skipped across the scratch battery.
Port of 3f72fd9c8e from #132455 onto the merged audit-trail code (#133952):
the doomed list is a candidate list, never a verdict. Each entry is
re-validated with the same idleness walk (subtree_touched_since) immediately
before its deletion — after the worktree scan, so the re-check sits as close
to the delete as the loop allows — and anything touched since selection
(a resumed writer whose cwd is outside scratch, invisible to the reap) is
rescued, recorded in scratch-prune.log alongside the existing records, and
never counted as removed. An entry that vanished since selection is recorded
as vanished, not rescued. Rescued lines go through %r like every other
record, so a control character in a name cannot split or forge one.
Scope, stated honestly: this closes the selection-to-deletion window. A
write landing during the rmtree call itself (after re-validation) can still
be lost — that is the atomic-delete residual #132401's triage set aside as
needing a rename-based design; it is unchanged here.
Red/green proven on this head: test_midprune_resumed_write_is_rescued fails
at the main base (entry destroyed with the fresh write inside) and passes
here. Scratch battery: 23 passed, 2 skipped (Linux-only) on this host.
Refs #132401. Split requested at #133952 (https://github.com/NousResearch/hermes-agent/pull/133952#issuecomment-6021214057).
Salvages the F1 half of #132455.
Review of #133952 (ehz0ah) showed a legal POSIX name containing a newline was written across two lines of scratch-prune.log, so one record split and the tail of the name could pass for a record of its own. Paths and process names are now logged with %r, so control characters are written escaped and each record stays one line.
Refs #132401.
Review of #133952 (andrexibiza) reproduced two ways a deletion could leave no line in scratch-prune.log: a legal POSIX name that is not UTF-8 hit the strict utf-8 handler and the whole record was dropped, and when two prunes of one home rolled the log over at once, the second rename found the file gone and the record was dropped with the FileNotFoundError.
The handler now writes with errors=backslashreplace, and its rollover treats a source renamed away by a concurrent prune as already rotated and writes the record to the new file.
Refs #132401.
The idle prune deleted entries and TERM/KILLed processes with one aggregate INFO line, invisible at the default WARNING level and absent entirely at boot, before setup_logging() runs. It also counted an entry as removed even when rmtree (ignore_errors) left it behind.
Each removed entry (path, size), each signal sent (pid, name, cwd) and each entry that could not be removed now gets a line in <home>/logs/scratch-prune.log, through a RotatingFileHandler of its own (fixed 5 MB x 3, as tool_calls.log), so the boot-time prune is recorded too. The handler belongs to one prune and is closed when it ends: no process holds the file open between prunes (an open handle blocks rollover on Windows), and prunes of two homes never share it. Only entries that are really gone are counted. A process that exited but is not yet collected (zombie) is not sent KILL.
Refs #132401. Salvages #132455.
Co-authored-by: Justin Sharpe <2106314+justinsharpe@users.noreply.github.com>
Change-detectors, tautologies, source-reading tests, redundant duplicates,
mock-echo tests and dead/unrunnable tests. Per-test rationale in the lane
ledger (category + reason for every removal).
`get_scratch_dir(home)` now forwards the home it resolved into the permission
policy, so a caller that already knows its home (boot scratch setup,
`hermes doctor` for another profile) reads the `.managed` marker from that
home instead of re-consulting `get_hermes_home()`.
Three `TestScratchDirPermissionPolicy` cases modelled the opposite shape: they
put the marker in `HERMES_HOME` and then passed an unrelated base
(`tmp_path/cache/scratch`, outside that home) to `get_scratch_dir`, so the
scratch dir under test was never the one the marker governed. Align them with
the contract the fix documents — the scratch dir's own home carries the
marker — keeping the coverage each case had (marker file, empty marker,
unreadable marker ⇒ pre-existing mode untouched).
Add the invariant that replaces the coupling they dropped: a marker in the
effective home must not exempt a *different* home's scratch dir from the
policy. Red on the parent test file, green with the aligned cases.
The macOS runner uses Bash 3.2, which rejects parameter case conversion.
Normalize the GHCR owner with portable tr and verify mixed-case input.
The macOS runner can clear setgid from a directory when chmod applies 2770.
Compare scratch permissions with the native chmod result while preserving all
bits the host accepts.
Tests: scripts/run_tests.sh tests/scripts/test_termux_build_driver.py tests/test_scratch_dir.py
Shell: bash -n scripts/termux/build_builder_image.sh
Deleting an idle scratch entry left two things behind. Headless browsers
started by a lane's e2e run kept running for days with a "(deleted)" cwd
(about 20 on one host; the process registry never knew them because they
were grandchildren of a shell that had exited). Repos whose linked
worktree lived in the entry kept a dangling registration until someone
ran `git worktree prune` by hand (10 in one repo).
The prune now lives in hermes_constants_scratch (hermes_constants keeps the
entry point). Before an idle entry is removed, same-user processes whose
cwd is inside it, or inside any scratch path that no longer exists, are
TERMed then KILLed; the deleted-cwd sweep runs on every pass so orphans
from earlier deletions are caught too. `.git` files found in the entry
name their repo, which gets `git worktree prune` after the rmtree.
cache/terminal used its own 72h fixed-age sweep; it now shares the 24h
idle rule and the subtree check. `hermes doctor` warns about cache-root
directories over 1 GiB that neither pruner covers, since finished campaign
trees parked there sat for weeks (95 GB on one host). System-prompt
scratch line updated to match.
The scratch pruner deleted top-level entries whose own mtime was older than
72h. A directory's mtime only moves when a direct child is added or removed,
so a lane writing deep inside its tree looked untouched and could lose a live
worktree at the deadline, while finished trees (7 GB clones with their own
venv per campaign lane) sat for three days: 791 entries / 57 GB after four
days of campaigns on one host.
Retention is now idle-based: an entry stays while anything anywhere in its
subtree was written in the last 24h and goes a day after the last write. The
walk short-circuits at the first recent mtime, so live trees cost one stat
and only a truly idle tree pays for a full walk, once, right before deletion.
Symlinks are not followed so a link into the repo cannot keep an entry alive.
The OS lanes are marker-driven: list_os_marked_tests.py picks the files
a lane imports from their platforms() specs and the lane selects with
-m platforms. A test gated with skipif(sys.platform != "win32") is
therefore never imported on the Windows lane and skipped everywhere else
— it runs on no host. skipif(sys.platform == "win32") tests were merely
invisible to the lane bookkeeping, but the rule the tree now follows is
one host marker, never a bare skipif.
Mechanical mapping, semantics preserved: skip-on-Windows → "posix",
skip-off-Windows → "windows", skip-off-Linux → "linux", skip-on-macOS →
"not macos". The former skip reasons stay as trailing comments. A
non-host condition (os.geteuid() == 0) stays a separate skipif beside
the marker, spelled getattr(os, "geteuid", ...) so the decorator still
imports on Windows.
Where the conversion would stack two platforms() marks on one test (the
conftest rejects that at collection) the narrower mark wins:
- test_update_wedged_gateway: the class is already platforms("linux");
its per-test "needs UNIX sockets" marks were redundant and are gone.
- test_process_registry.TestSystemdCgroupIsolation: the class-level
skip-on-Windows moves onto the 11 methods that had no host mark; the
11 platforms("linux") methods keep theirs.
- test_file_ops_single_roundtrip: the two fifo tests drop their
platforms("linux") in favour of the module's "posix" (mkfifo exists on
macOS; both tests already skip when it does not).
- test_linux_desktop_entry / test_gateway_job_teardown_live: duplicate
or wider marks removed.
Review follow-up: get_managed_system, the container/chmod-skip check and
the HERMES_UID/GID chown (_resolve_hermes_uid_gid/_chown_to_hermes_uid) now
live only in hermes_constants — the import-safe module apply_secure_dir_policy
already needed them in — and hermes_cli.config re-exports them, so there are
no 'keep in sync' copies and _secure_file skips on the same canonical
_detect_container signal as _secure_dir/get_scratch_dir. The dead config
copies are deleted and test_ensure_hermes_home_uid.py drives the new
symbols; one invariant test pins the single implementation.
Address the review findings on #117359:
- _is_managed_home(): drop "" from the not-managed tuple — the legacy
NixOS empty/unreadable marker maps to managed in
hermes_cli.config.get_managed_system, and the delegation was silently
re-enabling chmod on managed installs (#77579)
- _container_or_chmod_skipped(): widen to the canonical _detect_container
signals so Podman/containerd/K8s runtimes without HERMES_CONTAINER
keep operator modes (the sharing case #117347 sets out to honor)
- _chown_dir_to_hermes_uid(): docstring no longer claims a literal
contract with hermes_cli.config._chown_to_hermes_uid (no win32
early return here; os.chown AttributeError makes it a no-op)
Red/green: the three new parity tests fail with the fix stashed.
get_scratch_dir unconditionally chmod'ed <home>/cache/scratch to 0700,
bypassing the policy _secure_dir() applies everywhere else: managed
installs are left alone, containers only apply an explicit
HERMES_HOME_MODE, and HERMES_UID/HERMES_GID ownership is honored. On a
group-shared home (setgid 2770) that stripped the setgid bit and broke
the shared-group inheritance the operator configured (#77579, #117347).
Move the policy primitive to hermes_constants.apply_secure_dir_policy()
(import-safe, stdlib-only) and route both get_scratch_dir() and
hermes_cli.config._secure_dir() through it, so one implementation
serves all callers including the cron.jobs delegation chain.
Fixes#117347
Hermes and everything it launches (browser profiles, PTY probes, skill scripts,
tempfile defaults in delegated code) wrote to the system temp dir, which is a
RAM-backed tmpfs on most Linux hosts and containers and fills under agent load.
- hermes_constants.get_scratch_dir(): HERMES_HOME/cache/scratch (0700), entries
older than 72h pruned once per process / once per hour across processes.
- apply_scratch_tmp_env(env) / export_scratch_tmp_env(): TMPDIR/TMP/TEMP point at
the scratch dir when the user or OS has not set them; a value Hermes itself
exported (== HERMES_SCRATCH_DIR) is re-derived for a re-homed process or a
child served under another profile, so profiles never share scratch.
- hermes_bootstrap runs the export on import (every entry point); hermes_cli.main
re-runs it after --profile resolution; the subprocess HOME contract
(apply_subprocess_home_env) and the routed-home rewrites in code_execution_env
and served_profile_child_env apply it to child envs.
- The runtime-environment prompt block names the scratch dir so the model stops
reaching for the system temp dir by reflex; hermes doctor reports the dir, its
size and whether a user TMPDIR overrides it.
Teknium