Greptile r2 on #133283:
- #1: probe_store built one byte string the size of jobs.json; a large store made cron status,
doctor and the idle recovery check allocate that much. It now writes the same total in 64 KiB
chunks.
- #2: the host-wide store gauges read every degraded record, so a profile that left this
gateway while its store was down kept it at writable=0 until a save that would never come.
register_ticked_homes already discards a departed home's worker pool; it now also drops that
home's store state (store_health.forget_homes). The export test pins it.
The store gauges called degraded_record() in the exporter's (launch profile's) scope, so a
secondary profile's full or read-only store in a multiplexed gateway exported writable=1 and
skipped_runs=0 (Greptile r1 #4). Every served profile's ticker runs in this process, so the
gauges are now host-wide: writable is 0 while any store is degraded and skipped_runs sums them.
Review cleanup, no behaviour change:
- warn_store_unwritable only resolved the active store before calling
store_health.note_unwritable; note_unwritable now defaults cron_dir to the active store and
every caller uses it directly (scheduler_tick no longer imports both).
- The advance-skip split no longer zips a parallel bool list twice, and reads
(schedule or {}) so a null schedule cannot raise inside the OSError handler.
- skipped_runs metric: a walrus instead of getattr around an Optional record.
- Notices: runner._served_profile_homes is declared on the reconcile mixin, so no getattr.
- cron status loads the active jobs once (skipped count + summary) instead of twice.
Gate r4 quality: the store test's `<= _OBSERVABLE_METRIC_NAMES` assert only
mirrored the constant, while the existing all-gauges-registered test's
hand-built cron snapshot never got the two new gauges. Drop the constant
re-check and add hermes.cron.store.writable / skipped_runs to that snapshot,
so the parity check covers them (still red when either is unregistered).
Gate finding C12 r3 #3 (2ab, toothless): nothing pinned
hermes.cron.store.writable / skipped_runs, and pinning them showed the
gauges never left the process. _OBSERVABLE_METRIC_NAMES is the list the OTLP
exporter observes, and a gauge missing from it is silently dropped, so
dashboards never saw an unwritable store. Register both, and add one test:
a degraded store exports writable=0 with skipped_runs=1, a cleared store
exports writable=1, and both names are registered. The test goes red with
de334a6b37's cron_health hunk reverted and also with this registration
reverted.
Change-detectors, tautologies, source-reading tests, redundant duplicates,
mock-echo tests and dead/unrunnable tests. Per-test rationale in the lane
ledger (category + reason for every removal).
docs/ was not the documentation site; it was a grab bag of long-form
design notes, wire contracts and observability guides that landed with
feature PRs because their authors needed somewhere to put them. Root
AGENTS.md already says long-form dev docs live in
website/docs/developer-guide/; this moves the 14 living documents there
(or to the matching user-guide section) so they are published, searchable
and linked from the sidebar instead of being found by grep only.
Developer guide: micro-compaction, gateway-session-lifecycle (was
session-lifecycle), state-db-recovery, multiplexing-gateway,
chronos-managed-cron-contract, relay-connector-contract, observer-hooks
(was observability/README), gateway-monitoring (observability/monitoring),
relay-shared-metrics, middleware, streaming-tts, billing-lifecycle.
User guide: egress/network-isolation (was security/network-egress-
isolation), features/kanban-multi-gateway (was kanban/multi-gateway).
Each page got title/description frontmatter and a sidebar entry; repo-
relative links became site links or GitHub blob URLs; two MDX brace
hazards escaped. Every in-tree pointer (module docstrings, config
comments, the relay conformance test's Path, the monitoring-doc test,
gateway-internals, cron-internals, kanban docs, .dockerignore, AGENTS.md)
now names the new location. `docusaurus build` passes with no unresolved
links on the moved pages.
redact_for_egress's bearer sweep matched any run of token characters after
the word "Bearer", so ordinary prose ("I'm the bearer of bad news") came
back as "Bearer [redacted] bad news" on every chat and A2A reply. The
gateway and A2A sweeps this PR replaced always required 20+ chars; only
monitoring was floor-less. Restore the floor on the opaque branch and keep
the bracket branch that folds an already-masked residue to one marker.
Prepare dependency generations before selecting them. Keep shipped tool
bytes separate from writable additions, and store facts beside their entries.
Validate proposed plugin sets before config publication. Restore the previous
config if the facts write fails.
Consolidate duplicate updater, backup, setup, and voice helpers. Repair
launcher selection, dependency consumers, download ownership, update feeds,
and native Windows process and file handling.
Verification: 206 changed/prior-failing Python files reported 4630 passed,
one failed, and 330 skipped. Fix the remaining Hindsight fixture boundary.
The final targeted rerun reported 234 passed and two skipped. The store
review regression batch reported 83 passed and one skipped. Desktop
TypeScript checks, 56 selected Electron tests, 24 release tests, and the
removed-import/compatibility guards passed.
This is an integration checkpoint, not full audit acceptance. The complete
Python suite has not run on this fixed tree. Crash-atomic plugin publication,
generation cleanup, receipt correlation, and packaged lifecycle acceptance
remain open in docs/pm-audit-status.md.
_resource_attributes() in otlp_exporter.py built its own hardcoded
resource dict (service.name/instance.id/telemetry.scope only) instead
of reusing gateway_health_export.py's _runtime_resource_attributes(),
which already applies the resource_attributes allowlist from config.
Result: operator-configured attributes like deployment.environment.name
reached metrics and diagnostic logs but never spans.
Span resource building now delegates to the same
_runtime_resource_attributes() helper metrics/logs already use,
removing the duplicate implementation instead of patching it in place.
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
Complements the task-granular background_work with the async-delegation
UNIT count (each dispatch/batch = 1), recovering the pool-slot semantics
active_count() gives. Together: background_work = real concurrent subagent
load (batch expanded), background_delegations = slot pressure to alert
against delegation.max_concurrent_children. Registered in metric_names,
documented, and covered by a task-vs-unit contract test.
Extend the cron-export test to assert background_work membership (behavior
contract, not a frozen list), and add a regression guard that every gauge
emitted in the runtime snapshot is also registered in the observable
metric_names list — the silent-drop trap the extension guide documents.
- Cut the leftover telemetry.* DEFAULT_CONFIG block (nothing reads it) and
the legacy telemetry-key fallback in policy.py.
- install_id: persist the minted UUID back to config.yaml on first use so
service.instance.id survives gateway restarts (fail-open when the write
is not possible); regression test covers the restart path.
- Remove the no-op gateway_health_export.redaction config keys. Redaction
is always-on by design and deliberately not configurable; status output
now says so.
- Collapse redaction to one unconditional secrets+PII scrub: drop the
none/pii content modes (they served the dropped trajectories plane) and
fold gateway_health.py's duplicate bearer/token/email/phone regex layer
into agent/monitoring/redaction.py.
Salvages the event-spine foundation from feat/telemetry-observability
(emitter, typed events, OTLP streaming, redaction — authorship preserved in
the preceding commits) and scopes it to the plane enterprise operators need
today: gateway Service Health Monitoring plus redacted Operational
Diagnostics, exported over OTLP.
Dropped from the salvaged branch, deliberately:
- run/model/tool trajectory capture (plugins/telemetry hooks, tel_spans)
- the local JSONL + state.db tel_* store (monitoring is egress, not storage)
- usage rollups/metrics, /insights integration, bulk export
- hermes telemetry CLI (replaced by hermes monitoring status)
Those planes — shared client usage metrics and enterprise trace telemetry —
are being designed on the NeMo Relay integration with distinct consent,
policy, and export boundaries; this keeps the monitoring plane content-free
and independently enableable.
Renames agent/telemetry -> agent/monitoring, config telemetry.* ->
monitoring.*, and pins the otlp extra at OpenTelemetry 1.39.1 (matching
uv.lock; 1.30.0 conflicts with mistralai>=2.4 on opentelemetry-api).
Maciej Lisiewski