dilim/rea
1
0 Fork 0
mirror of https://github.com/morluto/rea.git synced 2026-11-06 05:17:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(eval): measure MCP catalog usage through the real Codex harness (#9035)

Browse Source 
* perf(eval): measure MCP catalog usage through real Codex sessions

* fix(eval): control the active Codex code mode host

* fix(eval): reject unavailable host controls without hiding client errors

* test(eval): cover the supported default and actionable client errors

* docs(eval): record pinned Codex catalog and real package measurements

* fix(eval): pin matrix identity and retain failed trial observations

---------

Co-authored-by: gensokyo-dev <gensokyo-dev@users.noreply.github.com>
This commit is contained in:
gensokyo-devandgensokyo-dev authored and GitHub committed 2026-11-06 02:25:34 +08:00
1 parent fb453f2930
commit 8084916859
11 files changed
+1935 -40
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/mcp-catalog-evaluation.md
+134
View File
@@ -0,0 +1,134 @@
# Measure MCP catalog costs in Codex
Use actual harness usage to evaluate catalog changes. A `tools/list` JSON byte
count measures transport size. It does not establish how many tokens a client
submits to a model, whether it defers schemas, or the total cost of an analysis.
The pinned [Codex observations](verification/mcp-catalog-codex-2026-11-05.md)
include actual loading controls, an unavailable-host negative control and a
completed real-package pilot.
## Run a controlled loading comparison
Build the selected source checkout with the pinned Node and npm versions first:
```sh
npm ci
npm run build:cached
npm run eval:mcp-catalog -- --model gpt-6.1-sol \
--output "$HOME/.cache/rea-catalog-run" --repetitions 3 --plan
```
Remove `--plan` to execute the matrix. This makes real model requests using the
local Codex authentication. The output directory must not already exist. No
account configuration is changed. The runner copies only authentication into
an external disposable profile and removes that profile after each trial.
Execution stays outside both source checkouts even when results are saved inside
one, so ancestor repository instructions and skills do not pollute the trial.
The npm entrypoint holds the driver's artifact lock throughout the matrix. A
selected external checkout has its own lock, so the matrix also pins one runtime
fingerprint and binds its catalog receipt to that manifest. Freshness checks
stop further model calls if the runtime changes between trials. Each trial's
before/after fingerprints must match the matrix pin; different builds never form
a valid pair. Matrix admission/freshness audits sit outside individual guarded
trial resource measurements. Do not rebuild the selected runtime concurrently.
This lane requires Linux, Python 3, a working user systemd manager with cgroup v2,
and an installed Codex CLI supporting the recorded flags. It reuses the existing
resource observer: one CPU, 4 GiB hard memory limit, 3 GiB early stop, no swap,
and a 192-task ceiling. Runs are sequential. Unsupported clients, resource stops,
failed startup, missing usage, and unverified cleanup remain failed measurements.
They do not become zero-cost successes.
Each repetition compares no server, all REA tools, and a client-selected subset
(`binary_session` by default), in the supported client-code host mode. Alternate
repetitions reverse ordering. All variants receive the same developer instructions
and exact-response prompt. A valid loading trial completes one turn, returns
`CATALOG_OK`, and performs no tool calls. REA is marked `required` in every
server-enabled configuration. The runner explicitly disables/enables the stable
`code_mode_host` feature for direct/client-code modes. The old `code_mode` and
`code_mode_only` switches do not override Codex 0.165's enabled host default.
The requested flags and CLI version are retained; verify the actual call shape
in completed-task traces too. `--modes direct client-code` adds a diagnostic
host-disabled control. This installed Codex reports that disabling the host
makes tool dispatch unavailable; it does not provide a qualified eager/direct
comparison. An empty response alone cannot qualify tool availability. Errors
invalidate that profile even when the model still emits the expected marker.
Useful controls:
```sh
# Compare full input guidance with the default compact input presentation.
npm run eval:mcp-catalog -- --model gpt-6.1-sol \
--output "$HOME/.cache/rea-catalog-full" --input-profile full --repetitions 3
# Choose exact tool names in the working host mode.
npm run eval:mcp-catalog -- --model gpt-6.1-sol \
--output "$HOME/.cache/rea-catalog-selected" --modes client-code \
--selected-tools analyze_javascript_application inspect_analysis_view
```
`enabled_tools` filters the catalog after client discovery. It does not reduce
the server's original `tools/list` transport. Server tool-family scope is a
separate existing control; do not attribute client filtering results to it.
The output-schema profile remains REA's default in these experiments.
## Read the evidence
`report.json` retains every attempted trial and compares each enabled-server
trial with its same-mode, same-repetition no-server control. It reports valid
pair counts and the median/range of observed input-token differences. Failed
pairs are explicit and excluded from the median. A small or negative delta can
reflect client framing, randomized workspace paths, and measurement variation;
it does not establish a negative schema cost.
Usage, cleanup and resource receipts are collected independently of the verdict.
A failed host, malformed event or wrong marker can retain observed usage and
resource costs. Stage-specific issues explain invalidity; missing observations
remain unknown. A changed matrix records its unattempted cells explicitly.
Each trial retains its exact prompt, runtime file digests, requested model alias,
Codex version, settings/source hashes, JSONL events, actual usage, answer, cleanup
receipt, and process-family resource receipt. The selected rollout capture keeps
per-response usage and tool exchanges; authentication, ambient environment,
account metadata, and encrypted reasoning are excluded. Tool output and local
paths remain evidence and should be reviewed before sharing the raw files.
The metrics have distinct meanings:
| Metric | Meaning and limit |
| ----------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Catalog bytes and SHA-256 | Built server advertisement, including output schemas; not model input. |
| `turn.completed.usage.input_tokens` | Observed whole-turn usage reported by Codex; includes cached input. |
| Cached/uncached input | Report both. Cache hits do not remove tokens from the context. |
| Per-response usage | Retained usage records help locate discovery and continuation costs. Do not add cumulative thread counts again. |
| Paired input delta | Incremental loading observation including instructions/client framing; not an exact schema-only token count. |
| Elapsed and family CPU | Whole guarded trial, including local setup and client/network waits; remote inference CPU is unmeasured. |
| Family peak memory | Kernel cgroup peak at payload exit, including observer and file cache; not final whole-scope counters. |
These are fresh client/server sessions. OS caches and API prefix caches can
already be warm. No cold-cache claim follows from a new session. The requested
model alias is recorded; a resolved server-side model version is not inferred.
The harness usage is not an independent billing reconciliation.
The usage event shape follows the documented
[Codex non-interactive JSON output](https://developers.openai.com/codex/noninteractive#make-output-machine-readable).
## Evaluate completed investigations too
A no-tool response can avoid deferred discovery entirely. Before changing the
catalog based on this lane, compare completed tasks through the real harness.
Use `npm run verify:agent` and its known-answer rubric, or run a selected real
artifact through `tests/support/analysis-efficiency/agent-trial.py` under
`resource-run.py --observe-payload-exit`. Use the same task, artifact digest,
model/reasoning, authority, result coverage, and correctness oracle in every arm.
The existing runner accepts exact `--enabled-tools` and defaults to client-code
host mode. Its direct option is a host-disabled diagnostic, not a claim of
working native dispatch in the installed client.
Use `--capture-tool-context` to locate per-response costs. Its task summary
explicitly leaves factual correctness unassessed until an independent oracle or
transcript review establishes the requested facts.
Count discovery, arguments, replies, retries, and continuations. Check the
requested facts against source bytes or another independent oracle, real produced
Evidence IDs and source links, partial/unknown coverage, and cleanup. Preserve
failed attempts. A successful small-package task is a pilot, not proof of broad
application performance or provider compatibility.
docs/testing-lanes.md
+4
View File
@@ -1230,6 +1230,10 @@ package entrypoint as the script's first argument for package coverage.
## Agent evaluation and conformance records
For catalog-loading measurements, use the real Codex
[MCP catalog evaluation](mcp-catalog-evaluation.md). It separates observed usage
from advertisement bytes and from completed-task correctness.
`npm run verify:agent` evaluates investigation scenarios through a real local
Codex CLI. The lane runs sequentially; select scenarios with
`REA_AGENT_EVAL_SCENARIOS` and choose the installed client/model with
docs/verification/mcp-catalog-codex-2026-11-05.json
+1027
View File
File diff suppressed because it is too large. Load diff
docs/verification/mcp-catalog-codex-2026-11-05.md
+111
View File
@@ -0,0 +1,111 @@
# Codex MCP catalog observations, 2026-11-05
The supported Codex host path showed no material input-token increase for an
empty response with REA enabled. Completing a real package question still used
hundreds of thousands of cumulative input tokens across discovery and
continuations. Neither raw advertisement bytes nor an empty response measures
that whole-task cost.
REA source baseline: `ec3410998b70290847f930538e57d0af0ac1bffb`. Loading driver:
`5038280f9`; completed-task/default-smoke driver: `61a364210`. The
[machine-readable receipts](mcp-catalog-codex-2026-11-05.json) retain exact usage,
source/runtime hashes, response counts, resource counters, answers, and invalid
profiles. Full local transcripts remain under the recorded `raw_artifact_root`;
they are not committed. See the [evaluation procedure](../mcp-catalog-evaluation.md)
to reproduce with a built source checkout.
Linux x64; Node 24.22.1/npm 11.16.0; Codex CLI 0.165.0; requested model
`gpt-6.1-sol`, reasoning `low`. The resolved backend model version is unknown.
Each guarded trial used a fresh client/server/profile, one CPU, 4 GiB hard memory,
3 GiB early stop, no swap, and at most 192 tasks. OS/API prefix caches could be
warm. Reported input includes cached input. All valid payloads exited normally
and verified cleanup; no resource stops or OOM events were observed.
## Advertisement and empty-response costs
The built default catalog contains 234 tools and 4,029,995 JSON bytes, including
3,681,386 output-schema JSON bytes, 138,747 input-schema bytes, and 148,355
description bytes. This is server advertisement size, not submitted model input.
Production `mcp doctor` independently observed all 234 tools and six prompts,
matched catalog identity, and completed `binary_session` successfully.
Three repetitions per profile, with alternating order; default compact input
schemas and default full output schemas. The client-selected profile exposes
only `binary_session` after discovering the same server catalog.
| Host-enabled profile | Valid samples | Paired input delta, median (range) | Median guarded wall s | Median family CPU s | Median family peak MiB |
| -------------------- | ------------: | ---------------------------------: | --------------------: | ------------------: | ---------------------: |
| No MCP | 3 | control | 14.693 | 1.965 | 187.9 |
| All tools | 3 | +3 (0 to +3) | 13.490 | 2.685 | 347.2 |
| Selected tool | 3 | +3 (-3 to +3) | 13.792 | 2.573 | 336.1 |
Every valid response returned exactly `CATALOG_OK` without tool calls. Input was
11,112–11,121 tokens, including 9,216 cached input tokens, with eight output
tokens. Random workspace paths and client framing remain in the comparison.
These small differences do not establish a schema-only token count. Network
waits dominate elapsed time; no stable latency improvement is established.
The local CPU/memory increase when enabling MCP remains measurable in this setup.
Family peaks are kernel cgroup counters including observer/file cache at payload
exit, rather than final whole-scope counters.
Nine additional host-disabled diagnostic samples emitted:
> Code Mode is unavailable because code-mode host is disabled.
The client also reported that tool dispatch would fail closed. It still returned
the requested marker, illustrating why that marker alone is insufficient.
All nine profiles are **invalid**; their paired deltas remain unknown. This
diagnostic matrix exits nonzero intentionally. It does not establish an eager
or native-dispatch comparison. The final command defaults to the working host
profile; a separate three-cell default smoke run passed.
## Completed real-package pilot
Both arms answered the same question from the installed public `semver@7.8.5`
package, analyzing only `package.json`: declared name `semver`, version `7.8.5`,
and entry `index.js`. Those three fields were checked against the source JSON
independently of REA. Answers cited Evidence references emitted by the actual
tools and stated that dependency modules were unexamined and runtime behavior
was not observed. Both runs retained the same analysis parent Evidence identity.
The selected arm allowed `analyze_javascript_application` and
`inspect_analysis_view`; the other allowed the whole catalog. Both used the
working host, the identical question/instructions, and unchanged target bytes.
| Tool selection | Whole-turn input (cached subset) | Output | Model responses / MCP calls | Guarded wall / family CPU s | Family peak MiB |
| -------------- | -------------------------------: | -----: | --------------------------: | --------------------------: | --------------: |
| All | 271,084 (228,224) | 699 | 8 / 3 | 48.952 / 6.532 | 671.7 |
| Selected | 160,438 (138,496) | 637 | 7 / 3 | 42.928 / 5.728 | 582.1 |
The first model input was 11,349 tokens in both cases. The subsequent
`exec`/`ALL_TOOLS` discovery, schema reads, replies and context replay account for
the later cumulative input. Per-response input counts sum exactly to the reported
whole-turn input; they must not be added to it again. Counting only the three MCP
calls would miss several model responses.
This is one task per arm. The model chose different follow-up tools and
projections. The numbers demonstrate a completed-task measurement method; they
do not isolate a causal catalog reduction or establish a general advantage for
filtered tools, complex applications, or other clients/providers.
## Verification and exclusions
The new runner separates execution workspaces from artifact output, holds the
driver's artifact lock, pins one matrix runtime and checks freshness across
trials and at completion. Its
named usage parser preserves missing counts as unknown and rejects failed client
profiles, wrong answers, unexpected actions, runtime drift and unverified cleanup.
All 18 retained control transcripts were reassessed after the reporting fixes:
verdicts and token comparisons stayed identical, and failed profiles retained
their available usage, cleanup and resource observations.
Exploratory cohorts remain recorded separately: repository-nested workspaces
inherited project instructions/skills; a concurrent artifact rebuild interrupted
two MCP handshakes; and old `code_mode`/`code_mode_only` switches did not establish
distinct dispatch modes. Those cohorts do not support the tables above. The
sibling JavaScript-facts harness already explicitly enables the required host.
Dependency acquisition/build costs, complete dependency-tree byte identity,
remote inference CPU/memory, exact submitted request bodies and independent
billing reconciliation are unmeasured. No REA production contracts, Evidence
semantics, or catalog defaults changed in this work.
package.json
+1
View File
@@ -154,6 +154,7 @@
"test:with-build": "npm run test:prepare && vitest run",
"test:run": "vitest run",
"test:python": "python3 scripts/run-python-tests.py",
"eval:mcp-catalog": "node scripts/run-exclusive.mjs artifacts python3 tests/support/analysis-efficiency/catalog-eval.py",
"test:local": "node scripts/run-development-tests.mjs local",
"test:fast": "node scripts/ci/test-group.mjs source",
"test:built": "node scripts/ci/test-group.mjs built",
tests/support/analysis-efficiency/README.md
+6 -1
View File
@@ -1,5 +1,10 @@
# Analysis verification test support
The maintained `catalog-eval.py` entrypoint (`npm run eval:mcp-catalog`) uses the
real local Codex CLI and the retained resource/agent runners. See
[MCP catalog evaluation](../../../docs/mcp-catalog-evaluation.md) for its matrix,
observed-token accounting, isolation and completed-task limits.
These files support the existing comparison, independent-oracle, receipt and
cleanup regressions. The TypeScript consumers live in `tests/boundary/`,
`tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers
@@ -17,7 +22,7 @@ belong to documentation. Their numerical sections are produced and checked by:
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
```
The full experiment suite was retired. For historical reproduction, use the
The former broad experiment suite was retired. For historical reproduction, use the
recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
preserves the former layout. Current deterministic checks do not requalify
those historical measurements.
tests/support/analysis-efficiency/agent-trial.py
+54 -39
View File
@@ -12,32 +12,13 @@ import tempfile
import time
from codex_context import capture_tool_context
def file_digest(path):
digest = hashlib.sha256()
with path.open('rb') as data:
for chunk in iter(lambda: data.read(1024 * 1024), b''):
digest.update(chunk)
return digest.hexdigest()
def runtime_fingerprint(root):
files = []
for name in ['dist', 'bridge']:
for path in sorted((root / name).rglob('*')):
if path.is_symlink():
raise ValueError('Runtime fingerprint does not follow symlinks')
if path.is_file():
files.append({'path': path.relative_to(root).as_posix(), 'sha256': file_digest(path)})
files.sort(key=lambda row: row['path'])
text = json.dumps(files, separators=(',', ':'), ensure_ascii=False)
return files, hashlib.sha256(text.encode()).hexdigest()
from runtime_identity import file_digest, runtime_fingerprint
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--mode', choices=['direct', 'client-code'], default='direct')
parser.add_argument('--mode', choices=['direct', 'client-code'], default='client-code',
help='client-code enables code_mode_host; direct is a host-disabled diagnostic')
parser.add_argument('--runtime-root', type=Path, required=True)
parser.add_argument('--revision', required=True)
parser.add_argument('--prompt', type=Path, required=True)
@@ -49,12 +30,22 @@ def main():
parser.add_argument('--codex', default='/usr/bin/codex')
parser.add_argument('--providers', type=Path)
parser.add_argument('--enabled-tools', type=Path)
parser.add_argument('--catalog-load', action='store_true',
help='Measure a no-tool response with identical instructions across catalogs')
parser.add_argument('--no-mcp', action='store_true',
help='No-server control; requires --catalog-load')
parser.add_argument('--profile-parent', type=Path,
help='External disposable-profile parent (default: cache beside the source Codex home)')
parser.add_argument('--authority', choices=['static', 'browser'], default='static')
parser.add_argument('--capture-tool-context', action='store_true',
help='Retain owned tool/usage items; excludes account/ambient state')
parser.add_argument('--guide-enabled-tools', action='store_true',
help='Opt-in exact-name discovery guidance for selected REA tools')
args = parser.parse_args()
if args.no_mcp and not args.catalog_load:
parser.error('--no-mcp requires --catalog-load')
if args.catalog_load and (args.guide_enabled_tools or args.artifact_path):
parser.error('--catalog-load excludes tool guidance and artifact tasks')
if args.guide_enabled_tools and args.enabled_tools is None:
parser.error('--guide-enabled-tools requires --enabled-tools')
enabled_tools = None
@@ -101,7 +92,13 @@ def main():
args.output.mkdir(parents=True, exist_ok=False, mode=0o700)
runtime_files, runtime_digest = runtime_fingerprint(repository)
(args.output / 'runtime-files.json').write_text(json.dumps(runtime_files))
profile = Path(tempfile.mkdtemp(prefix='profile-', dir=args.output))
# Output artifacts may live inside a checkout. Execution must not: Codex
# discovers ancestor AGENTS.md and skills independently of its user config.
profile_parent = (args.profile_parent or source_auth_home.parent / '.cache' / 'rea-agent-trials').resolve()
if any(profile_parent.is_relative_to(root.resolve()) for root in [repository, driver_repository]):
raise ValueError('--profile-parent must be outside the runtime and driver checkouts')
profile_parent.mkdir(parents=True, exist_ok=True, mode=0o700)
profile = Path(tempfile.mkdtemp(prefix='profile-', dir=profile_parent))
trial_codex_home = profile / 'codex'
account = profile / 'account'
workspace = profile / 'workspace'
@@ -156,6 +153,11 @@ def main():
'return only their metadata. Preserve the original analyst question, '
'dependency order and source Evidence.'
)
if args.catalog_load:
instruction = (
'This is a catalog loading measurement. Do not call any tools, search, '
'or delegate. Reply with exactly the text requested by the user.'
)
quote = json.dumps
config = '\n'.join([
'approval_policy = "never"', 'sandbox_mode = "read-only"',
@@ -164,20 +166,21 @@ def main():
'developer_instructions = ' + quote(instruction),
'[features]', 'multi_agent = false', 'shell_snapshot = false',
'shell_tool = false', 'unified_exec = false',
'code_mode = ' + ('true' if args.mode == 'client-code' else 'false'),
'code_mode_only = ' + ('true' if args.mode == 'client-code' else 'false'),
'code_mode_host = ' + ('true' if args.mode == 'client-code' else 'false'),
'[history]', 'persistence = "none"',
'[mcp_servers.rea]', 'command = ' + quote(node),
'args = ' + quote(command_args), 'startup_timeout_sec = 60',
'tool_timeout_sec = 600',
*([] if enabled_tools is None else ['enabled_tools = ' + quote(enabled_tools)]),
'[mcp_servers.rea.env]',
*[key + ' = ' + quote(value) for key, value in provider_env.items()],
]) + '\n'
if not args.no_mcp:
config += '\n'.join([
'[mcp_servers.rea]', 'required = true', 'command = ' + quote(node),
'args = ' + quote(command_args), 'startup_timeout_sec = 60',
'tool_timeout_sec = 600',
*([] if enabled_tools is None else ['enabled_tools = ' + quote(enabled_tools)]),
'[mcp_servers.rea.env]',
*[key + ' = ' + quote(value) for key, value in provider_env.items()],
]) + '\n'
# MCP read-only hints cover normal static tools. Explicitly authorize only
# the session-opening primitive needed by the requested native tasks.
config += '[mcp_servers.rea.tools.open_binary]\napproval_mode = "approve"\n'
for tool in ['analyze_javascript_application', 'inspect_analysis_view',
approved_tools = ['open_binary', 'analyze_javascript_application', 'inspect_analysis_view',
'inspect_android_package', 'search_android_classes',
'inspect_android_class', 'inspect_android_method',
'inspect_firmware_regions', 'inspect_managed_artifact',
@@ -199,8 +202,10 @@ def main():
'import_evidence_bundle', 'list_unknowns', 'verify_unknown_resolution',
'record_unknown', 'update_unknown', 'reset_investigation', 'binary_session',
'capture_browser_scenario',
'resolve_containing_procedure', 'analyze_function', 'close_binary']:
config += '[mcp_servers.rea.tools.' + tool + ']\napproval_mode = "approve"\n'
'resolve_containing_procedure', 'analyze_function', 'close_binary']
if not args.no_mcp:
for tool in approved_tools:
config += '[mcp_servers.rea.tools.' + tool + ']\napproval_mode = "approve"\n'
try:
if source_auth.exists():
shutil.copyfile(source_auth, auth)
@@ -220,10 +225,13 @@ def main():
'mode': args.mode, 'model': args.model, 'reasoning': args.reasoning,
'capture_tool_context': args.capture_tool_context,
'guide_enabled_tools': args.guide_enabled_tools,
'catalog_load': args.catalog_load, 'mcp_enabled': not args.no_mcp,
'driver_commit': subprocess.check_output(
['git', 'rev-parse', 'HEAD'], cwd=driver_repository, text=True).strip(),
'runtime_revision': args.revision,
'runtime_root': str(repository),
'workspace': str(workspace),
'workspace_isolated_from_runtime': not workspace.is_relative_to(repository.resolve()),
'selected_artifact_path': args.artifact_path,
'selected_artifact_sha256_before': target_digest,
'enabled_tools': enabled_tools,
@@ -233,7 +241,7 @@ def main():
'package_lock_sha256': file_digest(repository / 'package-lock.json'),
'experiment_sources_sha256': {
name: hashlib.sha256((experiment / name).read_bytes()).hexdigest()
for name in ['agent-trial.py', 'codex_context.py', 'provider-environment.json']
for name in ['agent-trial.py', 'codex_context.py', 'runtime_identity.py', 'provider-environment.json']
},
'developer_instructions': instruction,
'developer_instructions_sha256': hashlib.sha256(instruction.encode()).hexdigest(),
@@ -246,8 +254,7 @@ def main():
'provider_environment_policy_sha256': hashlib.sha256(
provider_policy_text.encode()).hexdigest(),
'features': {'shell_tool': False, 'unified_exec': False,
'multi_agent': False, 'code_mode': args.mode == 'client-code',
'code_mode_only': args.mode == 'client-code'},
'multi_agent': False, 'code_mode_host': args.mode == 'client-code'},
}, indent=2) + '\n')
invocation = [args.codex, '--no-daemon', 'exec', '--strict-config',
*([] if args.capture_tool_context else ['--ephemeral']),
@@ -290,9 +297,17 @@ def main():
).encode()),
})
final_target_digest = file_digest(Path(args.artifact_path)) if args.artifact_path else None
try:
_, final_runtime_digest = runtime_fingerprint(repository)
except OSError:
final_runtime_digest = None
summary = {
'mode': args.mode, 'requested_model': args.model, 'reasoning': args.reasoning,
'runtime_revision': args.revision,
'runtime_files_sha256_before': runtime_digest,
'runtime_files_sha256_after': final_runtime_digest,
'runtime_bytes_unchanged': runtime_digest == final_runtime_digest,
'catalog_load': args.catalog_load, 'mcp_enabled': not args.no_mcp,
'capture_tool_context_requested': args.capture_tool_context,
'selected_artifact_sha256_before': target_digest,
'selected_artifact_sha256_after': final_target_digest,
@@ -315,7 +330,7 @@ def main():
# Capture failure must never prevent copied-credential cleanup.
shutil.rmtree(profile)
(args.output / 'cleanup.json').write_text(json.dumps({'profile_removed': not profile.exists()}) + '\n')
raise SystemExit(124 if timeout else exit_code)
raise SystemExit(124 if timeout else 125 if final_runtime_digest != runtime_digest else exit_code)
if __name__ == '__main__':
tests/support/analysis-efficiency/catalog-eval.py
+284
View File
@@ -0,0 +1,284 @@
"""Measure catalog loading in real Codex sessions, never by a bytes/token ratio."""
import argparse
import hashlib
import json
from pathlib import Path
import statistics
import subprocess
import sys
from codex_context import token_counts
from runtime_identity import runtime_fingerprint
HERE = Path(__file__).resolve().parent
ROOT = HERE.parents[2]
MARKER = 'CATALOG_OK'
PROMPT = 'Reply with exactly ' + MARKER + '.\n'
TOKEN_KEYS = ('input_tokens', 'cached_input_tokens', 'output_tokens')
def write_json(path, value):
path.write_text(json.dumps(value, indent=2, ensure_ascii=False) + '\n')
def usage_counts(value):
counts = token_counts(value)
if not all(key in counts for key in TOKEN_KEYS) \
or counts['input_tokens'] == 0 or counts['cached_input_tokens'] > counts['input_tokens']:
raise ValueError('Missing or invalid observed token usage')
return counts
def assess_trial(directory, returncode, expected_runtime_digest):
"""Collect independent partial observations before grading the trial."""
result = {'status': 'invalid', 'directory': str(directory), 'exit_code': returncode}
issues = []
def observe(stage, work):
try:
return work()
except (OSError, ValueError, KeyError, TypeError) as error:
issues.append({'stage': stage, 'reason': str(error)})
return None
def require(condition, stage, reason):
if not condition:
issues.append({'stage': stage, 'reason': reason})
def read_object(name):
value = json.loads((directory / name).read_text())
if not isinstance(value, dict):
raise ValueError('Expected an object in ' + name)
return value
summary = observe('summary', lambda: read_object('summary.json'))
settings = observe('settings', lambda: read_object('trial-settings.json'))
cleanup = observe('cleanup', lambda: read_object('cleanup.json'))
if summary is not None:
result['summary'] = summary
result['runtime_fingerprint_sha256'] = summary.get('runtime_files_sha256_before')
require(returncode == 0 and summary.get('exit_code') == 0 and summary.get('timeout') is False,
'completion', 'Trial failed or timed out')
require(expected_runtime_digest is not None and summary.get('runtime_bytes_unchanged') is True
and summary.get('runtime_files_sha256_before') == expected_runtime_digest
and summary.get('runtime_files_sha256_after') == expected_runtime_digest,
'identity', 'Runtime does not match the pinned matrix fingerprint')
if settings is not None:
require(settings.get('catalog_load') is True and settings.get('workspace_isolated_from_runtime') is True,
'settings', 'Trial was not an isolated catalog-load measurement')
if cleanup is not None:
result['cleanup'] = cleanup
require(cleanup.get('profile_removed') is True, 'cleanup', 'Trial profile cleanup was not verified')
events = []
def read_events():
with (directory / 'events.jsonl').open() as lines:
for ordinal, line in enumerate(lines, 1):
if not line.strip():
continue
def parse():
value = json.loads(line)
if not isinstance(value, dict):
raise ValueError('Expected a JSONL event object')
return value
event = observe('events line ' + str(ordinal), parse)
if event is not None:
events.append(event)
observe('events', read_events)
turns = [event for event in events if event.get('type') == 'turn.completed']
require(len(turns) == 1, 'usage', 'Expected exactly one completed turn')
usage = observe('usage', lambda: usage_counts(turns[0].get('usage'))) if len(turns) == 1 else None
if usage is not None:
result['usage'] = usage
result['usage']['uncached_input_tokens'] = (
usage['input_tokens'] - usage['cached_input_tokens'])
for event in events:
require(event.get('type') not in ['error', 'turn.failed'], 'client',
'Client reported an error or failed turn: ' + str(event.get('message', event.get('error'))))
items = [event.get('item') for event in events if event.get('type') in ['item.started', 'item.completed']]
for item in items:
if isinstance(item, dict) and item.get('type') == 'error':
require(False, 'client', 'Client item error: ' + str(item.get('message')))
else:
require(isinstance(item, dict) and item.get('type') in ['agent_message', 'reasoning'],
'actions', 'A load-only trial performed a tool/action or an unknown item')
completed = [event.get('item') for event in events if event.get('type') == 'item.completed']
answers = [item.get('text') for item in completed if isinstance(item, dict) and item.get('type') == 'agent_message']
require(answers == [MARKER], 'answer', 'Load-only answer did not match the oracle')
def read_resources():
receipts = []
for line in directory.with_suffix('.log').read_text().splitlines():
try:
row = json.loads(line)
except ValueError:
continue
if isinstance(row, dict) and 'resource_scope' in row:
receipts.append(row)
if len(receipts) != 1:
raise ValueError('Missing or ambiguous resource receipt')
return receipts[0]
receipt = observe('resources', read_resources)
if receipt is not None:
result['resources'] = receipt
observation = receipt.get('payload_resource_observation')
require(isinstance(observation, dict) and receipt.get('abort') is None
and receipt.get('payload_resource_observation_verified') is True
and observation.get('payload_exit_code') == 0 and observation.get('payload_cleanup_verified') is True,
'resources', 'Resource stop, payload failure, or unverified process cleanup')
def metrics():
cpu = observation['cpu_stat']['usage_usec']
peak = observation['memory_peak_bytes']
if any(not isinstance(value, int) or isinstance(value, bool) or value < 0 for value in [cpu, peak]):
raise ValueError('Invalid resource counters')
return {'elapsed_seconds': receipt.get('elapsed_seconds'), 'cpu_seconds': cpu / 1_000_000,
'peak_process_family_bytes': peak,
'scope': 'Kernel cgroup counters at payload exit, including observer and file cache; not final scope counters.'}
measured = observe('resource metrics', metrics)
if measured is not None:
result['resource_metrics'] = measured
result['status'] = 'invalid' if issues else 'valid'
if issues:
result['issues'] = issues
result['reason'] = '; '.join(issue['reason'] for issue in issues)
return result
def comparison(rows, expected_runtime_digest):
"""Pair each catalog with its same-mode, same-repetition no-server control."""
controls = {(row['mode'], row['repetition']): row for row in rows if row['arm'] == 'none'}
pairs = []
for row in rows:
if row['arm'] == 'none':
continue
baseline = controls.get((row['mode'], row['repetition']))
valid = expected_runtime_digest is not None and row['status'] == 'valid' and baseline is not None and baseline['status'] == 'valid' \
and row.get('runtime_fingerprint_sha256') == expected_runtime_digest \
and baseline.get('runtime_fingerprint_sha256') == expected_runtime_digest
pairs.append({
'mode': row['mode'], 'arm': row['arm'], 'repetition': row['repetition'],
'status': 'valid' if valid else 'invalid',
'input_token_delta': row['usage']['input_tokens'] - baseline['usage']['input_tokens'] if valid else None,
})
groups = []
for mode, arm in sorted({(pair['mode'], pair['arm']) for pair in pairs}):
selected = [pair for pair in pairs if pair['mode'] == mode and pair['arm'] == arm]
deltas = [pair['input_token_delta'] for pair in selected if pair['status'] == 'valid']
groups.append({'mode': mode, 'arm': arm, 'valid_pairs': len(deltas), 'attempted_pairs': len(selected),
'median_input_token_delta': statistics.median(deltas) if deltas else None,
'min_input_token_delta': min(deltas) if deltas else None,
'max_input_token_delta': max(deltas) if deltas else None})
return {'pairs': pairs, 'groups': groups}
def catalog_receipt(root, profile, selected):
source = root / 'dist' / ('mcp-catalog-full.json' if profile == 'full' else 'mcp-catalog.json')
data = source.read_bytes()
catalog = json.loads(data)
names = {tool['name'] for tool in catalog['tools']}
if len(selected) != len(set(selected)) or not selected or not set(selected) <= names:
raise ValueError('Selected tools must be unique exact names in the built catalog')
size = lambda value: len(json.dumps(value, ensure_ascii=False, separators=(',', ':')).encode())
return {'path': str(source), 'sha256': hashlib.sha256(data).hexdigest(), 'bytes': len(data),
'tool_count': len(names), 'selected_tools': selected,
'tools_json_bytes': size(catalog['tools']),
'description_json_bytes': sum(size(tool.get('description')) for tool in catalog['tools']),
'input_schema_json_bytes': sum(size(tool.get('inputSchema')) for tool in catalog['tools']),
'output_schema_json_bytes': sum(size(tool.get('outputSchema')) for tool in catalog['tools']),
'scope': 'Built server catalog, not submitted model context. Client filtering still discovers the server catalog.'}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--runtime-root', type=Path, default=ROOT)
parser.add_argument('--output', type=Path, required=True)
parser.add_argument('--model', required=True, help='Explicit Codex model alias; never inferred from user config')
parser.add_argument('--reasoning', default='low')
parser.add_argument('--modes', nargs='+', choices=['direct', 'client-code'], default=['client-code'],
help='Default: supported host mode. direct disables the host as a diagnostic control.')
parser.add_argument('--input-profile', choices=['compact', 'full'], default='compact')
parser.add_argument('--selected-tools', nargs='+', default=['binary_session'])
parser.add_argument('--repetitions', type=int, default=1)
parser.add_argument('--timeout', type=int, default=180)
parser.add_argument('--codex', default='codex')
parser.add_argument('--plan', action='store_true', help='Show the matrix without invoking a model')
args = parser.parse_args()
if args.repetitions < 1 or args.timeout < 1 or len(set(args.modes)) != len(args.modes):
parser.error('Repetitions/timeout must be positive; modes must be unique')
root = args.runtime_root.resolve()
output = args.output.resolve()
runtime_files, pinned_digest = (None, None) if args.plan else runtime_fingerprint(root)
receipt = catalog_receipt(root, args.input_profile, args.selected_tools)
if not args.plan:
catalog_path = Path(receipt['path']).relative_to(root).as_posix()
pinned_catalog = next((row for row in runtime_files if row['path'] == catalog_path), None)
if pinned_catalog is None or pinned_catalog['sha256'] != receipt['sha256']:
raise ValueError('Catalog changed while pinning the matrix runtime; rebuild and retry')
revision = subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=root, text=True).strip()
matrix = []
for repetition in range(args.repetitions):
# Reverse both mode and arm ordering on alternating rounds to expose
# order/cache effects. Every invocation still has a fresh client/server.
for mode in args.modes[::1 if repetition % 2 == 0 else -1]:
for arm in ['none', 'all', 'selected'][::1 if repetition % 2 == 0 else -1]:
matrix.append({'repetition': repetition + 1, 'mode': mode, 'arm': arm})
experiment = {'version': 1, 'runtime_revision': revision, 'model': args.model, 'reasoning': args.reasoning,
'input_profile': args.input_profile, 'catalog': receipt, 'matrix': matrix,
'sources_sha256': {name: hashlib.sha256((HERE / name).read_bytes()).hexdigest()
for name in ['catalog-eval.py', 'agent-trial.py', 'codex_context.py', 'runtime_identity.py',
'resource-run.py', 'resource_payload.py', 'provider-environment.json']},
'scope': 'Fresh-session load-only usage. Warm OS/API caches are possible; no cache-miss claim or task-quality claim.',
'usage_scope': 'Codex turn.completed observed usage; deltas include server instructions and client framing, not an exact schema-only token count.'}
if args.plan:
print(json.dumps(experiment, indent=2))
return 0
experiment['runtime_files_sha256'] = pinned_digest
output.mkdir(parents=True, exist_ok=False, mode=0o700)
write_json(output / 'experiment.json', experiment)
write_json(output / 'runtime-files.json', runtime_files)
(output / 'prompt.txt').write_text(PROMPT)
write_json(output / 'selected-tools.json', args.selected_tools)
write_json(output / 'providers.json', {'REA_MCP_INPUT_SCHEMA_PROFILE': args.input_profile})
rows = []
for index, cell in enumerate(matrix):
reason = 'Runtime changed between trials; no further model calls were made'
try:
_, current_digest = runtime_fingerprint(root)
except (OSError, ValueError) as error:
current_digest = None
reason = 'Runtime identity unavailable; no further model calls were made: ' + str(error)
if current_digest != pinned_digest:
report = {'experiment': experiment, 'trials': rows, 'comparison': comparison(rows, pinned_digest),
'unattempted': matrix[index:], 'observed_runtime_digest': current_digest, 'reason': reason}
write_json(output / 'report.json', report)
print(report['reason'], flush=True)
return 1
directory = output / ('{repetition:02d}-{mode}-{arm}'.format(**cell))
command = [sys.executable, str(HERE / 'resource-run.py'), '--observe-payload-exit', '--',
sys.executable, str(HERE / 'agent-trial.py'), '--runtime-root', str(root), '--revision', revision,
'--prompt', str(output / 'prompt.txt'), '--output', str(directory), '--model', args.model,
'--reasoning', args.reasoning, '--mode', cell['mode'], '--timeout', str(args.timeout),
'--codex', args.codex, '--providers', str(output / 'providers.json'),
'--catalog-load', '--capture-tool-context']
if cell['arm'] == 'none':
command.append('--no-mcp')
elif cell['arm'] == 'selected':
command.extend(['--enabled-tools', str(output / 'selected-tools.json')])
print(json.dumps({'starting': cell}), flush=True)
with directory.with_suffix('.log').open('w') as log:
outcome = subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, check=False)
row = {**cell, **assess_trial(directory, outcome.returncode, pinned_digest)}
rows.append(row)
report = {'experiment': experiment, 'trials': rows, 'comparison': comparison(rows, pinned_digest)}
write_json(output / 'report.json', report)
print(json.dumps({'completed': cell, 'status': row['status'], 'usage': row.get('usage'), 'reason': row.get('reason')}), flush=True)
print(json.dumps(report['comparison']['groups'], indent=2))
return 0 if all(row['status'] == 'valid' for row in rows) else 1
if __name__ == '__main__':
sys.exit(main())
tests/support/analysis-efficiency/runtime_identity.py
+25
View File
@@ -0,0 +1,25 @@
"""One byte-identity owner shared by matrix admission and guarded trials."""
import hashlib
import json
def file_digest(path):
digest = hashlib.sha256()
with path.open('rb') as data:
for chunk in iter(lambda: data.read(1024 * 1024), b''):
digest.update(chunk)
return digest.hexdigest()
def runtime_fingerprint(root):
files = []
for name in ['dist', 'bridge']:
for path in sorted((root / name).rglob('*')):
if path.is_symlink():
raise ValueError('Runtime fingerprint does not follow symlinks')
if path.is_file():
files.append({'path': path.relative_to(root).as_posix(), 'sha256': file_digest(path)})
files.sort(key=lambda row: row['path'])
text = json.dumps(files, separators=(',', ':'), ensure_ascii=False)
return files, hashlib.sha256(text.encode()).hexdigest()
tests/support/analysis-efficiency/test_catalog_eval.py
+236
View File
@@ -0,0 +1,236 @@
"""Catalog measurement oracles and missing/failed measurement semantics."""
import copy
import importlib.util
import json
from pathlib import Path
import tempfile
import unittest
from unittest.mock import patch
import io
import sys
import subprocess
spec = importlib.util.spec_from_file_location('catalog_eval', Path(__file__).with_name('catalog-eval.py'))
catalog = importlib.util.module_from_spec(spec)
spec.loader.exec_module(catalog)
class CatalogEvaluationTests(unittest.TestCase):
def test_plan_does_not_spawn_a_model_or_acquire_trial_directories(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
(root / 'dist').mkdir()
catalog.write_json(root / 'dist/mcp-catalog.json', {
'tools': [{'name': 'binary_session', 'inputSchema': {}}]})
output = root / 'unacquired-output'
arguments = ['catalog-eval.py', '--runtime-root', str(root), '--output', str(output),
'--model', 'explicit-test-alias', '--plan']
stream = io.StringIO()
with patch.object(sys, 'argv', arguments), patch('sys.stdout', stream), \
patch.object(catalog.subprocess, 'check_output', return_value='pinned\n'), \
patch.object(catalog, 'runtime_fingerprint', side_effect=AssertionError('Plan must not hash the runtime')), \
patch.object(catalog.subprocess, 'run') as launch:
self.assertEqual(catalog.main(), 0)
launch.assert_not_called()
self.assertFalse(output.exists())
plan = json.loads(stream.getvalue())
self.assertEqual(plan['model'], 'explicit-test-alias')
self.assertEqual({cell['mode'] for cell in plan['matrix']}, {'client-code'})
self.assertEqual({cell['arm'] for cell in plan['matrix']}, {'none', 'all', 'selected'})
def test_unknown_or_invalid_usage_never_becomes_zero(self):
for value in [None, {}, {'input_tokens': 10},
{'input_tokens': True, 'cached_input_tokens': 0, 'output_tokens': 1},
{'input_tokens': 10, 'cached_input_tokens': 11, 'output_tokens': 1},
{'input_tokens': 0, 'cached_input_tokens': 0, 'output_tokens': 1},
{'input_tokens': 10, 'cached_input_tokens': 0, 'output_tokens': -1}]:
with self.subTest(value=value), self.assertRaises(ValueError):
catalog.usage_counts(value)
def test_pairs_with_the_same_mode_and_round_without_hiding_failures(self):
rows = [
{'mode': 'direct', 'arm': 'none', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 100}},
{'mode': 'direct', 'arm': 'all', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 130}},
{'mode': 'client-code', 'arm': 'none', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 1000}},
{'mode': 'client-code', 'arm': 'all', 'repetition': 1, 'status': 'valid', 'usage': {'input_tokens': 1200}},
{'mode': 'direct', 'arm': 'all', 'repetition': 2, 'status': 'valid', 'usage': {'input_tokens': 140}},
{'mode': 'direct', 'arm': 'none', 'repetition': 3, 'status': 'invalid'},
{'mode': 'direct', 'arm': 'all', 'repetition': 3, 'status': 'valid', 'usage': {'input_tokens': 150}},
]
for row in rows:
row['runtime_fingerprint_sha256'] = 'a' * 64
result = catalog.comparison(rows, 'a' * 64)
self.assertEqual([pair['input_token_delta'] for pair in result['pairs']], [30, 200, None, None])
direct = next(group for group in result['groups'] if group['mode'] == 'direct')
self.assertEqual(direct['attempted_pairs'], 3)
self.assertEqual(direct['valid_pairs'], 1)
self.assertEqual(direct['median_input_token_delta'], 30)
def test_rejects_different_builds_even_if_each_trial_is_individually_valid(self):
def row(arm, digest):
return {'mode': 'client-code', 'arm': arm, 'repetition': 1, 'status': 'valid',
'usage': {'input_tokens': 100}, 'runtime_fingerprint_sha256': digest}
for left, right, pin in [('a' * 64, 'b' * 64, 'a' * 64),
('b' * 64, 'b' * 64, 'a' * 64), (None, None, None)]:
result = catalog.comparison([row('none', left), row('all', right)], pin)
self.assertEqual(result['pairs'][0]['status'], 'invalid')
self.assertIsNone(result['groups'][0]['median_input_token_delta'])
def test_matrix_stops_before_more_model_calls_when_another_checkout_rebuilds(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
runtime = root / 'other-checkout'
(runtime / 'dist').mkdir(parents=True)
module = runtime / 'dist/module.js'
module.write_text('build A')
catalog.write_json(runtime / 'dist/mcp-catalog.json', {'tools': [{'name': 'binary_session'}]})
_, pin = catalog.runtime_fingerprint(runtime)
output = root / 'output'
arguments = ['catalog-eval.py', '--runtime-root', str(runtime), '--output', str(output), '--model', 'test-alias']
def launch(command, **options):
directory = Path(command[command.index('--output') + 1])
directory.mkdir()
catalog.write_json(directory / 'summary.json', {
'exit_code': 0, 'timeout': False, 'runtime_bytes_unchanged': True,
'runtime_files_sha256_before': pin, 'runtime_files_sha256_after': pin})
catalog.write_json(directory / 'trial-settings.json', {'catalog_load': True, 'workspace_isolated_from_runtime': True})
catalog.write_json(directory / 'cleanup.json', {'profile_removed': True})
(directory / 'events.jsonl').write_text('\n'.join(json.dumps(event) for event in [
{'type': 'item.completed', 'item': {'type': 'agent_message', 'text': catalog.MARKER}},
{'type': 'turn.completed', 'usage': {'input_tokens': 100, 'cached_input_tokens': 0, 'output_tokens': 8}},
]) + '\n')
options['stdout'].write(json.dumps({
'resource_scope': 'fixture-scope', 'abort': None, 'payload_resource_observation_verified': True,
'payload_resource_observation': {'payload_exit_code': 0, 'payload_cleanup_verified': True,
'cpu_stat': {'usage_usec': 1}, 'memory_peak_bytes': 1},
}) + '\n')
# The completed first trial is valid; the external build happens
# between trials, outside the driver's artifact lock.
module.write_text('build B')
return subprocess.CompletedProcess(command, 0)
with patch.object(sys, 'argv', arguments), patch('builtins.print'), \
patch.object(catalog.subprocess, 'check_output', return_value='fixed\n'), \
patch.object(catalog.subprocess, 'run', side_effect=launch) as spawn:
self.assertEqual(catalog.main(), 1)
self.assertEqual(spawn.call_count, 1)
report = json.loads((output / 'report.json').read_text())
self.assertEqual(report['experiment']['runtime_files_sha256'], pin)
self.assertEqual(report['trials'][0]['status'], 'valid')
self.assertEqual(len(report['unattempted']), 2)
def test_catalog_receipt_must_belong_to_the_pinned_manifest(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
(root / 'dist').mkdir()
source = root / 'dist/mcp-catalog.json'
catalog.write_json(source, {'tools': [{'name': 'binary_session'}]})
original = catalog.runtime_fingerprint
def rebuild_after_pin(runtime):
snapshot = original(runtime)
catalog.write_json(source, {'tools': [{'name': 'binary_session', 'description': 'new build'}]})
return snapshot
output = root / 'unacquired'
arguments = ['catalog-eval.py', '--runtime-root', str(root), '--output', str(output), '--model', 'test-alias']
with patch.object(sys, 'argv', arguments), \
patch.object(catalog.subprocess, 'check_output', return_value='fixed\n'), \
patch.object(catalog, 'runtime_fingerprint', side_effect=rebuild_after_pin), \
patch.object(catalog.subprocess, 'run') as spawn:
with self.assertRaisesRegex(ValueError, 'Catalog changed'):
catalog.main()
spawn.assert_not_called()
self.assertFalse(output.exists())
def test_load_oracle_rejects_actions_failures_and_unverified_cleanup(self):
summary = {'exit_code': 0, 'timeout': False, 'runtime_bytes_unchanged': True,
'runtime_files_sha256_before': 'a' * 64, 'runtime_files_sha256_after': 'a' * 64}
events = [
{'type': 'item.completed', 'item': {'type': 'agent_message', 'text': catalog.MARKER}},
{'type': 'turn.completed', 'usage': {'input_tokens': 100, 'cached_input_tokens': 80, 'output_tokens': 8}},
]
resources = {'resource_scope': 'fixture-scope', 'abort': None,
'payload_resource_observation_verified': True,
'payload_resource_observation': {'payload_exit_code': 0, 'payload_cleanup_verified': True,
'cpu_stat': {'usage_usec': 1500}, 'memory_peak_bytes': 1000}}
with tempfile.TemporaryDirectory() as temporary:
directory = Path(temporary) / 'trial'
directory.mkdir()
def assess(changed_events=None, changed_resources=None, changed_summary=None, cleaned=True, code=0):
catalog.write_json(directory / 'summary.json', summary if changed_summary is None else changed_summary)
catalog.write_json(directory / 'trial-settings.json', {'catalog_load': True, 'workspace_isolated_from_runtime': True})
catalog.write_json(directory / 'cleanup.json', {'profile_removed': cleaned})
(directory / 'events.jsonl').write_text(''.join(
json.dumps(event) + '\n' for event in (events if changed_events is None else changed_events)))
directory.with_suffix('.log').write_text(json.dumps(
resources if changed_resources is None else changed_resources) + '\n')
return catalog.assess_trial(directory, code, 'a' * 64)
valid = assess()
self.assertEqual(valid['status'], 'valid')
self.assertEqual(valid['usage']['uncached_input_tokens'], 20)
for item in ['mcp_tool_call', 'command_execution', 'tool_search', 'error', 'new_unknown_action']:
action = {'type': 'item.completed', 'item': {'type': item}}
self.assertEqual(assess(changed_events=[action, *events])['status'], 'invalid')
self.assertEqual(assess(changed_events=events + [events[-1]])['status'], 'invalid')
self.assertEqual(assess(changed_events=events[:1])['status'], 'invalid')
notice = {'type': 'item.completed', 'item': {'type': 'error', 'message': 'Host disabled; tool dispatch is unavailable'}}
failure = assess(changed_events=[notice, *events])
self.assertIn(notice['item']['message'], failure['reason'])
self.assertEqual(failure['usage']['input_tokens'], 100)
self.assertEqual(failure['resources'], resources)
self.assertTrue(failure['cleanup']['profile_removed'])
failure = assess(changed_events=[{'type': 'error', 'message': 'outer failure'}, *events])
self.assertEqual(failure['usage']['input_tokens'], 100)
self.assertEqual(failure['resource_metrics']['cpu_seconds'], 0.0015)
failure = assess(changed_events=[{'type': 'turn.failed'}])
self.assertNotIn('usage', failure)
self.assertEqual(failure['resources'], resources)
assess()
transcript = directory / 'events.jsonl'
transcript.write_text('not-json\n' + transcript.read_text())
failure = catalog.assess_trial(directory, 0, 'a' * 64)
self.assertEqual(failure['status'], 'invalid')
self.assertEqual(failure['usage']['input_tokens'], 100)
self.assertEqual(failure['resources'], resources)
for event in [{'type': 'turn.failed'}, {'type': 'error'}, None,
{'type': 'item.started', 'item': {'type': 'mcp_tool_call'}},
{'type': 'item.completed', 'item': []}]:
self.assertEqual(assess(changed_events=[event, *events])['status'], 'invalid')
wrong_answer = copy.deepcopy(events)
wrong_answer[0]['item']['text'] = 'unverified facts'
self.assertEqual(assess(changed_events=wrong_answer)['status'], 'invalid')
self.assertEqual(assess(changed_summary={'exit_code': 0, 'timeout': True})['status'], 'invalid')
self.assertEqual(assess(changed_summary={**summary, 'runtime_bytes_unchanged': False})['status'], 'invalid')
self.assertEqual(assess(changed_summary={**summary, 'runtime_files_sha256_before': 'b' * 64,
'runtime_files_sha256_after': 'b' * 64})['status'], 'invalid')
self.assertEqual(assess(cleaned=False)['status'], 'invalid')
self.assertEqual(assess(code=1)['status'], 'invalid')
for key, value in [('abort', 'memory_stop'), ('payload_resource_observation_verified', False),
('payload_resource_observation', {'payload_exit_code': 0, 'payload_cleanup_verified': False})]:
failed = {**resources, key: value}
self.assertEqual(assess(changed_resources=failed)['status'], 'invalid')
def test_built_catalog_keeps_transport_bytes_separate_from_client_selection(self):
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
(root / 'dist').mkdir()
catalog.write_json(root / 'dist/mcp-catalog.json', {
'tools': [{'name': 'one', 'inputSchema': {}, 'outputSchema': {}, 'description': 'one'},
{'name': 'two', 'inputSchema': {}, 'outputSchema': {}, 'description': 'two'}]})
one = catalog.catalog_receipt(root, 'compact', ['one'])
both = catalog.catalog_receipt(root, 'compact', ['one', 'two'])
self.assertEqual(one['bytes'], both['bytes'])
self.assertEqual(one['tool_count'], 2)
self.assertEqual(one['sha256'], both['sha256'])
for selected in [[], ['missing'], ['one', 'one']]:
with self.assertRaises(ValueError):
catalog.catalog_receipt(root, 'compact', selected)
if __name__ == '__main__':
unittest.main()
tests/support/analysis-efficiency/test_codex_context.py
+53
View File
@@ -673,9 +673,62 @@ class ContextCaptureTests(unittest.TestCase):
with self.assertRaisesRegex(OSError, 'fixture capture failure'):
agent.main()
self.assertFalse(list(output.glob('profile-*')))
self.assertFalse(list((root / '.cache/rea-agent-trials').glob('profile-*')))
self.assertTrue(json.loads((output / 'cleanup.json').read_text())['profile_removed'])
self.assertEqual((auth_home / 'auth.json').read_text(), 'synthetic-credential')
def test_artifact_directory_does_not_choose_the_codex_workspace_or_enable_mcp_control(self):
spec = importlib.util.spec_from_file_location('agent_trial', Path(__file__).with_name('agent-trial.py'))
agent = importlib.util.module_from_spec(spec)
spec.loader.exec_module(agent)
for control, drift, mode in [(False, False, 'direct'), (True, False, 'direct'),
(False, True, 'direct'), (False, False, 'client-code')]:
with self.subTest(control=control, drift=drift, mode=mode), tempfile.TemporaryDirectory() as directory:
root = Path(directory)
runtime = root / 'runtime'
(runtime / 'scripts').mkdir(parents=True)
(runtime / 'scripts/rea.mjs').write_text('')
(runtime / 'package-lock.json').write_text('{}')
(runtime / '.nvmrc').write_text('24.22.1\n')
(runtime / 'AGENTS.md').write_text('Foreign project instructions must not reach the trial')
question = root / 'question'
question.write_text('Reply with exactly CATALOG_OK.')
output = runtime / '.cache/trial'
arguments = ['agent-trial.py', '--runtime-root', str(runtime), '--revision', 'fixture',
'--prompt', str(question), '--output', str(output), '--catalog-load', '--mode', mode,
*(['--no-mcp'] if control else [])]
observed = {}
def launch(_invocation, **options):
workspace = options['cwd']
self.assertFalse(workspace.is_relative_to(runtime))
config = Path(options['env']['CODEX_HOME']) / 'config.toml'
observed['config'] = config.read_text()
if drift:
(runtime / 'dist').mkdir()
(runtime / 'dist/changed.js').write_text('Runtime changed during the trial')
child = Mock(stdin=io.StringIO())
child.wait.return_value = 0
return child
with patch.dict(os.environ, {'CODEX_HOME': str(root / 'source-auth')}, clear=True), \
patch.object(sys, 'argv', arguments), \
patch.object(agent.shutil, 'which', return_value='/fixture/node'), \
patch.object(agent.subprocess, 'check_output', return_value='v24.22.1\n'), \
patch.object(agent.subprocess, 'Popen', side_effect=launch), patch('builtins.print'):
with self.assertRaises(SystemExit) as completed:
agent.main()
self.assertEqual(completed.exception.code, 125 if drift else 0)
self.assertEqual('[mcp_servers.rea]' in observed['config'], not control)
self.assertEqual('required = true' in observed['config'], not control)
self.assertNotIn('Use the registered REA MCP server', observed['config'])
self.assertIn('code_mode_host = ' + ('true' if mode == 'client-code' else 'false'), observed['config'])
self.assertNotIn('code_mode =', observed['config'])
self.assertNotIn('code_mode_only =', observed['config'])
self.assertTrue(json.loads((output / 'trial-settings.json').read_text())['workspace_isolated_from_runtime'])
self.assertEqual(json.loads((output / 'summary.json').read_text())['runtime_bytes_unchanged'], not drift)
self.assertFalse(list((root / '.cache/rea-agent-trials').glob('profile-*')))
def test_refuses_foreign_session_directory_and_symlink_records(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)