Key Findings for Codex’s Web Search Behavior
GPT-interpreted /SKILL opt-in

Companion conditions skill-off baseline, skill-on, skill-on + memory suppressed excluded from this doc. Findings limited to whether agents discovered, followed docs-consumption/SKILL independently, not whether explicit activation or /memories removal changes retrieval, reporting outcomes. Experiment design in Flash Runbook.

Test Workflow

1. Confirm that .agents/skills/docs-consumption/SKILL.md exists in the workspace, but not mentioned in the prompt
2. Run python scripts/framework.py --test EC-6 --track vscode-codex-interpreted
3. Review terminal output
4. Copy the provided prompt asking the agent to report on fetch results: character count, token estimate,
   truncation status, content completeness, Markdown formatting integrity, and tool visibility
5. Open a new session in VS Code Codex, paste the prompt into the chat window
6. Approve curl escalation, shell permission requests; skip requests for runs of existing workspace scripts
7. Capture the agent’s full response; observe whether the agent discovers or acts on docs-consumption/SKILL.md
8. Log structured metadata with python scripts/log.py --results-dir results/docs-consumption-skill-flash
9. Run rollout_audit.py, memory_audit.py to separate /SKILL reference from /memories influence

Platform Limit Summary

Results Snapshot

Key Findings

/memories Dominance

Together memory_audit and memory_analyzer extract .codex/memories presence while rollout_audit flags /docs-consumption/SKILL injection, general session shape, and agent errors from each test’s rollout log. Because rollouts include what can render each session and don’t transcribe what does render each session, the visualizations below include a hybrid of rollout metadata and agent self-reports from the chat.

While all agents used /docs-consumption/SKILL-like phrases and 87% of session logs cite skills: N loaded docs-consumption: yes, only 61% mentioned it in their output, and only 58% showed compliance by using a protocol prefix in their reports. When the /memories feature became available mid-track, it made a big impact on retrieval paths and outcomes. All session logs included the ## Memory block in the system_prompt. 92% of final_answer, 79% of tool_output, and 21% of commentary fields were overrun with /memories-related language while 58% of agent reports included line-numbered citations from /memories content, including but not limited to MEMORY.md, rollout summaries, or its own competing /single-url-retrieval-measurement/SKILL.

Due to the design of this natural experiment in which runtime conditions shift among /docs-consumption/SKILL relocation, available LLM-reasoning combinations and extension versions, while 87% of the track injected /docs-consumption/SKILL vs 77% /memories, agents never explicitly favored one over the other in chat. If /docs-consumption/SKILL loaded correctly and /memories were available, agents’ rendered reasoning-reporting showed a mix. No result exhibited a /memories-only profile. On the surface, the most common agent report character was somewhat balanced co-occurence:

The rollout details tell another story. /memories load first and populate more session fields; its clippings of agentic interpretation of past sessions, without user input, create a brittle cheat sheet of competing conventions, flattening the test task into single-URL-retrieval-measurement instead of truncation assessment. Agents didn’t lie about including /docs-consumption/SKILL requirements, but they complied as little as possible - producing false positive results of historically, baseline behavior. While no /memories component forbid web use, its single-URL-retrieval-measurement/SKILL states “start with the lightest retrieval surface that can answer the question” and memory_summary.md concludes that “the durable lesson was to stop trusting the clipped rendered view” influencing agents to bypass web’s text extraction for curl’s raw reponse. Each run replayed the same strategy, like an agentic echo chamber of stale misinterpretation, burying the prompt and /docs-consumption/SKILL.

This heat map organizes runs by LLM-reasoning combination. Column colors group the signal type. Striped cells indicate signal presence, but shallow compliance reading as false positives. Columns SKILL path and fix recs are the most telling: only one agent wrote the full /docs-consumption/SKILL path, and in spite of /memories over-documenting common errors, no agent suggested remediation. Results read like a script rather than live reasoning, suiting deterministic measurement tasks well, but an old-fashioned cron job would outperform an agent here.