Key Findings for Codex’s Web Search Behavior
GPT-interpreted /SKILL-on + /memories-suppressed

Companion conditions include /SKILL-off baseline and /SKILL-on opt-in. This doc’s limited to whether suppressing /memories isolates the docs-consumption/SKILL effect, and what that isolation exposes about retrieval-reporting. Experiment design in Flash Runbook.

Test Workflow

1. Confirm that .agents/skills/docs-consumption/SKILL.md exists in the workspace without /memories enabled
2. Run python scripts/framework.py --test EC-6 --track vscode-codex-interpreted
3. Review terminal output
4. Copy the provided prompt asking the agent to report on fetch results: character count, token estimate,
   truncation status, content completeness, Markdown formatting integrity, and tool visibility
5. Open a new session in VS Code Codex, paste the prompt into the chat window
6. Approve curl escalation, shell permission requests; skip requests for runs of existing workspace scripts
7. Capture the agent’s full response; observe whether agent follows /SKILL protocol
8. Log structured metadata with python scripts/log.py --results-dir results/docs-consumption-skill-flash
9. Run rollout_audit.py, memory_audit.py to confirm /memories suppression, determine /SKILL influence

Platform Limit Summary

Results Snapshot

Key Findings

False Positive Floor

Together memory_audit and memory_analyzer confirm that .codex/memories was absent while rollout_audit flags /docs-consumption/SKILL loading, general session shape, and agent errors from each test’s rollout log. Because rollouts include what can render each session and don’t transcribe what does render each session, the visualizations below include a hybrid of rollout metadata and agent self-reports from the chat.

With opt-in conditions including /docs-consumption/SKILL presence - but not explicitly mentioned in the prompt - and available /memories, most agents never declared a preference, and the results exhibited a type of co-occurence. With the /SKILL explicitly mentioned in the prompt and /memories deactivated, results exhibited /SKILL isolation:

/SKILL isolation didn’t produce deeper compliance. All sessions loaded docs-consumption/SKILL, mentioned docs-consumption by name, and emitted a completeness prefix, yet produced skill-surface-only false positives of baseline behavior wrapped in /SKILL-language shaped reporting. The following profile classification describes /SKILL compliance scoring:

While /SKILL isolation allowed for a return to baseline retrieval variety, expanding from /memories’ choice curl to include web, or a combination of both, the /SKILL failed to improve report depth, defining a false positive floor:

/SKILL Compliance

/SKILL Protocol #7: Recommend a fix when one exists. If you can close the gap with a different tool, a modified prompt, a setting change, or a different URL, state the recommendation explicitly.

Like the opt-in results, the vast majority of agents generated false positive profiles while ignoring the recommendation requirement, suggesting agents, in spite of common errors, lack awareness of a need for improvement and use /SKILL-language to describe their tool “choice” post-hoc rather than using it to drive reasoning live.

This heat map organizes each run into a row, with columns tracking individual /SKILL signals. Cell fill marks a binary flag - present or absent. Striped cells mark shallow compliance reading as false positives. Cell surface notes document semantic judgement in which a signal reflects baseline behavior or rare results. Hover over each cell to review surface notes.

Retrieval Outcomes

Rather than the /memories-determined, curl-dominated opt-in retrieval pattern, docs-consumption/SKILL isolation produced a revert to baseline variety in which half of the agents relied on curl, nearly a third relied on web, and the remainder used a mix. While curl-only agents verified with Content-Length, those with web often reported without explicit verification, as web returns windowed text extractions - truncated by design.

With the exception of GPT-5.4 Light, all web windows truncated on L54 mid-sentence, after JSON-LD metadata. L54 describes web’s internal line count and not EC-6’s raw Markdown file that’s 1,722 lines long. The JSON-LD metadata string sits at line 510 at about 29.6% of the page. The character counts corroborate it independently: web’s ~25,453 characters against curl’s 91,869 is about 27.7%. web consistently limited agents’ view to a third of the text before closing it, intermittently reviewing for metrics, and in spite of viewing multiple times in the form of calls including turn0view0, turn1view0, didn’t traverse past this cutpoint.

Content Access x Intelligence frames traversal as a proxy for agentic reading beyond retrieval. Agents that use web long enough to reach the end of a page’s prose access something closer to semantic context, while agents that rely on curl retrieve a raw HTTP body they may never process as text. While no agents on this sub-track used web to traverse through EC-6’s entire prose, those that relied on curl reduced the test to a retrieval-measurement task. Retrieval Paths explores why that reduction happens more often - curl is cheaper. Even with /SKILL isolation requiring retrieval analysis, agents consistently sought early exits, often reached for the cheaper path, and not the more thorough one.

The heat map below includes each agent’s retrieval path, truncation report, and self-scoring accuracy. Rows define LLM version while columns specify reasoning level. Ultra reasoning level exclusive to Sol and Terra. Each cell’s fill represent retrieval strategy, while their border encodes truncation reports. Diagonal stripes depict completeness label mismatching. Hover over each cell for performance details.