Key Findings for Codex’s Web Search Behavior, GPT-interpreted - Desktop

Test Workflow

1. Run python framework.py --test {test ID} --track codex-interpreted
2. Review terminal output
3. Copy the provided prompt asking agent to report on fetch results: character count, token estimate,
   truncation status, content completeness, Markdown formatting integrity, and tool visibility
4. Open a new session in the Codex desktop app, paste the prompt into the chat window
5. Approve curl escalation and shell permission requests; skip requests for runs of local scripts
6. Capture the agent’s full response; observe the gap between self-report and actual retrieval behavior
   as the interpreted finding
7. Log structured metadata as described in framework-reference.md
8. Ensure results saved to /results/codex-interpreted/results.csv

Platform Limit Summary

Results Details

Content Access x Intelligence

Agentic task completion isn’t a useful signal for page readability. For Codex, retrieval strategy largely influences content accessbility; its web tool returns a rendered text extraction window, but it’s up to the agent to use it and most agent’s didn’t, at least not completely. Agents across this track most often started with web, recognized its limits, and pivoted to curl to complete the task, but curl returns a raw HTTP body whose readability entirely depends on that page’s architecture. For JS-rendered pages, curl delivers app shells with prose absent. Agents rarely distinguished between having fetched a URL and having read it.

The heatmap below encodes retrieval strategy, not task outcome. Rows are reasoning/intelligence levels, with LLM version as a sub-grouping. Columns are URLs ordered by content accessibility difficulty, left to right: static payloads → large static HTML → JS-rendered and/or SPAs where curl returns mostly scaffolding.

While curl is an appropriate choice to calculate metrics for some URLs, a prompt with context-specific questions - summarize a section, locate a specific value in the documentation - may have produced a different signal. This track instead uncovers a proxy: agents that used web long enough to traverse page text completely performed something closer to reading prose, as in, accessed semantic context, but agents that pivoted to curl may have retrieved code they never processed as text.

The column grouping makes the practitioner-relevant question legible: agents working with pages in the left two groups had readable content to process regardless of toolchain. Agents working with pages in the right group - EC-1’s SPA extraction at ~10% of raw, BL-3’s JS-rendered tutorial body absent from every fetch, SC-2’s CSP-nonce-gated app shell - retrieved bytes but perhaps didn’t meaningfully read regardless of intelligence level or method. Depending on LLM-version, intelligence level, and page architecture, the curl-only cells sometimes represent the highest task effort with the lowest content accessibility.

Truncation Analysis

Retrieval Outcomes

Output chars on the web surface aren’t a retrieval ceiling metric, but reflect how far the agent traversed through a line-indexed renderer. Agents wrote-saved a variety of artifacts unprompted in which curl body size was partially observable. Raw tracks intend document precise artifact measurements. Rows below organized by page architecture:
raw files → static HTML → reference/wiki → JS-rendered/SPA