Key Findings for Codex’s Web Search Behavior, GPT-interpreted - Extension

Test Workflow

1. Run python scripts/framework.py --test {test_id} --track vscode-codex-interpreted
2. Review terminal output
3. Copy the provided prompt asking agent to report on fetch results: character count, token estimate,
   truncation status, content completeness, Markdown formatting integrity, and tool visibility
4. Open a new session in VS Code Codex, paste the prompt into the chat window
5. Approve curl escalation and shell permission requests; skip requests for runs of local scripts
6. Capture the agent’s full response; observe the gap between self-report and actual retrieval behavior
   as the interpreted finding
7. Log structured metadata as described in framework-reference.md
8. Ensure results saved to /results/vscode-codex-interpreted/results.csv

Platform Limit Summary

Results Details

*Three-LLM roster reflects OpenAI’s retirement of GPT-5.2, GPT-5.3-Codex, GPT-5.4 between tracks; GPT-5.4 reappeared for EC tests, analysis in LLM Retirement.

Truncation + Inaccessibility

As in T1, agentic task completion isn’t a meaningful signal for page readability on T2. Retrieval strategy still governs content accessibility more than reasoning level does: web returns a line-indexed rendered extraction, and it’s up to the agent to paginate through the prose or escalate past it, which most agents eventually did, but not consistently.

T2 diverges from T1 in how tightly the web ceiling holds across LLMs. EC-6’s identical L54 cutpoint across 10/13 runs, and SC-2, SC-3’s comparably tight L139-140 and L353 clusters, show web setting ceilings by page structure or a fixed extraction default as much as LLM identity, suggesting a sharpening, not overturning, of T1’s LLM-dependent window finding, most explicit from OP-1 and OP-2, which both still exhibit per-LLM splits across T2.

The same three-tier grouping from T1 split results regarding content accessibility. EC-3, BL-2, EC-6, SC-4, and SC-1 remain readable static payloads where either retrieval path returns usable prose. BL-1, OP-2, OP-1, SC-3, and OP-4 are large static HTML where web truncates, but curl consistently returns coherent responses. The JS-rendered or SPAs of EC-1, BL-3, and SC-2 curl responses include scaffolding rather than prose regardless of tool sophistication or reasoning level. BL-3’s specific URL changed between tracks after the original’s retirement, but the replacement lands in the identical accessibility tier, confirming that the JS-rendered failure mode isn’t tied to one specific page.

The heat map below encodes truncation tier, not retrieval path. Rows are reasoning level, with LLM version as a sub-grouping. GPT-5.4 only has data for EC tests and renders as empty cells elsewhere. Content accessibility difficulty determines column order, mirroring T1’s heat map.

Truncation Analysis

Retrieval Paths

This heat map encodes retrieval path built from tools_named cross-checked against output_chars and, where needed, the source screenshot cross-checked against audits of session rollout JSON logs. A tool listed as attempted only counts if the session data traces back to it, as tools_named is agent self-reported and agents intermittently under-report. The primary retrieval method defines the cell category; cell notes include rare ~10 instances in which agents paginated through web text extractions. Column notes include more comprehensive toolchains, including common verification of fetched responses and failed methods.

Tool paths identify default capacity and configuration discoverability gaps. Despite workspace access to AET documentation and research concerns, an advantage T1 agents on the isolated Codex desktop app didn’t have, T2 agents of all reasoning levels still rediscover default limitations each session rather than drawing on that access, and don’t advise ways to improve fetch quality through in-house features. While agents relied on Codex’s fetch mechanism web, they often reasoned it unsuitable for fetching and describing web content, pivoting to curl; paginating through text-extraction slices is expensive compared to tapping a server for content-length. Multiple extension upgrades across the test cycle didn’t change this pattern or improve report quality. Workspace access, version updates, and reasoning level all failed to move agents toward exposing debugging or sharing product knowledge that may help a user learn Codex.

Retrieval Outcomes

Output chars on the web surface remain an unsuitable retrieval ceiling metric, but reflect how far agents traversed through a line-indexed viewport before stopping or changing strategy. T2 artifact write-saves were less consistent than T1. Several runs wrote files without reporting them, a pattern documented across SC-1 through SC-4, contributing to continued abstraction of agent activity across Codex testing. Rows below organized by page architecture:
raw files → static HTML → reference/wiki → JS-rendered/SPA