WikifitaGitHub live67e8de5
pesquisa · openai-research/probes

Probes — Methodology, Attempts, and Runtime Validation

The versioned harness probes, the validation methodology, the real captured traces, and the runtime experiments

Baixar raw

Probes — Methodology, Attempts, and Runtime Validation

Part of openai-research. Versioned code at ~/workdir/deepseek-harness/scripts/probes/ (excluded from the harness lint/typecheck because they are hand-run research instruments).

Methodology (hard rules learned)

  1. RE 100% before the first call — a probe only births crystal-clear; failure = go back to RE, never retry on unproven "intermittency" (the inference trap)
  2. Max 3 attempts per cycle, counting complete execution
  3. Manual edits always — no multi-line patch scripts (escape sequences are error-prone)
  4. Primary source before vendor — desktop app > codex-rs > vendors

Inventory

ProbeFunctionResult
openai-codex-link-session.tsSelf-contained OAuth PKCE (browser login), exchange, derived-session tests✅ validated 2×
openai-link-session-cookies.tsSet-Cookie forensics of link-session (__oailb JWT 1h)✅
openai-sentinel-credentials.tsSentinel prepare credential matrix (cookies vs Bearer vs both)✅ persona: chatgpt-paid
openai-codex-authorization.tsSelf-contained probe: preemptive refresh, X-OAI-IS chain, catalog, cross-surface, Codex generation opt-in (A–E)✅ A–E validated
openai-chat-submode-flow.tsFull chat submode flow (warm-up, sentinel harness heartbeat, prepare, conversation)⚠️ STALE — built on the static RE: replicates the web/sentinel route with fingerprint placeholders ('probe', 'probe-build') and web PoW; does NOT match the validated desktop facts (attestation challenge mandatory, effort standard/extended, supported_encodings: ['v1'], hyphenated slugs). Rewrite against chat-submode-protocol/attestation-flow when the standalone needs it
openai-sentinel-credentials.tsSentinel prepare credential matrix⚠️ STALE — fingerprint slots are placeholders (still usable only as a matrix probe of header acceptance; the real fingerprint is in sentinel-fallback-validation)
openai-chat-standalone-a.tsA0 attempt (PKCE manual)⚠️ historical — the A0 line is documented as blocked; superseded by A1 (see flow-definitions)
openai-chat-standalone-a1.tsA1 session-keeper probe⚠️ the validated path is the manual sequence (flow-a-manual-step-by-step); the A1 file still carries the earlier mixed-transport implementation and is a reference, not the current path

The 2026-08-24 cycle — the 3 attempts (now superseded by runtime traces)

Attempt 1: A–E ✅ (codex catalog with plan gating; /me web with Bearer; link-session + session marker; Codex low/high generation with complete x-codex-* and sticky x-codex-turn-state). F ❌ 422 "Invalid conversation body" (minimal body, thinking_effort: low invalid for the web architecture).

Attempt 2: F with web_effort: high ❌ same 422 (minimal message without the site-schema fields).

Attempt 3: F with the full t6r message ❌ same 422.

Root cause (closed with evidence — 2026-08-25 runtime traces)

The three attempts were bypass-patching the body (invalid effort → minimal message → complete message) — the methodological error documented in the RE directive. The post-cycle analysis closed the root cause: the request pipeline was too isolated. In the real desktop app, the prepare is a typing throttle (runs on every keystroke with partial text), and each real send requires the DeviceCheck attestation challenge (which the probes never had). The static RE reconstruction missed:

  • The app_attest_challenge (mandatory, from /ios/attestation_challenge + x-sentinel-dc DeviceCheck token)
  • The client_prepare_state: 'none' on the actual conversation send
  • The supported_encodings: ['v1'] field
  • The hyphenated model slugs (gpt-5-5, not gpt-5.5)
  • The UI effort mapping (standard/extended = Medium/High, not low/medium/high)

These were only discoverable with runtime capture (the bezetacil harness on the real app), which is now the primary validation method going forward.

The validated runtime data (2026-08-25, bezetacil v4)

  • 40 captured /f/conversation* traces (prepares + conversations + the handoff/tool loop)
  • Full attestation flow validated: DeviceCheck addon → x-sentinel-dc → challenge 200 → app_attest_challenge in body
  • The JWT claims decoded (apple amr, connectors scopes, 10-day window)

Reuse audit of the existing probes (2026-08-26, for the flow-A standalone probe)

What each file is good for NOW (values that contradict today's evidence are flagged):

ProbeReusableMust NOT reuse
openai-codex-link-session.tsPKCE + exchange + token-file store + masks (valid; validated 2×)CODEX_CLIENT_VERSION 0.144.1 (today: 0.150.0); the browser login itself is unnecessary when ~/.codex/auth.json exists
openai-codex-authorization.tsrefresh (JSON body, preemptive 5min/8d — exact codex-rs semantics), token store, random device-id file, /me cross-surfaceUA Chrome/136. The X-OAI-IS "wraps every request" claim (desktop: 0 occurrences in real captured headers — it was the web-flow/vendor path); CODEX_CLIENT_VERSION 0.144.1; openai-web-flow.ts reference (file does not exist)
openai-chat-submode-flow.tspipeline skeleton (warm-up → prepare → conversation → SSE)everything protocol: sentinel+PoW path instead of attestation; fingerprint with fake slots (probe/probe-build); UA Chrome/136; sentinel heartbeat as part of the flow (present, but only in fallback) — see flow-a-standalone for the real values
openai-sentinel-credentials.tsmatrix ideafingerprint still placeholder (stale)
openai-link-session-cookies.tsSet-Cookie forensics (self-limited)—

Plan: the flow-A probe is built NEW (no edits to these); it reuses the valid helpers (refresh/token-store/Jar) and the real surfaces; it is not yet executed.

Cost of the cycle

~50 requests accumulated across the day (including the exploratory rounds before the 3-attempt discipline — the methodological error that motivated the rules). No persistent 401/403 on the account after the cycle.

Cross-references