第7章 State and Persistence

The JSONL Rollout Format

{"type":"session_meta","payload":{"id":"0193d5e2-...","timestamp":"2026-04-04T12:34:56Z",...}}
{"type":"turn_context","payload":{"cwd":"/Users/zhangsan/workspace",...}}
{"type":"event_msg","payload":{"UserMessage":{"message":"> add user authentication"}}}
{"type":"response_item","payload":{"role":"assistant","content":[...]}}
{"type":"event_msg","payload":{"TurnComplete":{"turn_id":"turn-1"}}}

Each Codex conversation persists as a rollout file in JSONL (newline-delimited JSON) format. Every line is a complete JSON object representing a single RolloutItem—either session metadata, turn context, an event, or a model response item. This design choice reflects several deliberate trade-offs.

Why JSONL

JSONL emerged from the requirement that conversation history must be human-readable and append-only. When you're debugging a failed agent interaction or inspecting a conversation from months ago, you need to be able to open the file in a text editor and understand what happened. Binary formats like SQLite or Protocol Buffers would require specialized tooling.

The append-only constraint is equally important. Codex writes to rollout files as the conversation progresses, streaming each RolloutItem to disk as it occurs. Appending to a text file is a simple, atomic operation that doesn't require file locking or complex transaction handling. If Codex crashes mid-turn, the worst case is an incomplete final line—not a corrupted database.

🤿 深入底层：The JSONL format enables lazy loading of conversation history. When resuming a thread, Codex reads only the SessionMeta line first, then incrementally parses additional items as needed. For long conversations with thousands of items, this avoids the memory cost of loading everything upfront.

Rollout Item Structure

The RolloutItem enum captures five distinct item types:

#[serde(tag = "type", content = "payload", rename_all = "snake_case")]
pub enum RolloutItem {
    SessionMeta(SessionMetaLine),      // Thread identity and configuration
    TurnContext(TurnContextItem),      // Per-turn execution context
    EventMsg(EventMsg),                // Lifecycle events (turn start/complete)
    ResponseItem(ResponseItem),        // Model API responses
    Compacted(CompactedItem),          // Mid-turn history compression
}

The SessionMeta line always appears first, containing the thread's UUID, creation timestamp, working directory, model provider, and CLI version. The TurnContext items appear once per real user turn, capturing the sandbox policy, approval mode, and model in use. EventMsg items mark turn boundaries and track token usage. ResponseItem and Compacted items carry the actual conversation content.

Trade-offs

The JSONL format's primary disadvantage is the lack of random access. Finding the 500th item requires parsing 499 preceding lines. This is why Codex maintains a separate SQLite database for metadata—fast queries would be impractical against raw JSONL files.

Another trade-off is file size. JSON is verbose compared to binary formats. A rollout with code blocks and large context files can grow to multiple megabytes. However, disk space is cheap compared to the value of human-debuggable history.

Manual Inspection

You can inspect your own rollouts directly:

# List recent sessions
ls -lt ~/.codex/sessions/2026/*/rollout-*.jsonl

# View the first line (session metadata)
head -1 ~/.codex/sessions/2026/04/04/rollout-*.jsonl | jq

# Count rollout items per session
for file in ~/.codex/sessions/2026/*/*.jsonl; do
    echo "$(wc -l < "$file") $file"
done | sort -rn

⚠️ 避坑指南：Never modify a rollout file while Codex is running. The runtime does not detect external changes, and your edits will be overwritten on the next append. Use the read-only view commands or copy the file elsewhere before editing.

SQLite Schema Design

While JSONL files store the complete conversation history, SQLite stores queryable metadata extracted from those files. The state database lives at ~/.codex/state_5.sqlite (versioned to enable schema migrations) and contains three primary tables.

Database Schema

-- Core thread metadata extracted from rollouts
CREATE TABLE threads (
    id TEXT PRIMARY KEY,                    -- Thread UUID
    rollout_path TEXT NOT NULL,             -- Path to JSONL file
    created_at INTEGER NOT NULL,            -- Unix timestamp
    updated_at INTEGER NOT NULL,            -- Last modification time
    source TEXT NOT NULL,                   -- "cli", "exec", "app_server"
    agent_nickname TEXT,                    -- Sub-agent nickname
    agent_role TEXT,                        -- Sub-agent role
    agent_path TEXT,                        -- Canonical agent path
    model_provider TEXT NOT NULL,           -- "openai", "ollama", etc.
    model TEXT,                             -- Latest model used (e.g., "gpt-5")
    reasoning_effort TEXT,                  -- "low", "medium", "high"
    cwd TEXT NOT NULL,                      -- Working directory
    cli_version TEXT NOT NULL,              -- Codex version
    title TEXT NOT NULL,                    -- First user message
    sandbox_policy TEXT NOT NULL,           -- "read-only", "workspace-write"
    approval_mode TEXT NOT NULL,            -- "never", "always", "on-request"
    tokens_used INTEGER NOT NULL DEFAULT 0,
    first_user_message TEXT,                -- Truncated preview
    archived INTEGER NOT NULL DEFAULT 0,
    archived_at INTEGER,
    git_sha TEXT,                           -- Git commit hash if available
    git_branch TEXT,                        -- Git branch name
    git_origin_url TEXT                     -- Git repository URL
);

-- Agent job orchestration for batch processing
CREATE TABLE agent_jobs (
    id TEXT PRIMARY KEY,
    name TEXT NOT NULL,
    status TEXT NOT NULL,                   -- "pending", "running", "completed", "failed"
    instruction TEXT NOT NULL,
    output_schema_json TEXT,
    input_headers_json TEXT NOT NULL,
    input_csv_path TEXT NOT NULL,
    output_csv_path TEXT NOT NULL,
    auto_export INTEGER NOT NULL DEFAULT 1,
    max_runtime_seconds INTEGER,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL,
    started_at INTEGER,
    completed_at INTEGER,
    last_error TEXT
);

-- Individual work items within each job
CREATE TABLE agent_job_items (
    job_id TEXT NOT NULL,
    item_id TEXT NOT NULL,
    row_index INTEGER NOT NULL,
    source_id TEXT,
    row_json TEXT NOT NULL,
    status TEXT NOT NULL,
    assigned_thread_id TEXT,
    attempt_count INTEGER NOT NULL DEFAULT 0,
    result_json TEXT,
    last_error TEXT,
    created_at INTEGER NOT NULL,
    updated_at INTEGER NOT NULL,
    completed_at INTEGER,
    reported_at INTEGER,
    PRIMARY KEY (job_id, item_id),
    FOREIGN KEY(job_id) REFERENCES agent_jobs(id) ON DELETE CASCADE
);

Indexing Strategy

Query performance depends on well-chosen indexes. The threads table uses a composite index pattern optimized for the most common queries—listing threads by creation or update time:

-- Support "recent threads first" ordering
CREATE INDEX idx_threads_created_at ON threads(created_at DESC, id DESC);
CREATE INDEX idx_threads_updated_at ON threads(updated_at DESC, id DESC);

-- Filter by archive status
CREATE INDEX idx_threads_archived ON threads(archived);

-- Filter by session source
CREATE INDEX idx_threads_source ON threads(source);

-- Filter by model provider
CREATE INDEX idx_threads_provider ON threads(model_provider);

The agent_jobs and agent_job_items tables use status-based indexes to enable efficient polling for available work:

-- Find jobs by status, recency
CREATE INDEX idx_agent_jobs_status ON agent_jobs(status, updated_at DESC);

-- Find claimable items within a job
CREATE INDEX idx_agent_job_items_status ON agent_job_items(job_id, status, row_index ASC);

☕ 茶水间时间：You might wonder why the threads index includes id DESC alongside created_at DESC. This creates a deterministic sort order when multiple threads share the same timestamp—which happens often during automated testing or rapid session creation. Without the id tiebreaker, pagination could skip or duplicate threads across page boundaries.

Query Patterns

The database supports three primary query patterns. Time-ordered listing retrieves threads for the TUI and app-server:

SELECT id, title, created_at, updated_at, tokens_used
FROM threads
WHERE archived = 0
  AND first_user_message <> ''
  AND source IN ('cli', 'exec')
ORDER BY created_at DESC, id DESC
LIMIT 51;  -- Fetch one extra to detect if another page exists

Thread lookup by ID finds the rollout path for resuming conversations:

SELECT rollout_path, archived_at
FROM threads
WHERE id = ?
  AND archived = 0;

Sub-agent discovery traverses the thread spawn hierarchy:

WITH RECURSIVE subtree(child_thread_id, depth) AS (
    SELECT child_thread_id, 1
    FROM thread_spawn_edges
    WHERE parent_thread_id = ? AND status = 'open'
    UNION ALL
    SELECT edge.child_thread_id, subtree.depth + 1
    FROM thread_spawn_edges AS edge
    JOIN subtree ON edge.parent_thread_id = subtree.child_thread_id
    WHERE edge.status = 'open'
)
SELECT child_thread_id
FROM subtree
ORDER BY depth ASC, child_thread_id ASC;

Migration System

Schema evolution uses SQLx's migration system. Each migration lives in codex-rs/state/migrations/ with a filename like 0001_threads.sql or 0014_agent_jobs.sql. When Codex starts, it runs any unapplied migrations automatically, then cleans up legacy database files.

The version number in the filename (state_5.sqlite) increments when backward-incompatible schema changes occur. This allows new Codex versions to start fresh while preserving older databases for potential rollback or migration scripts.

The Backfill Pipeline

The journey from JSONL rollout to SQLite row is orchestrated by the backfill pipeline. This asynchronous process scans rollout files, extracts metadata, and populates the state database—all without blocking the main Codex runtime.

Async Scanning

When Codex initializes, it checks whether the state database exists and whether backfill has completed. If not, it spawns a background task via tokio::spawn that runs independently of the main thread:

// codex-rs/rollout/src/state_db.rs
pub async fn init(config: &impl RolloutConfigView) -> Option<StateDbHandle> {
    let runtime = codex_state::StateRuntime::init(...).await?;
    let backfill_state = runtime.get_backfill_state().await?;
    if backfill_state.status != BackfillStatus::Complete {
        tokio::spawn(async move {
            metadata::backfill_sessions(runtime_for_backfill.as_ref(), &config).await;
        });
    }
    Some(runtime)
}

This design ensures that backfill never blocks startup. Even with thousands of rollout files to process, the TUI or app-server becomes available immediately. The database gradually populates in the background, with query results improving as more threads are indexed.

Job Queue

The backfill system uses a lease-based claiming mechanism to ensure only one process processes rollouts at a time. This is critical because Codex might run multiple instances simultaneously—e.g., a TUI session plus an exec-mode job, or multiple IDE connections via app-server.

The backfill_state table tracks the single global state:

CREATE TABLE backfill_state (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    status TEXT NOT NULL,                   -- "pending", "running", "complete"
    last_watermark TEXT,                    -- Last processed file path
    last_success_at INTEGER,
    updated_at INTEGER NOT NULL
);

INSERT INTO backfill_state (id, status, ..., updated_at)
VALUES (1, 'pending', NULL, NULL, CAST(strftime('%s', 'now') AS INTEGER))
ON CONFLICT(id) DO NOTHING;

Claiming the backfill worker slot uses an atomic compare-and-swap operation:

UPDATE backfill_state
SET status = 'running', updated_at = ?
WHERE id = 1
  AND status != 'complete'
  AND (status != 'running' OR updated_at <= ?);

If this query affects exactly one row, the claim succeeded. Otherwise, another process either completed backfill or holds an active lease. The lease expires after 900 seconds (15 minutes) in production, allowing a new worker to take over if the previous one crashed.

Error Handling

Rollout files can be malformed—incomplete JSON, missing required fields, or corrupted by disk errors. The backfill pipeline handles these gracefully:

for rollout in batch {
    stats.scanned += 1;
    match extract_metadata_from_rollout(&rollout.path).await {
        Ok(outcome) => {
            if outcome.parse_errors > 0 {
                metric_client.counter(DB_ERROR_METRIC, outcome.parse_errors, ...);
            }
            runtime.upsert_thread(&outcome.metadata).await?;
        }
        Err(err) => {
            stats.failed += 1;
            warn!("failed to extract rollout {}: {err}", rollout.path.display());
        }
    }
}

Parse errors increment a counter but don't stop processing. A malformed line is skipped, but the rest of the file still contributes to the metadata. This best-effort philosophy means a partially corrupted rollout can still appear in search results, even if some history is lost.

Retry Logic

The backfill process checkpoints progress after each batch of 200 files. If Codex crashes or is interrupted, the next run resumes from the last checkpoint rather than reprocessing everything:

for batch in rollout_paths.chunks(BACKFILL_BATCH_SIZE) {
    for rollout in batch {
        // ... process rollout
        last_watermark = Some(rollout.watermark.clone());
    }
    runtime.checkpoint_backfill(&last_watermark).await?;
}
runtime.mark_backfill_complete(last_watermark.as_deref()).await?;

The last_watermark field stores the rollout file path as a string. Since filenames contain timestamps (rollout-2026-04-04T12-34-56-0193d5e2...jsonl), this provides a natural ordering that determines which files still need processing.

Search Enablement

Once backfill completes, the SQLite database becomes the primary source for thread listing and search. The TUI's thread browser queries threads directly instead of scanning the filesystem. This reduces startup time from seconds (directory traversal + JSON parsing) to milliseconds (indexed SQL query).

Search across thread titles and first user messages uses SQLite's instr function for case-insensitive substring matching:

SELECT id, title, first_user_message, created_at
FROM threads
WHERE archived = 0
  AND instr(title, ?) > 0
ORDER BY created_at DESC, id DESC
LIMIT 51;

This is fast even with thousands of threads, but it's still a full table scan for the instr filter. Future versions might add full-text search via FTS5 for better performance on large installations.

Thread Management and Lifecycle

The ThreadManager tracks active conversations in memory, coordinating creation, forking, and archival. Each thread has a corresponding CodexThread object that holds the runtime state, while the rollout file and SQLite row provide durable persistence.

Thread Creation

When you start a new conversation, ThreadManager::spawn_thread creates a Codex agent instance and waits for the first event—the SessionConfigured message that contains the new thread's UUID and rollout path:

let CodexSpawnOk { codex, thread_id, .. } = Codex::spawn(...).await?;
let event = codex.next_event().await?;
let session_configured = match event {
    Event { msg: EventMsg::SessionConfigured(cfg), .. } => cfg,
    _ => return Err(CodexErr::SessionConfiguredNotFirstEvent),
};

let thread = Arc::new(CodexThread::new(
    codex,
    session_configured.rollout_path.clone(),
    watch_registration,
));
threads.insert(thread_id, thread.clone());

The thread is immediately registered in the manager's internal HashMap and persists its SessionMeta line to the rollout file. SQLite metadata is populated later by the backfill pipeline or via incremental updates as items are appended.

⚠️ 避坑指南：The SessionConfigured event must be the first event from a spawned Codex instance. This invariant is enforced by the match statement—if any other event arrives first, the thread creation fails with SessionConfiguredNotFirstEvent. This prevents creating threads with indeterminate identity or missing configuration.

Forking

Thread forking creates a new conversation that starts from a snapshot of an existing thread's history. Codex supports two fork modes via the ForkSnapshot enum:

pub enum ForkSnapshot {
    TruncateBeforeNthUserMessage(usize),  // Cut before nth user message
    Interrupted,                           // Cut now, simulating interrupt
}

The TruncateBeforeNthUserMessage mode extracts a committed history prefix, useful for branching a conversation at a specific point. The Interrupted mode appends a synthetic <turn_aborted> marker, as if the source thread were interrupted at that moment.

Forking operates entirely on the persisted rollout history. The new thread receives a fresh UUID and rollout file, but its initial items are copied from the source:

let history = RolloutRecorder::get_rollout_history(&path).await?;
let snapshot_state = snapshot_turn_state(&history);
let history = match snapshot {
    ForkSnapshot::TruncateBeforeNthUserMessage(n) => {
        truncate_before_nth_user_message(history, n, &snapshot_state)
    }
    ForkSnapshot::Interrupted => {
        // Append turn_aborted marker if mid-turn
        if snapshot_state.ends_mid_turn {
            append_interrupted_boundary(history, snapshot_state.active_turn_id)
        } else {
            history
        }
    }
};

The fork's SQLite row records the parent-child relationship in the thread_spawn_edges table, enabling the hierarchical queries shown earlier.

Archival

Threads are archived rather than deleted. Archival moves the rollout file from sessions/ to archived_sessions/ and updates the SQLite row's archived_at timestamp:

pub async fn mark_archived(
    &self,
    thread_id: ThreadId,
    rollout_path: &Path,
    archived_at: DateTime<Utc>,
) -> anyhow::Result<()> {
    let Some(mut metadata) = self.get_thread(thread_id).await? else {
        return Ok(());
    };
    metadata.archived_at = Some(archived_at);
    metadata.rollout_path = rollout_path.to_path_buf();
    self.upsert_thread(&metadata).await
}

Archived threads are excluded from default thread listings but remain searchable. The archived flag acts as a soft delete—you can recover an archived thread by setting archived_at back to NULL.

ThreadManager API

The ThreadManager provides a clean API for thread lifecycle operations:

// Create a fresh thread
thread_manager.start_thread(config).await?

// Resume from an existing rollout
thread_manager.resume_thread_from_rollout(config, rollout_path, ...).await?

// Fork with snapshot control
thread_manager.fork_thread(ForkSnapshot::TruncateBeforeNthUserMessage(3), ...).await?

// Retrieve an active thread
thread_manager.get_thread(thread_id).await?

// Graceful shutdown with timeout
thread_manager.shutdown_all_threads_bounded(Duration::from_secs(30)).await

Shutdown is bounded—each thread gets a fixed timeout to flush pending writes and close resources. Threads that complete shutdown are removed from the manager; those that time out remain tracked for potential retry.

Thread States

Each CodexThread progresses through these conceptual states:

1. Initializing: Agent is spawning, waiting for SessionConfigured
2. Idle: No active turn, waiting for user input
3. Active: Processing a turn, streaming model responses
4. Interrupted: Turn was aborted (user cancellation or error)
5. Shutdown: Thread is closing, writing final items

The SQLite threads table doesn't directly track these states—they're transient runtime information. However, the combination of archived flag, updated_at timestamp, and presence in the ThreadManager's internal map provides enough information to infer whether a thread is currently active, idle, or already closed.

---

With core systems understood—configuration, tools, skills, and persistence—we're ready to zoom in on specific components. The next part examines the heart of Codex CLI: the core agent orchestration that weaves these systems together into a coherent coding agent.

Chapter Exercises

1. Query the state database: Use sqlite3 ~/.codex/state_5.sqlite to find all threads created in the last 24 hours with tokens_used > 10000.

2. Inspect a rollout file: Find your most recent rollout and examine the first 20 lines. Can you identify each RolloutItem type?

3. Trigger backfill observation: Delete your state_5.sqlite file and restart Codex. Watch the logs for backfill progress messages.