VibeAIHub

Openclaw · EN · Part 4

Ch 13 Testing And Quality Assurance

13-ch-13-Testing-and-Quality-Assurance.md

第13章 Testing and Quality Assurance

"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."

— Brian Kernighan

OpenClaw's test suite numbers 3,500 test files spanning fifty-eight separate Vitest configurations. This isn't an accident—it's the foundation that allows twenty messaging channels, hundreds of plugins, and a distributed Gateway architecture to move forward without breaking. When you add a feature that affects Telegram, Discord, the CLI, and three provider plugins simultaneously, you need confidence that nothing else quietly collapsed.

This chapter guides you through OpenClaw's testing philosophy, from writing your first unit test to navigating the performance guardrails that keep the test suite runnable. You'll learn why tests are colocated with source, how to structure tests that survive refactoring, and when to reach for Docker-based E2E validation.

Testing Architecture Overview

OpenClaw uses Vitest as its test framework with V8 coverage collection. The 70% coverage threshold is enforced through CI gates but the architecture goes deeper: tests are organized into layers (unit, integration, E2E), colocated with their source, and split across domain-specific configurations.

The architecture follows three principles:

  1. Colocation: Tests live next to the code they test (src/gateway/protocol/index.test.ts tests index.ts). This keeps tests in sync during refactors and makes the test surface discoverable.

  2. Domain partitioning: Instead of one monolithic test configuration, OpenClaw uses fifty-eight configs (vitest.gateway.config.ts, vitest.channels.config.ts, etc.) each scoped to a domain. This allows targeted execution and faster CI parallelization.

  3. Isolation control: Most configs use isolate: false for performance (workers share module state) while hot paths like gateway and channels use explicit forks pools for true isolation.

Rendering diagram...

Coverage Thresholds

The V8 provider enforces 70% thresholds for lines, functions, and statements (55% for branches). These are documented in vitest.shared.config.ts:

// [ch-13-code1] from vitest.shared.config.ts
coverage: {
  provider: "v8" as const,
  reporter: ["text", "lcov"],
  all: false,
  thresholds: {
    lines: 70,
    functions: 70,
    branches: 55,
    statements: 70,
  },
  include: ["./src/**/*.ts"],
  exclude: [
    `${BUNDLED_PLUGIN_ROOT_DIR}/**`,
    "apps/**",
    "ui/**",
    "test/**",
    "src/**/*.test.ts",
    // ... more exclusions
  ],
}

🤿 深入底层：The all: false setting means coverage only counts files that are actually exercised by tests, not every file in src/. This is intentional—OpenClaw's coverage surface is the "critical path" code, not the entire codebase. New exclusions should be justified through the same lens.

Test Layers

OpenClaw's tests organize into three layers:

  1. Unit tests (*.test.ts): 3,500+ files testing individual functions, classes, and modules. Run via pnpm test or pnpm test:fast (unit-only).

  2. Integration tests: Domain-specific configs test component interactions (vitest.gateway.config.ts covers gateway-protocol, server-methods, and client integration).

  3. E2E tests (*.e2e.test.ts): Docker-based tests running live Gateway instances, real channels, and actual provider calls. Run via pnpm test:docker:all or individual e2e scripts.

Writing Effective Tests

The structure of an OpenClaw test follows Vitest conventions with project-specific patterns for fixtures, mocking, and cleanup. Let's walk through a well-structured test:

// [ch-13-code1] Well-structured test example
import { describe, expect, it, beforeEach, afterEach } from "vitest";
import { createExpiringMapCache, resolveCacheTtlMs } from "./cache-utils.js";

describe("resolveCacheTtlMs", () => {
  it("accepts exact non-negative integers", () => {
    expect(resolveCacheTtlMs({ envValue: "0", defaultTtlMs: 60_000 })).toBe(0);
    expect(resolveCacheTtlMs({ envValue: "120000", defaultTtlMs: 60_000 })).toBe(120_000);
  });

  it("rejects malformed env values and falls back to the default", () => {
    expect(resolveCacheTtlMs({ envValue: "0abc", defaultTtlMs: 60_000 })).toBe(60_000);
    expect(resolveCacheTtlMs({ envValue: "15ms", defaultTtlMs: 60_000 })).toBe(60_000);
  });
});

describe("createExpiringMapCache", () => {
  let now = 1_000;
  const clock = () => now;
  
  beforeEach(() => {
    now = 1_000; // [1] Reset shared state
  });

  it("expires entries on read after the TTL", () => {
    const cache = createExpiringMapCache<string, string>({ // [2]
      ttlMs: 5_000,
      clock, // [3] Inject test clock
    });

    cache.set("alpha", "a");
    expect(cache.get("alpha")).toBe("a");

    now = 6_001; // [4] Advance time
    expect(cache.get("alpha")).toBeUndefined();
    expect(cache.size()).toBe(0);
  });
});

Key patterns:

  • [1] Reset shared state in beforeEach to avoid test pollution
  • [2] Type the generic parameters explicitly for better documentation
  • [3] Inject test doubles (clock, random, etc.) as constructor options
  • [4] Advance test time deterministically rather than using real delays

Test Helpers and Fixtures

The test/helpers/ directory provides reusable test infrastructure:

HelperPurpose
withTempHome()Isolated $HOME directory with auto-cleanup
createCapturedIo()Capture stdout/stderr without affecting global streams
spawnGatewayInstance()Launch a real Gateway for E2E tests
connectGatewayClient()Connect a WebSocket client with auth

Example using withTempHome:

import { describe, expect, it } from "vitest";
import { withTempHome } from "../../test/helpers/temp-home.js";
import { readConfig } from "./io.js";

describe("config I/O", () => {
  it("writes and reads config in isolated home", async () => {
    await withTempHome(async (home) => {
      const configPath = `${home}/.openclaw/openclaw.json`;
      await writeConfig(configPath, { gateway: { port: 18789 } });
      
      const config = await readConfig(configPath);
      expect(config.gateway?.port).toBe(18789);
    });
  });
});

⚠️ 避坑指南：Never mix await import(...) and static import for the same module. If you need lazy loading, create a dedicated *.runtime.ts boundary file and import that instead. Mixing import styles breaks Vitest's module caching and causes intermittent test failures.

Mocking Strategies

OpenClaw uses Vitest's vi.mock() hoists for stable module mocking. The pattern in test/setup.shared.ts shows how to mock external dependencies:

// test/setup.shared.ts
vi.mock("@mariozechner/pi-ai", async () => {
  const original = await vi.importActual<typeof import("@mariozechner/pi-ai")>("@mariozechner/pi-ai");
  return {
    ...original,
    getOAuthApiKey: () => undefined,
    getOAuthProviders: () => [],
    loginOpenAICodex: vi.fn(),
  };
});

Key rules:

  • Mock heavy dependencies once in setup files
  • Use vi.importActual() to preserve real exports you don't want to mock
  • Never mock openclaw/plugin-sdk/* barrels in hot tests—create local runtime seams instead

Running Tests Locally and in CI

Basic Execution

The primary test command runs all suites:

pnpm test              # Run all tests
pnpm test:fast         # Unit tests only
pnpm test:coverage     # With coverage report

Targeted execution uses the native root-project entrypoint:

pnpm test src/gateway/protocol/index.test.ts        # Single file
pnpm test src/gateway/protocol/ -t "additionalProperties"  # Filter by name

Live Tests

Live tests require real API keys and validate against actual providers:

OPENCLAW_LIVE_TEST=1 pnpm test:live
# Or specific domains:
OPENCLAW_LIVE_TEST=1 pnpm test:live -- src/agents/auth-profiles/anthropic.setup-token.live.test.ts

Live tests default to quiet mode—keep [live] progress markers but suppress profile chatter. Full logs:

OPENCLAW_LIVE_TEST_QUIET=0 OPENCLAW_LIVE_TEST=1 pnpm test:live

☕ 茶水间时间：Live tests are deliberately opt-in (OPENCLAW_LIVE_TEST=1) because they burn actual API quota and may expose rate limits in CI logs. When contributing to OpenClaw, run live tests before pushing if you touched auth, provider selection, or model payload construction.

Docker-based E2E Tests

Docker E2E tests validate the full stack: build → install → run real commands → assert results:

# All E2E tests (builds Docker image, runs live gateway, onboarding, etc.)
pnpm test:docker:all

# Individual E2E suites:
pnpm test:docker:live-models      # Real provider calls
pnpm test:docker:live-gateway     # Gateway server with real nodes
pnpm test:docker:onboard          # Installation + onboarding flow
pnpm test:docker:plugins          # Plugin installation contracts

These tests live in scripts/e2e/*.sh and use Docker to ensure clean environments:

# scripts/e2e/onboard-docker.sh (excerpt)
docker run --rm \
  -v "$PWD:/repo" \
  -e OPENCLAW_E2E_MODELS="$OPENCLAW_E2E_MODELS" \
  openclaw-e2e \
  bash -c "cd /repo && pnpm install && pnpm build && openclaw onboard"

CI Configuration

GitHub Actions workflows run tests in parallel across multiple jobs:

# .github/workflows/test.yml (simplified)
jobs:
  test:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]  # Split tests across 4 runners
    steps:
      - uses: actions/checkout@v4
      - run: pnpm install
      - run: pnpm test --shard=${{ matrix.shard }}/4

CI uses fixed worker counts (ciWorkers = isWindows ? 2 : 3) and runs coverage on every push to main.

Test Performance and Guardrails

OpenClaw's 3,500 test files can take hours if run naively. The test suite uses aggressive performance guardrails to keep runtimes reasonable:

Module Loading Patterns

The most common performance killer is heavy module loading in test loops. Avoid this pattern:

// ❌ BAD: Reloads heavy module every test
beforeEach(async () => {
  vi.resetModules();
  await import("../../src/plugins/manifest.js");
});

Instead, import once and reset mocks:

// ✅ GOOD: Import once, reset mocks per test
let loadPluginManifest: typeof import("../../src/plugins/manifest.js").loadPluginManifest;

beforeAll(async () => {
  const module = await import("../../src/plugins/manifest.js");
  loadPluginManifest = module.loadPluginManifest;
});

beforeEach(() => {
  vi.clearAllMocks();
});

vi.mock() Usage

Static vi.mock() hoists are preferred over vi.doMock() for heavy modules. Hoisted mocks are evaluated once at module load time:

// ✅ GOOD: Hoisted mock (evaluated once)
vi.mock("../../src/infra/device-identity.js", () => ({
  loadOrCreateDeviceIdentity: vi.fn(() => mockDeviceIdentity),
}));

// ❌ BAD: Dynamic mock (evaluated every call)
beforeEach(() => {
  vi.doMock("../../src/infra/device-identity.js");
});

⚠️ 避坑指南：Never pair vi.mock() hoists with vi.resetModules() + fresh await import() in beforeEach. This forces Vitest to pay the module graph cost per test. Import heavy modules once in beforeAll, then reset only the mocks/runtime state in beforeEach.

Hot Test Identification

OpenClaw tracks test import durations to identify hot tests:

pnpm test:perf:imports    # Report import time per test
pnpm test:perf:hotspots   # List the slowest test files

Output example:

[src/gateway/server-methods/agent.test.ts] 450ms import
[src/agents/pi-embedded-runner.test.ts] 380ms import
[src/plugins/registry.test.ts] 320ms import

Tests above ~200ms import time are candidates for refactoring:

  1. Extract heavy runtime code behind *.runtime.ts boundaries
  2. Mock the boundary instead of importing real implementation
  3. Split large test files into focused suites

Worker and Memory Management

The default worker count derives from host capabilities:

// From vitest.shared.config.ts
function resolveLocalVitestMaxWorkers(env, system, pool, processStats) {
  const cpuCount = system.cpuCount ?? 1;
  const totalMemoryGb = system.totalMemoryBytes / 1024 ** 3;
  
  let inferred = cpuCount <= 4 ? 1 
                : cpuCount <= 8 ? 2 
                : cpuCount <= 12 ? 3 
                : cpuCount <= 16 ? 4 
                : 6;
  
  // Reduce for memory-constrained hosts
  if (totalMemoryGb <= 16) inferred = Math.min(inferred, 2);
  
  // Throttle if system is already busy
  if (processStats.otherVitestWorkerCount >= 2) {
    return { maxWorkers: 1, fileParallelism: false, throttledBySystem: true };
  }
  
  return { maxWorkers: inferred, fileParallelism: true, throttledBySystem: false };
}

Manual override:

OPENCLAW_VITEST_MAX_WORKERS=1 pnpm test  # Force single worker
OPENCLAW_VITEST_MAX_WORKERS=8 pnpm test  # Force 8 workers (max)

Performance Refactoring Example

The gateway tests were a notorious hotspot. The fix involved extracting a gateway-runtime.ts seam:

// src/gateway/test-helpers.runtime.ts (new boundary)
export async function spawnGatewayForTest(config: GatewayConfig) {
  // Heavy runtime: spawn process, wait for port, establish WebSocket
  const gateway = spawn("node", ["dist/index.js", "gateway", ...]);
  await waitForPort(config.port);
  return gateway;
}

// src/gateway/server.test.ts (refactored)
import { spawnGatewayForTest } from "./test-helpers.runtime.js";
vi.mock("./test-helpers.runtime.js"); // [1] Mock the heavy runtime

it("handles node connect", async () => {
  vi.mocked(spawnGatewayForTest).mockResolvedValue(mockGateway); // [2]
  // Test logic...
});

Before: 8 seconds for 10 tests. After: 400ms. The heavy runtime code still gets exercised in *.e2e.test.ts files—unit tests just no longer pay the cost.

Chapter Exercises

  1. Write a unit test for src/config/cache-utils.ts adding a new clear() method that expires all entries. Follow the existing test structure with a fake clock.

  2. Run live tests with real API keys for your configured provider. Set OPENCLAW_LIVE_TEST=1 and run pnpm test:live -- src/agents/auth-profiles/. Note any failures and investigate whether they're credential issues or test bugs.

  3. Identify a test performance issue by running pnpm test:perf:imports. Pick a test over 200ms import time and create a *.runtime.ts seam that allows unit tests to mock the heavy code while preserving E2E coverage.

With testing understood—the philosophy, the patterns, the performance guardrails—you have the foundation to contribute confidently. Tests aren't a gate; they're a safety net that lets the project move fast without breaking. Next, we'll explore the full contribution workflow: how to find issues, write PRs, and get your changes merged into main.