Building a Test Harness to test AI Produced Code

Two column map linking each architecture layer to a Microsoft or GitHub tool from Copilot to Azure AI Foundry

Why Coding Agents Can’t Validate Their Own Output (And How to Fix the 80/20 Inversion)

A passing unit test does not prove a feature works. It only proves a function returned the value its author anticipated—it says nothing about whether a modal renders, a button is clickable, or a user can complete a transaction.

Coding agents are probabilistic text generators, not visual, stateful execution runtimes. Asking an agent to evaluate its own code creates an immediate conflict of interest: a player cannot act as the referee.

1. The Three Failure Modes of Agent Self-Validation

The Context Bottleneck: Real-world testing requires institutional memory, edge-case histories, team conventions, and framework quirks. Attempting to fit years of enterprise testing context into an LLM context window causes hallucination rather than reasoning.
No Hands and No Eyes: Language models output text. They cannot natively click, scroll, inspect rendered DOM elements, or detect collapsed CSS layouts in a live browser runtime.
Brittle Script Generation: When agents write automated test scripts directly, they generate hardcoded selectors and arbitrary timeouts. Minor UI updates break these tests immediately, causing massive maintenance debt.

2. The Inverted SDLC: Verification is the New Bottleneck

Traditional software development spent roughly 80% of effort on development and 20% on validation. Because code creation was slow, validating human output was manageable.

AI agents have inverted this ratio:

Code Generation (20%): Instantaneous and cheap.
Code Validation (80%): The primary operational bottleneck, requiring continuous verification against high-volume, machine-generated pull requests.

Traditional SDLC: [========= Development (80%) =========] [== Testing (20%) ==]
AI-Era SDLC:      [== Development (20%) ==] [========= Validation (80%) =========]

3. Reference Architecture: The 4-Stage Agentic Quality Pipeline

To reliably ship machine-generated code, organizations must place a deterministic validation layer between code generation and production deployment.

Code snippet

graph LR
    classDef stage fill:#F8FAFC,stroke:#0F172A,stroke-width:2px,color:#0F172A;
    classDef loop fill:#FFFFFF,stroke:#2563EB,stroke-width:2px,stroke-dasharray: 5 5;

    subgraph PIPELINE [Agentic Quality Lifecycle]
        direction LR
        P1["1. Planning<br>(Requirement to Intent Matrix)"]:::stage
        P2["2. Authoring<br>(Deterministic Intent & CLI Validation)"]:::stage
        P3["3. Execution<br>(HyperExecute Across Real Browsers/Devices)"]:::stage
        P4["4. Analysis<br>(AI Error Classification & Auto-Healing)"]:::stage

        P1 --> P2 --> P3 --> P4
    end

    P4 -.->|Insights & Flakiness Signals| P1

1. Planning: Translates user stories, tickets, and PRDs into structured test coverage matrices.
2. Authoring: Converts plain-English intents into executable browser actions without hardcoded selectors.
3. Execution: Distributes tests across real browsers, mobile emulators, API endpoints, and accessibility grids.
4. Analysis: Classifies errors, eliminates flaky test noise, and feeds runtime telemetry back into the planning stage.

4. Operationalizing Deterministic Validation: Kane CLI

Implementations like Kane CLI (developed by TestMu AI) bridge the gap between probabilistic agents and physical browser runtimes through intent-based automation:

Vision-Based Waiting: Replaces brittle sleep timers by visually waiting for DOM stability and rendered elements before executing subsequent actions.
Dynamic Auto-Healing: Rebuilds broken selectors on the fly using role- and type-based locators when UI layouts shift.
Playwright & Test.md Artifacts: Translates exploratory natural language runs into standard Playwright test files and human/agent-readable Markdown (Test.md) specifications.
Machine-Readable Event Streams: Emits structured newline-delimited JSON (ndjson) so external coding agents can programmatically parse test outcomes.

+-----------------------------------------------------------------------------+
|                            KANE CLI EXECUTION MODES                         |
+-----------------------------------------------------------------------------+
|                                                                             |
|  [ 1. INTERACTIVE TUI ]       [ 2. HEADLESS CI RUN ]    [ 3. AGENT SKILL ]  |
|  Human developer cockpit      Automated PR gating       Inline tool for     |
|  Live browser exploration     Standard exit codes       Claude Code, Codex, |
|  Real-time step streaming     CI/CD pipeline runs       Gemini, & Cursor    |
|                                                                             |
+-----------------------------------------------------------------------------+

Key Strategic Takeaway

Agents generate output; deterministic environments produce proof. By separating the creative generator (the coding agent) from the deterministic validator (the browser execution harness), engineering teams can safely scale automated development without drowning in unverified code.