Closing the Loop on Harness Engineering

Up until very recently, improving my agent has been the same loop over and over again: Make small changes to the harness (prompt changes, tool changes), run the agent again, inspect what happened, repeat.

Removing the human doesn’t require one magical autonomous agent. It requires turning each of the things the human does in the loop into an interface an agent can use.

I wanted to remove myself from that loop.

The Loop

Let’s go back to the loop: there are multiple points that need to be handled before we can get ourselves out of the loop.

In the diagram, I have the basic loop that consists of 4 phases (no matter if a human is involved).

1. First is running the agent, which is self-evident, in order to test the agent, we need to run it.

2. Then after the run has completed, inspect its output.

3. Then based on the output, decide whether we are done or the agent passed our judgement.

4. Finally, if the agent did not work correctly, start fixing the harness.

Giving the Agent a Way to Run

The first requirement is simple: the coding agent needs a way to invoke the system it is working on.

I don’t want the testing framework to own my agent implementation or just wrap the model calls. I want to run real systems.

In Apo, the boundary is the adapter. So the responsibility shifts, I give an interface, the target system implements it. This is through a fixed lifecycle: initialize to load your inputs, startSession to open the session, where you then implement how turns in the system are handled.

import { defineAdapter } from "@apo-ai/sdk/agent-task";
import { z } from "zod";

export const myAdapter = defineAdapter({
name: "my-agent",
deliverables: { result: z.string() },
async startSession(ctx) {
  return {
    async sendUserTurn(turn, { trace, parentSpanId }) {
      const response = await runMyAgent(String(turn), { trace, parentSpanId });
      return { response };
    },
  };
},
async collectDeliverables(ctx) {
  return { result: /* mine the session state */ "" };
},
});

What this achieves is that Apo can actually run your real system at the level you wish. The adapter is a prerequisite for running your system. What is interesting is when we connect it to the CLI:

Now, we can just give the CLI to the agent and Voilà, the agent can run your system without touching the UI and the browser.

Giving the Agent a Way to Inspect

In order for the agent to work on its own, running is not sufficient. The agent also needs to know what happened in the run!

There are already mature solutions for tracing and observability for the agent like Langfuse, what we have in Apo are similar concepts for tracing.

The important part is not building another tracing system. What we want is that our agents have the trace available. Apo consumes OpenTelemetry data during the run, and exposes the resulting trace through the CLI.

import { generateText } from "ai";
import { createOpenAI } from "@ai-sdk/openai";
import { registerApoTracing } from "@apo-ai/sdk/agent-task";

// Register once at module load — idempotent.
await registerApoTracing();

const client = createOpenAI({ apiKey, baseURL: "https://openrouter.ai/api/v1" });

async sendUserTurn(turn, { trace, parentSpanId }) {
  const result = await generateText({
    model: client.chat("google/gemini-2.5-flash-lite"),
    system: SYSTEM_PROMPT,
    messages,
    tools,
    experimental_telemetry: { isEnabled: true }, // ← that's it
  });
  return { response: result.text };
}

Once the telemetry flows, the run records a trace the agent can read from the CLI. This is the trace behind that failing run above, one model call, and nothing nested under it.

Giving the Agent a Way to Judge

Now your coding agent can run your system, and trace your system. However, it doesn’t know what you want before you tell it. There needs to be a golden standard on how the agent should behave. How I chose to do it is by tests, “unit-test-looking” framework tests.

Agent behaviour isn’t always reducible to an exact value. Some requirements are deterministic, like “did it call read_file”, while others are semantic, like “did it extract the relevant party name from the document”.

So Apo lets both kinds of checks live side by side.

test("reads-source-first", (t) => {
  t.calledTool("read_file", { input: { path: "spec.md" } });
  t.noFailedActions();
});

test("answer-is-correct", async (t, { deliverables }) => {
  t.check(deliverables.answer, matches(answerSchema));
  await t.judge(
    deliverables.answer,
    "PASS when the answer is accurate to the source and adds nothing false.",
  );
});

The syntax is not the most important part here — what this achieves is. This allows more like “TDD” approach for harness engineering. We can define our wanted behaviour from the agent with the example.eval.ts file, and let the agent run the tests until they pass.

Here’s what a red state looks like when the agent runs those tests — the deterministic one green, the judged one red, with the judge’s diff:

If you look at the test, you can see the deliverables part in the second test. I go all-in on the notion of deliverables: usually agents don’t just talk and give an answer, they also create files or make changes to the database. In Apo these are called “deliverables”, and they are caught in the adapter lifecycle. This allows us to test the actual result, not just what the agent said — for example, does the newly created DOCX contain the correct information, or did the DB status for the user change.

Test the deliverable, not the conversation!

Giving the Agent a Way to Fix

Final step is the easiest. Your coding agent can run, can look at traces, and can see your golden standard. Now it just needs to run in your codebase and use the CLI to fix the problems found. This loop ends when the agent has found the problems and has made the changes so the tests start to pass again.

Here’s that whole loop as a mocked agent session.

Closing the Loop

The code hasn’t become irrelevant. However, I’m increasingly engineering directly in the feedback loop around the code.

If the harness can run the real system, expose what happened, judge the outcome, and fix the harness with enough information to change it, then my job becomes increasingly about specifying the constraints the loop should optimize.

Recently, I find myself reviewing changes to eval.ts more carefully than changes to the implementation itself.

You can try all this yourself! Apo is open-source and free: docs.test-apo.online

apo — Opinionated agent testing framework End-to-end testing for agent systems. apo runs your real agent, checks what it actually produced, and shows you exactly what went wrong when something breaks. docs.test-apo.online