Harness Engineering in Practice: Building a Self-Improving AI Development System
Preface
The AI development community has converged on a powerful insight: Agent = Model + Harness. The harness is everything except the model itself — orchestration, tools, permissions, memory, context management, and workflow automation.
Even a frontier model running in a loop across multiple context windows will underperform without a well-designed harness. The model tends to do too much at once or declares the job done prematurely; the harness imposes structure on that tendency.
This article documents my journey building a production-grade harness over 700+ Claude Code sessions across 21 days, operating a financial trading system with 36+ Docker containers. I’ll share the architecture, the friction data, and the self-improvement loop that makes the harness get better over time.
The next frontier isn’t better models — it’s better harnesses.
The 5-Layer Harness Architecture
Each layer addresses a different failure mode. Let me walk through them bottom-up.
Layer 1: Behavior Specification (CLAUDE.md)
Problem it solves: The model proposes approaches that violate architectural decisions or repeat past mistakes.
CLAUDE.md is the “constitution” of your AI agent. Mine evolved from a simple style guide into a 200+ line operational manual containing:
Architecture Principles (Non-Negotiable)
## Architecture Principles - Event-driven first, polling forbidden: All state changes must be driven by events/callbacks/TTL expiry. Never use polling. - Prefer Pub/Sub, Callback, Cache TTL expiry over periodic checking. - When designing new features, if polling seems needed, propose an event-driven alternative first. This rule exists because the model proposed polling-based solutions twice before I corrected it. The correction led to a 98.8% CPU reduction across services. Without the rule in CLAUDE.md, every new session risks the same mistake.
Bug Prevention Rules (Learned from Incidents)
After a deployment introduced 3 simultaneous production bugs, I distilled 8 defensive rules:
### 1. Shared state writes must filter by source - Before updating store, MUST check symbol === currentSymbol ### 2. Multi-leg trade parameters must be explicit - lots, contract_multiplier MUST NOT rely on defaults ### 3. Background tasks need triple guards All scheduled tasks MUST verify: 1. Trading hours: is_trading_time() 2. Data freshness: is_tick_stale(symbol, 120) 3. Quote completeness: both legs have price + BidAsk Why CLAUDE.md Alone Isn’t Enough
The uncomfortable truth from my data: CLAUDE.md covered ~90% of friction scenarios, yet “wrong approach” still occurred 31 times.
Rules exist ≠ rules are followed. This is why we need the remaining 4 layers.
Layer 2: Automated Guardrails (Hooks)
Problem it solves: Syntax errors and security issues slip through despite written rules.
Hooks are shell commands that execute at lifecycle events — before/after tool calls, on session stop. They provide automated enforcement that doesn’t rely on the model remembering to check.
My Hook Architecture
| Hook | Trigger | Function |
|---|---|---|
syntax-validator | PostToolUse (Edit/Write) | .py py_compile + .json json.load after every edit |
commit-guard | PreToolUse (git commit) | Scans staged files for hardcoded secrets and debug statements |
wiki-auto-commit | PostToolUse (Edit/Write) | Auto-commits knowledge base changes |
cost-tracker | Stop | Records session cost metrics |
The Syntax Validator Pattern
# PostToolUse hook: validates syntax after every file edit def validate_python(filepath): result = subprocess.run( [sys.executable, "-m", "py_compile", filepath], capture_output=True, text=True ) return result.returncode == 0, result.stderr.strip() def validate_json(filepath): try: with open(filepath, "r", encoding="utf-8") as f: content = f.read().strip() if not content: return True, "skip (empty)" json.loads(content) return True, "JSON OK" except json.JSONDecodeError as e: return False, f"line {e.lineno} col {e.colno}: {e.msg}" # Route by extension validators = { ".py": validate_python, ".json": validate_json } Hook Timing Subtlety
PostToolUse fires after the tool executes — block tells the model the operation failed, but the file is already on disk. This is the correct trade-off:
- PostToolUse: Can inspect results, can’t prevent the write
- PreToolUse: Can prevent execution, can’t see results
For syntax validation, PostToolUse is ideal — immediate feedback with exact error message, forcing the model to fix before proceeding.
Layer 3: Workflow Templates (Skills)
Problem it solves: Multi-step processes get steps skipped under time pressure.
Skills are reusable prompt workflows triggered by /commands. They encode institutional knowledge about how to perform complex operations.
The Deploy Skill
My deploy workflow has 8 mandatory steps. Before encoding it as a skill, I frequently skipped health checks and notifications:
Skills can reference each other, creating a workflow graph:
User Request -> /deploy |-- Step 5 calls /health-check |-- Step 7 calls /devdiary |-- Writes to knowledge base The SBE Skill: Specification by Example as Development Entry Point
Traditional TDD has a fundamental flaw in the agentic era: the AI writes both the test and the implementation, grading its own homework. The test becomes a self-fulfilling prophecy rather than a genuine specification.
I solved this by creating a /spec-by-example (SBE) skill that serves as the mandatory entry point for all new feature development:
Old workflow: Requirement → Impact Analysis → BDD Generation → Implement New workflow: Requirement → /sbe (Knowledge + Examples + Human Gate) → Implement The skill runs 6 steps:
- Requirement Parsing — AI restates the requirement in its own words, exposing misunderstandings early
- Knowledge Exploration — Queries wiki MCP + greps codebase to map system boundaries
- Example Mapping — Generates concrete Given/When/Then examples in 3 categories:
- Happy path (normal behavior)
- Edge cases (boundary conditions)
- Error paths (failure handling)
- Human Confirmation Gate — Presents all examples to the user. Cannot be skipped. The user validates, corrects, or adds examples before any code is written
- Spec Persistence — Writes confirmed spec to
specs/sbe/as the single source of acceptance criteria - Transition to Development — Converts confirmed examples into test skeletons, then implements
The key insight: in the agentic era, the specification IS the human’s contribution. The AI generates examples to make its understanding explicit; the human confirms or corrects. Only then does the AI code — against specifications it doesn’t own.
This eliminated the “misunderstood request” friction category entirely: the misunderstanding surfaces in Step 3-4, not after deployment.
Layer 4: Knowledge Persistence
Problem it solves: Each new session starts from zero, repeating past mistakes.
I built a 3-tier knowledge system:
Tier 1: Session Memory (MEMORY.md)
Cross-session memories organized by type:
- User profile: Role, preferences, expertise
- Feedback: Corrections (positive AND negative)
- Project state: Current initiatives, decisions
- References: Pointers to external systems
memory/ ├── MEMORY.md # Index (always loaded) ├── user_profile.md # Who the user is ├── feedback_no_polling.md # "Never propose polling" ├── feedback_verify_all_code_paths.md # "grep ALL occurrences" ├── project_regime_redesign.md # Current project status └── ... (40+ memory files) Tier 2: Knowledge Wiki (MCP Integration)
A structured wiki following the Karpathy LLM Wiki architecture:
Schema Layer (CLAUDE.md) — defines structure | raw/ — Immutable source (LLM reads only) | wiki/ — LLM-compiled knowledge (LLM owns completely) Exposed via MCP with 3 tools: search(), get_concept(), related().
Tier 3: Dev Diary
Every task gets a structured entry: Problem → Analysis → Decision → Implementation → Pitfalls → Insights. This creates searchable institutional memory.
Layer 5: Self-Improvement Loop
Problem it solves: The harness itself has blind spots that only surface after hundreds of sessions.
Baseline Tracking
After analyzing 702 sessions, I created a wiki page tracking friction metrics:
| Friction Type | Baseline | Target |
|---|---|---|
| Wrong Approach | 31 | ≤ 10 |
| Buggy Code | 20 | ≤ 8 |
| Full Achievement Rate | 62% | ≥ 75% |
| Command Failed | 196 | ≤ 100 |
What the Data Revealed
The most surprising finding: the bottleneck wasn’t missing rules — it was inconsistent enforcement.
This shifted my investment from “write more rules” (Layer 1) to “build automated guardrails” (Layer 2). The syntax validator catches errors at edit-time. The deploy skill enforces verification steps that were previously skipped.
Real-World Metrics
After 21 days and 702 analyzed sessions:
| Metric | Value |
|---|---|
| Total sessions | 1,195 (702 analyzed) |
| Messages | 10,276 (467/day) |
| Full achievement rate | 62% |
| Files touched | 541 |
| Code changes | +31,011 / -2,349 lines |
| Commits | 159 |
| Top tool | Bash (2,810 calls) |
| Remote operations | 620 messaging calls |
What Helped Most
| Capability | Sessions |
|---|---|
| Multi-file changes | 41 |
| Good debugging | 39 |
| Good explanations | 36 |
| Proactive help | 17 |
Primary Friction Sources
| Type | Count | Root Cause |
|---|---|---|
| Wrong Approach | 31 | Model defaults to simpler patterns |
| Buggy Code | 20 | Syntax errors, wrong parameters |
| Misunderstood Request | 4 | Ambiguous terse commands |
Lessons Learned
1. Rules Exist ≠ Rules Are Followed
CLAUDE.md covered 90% of friction scenarios. The model still made those mistakes 31 times. Automated enforcement (hooks) is worth 10x more than written rules.
2. Match the Harness to Your Interaction Style
I operate via short mobile commands expecting end-to-end autonomous execution. My harness is optimized for this: skills encode complete workflows, memory preserves cross-session context, hooks catch errors that brief commands can’t prevent.
Pair-programming style needs a different harness — more explanation, less autonomy.
3. Measure the Harness, Not Just the Output
Before Layer 5, I had no idea “wrong approach” was my #1 friction at 31 instances. I thought it was buggy code. Measuring changed my investment priorities.
4. Event-Driven Thinking Applies to the Harness
- Hooks are event-driven (triggered by tool use) — effective
- Rules are passive (model must remember) — weaker
- Skills are command-driven (explicit invocation) — effective
The most effective harness components are reactive, not declarative.
5. Knowledge Systems Need Separation of Concerns
- Memory: Cross-session context (who, what, preferences)
- Wiki: Structured evolving knowledge (concepts, architecture)
- Dev Diary: Immutable learning records (what happened, why)
Mixing these creates a mess. Separating lets each serve its purpose.
Getting Started
Start with these layers in order:
- CLAUDE.md — Your top 5 non-negotiable rules. Add as you discover friction.
- One hook — A syntax validator for your primary language. Immediate ROI.
- One skill — Your most repeated multi-step workflow.
- Memory — Start recording corrections. Even 10 feedback memories improve consistency.
- Measurement — Run
/insightsafter 50+ sessions. Let data guide investment.
The harness doesn’t need to be perfect on day one. The key is to start measuring early.
Conclusion
Harness Engineering is the discipline of building the operating system around an AI agent. The model is the engine; the harness is the chassis, steering, brakes, and navigation.
After 700+ sessions, my harness evolved from a simple CLAUDE.md into a 5-layer system with automated guardrails, reusable workflows, persistent knowledge, and a self-improvement loop.
The harness that measures itself is the harness that improves.
Built with Claude Code. Measured with /insights. Improved through PDCA.
References: