Agent from Scratch Part 2: Orchestration

The Problem with Raw LLM#

Now that IO is hooked up, you’d think the agent should work. It doesn’t.

A raw LLM is pretty much Q&A — there’s no skill, no action. It just answers your input with predicted tokens. Impressive, but useless as an agent.

To resolve that, we need prompt engineering. This is probably the only truly unique part of an LLM-powered agent system. Everything else borrows from existing software patterns.

Two Types of Prompts#

In my implementation I gave the agent two core prompts: emotional support and productivity.

The difference lands on what we ask the agent to do:

• Emotional support prompt: “Say something nice, be supportive.” The prompt helps the agent recognize that its job is comfort, not tasks. From the model’s point of view, we’ve provided context, so it can make a better prediction.

• Productivity prompt: Way more complex. This is where the “harness system” lives.

The Harness System#

Harness engineering = creating a bash-style execution environment where:

• We have skills (bash commands)
• We define how to trigger them (accurate match vs. model-generated)

In my example, I created a set of skill schemas. It’s a bit old-fashioned compared to plain-text skills I’ll cover later, but they do the same thing at their core.

The full execution loop looks like this:

1
LLM reads local env
2
  → finds function it can use
3
  → understands the task
4
  → does it
5
  → validates and responds to user
6
  → user annotates (correct / incorrect)
7
  → agent learns from execution

The abstraction isn’t that different from humans: fail more, learn more. And eventually there will be an “aha moment.”

Distillation is the Real Secret#

I almost forgot to mention the most important thing: “learn from other people’s success or failure” is the best way to describe what good orchestration enables.

When you capture agent execution logs — what it tried, whether it worked, what the user annotated — you have a distillation dataset. That’s exactly what powerful models are trained on: human-annotated traces of good decisions.

Keep It Simple#

Data is king. Keep the flow simple but logical. Let the agent figure out the best way to do things — don’t over-engineer the orchestration layer.

A complex orchestration system you built becomes a constraint the agent has to work around. A simple harness the agent can reason about is a tool the agent can use.

GitHub#

Full code at github.com/Czhang0727/agent-from-scratch.

Part 3 covers skills — the user manuals that tell your agent which tools to use and when.