What Is Agent Observability? The Four Things to Log

Agent observability is the ability to answer, after the fact, what an agent did and why it did it. Not whether it finished. Not how long it took. What specific actions it took, in what order, on what evidence.

Ordinary application monitoring answers "is it up" and "is it slow". Those questions remain useful and stop being sufficient the moment software starts making its own decisions, because an agent can be fast, healthy, returning 200s, and doing entirely the wrong thing.

Why agent observability is a different problem

A normal request is deterministic. Same input, same path, same output, and if something breaks the stack trace tells you where.

An agent run is a sequence of decisions. The same prompt can produce a five-step run on Monday and an eleven-step run on Tuesday. There is no stack trace, because nothing crashed. The failure is semantic: it called the wrong tool, read the wrong file, believed a bad result, or stopped early.

Debugging that requires a record of reasoning and action, not a record of errors. Your error tracker will show you nothing, because there was no error.

The four things to record

1. The full input, including what you injected

Not just the user's message. The system prompt, the tool definitions, retrieved context, conversation history, and anything your code inserted before the model saw it.

This is the record people skip because it is large and repetitive. It is also the only way to answer "why did it think that", because the answer is almost always something you put there.

2. Every tool call, with its arguments

Which tool, called with what, in what order. This is the part most teams do capture, and it gets you about halfway.

3. Every tool result, in full

This is the record most teams miss, and missing it is the single most expensive gap in practice.

A log showing `search_documents(query="refund policy")` tells you the agent looked. It does not tell you what came back. If the search returned nothing, or returned three irrelevant documents, the agent's subsequent confident wrong answer is fully explained and you cannot see it.

The debugging failure this causes is specific and maddening: you look at the trace, every tool call is sensible, the final answer is wrong, and you conclude the model is unreliable. The model was reliable. It was reasoning correctly over bad inputs that you did not record.

Log the results. Truncate large ones if you must, but log that you truncated and log the size.

4. The stopping reason

Why did the run end? Completed, hit a step limit, hit a token limit, refused, errored, or was cancelled. These produce very different outcomes and look identical in a log that only records "finished".

An agent that stopped at a step cap looks exactly like an agent that decided it was done, which is why an agent that did nothing is so often misdiagnosed.

A minimal record per step

run_id, step_number, timestamp
model, temperature, token counts (in / out / cached)
tool_name, tool_args
tool_result (full or truncated, with original size noted)
stop_reason

Five fields of that are free, because your provider returns them. The tool result is the one that takes deliberate work, and it is the one that pays.

What to do with the data

Three uses, in increasing order of value:

• **Debugging a specific run.** Pull the trace, read it in order, find the step where the picture went wrong. This is the obvious use.

• **Finding patterns across runs.** Which tool fails most often. Which step number runs tend to die at. What fraction of runs hit the step cap, which is a number that tells you whether your cap is a safety net or a ceiling.

• **Catching silent degradation.** An agent whose average step count creeps from six to nine over a month is getting worse at something, and nobody will notice from the outcome alone because it still finishes.

That third one is the reason to build this before you need it. Degradation does not announce itself, and without a baseline you cannot see it.

Where this fits

Observability is downstream of architecture. Agentic infrastructure covers the parts an agent needs to run at all, and the agent harness is the component that is usually in the best position to record this, because it sits between the model and the tools.

If you are running agents unattended, this stops being optional. How long to let an agent run unattended is a question you can only answer responsibly if you can reconstruct what happened while you were not watching.

FAQ

Is agent observability the same as LLM monitoring?

Monitoring usually means tracking latency, cost and error rates in aggregate. Observability means being able to reconstruct a single run. You want both, and the aggregate numbers are the easier half.

How much should I log?

More than feels comfortable at first, then trim. The common regret is a run that cannot be explained because a field was dropped for storage reasons. Storage is cheaper than a week of guessing.

Does logging prompts create a privacy problem?

Yes, and it needs handling rather than avoiding. Prompt and result logs contain whatever users sent, which makes them subject to the same rules as any other user data, including where they are processed. Decide retention and access before you turn capture on, not after.

What is the smallest useful version?

Run ID, step number, tool name, tool arguments, tool result, stop reason. Six fields in a table. That alone answers most of the questions you will actually have.

Sources: the OWASP Top 10 for LLM applications treats inadequate logging of model and tool activity as a contributing factor across several of its listed risks.