Insights

Observability for coding agents: what to record in each session

Published 9 October 2026

Observability for coding agents is the structured record of every session in which an agent works on a company's code: the request, the steps it took, the model it used, the cost, the result and the authorship. With that record, a company can audit how AI is used in development and see what to improve. That trail turns speed into an operation you can audit, with governance, controllable cost, continuous improvement and autonomy under human supervision.

Why each session needs a trail

Coding agents are not autocomplete. They plan, call tools, read repositories, change files and chain decisions together. Without a record, an organization gains throughput and loses the ability to answer three basic questions:

What was asked, and by whom?
What did the agent do, with which model and at what cost?
What went into production, and who approved it?

These questions are the minimum for treating AI in development as a trustworthy system instead of a black box.

The NIST AI Risk Management Framework (AI RMF 1.0) lists "accountable and transparent" among the characteristics of trustworthy AI. In the framework's words, "accountability presupposes transparency": appropriate information about the system and its outputs, based on the stage of the life cycle and tailored to the role of each actor. In software development, our reading is that this calls for a session trail, which a policy document alone does not provide.

What to record in each session

A coding-agent session starts with the request and ends with the result, accepted or rejected. A useful record covers six fields, and each one supports a business decision.

Request

The task statement, the context available (repository, branch, tickets) and who asked for it. Without the request, there is no way to audit whether the agent solved the right problem.

Steps

The sequence of actions: planning, file reads, tool calls, proposed diffs, tests run, retries. Steps are the operational equivalent of a trace: they show the path, and not only the end.

Open conventions already describe this shape. The OpenTelemetry GenAI semantic conventions for agents, still marked as in development, define spans such as invoke_agent, plan and execute_tool, with attributes for the model, tokens and errors. Instrumenting the session along those lines saves the internal team from inventing a format of its own.

Model

Which model, or set of models, ran each step. The model changes quality, latency, cost and risk profile. Without this field, periodic reviews turn into opinion.

Cost

Input and output tokens, and the cost in money per generation and per session. LLM observability tools already work this way: Langfuse's token and cost tracking documentation describes recording usage and cost for every model call and monitoring spend across models, use cases and time. In a company, this becomes an AI budget the CFO can follow.

Result

The artifact produced (diff, pull request, report), its status (accepted, rejected, rolled back) and the evidence of verification (tests, human review). A result without a verdict does not close the session.

Authorship

Who asked, which agent executed, who supervised and who promoted the change to production. Authorship closes the accountability loop: responsibility stays with people when AI writes the code.

What the evidence says about speed, and where supervision comes in

Productivity with AI assistance has already been measured under controlled conditions. In an experiment with GitHub Copilot (Peng, Kalliamvakou, Cihon and Demirer, 2023), developers completed a programming task 55.8% faster than the control group (95% confidence interval: 21% to 89%). It is a study of one task with one assistant, and not an Ithaca measurement.

Supervision and the session trail play a different role: they get that kind of gain to production with code that has been read, clear authorship and every session logged. Session observability is what makes that combination work at scale, because a team can only supervise, budget and improve what it can see.

What this changes in the company

Governance and audit

With the request, steps, model, cost, result and authorship on record, leadership can answer internal audit, risk and compliance without rebuilding the story from chat history.

Controlled cost

Sessions without aggregated cost turn into a surprise invoice. With cost per model and per type of task, the team cuts expensive paths, picks models by criticality and sets ceilings, without slowing the operation down blindly.

Continuous improvement

Periodic reviews work from the trail: which prompts fail, which tools misfire, where people reject the output, where the model is more than the task needs. Without a trail, "improving how we use AI" stays anecdotal.

Autonomy with safety

Ithaca operates coding agents at level 4 of the scale Dan Shapiro described, the same scale used on the dark factory page. In Ithaca's practice, AI executes, and people supervise and read the code. A dark factory is level 5, where the spec becomes software with no human reading. The platform was built to stop at level 4, and the client's team decides how far autonomy goes.

How the platform puts the trail into practice

Ithaca runs an agentic development platform. Every session is logged, with its authorship, and observability follows the session from request to response, with the duration, the cost and the model of each step.

On legacy systems the same discipline applies: the Knowledge base reads the existing system before any new line is written, the new system is built beside the old one, and the switch happens only after it has been checked. The method is in Legacy modernization without stopping the business.

Dr. Consulta modernized its subscription platform, a system with years in production, using the agentic development platform, with its own team supervising. Compared with the period before the modernization, infrastructure cost is 75% lower, and one feature that used to take about two weeks took up to three days. The feature is a representative example, not a measurement of the whole backlog. Maintenance went from work done by hand to supervising the AI.

These are results of one deployment of the platform, not a benchmark, and they do not replace the session trail. The full case is in the Dr. Consulta case study.

From telemetry to business results

Recording the session is not an end in itself. The value shows up when the trail feeds four routines leadership already knows.

Inventory and policy. Without an inventory of sessions, there is no way to know how many agents are running, on which systems and with which models. The inventory becomes a starting point for the Govern function of the NIST AI RMF, which covers roles and responsibilities, the organization's risk tolerance and periodic review.
AI budget. Cost per session, per repository and per type of task makes comparisons possible: refactoring versus new features, a larger model versus a smaller one, an automatic run versus a supervised session. The team cuts what does not deliver and reinforces what does, with numbers instead of gut feeling.
Incidents and rollback. When an agent-generated commit fails in production, the question shifts from "who touched it?" to "which session, which steps, which approval?" Diagnosis starts from the session record instead of from memory.
Team training. Reviews with the trail on the table teach the team to ask better, reject better and set autonomy limits based on what the session actually did.

What not to confuse

Three confusions come up often in conversations with executives.

Agent observability is not a chat log. Pasting the assistant's history into a file is not enough. It lacks structure: IDs, tool spans, the model, tokens, acceptance status and who promoted the change.
Tracking the session is not monitoring developers. What gets measured is the agent's session and the artifact that enters the repository, not individual productivity under surveillance. The goal is accountability for the AI system, in line with what NIST describes as transparency suited to each actor's role.
High autonomy is not the same as a dark factory. Raising coding agents' autonomy is only responsible if people keep reading the code and the session stays auditable. Without both, the organization crosses the threshold where nobody can sustain what was written.

The record is telemetry; the policy lives in the guide to AI governance in software development.

A quick checklist for executives

Before scaling coding agents, confirm that:

Every session has an ID and is linked to whoever asked for it
Steps, including tool calls, are stored for the defined retention period
The model and the tokens or cost go into the same record
The result includes the acceptance status and the evidence of verification
Authorship covers the request, the execution, the supervision and the promotion
People read the code that goes to production (level 4, not a dark factory)

If any item fails, there is speed, but the measurement that lets you scale with governance is still missing.

FAQ

What is observability for coding agents?

Observability for coding agents is the structured record of every session in which a coding agent works: request, steps, model, cost, result and authorship. It is used to audit, control spend and improve the use of AI with evidence.

Is LLM observability the same as traditional APM?

No. Classic APM watches services, latency and application errors. Observability for coding agents also covers planning, tools, tokens, the model and the human decision: the full cycle of an agentic session.

Do we need an open standard, or is an internal format enough?

An internal format works in the short term. Open standards, such as the OpenTelemetry GenAI conventions for agents, reduce lock-in and make it easier to integrate with the telemetry stack the company already uses.

How does this relate to the NIST AI RMF?

The AI RMF treats accountability and transparency as characteristics of trustworthy AI and organizes activities into Govern, Map, Measure and Manage. Recording sessions is a concrete way to Measure and Manage in the software development life cycle: there is something to measure, something to prioritize and a trail for incident response.

Where does the platform run, and does our code leave the network?

The platform is deployed in the client's environment, on a cloud account the client controls or in its own data center, and the repository is not sent to an Ithaca SaaS. Model calls follow the client's policy for provider, region and contract; during inference, code excerpts may go to the chosen model provider under that policy.

Book a demo

Sources

NIST AI RMF 1.0 (NIST AI 100-1). Accountability and transparency as characteristics of trustworthy AI; the Govern, Map, Measure and Manage functions.
OpenTelemetry GenAI, agent spans. Open conventions, in development, for agent spans (invoke_agent, plan, execute_tool) and usage attributes.
Langfuse, token and cost tracking. Market practice: record usage and cost for every model call and monitor spend across models, use cases and time.
Peng, Kalliamvakou, Cihon and Demirer (2023). Controlled evidence of speed-up: 55.8% faster (95% CI 21% to 89%); a study of one task, not an Ithaca measurement.
Dan Shapiro (2026), The Five Levels. The five-level autonomy scale; level 5 is the dark factory.
Ithaca, Dark factory. Level 4 against the dark factory.
Ithaca, Legacy modernization. The legacy method and the Dr. Consulta case.
Ithaca, Agentic development platform. What the platform is and what it is not.
Ithaca, Harness engineering for enterprise coding agents. Harness, model and observability, told apart.