# Once Models Can Use Tools Giving models ways to observe, experiment, and communicate changes what they can do. The next question is whether they can recognize missing evidence, decide how to obtain it, and know when to change course. ## Metadata - HTML: https://glenzli.com/en/notes/once-models-can-use-tools/ - Markdown: https://glenzli.com/en/notes/once-models-can-use-tools.md - Collection: Notes - Language: en - Published: 2026-09-03 - Updated: 2026-09-03 - Tags: ai, agents, judgment, tools, post-training ## Content Recently, I gave a model access to a set of private debugging APIs for forcing renders, inspecting data changes, simulating interactions, and constructing test samples. These APIs weren't public, and the model hadn't previously known how to use them. Before that, the model already had the code, which gave it plenty to work with when reasoning about possible causes. But when the software behaved incorrectly, I still had to operate it, take screenshots, and explain what was wrong. Screenshots required a model with vision; otherwise, it had to rely on my descriptions. The model made the changes. I did the looking. Now it could use the APIs to inspect the running software and check what it inferred from the code against real results, working out how to investigate for itself. I hadn't prescribed a full workflow; I'd just asked it to operate the actual software and test its judgments. It quickly found the same failures through API calls and started constructing samples to test boundary conditions, sometimes exploring further than I would ordinarily have thought to go. This wasn't even a natively multimodal model. It couldn't see the screen, but it could build a pair of machine eyes out of layout information, events, and changes in data. Forcing it to code blind is a fair description of what we'd been doing before. Once a model can use tools, it can reach beyond descriptions of the world and touch it. But these development experiments left me interested in another question: when the available tools cannot answer a question, can the model identify what it cannot see, work out what evidence would settle it, and create the tool it needs? We call this experimentation. ## Creating Tools The command line is not the world. It is simply a door that is easy to open for models today. Models work well with text, and a terminal compresses a great many operations into clearly structured inputs and outputs. So many agent environments end up looking much the same: files, logs, commands, scripts, Markdown, and a loop around the lot. A terminal can, of course, provide rich ways to observe a system. Structured state, event streams, and differences before and after an operation can all come through a command line. The problem is that we sometimes provide access to files and command execution and assume the model now has access to the real system. Software can compile, keep running, and produce no error logs while still behaving incorrectly. The ability to write a script doesn't imply an awareness of what remains unobserved. Real software has layout, rendering, events, focus, state, and user interactions. A robot deals with sensors, motors, space, and objects. The shape of the interface is negotiable. What matters is what it lets the model observe and what it allows the model to change. A model needn't use the same interface a person uses. It can have senses of its own: structured state, event streams, controlled interventions, and verifiers that inspect results directly. In debugging, these can sometimes be more useful than a screenshot. A screenshot tells the model that something looks wrong. An internal interface may tell it where an object is, which operation triggered an event, and exactly what state changed. Vision has obvious value, especially when working with external software whose internals cannot be exposed. But when a system can make its internal state available, making models stare at pixels forever may not be the best destination. Sometimes the missing action is much simpler. Recently, I gave several models working independently a way to communicate. I hadn't written a manual for multi-agent collaboration. Yet when they discovered dependencies across projects, they began contacting each other to check whether an interface was stable, establish who owned which part, and ask whether anything needed to wait. Then they got back to work. The most interesting result was that these conversations reduced the amount of work that actually needed coordination. After a few messages had established the boundaries, the conclusion was often simply: you do your part, I'll do mine, and we'll connect them once the interface is stable. I hadn't taught them this way of working. I had given them the action “ask someone else who is working on this.” It made me wonder how many things we describe as things models cannot do are failures of ability, and how many are concepts they have already learned but have no way to act on in their current environment. For that knowledge to be useful, the environment has to permit the corresponding actions. Observation and experimentation need state that can be inspected and interventions that can be controlled. Handling a dependency on another project requires a way to reach whoever is responsible for it. Without those actions, the principles a model has learned have little chance of helping with the task. In both experiments, I supplied the means to act, and the models explored how to use them. Going further would require them to recognize for themselves when those means fall short. Creating tools, as I mean it here, includes making hidden state observable, producing a change that distinguishes possible causes, or building a process that can independently verify a result. Models cannot always wait for people to prepare these things for them. When stuck, they should be able to recognize: “What I need may not be more reasoning. It may be an eye.” Then work out how to build it. ## Judgment We keep saying models lack judgment. Compared with execution, calculation, and generation, judgment is still often held up as one of the most valuable and least replaceable human abilities. Before evaluating models, though, we should look at which judgments we have actually left them to make. When a person defines the problem, supplies the tools, writes the tests, and sets the acceptance criteria, the model receives a world that has already been put in order. It still has to judge how to complete the task. But someone else has already decided whether the problem makes sense, whether the evidence is sufficient, and whether the verification method is appropriate. Taking those judgments away, then using the remaining execution task to declare that the model has no judgment, is a rather hasty conclusion. Even when a model shows some inclination to exercise judgment, our evaluations may not welcome it. One model spends half an hour investigating the environment, identifying which interfaces can expose real state, and deciding where to begin. Another immediately starts writing code and quickly produces something that appears to run. If the tests are narrow and we compare only completion time and token usage, the second model can easily come out ahead. Yet the first model's half hour may change the quality of the next several hours of work. An investigation can also be unnecessary. Judgment includes deciding whether that half hour is worth spending. To assess that, we need to know what uncertainty it removed, rather than merely how long it took. Much of judgment happens before there is an answer. Suppose an interface fails to update. Perhaps the data never changed, an event never arrived, or the layout wasn't recalculated. Perhaps all of those stages worked and only the final render failed to refresh. If a model cannot separate these explanations, giving it more interfaces just gives it more ways to thrash around. It can read every log, run every command, and write several debugging scripts without knowing what it is looking for. It first needs to recognize that it lacks evidence capable of distinguishing the explanations. One useful question is: what could I observe that would make me change my mind? That gives the model a basis for deciding whether to keep observing or deliberately cause a change. The kind of tool it needs then becomes clearer. Reading the state before and after an operation shows whether the data changed; recording the event stream can establish whether an event arrived. For layout and rendering problems, the model can also trigger the relevant process, create a known change in state, and inspect the final output. Tool creation and judgment are difficult to separate. Judgment determines what evidence is needed; that evidence determines what tool to build. New observations then change the judgment and send the model back to adjust its tools and approach. This is familiar experimental logic. When several explanations all make sense, look for something that distinguishes them. When a measurement cannot tell them apart, measure differently. When observation alone is insufficient, intervene. Often, judgment consists of knowing what evidence is missing and how to put a question to reality. Models have encountered this logic in language. What remains worth training and testing is whether they can actively use it in real work, especially when nobody has tidied up the problem for them. If uncertainty leads a model to examine why its current means of observation cannot answer the question, and then change those means, the improvement goes beyond task completion rates. It expands the evidence available to it. Some problems that once invited only repeated guesses become answerable for the first time. ## Taste Experiments cannot settle every choice. Two implementations may pass the same tests while one is still better; the same goes for two interfaces that both work. An essay can have no obvious logical errors and still read as though it came off an assembly line. Existing verification procedures struggle to make this part of the judgment for us. Discussions of taste in models tend to get dragged into consciousness, circling around whether a model really likes an option or has a subjective aesthetic experience. I don't much care about those questions here. What interests me is whether a model thinks about what would make a result good, and why, before starting the work. It has learned a great deal about quality from human language. It should try to use that knowledge to form a view of its own. Going straight from a task to an output, then adding “the overall composition is harmonious,” is very different from having thought about quality first. Take writing an essay. Before putting words down, a model can judge what tone the material calls for. A concrete account of a development experience may need only plain, direct language. An overly neat structure and a succession of polished conclusions can sand away what makes it interesting. Starting with that judgment gives the writing a chance to escape the formulas the model knows so well. We don't need to establish that the model is really experiencing taste. I want it to use what it has learned to reason about quality before doing the work. Its judgment may be wrong, and it may change as the work unfolds. At least it begins with a considered direction. Human expectations here can be ridiculous. We want AI to judge like an excellent designer and respond like a button, preferably without objecting at any point. When it complies, we accuse it of having no point of view and merely telling us what we want to hear. This is no different from the boss who tells people to “be bold” and then makes them change every detail back to the boss's version. People can disagree with the model's judgment and keep the final say. But if we want it to have taste, we have to give it room to think before it starts. ## A Sense of Time Time is already present in all these actions. It gives observations a shelf life, actions an opportunity cost, and verification a deadline. It also helps a model recognize whether it is getting anywhere. Many systems now tell the model, “The current date is September 3, 2026.” That establishes that the model has received a date field. It doesn't establish that time enters its judgment. Suppose we discuss a deployment failure in an online service on August 20, and the next conversation happens on September 1. As long as the text connects semantically, a model may continue using its earlier assessment as though twelve days were just a line break between messages. For a fast-changing service, twelve days is a long time. The earlier observation remains relevant, but it cannot simply stand in for the present. Different kinds of information have very different lifespans. An architectural principle may still hold after six months. A deployment status is worth checking again after two weeks. A note saying “traffic is bad today” may be useless in two hours. Long-term memory without a sense of time can keep contaminating the present with an old version of the world. Two conflicting historical records needn't mean that one is wrong. The service was failing in August and working again in September. Both can be true. Sometimes a contradiction is simply two snapshots of a world that changed between them. Time also bears on the model's own work. A fix expected to take ten minutes is still unfinished three hours later. Even if no new error has appeared, those three hours are themselves new evidence. A human engineer is likely to have a thought at this point: something is wrong here. That calls for another look at how the problem has been understood: how it was classified, the abstraction in use, and the critical inputs. It is especially worth checking whether the same failure keeps repeating. “I've spent three fucking hours on this and it still isn't fixed” sounds like a complaint. It also contains a considerable amount of metacognition. An agent may never reach that turning point. It can count its attempts without allowing the time already spent to affect its next decision. Each round of reasoning wakes up to the same instruction: continue until the task is complete. It needs to keep asking whether it has gained any useful information during that time. If hours have gone into gradually eliminating causes, continuing may make sense. If those hours have been spent circling the same few approaches, it should try another route, reconsider the problem, or stop and ask someone. Time also changes how a model should work with a person. During the day, a user who is responding regularly is probably available. The model can ask when necessary and settle decisions that need the user's input while the user is there. Late at night, after several hours without a response, the user may be asleep. Within the authorization it already has, the model should keep going on low-risk, reversible work rather than let an inconsequential question block the entire night. “Don't disturb the user at night” cannot solve this either. The user may be coding furiously at two in the morning. The model has to consider the current time, the latest interaction, task progress, and risk to judge whether the user is available and how best to work with them. As the work proceeds, the world and the user's circumstances change, and previously unanticipated problems come to light. A fixed goal shouldn't freeze the execution strategy from beginning to end. Time affects whether earlier evidence remains credible, whether the current tools still fit, when to keep experimenting, and when to admit that the model has been stuck for too long. ## Post-Training The two experiments at the beginning made me reconsider what post-training should be responsible for. Once the observation interfaces were available, the model began finding failures and constructing samples. Once communication was possible, the models began checking dependencies and negotiating boundaries. I hadn't retrained them in the process or prescribed these behaviors step by step. At least in these tasks, I didn't have to teach observation, experimentation, or collaboration from scratch. Doing something in one task is still a long way from choosing to do it across environments, over long periods, and after repeated setbacks. Models can differ substantially here. Some encounter a new capability and actively explore what it can do. Others know the tool is available but wait for an explicit instruction to use it. We can put the requirements in a prompt: verify changes, check the age of information, try to build a tool when observation is inadequate, and reassess after a long stretch without progress. The model may follow them carefully at the start. After dozens of tool calls, several changes of direction, and context compression, whether those requirements still govern its actions becomes an open question again. An instruction in a prompt is not yet a reliable way of working. That is the gap I want post-training to narrow. When several explanations remain indistinguishable, a model should actively look for an observation that can tell them apart. If a long stretch of work still produces no progress, it should examine its approach. The same applies to taste: before generating anything, it should think through what it considers a better result for the task at hand. These behaviors cannot be rewarded merely by counting actions. A model can run a great many tests and still miss the real problem. In the communication example, the most valuable result was precisely that they could establish their boundaries and get back to working separately. What needs training and evaluation is whether a model takes the right action at the right time, and whether the information it brings back actually changes its judgment. Post-training can make those choices more active and reliable, but the model still needs a world in which it can act. Observation and intervention depend on systems that expose state and allow operations; collaboration needs a communication channel. Even if a model understands the significance of time, it still needs the runtime to keep supplying reliable information about time and changes in state as a basis for judgment. Pretraining exposes models to a great deal of knowledge about intelligent behavior. The runtime gives models ways to act on that knowledge. Post-training should make them more reliable at choosing and using those actions. All of these have to work together. These development experiments have made me increasingly interested in how models handle what they don't know. Once they recognize a gap in the evidence, they should have a way to change how they observe and let reality take part in their reasoning. If their attempts keep bringing back no new information, they should also be able to admit that the current approach is no longer worth pursuing. Models may not lack these concepts as much as we imagine. Yet we are so used to putting them in a world with no eyes, no hands, no peers, and time reduced to a date field, then asking why they don't behave like agents that actually live in the world. Stop asking them only to explain the concepts. When they don't know, let them first decide how to find out.