Engineering conversational agents for GUI-coupled applications

When we started building Trova's conversational interface, we assumed the hard part would be the recruitment domain: distilling job descriptions, planning candidate research, searching, shortlisting. It wasn't. The hard part was that our users don't just talk to the agent. They click buttons, prune lists, add items, and link relationships through a GUI, just like they would in any other enterprise software.

A conversational agent embedded in an enterprise application has to reconcile three sources of user state. There's the conversation itself, which the model sees in full. There's the rendered interface, where users scroll, reorder, dismiss, and select without saying a word. And there's the database, which holds the ground truth about the application's objects (assignments, candidates and reports, in Trova's case). Each of these evolves on its own timeline, and each is deeply intertwined with the others. The model can only reliably observe one of them, because of how the information actually reaches it.

We evaluated the orchestration frameworks available to us: LangGraph, AutoGen (since succeeded by Microsoft Agent Framework) and CrewAI. They're good at what they're built for: tool execution, workflow management, multi-agent coordination with human-in-the-loop. None of them address the problem we kept hitting, which is reliably interpreting inferred, referentially ambiguous instructions in a multi-turn conversation that's coupled to a multi-action interface. The naive fix, passing every GUI and database event into a long-context LLM pipeline, sounds simple in theory. If it were that simple in practice, there would be far fewer products with an AI chat window bolted onto the corner of the screen. This post covers the architectures we tried, why the obvious ones failed, and the pattern that eventually worked.

Why this problem is different

Here's a minimal version of the failure mode. A user reviews a list of candidates the agent has surfaced, then manually removes two of them directly in the interface, with no message and no conversation turn. A few exchanges later they ask the agent to add back "the one I removed earlier." Nothing in the conversation transcript reveals that a removal ever happened; it happened entirely in the GUI. Without deliberate handling, the model has no way to resolve "the one I removed," and it either fails the request or, worse, guesses.

We also looked at giving the model the interface itself: computer-use style systems that observe the screen and drive the machine directly, which sidesteps the reconciliation problem by removing separate state altogether. These are legitimate systems for their intended use case, but handing an agent control of a user's screen was inconsistent with the trust model we wanted for a B2B recruitment product, where the application should constrain the agent, not the other way round.

A single agent with many tools

Our first attempt was the obvious one: a single reasoning LLM with tool-calling. We gave it tools for every step of the workflow: ingesting a job description and supporting documents, generating a candidate research plan, accepting conversational edits to that plan, running a candidate search, letting the user modify the resulting list conversationally, adding candidates to the assignment, searching the web for questions outside its own knowledge, and querying the application database.

On top of the tools, the model had to answer open questions about the role, the candidates, the industry, and the research plan, while respecting guardrails around hallucination and data privacy. The system and user prompts grew large and complex quickly.

Testing showed a consistent pattern: the model followed explicit instructions well, and failed frequently on inferred ones. That is the "that one," "the second one," "go back to the earlier version" class of reference, which assumes shared context the model never had.

We tried closing the gap with checkpoints: after significant interface changes, we serialised UI state back into the model's text memory. This helped, but it was expensive, because every turn now carried the accumulated history of UI state as tokens. Reconstructing a user's intended operation from a text description of all interface events consumed a large share of the context budget, and it still missed interactions once a conversation ran long enough for earlier events to fall outside the context window.

Delegating ambiguity to a smaller model

Our second architecture split the two jobs the single model had been doing badly at once. The main conversational LLM continued to run the workflow. When it hit an ambiguous or inferred instruction, it delegated to a smaller model, a deducer, whose only job was to work out what the user meant and return the steps needed to resolve the request.

The first difficulty was the trigger. There's no established theory for when a conversational model should recognise its own confusion and delegate rather than guess. Frontier model providers don't expose a reliable confusion signal, so we couldn't detect ambiguity from the model's internals. We had to design for it at the architecture level instead.

The pattern worked better than expected on the problem it targeted. Context and memory bloat improved substantially, because the main model no longer carried the full reconstruction burden, and the deducer received a focused question instead of the entire transcript. It didn't resolve everything, though. The failure mode that survived was branching across turns. A representative session looked like this:

User: Show me the shortlist again.

Agent: Displays eight candidates.

User: What has the third one published recently?

Agent: Runs a web search, summarises the publications.

User: Actually, change the criteria. I want operational experience weighted higher, and rerun it.

By the third turn, resolving "the criteria" and "rerun it" means stitching together the original search parameters, an intervening web search that had nothing to do with them, and whatever the user did in the interface in between. The deducer got a focused question, but the context it needed was now spread across the whole session.

Due to recent advancements in System One models, we are planning to update this model with Jev for the next release.

Context is not the answer

Our first instinct was to give the deducer more history, summarising the accumulated GUI and database events into its context. This helped briefly, then stopped helping.

The pattern repeated across experiments: each increase in historical context recovered a handful more cases while adding latency to every request, and the marginal cases it recovered got stranger, not simpler. We concluded that a bigger context window for the deducer was neither a durable fix nor a comprehensive one.

The reason is structural, not a matter of scale. Ambiguity resolution doesn't improve with more context, because the deducer isn't designed to see everything; that's precisely the problem it exists to solve. A small model patched in to course-correct a large model with a windowed context is a temporary fix, not an architecture.

Letting the interface carry meaning

While building a faster human-in-the-loop conversational flow, we introduced interface elements that removed the need for users to instruct the model through text at all. We'd been treating the GUI purely as a rendering target, a place where the agent's outputs showed up. We started treating it as part of the conversation instead. Every interaction we moved out of text and into a direct interface action also removed a conversation turn, which is what actually drove context and memory usage down: cost scaled with turn count, not with the sophistication of any single turn.

What used to be a conversation turn, adding one or two candidates to an assignment, became an "Add to assignment" button on the candidate card. The user no longer spent a turn asking the agent to do it. A full round-trip through the model collapsed into a single structured event in the context: "user added candidates to the assignment," with a candidate ID array attached.

A button click is an utterance the model doesn't have to interpret. When a candidate list renders with explicit per-candidate actions, "remove the second one" stops being a sentence the deducer has to resolve and becomes a click on a specific, already-identified row. When a research plan renders with editable criteria fields, "change the criteria and rerun it" becomes a structured edit event with no referential ambiguity in it at all. Each interface element we added this way eliminated an entire class of inference, not just a single case.

We want to be precise about the claim here: this reduced the problem, it didn't solve it. Users still type inferred instructions, the deducer still handles them, and the branching failure mode from the previous section still exists. What changed is volume and stakes. The interface absorbs the references that are cheapest to make unambiguous, so the model's interpretive budget goes to the requests that genuinely need it.

Lessons

Treat the interface as state, not output. The agent needs to know what the user saw and did, not just what they typed. Any architecture that treats the conversation as the whole interaction will fail on everything that happens outside it.

Delegation beats accumulation. A small model given a focused question outperformed a large model given everything. Separating conversation management from ambiguity resolution improved both.

Design away the ambiguity you can't resolve. The cheapest inference to handle is the one the UI made unnecessary in the first place. Interface design turned out to be an ambiguity-reduction tool, not just a usability concern.

If you're working on similar problems, we'd like to hear from you.

Acknowledgements

This work was carried out by the Trova engineering team at Evolve AI Labs.