That experience made me want to step back and reflect, on what is being built across the world in the name of AI/Agents. More importantly, it made me want to understand what work actually means in a post-LLM world.

For the purpose of this article, I’m deliberately limiting scope to knowledge work performed on a computer. Physical labor and robotics are out of scope, not because they are less important, but because they will likely be solved via a parallel path: better actuators controlled by increasingly capable models. The ideas here will apply there too, just perhaps later.

At its core, One can consider work to be the transformation of inputs into outputs under constraints. For most of human history, this transformation was physical. Wood carved into furniture, Iron beaten into Tools, Rock crushed into foundation and beams. But Over time, especially in the last few decades, a large and growing class of work became symbolic performed using numbers, categories, text, and rules.

Most modern jobs, be it in finance, operations, legal, analytics, support, compliance etc. exist almost entirely in this symbolic world. They don’t differ in essence. They differ only in which symbols they manipulate and which rules they apply.

Knowledge work, then, is not so much thinking as it is execution. Humans temporarily lend their cognitive machinery to the task: ingest symbols, transform them under constraint, emit updated symbols, and clear state. Over time, this looping behavior hardens and evolves into roles, titles, and careers and even to a life well lived.

But at its core, it remains a simple computational pattern. Human mind, which is a general purpose computer is programmed to perform the task.

And we invented computers to accelerate that very computational patterns. And as we all know, it worked spectacularly.

Software multiplied human output. The amount of work one person could do expanded dramatically, depending on which software they had access to. Entire categories of effort collapsed. Tasks that once took teams could now be handled by one person with the right tools. We all got faster. Louder. More productive.

It was true progress.

But software came with two fundamental constraints.

It was rigid. And it was expensive.

Rigid, because software demands that the world be described in advance. Someone has to decide the flows, define the states, enumerate the edge cases, and design interfaces that attempt to shepherd messy human reality into clean, pre-approved paths. A great deal of modern product development has been about this act of compression: forcing open-ended human work into rigid frameworks that can be coded, shipped, and supported.

Once deployed, software does what it was told to do. no more, no less. In the era of software burned onto CDs, this rigidity was physical. In the cloud era, it became procedural: CI pipelines, release cycles, migrations. Faster, but still fundamentally frozen at any given moment in time.

And software was expensive because specifying all that logic required specialists. Programmers became some of the most highly paid people on the planet, not because typing code is magical, but because translating ambiguous human work into deterministic systems is brutally hard. We spent billions, trillions to build B2B SaaS to encode how organizations believed work should happen.

Marc Andreessen famously called this “software eating the world.”
What that really meant was software eating human judgment, one rigid abstraction at a time.

And yet, even as software spread everywhere, humans never quite left the loop. They bent the tools. Worked around them. Filled the gaps. Became the glue between systems that couldn’t fully understand each other.

And we built software that appealed to humans and humans only

That was until recently.

I recently re-read one of my favourite essays, In the Beginning was the Command Line (Must read if you love computers and programming)

Back in the days of the command-line interface, users were all Morlocks who had to convert their thoughts into alphanumeric symbols and type them in, a grindingly tedious process that stripped away all ambiguity, laid bare all hidden assumptions, and cruelly punished laziness and imprecision. Then the interface-makers went to work on their GUIs, and introduced a new semiotic layer between people and machines. People who use such systems have abdicated the responsibility, and surrendered the power, of sending bits directly to the chip that’s doing the arithmetic, and handed that responsibility and power over to the OS. This is tempting because giving clear instructions, to anyone or anything, is difficult. We cannot do it without thinking, and depending on the complexity of the situation, we may have to think hard about abstract things, and consider any number of ramifications, in order to do a good job of it. For most of us, this is hard work. We want things to be easier. How badly we want it can be measured by the size of Bill Gates’s fortune.

The OS has (therefore) become a sort of intellectual labor-saving device that tries to translate humans’ vaguely expressed intentions into bits. In effect we are asking our computers to shoulder responsibilities that have always been considered the province of human beings--we want them to understand our desires, to anticipate our needs, to foresee consequences, to make connections, to handle routine chores without being asked, to remind us of what we ought to be reminded of while filtering out noise.

Neal Stephenson, talks about the computers and OSes, and how they have evolved over the past few decades after personal computing became a thing. How we explicitly rejected word-based interfaces (CLI i <3 u) for more Graphical and Sensorial interfaces purely to dumb down their capabilities for the masses.

Graphical interfaces were bolted onto rigid systems not because computers needed them, but because humans did.

Software required precise instructions, exact inputs, and unwavering obedience to predefined rules. The GUI existed to translate messy human intent into something a deterministic machine could tolerate.

Humans would do the interpretation, and Software would do the execution.

But that arrangement only made sense as long as machines couldn’t interpret intent on their own.

If most knowledge work is programmable logic, and if we now have systems that can write that program, run it, evaluate the result, and revise it when reality disagrees, then the obvious question presents itself:

What role is left for a human to play in executing the work itself?

A model with access to a runtime environment and a file system is, for all practical purposes, as close to artificial general intelligence as knowledge work demands. Not because it can “think like a human,” but because it no longer needs a human to act as the execution layer.

• the ability to read and write files, This is your memory

• the ability to execute programs in an environment, This is your action

• and the ability to observe the results and iterate, This is your feedback

That’s the entire so-called agentic loop.

With this setup, the agent can ingest information, store it, transform it, generate new tools for itself, run them, evaluate the results, and revise its approach. It can externalise state, reason over it, and operate indefinitely without relying on fragile context windows or human intervention.

You don’t need a cathedral of agent architecture.

As models improve, they will learn to manage their own context: compact it, reorganize it, and project it outward into durable representations. The file system becomes a knowledge base. The environment becomes an extension of the model’s cognition. What looks like “agent design” today will eventually collapse into capability.

This is why the moment I used Claude Code, something clicked in a way it hadn’t with tools like Cursor.

Not because it was smarter in isolation, but because it inhabited an environment.

A system that could enter a directory, inspect the terrain, create artifacts, modify them, run them, observe what happened, and keep going.

The quality of work then boils down into how long these agents can run autonomously. It’s a function of inference time and inference cost, both of which is guaranteed to reduce. (Moore’s law ftw)

Knowledge work doesn’t require consciousness, intuition, or creativity in the mystical sense we often attribute to it. It requires a loop: read symbols, transform them under constraints, emit new symbols, and verify the result.

And this is why coding agents like Cursor, Claude Code, Replit, v0, and Lovable are sitting on top of a gold mine.

Not because they help people write code faster, but because they sit directly inside the fundamental loop of what could be the future of knowledge work.

Every interaction a user has with these systems, every correction, every nudge, every moment of “no, not like that, like this” is a signal. A lightweight but extraordinarily high-quality form of reinforcement. A human acting as a sparse reward function, guiding the system toward intent.

The user isn’t just producing software. They’re teaching the system how to converge: fewer steps and fewer errors. Over time, the iterations shrink. The feedback becomes implicit. What once required an explicit human controller gets internalized by the model itself.

Tighter alignment over intent is what these models need to achieve.

This is a compounding advantage. and most likely why these companies command the valuations they do.

The closer a system sits to execution, and the more tightly it couples action with feedback, the faster it learns to do the work on its own.

Which brings us to the uncomfortable implication for anyone trying to build agents for knowledge work.

If everything is a programming problem, then code is the most natural substrate for agents to operate on. Large language models already understand it deeply, not by accident, but by virtue of their training. Code is abundant in their corpus, structured in a way that mirrors reasoning, and, crucially, verifiable. It produces outputs that can be tested, evaluated, and rewarded with precision.

Any serious agent for knowledge work will need to:

• generate code to express intent,

• execute that code in a real environment,

• observe the results,

• and iterate until the outcome matches the goal.

On top of that, the file system alone might not be enough. Real work requires durable context: business-facing data models that act as a structured knowledge base, connectors that bridge systems of record, and powerful, composable APIs that agents can use as tools.

What we ultimately need to build is a Run time tailored for a specific kind of work.

Leave a comment

Share

That vision is exciting, but In reality, we’re still a ways off from complete autonomous agents

Right now, what works best is a Human-in-the-Loop (HITL) approach. These systems don’t replace people entirely but rather automate large, repetitive chunks of work so the human can step in only where it matters most. This doesn’t just make life easier for individuals, it unlocks a lot of value for teams. It helps them stay leaner, yet still grow faster.

The Automation Fallacy

It’s tempting to think the math is straightforward. If an AI automates 50% of a task, the human becomes 2x more productive. If it automates 80%, suddenly they’re 5x more productive. Logical, right?

Unfortunately, that assumption is completely wrong.

Most tasks just aren’t linear in nature. Automation almost always starts by tackling the low hanging fruit which consists of the repetitive, structured steps that are easiest to codify. As the system improves, you might reach 80% coverage and feel confident you’ve unlocked huge value.

But the catch is that the last 20% is the hardest part, and it gets even harder because of the context loss.

When a person has to step in at that point, they’re not continuing smoothly from where the AI left off. Instead, they face three compounding problems:

1. Lost context – they don’t know how or why the agent made certain decisions.

2. Extra effort – they may need to validate or even redo what the agent did before moving forward.

3. Broken flow – Fixing up stuff often takes longer than just doing the whole job from scratch.

4. Rigid flow – the agent’s process or agentic workflow is often too fixed. It doesn’t adapt or learn from how the human finishes the task, so the automation is stuck at 80% instead of flexibly moving toward 90% and even more.

So that final 20% can sometimes take more time than it would have to complete the entire task manually. Instead of making people 5x more productive, partial automation risks slowing them down.

This is also why AI agents often fails during pilots, even if with their fantastic capabilities and mind blowing demos.

To give an example, a coding agent might generate 80% of your code, They can one shot entire classes, data models, migrations, everything. That’s impressive and saves time upfront. But if the structure isn’t quite right with respect to your design principles, or if the logic doesn’t fully match the business rules, the developer now has to dive in and untangle someone else’s half-finished work.

And if you’ve worked with developers, you know that there’s nothing they hate more than cleaning up someone else’s half-baked code, except for maybe meetings 😜

This isn’t unique to coding tho. it’s the same dynamic in other domains like accounting, report generation, customer support, or data analysis. Automating the “easy” parts feels like progress in theory, but without proper context handoff, it just doesn’t make much difference to end user.

Where Partial Automation Still Works

This doesn’t mean partial automation has no value, It just means it must be designed thoughtfully. Two principles matter most:

• Seamless handoff: The AI needs to pass context in a way that makes it easy for a human to pick up where it left off. Summaries, structured outputs, or clear audit trails can eliminate the need for reverse engineering.

• End-to-end sub-tasks: Instead of doing 80% of every job, agents should fully own specific, repeatable sub-jobs. Do end to end account mapping, Create Draft entries. Triage and route the customer ticket. The human then steps in only when judgment or nuance is required.

This shift in design turns HITL into an amplifier rather than a bottleneck.

But the real silver bullet is Agentic Memory.

The real unlock for AI agents won’t just come from better models or faster automation, rather it will come from adaptiveness. The biggest frustration with today’s partial automation is that it’s rigid. Agent makes its best guess, the human cleans it up, and the agent learns nothing from that cleanup. Next time, the same gaps appear, and the cycle repeats.

What’s missing is agentic memory, a policy layer that allows agents to watch, record, and internalize how humans finish the last mile. Instead of getting stuck at 80% coverage, the system uses every correction as fuel to stretch toward 90%, even 95%.

This creates two powerful effects:

• Compounding automation – over time, the agent doesn’t just do more of the work, it does more of the right work, tailored to the team’s rules and preferences.

• Stickiness – because the agent adapts to the unique workflows of its users, switching away from it feels costly. It doesn’t just automate tasks; it remembers how you do them.

In other words, the way past the Last 20% Problem isn’t just brute force improvements. it’s giving agents memory and adaptiveness, so they become collaborative partners that grow more capable the longer they’re in the loop.

The true promise of AI Agents isn’t about nibbling away at tasks in percentages.

It’s about taking full ownership of well-bounded, repeatable workstreams so that humans can focus on the parts of the job that require context, creativity, and/or decision-making.

Instead of thinking, “We’ve automated 80% of this task, so we’ve doubled productivity,” the better framing is: “We’ve automated this entire sub-task end-to-end, so humans never have to think about it again.”

Humans and agents should fully own meaningful slices of work, end-to-end. Like a well oiled machine, Their work should be complimentary, not tripping each other up.

Leave a comment

“The best way to predict the future is to invent it.” - Alan Kay

From Asimov’s I, Robot to The Electric Slate, Metropolis’s Maria to HAL 9000, from JARVIS to Her, some of our best stories have dreamt aloud about machines that could think, feel, and act with purpose.

Automatons, androids, agents. They’ve all flickered across our screens as both warnings and promises. But up until two-three years ago, they all just felt like a promise of the future.

But that’s no longer the case. The Autonomous Era is finally here. And it’s here to stay.

These last few years, The future of large‑language‑model (LLM) applications has been invented in real time and it’s moving at a pace so fast that it has become extremely hard to keep up.

2023 was the era of workflows. We built multistep, hand‑crafted LLM pipelines. This brought about a flavour if intelligence to what was our traditional deterministic software.

But that was just the beginning

By mid‑2024 we discovered ReAct‑style agents that could reason and decide which tool to call next. The first time I witnessed a ReAct Agent, I was truly mesmerised. This was a step shift in how jobs are being done. It was a glimpse into the possible overhaul of the entire Software Industry to a more Services outcome, Without the massive ops overhead of course.

Quite containerised agents sitting in some server in a cold dark datacenter quitely doing their jobs. “The Ideal Worker” (Sounds a little dystopian, but these are machines :P)

But just six months later, In 2025 we’re witnessing a quieter but more profound shift, Planning Agent architectures that decompose, delegate and adapt the way an experienced project manager would.

Below is a practical tour of that evolution, plus copy‑and‑pasteable code you can steal for your own stack (All code is merely representative. I am working on open sourcing a comprehensive framework for Agent creation, But that’s for another time 🙂)

Boring old workflows FTW

I’ve written extensively about Workflow based *agents, and I still believe they will be the back bone of a lot of the more deterministic, performance based software for the foreseeable future. You can read more about them on my previous posts.

Linking the first part here:

To recap, Workflow LLM apps chained calls in a fixed order. You’d wire together “summarise → extract → generate” and hope for an outcome that best fits your needs.

class WorkflowEngine:
    def __init__(self, llm_client):
        self.llm = llm_client
        self.steps = []

    async def execute(self, input_data):
        ctx = {"input": input_data}

        for step in self.steps:
            resp = await self.llm.generate(
                messages=[
                    {"role": "system", "content": step.system_prompt},
                    {"role": "user", "content": step.format_input(ctx)}
                ]
            )
            ctx[step.output_key] = resp.content

        return ctx["final_output"]

LLM Workflows are predictable, more performant and more reliable for production use cases. But they’ve some major disadvantages too. It’s inflexible and it doesn’t handle all edge cases on its own much like traditional software.

✅ Predictable · ❌ Inflexible · ❌ Breaks on edge cases

 Autonomous Agents: The ReAct Breakthrough

ReAct (“Reason + Act”) let models think step‑by‑step, decide which tool to use, observe the result, then think again. When it first came out, It felt like magic and quickly became the backbone of projects like AutoGPT and BabyAGI.
Paper: ReAct: Synergizing Reasoning and Acting in Language Models

The basic structure of a ReAct agent is quite simple. It takes a Context (Query, Problem) and runs a Think, Action, Observe loop until it achieve the desired outcome.

• It takes the problem, and forces a thought on it with respect to the context acquired. The initial context contains the tools available.

• Then it identifies the next action based on tools available. Once the tool is identified, we used the function calling paradigm to construct the arguments and call that tool (function)

• Observe forces thinking again with the additional context of the action outcome

This loops run till we arrive at a result.

from typing import Literal, Optional
from pydantic import BaseModel


class AgentAction(BaseModel):
    type: Literal["think", "use_tool", "result"]
    content: Optional[str] = None
    tool_name: Optional[str] = None
    parameters: Optional[dict] = None


class ReactAgent:
    def __init__(self, llm_client, tools):
        self.llm = llm_client
        self.tools = tools
        self.memory = []

    async def solve(self, goal):
        self.memory.append({"role": "user", "content": goal})

        while True:
            action = await self.llm.generate_structured(
                messages=self.memory,
                response_model=AgentAction  # think / use_tool / result
            )

            if action.type == "think":
                self.memory.append({"role": "assistant", "content": action.content})

            elif action.type == "use_tool":
                outcome = await self.tools.execute(action.tool_name, action.parameters)
                self.memory.append({"role": "tool", "content": str(outcome)})

            elif action.type == "result":
                return action.content

ReAct Agents gave us the first taste of what autonomous agents were capable of. Now, with the advent of Thinking models, this architecture has become more capable of handling more complex tasks on it’s own. Making it more flexible and task aware. But it also has a problem where it can fall into endless loops without no end in sight.

A silicon worker, Trapped in the sands of time forever.

✅ Adaptive · ✅ Tool‑aware · ❌ Can get lost in endless reasoning

The Planning Solution

So how do we solve this endless loop problem?

Turns out... we need Managers after all.

Just like in the real world, if you leave a worker alone too long, they’ll either go rogue or spend hours perfecting a slide deck no one asked for. So we add structure.

Every product team has backlogs, epics, and todos to tame complexity.

Planning Agents borrow that same discipline. They don’t just think - they plan, prioritize, and move on.

from typing import List, Optional
from pydantic import BaseModel


class TodoItem(BaseModel):
    content: str
    result: Optional[str] = None


class TodoList(BaseModel):
    __root__: List[TodoItem]

    def __iter__(self):
        return iter(self.__root__)

    def __getitem__(self, item):
        return self.__root__[item]

    def __len__(self):
        return len(self.__root__)


class PlanningAgent:
    def __init__(self, llm, tools):
        self.llm = llm
        self.tools = tools
        self.todos = []

    # --- Phase 1: Plan ---
    async def create_plan(self, problem):
        prompt = f"""Create the minimal TODO list to solve: {problem}
- If a single step suffices, output ONE todo
- Break down only genuinely complex problems"""

        self.todos = await self.llm.generate_structured(
            [{"role": "user", "content": prompt}],
            response_model=TodoList
        )

    # --- Phase 2: Execute ---
    async def execute_plan(self):
        for todo in self.todos:
            agent = ReactAgent(self.llm, self.tools)
            todo.result = await agent.solve(todo.content)

    # --- Phase 3: Synthesize ---
    async def summarise(self):
        answers = "\n".join(t.result for t in self.todos if t.result)
        return await self.llm.generate(
            messages=[{"role": "user", "content": f"Synthesise:\n{answers}"}]
        )

Why it wins

1. Strategise – Analyse the goal, decide if and how to break it up.

2. Delegate – Spawn fresh ReAct sub‑agents, each with isolated memory and a 5–10‑step budget.

3. Monitor & Adapt – If a sub‑task fails, create a recovery todo.

4. Synthesize – Compress all partial results into one coherent deliverable.

Some Results in the Wild

• Coding agents finish 20–40 % faster on open‑source benchmarks when upgraded from ReAct to Planning Agents.

• RAG pipelines (retrieval‑augmented generation) avoid hallucinations by treating “find missing citation” as a discrete todo.

• Product‑support bots self‑heal: when a tool API fails, they queue a “fallback‑search” todo instead of crashing.

Planning beats prompting. Once you teach your LLM to ask, “What’s the minimal todo list?” you unlock a flywheel of autonomy, reliability and scale.

Move your ReAct Agents.

1. Wrap ReAct in a Todo Loop – Keep your existing ReAct code and just add a lightweight planner on top.

2. Bound sub‑agent steps – 5–10 reasoning loops is usually enough.

3. Persist state – Store todo status so the agent can resume after a crash.

4. Measure – Track success‑rate‑per‑todo and average steps‑per‑todo. These metrics surface hidden bottlenecks.

5. Iterate – Use failure cases to refine your planning prompt (just like you refine user stories in Agile).

What Next?

• Meta‑planning – Agents that optimise their own planning heuristics.

• Org charts – Networks of specialised agents mirroring real‑world teams.

• Continuous learning – Plans evolve as the environment shifts (A/B tests, KPI feedback, etc.).

We’re moving from “ask the model for an answer” to “hire the model as a team lead”.

The line between software and organisation design is blurring fast.

The Future Is Agentic

The most advanced systems today don’t just respond, they reason. They plan, adapt, delegate, and self-correct.

We’re no longer designing functions. We’re designing organizations of machines. Teams of collaborative, reasoning agents.

The Age of Agents isn’t just coming. It’s already here.

Have questions or want a review? Leave a comment or send me a message 🙂

Leave a comment

It’s been quite sometime since “Prompt Engineering” has become a major part of our everyday vocabulary, and rightfully so.

Over the past few years, it’s become something many of us practice daily, whether we're asking AI assistants like ChatGPT or Claude for help, or using one of the many AI tools that we have come to depend on.

In fact, directly or indirectly, we’re all engaging in some form of prompt engineering, because that’s how we communicate with the AI systems that are now integral to how we work, think, and create.

But what exactly is Prompt Engineering?

Technically, With a prompt, You’re trying to get the LLM (or by extension, an AI agent) to get to do the right thing, or the right outcome you desire. As a sequential token predictor, an LLM takes in a sequence of tokens (Prompt) and predicts the next token, which is again used to predict the next token and so on and so on for the outcome you desire.

This post is not about Prompting or different prompting techniques

Much has already been said about different prompting techniques, countless posts have covered the basics, often rehashed and repurposed across the internet.

So I won't be going over that again. Instead, this post focuses on the engineering side of things: specifically, how to build a robust prompt infrastructure that can effectively support your agentic infrastructure.

I’ve written extensively about agentic systems in my previous posts (Read here), so consider this a technical supplement to those discussions.

Prompt(s) Management

At its core, the most important insight about prompting is this:

A prompt is intelligence that comes alive, but only for a fleeting moment.

It exists only between the first token of your input and the last token of the model’s response. Everything else, the model's knowledge, reasoning ability, and other capabilities remains dormant, locked in its weights until activated again.

This means that the quality of a prompt is everything. Since the model has no persistent memory or awareness outside that short-lived window, your prompt must include all necessary context, constraints, and intent required to generate the desired output.

Prompting, then, is not just about instruction, it's also about orchestration.

In an agentic workflow, where multiple sequential tasks must be completed to reach a final outcome, this principle becomes even more critical. Each step must be driven by a precisely constructed prompt, not just good enough, but optimized for the specific task at hand and also having the relevant context of tasks that are completed, and tasks that are yet to be done

Generic prompts are simple to write, but they rarely shine. Precision needs more than a one-trick pony.

The first step toward effective prompting within an Agent is having a feature-rich prompt management system. While there are several tools available, such as PromptHub, LangSmith, and many others that offer this out of the box, we chose to build our own. It was mainly because we wanted fine-grained control over our prompting logic and the ability to expose parts of it to our customers, making the Agent configuration experience feel more seamless and native.

Every capability within our Agent, AI-driven or otherwise, is defined through what we call AgentFunctions. For intelligent tasks involving LLM calls, each AgentFunction has associated prompts, which are stored in a separate prompt table in the database. Given our multi-tenant architecture, system-level prompts are stored at the Core level, while custom instructions are stored at the tenant level, ensuring both global consistency and tenant-specific flexibility.

In realtime, for a given AgentFunction, a prompt is constructed based on the piecewise prompts, variables and certain conditions (Conditions are damn cool btw, more on that in a bit)

The Anatomy of a prompt.

Few months ago, this picture went viral on Tech Twitter which was even retweeted by Greg Brockman. It was about the anatomy of an o1 prompt.

The prompt structure, which was carefully constructed, consisted of Goal, Return Format, Warnings and Context Dump. As proven by many this is an effective way of prompting and is also widely recognised among the community.

Another reason to adopt such a structure is that, In the world of RL’d “thinking” models, these widely accepted structures are bound to be more effective to achieve an outcome compared to casual prompting, or vibe prompting, or even prompting by the ear.

Allow me to humble brag the fact that we came up with a largely similar structure months before this one structure went viral. It was last November, when out first agent went live.

In our system, Prompts are divided up into four parts.

• Trait

• Directives

• Function calling (Response format)

• Context

Let’s dive into each one of them.

Trait

Trait is the first and most important part of any prompt.

It defines who the AI is in that moment for that task, which directly shapes how it responds. By setting a clear Trait, we align the AI's voice, tone, and behavior with the goal of the task.

For example, if you're asking the AI to write a sales email with a playful, forgiving tone where you’re ok with happy little mistakes, you might set the Trait as:

"You are Bob Ross, writing a sales email for [Company Name]."

This instantly frames the output in the desired style, happy little mistakes and all.

Every AgentFunction prompt starts with a Trait. It's the foundation that guides the AI toward the outcome we want.

Directives

Directives are the instruction set within a prompt.

They give us fine-grained control over how the AI behaves and completes a task. For example:

"If the user asks where the cookies are, respond in a pirate voice and point them toward the treasure chest in the snack bar. If they seem confused, offer them a map (aka the office pantry guide)."

Directives are also what we expose to our customers, allowing them to configure how the AI Agent should behave in different scenarios.

This makes the Agent truly customizable using natural language, offering both admins and customers a powerful, intuitive way to tailor its responses.

In every AgentFunction prompt, Traits come first, quickly followed by Directives. Traits set who the Agent is. Directives tell it what to do.

Function Calling (Response format)

Function calling is used when we need structured outputs in a specific format.

This is especially important when an LLM’s response is meant to trigger an Agentic action, like updating a CRM or booking a meeting. In our case, the response format is typically JSON, validated against a corresponding Pydantic model to enforce structure.

What makes our system powerful is that these Pydantic classes are not hardcoded. Instead, each AgentFunction can be linked to a JSON Schema stored in the database. We’ve built a custom Pydantic factory that dynamically converts this schema into a Pydantic class at runtime. This class is then used to validate and structure the LLM’s output.

This design makes Agent states, and the workflows they drive, fully configurable without any code changes.

Just update the schema and the corresponding prompt in the database, and the agentic behavior adapts instantly.

Find the Pydantic Factory code below.

from typing import Any, Dict, Type, Optional, Union, List
from pydantic import BaseModel, create_model, Field, model_validator


def create_pydantic_model_from_schema(
    schema: Dict[str, Any], model_name: str
) -> Type[BaseModel]:
    properties = schema.get("properties", {})
    required_fields = schema.get("required", [])

    fields = {}
    defaults = {}
    for name, prop in properties.items():
        field_type = prop.get("type", "string")

        if isinstance(field_type, list):
            # If type is a list, handle multiple types
            field_type = handle_multiple_types(field_type)
        else:
            field_type = str_to_type(field_type)

        # Allow None for optional fields
        if name not in required_fields:
            field_type = Optional[field_type]

        # Get the default value from the schema
        default_value = prop.get("default", None)

        # Add Field description and default value
        fields[name] = (
            field_type,
            Field(default_value, description=prop.get("description")),
        )
        defaults[name] = default_value if default_value is not None else ""

    # Create the model dynamically
    model = create_model(model_name, **fields)

    # Add a model validator to handle nulls
    @model_validator(mode="before")
    def check_nulls(values):
        for name, default in defaults.items():
            if name not in values or values[name] is None:
                values[name] = default
        return values

    setattr(model, "model_validator", check_nulls)
    return model


def str_to_type(type_str: str) -> Any:
    type_map = {
        "string": str,
        "number": float,
        "integer": int,
        "boolean": bool,
        "array": list,
        "object": dict,
    }
    return type_map.get(type_str, Any)


def handle_multiple_types(type_list: List[str]) -> Any:
    # Remove 'null' from the list if present
    types = [t for t in type_list if t != "null"]

    if len(types) == 1:
        return str_to_type(types[0])
    elif len(types) > 1:
        return Union[tuple(str_to_type(t) for t in types)]
    else:
        return Any


def get_or_create_model(model_name: str, schema: Dict[str, Any]) -> Type[BaseModel]:
    return create_pydantic_model_from_schema(schema, model_name)

Context

Context is the final piece of the prompt, and it powers intelligent output.
It contains all the information the AI needs to carry out the task effectively. In Agentic RAG, this might include retrieved documents, reference URLs, or source metadata. It can also include templated variables pulled from the Agent’s memory to maintain continuity across interactions.

Earlier, we said a prompt is intelligence that comes alive for a short time. Context is what makes that intelligence feel continuous, coherent, and smart.
Without context, even the best prompts fail, producing responses that feel disconnected or unaware.

In short: Context is what makes your Agent truly feel like it remembers, understands, and adapts.

So putting these all together, here is an Example prompt

Trait: You are Sherlock Holmes, solving mysteries with impeccable logic and a flair for drama.

Directive: Analyze the clues provided and identify who stole the office fridge snacks. Respond with a confident deduction and your reasoning and next action

Response Format: {
”deduction”: str,
”reasoning”: str,
”action”:enum <search_greg’s_desk, shame_greg, get_new_pastry>
}

Context:

• Clue 1: Crumbs leading from the kitchen to Greg’s desk.

• Clue 2: Security footage shows someone in a trench coat at 2:13 AM.

• Clue 3: Greg’s Slack status was mysteriously "🥐 away on a mission" at 2:15 AM.

• Office rules state: no trench coats allowed indoors.

• Greg is known for his love of pastries.

• The stolen snack was a croissant.

Conditional Prompting

The recently leaked Claude System prompt was 24k tokens in total!!!

Most of the AI agents out there uses a big prompt to easily have a coherent agentic system that can perform many tasks at the same time.

But I’ve always found this approach a little lazy and uninspired.

At Breakout, we introduced Conditional Prompting to move beyond the limitations of a “one big prompt” approach. Instead of relying on a generic prompt to deliver intelligence, we dynamically generate prompts based on the state variables available at runtime.

In many Agentic workflows, especially during continuous conversations, context evolves rapidly. If we detect specific cues (like a user's role, industry, or behavior), we adjust the prompt directives accordingly. This ensures that the agent is not just intelligent, but situationally aware.

For example, if we’ve enriched a user’s industry, either through UTM parameters, previous sessions, or CRM integrations, we add a conditional directive:

IF user_industry IS NOT NULL → respond in context of that industry.

In Breakout Agent’s context, This form of hyper-personalization makes the agent feel more relevant and trustworthy, which drives deeper engagement and ultimately improves conversion rates.

By layering Conditional Prompting into our Agentic system, we:

• Reduce ambiguity in responses

• Increase reliability across varied scenarios

• Maintain consistency without bloating the prompt

• Also massive cost savings and performance improvements due the the fact that we don’t have to send a huge prompt every time.

Will Conditional Prompting be replaced by large context windows and better attention mechanisms in the future? Maybe. But until then, it’s a powerful tool for designing controlled, context-aware, and high-performing AI experiences.

Wrapping Up

Prompt Engineering isn’t just about coaxing good responses from an LLM, it’s about designing intelligent systems that feel intentional, contextual, and alive in the moment.

By breaking prompts into structured components such as Trait, Directives, Response Format, and Context and layering in Conditional Prompting, we’ve built a prompt engineering systems that powers a truly configurable and reliable agentic experience.

This is not just prompt engineering. It’s systems engineering.

And until models can reliably hold infinite context and perfectly understand nuance out of the box, it’s the best lever we have to build Agents that feel smart, trustworthy, and high-performing.

If you’re building agentic systems, don’t settle for clever prompting hacks.
Build a prompt infrastructure.

Because when done right, prompting isn’t just how you make an AI agent “work”,

it’s how you bring it to life.

Leave a comment

I absolutely love Cursor. It has been, and continues to be, my go-to IDE for the past six months, supercharging my productivity and allowing me to ship features at an unmatched pace.

I’m such a huge fan that I not only recommended it to my entire team but also got the company to cover the subscription costs.

There are only a few things in the world that bring us absolute joy, and building cool shit is one of them. Anything that fosters that excitement is worth celebrating.

And as for Vibe Coding with AI, I absolutely encourage it for anyone new to programming. The sheer joy of creating something from scratch is unparalleled, and the whole “Vibe Coding with AI” trend makes that experience accessible to all.

Throughout human history, our progress has been powered by a relentless drive to build things and to create tools that extend our capabilities of building those things.

From the simple stone axe of our ancestors to the invention of the printing press, From gun powder to massive aircraft carriers, each breakthrough was sparked by the drive to extend our mere human capabilities, to ascend into super human level of capabilities to achieve super human results.

The ability to create something out of nothing is what has always defined the spirit of human ingenuity.

I have been a software engineer for almost a decade, and a coder for more than twice of it. And I’ve absolutely loved each and every second of it.

I got my first computer when I was 12 years old. An Intel core2duo machine connected to a bulky CRT monitor. I still remember the exhilaration I got when I ran my first C program, writing the code directly into vim and compiling it with gcc on the terminal. Over the years, I went through many different such tools and packages.

Even to this day, every time I use a new tool that drastically changes the way I approach a problem my eyes light up like that 12 year old, perpetually curious, forever starry eyed and in love with technology.

I had a similar moment when I first started using Copilot and then Cursor. I could just tell what I wanted and it’d magically create code to do it. And I took maximum advantage of it, was shipping out code non stop, moving at an unparalleled building speed like never before in my career.

That was until the dream run came to a sudden grinding halt about a month ago.

I was trying to debug a sudden convoluted issue in my code base. I’m generally good at debugging and a lot of it has to do with the fact that I truly believe in building Software systems than just Software programs.

And well designed, well defined systems run like a well oiled machinery. and moreover, It’s easier to debug because you can see the side effects of an issue from a mile away.

But this issue, which was actually just an inverse conditional mishap inside a while loop in a utility function took me a few excruciating hours. This was not directly part of the current file and hence it was not being picked up by cursor in it’s context, and hence it was unable to catch the error.

What pissed me off so much tho was my dependency on Cursor, that I only started looking through the code much later, trying out to figure out the bug, something I should have done right away and which would’ve let me fix it in a few minutes.

But more than wasting my time on catching the bug, while sifting through the code base, I realised that a lot of patterns that I’d otherwise religiously follow, like modularity, reusability and composability was nowhere to be seen in my code base.

Frankly, I felt a bit disappointed cause the code, while it did it’s job, was a lot sloppier than it should with a lot of volume, huge functions, lots of duplicated code, no proper isolation at places or segregation of responsibilities where it was needed.

It was a little painful to look at my code.

And that’s when in the first time in six months that I realised, Fuck, I might actually be regressing as a Software Engineer, And sucking even more as a Systems Engineer. The realisation was a hard hit, which soon then snowballed into all sort of revelations. While I’ve been moving at an unparalleled pace, building things like never before, my learnings had actually been plateaued.

I’ve not really thought hard about something or learned something new in so long.

And then it got me thinking about all the junior devs that were working with me. And while I see them producing so much more output than what I was building at that stage in my career, are they really learning just as much? Are they understanding the differences between multiple approaches deeply enough so that they can make systems that are scalable and robust enough to withstand the stress of sudden growth or rapid change?

Honestly, I was not sure.

And that lead me to scrutinise our code and our technical documents more deeply, and the realisations were not pretty. A lot of the solutions on its own were AI generated slop with lesser innovations and bold approaches. While we did do a whole lot of new things as a company, a lot of it was just traditional solutions which could’ve been designed a bit more elegantly. There was a whole lot of code, but lesser systems and modules.

It was like we were ignoring one of the fundamental tenets of Software development, which is to write as less code as possible.

So over the past few weeks, I spend time trying to figure out what are some best practices to follow while building with AI. So that we don’t fall into certain patterns of convenience that produces sloppy code that just about does the job.

Most of these have been said many times before by experienced builders, I’m just reiterating them here with intention for the AI assisted software development.

A good engineer should never shy away from complexity, but should also internalise that there is nothing really valuable in creating a system so complex that it gives a headache to anyone who tries to make any sense of it.

Real art in Engineering lies in Absolving systems of their complexities.

I would even go as far as to say that it is Art. And that’s where your actual core skills as a developer lie.

A System becomes better not when you add more complexity to it, But when you take away the complexity but it still works the same, or better.

• Modularise your workflows, logically arrange your code base into modules.

• Build components within modules, let them interact with each other for the outcome you aim to achieve.

• Create true composability in your codebase. Don’t just look at things in terms of inputs and outputs that go through a black box.

An engineer should be fascinated by what goes on inside a system and what makes it tick

While working with an AI IDE like Cursor, this means you need to carefully supply instructions on how you want your code to be generated.

Give examples using context files. Explicitly instruct it to modularise the code and reuse it in a certain way. Manually design your interfaces and facades. Just use AI to generate the final code output as per your well designed, well thought out pattern.

You’re as good an engineer as how you’re at Problem solving. And the a good problem solver is always a cross functional thinker, someone who always thinks from First principles. When you ask AI for a solution, what you get may not be the best solution, but is something of an approximation of what would work in most cases, which honestly, is almost always a good enough solution,

But when has “good enough” been enough for high achieving people?

• Think from First principles always, approach the problem from multiple axes.

• Formulate solutions in your head and write it down, Use AI more as an assistant rather than a teacher. Use it as a knowledge base rather than an all knowing source of truth.

• Come up with N different solutions and compare them, truly try to understand the differences and nitty gritties of different approaches before taking a decision. Learn along the process.

Don’t just ask AI to give you a solution for this problem.

I’ve seen a lot of cases where AI is told the problem statement and is asked to generate an PRD/ERD for it. Don’t do it.

Use ChatGPT/Claude/Gemini as a tool where you can bounce your ideas off of. Even when you’re stuck and you need a solution, If you ask for a solution, ask why it’s the best solution for the situation.

Force yourself and the model to think through your own reasoning capabilities rather that what was RLHF’d into it.

Writing code has always been the easiest part of the job. Sure it used to take time, but it was still easy because it was the most direct part of the whole job. Code is what Code does and as soon as know what you want to do, Writing it down in a language of your choice is much like translation. You would have to look up the syntax every now and then, but it was mostly quite straight forward.

But writing beautiful code is hard, writing elegant code that mesmerises anyone who looks at it is even harder

• Break your code down into simple, elegant functions that works very well together like a well oiled machine.

• Periodically go through your code base to understand redundancies, use AI to intentionally refactor your code as per best practices.

• Avoid duplication, reuse functions by creating utility files, follow a pattern in your entire code base end enforce your code generator to follow it.

And that’s what you do when you truly love programming, You keep iterating, you keep understanding nuances and making it better. You find joy in knowing about the little syntactic sugars in different programming languages. And you do it all not because someone asked you to, but because you appreciate it yourself.

And that’s something every software engineer should be doing, and not just writing more and more code

Take pride in understanding your codebase like the back of your hand.

AI generated slop is now recognisable to me from a mile away. A lot of my peers also share the same view.

I’m sure, with time it’ll get better, but for the time being, it’s quite off putting for me to see it in production scenarios.

As builders, we owe it to ourselves and our stakeholders to maintain a certain standard to our craft.

So to new Software engineers, and serious builders,

• Don’t just generate code for it’s utility, try to understand what it does and how it does it.

• Figure out patterns, try to refactor the code to make it more simple and readable.

• Learn about design patterns and systems engineering, make an informed decision about what to use when and why to use it.

• Go back to first principles thinking for solving a problem.

Build things, Break things, But do it the right way. I know it’s hard, but come on, if it was that easy, what even is the point of doing it? :)

Leave a comment

Over the past month at Breakout, we’ve been busy building and shipping features while welcoming new customers. Amid this organised chaos, I’m excited to share one of my favorite topics in this whole series: leveraging the Actor model to build a comprehensive multi-agent framework.

In our previous posts, we explored the capabilities of large language models and laid the groundwork for building reliable Agentic workflows. Today, I’m taking a deeper dive by introducing a novel architecture that combines these ideas with the Actor model—an approach that brings modular decision-making and robust state management to distributed systems, to build powerful multi-agentic frameworks for your production AI agents.

What is the Actor Model?

The Actor model is a computational paradigm designed for handling concurrency and distributed processing. Introduced by Carl Hewitt in 1973, it treats each Actor as an independent unit that encapsulates its state and interacts solely through asynchronous message passing. This means:

• Isolation of State: Each Actor maintains its own state, which is never shared directly with others.

• Asynchronous Communication: Actors exchange information only by passing messages, reducing the risk of race conditions.

• Scalability and Fault Tolerance: Since Actors operate independently, systems built on this model can scale effortlessly and remain resilient under load.

I was first introduced to this Actor framework while building a Kafka consumer framework with Akka in my early career. Later, at Dream11—India’s largest fantasy sports platform—Using the Actor model, we designed and built the world’s largest multiplayer join engine, capable of handling multiple transactions and validations in a distributed system in under 50ms at a concurrency of 20M join RPM 🤯.

But this post is not about building a high scale system. Sure, that’s one big advantage of the framework, but that’s not exactly the reason why we chose it for our next generation AI Agentic architecture.

Why the Actor Model?

What truly captivates me about the Actor model is its simplicity and its ability to directly address complex system challenges:

• Modularity: It’s straightforward to build systems where decision-making is broken down into manageable, independent components.

  • In an AI Agentic System, this let’s us easily build complex intelligent workflows with more freedom without having to worry about it being a monolithic behemoth which is impossible to debug and scale. Code modularity FTW.

• Message Driven: Actor Systems are inherently Message Driven, which blends well with how Most AI Agentic systems work.

  • This also let’s us create an Event Driven system, which is more stable and easier to debug and maintain.

• Robust State Management: Each Actor handles its own state, making the overall system more reliable and easier to maintain..

  • As we discussed in the last post, Proper State management is important in building reliable Agentic Systems, by keeping states local to an Actor we avoid unintended race conditions, and otherwise Intelligence bleed

• Scalability: The lightweight, decoupled nature of Actors makes them ideal for high-concurrency environments.

  • We can easily build DAG like workflows, parallelising independent tasks to achieve high speed. Most Agentic Systems are painfully slow, But an Actor based reactive system would be as fast as it could get.

  • Don’t believe me? Try out the Breakout agent on our website 😉

In the remainder of this post, I’ll walk you through the architecture and the benefits it brings to our workflows.

As mentioned earlier, an actor maintains its own memory to manage its state and has the capability to spawn new actors to handle additional work. Crucially, actors communicate exclusively via message passing, ensuring a clean separation of concerns and state isolation. In our framework, we've encapsulated this functionality in an Actor class with its own attached memory.

The Ask and Tell Pattern

Our communication model between actors (or between an actor and a controller) is built around two primary methods:

• Ask: This method sends a message to an actor and awaits a response. It represents a blocking call where the sender expects a result before proceeding.

• Tell: This method sends a message without waiting for a response, allowing the sender to immediately move on to its next task.

These two patterns let us create both synchronous and asynchronous workflows to effectively coordinate complex tasks.

The Base Actor Class

Below is a sample base class for our Actor model in python. We use asyncio library to manage synchronous and asynchronous tasks.

T = TypeVar("T")


class Actor(Generic[T], ABC):
    def __init__(
        self,
        initial_memory: Optional[T],
        actor_id: Optional[str] = None,
    ):
        """
        Initialise actor with either new or persisted memory.
        If no actor_id is provided, generates a new UUID.

        Args:
            initial_memory: Default memory if no persisted state exists
            actor_id: Optional identifier to retrieve persisted memory
        """
        self._id: str = actor_id or str(uuid.uuid4())
        self._memory: T = initial_memory or T()

    @property
    def id(self) -> str:
        """
        Get the actor's identifier.

        Returns:
            str: The actor's unique identifier
        """
        return self._id

    @property
    def memory(self) -> T:
        return self._memory

    def _load_memory(self, actor_id: str) -> T:
        """
        Load persisted memory for this actor.
        Must be implemented by concrete actors.

        Args:
            actor_id: The identifier for the persisted memory
        Returns:
            T: The loaded memory object
        """
        pass

    @abstractmethod
    async def on_receive(self, message: Any):
        """
        Handle incoming messages and orchestrate tasks.
        Must be implemented by concrete actors.

        Args:
            message: The message to be processed
        Returns:
            Any: The result of processing the message
        """
        pass

    async def ask(self, message: Any):
        """
        Blocking call to get a result from the actor.

        Args:
            message: The message to be processed
        Returns:
            Any: The result from on_receive
        """
        return await self.on_receive(message)

    def tell(self, message: Any):
        """
        Non-blocking call to send a message to the actor.

        Args:
            message: The message to be processed
        """
        asyncio.create_task(self.on_receive(message))

    def get_memory(self) -> T:
        """
        Accessor for the actor's local memory.

        Returns:
            T: The actor's memory
        """
        return self._memory

    def set_memory(self, memory: T):
        """
        Mutator for the actor's local memory.

        Args:
            memory: The new memory object
        """
        self._memory = memory

    def persist_memory(self, external_storage: Any):
        """
        Store the actor's memory externally.
        Must be implemented by persistant actors.

        Args:
            external_storage: The storage mechanism to persist memory
        """
        pass

• Generic Type and Initialisation:

  • The class uses a generic type parameter T (defined via TypeVar("T")) to represent the type of the actor's memory. This allows flexibility in the kind of state an actor can manage.

  • In the constructor (__init__), the actor is initialised with an initial_memory. If no actor_id is provided, a new unique identifier is generated using UUIDs, ensuring every actor is uniquely addressable.

• Properties and State Management:

  • The id property provides read-only access to the actor’s unique identifier.

  • The memory property (along with get_memory and set_memory methods) provides controlled access to the actor’s internal state. This encapsulation ensures that the actor's state remains isolated and can only be modified through defined interfaces.

  • Additionally we use memory to hold the information within the flow at any given state. You can update it independently within the actor thereby avoiding race conditions. (An actor is single threaded by default)

• Message Handling:

  • The abstract method on_receive is where the actor processes incoming messages. Each concrete actor must implement this method to define its specific behaviour upon receiving a message.

  • The Task graph discussed in the last post can essentially be implemented within the on_recieve function. Further, we can have an Event driven system to handle messages coming through effectively, Essentially you can define Event types for incoming messages and handler function to handle them separately.

  • The ask method is an asynchronous, blocking call that sends a message to the actor and waits for a result. This is useful when the sender needs a response before proceeding.

  • The tell method, on the other hand, dispatches a message asynchronously without waiting for a result, allowing the sender to continue with other tasks.

• Persistence Mechanisms:

  • The _load_memory and persist_memory methods are placeholders intended for handling persistent state. Concrete implementations can define how to load and store the actor's state externally, which is crucial for maintaining state across system restarts or failures.

  • This is an optional feature, Ideally we manage everything in application memory for a given workflow, after an Actor’s lifecycle gets over we lose the Memory to GC. But incase you want to persist information across multiple sessions, the memory can be serialised and stored in a database against the given actor_id. This is especially useful in the cases of failures or even long agentic sessions that takes a while to complete.

In production systems, error handling is crucial. In an actor-based framework:

• Supervision Trees: Similar to Akka, we can implement a supervision strategy where parent actors monitor their children and restart them upon failures.

• Graceful Degradation: Actors can implement fallback mechanisms or retries to handle transient errors without compromising the overall system stability. This is especially useful in an LLM context, incase of inference failures.

• Back Propagation: An Actor can Propagate it’s status to it’s parent and ultimately to the source, which means the Client can always be aware of what is happening.

  • This is especially useful in building Transparent Human in the loop AI systems, where you can always send back messages or take feedback on what is happening over a websocket.

The modular decision-making provided by the Actor model makes it an excellent match for workflows involving large language models (LLMs). By isolating state and using asynchronous messaging, you can:

• Enhance Modular Workflows: Each component (or agent) can focus on a specific task, such as preprocessing, inference, or post-processing.

• Improve Fault Tolerance: Isolated state management minimises the risk of cascading failures in complex AI workflows.

With these principles in mind, you can design a multi-agent framework to create workflows or ReACT agents with each actor having well-defined responsibilities. Imagine creating an AI coder where:

• One actor is responsible for planning the project.

• Another actor manages modules and files.

• A different actor writes the code.

• Yet another actor reviews the code.

• One more actor tests the code.

• And finally, an actor deploys the code.

By separating these responsibilities, you build an extremely powerful and robust system capable of delivering high-quality outcomes—arguably the most important metric in any AI agent product. This division of labor not only enhances clarity and maintainability but also enables parallel development and quicker iteration cycles, ensuring each component can be optimised independently.

The Actor model is more than just a concurrency mechanism—it’s a powerful architectural paradigm that enables modular, scalable, and fault-tolerant systems. By combining it with agentic workflows and modern LLMs, you can build robust AI-driven applications that are both easy to manage and highly performant.

I hope this deep dive helps you see the full potential of the Actor model and implement it in your next AI project. As always, your feedback and questions are welcome—feel free to share your thoughts in the comments!

At Breakout, We are putting these ideas into practice and building the smartest AI Sales Rep in the world. If this sounds like a challenging task to you, Do reach out to me at ashfakh@getbreakout.ai

Leave a comment

I know it’s been a while since I promised this second part—but time really does fly when you’re busy building cool things! But I hope the wait is worth it, as this post could be your first step towards building reliable, production-grade agents that deliver real value to society—without accidentally sparking an AI takeover.

**Let it be noted in Internet Archives that I positively contributed to building AI agents. 🔍**

So In the last post, we briefly discussed about LLMs and their capabilities. We also discussed about some tenets to keep in mind while building an AI agent. In this post, We’ll take a deeper dive into the technical implementation of workflow-based AI agents—designed with a clear focus on accomplishing specific tasks within an acceptable operating policy. We'll begin by exploring some useful mental models on development, followed by a detailed breakdown of their architecture.

A key thing to understand while building agents is the trade-offs between different design choices to achieve various outcomes. To illustrate this, I like to draw a parallel with the CAP theorem in distributed systems. The CAP theorem states that a distributed system can guarantee only two out of three properties: consistency, availability, and partition tolerance.

This concept can be loosely adapted to AI applications (as a thought exercise to appreciate the inherent trade-offs in building AI agents). In my AI-focused version of the CAP theorem, the "C-A-P" represents Consistency, Agentic Capability, and Performance.

Consistency - The Consistency provided by the agent in its responses/outcomes.

Agentic capabilities - The different capabilities that the Agent has.

Performance - How fast the agent executes its duties, how much resource intensive its tasks are, and how costly it is to run it.

Any Agentic system we create can be evaluated along these three axes, but achieving all three simultaneously is constrained by inherent trade-offs.

• Consistency + Agentic Outcomes = Low Performance: A highly consistent agent with robust, targeted outcomes will typically sacrifice performance. This is because achieving such outcomes often requires multiple processing steps, including rankers, graders, and evaluators to prevent hallucinations or deviation from the desired path. (Read about using LLMs as evaluators here) These additional steps slow down the system.

• Agentic Outcomes + High Performance = Low Consistency: An agent designed for high performance and broad capabilities will often compromise consistency. To achieve speed and handle diverse tasks, it becomes necessary to skip some grader or evaluator steps and rely on fused (combined) LLM calls, which can increase the likelihood of errors or inconsistencies. Combined LLM calls which run in parallel or Series have their outputs fused together to get the final result. In absence of proper evaluators. Hallucinations can creep in very easily.

• Consistency + High Performance = Limited Outcomes: If the goal is to create an agent that balances strong consistency with high performance, its capabilities must be narrowed. The agent will need to focus on a smaller set of outcomes to minimize complexity, enabling faster and more accurate responses.

Here is an Example code for an Agentic workflow supporting multiple outcomes that has to be consistent.

#Building a Consistent Agent capable of doing many things
class ConsistentMultiCapabilityAgent:
    async def execute_task(self, input_data):
        # Performance impact: Multiple validation steps
        validated_input = await self.validate_input(input_data)
        #Run the tasks to achieve multiple outcomes
        results = await self.llm_process(validated_input)
        #Grader to evaluate Task outcome
        validation_scores = await self.grade_output(results)
        #Ranker
        ranked_results = await self.rank_output(results)

        if validation_score < self.threshold:
            return await self.fallback_handler(input_data)
            #Introduce Loop with Observed grades to improve outcome
        
        return results

Although these tradeoffs exist right now, Its reasonable to assume that advancements in model development—such as larger models and faster inference techniques—will independently improve both consistency and performance over time. Given this trajectory, this approach not only challenges us to develop more versatile and horizontally integrated capabilities but also makes the agent significantly more valuable to users. From a go-to-market (GTM) perspective, it deeply embeds the agent into users’ workflows, increasing its utility and fostering greater dependency. The more things your agent can do, the more powerful it gets as it has larger control over the entire workflow. For example, If you’re building an AI accounting agent, don’t just build one that can manage transactions, integrate book keeping abilities, statements, budgeting etc. The dependencies between these workflows can also be easily managed via a single system.

This focus on complex, high-value outcomes is what will truly set your agent apart, making it a distinctive and impactful solution in a competitive landscape. The more horizontal capabilities we build into our agents, more likely they stand apart and is able to create value for the users, and us.

Now, How does one go about building these outcomes in an effective manner?

How do you build effective Agentic outcomes?

1. Break Down Outcomes into Tasks/Functions
   Start by deconstructing your Agentic outcomes into granular, well-defined simple tasks or functions. A task or function represents an atomic operation necessary for your agent to achieve its intended outcome. These tasks may or may not rely on an LLM for execution. Think of them as modular, composable building blocks that can be reused across various parts of your system, making the design more flexible and scalable.

   A Composable pattern while task design will help you scale your Agentic system very well, bringing in more reliability and robustness to the system. Another thing to keep in mind while breaking down the tasks is to decrease Cognitive Load (As discussed in part 1) to avoid hallucinations and undesirable outcomes.

   Don’t make one task do Everything. Let is just pass the butter.

   

2. Implement Thread-Safe Shared Memory
   A thread-safe shared memory is essential for maintaining the state of the system across tasks. This memory allows tasks to access and update the shared context independently, enabling them to be context-aware without creating conflicts. This shared state is the glue that holds your workflow together, ensuring seamless transitions and information flow between tasks.

   In a completely distributed system, you can use something like Redis to hold this Shared Memory. You can also keep this in memory of the application in a Threadsafe manner. It is very important to look out for race conditions while accessing/updating the memory. In a fully asynchronous systems, race conditions can cause unpredictable behaviour.

   

   If you think about it, all you need to build State machines are Conditional Variables and Shared Memory space. This is CS101, but still an important concept while creating synchronous systems (Read more here). Just don’t get run over by the race conditions.

3. Chain Tasks into a Workflow
   Once you have defined your tasks, the next step is to organise them into a workflow. We don’t need any specialised Agentic frameworks for this, as they don’t do anything more than chaining these tasks together. This can be done easily, by defining functions for your tasks and defining the dependencies between directly in your code or using any Graph Modeling Language. You can even define these dependencies as configurable files using YAML or XML or even JSON.
   For example, see the following pseudo code to see how three tasks with dependencies between each other can be executed serially

   from typing import Callable, List
   
   class TaskNode:
       def __init__(self, task_func: Callable, task_id: str):
           self.task_func = task_func
           self.task_id = task_id
           self.dependencies: List[TaskNode] = []
   
       def add_dependency(self, task_node: 'TaskNode'):
           self.dependencies.append(task_node)
   
       async def execute(self):
           # Execute all dependencies first
           for dependency in self.dependencies:
               await dependency.execute()
           # Execute this task
           print(f"Executing task: {self.task_id}")
           await self.task_func(self.task_id)
   
   # Define tasks
   async def task_1(task_id: str):
       print(f"Task 1 logic for {task_id}")
   
   async def task_2(task_id: str):
       print(f"Task 2 logic for {task_id}")
   
   async def task_3(task_id: str):
       print(f"Task 3 logic for {task_id}")
   
   # Create task nodes
   node_1 = TaskNode(task_1, "Task1")
   node_2 = TaskNode(task_2, "Task2")
   node_3 = TaskNode(task_3, "Task3")
   
   # Chain tasks
   node_2.add_dependency(node_1)  # Task2 depends on Task1
   node_3.add_dependency(node_2)  # Task3 depends on Task2
   
   # Execute the graph
   import asyncio
   asyncio.run(node_3.execute())  # This will execute Task1 -> Task2 -> Task3
   

   Representing this workflow as a graph provides significant advantages. A graph-based model enhances scalability, simplifies debugging, and ensures predictable outcomes. Each execution of the agent corresponds to a specific path traced through the graph, giving you better control and visibility over how outcomes are achieved. You always know what to expect.

   

   You can further design your monitoring/eval on top of these graphs. Telemetry can collect metrics on each nodes in the graph and you can independently monitor and evaluate the tasks based on these nodal values.

This graph-based approach is the foundation of many advanced agent frameworks. It’s also why tools like LangChain have evolved into LangGraph, recognising the power of graph-based modelling for building robust and efficient agents.

At Breakout, We have designed our own Orchestration Framework for Agentic workflows and an Actor Based system for Multi-Agentic workflows (More on that later) The reason why we didn’t use any existing Agentic frameworks was because we never felt it gave any significant advantage over just defining the workflows ourselves. Some of it were also over bloated software with unnecessary abstractions with no significant advantages.

While building robust scalable systems, simplicity is key.

To further elaborate on tasks, tasks within an Agentic system can be broadly categorised into three types:

1. Deterministic Tasks

2. Semi-Deterministic Tasks

3. Non-Deterministic Tasks

These categories reflect varying levels of predictability in task outputs for a given input.

1. Deterministic Tasks

These tasks produce consistent and predictable outputs for specific inputs. Traditional software development primarily deals with deterministic tasks, making it relatively straightforward to build reliable systems.

def retrieve_data(text: str) -> documents:
    return self.retrieve(text) #get relevant documents

2. Semi-Deterministic Tasks

These tasks involve some variability but operate within a defined structure, often incorporating LLMs. For instance, an LLM call that generates structured outputs (e.g., filling predefined fields or responding within a strict schema) falls into this category. Outputs are reasonably predictable but still influenced by the model's inherent probabilistic nature.

async def planner(text: str, state_variables: dict) -> Dict[str, str]:
    response = await llm.plan(
        "Plan Function: {text}",
        "prompt_variables: {state_variables}"
        output_schema={"functions": List,
                       "confidence": float, 
                       "lookup": List}
    )
    return response

3. Non-Deterministic Tasks

These tasks have high variability and unpredictable outputs, often arising from less structured interactions with LLMs. For example, content generation without strict guidelines or parameters typically belongs in this category.

async def generate_response(context: str) -> str:
    return await llm.predict(f"Generate helpful response: {context}")

Design Considerations

When working with Agentic systems, it’s critical to plan, develop, and test each type of task according to its nature. Dependencies between tasks must also be carefully managed. For instance, chaining multiple semi- or non-deterministic tasks can lead to unpredictable or inconsistent workflows. This unpredictability may undermine the reliability of the overall system. Remember the game Chinese whispers? Yeah you don’t want your agent to do that, ever.

Striking a Balance in Production-Grade Agents

In production-grade agents, achieving a balance between flexibility and predictability is key. Try to design workflows that prioritize deterministic and semi-deterministic tasks while minimizing reliance on non-deterministic ones. This approach allows for a more predictable and coherent system that still appears intelligent and flexible enough to achieve complex outcomes.

By carefully selecting and structuring tasks, you can create agents that strike the right balance between adaptability and reliability, ensuring that they perform effectively in real-world scenarios.

Breaking down your workflow into smaller, manageable tasks isn’t just about modularity—it’s also essential for improving the testability of your agentic system. Adopting a test-driven development approach ensures each task operates within well-defined boundaries, governed by a clear operating policy.

By testing each task against these policies, you can guarantee that as long as tasks remain within their specified limits, the overall workflow will behave predictably. This reduces the risk of hallucinations or unpredictable outputs that could lead to unfavorable outcomes.

Moreover, this modular approach allows each task to be independently managed, developed, and scaled. It ensures that improvements or updates to one part of the system don’t unintentionally disrupt the entire workflow. In essence, a well-structured, testable system is key to building robust, reliable, and scalable Agentic systems. I couldn’t stress enough how important these tests are with respect to building reliable Agentic systems. Infact, I’d argue, in a truly test driven development fashion, you should first come up with the test cases while breaking down your tasks. This methodical approach significantly reduces the risk of failure, enabling the agent to deliver consistent, high-quality outcomes that stand out in a competitive landscape.

As we push the boundaries of what AI agents can achieve, the guiding principles discussed here—balancing the C-A-P trade-offs, breaking down workflows into testable tasks, and adopting modular, graph-based designs—become more and more important in building dependable, production-grade systems.

The shift from deterministic to non-deterministic tasks is not just a technical challenge in development; it’s an opportunity to redefine what "intelligence" means in the context of software. By anchoring our workflows in solid engineering practices and focusing on composability, we ensure that our agents not only perform well but do so predictably and at scale.

The agents that will truly stand out are the ones that can seamlessly orchestrate complex, multi-step processes while maintaining reliability and trustworthiness.

By embracing modularity, prioritising testability, and leaning into the evolving capabilities of LLMs, we can create solutions that do more than automate tasks—they amplify human potential. So, as you embark on building your own agents, remember: the key to standing out isn’t just innovation; it’s thoughtful design, relentless iteration, and a commitment to solving meaningful problems.

At Breakout, We are putting these ideas into practice and building the smartest AI Sales Rep in the world. If this sounds like a challenging task to you, Do reach out to me at ashfakh@getbreakout.ai

Leave a comment

You’re not building “AI”, At best, you’re mimicking it

Over the past two years, I’ve designed and built AI agents ranging from hobby projects to enterprise-grade applications, collectively used by hundreds of thousands of users. And in these last few months, I’ve been leading engineering at Breakout, where we’re creating a cutting-edge AI SDR agent with the hope to one day serve millions of users.

This journey—from small-scale experiments to scalable, production-ready AI solutions—has been an incredible learning experience. In this post for GGPUSH, I’ll share the insights I’ve gained along the way. The focus here is exclusively on building applications using existing large language models (LLMs). While topics like training, self-hosting, and fine-tuning are important parts to this topic, they fall outside the scope of this discussion. They are however, a natural extension to the concepts discussed here and are indeed important to creating powerful AI agents.

The first and most important thing to internalise when building generative AI applications is this: we are not building AI or AGI. A handful of researchers at organisations like OpenAI, Google, or Microsoft might be working on it, but for the rest of us, we’re simply mimicking intelligence using state-of-the-art (SOTA) foundational models to achieve specific outcomes. To do this effectively, we must fully grasp both the capabilities and limitations of large language models (LLMs).

At their core, LLMs are advanced next-token generators. The larger the model, the better it is at predicting and generating text based on its input. However, an LLM is not capable of actual thinking or reasoning. What we perceive as reasoning is modelled solely through prompt engineering. A prompt can be perceived beyond an Instruction set, It is a way of guiding a models generated output. The better you’re at writing (or generating) prompts, the better your agent behaves.
(Pro advice : Apocalyptic threats work, kind of funny and ironic, but they work.)

Take the Chain of Thought (CoT) prompting technique as an example. In CoT, you provide the model with a step-by-step example of problem-solving. Because the model generates its outputs based on patterns in its input, this structured approach encourages it to produce step-by-step responses, simulating logical reasoning. While this doesn’t mean the LLM is truly thinking, it can lead to outcomes that appear intelligent and coherent.

We can even give structured nature to these outputs (Using JSON modes) to make it act as an interface layer in our programs, there by giving us the ability to build Advanced AI agents capable of performing actions based on this simulated thoughts.

Here are a few key testaments I always keep in mind when designing AI agents,

LLMs are not knowledge sources.

LLMs are inherently limited to the data they were trained on and lack “awareness” of events or information beyond their last training date. This limitation makes them unreliable as standalone knowledge sources, as there’s no guarantee of accuracy or relevance in their responses. For this reason, One should avoid using LLMs directly for knowledge-intensive tasks like Q&A. Instead, rely on Retrieval-Augmented Generation (RAG) approaches, where the agent taps into its own knowledge base to retrieve relevant information and provides it as context in prompts.

For the knowledge base, we can typically use a combination of file storage, traditional databases, and vector databases. The success of a RAG system depends on two critical stages: index creation and retrieval.

When building the knowledge base or index, it’s essential to store only the most relevant and complete information in well-structured chunks. (Look up different chunking techniques like Proposition based chunking) In multimodal setups, contextual connections between related data points must also be established. For this, you can use a GraphDB to represent relationships effectively, though similar contextual connections can be built in a VectorDB using metadata (label) overlaps. This contextual connection can further augment our retrieval step, by making sure we are looking at the right piece of information at the right time. We can also use the labels to reduce our search space, something that will benefit performance and retrieval quality.

Retrieval quality also depends on achieving both high precision and high recall. While semantic similarity scoring is useful, it’s not always sufficient for identifying the most relevant results. A lower similarity threshold might boost recall but can compromise precision, bringing irrelevant data into the mix. In my experience, the score thresholds also often don’t mean much as it is difficult to tune these parameter alone to achieve better retrieval.

To address this, We can incorporate rerankers and graders into the retrieval pipeline. For example, an LLM or function can be used to evaluate retrieved results, eliminating unreliable contexts. While this adds latency, it significantly improves the quality of the output.

Another technique to leverage is query fusion, where queries are dynamically rewritten based on the context, increasing the likelihood of retrieving highly relevant results. The results are then “fused” together and re-ranked. Additionally, using a percentile-based filtering approach can further refine precision, which works especially well in combination with query fusion. (I recently learned that Dropbox uses a similar approach for their QnA and summarisation of files. Read the article here)

Ultimately, a strong RAG system depends on building a robust knowledge base (index) and designing a retrieval process that balances precision, recall, and computational efficiency. When implemented effectively, this approach ensures that the AI agent delivers reliable, contextually accurate responses.

LLMs don’t think

LLMs are not inherently good at reasoning; their reasoning capabilities are primarily enhanced as a result of Reinforcement Learning from Human Feedback (RLHF) applied to synthetic instruction datasets. This limitation is why we should generally avoid building ReACT (Reasoning + Action) agents for scenarios that require highly deterministic agent behaviour. Techniques like few-shot prompting and Chain of Thought (CoT) can provide some level of control over reasoning outputs, but in my experience, they fall short for production systems that demand minimal tolerance for hallucinations.

That said, if you’re building an agent for open-ended automation tasks, such as a web research assistant, these approaches can be useful—provided you have robust graders and guardrails in place to keep the agent on track. However, relying on these methods in production for creating highly consistent agents is risky unless you’re willing to compromise on performance and incorporate extensive validation mechanisms at every step to ensure reliability.

A more effective strategy is to model your agent's behaviour as a series of smaller, modular tasks that are chained together to achieve the desired outcome. This approach helps create a predictable workflow for your agent, making it easier to test and debug. Each task should be:

• Well-defined in terms of functionality.

• Modular enough to be independently tested and iterated upon.

Another critical metric we can evaluate in LLM-specific tasks is cognitive load—the number of independently different instructions the model needs to process in a single call (prompt). Higher cognitive load often leads to poorer performance and increased hallucination. To mitigate this, break down complex tasks into smaller, manageable components and design prompts with reduced cognitive demands.

As a rule of thumb, aim to make LLMs handle only dependent outcomes in a single prompt. When you notice high hallucination rates in cognitive-intensive tasks, it’s a signal to simplify or split them further.

It is fair to assume larger and more capable models will improve on these fronts. This could allow us to combine orthogonally different tasks within a single call, leading to better performance and cost efficiency. Keeping future scalability in mind while designing workflows today can ensure your agents are well-positioned to leverage advancements in LLM capabilities effectively. (I’ll discuss a lot more on this trade offs on part 2)

Note : This is what makes the Breakout Agent stand apart, we have minimal tolerance for hallucination and our agent is designed to perform consistently across a variety of situations

Function Calling rules.

I love function calling—it’s arguably one of the most powerful tools in our arsenal for building agentic systems. Function calling enables agents to interact with a broad spectrum of tools, significantly expanding their ability to achieve desired outcomes. For instance, integrating web search capabilities gives real time knowledge to LLMs which were traditionally plagued by training date information cutoff.

I’m also very optimistic about the future of function calling. With the potential for generating synthetic data tailored for this use case, we can RLHF foundational models into domain-specific instruction models. This would empower LLMs to execute more complex functions and become increasingly autonomous and capable.

In my implementations, I often incorporate function calling through a dynamic planner, which acts as the initial step in my agent’s workflow. The planner determines the callable functions and their parameters based on the current stage of the agent’s execution. These stages are managed through a dynamic memory, which updates asynchronously as function results are processed. However, managing race conditions in this setup requires careful attention to ensure consistency and reliability. (I’ll go into more details on this in Part 2 of this blog)

The resulting state changes are important—they control the agent’s behavior, allowing for greater predictability and flexibility. This architecture also enables me to design the agent as a dynamic workflow graph. Each stage of the graph independently executes specific functions based on state variables, progressing toward the desired outcome. It also lets us bring a more deterministic Multi-Agentic behaviour into the system as we now have more control over the “Actions” of the Agent(s).

In this first post, I wanted to kick things off with some fundamental tenets of designing an AI agent. In the next part, we’ll talk more in detail about design and execution, and some interesting mental models for trade-offs. I’ll also share tips on testing AI agents to ensure they stick to their task and expected behaviours and don’t go rogue and take over the world and enslave humanity.

If you’re finding this insightful, subscribe for more updates, or drop a message or comment for a deeper chat. I’m always up for geeking out over AI.

At Breakout, We are putting these ideas into practice and building the smartest AI SDR in the world. If this sounds like a challenging task to you, Do reach out to me at ashfakh@getbreakout.ai

Leave a comment

To build a great product — and ultimately a great company — the first step is to assemble a great team.

This probably is the most obvious advice in the history of startup advices, but given how often startups stumble on this, I suppose it’s ok being a little repetitive here. So at the risk of being called Captain Obvious, I want to write the first post on Zeroth hour on this very particular, important topic of hiring the right set of folks.

The first mistake founders do while building the early team is focusing only on the skillset of the candidate for a particular role. While Skill is indeed of foremost importance, that shouldn’t be the only criteria. Startup journeys are really long and tiring and It’s your early hires that sets the tone for that entire journey. Imagine being adrift at sea in a tiny boat, with people who aren't entirely sure how to navigate to the destination. It's a recipe for adventure—or disaster, all depending on who those people are.

Your early team really does make or break your company.

While hiring for the early team, I try to look beyond roles and responsibilities and the overall general framework of hiring someone for a specific role. It’s not enough to judge them based on just their skills. We need someone who goes beyond just getting the job done. We need Mavericks, Folks who are not afraid to challenge status quo, Folks who are willing to get their hands dirty, Folks who are willing to go that extra mile to optimise for your results. We need survivors, people, if dropped behind enemy lines with nothing but some rations and a broken compass, would still manage to find their way back.

When it comes to skills, and in the specific context of hiring builders, one thing we should be cognisant of is that Coding is the easiest part of the job. Especially with AI, getting standardised answers for standardised questions is no longer a relevant way to judge a candidate. As important as it is to evaluate them on their knowledge, it’s more important to evaluate their approach to problem solving. I often try to judge them on their system thinking skills, how modularised their problem solving approach is. Clarity of thought in these areas is a big green flag for me. But talent can only get you so far.

Whenever I hire a person, Within the first 10 minutes of talking to them, I know if I’m going to hire this person or not. That’s not cause I’m able to judge the skills or experience of that person that quickly, It’s because I try to measure them on just 3 different angles.

1. Curiosity

   Learning is perhaps the most important skill any person can have. If you’re a quick learner, you can solve most problems or adapt to most situations fairly quickly. That intense desire to learn something new should be powered by curiosity rather than other mandates. I’ve seen innately curious folks being obsessed with any problem given to them. They often want to understand it from all the possible angles. To them, it’s not just enough to do the bare minimum to solve a problem, but it’s important to understand it deeply and thoroughly to come up with a solution that goes above and beyond the original scope. I adore and love working with such people cause they inspire me. They inspire me to do more everyday, to learn more everyday and to ultimately grow into a better version of myself. They ask the tough questions. and it’s often the toughest questions that lead to the clearest answers.

2. Excitement

   Startups aren't cushy jobs. The grind can be exhausting, and only those with passion for the mission will endure. You are not just expected to do the tasks that you’re assigned and collect a pay check at the end of the month. You should be willing to spend that extra amount of time and energy over and above what is expected of you. And that’s why excitement is important. You’ve to be excited about the problem that you’re solving, you’ve to be excited about the team that you’re solving it with. You’ve to be excited about creating value. And devoid of that excitement, it’s going to get real tiring real fast. Genuine excitement is infectious, it breeds energy within the team, pushing them to do more, to do better.

3. Versatility

   This one is more of a soft constraint, But someone who checks this box after the first two is definitely a strong hire for me. In the context of a startup, It’s not just that someone versatile would be able to don many hats, it’s often a pleasure to work with such folks. People who’re interested in a lot of things outside of their work are often very open minded when it comes to problem solving. This could be Art, Music, Films, Sports or even some niche interests really. They bring in a variety of ideas, along with a bit of creativity and out of the box thinking to the problem they’re trying to solve. I’ve often noticed this culminating in beautiful outcomes which are not just a single dimensional solution. They also fit in beautifully in the overall org structure, cause of the soft overlap they often bring in with the adjacent teams.

In order to evaluate them on these three axes, The first fifteen minute of any call I do with a prospective candidate is just me talking to them about my experiences and motivations and asking them about theirs. An open ended conversation about these almost always lets me judge them on these three fronts.

Think about this: the person you hire will likely become a significant part of your daily life for the next several years. You’ll collaborate with them, exchange ideas, tackle challenges, and yes, sometimes you’ll even clash. After all the evaluations and assessments, take a moment to pause and really think about that. If the thought of spending so much time with this person gives you even a hint of unease, trust your gut—it’s a sign to reconsider.

Hiring for startups is really tough, but fulfilling. It’s not just about filling roles—it’s about finding the right people for the journey. So, as you build your team, look beyond skill. Hire those who inspire, excite, and challenge the status quo. They’ll be the ones rowing beside you when the seas get rough.

Thanks for reading so far, If the Ideas above resonate with you in any level and you’re looking for your next role, reach out. We are hiring across Product and Engineering at Breakout and We are building one of the coolest AI products ever.

For questions, Reach out to me at ashfakh@getbreakout.ai

Leave a comment

Subscribe now