What it is
Orchestration is the job of coordinating a larger AI system: deciding which steps run, in what order, which ones can happen at the same time, where each result goes next, and what happens when something breaks. The part of the system that does this is called the orchestrator.
The easiest way to picture it is a busy restaurant kitchen. The head chef doesn't cook every dish. They read each order as it comes in, call out who does what ("two steaks, one fish, table six"), tell the grill and the fryer to start at the right moments so everything lands together, taste a plate before it leaves the pass, and decide what to do when the fish runs out. The cooks are the specialists. The head chef is the orchestrator. An orchestra conductor is the same idea: the conductor plays no instrument, but decides when each section comes in and keeps them all in time.
In an AI system the cooks are the individual pieces: a single agent, a call to a language model, a tool such as a web search or a spreadsheet, a subagent sent off on a narrow job, or a person who must approve something. The orchestrator is whatever sits above them and runs the whole service.
There are two broad ways to build that head chef.
Code-driven orchestration. A fixed script decides. The steps and their order are written down in advance, by a programmer or in a visual workflow automation tool: first transcribe, then summarise, then write three social posts at once, then check them. The AI does the work inside each step, but it never decides what comes next. This is predictable. It costs about the same every time, takes about the same time, and when it breaks you can see which step broke. OpenAI's Agents SDK documentation says orchestrating via code makes tasks "more deterministic and predictable, in terms of speed, cost and performance."
Model-driven orchestration. A language model decides. You give it a goal and a list of tools and helpers, and it chooses the next step, looks at the result, and chooses again. This copes with tasks you can't map in advance, like open-ended research, but it is less predictable. It may take three steps or thirty, and it may go down a wrong path before noticing.
Anthropic's December 2024 guide "Building effective agents", covered in more depth in the Prompt Chaining article, draws the same line with different words. It calls the first kind "workflows", where models and tools are "orchestrated through predefined code paths", and the second kind "agents", where models "dynamically direct their own processes and tool usage". Most real systems mix the two: a fixed outer plan, with a model making choices inside one or two of the steps.
Whichever kind you build, a good orchestrator handles the same list of jobs.
Routing. Deciding where a piece of work should go. A customer email about a refund goes to one set of steps, a question about delivery goes to another. Often a small, cheap model classifies the input and ordinary code sends it down the right path.
Parallel steps. Running independent pieces at the same time. Writing a LinkedIn post, an Instagram caption and a newsletter blurb from the same summary doesn't need to happen one after another, so the orchestrator can start all three together and wait for them all to finish. Steps that depend on each other still have to wait their turn.
Retries and fallbacks. Services fail. A model is overloaded, a website doesn't respond, an answer comes back in the wrong format. A retry tries the same step again, usually after a short pause. A fallback is plan B: a different model, a simpler method, or a message to a person saying "I couldn't do this part".
Timeouts. A limit on how long any step may take. Without one, a single stuck step can hold up everything behind it indefinitely.
Cost budgets. A limit on how much the whole run may spend. AI models charge by tokens, the small chunks of text they read and write, and a system that loops or launches many helpers can spend far more than you expected. Anthropic reported in June 2025 that, in its data, agents used about four times as many tokens as ordinary chats, and multi-agent systems about fifteen times as many. A budget, and a maximum number of steps or loops, turns "it went on for hours" into "it stopped and told me".
State. Keeping track of what's done. The orchestrator needs to know which steps have finished, what each produced, and what's still waiting. Good state means that if something crashes at step seven, you can restart from step seven instead of paying for steps one to six again.
Logging. Writing down what happened: which step ran, what went in, what came out, how long it took and what it cost. When the final result is wrong, the log is how you find out which step went wrong. This is the heart of observability.
Software engineers had a version of this debate long before AI. When a business process is split across many small services, you can have a central orchestrator that tells each service what to do, or you can use "choreography", where each service listens for events and reacts on its own, like dancers who know the routine and follow the music without a conductor. Microsoft's architecture guidance sums up the trade-off: a central orchestrator gives one place to see the status of the whole job, but it can become a bottleneck and a single point of failure, while choreography removes that central point but makes the whole thing harder to follow and to monitor. For most AI projects built by small teams, a clear orchestrator wins, because being able to see and stop the whole job matters more than removing the single point of control.
Why it matters
As soon as an AI system does more than one thing, somebody or something is orchestrating it. If you paste a transcript into a chatbot, copy its summary into a second chat to write social posts, and then check the posts yourself, you're the orchestrator. That's fine for a one-off. It stops being fine when you do it every week, when there are ten steps instead of three, or when the steps run at night while you're asleep.
Orchestration is also where most of the reliability of an AI system comes from. A single model call can't retry itself, notice it has spent too much, or save its progress. Those things are properties of the system around the model. A clever model inside a badly orchestrated system still produces a mess: steps run in the wrong order, one failure silently ruins everything after it, or a loop quietly spends money for hours. A research team at UC Berkeley that studied multi-agent systems in 2025 (their paper is described in the Multi-agent Systems article) found that many failures came from system design, such as agents repeating steps or not knowing when the job was finished, and concluded that better prompts alone would not fix them.
It also matters because the choice between code-driven and model-driven orchestration is really a choice about control, cost and risk. A fixed script is cheaper and easier to trust, but it can't handle surprises. A model in charge can handle surprises, but it can also make surprising decisions. Knowing the difference lets you use the right one, and lets you ask sensible questions when a vendor sells you "an AI agent that runs your whole content pipeline". Which parts are fixed? What happens when a step fails? Is there a spending limit? Can I see a log?
Finally, orchestration is where people fit in. A well-designed orchestrator has clear places where the work pauses and waits for a person to approve it before anything is published, sent or paid. That's the practical meaning of human in the loop, and it's much easier to build into the plan from the start than to bolt on later.
Real-world examples
LangGraph: orchestration as a map with loops (January 2024 onwards)
LangChain, a popular free toolkit for building AI applications, started with "chains": fixed sequences of steps, one after another. In January 2024 the LangChain team announced LangGraph, a module for building what they called "cyclical graphs". Their launch post explained the problem it solved. Chains, like most data orchestration tools, were directed acyclic graphs: maps of steps where the arrows only point forward. An agent, though, often needs to loop: try a tool, look at the result, and go back to an earlier step. LangGraph let developers draw the orchestration as a map of steps (called nodes) with arrows between them, including arrows that loop back, and a shared state object that every step reads from and writes to. The launch examples included adding a human approval step before the agent was allowed to call a tool.
That design choice, a graph of steps with shared state, captures both halves of orchestration. The arrows are fixed code, so you can see and control the possible paths. A model can still choose which arrow to follow at the points where you let it.
On 22 October 2025 LangChain released version 1.0 of LangGraph. The announcement put three features first: durable state, so that a run interrupted by a server restart picks up where it left off; built-in saving and resuming at any point; and first-class support for pausing for human review. It named Uber, LinkedIn and Klarna among companies using LangGraph in production. Those three features are exactly the unglamorous parts of orchestration, state, recovery and human checkpoints, that matter most once a system leaves the demo stage.
Microsoft merges AutoGen and Semantic Kernel into one orchestrator (October 2025)
Microsoft had two separate open-source projects for building AI agents. AutoGen, from Microsoft Research, was known for experiments with agents talking to each other. Semantic Kernel was the more business-focused toolkit. On 1 October 2025 Microsoft announced the Microsoft Agent Framework, which its announcement said "converges" the two into a single framework. It reached version 1.0 on 3 April 2026.
The announcement reads like a checklist of what a serious orchestrator is expected to do. It listed ready-made patterns: sequential (step by step), concurrent (in parallel), group chat (agents discussing), handoff (passing control to a specialist, covered in the Handoffs article), and a manager pattern called Magentic, where a lead agent plans and assigns work. It described workflows that can "pause, resume, and recover from interruptions", with retry and error-handling logic. It let builders mark tools as needing human approval, so the system stops and asks before using them. And it recorded every agent action and tool call using OpenTelemetry, an open standard for logging what software does.
The most useful line for a non-technical reader is in Microsoft's own documentation for the framework. Its table on when to use an agent and when to use a workflow ends with a plain piece of advice: "If you can write a function to handle the task, do that instead of using an AI agent." In other words, even the company selling an agent framework tells you to reach for the simplest option first.
From Swarm to the Agents SDK: OpenAI's two ways to orchestrate (October 2024 to March 2025)
In October 2024 OpenAI published Swarm on GitHub, a small, experimental framework for coordinating several agents that pass work between each other. It was openly positioned as a teaching example, not a product. An OpenAI team member said publicly that it wasn't an official product and wasn't meant for production, and the repository describes it as an "educational framework". On 11 March 2025 OpenAI released the Agents SDK, which its announcement said was inspired by Swarm, and the Swarm page now tells readers it has been replaced by the SDK.
What makes the Agents SDK documentation worth reading is that it names the two approaches this article describes. "Orchestrating via LLM" lets the model plan and decide which tools and agents to use. "Orchestrating via code" uses ordinary programming to decide the flow. The documentation lists the typical code patterns: using a model to classify a task and then routing it, chaining one agent's output into the next, running a checker in a loop until the work meets the bar, and running independent agents in parallel. It suggests mixing the two as the task demands.
Durable execution: borrowing reliability from older software (July 2025)
Long before language models, companies that run payments, bookings and deliveries needed processes that survive crashes. Tools for "durable execution" record each completed step, so that if a machine fails halfway through, the process resumes from the last completed step rather than starting again. Temporal is one well-known example. On 30 July 2025 Temporal announced an integration with OpenAI's Agents SDK, developed with OpenAI, so that agents built with the SDK could run inside Temporal's durable workflows. The announcement put it bluntly: "AI agents are distributed systems". In plain terms, they have many moving parts that can fail independently, so they need the same crash protection as any other serious business process.
Anthropic's June 2025 account of its multi-agent research system, described in the Subagents article, made the same point from the inside. Because agents keep state across many steps, the team wrote, minor system failures can be catastrophic, so they built the system to resume from where an agent was when an error occurred instead of restarting from scratch. They also found that telling the agent when a tool was failing, and letting it adapt, worked surprisingly well. Code handled the recovery; the model handled the judgement.
An everyday scenario: the podcast that ran itself, almost
This is an illustration, not a documented case. A creator records a weekly podcast and sets up an automated pipeline: when a new episode is uploaded, it's transcribed, summarised, and turned into show notes, three social posts and a newsletter section. The first version is a straight line, one step after another, with no checks. It works for four weeks. In week five the transcription service is down for an hour. The summary step receives an empty transcript, confidently invents a summary of an episode it never heard, and the social posts go out on schedule. Nothing crashed, so nothing alerted anyone.
The second version has an orchestrator with rules. If the transcript is shorter than a thousand words, stop and retry in thirty minutes, up to three times, then email the creator. The three social posts and the newsletter section are written in parallel, since they only need the summary. Nothing is published until the creator taps "approve" on a preview. And every run writes a one-line log entry per step. The pipeline is barely more complicated, but it now fails loudly and safely instead of quietly and publicly.
Try it yourself
You don't need any software beyond an ordinary chatbot such as ChatGPT, Claude or Gemini. You'll design an orchestrator on paper, then run it by hand, playing the orchestrator yourself. Doing it manually is the best way to see which decisions a real orchestrator has to make.
- Pick a creator task with several outputs. The example here is turning one podcast episode (or a video, or a long blog post) into show notes, three social posts and a newsletter section. Have the transcript or text ready. If you don't have one, use any long article you've written.
- Draw the steps on paper. Write each step in a box: "clean up transcript", "summarise", "show notes", "social posts", "newsletter section", "final check". Draw arrows for the order. Circle the steps that can run at the same time because they only need the summary (show notes, social posts and newsletter). That's your parallel section.
- Add a check after the risky steps. For each box, write one sentence: "How would I know this step worked?" For the summary it might be "mentions the episode's three main points and invents nothing". For the social posts, "each under 280 characters, no claims that aren't in the episode".
- Write your failure rules. Next to each check, write what happens if it fails: retry once with a clearer instruction, fall back to doing that part yourself, or stop the whole run. Also write a budget, for example "no more than 12 chatbot messages in total", and a stopping rule, "if the summary fails twice, stop".
- Run the first step. Open a new chat and paste: “
You are step 1 of a content pipeline. Read the transcript below and write a summary of 150 to 200 words that covers the main points in the order they were discussed. Do not add any facts, names or numbers that are not in the transcript. At the end, list the three main points as bullet points. Transcript: [paste your transcript here]“ - Check it against your rule. Compare the summary with what you know was said. If it invented something, that's a failure: apply your rule from step 4. Retry once with "You added [the invented detail], which is not in the transcript. Rewrite the summary without it." Count every message you send against your budget.
- Run the parallel steps in separate chats. Open three new chats, one for each output, and paste only the approved summary, not the whole transcript. This is what an orchestrator does: it passes each worker exactly what it needs. For example: “
You are writing social posts for a podcast. Using only the summary below, write three different posts for LinkedIn, each under 280 characters, each highlighting a different main point. Do not add facts that are not in the summary. Summary: [paste the approved summary]“ Do the same for the show notes and the newsletter section, each in its own chat with its own clear instruction. - Run the final check. In one more chat, paste all three outputs and the summary and ask: “
Check the three pieces below against the summary. List anything in the pieces that is not supported by the summary, any contradictions between the pieces, and any piece that breaks its length limit. Do not rewrite anything; only report problems.“ - Keep a log as you go. On your sheet of paper, note each step, whether it passed its check first time, how many messages it took, and roughly how long. At the end, total your messages and compare them with your budget.
- Decide what should be fixed and what needs judgement. Look at your log. Steps that always passed first time are good candidates for a fixed script or an automation tool. Steps where you had to think about what to do next are where a model or a person needs to stay in charge. You've just made the core design decision of orchestration.
Common mistakes and misunderstandings
"An orchestrator is the same as a smarter agent." Not quite. An orchestrator can be a plain script with no AI in it at all, or an agent making decisions, or a mix. The defining feature is that it coordinates other pieces. Plenty of excellent orchestrators are a few lines of ordinary code.
Reaching for an orchestrator when a simple chain would do. If your task always runs the same steps in the same order, a straight prompt chain is easier to build, cheaper to run and easier to fix. Microsoft's guidance suggests using the lowest level of complexity that reliably meets your needs, and Anthropic's guide says to add multi-step agentic systems only when simpler solutions fall short. An orchestrator starts to earn its keep when you need routing, parallel work, recovery from failures, or several different specialists.
Letting the model decide everything. Model-driven orchestration is tempting because it looks flexible, but every decision the model makes is a decision you can't fully predict or repeat. Keep the parts you already know, such as the order of steps, the approvals and the spending limit, in fixed rules, and let the model decide only where judgement is genuinely needed.
No stopping rule. A loop of "write, check, rewrite" can go on indefinitely if the checker is never satisfied, and two agents can pass work back and forth without end. Anthropic's guide recommends stopping conditions such as a maximum number of iterations, and Microsoft's patterns guide warns specifically about infinite loops, telling builders to cap iterations and decide what happens when the cap is reached. Every loop needs a maximum count, a time limit and a cost limit.
Treating a missing result as a success. The most dangerous failures are silent ones: a step returns an empty answer, a malformed answer or an error message, and the next step carries on as if it were real. Check each important output before passing it on, and make the orchestrator stop or retry when a check fails.
Retrying things that aren't safe to repeat. Retrying a summary is harmless. Retrying "send the newsletter" or "charge the customer" can mean doing it twice. Before you let an orchestrator retry a step, ask whether running it twice would cause harm, and if so put a person or a "has this already happened?" check in front of it.
Running dependent steps in parallel. Parallel steps save time only when they genuinely don't need each other's output. Cognition's "two Flappy Birds" example, in the Subagents article, shows what happens when pieces that must fit together are made separately without shared context.
No log. Without a record of what each step received and produced, a wrong final result is a mystery. Even a spreadsheet with one row per step is far better than nothing.
Forgetting that frameworks change. The tools named in this article are updated often, and their names, features, terms and prices change. Treat them as examples of the ideas, and check each one's current documentation and terms before relying on it.
Related elements
- Agent: the decision-making worker an orchestrator often coordinates, or sometimes is.
- Prompt Chaining: the simplest orchestration, a fixed line of steps, and often all you need.
- Workflow Automation: the no-code tools where most people first build a code-driven orchestrator.
- Multi-agent Systems: teams of agents that need orchestration to work together without chaos.
- Subagents: helpers an orchestrating agent launches for focused jobs.
- Handoffs: passing control from one agent to another, one of the main orchestration patterns.
- Skills: packaged know-how that an orchestrator can call on for a particular kind of task.
- Triggers: the events that start an orchestrated run, such as a new upload or a new email.
- Scheduled Tasks: runs that start at set times, where good failure handling matters most because nobody is watching.
- Planning: how a model breaks a goal into steps, the core of model-driven orchestration.
- Tool Use: the actions an orchestrator schedules, retries and limits.
- Observability: the logs and traces that show what an orchestrator actually did.
- Human in the Loop: the approval points a good orchestrator builds into its plan.
- Autonomy Levels: how much the orchestrator is allowed to decide and do without asking.
- Tokens: the unit of cost that a budget in the orchestrator keeps under control.