← All posts

Managing State and Recovery in Multi-Agent Orchestration with Microsoft Foundry

7 min read4.6k
#AgenticAI#Orchestration#Foundry#Microsoft#DistributedSystems#EnterpriseAI#Azure

Once you've decided a workload belongs in Foundry Agent Service (Part 6), the next problem is orchestration mechanics: multiple agents that need to share state, hand off work to each other, and recover when one of them fails partway through a task. This is where most multi-agent prototypes fall apart moving to production — they work when every agent succeeds, and nobody's tested what happens when one doesn't.

Orchestration flow with shared state

Shared state: don't pass full conversation history between agents

The naive approach — passing the entire conversation transcript to every agent in the chain — works for a demo and falls over at scale: token costs balloon, and agents downstream start reacting to context meant for a different agent's task. Instead, define an explicit shared state object that each agent reads from and writes to:

python
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class SupportTaskState:
    task_id: str
    customer_query: str
    order_id: Optional[str] = None
    fraud_check_result: Optional[dict] = None
    resolution: Optional[str] = None
    escalated_to_human: bool = False
    completed_steps: list = field(default_factory=list)

def run_agent_step(agent_name, state: SupportTaskState, agent_fn):
    result = agent_fn(state)
    state.completed_steps.append(agent_name)
    return result

Each agent gets exactly the fields relevant to its task, not the raw conversation — this keeps prompts smaller and prevents an agent from being confused by context that belongs to a different stage of the workflow.

Handoff pattern: explicit, not implicit

Don't let agents decide amongst themselves who goes next based on free-text reasoning alone — make the handoff a structured decision with a small, closed set of valid targets:

python
handoff_schema = {
    "type": "object",
    "properties": {
        "next_agent": {
            "type": "string",
            "enum": ["fraud_check_agent", "refund_agent", "human_handoff", "complete"]
        },
        "reason": {"type": "string"}
    },
    "required": ["next_agent", "reason"]
}

def orchestrate(state: SupportTaskState):
    while state.resolution is None and not state.escalated_to_human:
        current_agent = determine_current_agent(state)
        handoff = current_agent.decide_next_step(state)  # validated against handoff_schema
        if handoff["next_agent"] == "human_handoff":
            state.escalated_to_human = True
        elif handoff["next_agent"] == "complete":
            state.resolution = state.resolution or "resolved"
        else:
            state = advance_to(handoff["next_agent"], state)
    return state

Constraining handoffs to an enum, validated the same way as the tool-call schemas from Part 5, prevents an agent from routing to a nonexistent or unintended target — a surprisingly common failure when handoff decisions are left as free text.

Recovery: what happens when an agent fails mid-task

This is the part demos never test. If fraud_check_agent times out or throws, what's the right behavior — retry, escalate, or roll back? The answer depends on whether the failed step had side effects.

python
def run_agent_step_with_recovery(agent_name, state: SupportTaskState, agent_fn, has_side_effects=False):
    try:
        return agent_fn(state)
    except AgentTimeoutError:
        if has_side_effects:
            # don't blindly retry something that may have partially executed —
            # check whether the side effect actually completed before deciding
            if not verify_side_effect_completed(agent_name, state):
                return retry_agent_step(agent_name, state, agent_fn)
            else:
                state.completed_steps.append(agent_name)
                return state
        else:
            return retry_agent_step(agent_name, state, agent_fn, max_retries=2)
    except AgentFailure:
        state.escalated_to_human = True
        log_failure_for_human_review(agent_name, state)
        return state

The key distinction: agents with side effects (refund processing, sending communications) need a verification step before retry — don't just retry blindly and risk a duplicate action, the same idempotency concern from Part 5 but now at the agent-orchestration level rather than the single-tool-call level.

Timeout budgets across a chain of agents

A subtlety that only shows up once you have four or five agents chained together: if each agent has its own reasonable-looking individual timeout (say, 10 seconds each), a request that touches all of them in sequence can take 40-50 seconds end-to-end in the worst case — well past what a synchronous customer-facing request should tolerate, even though no single agent's timeout looks unreasonable in isolation.

python
class TimeoutBudget:
    def __init__(self, total_budget_seconds):
        self.remaining = total_budget_seconds
        self.start = time.monotonic()

    def remaining_for_step(self, reserve_for_later_steps=0):
        elapsed = time.monotonic() - self.start
        return max(0, self.remaining - elapsed - reserve_for_later_steps)

def run_chain_with_budget(agents, state, total_budget_seconds=15):
    budget = TimeoutBudget(total_budget_seconds)
    for i, agent in enumerate(agents):
        remaining_steps = len(agents) - i - 1
        reserve = remaining_steps * MIN_TIME_PER_AGENT
        step_timeout = budget.remaining_for_step(reserve_for_later_steps=reserve)
        if step_timeout <= 0:
            state.escalated_to_human = True
            log_budget_exhaustion(agent.name, state)
            break
        state = run_agent_step_with_timeout(agent, state, timeout=step_timeout)
    return state

Think in terms of a total request budget divided across the chain, not per-agent timeouts chosen independently — and build in an explicit fallback (escalate to human, or return a partial result with a "still processing" status) for when the budget runs out partway through, rather than letting the last agent in the chain simply time out and produce a confusing partial failure.

Circuit breakers for consistently failing agents

If a specific agent (say, the fraud-check agent, because its upstream dependency is degraded) fails repeatedly across many requests, retrying it on every single request wastes the timeout budget above and delays every user hitting that path. A circuit breaker — stop calling a consistently failing dependency for a cooldown period, and fail fast instead — is standard practice for service dependencies, and it applies just as directly to agent-to-agent calls:

python
class AgentCircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown_seconds=30):
        self.failures = 0
        self.threshold = failure_threshold
        self.cooldown = cooldown_seconds
        self.open_until = None

    def call(self, agent_fn, state):
        if self.open_until and time.monotonic() < self.open_until:
            raise CircuitOpenError("Agent circuit open, failing fast")
        try:
            result = agent_fn(state)
            self.failures = 0
            return result
        except AgentFailure:
            self.failures += 1
            if self.failures >= self.threshold:
                self.open_until = time.monotonic() + self.cooldown
            raise

When the circuit is open, route directly to human escalation rather than attempting the failing agent — this fails fast for the user instead of making them wait through a doomed timeout, and it protects the degraded downstream dependency from additional load while it recovers.

Compensating actions: the pattern for partial failure

If agent 3 of 4 fails after agents 1 and 2 already produced side effects (e.g., a refund was approved but the confirmation email agent failed), you need a compensating action, not just a retry of step 3:

python
compensation_map = {
    "refund_agent": "reverse_refund_if_unconfirmed",
    "notification_agent": "resend_notification",
}

def handle_partial_failure(failed_agent, state: SupportTaskState):
    compensation_fn = compensation_map.get(failed_agent)
    if compensation_fn:
        execute_compensation(compensation_fn, state)
    else:
        state.escalated_to_human = True

Define compensating actions at design time for every agent that has a side effect — retrofitting this after a production incident is much more painful than designing for it up front.

Testing failure modes deliberately

Build integration tests that deliberately inject failure at each step of the chain — timeout agent 2, throw an exception in agent 3, simulate a partial side-effect completion — and assert the system reaches a safe state (either successful compensation or a clean human escalation, never a silent inconsistent state).

What's next

With orchestration and recovery handled, Part 8 turns to the knowledge layer these agents depend on: benchmarking Foundry IQ's managed retrieval against a custom pipeline to decide which is the right foundation for this system.

References