← All posts

Designing Reliable Function-Calling Schemas for Production in Microsoft Foundry

6 min read4.4k
#LLMOps#Azure#Reliability#Microsoft#Generative AI#Foundry#LLM#AI Architecture

Function-calling demos work because the model is well-behaved and the test queries are clean. Production breaks this in three specific ways: malformed arguments, calls to the wrong tool when two tools are semantically similar, and side effects that need to be idempotent when a call gets retried. This post covers schema design and handling code for all three.

Tool call lifecycle

Schema design: be more restrictive than feels necessary

A common mistake is under-constraining the schema because "the model is smart enough to figure it out." In practice, tighter schemas measurably reduce malformed calls:

python
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Retrieve the current status of a customer order. Only call this when you have a specific order ID — do not guess or fabricate one.",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {
                        "type": "string",
                        "pattern": "^ORD-[0-9]{8}$",
                        "description": "Order ID in the format ORD-XXXXXXXX. Must come from the conversation or a prior tool result, never inferred."
                    }
                },
                "required": ["order_id"],
                "additionalProperties": False
            }
        }
    }
]

Three things doing real work here: the regex pattern, additionalProperties: False (rejects extra fields the model might hallucinate), and an explicit instruction in description not to fabricate the ID. None of these are optional in production — each one closes a specific failure mode we've seen in real logs.

Handling malformed calls without crashing the conversation

Even with a tight schema, models occasionally emit arguments that don't validate — wrong format, missing required field, or (less often) a call to a tool name that doesn't exist. Validate before executing, and feed the failure back to the model rather than crashing:

python
import json
import jsonschema

def execute_tool_call(tool_call, tool_registry):
    tool_name = tool_call.function.name
    if tool_name not in tool_registry:
        return {"role": "tool", "tool_call_id": tool_call.id,
                "content": json.dumps({"error": f"Unknown tool: {tool_name}"})}

    try:
        args = json.loads(tool_call.function.arguments)
    except json.JSONDecodeError:
        return {"role": "tool", "tool_call_id": tool_call.id,
                "content": json.dumps({"error": "Arguments were not valid JSON"})}

    schema = tool_registry[tool_name]["schema"]
    try:
        jsonschema.validate(args, schema)
    except jsonschema.ValidationError as e:
        return {"role": "tool", "tool_call_id": tool_call.id,
                "content": json.dumps({"error": f"Validation failed: {e.message}"})}

    result = tool_registry[tool_name]["fn"](**args)
    return {"role": "tool", "tool_call_id": tool_call.id, "content": json.dumps(result)}

Feeding the validation error back as a tool result (rather than raising an exception up the stack) lets the model self-correct on the next turn — this recovers a surprising fraction of malformed calls without any human intervention.

Disambiguating similar tools

When two tools are semantically close (get_order_status vs. get_shipment_status), models mis-select more often than you'd expect, especially under longer conversation history. Two mitigations that actually move the needle:

  1. Make descriptions mutually exclusive, not just individually clear — explicitly state what each tool is not for: "get_order_status: use for payment/processing status. Do NOT use for shipping/delivery tracking — use get_shipment_status instead."
  2. Reduce the active tool set per turn where possible. If you can infer from conversation state that shipping tools are irrelevant (order hasn't shipped yet), don't include them in that turn's tool list at all. Fewer candidates, fewer mis-selections.
Failure modeMitigationWhy it works
Malformed JSON argumentsValidate before execution, return error as tool resultModel self-corrects on next turn instead of crashing the flow
Wrong tool selected among similar optionsMutually exclusive descriptions + narrower active tool setFewer candidates, clearer boundaries reduce mis-selection
Duplicate side effects on retryIdempotency key per logical callRetried calls return cached result instead of re-executing

Streaming responses and parallel tool calls complicate the happy path

Everything above assumes a single tool call resolved before the model continues. Production systems usually deal with two additional wrinkles: streaming responses (where you want to start executing a tool call before the full response finishes streaming) and parallel tool calls (where the model requests multiple tools in one turn, which recent model versions do fairly often to save round-trips).

python
async def handle_streaming_tool_calls(stream, tool_registry, idempotency_store):
    accumulated_calls = {}

    async for chunk in stream:
        if chunk.tool_call_delta:
            call_id = chunk.tool_call_delta.id
            if call_id not in accumulated_calls:
                accumulated_calls[call_id] = {"name": "", "arguments": ""}
            accumulated_calls[call_id]["name"] += chunk.tool_call_delta.function.name or ""
            accumulated_calls[call_id]["arguments"] += chunk.tool_call_delta.function.arguments or ""

        if chunk.finish_reason == "tool_calls":
            # Execute all accumulated calls — potentially in parallel if they're independent
            results = await asyncio.gather(*[
                execute_tool_call_async(call_id, call_data, tool_registry, idempotency_store)
                for call_id, call_data in accumulated_calls.items()
            ])
            return results

The subtlety worth calling out: parallel tool calls are only safe to execute concurrently if they're actually independent. If one tool call reads state that another tool call in the same batch writes, executing them concurrently introduces a race condition identical in spirit to the parallel-node ordering bug from Part 2. Before parallelizing, explicitly classify your tools as read-only or side-effecting, and only run side-effecting calls concurrently if you can prove they don't share mutable state — otherwise, serialize them even at some latency cost. This is a case where the "obvious" performance optimization (parallelize everything) is the wrong default; correctness has to be verified per tool pair, not assumed.

Idempotency: the part everyone forgets until it bites them

If a tool call has a side effect (charging a card, sending an email, updating a record) and the call gets retried — due to a timeout, a retry-with-backoff policy, or the model itself re-issuing the call after a partial failure — you need idempotency, or you'll double-charge or double-send.

python
def execute_with_idempotency(tool_call, tool_registry, idempotency_store):
    idempotency_key = tool_call.id  # stable per logical call

    cached = idempotency_store.get(idempotency_key)
    if cached is not None:
        return cached  # already executed; return prior result, don't re-run

    result = execute_tool_call(tool_call, tool_registry)
    idempotency_store.set(idempotency_key, result, ttl_seconds=3600)
    return result

For genuinely external side effects (payment APIs, email providers), pass this same idempotency key through to the downstream API if it supports one — most payment processors do — so the protection holds even across process restarts, not just within a single conversation's memory.

Testing this properly

Build a fuzz-test harness that deliberately sends malformed tool call arguments (wrong types, missing fields, out-of-pattern strings) through your validation path, and confirm every one produces a graceful tool-result error rather than an unhandled exception. This is the test suite that catches the incident before production traffic does.

What's next

With reliable single-tool-calling in place, Part 6 tackles the architectural decision that determines how far you can scale this: Foundry Agent Service vs. Copilot Studio, and when you actually need both.

References