Park Whisperer
Azure OpenAI Assistants API

The most compelling property of the Assistants API is that the conversation history lives server-side in Thread objects. The client only needs to store a thread_id — not reconstruct or send the full message history on every turn. For a multi-turn park planning conversation that accumulates dozens of messages and multiple tool call results, this matters: passing the full history in the messages array on every call adds tokens and latency proportional to conversation length.

In practice, the Thread-based model works well for long-running planning sessions where a guest asks follow-up questions across many turns. The server-side state also means if the client drops the connection mid-turn, the Run can continue and the result is retrievable later — a resilience property that the stateless chat completions model doesn't have.

The Assistants API Run lifecycle requires polling: create a Run, poll until requires_action or completed, execute tools, submit outputs, poll again. Each poll is an HTTP round-trip. A conversation turn requiring three tool calls involves: create run → poll → submit three tool outputs → poll → retrieve messages. This polling loop adds latency and complexity compared to streaming chat completions where the tool call response arrives in the stream and the agentic loop runs in the same function execution.

The production system uses streaming chat completions inside an Azure Function, which allows token-by-token streaming to the client — users see the response building in real time. The Assistants API (at beta/2024-05-01-preview) did not support streaming for runs with tool calls, which made the UX feel slower even when total latency was similar.

get_park_shows and get_release_events were present in this v4 Assistants version that weren't in early production iterations. Show schedules are straightforward — the agent can tell a guest when Fantasmic plays tonight. The release events tool tracks when Lightning Lane return times become available throughout the day, which is a more advanced use case: guests asking "when should I try to book TRON Lightning Lane?" can get data-driven timing recommendations rather than generic advice.

The 16th tool, get_park_entities, provides the reference entity data (IDs, names, types) that other tools use for filtering. In practice, the agent rarely needs to call this explicitly — it uses entity IDs from the system context — but having it as a callable tool means the agent can resolve an ambiguous attraction name to an ID before calling other tools. This resolved a class of "attraction not found" errors when guests used informal names.

Separating the agent configuration into version-controlled JSON files (agent-config.json, agent-functions.json) and deploying via script is the most durable practice from this version. It meant that changes to the system prompt or tool definitions were tracked in git, reviewable as diffs, and reproducible — rather than being made directly in the Azure portal where changes leave no audit trail.

The agent-assistant-config.json snapshot pattern (reading back the deployed state after update) is equivalent to a Terraform state file: it records what was actually deployed, allowing drift detection between what's in agent-config.json and what the Azure OpenAI service has. This pattern carried forward into the production deployment pipeline, where function app settings are read back and validated after each deploy.

Three concrete reasons drove the migration to the stateless chat completions approach for production:

• Streaming — chat completions support token streaming with tool use; the beta Assistants API did not. Users expect streaming in modern chat interfaces.
• Control — the agentic tool-call loop running inside an Azure Function is fully observable and debuggable. The Assistants API's Run lifecycle is opaque: if a Run fails or stalls, the error is often a generic timeout rather than a specific exception from the tool execution code.
• Thread management overhead — server-side Thread objects accumulate and require cleanup. For a high-traffic public product, managing Thread lifecycle (creation, expiry, deletion) adds operational burden that the stateless model avoids entirely. Each conversation in the production system is self-contained in the client-sent messages array.

The trade-off: the production system must send the full conversation history on every turn, which uses more tokens for long sessions. But the streaming UX improvement and reduced operational complexity were judged worth that cost.