AI Operations

Review conversations, monitor escalations, and manage your operator team.

Conversations reveal what your knowledge system handles well and where it has gaps. Operations Overview is where your team reviews conversation quality, tracks customer gaps, and manages operators. Handling escalations themselves happens on the Escalations tab in Operations Overview, a dedicated three-pane view for claiming, replying to, and resolving conversations. Escalations feed directly into your system's improvement cycle, and an Overview stat card here links straight to customer-facing gaps on the Customer Gaps tab of Requests.

ConversationsReview quality
Customer GapsDeep-link to Customer Gaps on Requests
TeamManage operators

The Overview tab shows metrics across all your agents.

Key Metrics

Pending12: Escalations waiting for an operator to claim
Urgent3: High-priority escalations needing immediate attention
Customer Gaps8: Topics where agents couldn't provide good answers. Click through to Customer Gaps on Requests to act on them.
Active5: Escalations currently being handled by operators

Priority Queue

Visual breakdown of escalations by severity: Urgent, High, Medium, Low

Audit AI responses for quality. Find conversations to review and flag issues.

Conversation Outcomes

ResolvedAI successfully answered
EscalatedHanded to human
AbandonedUser left
Low ConfidenceAI uncertain

Flagging Conversations

When you spot a problem, flag the conversation with a reason: Incorrect Answer, Inappropriate Tone, Missed Context, Guardrail Failure, Hallucination, or Other.

AI Reasoning Panel

For each AI response, see behind the scenes: retrieval method and sections considered, confidence level, guardrails checked and any that triggered, whether the response was modified by safety rules, and source documents used for the answer.

When customers ask your public AI agents questions they can't answer well, the system tracks them as customer gaps. Triaging these gaps, including search, filtering by status/topic/occurrence, sorting, and the gap detail panel with assign, dismiss, and resolve actions, now happens on the Customer Gaps tab of Requests rather than in Agent Operations.

The Overview tab here keeps a Customer Gaps summary stat card that deep-links to the Customer Gaps tab on Requests, so you can jump straight from a spike in gaps to resolving them.

Operations Overview uses a 3-tier role system for access control.

Admin: Full AccessEverything a Manager can do, plus operator management (add/remove/change roles).
Manager: Operations + SettingsHandle escalations and review conversations. Can edit thresholds, templates, and assignment rules.
Operator: Escalation HandlingClaim and resolve escalations, flag conversations. Cannot edit settings or manage team.

Operator Status

Operators can set their availability status: Online, Away, Busy, or Offline

Managers and Admins can configure operational settings in the Settings tab.

Threshold Tuner

Adjust when escalations trigger automatically:

Confidence ThresholdAuto-escalate if AI confidence below X%
Auto-Escalate RestrictionsEscalate when any guardrail triggers
Auto-Escalate User RequestEscalate when user asks for human

Response Templates

Pre-written responses operators can insert into live chats. Ensures consistent, quality messaging.

Team Assignments

Routing rules to automatically assign escalations to specific operators based on topic category (Billing → Billing Team), priority level (Urgent → Senior Operators), or trigger type (User Request → Customer Success).

Respond to urgent escalations quicklyUsers are waiting. Aim for under 5 minutes for urgent, under 1 hour for others.
Review random conversations weeklyDon't just wait for escalations. Spot-check normal conversations to catch issues early.
Resolve knowledge gaps promptlyEach unresolved gap = more failed conversations. Prioritize high-occurrence gaps.
Use assignment rules for routingDon't make operators hunt for their escalations. Route automatically by topic or skill.
Track patterns in escalationsFrequent escalations on the same topic? Add content or adjust guardrails.
Was this helpful?