Simply AskingDocs

AI Operations

Review conversations, monitor escalations, and feed knowledge gaps back into your system.

Conversations reveal what your knowledge system handles well and where it has gaps. The AI Operations portal is where your team reviews conversations, handles escalations, and identifies knowledge gaps. Escalations and knowledge gaps feed directly into your system's improvement cycle.

EscalationsHandle human handoffs
ConversationsReview quality
Unanswered QuestionsImprove coverage
TeamManage operators

The Overview tab shows metrics across all your agents.

Key Metrics

Pending12: Escalations waiting for an operator to claim
Urgent3: High-priority escalations needing immediate attention
Unanswered Questions8: Topics where agents couldn't provide good answers
Active5: Escalations currently being handled by operators

Priority Queue

Visual breakdown of escalations by severity: Urgent, High, Medium, Low

When an AI agent can't handle a conversation, it creates an escalation. Your team picks these up and resolves them.

Escalation Triggers

Restriction triggeredGuardrail with Escalate action
User requestUser asked to speak to human
Low confidenceAI confidence below threshold

Escalation Workflow

1

Claim

Click "Claim" to take ownership. Status changes to "Claimed" and removes it from others' queues.

2

Review

View the full conversation history, reason for escalation, and any context from the AI.

3

Resolve

Choose a resolution type and close the escalation.

Resolution Types

AnsweredProvided the answer
Action TakenDid something for user
ReferredHanded to another team
SpamNot a real inquiry
DuplicateAlready handled
No ResponseUser didn't reply

Audit AI responses for quality. Find conversations to review and flag issues.

Conversation Outcomes

ResolvedAI successfully answered
EscalatedHanded to human
AbandonedUser left
Low ConfidenceAI uncertain

Flagging Conversations

When you spot a problem, flag the conversation with a reason: Incorrect Answer, Inappropriate Tone, Missed Context, Guardrail Failure, Hallucination, or Other.

AI Reasoning Panel

For each AI response, see behind the scenes: retrieval method and chunks considered, confidence level, guardrails checked and any that triggered, whether the response was modified by safety rules, and source documents used for the answer.

When users ask questions the AI can't answer well, the system tracks them as knowledge gaps. Resolving gaps improves your agent over time.

Gap Information

QuestionThe unanswered question
OccurrencesHow many times this was asked
TopicCategory (e.g., Billing, Technical)
Last SeenWhen this was last asked

Resolution Options

Create DocumentWrite new content addressing the gap. Adds to your knowledge base so the agent can answer next time.
Add Approved ResponseCreate a canned response for this specific question. Agent uses it automatically for similar questions.
Mark Out of ScopeThis isn't something your agent should answer. Consider adding a guardrail to deflect these questions.
Already CoveredContent exists but the agent didn't find it. May indicate a search issue or poor document tagging.

The Operations portal uses a 3-tier role system for access control.

Admin: Full AccessEverything a Manager can do, plus operator management (add/remove/change roles).
Manager: Operations + SettingsHandle escalations, review conversations, manage knowledge gaps. Can edit thresholds, templates, and assignment rules.
Operator: Escalation HandlingClaim and resolve escalations, flag conversations, view knowledge gaps. Cannot edit settings or manage team.

Operator Status

Operators can set their availability status: Online, Away, Busy, or Offline

Managers and Admins can configure operational settings in the Settings tab.

Threshold Tuner

Adjust when escalations trigger automatically:

Confidence ThresholdAuto-escalate if AI confidence below X%
Auto-Escalate RestrictionsEscalate when any guardrail triggers
Auto-Escalate User RequestEscalate when user asks for human

Response Templates

Pre-written responses operators can insert into live chats. Ensures consistent, quality messaging.

Team Assignments

Routing rules to automatically assign escalations to specific operators based on topic category (Billing → Billing Team), priority level (Urgent → Senior Operators), or trigger type (User Request → Customer Success).

Respond to urgent escalations quicklyUsers are waiting. Aim for under 5 minutes for urgent, under 1 hour for others.
Review random conversations weeklyDon't just wait for escalations. Spot-check normal conversations to catch issues early.
Resolve knowledge gaps promptlyEach unresolved gap = more failed conversations. Prioritize high-occurrence gaps.
Use assignment rules for routingDon't make operators hunt for their escalations. Route automatically by topic or skill.
Track patterns in escalationsFrequent escalations on the same topic? Add content or adjust guardrails.
Was this helpful?