There's no way to detect or get notified when a Chat AI conversation takes an undesirable turn, for example when the AI tells a visitor that Chili Piper isn't the right solution for them, or handles an edge case in a way that wasn't intended. Admins currently have to manually review conversations on a recurring basis to catch these situations.

Admins should be able to configure rules or conditions that flag specific conversation outcomes, such as disqualification language, certain AI responses, or defined keywords, and trigger a notification so the conversation can be reviewed immediately.

Chat AI can go off-script in ways that are hard to predict, and by the time a manual review catches it, the damage is done. Real-time flagging and alerting would let teams identify and respond to problematic conversations quickly, improve AI training, and reduce reliance on manual audits.