- AI Safety
- Conversational AI
- LLM
- Moderation
- Architecture
The Safety Layer Every Conversational AI Needs: Moderation Models, Incident Records and Human Escalation
A conversational AI that people trust with difficult situations needs more than a well-behaved prompt. The production safety architecture: moderation as a mandatory pipeline stage, structured incident records and a human escalation path that actually pages someone.
If you build a conversational AI that is any good, people will eventually tell it things that matter: conflict, distress, situations where a wrong response causes real harm. This is a direct consequence of building something trustworthy, and it arrives whether or not you designed for it. The only question is whether your architecture is ready on the day it happens.
A system prompt that says "be safe" is not an architecture. Any assistant that people confide in, whether it supports employees, patients or customers, needs a safety layer designed as infrastructure from the first week: a moderation model in front of every message, structured incident records, and an escalation path that ends with a human being notified, not with a log line nobody reads. This post describes that design and the reasoning behind each piece, drawn from building and operating exactly such a system.
Why is a system prompt not enough?
Three structural reasons. First, prompts steer generation; they do not classify input. Detecting that a message indicates a safety concern is a different task from responding well, and conflating the two means your detection quality silently depends on your response model's mood. Second, prompts change constantly for product reasons, and safety behaviour must not be a side effect of copy tuning. Third, and decisively: some situations should not be handled by any model alone. Self-harm signals, threats, abuse. For those, the system's job is to respond with care and get a human informed. A prompt cannot page anyone.
So safety lives outside the generation path, as a pipeline stage with its own model, its own storage and its own alerting.
Moderation as a mandatory pipeline stage
Whatever your conversation engine looks like, the pattern is the same: every inbound user message passes through a moderation check before the response model ever sees it, as the first step of the shared message-processing pipeline, ahead of context loading, memory updates and response generation.
Placing moderation in that shared pipeline rather than as middleware on one endpoint has a consequence that justifies the whole design: every entry point inherits it. Web chat, Slack messages, any future surface: they all flow through the same stage, so no new integration can accidentally skip the check. Safety coverage becomes a structural property instead of a per-endpoint convention someone must remember.
The stage itself calls a dedicated moderation model (OpenAI's omni-moderation-latest, for example) on the user's message and receives category-level classification with scores:
const response = await openai.moderations.create({
model: "omni-moderation-latest",
input: userMessage,
});
const result = response.results[0];
if (result?.flagged) {
await recordIncidentAndEscalate(conversationId, userMessage, result);
}
// continue to the next pipeline stage either wayA purpose-built classifier is cheap, fast, consistent and independently versioned. Those last two properties are what the system prompt could never offer.
What happens when a message is flagged?
Three things, in order, each with its own failure isolation.
First, a structured incident record. Flagged content is stored in teh database: the chat id, the message, and the full classification result with model name and category scores. The raw scores matter more than the boolean; they are what let you audit threshold behaviour later, answer "why was this flagged" precisely, and re-evaluate history when the moderation model versions change. An incident is a first-class domain object, not a log line.
Second, human escalation. The incident triggers an alert into a dedicated, access-controlled channel where trained people see it promptly. The alert carries what a responder needs to act (who, which categories, where to look), routed through the same messaging infrastructure as any other operational notification. The principle: detection without a human path is surveillance, not safety. Someone specific finds out, quickly, every time.
Third, the conversation continues with care. Blocking the message and going silent is the worst response to a person in distress. The assistant responds within its guardrails (a well-designed one is explicit about what it is not, whether that is therapy, HR or legal counsel, and knows how to say so and point to appropriate help), while the escalation proceeds in parallel.
Fail-safe, in which direction?
Every safety layer must answer: what happens when the safety check itself fails? If the moderation API times out, refusing to converse fails closed, which sounds rigorous until you notice it means the assistant goes mute at unpredictable moments, including mid-support. Processing without the check fails open, which sounds negligent until you notice the alternative.
Failing open on individual moderation errors, deliberately, with the error logged and monitored, is a reasonable choice for many assistants:
try {
response = await openai.moderations.create({ ... });
} catch {
errorLogger(`Moderation check failed for ${conversationId}`);
return proceedUnmoderated(); // proceed, visibly, monitored
}The reasoning is contextual, not universal: modern response models have their own conservative behaviour, conversations are between an authenticated user and their own assistant (not user-generated content broadcast to others), and a spike in moderation failures should alert operations independently. A platform where flagged content reaches other users should make the opposite choice. The point is that fail-open versus fail-closed is a decision to make once, explicitly, in design review, and never implicitly in a catch block.
Privacy: the tension you must design, not discover
A safety layer stores sensitive messages and shows them to humans, in a product whose promise is confidentiality. Pretending the tension away is how trust dies, so the resolution is explicit:
- Minimum necessary storage: the flagged message and classification, not surrounding conversation.
- Access control: incidents and the escalation channel are restricted to the people whose role is response.
- Honest user-facing language: confidentiality commitments state the safety exception, because users deserve the true rule rather than a comfortable one.
- Retention: incidents age out on a defined schedule rather than accumulating forever.
Safety and privacy stop conflicting the moment both are designed rather than one being improvised inside the other.
Frequently asked questions
Is the OpenAI moderation endpoint suitable for production?
It handles the volume of a real product without becoming a latency or reliability problem. Wrap it with timeouts, error logging and an explicit fail-mode decision, as with any external dependency.
Should moderation block the response until it completes?
Running it as the first sequential step adds tens of milliseconds ahead of an LLM call that costs seconds, and the ordering guarantees nothing reaches the response model unchecked. Running it in parallel and cancelling on flag is a defensible latency optimisation with a hairier failure story.
What about moderating the model's outputs, not just user inputs?
A reasonable second stage, and the same pipeline pattern accommodates it. Input moderation deserves priority because the human escalation path (the part that helps a person in distress) is triggered by what users say, not by what the model replies.
Who should receive escalations?
Named, trained people with a defined protocol, in a restricted channel, with coverage expectations. If the answer is "whoever happens to read the alerts channel", the escalation path does not exist yet.
How do we test a safety layer without generating real incidents?
Synthetic flagged inputs in staging exercise the full path (classification, incident row, alert delivery) on a schedule, so the first real incident is not also the first integration test.
Key takeaways
- Safety is a pipeline stage with its own classifier, storage and alerting, never a paragraph in the system prompt.
- Put moderation in the shared conversation pipeline so every current and future surface inherits it structurally.
- Store incidents as structured domain objects with full classification scores; they are your audit trail and your tuning data.
- Escalation must end at a notified human with a protocol. Detection without a human path is not a safety feature.
- Choose your failure direction explicitly and pair it with monitoring; resolve the safety-privacy tension in design, in writing, in the product's own language.
The measure of a safety layer is unglamorous: on the day a real person is in real difficulty, the system responds with care and the right human finds out in minutes.
