Automation rate is a useful efficiency metric, but it isn't a measure of maturity. It counts the decisions humans didn't have to make, not whether those decisions were correct.
The goal I work toward is automating as much decision-making as the evidence supports, and reserving human judgment for cases where context, severity, or the cost of error requires it.
A classifier alone can not yet make that determination. The system also has to weigh confidence, severity, user history, behavioral signals, prior enforcement, reputation, age-related risk, and the consequences of a wrong decision.
Those inputs determine the response: automated enforcement where the evidence is clear; friction, deprioritization, or distribution limits where it is less certain; additional signals where a decision isn't yet supportable; and human review where the cost of an error is too high.
Consider what a single high-confidence score can describe: a credible threat, hyperbole between acquaintances, or a survivor recounting their own experience. The score is the same. The appropriate action is not. That is why decisions in child safety, credible threats, exploitation, and self-harm can't yet rest on model confidence alone.
The programs I've seen work best didn't measure success by how much review they eliminated. They looked at decision accuracy, cost per decision, and appeal overturn rate, along with whether reviewer time was concentrated where judgment mattered most.
Where the line between automated and human decisions sits also determines how much harmful material reviewers are exposed to. Designing it well includes limiting that exposure through rotation and support.
AI lets Trust & Safety teams operate at a scale manual review never could. Deciding where automated action ends and human judgment begins is the harder design problem.
My default: where a wrong decision can't be reversed, or a user's safety is at risk, a human owns the call. In those cases, automation prepares the case and a person closes it.
#TrustAndSafety #ContentModeration #ResponsibleAi