As more enforcement moves to automation, one of the most important questions a Trust & Safety leader can answer is how quickly they'd know if the system started getting things wrong.
Most programs already have one of the best early-warning signals for this and don't treat it that way: the decisions users contest and win.
Broken out by policy area, enforcement source, market etc., appeal overturn data shows where guidance is unclear, where automated decisions hold up worse than human review, where language and context are getting lost, and where a model has drifted. It often shows drift weeks before anything else does.
That makes it something you can govern with. A few rules I'd put in place:
- Automation doesn't expand into a new policy area until its overturn rate is at or below human review.
- Every model or threshold change gets an overturn baseline before launch and a check (x) weeks after. If the rate moves past an agreed limit, the change rolls back.
- When a policy area stays above its overturn target, the first thing reviewed is the guidance itself, before anyone looks at reviewer performance.
This signal has limits. It only measures over-enforcement, since nobody contests the harm you missed, so it needs to sit next to prevalence sampling. And appeal rates vary by market, so a low overturn rate where few people appeal isn't automatically good news.
Automation in T&S isn't slowing down. The programs that handle it well will be the ones that decide in advance what earns automation more scope - and what takes that scope away.