Moderator wellbeing

1. Step 01

   Treat the automation line as a wellbeing decision

   Where the line between automated and human decisions sits also sets how much harmful material your reviewers see. Designing it well includes limiting that exposure through rotation and support. Two effects pull in opposite directions.

   Automation can take people away from the material. Hash matching finds copies of known images without anyone having to study them fresh. Classifiers clear the obvious cases. Summaries, metadata and transcripts let a reviewer understand a case before opening the worst of it, or decide it without opening it at all.

   Automation also concentrates what's left. Where a wrong decision can't be reversed or someone's safety is at risk, a person owns the call. That means child safety, credible threats, exploitation and self-harm stay with people, and those are the most disturbing queues. As automation takes the clear-cut cases, each hour of human review can carry more severe content, not less.

   So measure exposure hours before and after every automation change, next to accuracy and cost. And design the case view so automation prepares the case: context first, the content itself last, and only as much of it as the decision needs. Chapter 7 covers where the line sits, and chapter 18 covers automation in depth.

2. Step 02

   Map everyone who's exposed

   It's easy to protect the front-line review team and forget everyone else. List every role that sees harmful material, in-house and at vendors:

   • Front-line reviewers, and the specialists and investigators who take their escalations
   • Quality reviewers, calibrators and appeals reviewers, who see the same cases twice
   • The on-call team and anyone who handles law enforcement requests
   • Policy writers who build example sets, and trainers who show them
   • Engineers and data scientists who inspect training data or classifier errors
   • Data labelers, AI output raters, red-teamers and evaluation graders
   • Support agents who read user reports before they're routed

   Then tag each queue with an exposure level, for example high, medium or low. Graphic violence, child sexual exploitation and self-harm are high. Voice review can be as hard as graphic video, so count it. The tag decides which limits, tools and support apply.

3. Step 03

   Limit exposure, and measure it per person

   You can't manage exposure you don't measure. Track graphic-exposure hours per reviewer: hours each person spends on high-exposure queues each week.

   • Log it from the case tool, not from self-reporting, so the number doesn't depend on anyone remembering, or wanting to admit, how long they spent.
   • Set daily and weekly caps with a clinical partner. There's no agreed, evidence-based number for how much is too much. The Technology Coalition's guidebook says limiting time exposed to child sexual abuse material is key, without naming a figure (Employee Resilience Guidebook). Start with your clinician's advice, then adjust on what you see in attrition, survey results and support use.
   • Decide what happens at the cap. The person moves to lower-exposure work, with no loss of pay or standing. If hitting the cap costs people anything, they'll hide their hours.
   • Alert leads before someone reaches the cap, not after.
   • Report the spread and the number of people over the cap. Averages hide individuals. A team can average well under the cap while the few people trained for the child-safety queue sit far above it.
   • Cross-train so the caps can hold. A cap nobody can cover for is a cap that gets broken in the first busy week.

   Look at productivity targets too. Don't run high-exposure queues on the same throughput targets as spam: a target built for volume pushes people through the worst material faster. Moderators' unions go further. The Global Trade Union Alliance of Content Moderators, formed in 2025, calls for eliminating all quotas for egregious content (UNI Global Union).

4. Step 04

   Make the tools reduce exposure by default

   Small interface choices change how much of each item a reviewer has to take in. Turn them on by default, and let reviewers turn detail up when a case needs it, rather than down after they've seen too much.

   Default	What it does
   Grayscale	Shows images and video without color
   Blur that lifts on hover	The reviewer reveals only the part they need, only for as long as they need it
   Muted audio	Sound plays only when the reviewer chooses, with a transcript where you have one
   Previews	Thumbnails or keyframes before the full video, and no autoplay
   A warning before opening	The suspected category is shown first, so nothing arrives as a surprise
   Hash matching	Known material is matched by its digital fingerprint, so it doesn't need fresh eyes to be identified
   Context first	Account history, reports and metadata appear before the content itself

   The research supports the first two, with a caveat. In a live review setting, simple grayscale transformations significantly changed the emotional impact of reviews without a significant drop in accuracy, while a full blur was challenging for reviewers (Karunakaran and Ramakrishan, 2019). In experiments with crowd workers, interactive blurring reduced emotional impact without sacrificing accuracy or speed (Das, Dang and Lease, 2020). So blur that reviewers control, not blur that gets in their way.

   You don't have to build these from scratch. ROOST's open-source review console, Coop, blurs images and video by default with hover to reveal, offers grayscale and muted video, lets admins set organization-wide wellness defaults, and matches uploads against NCMEC's hashes of known child sexual abuse material. For hash matching, Microsoft's PhotoDNA and Thorn's Safer are established options.

5. Step 05

   Give support people actually use

   A general employee assistance program, the counseling benefit offered to all staff, is built for everyday problems, not the specific effects of reviewing harmful content. People in these roles need more.

   • Before the work. Tell every candidate and new reviewer what they'll see and what support exists. Get informed consent before assigning anyone to child sexual abuse material, as the Technology Coalition's guidebook recommends, and do the same for other high-exposure queues.
   • Specialist counseling. A licensed clinician who understands this work, available during working hours. The guidebook advises that the care provider be outside the company, which helps with confidentiality.
   • A confidential way to step away. Clear opt-out rules, so someone can leave a queue for a while without explaining why to the whole team.
   • Rotation and real breaks. Rotation off high-exposure queues, either on a fixed schedule or triggered by the cap, whichever your staffing can hold, and wellbeing time counted as planned time away from the queue, so it isn't taken out of anyone's targets (chapter 7).
   • Debriefs after severe cases. After a self-harm emergency or a child safety case, offer a debrief and counseling, give time away from high-severity queues, and check exposure across the team. The tabletop scenario The post six friends saw rehearses that call.
   • Colleagues and meaning. In Spence's survey, supportive colleagues and feedback about the importance of the role softened the link between exposure and distress. Build time for peers to talk, and tell reviewers what their decisions led to: a child safeguarded, a network removed, a report that reached the right people.
   • Trained managers who model it. Leads should know the signs of secondary trauma and what to do next, and should take their own breaks and talk about it. When leads do, everyone else feels able to. UNI's protocols ask for trauma-informed training for supervisors as well as moderators.
   • After they leave. Support shouldn't end on someone's last day. The maturity model treats support after people leave review roles as the mark of a leading program.
6. Step 06

   Treat data labeling and AI training review as moderation work

   Data labeling is moderation work. It needs the same wellbeing support and escalation paths.

   The people who label training data for safety classifiers, rate model outputs, red-team AI products and grade evaluations see the same material your reviewers do. Many are contractors working through a platform, outside the Trust & Safety team, and may have no exposure limits, no blurring in the labeling tool and nobody to call.

   • Bring them into the program. Same exposure levels, caps, tools and counseling as your reviewers.
   • Give them an escalation path. A labeler who finds what looks like child sexual abuse material, or a real person at risk, needs a route to your restricted escalation path, not just a "skip" button. Chapter 12 covers what happens next.
   • Design datasets to need less exposure. Filter known material out by hash before anyone labels it, and label from text or metadata where that's enough.
   • Put it in the contract. Labeling vendors get the same wellbeing clauses as review vendors. Partnership on AI's Responsible Sourcing of Data Enrichment Services sets out what AI developers should consider when buying this work.
7. Step 07

   Write wellbeing standards into vendor contracts, and check them

   If you use a vendor, its staff may make most of your review decisions and see much of the worst content. A contract that says "wellness program" without numbers or a way to check is a promise, not a standard. Write in:

   • A maximum daily exposure to graphic content for each person, with logs you can see
   • Licensed counseling during employment and after someone leaves
   • Blurring and grayscale on by default in every review tool
   • Annual attrition for your account, split by queue
   • Support use and survey results, aggregated and anonymous
   • Your right to audit and visit
   • A price that pays for wellbeing time, not just decisions

   Choose vendors with this as a floor. In the vendor scorecard, reviewer wellness carries 15% of the weight and has a minimum: a score of 2 or lower rules a vendor out, whatever its total. Hard daily exposure caps per person, counseling during and after employment and blurring by default earn a 5.

   Then check it. Ask for exposure data by person, pseudonymized, not a summary slide. Compare attrition by queue against your own teams. Visit, and talk to reviewers without their managers in the room. When something goes wrong at the vendor, how you treat their people shapes the quality you get afterwards: if a site goes dark, keep paying and ask how you can help. Chapter 9 covers contracts in full.

8. Step 08

   Watch attrition and support use as early warnings

   Watch the people, not just the quality scores. Track attrition and wellness-support usage on a regular cadence, monthly for example, for in-house and vendor teams:

   • Attrition by exposure level. Monthly leavers divided by average headcount, times 12. If high-exposure queues lose people much faster than the rest, fix the work design, not the hiring.
   • Support sessions per person. From your provider, aggregated and anonymous only. Never individual records.
   • A regular anonymous pulse survey. Quarterly, for example. It explains what the numbers can't.
   • Other signals. Rising sick leave, quality dropping on one team, handle time climbing on severe queues, and people asking to move.

   Read low support use carefully. It can mean people are fine. It can also mean they doubt it's confidential, don't want to be seen using it, or can't fit it into a shift. Ask anonymously which.

   Make one senior leader accountable for reviewer wellbeing, and put these numbers in your regular executive review next to the operational ones. If nobody senior owns it, it's easy to cut in a tight quarter.

1. 01

   Relying on a general employee assistance program

   Provide specialist counseling from clinicians who understand this work.

2. 02

   Automating the easy cases and forgetting what's left

   Measure exposure after every automation change, because the human queue gets harder.

3. 03

   Reporting average exposure

   Report the spread and the number of people over the cap.

4. 04

   Caps that cost people pay or standing

   Move people to lower-exposure work at the cap, with nothing lost.

5. 05

   Training only one or two people on the worst queue

   Cross-train so caps can hold.

6. 06

   Forgetting the people outside review

   Labelers, raters, red-teamers, policy writers and engineers are exposed too.

7. 07

   Contracts that say "wellness" with no numbers

   Write in caps, counseling, tooling and audit rights, then check them.

8. 08

   Reading low counseling use as good health

   Ask anonymously why people do or don't use it.

9. 09

   Support that ends on someone's last day

   Continue it after people leave review roles.

From Steven's writing · 2 posts

1. Sep 29, 2026Reading roundup

   The era of voluntary child safety is ending

   India's push for age checks, Florida's case against OpenAI, Copilot data labeling, TikTok's Alabama settlement and Meta's New Mexico verdict. The question has moved from "do you have a policy?" to "can you prove it works?"

   Also in: 15. Working with Product, Legal, Comms and leadership, 16. Regulation and compliance, 18. AI in Trust & Safety

2. Sep 21, 2026Essay

   Automation rate isn't a measure of maturity

   Automate as much as the evidence supports. Where a wrong decision can't be reversed or someone's safety is at risk, automation prepares the case and a person closes it.

   Also in: 5. Detection and prevention, 7. Standing up review operations, 18. AI in Trust & Safety

Outside sources

• Technology Coalition: Employee Resilience Guidebookindustry guidance on protecting staff who handle child sexual abuse images, from hiring and consent to counseling and opting out.
• Steiger et al., The Psychological Well-Being of Content Moderatorsa CHI 2021 review of the research on moderators' mental health and the ways to support them.
• Spence et al., Content Moderator Mental Health, Secondary Trauma, and Well-beinga 2024 survey linking how often moderators are exposed to distress and secondary trauma.
• Das, Dang and Lease, Interactive Blurring Helps Moderators Reduce Exposure to Harmful Contentexperiments on blur designs that protect reviewers without slowing them down.
• UNI Global Union: Mental Health Protocols for content moderationthe eight protocols moderators' unions are asking the tech supply chain to adopt.
• Partnership on AI: Responsible Sourcing of Data Enrichment Servicesrecommendations for AI developers on the working conditions of the people who label their data.

1. Oct 4, 2026
   RevisedWith 18 other chapters

   Practical advice that varies by platform, such as cadences, sample sizes, targets and who owns what, is now set out as options with examples, so each team can choose what fits.

2. Oct 4, 2026
   RevisedWith 14 other chapters

   Added Steven's own calls from an interview: where Trust & Safety should report, what to automate first, the one number to track from day one, who makes the 2am call, and more. Practical choices that vary by platform are now laid out as options.

3. Oct 3, 2026
   DraftedWith 17 other chapters

   First full drafts of the other 18 chapters, built on Steven's posts, the handbook's principles and the Workbench's open content, with every legal and factual claim checked against its source. Stories from Steven's own work come next.

4. Oct 3, 2026
   AddedWith 14 other chapters

   Linked the first 16 posts to the chapters they inform, and set out the ten principles behind the handbook.