Detection and prevention

1. Step 01

   Map the detection stack against your harms

   No single layer catches everything. Each one has a job and a blind spot:

   Layer	What it catches	Where it falls short
   User reports	What users see and care about, with context only they have	Children rarely report. Report volume measures how easy reporting is as much as how much harm exists. Groups can mass-report a target.
   Rules	Known patterns: keywords, bad links, posting velocity. Fast, cheap and easy to explain.	Easy to evade with misspellings, symbols and coded language.
   Hash matching	Copies of known illegal images and video, matched by digital fingerprint	Only finds material that's already known. New material needs other layers.
   Classifiers	New content that looks like what the model was trained on, at scale	Accuracy varies by language and context, and drifts as behavior changes.
   Language models	Rules applied with context, and new policies without new training data	Slower and costlier per item. Need testing before they act (chapter 18).
   Behavior signals	Accounts, relationships and networks over time: account age, devices, who contacts whom, gifts and payments	Need data, engineering and a privacy review.

   If you host images or video, hash matching is usually the first proactive layer to add, because the lists already exist. A text-only product might start with rules and rate limits instead. Microsoft's PhotoDNA is free to qualified organizations for known child sexual abuse material. Members of GIFCT, the Global Internet Forum to Counter Terrorism, share hashes of terrorist and violent extremist content through its hash-sharing database. Chapter 9 covers choosing tools, including free and open-source ones.

   Rate each of your top harms against these layers with the coverage radar. Usually detection engineering owns the systems and policy owns what they're trying to find, though on a small team that may be one person. You're done when every high-risk harm has at least one layer that doesn't depend on a user report.

2. Step 02

   Detect the pattern, not the message

   Many serious harms don't show up in a single message. They show up as a sequence:

   • Grooming: an adult account friends many younger players it has no connection to, gifts currency early, asks about parents, moves to private voice, then pushes for another app (chapter 6).
   • A raid: dozens of new accounts post the same kind of abuse in one room within minutes.
   • A scam: a new account sends first messages to many strangers, builds trust, then asks for money, gift cards or a move off the platform.

   Each step can be innocent on its own. Together they're a pattern. Reviewing messages one at a time misses most of it.

   So decide what you're watching for each harm. Sometimes it's a single item. Often it's an account, a relationship between two accounts, or a group of accounts acting together. Keep a short signal history for each one, within a retention window you've agreed with Legal, so the third signal can be read next to the first two.

3. Step 03

   Let the response build as signals stack

   Because each signal is weak on its own, the response should get stronger as signals add up:

   Signals	Response	Who notices
   One	Log it. Nothing visible happens.	Nobody
   Two or more within a short window	Add friction: rate limits, a safety prompt, no new private channels, limits on gifts or payments	The user
   The high-risk step after that: a request to move off-platform, for images or for money, or a threat	Restrict the account or the contact, and send the full history to a trained reviewer	A reviewer

   This is the graduated approach in chapter 4 applied to detection. Act automatically where the evidence is clear and the action can be undone. Use friction and limits where it's less certain. Gather more signals where you can't support a decision yet. Send it to a person where an error would cost too much.

   When flagged cases outnumber reviewers, protective actions stay automatic and the queue is worked by risk. A restricted account waiting a day for review is a cost worth paying (chapter 7). Chapter 6 sets out this ladder in full for adult-to-minor contact.

4. Step 04

   Prevent before you remove

   The protections most likely to work act before any harmful message is sent. They're often cheaper than detection, and most users never notice them:

   • Safer defaults. Minors' accounts are private, and adults can't message minors they aren't connected to (chapter 6).
   • Rate limits. Cap posting, messaging, invites and friend requests, with stricter caps for new accounts.
   • Limits on new accounts. No links, no messages to strangers and no payouts until an account has built some trust.
   • Friction at the risky moment. A warning when someone asks for money, gift cards or a move to another app. A prompt before sending something your filters flag.
   • Proportionate responses to spikes. When a raid hits, slow down the accounts causing it rather than shutting the space for everyone (step 6).

   Put friction where the risk is, not everywhere. A limit that hits every user will be removed at the first growth review. A limit that hits new accounts messaging strangers will survive.

   These protections are easiest to add before launch, when changing a default is a quick edit to the spec. After launch, it means taking something away from users. That's why the abuse review belongs at design review (chapter 2, chapter 15). And measure what they prevent, such as contact that never happened, not just what you removed.

5. Step 05

   Set thresholds from data, and keep them current

   Every rule, classifier and signal ladder has a threshold. Set it from evidence, not instinct:

   1. Gather past confirmed cases and past false alarms for the harm.
   2. Replay the signals and see where each candidate threshold would have fired.
   3. Pick the point where precision holds and reviewers can keep pace. A threshold that sends ten times more cases than your team can review isn't protecting anyone.
   4. Write it down: the threshold, the numbers behind it, the owner and who signed it off. A threshold is policy (chapter 4).
   5. Revisit it every quarter. Offenders learn what triggers friction and route around it.

   Treat every change like a release. Take a baseline before the change, check again a few weeks after, and roll it back if precision or overturns move past a limit you agreed in advance (chapter 18).

   Measure precision on a regular cadence, for example weekly, by having experts re-check a random sample of what each system acted on. Estimate recall by labeling a sample of content the system didn't flag. Report both per policy area and per model version. A blended number hides failures in small, high-severity areas, and a rising proactive detection rate means nothing if precision fell at the same time.

6. Step 06

   Use rates as baselines, and alert on the jumps

   Volume swings with your audience. A big match, a launch or a holiday can multiply activity several times over in an evening. The rate of a category, such as abuse reports per 1,000 messages or one category's share of enforcement actions, usually holds steadier. So the rate makes a better baseline than the volume.

   When volume goes up and the rate stays put, that's a busy night. When one category's rate climbs while volume is normal, something coordinated is usually happening: a raid, or a group targeting one person.

   I'd set up an alert on that rate. Something simple works to start: if a category sits well above its usual level for that hour for 15 minutes or so, page whoever's on call.

   When it fires, respond in proportion. Locking the room punishes thousands of people for what a few hundred accounts are doing. Slowing down posting for accounts less than a week old can catch most of the people causing it and leave everyone else alone.

   Judge the alert by two numbers: how fast the team acts once it fires, and how many alerts turn out to be real coordinated activity. If most turn out to be reactions to something that happened in the game or the news, the threshold is too low. For new kinds of abuse that no alert covers yet, track how long they run before you notice, and log it in every incident review (chapter 13).

7. Step 07

   Share signals across platforms

   Harm that starts on one service often lands on another. Offenders make first contact in a game and move the child to a private messaging app. Nudify apps show the same shape: the image is made on one service, the tool is promoted on another and the harm lands on a third. No single platform sees the whole thing, so the response has to be coordinated across them.

   Programs that exist for this:

   • Lantern, run by the Tech Coalition since 2023, lets qualifying tech companies and financial institutions share signals about online child sexual exploitation and abuse. That includes content signals such as hashes and URLs, and incident signals such as attempts to move conversations with minors off-platform.
   • GIFCT's hash-sharing database lets member companies match content against hashes of known terrorist and violent extremist material. Members share hashes, not the content.
   • StopNCII.org lets adults create a hash of their intimate images on their own device, so participating platforms can find and remove matching copies. The image never leaves the device.
   • Take It Down, from NCMEC, does the same for nude or sexually explicit images taken when the person was under 18.

   By stage: an early team matches against the lists these programs provide. A growing team joins the ones it's eligible for. At scale, you contribute signals back. Even before you're eligible, design your case records so you could share well-structured signals later. If you'd rather not build detection infrastructure from scratch, ROOST, a nonprofit, publishes free, open-source tools, including the Osprey rules engine and Coop review console.

8. Step 08

   Set privacy limits before launch

   Detection reads people's data, so decide its limits before it runs, not after a complaint:

   • Narrow scope. Detect where the risk is. For grooming, that means adult-to-minor contact, not every conversation.
   • Metadata before content. Who contacts whom, how often and from how new an account often tells you enough to add friction. Read message content only when the pattern justifies it.
   • Agreed retention. Agree with Legal how long signals and evidence are kept, and delete on schedule.
   • Regional rules. Rules on scanning private messages vary by market, especially in the EU. Check each one before launch.

   In the EU, the GDPR's principles of data minimisation and storage limitation (Article 5(1)(c) and (e)) apply to detection data like any other personal data. Processing likely to result in a high risk to people's rights and freedoms needs a data protection impact assessment before it starts (Article 35). For each detection system, write down what it reads, why, who sees the results, how long anything is kept and which markets it runs in. That record is what Privacy and Legal will ask for, and what a regulator will ask for later.

1. 01

   Relying on user reports

   Add at least one proactive layer for every high-risk harm, starting with hash matching if you host images or video.

2. 02

   Reviewing messages one at a time

   Track accounts and relationships over time, and let the response build as signals stack.

3. 03

   Treating removal as the only response

   Use friction, rate limits and feature limits where the evidence is weaker, and keep removal for when it's clear.

4. 04

   Setting thresholds by instinct

   Test them on past confirmed cases and false alarms, write them down, and revisit them every quarter.

5. 05

   Celebrating a rising proactive detection rate on its own

   Read it next to precision. More catches can mean more good users caught by mistake.

6. 06

   Alerting on volume

   Alert on the rate of each category for that hour, and judge the alert by how many fires were real.

7. 07

   Locking the room

   Slow down the accounts causing the problem instead of punishing everyone in the space.

8. 08

   Scanning everything because you can

   Narrow the scope, use metadata first, agree retention and check each market's rules.

From Steven's writing · 6 posts

1. Oct 2, 2026Commentary

   Watch the rate, not the volume

   In Stream's live sports chat data, volume swung 7.5x but the rate of racist content held steady. Alert on rate jumps to spot raids, and slow down new accounts instead of locking the room.

   Also in: 7. Standing up review operations, 13. Crisis response

2. Sep 30, 2026Essay

   Grooming is a pattern, not a message

   Responses build as signals stack on adult-to-minor contact, with thresholds tested on past cases, a queue worked by risk, privacy limits agreed up front, and four numbers that show it works.

   Also in: 6. Child safety and age assurance, 7. Standing up review operations

3. Sep 29, 2026Share

   Nudify apps: harm no single platform sees in full

   The image is made on one service, the tool promoted on another and the harm lands on a third. That's why cross-platform signal sharing through Lantern matters.

   Also in: 6. Child safety and age assurance

4. Sep 28, 2026Essay

   Games are where kids socialize now, and regulators know it

   Grooming builds over weeks and usually moves off-platform. Protections that work act earlier and limit who can reach a child. Measure prevented contact, not just removals.

   Also in: 6. Child safety and age assurance, 16. Regulation and compliance

5. Sep 21, 2026Essay

   Automation rate isn't a measure of maturity

   Automate as much as the evidence supports. Where a wrong decision can't be reversed or someone's safety is at risk, automation prepares the case and a person closes it.

   Also in: 7. Standing up review operations, 14. Moderator wellbeing, 18. AI in Trust & Safety

Outside sources

• Tech Coalition: Lanterncross-platform signal sharing about online child sexual exploitation and abuse.
• GIFCT: Hash-sharing databasehow member companies share hashes of terrorist and violent extremist content.
• StopNCII.org: How it workson-device hashing so adults can stop intimate images being shared on participating platforms.
• NCMEC: Take It Downthe same approach for images taken when the person was under 18.
• ROOSTfree, open-source safety tools, including a rules engine and a review console.
• Microsoft PhotoDNAhash matching for known child sexual abuse imagery, free to qualified organizations.

1. Oct 4, 2026
   AddedAlso: 1. What Trust & Safety is for, 8. Hiring and structuring the team, 15. Working with Product, Legal, Comms and leadership

   Linked Steven's post on why treating Trust & Safety as an operations function is a mistake, and added it to chapter 8's view on where the team should report.

   Read the post: You can't moderate your way out of a systems problem →

2. Oct 4, 2026
   RevisedWith 18 other chapters

   Practical advice that varies by platform, such as cadences, sample sizes, targets and who owns what, is now set out as options with examples, so each team can choose what fits.

3. Oct 3, 2026
   DraftedWith 17 other chapters

   First full drafts of the other 18 chapters, built on Steven's posts, the handbook's principles and the Workbench's open content, with every legal and factual claim checked against its source. Stories from Steven's own work come next.

4. Oct 3, 2026
   AddedWith 14 other chapters

   Linked the first 16 posts to the chapters they inform, and set out the ten principles behind the handbook.