Writing · Safety by design and operations

What I'd tell myself in year one of Trust & Safety

1 min readEssayDiscuss on LinkedIn →

More than ten years in Trust & Safety, across social media, gaming and user-generated content. The work changed a lot in that time. My view of it changed more. If I could sit down with myself in year one, this is what I'd say.

1. You can't moderate your way out of a systems problem. Reputation, account history, age and behavior signals belong in one risk model, and safety belongs in the product from the start.

2. A ban doesn't close the case. The people you remove come back on fresh accounts, so map the network around them and watch for the return.

3. Measure what users experience, not how busy the team was. Removals and queue counts show activity, while exposure to harm and repeat offenders show impact.

4. Automation earns its scope. It takes on more only when its decisions hold up on appeal as well as people's do, and a person owns every call that can't be undone.

5. Every risk needs one named owner. When two teams think they share a harm, nobody owns it, and you find that out during the incident.

All five still shape how I'd build a Trust & Safety team today, whatever the platform.

What would you tell your first-year self in this work? I'd like to read yours, especially the ones I missed.

#TrustAndSafety #OnlineSafety #ContentModeration #Leadership

In the handbook

1. What Trust & Safety is forWhat is this function for, and how do you know it's working?
8. Hiring and structuring the teamWho do you hire, in what order, and where should the team report?
11. Measuring what mattersIs the program making people safer, and how would you prove it?
18. AI in Trust & SafetyWhere does AI help moderation, and how do you keep AI products safe?

Related

  1. Since 2024, the job is proving it works

    Since 2024 the question has moved from whether you have a policy to whether you can show it works: UK risk assessments that must be redone before significant changes, age assurance as the foundation, 48-hour removal of intimate images under the US TAKE IT DOWN Act, and free open-source tooling. Those duties land on different teams, so the leadership team should name one owner for the whole picture.

    2 min read · Essay · What's changing

  2. A good Trust & Safety program is invisible

    When Trust & Safety works, neither users nor the company get hurt, so nobody sees it and leadership finds it easy to devalue and cut. Make it visible by measuring what didn't happen: exposure to harm, repeat offenders, time to action on the worst cases, and what users who ran into harm do next.

    1 min read · Essay · Measuring what matters

  3. Age assurance: what should each check unlock?

    The method matters less than what each level of assurance lets a user do. Errors aren't equal, accounts change hands, and someone has to own the thresholds.

    2 min read · Essay · Child safety and age assurance