Writing · Measuring what matters

A good Trust & Safety program is invisible

1 min readEssayDiscuss on LinkedIn →

A good Trust & Safety program is invisible. It's invisible because neither the company nor its users are getting hurt.

On the user side, people aren't getting harassed, scammed or groomed, and they aren't leaving the platform because of it. On the company side, there are no regulator letters, no fines, no lawsuits, no ugly headlines, no advertisers or partners walking, and no app store trouble.

When the program works, none of that happens. So nobody sees it. That's exactly the job. It's also why leadership often finds it easy to devalue. With nothing going wrong, safety looks like a cost with no return, and it becomes an easy line to cut in a tough quarter, right up until something goes wrong.

The way to make it visible is to measure what didn't happen. How often harm actually reaches people, how many offenders come back after enforcement, how fast the worst cases get handled, and what users who ran into harm do next compared with users who didn't. I put the ones I'd use into a free, open metrics framework, 36 of them, linked in the comments.

Protecting users and protecting the company are the same work. How do you make the case for a program whose best result is nothing happening?

#TrustAndSafety #Leadership #OnlineSafety

In the handbook

1. What Trust & Safety is forWhat is this function for, and how do you know it's working?
11. Measuring what mattersIs the program making people safer, and how would you prove it?
19. Budgets, roadmaps and making the caseHow do you get the people and money the program needs?

Related

  1. Harassed players look like your best-retained users

    Raw retention data hides the cost of toxicity because harassed players are the most engaged. A matched cohort shows it, and safety exposure belongs on the retention dashboard.

    1 min read · Essay · Measuring what matters

  2. What I'd tell myself in year one of Trust & Safety

    Five lessons from more than ten years in the field: treat safety as a systems problem built into the product, don't treat a ban as a closed case, measure what users experience rather than how busy the team was, let automation earn its scope, and give every risk one named owner.

    1 min read · Essay · Safety by design and operations

  3. Since 2024, the job is proving it works

    Since 2024 the question has moved from whether you have a policy to whether you can show it works: UK risk assessments that must be redone before significant changes, age assurance as the foundation, 48-hour removal of intimate images under the US TAKE IT DOWN Act, and free open-source tooling. Those duties land on different teams, so the leadership team should name one owner for the whole picture.

    2 min read · Essay · What's changing