Writing · Safety by design and operations

Watch the rate, not the volume

Kenzie Wilson at Stream published a super interesting article with a week of moderation data from live sports chat during the NFL season. (A guilty pleasure read, because I love football)

Chat volume swung 7.5x across a single day, but the rate of racist content stayed between 65 and 100 per 1,000 enforcement actions at every hour. Game day added about 3%.

Kenzie's takeaway is that most of this comes down to capacity and configuration, and I agree. A steady rate also makes a good baseline. On a high volume night, volume goes up and the rate stays put. If the rate for one category climbs while volume is normal, that usually means something coordinated, like a raid or a group targeting one player.

I'd set up an alert on that rate. Something simple works to start: if a category sits well above its usual level for that hour for 15 minutes or so, page whoever's on call.

When it goes off, locking the room isn't a great first move in the middle of a big game. It punishes thousands of fans for what a few hundred accounts are doing. Kenzie found about half the accounts flagged each day are new, so I'd start by slowing down posting for accounts less than a week old. That catches most of the people causing it and leaves everyone else alone.

To know if it's working, I'd look at how fast the team acts once an alert goes off, and how many alerts turn out to be real coordinated activity. If most of them end up being a reaction to something in the game, the threshold is set too low.

Thanks to Kenzie for sharing real numbers!

Related

  1. What I'd tell myself in year one of Trust & Safety

    Five lessons from more than ten years in the field: treat safety as a systems problem built into the product, don't treat a ban as a closed case, measure what users experience rather than how busy the team was, let automation earn its scope, and give every risk one named owner.

    1 min read · Essay · Safety by design and operations

  2. If a regulator asked for your pre-launch risk record tomorrow

    Ofcom's investigation into whether Meta risk-assessed Instagram Instants before launch is a test any service with UK users could face. Do the assessment at design review with a named sign-off, decide how disappearing content gets reported, and file a dated record you could hand over within a day.

    1 min read · Essay · Safety by design and operations

  3. Launch week is the hardest test a safety program gets

    Most of what goes wrong in launch week is decided before day one. Pre-approve friction for new and unverified accounts, triage by severity, watch the rate of abuse rather than raw counts, and agree with product how much friction you'll accept, then review it after week one.

    1 min read · Essay · Safety by design and operations