Writing · Safety by design and operations

If a regulator asked for your pre-launch risk record tomorrow

If a regulator asked for your pre-launch risk record tomorrow, you should already know what it would show.

This week Ofcom opened an investigation into whether Meta carried out a suitable and sufficient illegal content risk assessment and children's risk assessment before launching Instagram Instants, where content disappears once viewed. It's an investigation, not a finding of a breach. Meta says it takes its obligations seriously, ran a risk analysis and briefed Ofcom on the feature several times before launch.

The Online Safety Act requires services to keep their risk assessments current, to do a fresh one before any significant change to the design or operation of the service, and to keep a written record of each. One feature launch is enough to put that to the test. If you have UK users, this is a question about your own process.

I'd want the assessment done at design review, in the same slot as the security review, with a named sign-off. By the time a feature is on a launch calendar, the decisions that create or remove risk have mostly been made. The abuse pre-mortem belongs where the design can still change.

Disappearing content deserves its own page. If something is gone after one view, the plain questions are how a user reports it afterwards and what my team can still see when they do. Whatever the answer, it's a design decision, and it needs an owner and a written mitigation before launch.

The record matters as much as the work. Every launch should leave a dated assessment behind it, with mitigations and the name against each one, filed so I could hand it over within a day. The assessment is product work. The record is the proof it was done.

Links in comments. Pre-mortum tool that I built in comments as well.

#TrustAndSafety #OnlineSafety #SafetyByDesign #ChildSafety #Compliance

Related

  1. What I'm reading: Ofcom and Instagram Instants, Ofcom's data demands in court, and Roblox in Kansas

    Ofcom is investigating whether Meta risk-assessed Instagram Instants before launch, and Meta, TikTok and X are challenging Ofcom's demands for moderation data in court. Kansas settled with Roblox for more than $10 million plus age checks and age-grouped chat, and OpenAI's own tests show its new EU text watermark weakens when 10% of the words are swapped.

    1 min read · Reading roundup · What's changing

  2. Since 2024, the job is proving it works

    Since 2024 the question has moved from whether you have a policy to whether you can show it works: UK risk assessments that must be redone before significant changes, age assurance as the foundation, 48-hour removal of intimate images under the US TAKE IT DOWN Act, and free open-source tooling. Those duties land on different teams, so the leadership team should name one owner for the whole picture.

    2 min read · Essay · What's changing

  3. What I'd tell myself in year one of Trust & Safety

    Five lessons from more than ten years in the field: treat safety as a systems problem built into the product, don't treat a ban as a closed case, measure what users experience rather than how busy the team was, let automation earn its scope, and give every risk one named owner.

    1 min read · Essay · Safety by design and operations