Writing · Steven Macchia

Every public safety claim is something someone can test

2026-10-16T15:10:31.286Z

A lot of the US pressure on games and chat platforms over child safety is coming from state attorneys general right now, mostly through consumer protection law. The question in those cases is whether what a company told parents about safety matches what actually happened on the platform.

Last Thursday, Florida's attorney general sued Discord under the state's Deceptive and Unfair Trade Practices Act. The complaint says Discord marketed itself as safe for minors, lists eight incidents since 2022 where it says an adult exploited a minor through the platform, and asks the court to bar Discord from marketing itself as safe without real age verification.

New Jersey brought a similar suit last year, and several states have sued Roblox since 2025. These are allegations, and none of the cases has been decided.

Every public safety claim, from the help center to a sales deck a partner saw two years ago, is something someone can test. So sit down with Legal, give one person ownership, and get Marketing in the room. List every claim, attach the evidence behind it (the control, the metric, when it was last tested), and fix or soften anything that has none.

Be exact about age checks, since a self-declared birthday and a verified age are different claims. And recheck the list whenever the product changes. A default flipped off can make a true sentence false without anyone touching the page.

Someone must own the list of public safety claims and actually check it against your product.

Floridas suit, as reported by WFSU: https://lnkd.in/eRuriyCn

#TrustAndSafety #ChildSafety #Compliance

A good Trust & Safety program is invisible

2026-10-15T15:16:48.294Z

A good Trust & Safety program is invisible. It's invisible because neither the company nor its users are getting hurt.

On the user side, people aren't getting harassed, scammed or groomed, and they aren't leaving the platform because of it. On the company side, there are no regulator letters, no fines, no lawsuits, no ugly headlines, no advertisers or partners walking, and no app store trouble.

When the program works, none of that happens. So nobody sees it. That's exactly the job. It's also why leadership often finds it easy to devalue. With nothing going wrong, safety looks like a cost with no return, and it becomes an easy line to cut in a tough quarter, right up until something goes wrong.

The way to make it visible is to measure what didn't happen. How often harm actually reaches people, how many offenders come back after enforcement, how fast the worst cases get handled, and what users who ran into harm do next compared with users who didn't. I put the ones I'd use into a free, open metrics framework, 36 of them, linked in the comments.

Protecting users and protecting the company are the same work. How do you make the case for a program whose best result is nothing happening?

#TrustAndSafety #Leadership #OnlineSafety

Parents' top gaming concern is the rule they set least

2026-10-13T13:34:39.115Z

Ofcom published new research today on kids and gaming, Growing Up Gaming. The bit that stuck with me: 55% of children who game online say they at least sometimes play with people they've never met offline. Talking to strangers is also parents' biggest concern, ahead of bullying, yet fewer than half (43%) set any rule about who their child can play with.

Tony Stower, Ofcom's online safety gaming lead, said: "Parents can do their bit, but ultimately the buck stops with gaming platforms." The law is moving the same way.

Platforms serving UK children already carry that duty under the Online Safety Act, and the EU's KIDS Act, proposed last month, would ban unsolicited contact from strangers in online games. The gap between parents' biggest concern and the rule they set least is the platform's job to close, with defaults.

What good defaults look like to me: a kid's account starts with contact limited to people they actually know. An adult trying to reach a kid they don't know hits some friction. And parents get one clear switch for who can contact their child, not a maze of settings.

The other piece is what teams measure. Most of us report how much got removed after the fact. I'd want a number for how often a stranger reaches a kid in the first place, by week, and I'd want it going down.

#TrustAndSafety #ChildSafety #GamingSafety #OnlineSafety

Ofcom's research: https://lnkd.in/eCMEajUX

What I'd tell myself in year one of Trust & Safety

2026-10-08T15:06:11.894Z

More than ten years in Trust & Safety, across social media, gaming and user-generated content. The work changed a lot in that time. My view of it changed more. If I could sit down with myself in year one, this is what I'd say.

1. You can't moderate your way out of a systems problem. Reputation, account history, age and behavior signals belong in one risk model, and safety belongs in the product from the start.

2. A ban doesn't close the case. The people you remove come back on fresh accounts, so map the network around them and watch for the return.

3. Measure what users experience, not how busy the team was. Removals and queue counts show activity, while exposure to harm and repeat offenders show impact.

4. Automation earns its scope. It takes on more only when its decisions hold up on appeal as well as people's do, and a person owns every call that can't be undone.

5. Every risk needs one named owner. When two teams think they share a harm, nobody owns it, and you find that out during the incident.

All five still shape how I'd build a Trust & Safety team today, whatever the platform.

What would you tell your first-year self in this work? I'd like to read yours, especially the ones I missed.

#TrustAndSafety #OnlineSafety #ContentModeration #Leadership

Financial sextortion moves in hours, so start with what the teen sees

2026-10-07T14:54:46.448Z

Most grooming detection is built for patterns that play out over weeks. Financial sextortion doesn't give you weeks. An account posing as another teenager makes contact, asks for an image almost right away, then demands money and threatens to send it to the teen's friends and family. It can happen inside a day.

NCMEC received an average of 137 financial sextortion reports a day in 2025, many involving teenage boys. It knows of at least three dozen teenage boys in the US who died by suicide after being targeted. If I were setting priorities, I'd start with what the teen sees in that first conversation.

Defaults first. Either under-18s can't get messages from accounts they aren't connected to, as Ofcom's UK codes already expect of higher-risk services, or new accounts get a hard cap on how many minors they can message. Then a notice when the sender is a stranger or seems to be in another country. Some real friendships will start slower, and messages from new accounts to minors they don't know will drop, which is the point. I'd take that trade, and agree with product up front how we'll measure it.

When a teen reports, they need a fast, shame-free answer: this isn't your fault, don't pay, and NCMEC's Take It Down can help get images removed. Staff it like the urgent queue it is, with someone on call at 2am who can make an emergency referral. In the US it's also a mandatory CyberTipline report, so flag any suicide risk in it.

#TrustAndSafety #ChildSafety #OnlineSafety #SafetyByDesign

What I'm reading: an open policy model, Xbox's adult checks, Discord in Brazil and KOSA

2026-10-07T11:08:30.988Z

Few T&S things I'm reading today:

- Musubi released PolicyLM-1.7B, an open-weights moderation model that takes your written policy and returns a decision in under 50ms. We've already added it to the Policy Stress Test on T&S Workbench so you can test your own policies against it.

https://lnkd.in/eYcbF7ri

- Xbox is emailing longtime users saying it has marked their accounts as verified adults based on account history alone. Big age assurance move, and nobody is ever going to appeal being called an adult.

https://lnkd.in/eRRaQPEM

- A Brazilian judge ordered Discord to strengthen its parental controls and age verification after a teen's death. The government's ~$100M lawsuit is still pending.

https://lnkd.in/emw3msrA

- UK MP Jess Asato, who is suing xAI over Grok deepfakes, told the NYC Council it's a design flaw, not user misuse.

https://lnkd.in/e82Ht4x8

- KOSA is stuck again. Rand Paul blocked unanimous consent, and the House and Senate versions still have to be reconciled before Jan 3.

https://lnkd.in/e98KCQHX

What's on your radar this week? Usually post this once a week but so much going on and the industry is moving so quickly now

#TrustAndSafety #OnlineSafety #AgeAssurance #ChildSafety #ContentModeration

If a regulator asked for your pre-launch risk record tomorrow

2026-10-06T14:45:06.704Z

If a regulator asked for your pre-launch risk record tomorrow, you should already know what it would show.

This week Ofcom opened an investigation into whether Meta carried out a suitable and sufficient illegal content risk assessment and children's risk assessment before launching Instagram Instants, where content disappears once viewed. It's an investigation, not a finding of a breach. Meta says it takes its obligations seriously, ran a risk analysis and briefed Ofcom on the feature several times before launch.

The Online Safety Act requires services to keep their risk assessments current, to do a fresh one before any significant change to the design or operation of the service, and to keep a written record of each. One feature launch is enough to put that to the test. If you have UK users, this is a question about your own process.

I'd want the assessment done at design review, in the same slot as the security review, with a named sign-off. By the time a feature is on a launch calendar, the decisions that create or remove risk have mostly been made. The abuse pre-mortem belongs where the design can still change.

Disappearing content deserves its own page. If something is gone after one view, the plain questions are how a user reports it afterwards and what my team can still see when they do. Whatever the answer, it's a design decision, and it needs an owner and a written mitigation before launch.

The record matters as much as the work. Every launch should leave a dated assessment behind it, with mitigations and the name against each one, filed so I could hand it over within a day. The assessment is product work. The record is the proof it was done.

Links in comments. Pre-mortum tool that I built in comments as well.

#TrustAndSafety #OnlineSafety #SafetyByDesign #ChildSafety #Compliance

What I'm reading: Ofcom and Instagram Instants, Ofcom's data demands in court, and Roblox in Kansas

2026-10-06T11:38:53.318Z

Lots going on in T&S this week. Here's what I'm reading:

- Ofcom is investigating Meta over Instagram Instants (the disappearing photo feature), asking whether they actually risk assessed it, including for kids, before it launched in May.

https://lnkd.in/emtugTf9

- Meta, TikTok and X are taking Ofcom to court over its data demands, things like how many posts got removed or downranked and how many people saw harmful content.

https://lnkd.in/eMuZrQaq

- Kansas AG got a $10M settlement out of Roblox over child safety.

https://lnkd.in/eMCmuYdE

- KOSA stalled in the Senate again, and age assurance keeps getting pulled into the debate.

https://lnkd.in/e_uAPjHu

- Consumer groups are pushing Congress to look into scams on tech platforms.

https://lnkd.in/ek__jbv2

- OpenAI is going to watermark ChatGPT and Codex text for EU users. They admit detection drops from about 92% to 66% if you swap out 10% of the words.

https://lnkd.in/eQdTMbge

What else should be on my list?

#TrustAndSafety #OnlineSafety

Launch week is the hardest test a safety program gets

2026-10-05T21:02:41.862Z

Launch week is the hardest test a safety program gets, and most of what goes wrong is decided before day one.

New users arrive all at once, and so do (many of) the people who prey on them. In a game thats raids on the busiest chats and scams in trade and gifting. In a social app its strangers testing the new DMs. Much of it comes from the newest accounts.

Run the abuse review alongside the security review, and decide the friction before launch: slower posting and lower limits for brand-new accounts, tighter rules for anyone unverified. Get product and legal to approve the thresholds in advance, so nobody is hunting for sign-off at 2am on launch night. And when a raid hits, slow down the newest accounts instead of locking the room for everyone.

Reports will spike with the user count, so triage by severity, child safety and threats first. Read the rate of abuse per user, not the raw counts. A jump in the rate for one category is the signal, and it usually means something coordinated.

The necessary conversation is with product. Friction on new accounts slows real newcomers in the week you most want them to stay. Id rather have that argument a month out than on launch night. Agree how much friction youll accept and for how long, then review after week one what fired, what youd pre-approve next time and what it cost new users.

#TrustAndSafety #SafetyByDesign #ProductManagement #GamingSafety

Since 2024, the job is proving it works

2026-10-05T15:05:31.132Z

Ask what changed in Trust & Safety since 2024 and most people say AI. Id say its what you have to prove. The question used to be whether you had a policy. Now its whether you can show it works.

Last month I wrote that you cant moderate your way out of a systems problem. That idea now comes with deadlines. Under the UK Online Safety Act, services with UK users owed written illegal content risk assessments by March 2025, and any significant change needs a fresh one before it ships. That last part turns a one-off filing into a record you have to keep current.

Age assurance is the foundation. Australia's social media minimum age of 16 took effect in December 2025, and the EU's July 2025 guidelines on minors recommend age estimation even where a platforms own minimum age is under 18. The method matters less than what each level of check unlocks, and since a threshold touches product, safety, legal and privacy at once, it needs one owner.

Removal runs on a clock too. Since May 2026 the US TAKE IT DOWN Act gives covered platforms 48 hours from a valid request to remove a nonconsensual intimate image, AI-made ones included. The tooling floor moved as well. ROOST launched in February 2025 and publishes free open-source review tools, open-weight models can classify content against a written policy, and were too small is a weaker answer now.

If a CEO asked me where to start, Id ask who owns all four before I picked one. In most companies the risk assessment sits with Legal, the age checks with Product and the takedown inbox with Support, and nobody sees them side by side until a regulator does.

The fix is one owner for the whole picture, named by the leadership team and agreed out loud, so nobody assumes someone else has it.

If you can't name that person today, start there. It's cheaper than finding out from a regulator.

#TrustAndSafety #OnlineSafety #AgeAssurance #SafetyByDesign #Compliance

Saturday reading: child safety gaps in games, Ofcom in court and California's new laws

2026-10-03T12:06:59.334Z

What I'm reading in Trust & Safety with my Saturday morning coffee ☕

1. BBC: Australia finds "significant" child safety gaps on Steam, Roblox, Fortnite and Minecraft

eSafety found Fortnite and Minecraft still mostly rely on self-declared age, and Valve was the only one not using industry-standard tech to detect known harmful content. Age assurance is the foundation here, because every other safeguard assumes you know who's a kid.

https://lnkd.in/er_Tce7u

2. Meta, X, and TikTok are challenging Ofcom’s information requests under the Online Safety Act in court. They are seeking a hearing on Monday. If the powers of regulators are limited, they will end up relying on platforms’ word instead of verifying the information.

https://lnkd.in/er9CyBNS

3. KATV: Experts Raise Concerns About California’s New Online Safety Laws for Children

The new regulations prohibit addictive feeds and mandate AI intervention when self-harm signals are detected. However, some mental health experts express skepticism about their effectiveness. While mandating “detect and intervene” is straightforward, accurately identifying and addressing self-harm in a child at 2 am presents significant challenges.

https://lnkd.in/ewnyJr26

What's on your reading list this weekend?

#TrustAndSafety #OnlineSafety #GamingSafety #AgeAssurance #ChildSafety

Watch the rate, not the volume

2026-10-02T15:37:22.135Z

Kenzie Wilson at Stream published a super interesting article with a week of moderation data from live sports chat during the NFL season. (A guilty pleasure read, because I love football)

Chat volume swung 7.5x across a single day, but the rate of racist content stayed between 65 and 100 per 1,000 enforcement actions at every hour. Game day added about 3%.

Kenzie's takeaway is that most of this comes down to capacity and configuration, and I agree. A steady rate also makes a good baseline. On a high volume night, volume goes up and the rate stays put. If the rate for one category climbs while volume is normal, that usually means something coordinated, like a raid or a group targeting one player.

I'd set up an alert on that rate. Something simple works to start: if a category sits well above its usual level for that hour for 15 minutes or so, page whoever's on call.

When it goes off, locking the room isn't a great first move in the middle of a big game. It punishes thousands of fans for what a few hundred accounts are doing. Kenzie found about half the accounts flagged each day are new, so I'd start by slowing down posting for accounts less than a week old. That catches most of the people causing it and leaves everyone else alone.

To know if it's working, I'd look at how fast the team acts once an alert goes off, and how many alerts turn out to be real coordinated activity. If most of them end up being a reaction to something in the game, the threshold is set too low.

Thanks to Kenzie for sharing real numbers!

Run the abuse pre-mortem at design review

2026-10-02T14:21:28.903Z

The best time to run an abuse pre-mortem, the session where a team works out how a feature could be misused before it ships, is at design review.

At that point, changing a default is easy. If new accounts shouldn't be able to message strangers, or should have a lower gifting cap in their first week, it's a quick edit to the spec. Once the feature has shipped and people are using it, the same change means taking something away from users, and that's a much harder decision to get through.

Being invited that early is something T&S earns. It means being selective: full reviews for launches that let strangers contact each other, move money or items between users, expose location, or change what minors can do, and a short checklist for everything else. It also means answers that come back in days. A review that slows every release won't stay in the launch plan for long.

Some safeguards cost the feature its own metric. A gifting limit on new accounts will reduce gifting, and buy-in is difficult (revenue is king). Agreeing the exit criteria up front keeps the follow-up simple.

If a launch goes ahead with a known risk, the person who owns the product outcome signs off on it, with a date to revisit. T&S makes sure the risk is in front of them when they decide, and takes it to the exec who owns safety risk when it's too serious to be one person's call.

For the program itself, one method is to track how many incidents in a features' first 90 days the review predicted, and how many accepted risks turned into incidents. If that second number keeps climbing, risks are being accepted at too low a level.

I built a free tool for running a pre-mortem at design stage. Link in the comments.

#TrustAndSafety #SafetyByDesign #ProductManagement #GameDev

Harassed players look like your best-retained users

2026-10-01T18:57:24.344Z

Pull the raw retention data at most game studios and harassed players look like some of the best-retained users.

They chat more, queue more and play longer, so they run into more abuse. A team that stops there concludes toxicity doesnt hurt retention, and the safety budget conversation ends before it starts.

Studies tell us more.

Take This and Nielsen found 61% of players have skipped spending in a game because of how other players treated them (THATS A MASSIVE REVENUE HIT), and 60% have quit a match or a game over harassment. But surveys rarely move a budget. A studios own data does, if its cut correctly.

The cut that works is a matched cohort. Take new players whose early sessions included an actioned incident, and compare their 7- and 30-day return against new players with clean sessions, matched on playtime, mode, region and platform. Without the matching, the analysis measures engagement instead of harm.

The same blind spot affects enforcement on high-value players. Banning a top spender hits the revenue report the next day, while the players they drove away stay invisible unless someone builds the cohort.

Numbers worth tracking:

- Share of new players whose first five matches include an actioned incident

- D7 and D30 return for exposed new players vs the matched group

- Voice chat opt-out rate in a players first week

etc.

Safety exposure belongs as a standard cut on the retention dashboard, owned jointly by T&S and data teams and reviewed alongside every other churn driver. Surface level metrics don't tell the whole story.

2023 Take This + Nielsen study in comments

#TrustAndSafety #GameDev #Userretention #CommunityManagement

A proper introduction

2026-10-01T13:19:16.614Z

With so many new connections and followers over the past couple of weeks since posting about the state of Trust & Safety, a proper introduction may be due.

I am Steven Macchia, a Trust and Safety and compliance leader with more than 10 years of experience protecting users in online communities, mostly in social media, gaming, and user-generated content. I have built safety functions from the ground up, led global regulatory compliance, and worked closely on child safety and youth protection.

Why I decided to start writing on LinkedIn:

The online safety landscape is moving faster than ever. New laws are taking effect, regulators are acting on them, platforms are changing their policies, and new threats keep appearing. All of this is happening at once and is spread across dozens of sources.

Practitioners rarely have time to track all of it, but every piece of it affects the work.

My goal is simple: one clear, reliable place to see what changed in Trust and Safety and why it matters.

What you can expect here

- Regular posts on the most important Trust and Safety, online safety, and regulatory news

- Context and perspective from someone who gets their hands dirty with work, not just headlines

- A particular focus on child safety, age assurance, gaming, and platform compliance

- Free resources, including T&S Workbench, my set of tools for Trust and Safety professionals at stevenmacchia.com (WIP)

I would also like to hear from you. What topics should I cover more closely? What developments are you watching? Tell me in the comments. Some of the best insights in this field come from people doing the work every day - as bad actors are continually evolving.

#TrustAndSafety #OnlineSafety #ChildSafety #Compliance #ContentModeration

Why early grooming detection matters

2026-09-30T22:59:48.148Z

This is why what Aiba is building matters. Tools that help platforms spot grooming early protect kids and the communities around them.

Launches get a security review. Almost none get an abuse review.

2026-09-30T20:08:15.901Z

Most product launches get a security review. Almost none get an abuse review: who will misuse this, how, and what do we ship on day one to stop it.

So I built a free tool that does one in about three minutes.

You answer 12 plain questions about what you're building and who uses it.

It gives you:

→ The abuse risks that apply, rated by how severe and how likely

→ The safeguards to ship before launch, with an owner for each, in priority order

→ The laws that likely apply in your markets

→ A launch plan you can send straight to Jira, Linear or GitHub

It runs in your browser. Nothing you type leaves your device. (Unless you opt-in to Claude functionality on some tools)

No sign-up.There are worked examples for a marketplace, a dating app, a teen social app and a few others if you'd rather look before you answer anything.

Its part of a larger project Ill share more of soon. For now Id like to know where this one is wrong: which risks are missing, which safeguards are naive, what a real launch review would ask that this doesn't.

Link in the first comment.

Grooming is a pattern, not a message

2026-09-30T14:22:08.427Z

Grooming in games almost never shows up in one message. It shows up as a pattern. An adult account friends a lot of younger players it has no connection to, gifts in-game currency early, asks about parents, moves the conversation to private voice, and eventually pushes for another app.

Each step can be innocent on its own, so the response has to build as signals stack. One signal on an adult-to-minor contact gets logged. Two or more in a short window add friction: gifting limits, a safety prompt, no new private channels. An off-platform request after that restricts contact and sends the full history to a trained reviewer.

Those thresholds should come from data, not instinct. Test them against past confirmed cases and past false alarms, set them where precision holds and the review team can keep pace, and revisit them every quarter, because offenders learn what triggers friction and route around it.

The hard calls happen when flagged relationships outnumber reviewers. Protective actions stay automatic through those weeks, and the queue gets worked by risk, starting with off-platform requests and players under 13. A restricted account waiting a day for review is a cost worth paying.

Privacy sets the other boundary. Limit pattern detection to adult-to-minor pairs, start with metadata before message content, agree retention limits with legal before launch, and check each region, since rules on scanning private messages vary, especially in the EU.

Four numbers show whether it works: time from first signal to protective action, share of confirmed cases caught before the off-platform move, precision at each step, and how long high-risk cases wait for review.

All of this gets tested the day a regulator, a parent or a court asks why a specific account was or wasnt restricted. That question is only answerable if one person owned the thresholds, each change was written down with the numbers behind it, and the decision to report to law enforcement was made by people trained for it.

#TrustAndSafety #ChildSafety #OnlineSafety #Gaming #ContentModeration

Nudify apps: harm no single platform sees in full

2026-09-29T18:46:20.700Z

Nudify apps are a growing problem, and a clear example of harm no single platform sees in full. The image gets made on one service, the tool gets promoted on another, and the harm lands on a third.

Thats why signal sharing through Lantern matters. Glad to see the Tech Coalition getting this in front of a wider audience. 👏

Age assurance: what should each check unlock?

2026-09-29T18:13:24.004Z

Most of the debate about age assurance in games is about which method to use. Face scans, IDs, parental consent. I think the more important question is what each one should let a user do.

Some methods that I've seen used and work: A birthday typed in at signup gets you locked-down defaults and not much else. An age estimate can open up text and voice chat with people your own age. Anything that lets an adult reach a minor privately should need a verified account or a parents sign-off. Friction goes where the risk is, and most users never feel it. (Obviously context and platform matter TREMENDOUSLY)

None of these methods is perfect, and the mistakes aren't equal. Put an adult in a teen group and they get annoyed, and some borrow a younger relatives account. Put a 15-year-old in an adult group and every protection built for them is gone. Optimizing for 13, 16 and 18 becomes one of the most important points.

Most age checks happen once, at sign-up or when a feature unlocks, but accounts don't stay with one person. They get shared, sold, or passed down to a younger sibling, and months later the person playing no longer matches the age on file.

Banning every account that looks off would hit a lot of legitimate users, and doing nothing leaves the gap open - these are off the table for most programs anyway. We can do way better. The better move is to put the account back on safer defaults right away and ask for a stronger check. The kid stays protected while it gets sorted out.

Someone also has to own these lines. Where the thresholds sit, and what evidence each one needs, affects product, safety, legal and privacy all at once. If nobody owns it, each team nudges it toward its own goals and nobody notices the drift. It deserves the same sign-off process as a policy change.

And completion rate is a weak metric on its own. Id rather know how many accounts in teen groups turn out to be adults, how many kids end up in adult groups, and how long it takes to catch an account thats changed hands.

The era of voluntary child safety is ending

2026-09-29T11:45:56.591Z

The era of voluntary child safety is ending. Courts, attorneys general and regulators are turning platform promises into binding obligations. AI products are now in scope too. Here's what I'm reading this morning:

1/ India The Supreme Court said protections for under-18s on social media must carry legal force. The government is now looking at using the IT Intermediary Rules to require age verification and parental consent.

→ Another major market moves toward mandatory age assurance.

🔗 https://lnkd.in/eb2cHuyR

2/ Florida vs. ChatGPT Floridas AG asked a court to temporarily halt ChatGPT development, citing COPPA violations and weak guardrails for minors.

→ AGs now treat GenAI like social platforms. AI age gating and safety evals need the same rigor as feeds.

🔗 https://lnkd.in/e9NPz8ag

3/ Copilot data labeling 404 Media reports that Prolific contractors review Microsoft Copilot prompts and images for quality tuning. Many are sexual, and some involve minors.

→ Data labeling is moderation work. It needs the same wellbeing support and escalation paths.

🔗 https://lnkd.in/e_5n5E4W

4/ TikTok and Alabama TikTok settled for at least $100M. It agreed to a 2-hour daily limit for minors, overnight restrictions and stronger parental controls.

→ Settlements are now writing product requirements. Expect other states to ask for the same terms.

🔗 https://lnkd.in/eKZKXSxu

5/ Meta in New Mexico A jury found Meta violated state consumer law more than 43M times with deceptive privacy claims. The penalty hearing is on Oct 1.

→ Your public safety claims are litigation evidence. What comms says has to match what enforcement actually does.

🔗 https://lnkd.in/enTemq7Z

The common thread: the question used to be Do you have a policy? Now its Can you prove it works? Which of these hits your roadmap first?

#TrustAndSafety #ContentModeration #OnlineSafety #ChildSafety #AISafety #AgeAssurance

Games are where kids socialize now, and regulators know it

2026-09-28T21:14:34.710Z

Games have become one of the main places kids socialize online, and regulators and courts are starting to treat them that way.

In April, Australia's eSafety Commissioner sent legal notices to Roblox, Minecraft, Fortnite and Steam asking what they are doing to prevent grooming. The regulator noted that offenders often make first contact in a game and then move children to private messaging apps. In the US, LA County sued Roblox in February, and several state attorneys general have reached settlements with the company. The consolidated federal case over child abuse claims now includes 182 lawsuits.

Grooming in games rarely shows up in a single message. It builds over weeks through friend requests, gifted in-game currency, requests to keep things private, and eventually a push to move to another app. Moderation that reviews messages one at a time misses most of that.

So the protections most likely to work act earlier and look at patterns. Roblox now requires a facial age check to use chat, and adults can't message users under 16 by default. That limits who can reach a child in the first place. And because the riskiest step happens off-platform, sharing warning signs between platforms matters too. The Tech Coalition's Lantern program was built for that, and Roblox and Discord were among its founding members.

The gaming companies that make real progress here will be the ones that measure how often risky contact is prevented, alongside how much content gets removed.

A banned account is not a closed case

2026-09-25T14:41:54.720Z

Online enticement is now one of the fastest-growing categories NCMEC tracks. In 2025, the CyberTipline received 1.4 million enticement reports, up more than 150% from the year before. Part of that jump comes from the REPORT Act making enticement mandatory to report, and that's the point: platforms are now expected to report behavior, not just files.

That changes the enforcement problem. Much of CSAM detection is built around content you can hash and match. Enticement is a pattern of contact over time, and the decisions around it are harder, including when and how visibly to act on an account.

Detection is starting to catch up. Thorn has added a grooming label to its conversation-level classifier, Roblox has open-sourced Sentinel, the system it uses to surface early grooming patterns in chat, and companies like Aiba are building behavioral detection for various online communites. Thats great progress, but for platforms - means will facing harder account decision more often, not less.

Timing matters because of how offenders operate. Thorn's 2022 grooming research found that nearly two in three minors had been invited by an online-only contact to move a conversation to a private chat on a different platform, and groomers routinely use that move to isolate victims. A visible ban on one platform doesn't end that relationship, and it can speed up the move to a place where no one is watching.

None of this argues for leaving harmful material up. Content comes down, gets preserved, and gets reported, and a child in immediate danger comes first. NCMEC escalated more than 53,000 urgent or imminent-danger reports to law enforcement last year.

The judgment call is on the account: act immediately and visibly, or restrict quietly while investigations maps linked accounts and law enforcement decides what it needs. Immature programs default to the ban because it's the only lever their tooling gives them, and it looks like success on a dashboard. In practice, the offender returns on a fresh account with none of the history that flagged them, linked accounts go unexamined, and the child is still reachable wherever the conversation already moved.

The industry is building the other half. Through the Tech Coalition's Lantern program, participating companies had shared more than 2 million signals by the end of 2025, supporting over 350,000 enforcement actions across platforms. If offenders work across services, a single platform's ban is one move in a longer sequence, and the response has to be coordinated to match.

A banned account is not a closed case. In enticement work, speed isn't the standard. A safe child and a fully mapped network are.

#ChildSafety #TrustAndSafety #ResponsibleAI

Appeal overturns: the early warning for automation

2026-09-24T14:48:33.759Z

As more enforcement moves to automation, one of the most important questions a Trust & Safety leader can answer is how quickly they'd know if the system started getting things wrong.

Most programs already have one of the best early-warning signals for this and don't treat it that way: the decisions users contest and win.

Broken out by policy area, enforcement source, market etc., appeal overturn data shows where guidance is unclear, where automated decisions hold up worse than human review, where language and context are getting lost, and where a model has drifted. It often shows drift weeks before anything else does.

That makes it something you can govern with. A few rules I'd put in place:

- Automation doesn't expand into a new policy area until its overturn rate is at or below human review.

- Every model or threshold change gets an overturn baseline before launch and a check (x) weeks after. If the rate moves past an agreed limit, the change rolls back.

- When a policy area stays above its overturn target, the first thing reviewed is the guidance itself, before anyone looks at reviewer performance.

This signal has limits. It only measures over-enforcement, since nobody contests the harm you missed, so it needs to sit next to prevalence sampling. And appeal rates vary by market, so a low overturn rate where few people appeal isn't automatically good news.

Automation in T&S isn't slowing down. The programs that handle it well will be the ones that decide in advance what earns automation more scope - and what takes that scope away.

Activity is easy to measure. Impact is harder.

2026-09-23T16:38:43.784Z

If your team took down 40% more harmful content last quarter than the quarter before, is that good news?

Most Trust & Safety teams would present it that way. The honest answer is that you don't know yet.

It could mean detection improved. It could mean harm on the platform grew. It could mean automation got more aggressive, and legitimate users are paying for it.

Volume alone can't tell you which.

This is one of the most common gaps in how T&S programs report to leadership. We measure how much work the system does, like removals, reports actioned, automation rate, and time to decision. Those numbers are easy to produce, and they almost always go up and to the right.

But what gets measured gets resourced. When a program is rewarded for activity, it optimizes for activity.

The metrics that matter are harder to build:

- Is exposure to severe harm going down?

- After enforcement, do bad actors stop, or just come back on new accounts?

- Are we catching high-risk behavior earlier?

- What are appeals telling us about where our policy or automation is wrong?

Operational metrics show the machine is running. Outcome metrics show it's working. A mature program reports both, and is willing to tell leadership when they disagree. That's the real leadership test. It takes discipline to walk into a business review and say, "Our numbers are up, and I'm not convinced we're safer."

But that's the conversation that earns credibility, directs investment to the right places, and keeps Trust & Safety from being judged like a cost center measured by throughput.

Activity is easy to measure. Impact is harder.

Automation rate isn't a measure of maturity

2026-09-21T13:22:14.043Z

Automation rate is a useful efficiency metric, but it isn't a measure of maturity. It counts the decisions humans didn't have to make, not whether those decisions were correct.

The goal I work toward is automating as much decision-making as the evidence supports, and reserving human judgment for cases where context, severity, or the cost of error requires it.

A classifier alone can not yet make that determination. The system also has to weigh confidence, severity, user history, behavioral signals, prior enforcement, reputation, age-related risk, and the consequences of a wrong decision.

Those inputs determine the response: automated enforcement where the evidence is clear; friction, deprioritization, or distribution limits where it is less certain; additional signals where a decision isn't yet supportable; and human review where the cost of an error is too high.

Consider what a single high-confidence score can describe: a credible threat, hyperbole between acquaintances, or a survivor recounting their own experience. The score is the same. The appropriate action is not. That is why decisions in child safety, credible threats, exploitation, and self-harm can't yet rest on model confidence alone.

The programs I've seen work best didn't measure success by how much review they eliminated. They looked at decision accuracy, cost per decision, and appeal overturn rate, along with whether reviewer time was concentrated where judgment mattered most.

Where the line between automated and human decisions sits also determines how much harmful material reviewers are exposed to. Designing it well includes limiting that exposure through rotation and support.

AI lets Trust & Safety teams operate at a scale manual review never could. Deciding where automated action ends and human judgment begins is the harder design problem.

My default: where a wrong decision can't be reversed, or a user's safety is at risk, a human owns the call. In those cases, automation prepares the case and a person closes it.

#TrustAndSafety #ContentModeration #ResponsibleAi

You can't moderate your way out of a systems problem

2026-09-18T19:06:56.160Z

One of the biggest mistakes companies make with Trust & Safety is treating it primarily as an operations function.

Hire moderators. Write policies. Build escalation queues. Respond when something goes wrong.

Those things are necessary.

But at scale, you cannot moderate your way out of a systems problem.

Trust & Safety has to become part of the product architecture. That means moving beyond individual moderation decisions and designing systems that understand risk across the entire user journey.

Reputation, account history, age, behavioral signals, content signals, prior enforcement, and appeals should not exist as disconnected workflows. They should inform a broader risk model that helps determine how the platform responds.

The same is true for automation. The challenge is not simply building a classifier or automating more moderation decisions. It is knowing where automation should act, where it should escalate, and where human judgment is worth the cost.

Over the last several years, I’ve worked across moderation operations, policy, child safety, investigations, compliance, and product.

The biggest lesson I’ve taken from that work is that mature Trust & Safety programs are not built around a moderation queue.

They are built around systems.

Reputation systems that change how risk is evaluated. AI-assisted moderation that surfaces content and behavior humans would otherwise never see. OCR, audio transcription, behavioral signals, and content classification working together instead of operating as isolated tools.

Policy translated into product requirements, enforcement logic, and measurable outcomes. And human review reserved for the places where context and judgment actually matter.

Trust & Safety will always require strong operations.

But the future of the field is not simply better moderation.

It is designing products where safety is part of the architecture from the beginning.

That intersection between Trust & Safety, AI, policy, and product is where I’ve spent much of my career, and it’s something I plan to write about more here.