Tech giants unite to fight CSAM with open-source AI

If you are running any online platform where kids hang out, moderation is an absolute nightmare. Smaller companies simply do not have the cash or the AI talent to build cutting-edge detection pipelines.

Google, OpenAI, Roblox, and Discord just teamed up to launch ROOST, short for Robust Open Online Safety Tools. It is a new non-profit backed by more than $27 million in philanthropic funding, designed to operate for at least the next four years.

Former Google CEO Eric Schmidt is also on board as a founding partner. The game plan is straightforward: build and release free, open-source AI tools that can identify, review, and report child sexual abuse material (CSAM).

Pooling tools to plug online blind spots

Right now, safety tech across the web is completely fragmented. Big tech platforms build proprietary defense systems, while smaller forums and indie apps are left largely defenseless.

ROOST plans to unify these fragmented efforts by offering vetted AI training datasets and large language model tools. Roblox's VP of engineering, trust, and safety, Naren Koneru, indicated that ROOST might even host moderation systems that smaller apps can simply call via API.

Each member is contributing its own tech stack. Discord is leaning on the Lantern cross-platform data-sharing project it joined with Meta and Google. Roblox is planning to open-source an updated version of its AI audio-moderation model later this year, targeting voice chat for bullying, racism, and sexting.

That said, clear technical details are still sparse. For instance, ROOST has not explained how its AI tools will interface with industry staples like Microsoft's PhotoDNA.

A preemptive strike against regulators

This coalition is not just doing charity out of goodwill; regulatory heat is turned all the way up.

According to the National Center for Missing and Exploited Children, suspected child exploitation reports jumped 12 percent between 2022 and 2023. Meanwhile, Roblox and Discord have faced brutal public scrutiny and lawsuits over adult predators messaging minors without supervision.

Generative AI has only added fuel to the fire, making synthetic exploitation material easier to create. Joining forces allows these companies to present a united front of self-regulation to lawmakers.

Alongside the announcement, Discord also rolled out a practical new feature called Ignore, which lets users hide messages and notifications from specific accounts without alerting them.

Why it matters

For indie developers, gaming studios, and community builders, ROOST could be a genuine lifesaver. Building contextual, multimodal trust-and-safety tools from scratch is financially out of reach for most early-stage products. Having battle-tested, open-source models or free APIs backed by OpenAI and Google makes baseline safety achievable on day one.

Across the industry, this moves the battle against bad actors from isolated silos to a shared defense layer. If exploitation material gets flagged on Discord, the intelligence and models can quickly protect Roblox or a tiny startup before abuse spreads.

We think open-sourcing these tools is a smart way to level the playing field, but it also carries real risk. Putting model architectures and training methodologies in the open gives bad actors a blueprint to stress-test adversarial attacks and evade filters. How ROOST balances transparency with defensive resilience will decide whether this coalition actually succeeds.