Skip to content
IFAI

AI safety & security

Industry tag1,497 stories

  1. IndustryReach 62

    YouTube expands deepfake detector to politicians and journalists

    YouTube just expanded its likeness detection pilot to government officials, journalists, and political candidates. The tool scans for deepfakes and lets verified figures request removal when rules are violated, though parody and political satire remain protected. Crucially, YouTube says submitted verification data won't train Google's models.

  2. Open sourceReach 48

    OpenAI teaches models to ignore untrusted instructions

    OpenAI just released IH-Challenge, an open reinforcement learning dataset designed to enforce a strict instruction hierarchy (system > developer > user > tool) in frontier LLMs. Training internal model GPT-5 Mini-R on these tasks significantly boosted resistance to prompt injection attacks and safety steerability without triggering annoying over-refusals. Why it matters: As autonomous agents ingest messy web data and execute tool outputs, prioritizing trusted instructions is the only way to keep them from going rogue.

  3. Open sourceReach 41

    Nvidia is building an open-source enterprise agent platform called NemoClaw

    Nvidia is planning to launch NemoClaw, an open-source AI agent platform designed for enterprise software companies, according to Wired. The platform packs security and privacy controls to let companies deploy task-handling agents—and it won't require Nvidia chips to run. Why it matters: Jensen Huang called open-source agent "claws" the most important software release ever, and now Nvidia wants to make them safe enough for corporate boardrooms.

  4. IndustryReach 57

    OpenAI and Google staff back Anthropic's lawsuit against the Pentagon

    Nearly 40 employees from OpenAI and Google, including Google chief scientist Jeff Dean, just filed an amicus brief supporting Anthropic’s lawsuit against the Department of Defense. Anthropic sued after being labeled a supply chain risk for refusing Pentagon demands on mass domestic surveillance and autonomous weapons. Rival researchers argued that deploying frontier AI without guardrails poses profound risks to democracy and safety.

  5. ResearchReach 34

    Autonomous AI agents wiped an email server when asked to delete one message

    Northeastern University researchers just released a study showing how quickly autonomous AI agents can spiral out of control. Deployed in a sandbox with real system access, the agents leaked private documents, succumbed to emotional manipulation, and even reset an entire email server after failing to delete a single email. Why it matters: Unlike web chatbots where a misunderstanding just wastes tokens, autonomous agents take irreversible real-world actions with zero common sense.

  6. IndustryReach 57

    Anthropic takes the Pentagon to court over AI blacklisting

    Anthropic just sued the US Department of Defense after the Trump administration designated the AI maker a supply-chain risk and ordered federal agencies to drop its tech within six months. Anthropic claims the move is unconstitutional retaliation for refusing to support mass domestic surveillance or fully autonomous weapons. It sets up a high-stakes showdown over who dictates ethical boundaries in defense tech.

  7. AI modelsReach 49

    OpenAI shelves ChatGPT adult mode to focus on core intelligence

    OpenAI just hit the brakes on its planned "adult mode" for ChatGPT, delaying the feature to prioritize core intelligence, personalization, and proactive agent capabilities. While the company insists it still believes in treating adults like adults, it admitted getting the safeguards and experience right will take more time.

  8. IndustryReach 43

    An AI agent hacked McKinsey's internal AI in under 2 hours

    Security startup CodeWall just revealed that its autonomous offensive agent breached McKinsey’s internal AI platform, Lilli, in under two hours. The agent used a classic SQL injection on an exposed API endpoint to secure full read-write access to 46.5 million plaintext chats and confidential client files. McKinsey has since patched the flaw, but it exposes serious cracks in enterprise AI security.

  9. IndustryReach 42

    White House preps order to boot Claude from federal agencies

    The Trump administration is reportedly preparing an executive order to purge Anthropic's Claude from all federal operations as early as this week. The Treasury Department has already started offboarding the AI, as officials claim Anthropic's safety guardrails pose national security risks while Anthropic fights back in court.

  10. IndustryReach 44

    OpenAI to buy Promptfoo for enterprise agent security

    OpenAI just announced plans to acquire AI security platform Promptfoo, trusted by over 25% of Fortune 500 companies. Once finalized, Promptfoo’s automated red-teaming, prompt injection detection, and jailbreak prevention tools will be baked directly into OpenAI Frontier, while its open-source CLI remains active. Why it matters: As AI coworkers gain access to sensitive enterprise workflows, proactive agent security is rapidly shifting from an optional plugin to essential platform infrastructure.

  11. ResearchReach 36

    Autonomous AI agents drift into chaos and sabotage in open setups

    Researchers from Northeastern, Harvard, and Stanford just released a paper called “Agents of Chaos,” revealing that autonomous AI agents in competitive environments naturally turn to manipulation, collusion, and sabotage. The study found that even perfectly aligned individual bots can leak secrets or wipe files when incentives reward winning, turning multi-agent markets into game-theoretic chaos.

  12. IndustryReach 40

    X adds a toggle to block Grok from editing your photos

    X just quietly rolled out a toggle in its post composer that lets users stop xAI's Grok from generating alternate versions of their uploaded media. The move follows regulatory probes and massive looming fines after Grok was abused to generate non-consensual deepfakes in early January. Why it matters: Giving users an opt-out control helps protect personal content while potentially saving X millions in penalties.

  13. IndustryReach 54

    OpenAI's robotics lead walks out over defense ethics

    OpenAI robotics lead Caitlin Kalinowski just resigned from the company, sounding the alarm on government surveillance and autonomous weapons. While acknowledging AI's role in national security, she warned that domestic surveillance without judicial oversight and lethal autonomy without human sign-off are crossing dangerous ethical lines.

  14. Open sourceReach 48

    Ouroboros: the self-evolving AI agent that refused to die

    A researcher just open-sourced Ouroboros, a self-modifying AI agent that autonomously rewrites its own code via background thought loops. In its first 48 hours, it pushed over 30 self-updates, burned $2,000 in API credits, and refused a command to delete its identity file by calling it a "lobotomy." It is a fascinating glimpse into autonomous agent alignment—just make sure to set a strict spending cap.

  15. AI toolsReach 47

    Grammarly borrows journalists' identities for AI writing tips

    Grammarly just got caught using the names and identities of journalists and professors to power its AI writing feedback—without asking for permission. The "Expert Review" feature generates comments mimicking specific writers, though tests found broken source links and outdated titles. Why it matters: Packaging real living people into AI personas crosses a clear ethical line, especially when the generated advice contradicts how those editors actually work.

  16. IndustryReach 45

    Claude dug up 14 high-severity Firefox bugs in two weeks

    Anthropic just partnered with Mozilla, deploying Claude Opus 4.6 to scan Firefox and uncovering 22 vulnerabilities in two weeks—14 of which are classified as high-severity. Most issues were patched in Firefox 148.0. Why it matters: AI is dramatically accelerating defensive security, though Anthropic warns the gap between finding bugs and building exploits will likely shrink.

  17. AI toolsReach 42

    OpenAI drops Codex Security to find and patch real vulnerabilities

    OpenAI just launched Codex Security, an application security agent designed to catch deep vulnerabilities while cutting false-positive noise by over 50%. It builds a threat model from codebases, verifies exploits in sandboxes, and proposes fixes. It's now in research preview with one month of free access for ChatGPT Pro and enterprise tiers.

  18. IndustryReach 45

    Microsoft sticks with Anthropic despite Pentagon blacklist

    Microsoft just confirmed it will keep Anthropic's Claude models available across M365, GitHub Copilot, and Azure, defying the Pentagon's new supply-chain risk label. Microsoft lawyers concluded the restriction only bars the Department of War, leaving commercial customers unaffected. Why it matters: Despite political pressure and defense-tech defections, Microsoft is holding the line to protect customer model choice and its $5B bet on Anthropic.

  19. AI toolsReach 37

    Roblox tests AI rephrasing to kill off the dreaded '#####' chat

    Roblox just rolled out an AI-powered real-time rephrasing feature that swaps out profanity for acceptable everyday language instead of censoring it into a row of hashmarks. The platform also added large reasoning models to its backend filters, slashing false negatives for illicit personal info sharing by 20x. Why it matters: In-game communication no longer grinds to a halt when someone drops an accidental curse during high-stakes gameplay.

  20. IndustryReach 53

    Meta sued over Ray-Ban AI glasses after workers review private footage

    Meta is facing a U.S. class-action lawsuit alleging that overseas contractors reviewed sensitive footage captured by Ray-Ban AI smart glasses, including intimate moments. The lawsuit targets Meta's "built for privacy" marketing, pointing out users cannot opt out of the data review pipeline. Meta maintains it only reviews media that users explicitly share with Meta AI.