OpenAI wants AI to predict human morality

Giving an algorithm the authority to predict ethical decisions sounds like the opening scene of a sci-fi dystopia. Yet, OpenAI is putting real cash behind it.

According to an IRS filing, OpenAI Inc., the company’s nonprofit arm, awarded a grant to Duke University researchers for a project titled "Research AI Morality." An OpenAI spokesperson confirmed the funding is part of a larger three-year, $1 million initiative aimed at "making moral AI," scheduled to wrap up in 2025.

A moral GPS for doctors and lawyers

Details remain tight-lipped. The study's lead investigator, Duke practical ethics professor Walter Sinnott-Armstrong, declined to discuss the specifics. But his past work speaks volumes.

Alongside co-investigator Jana Borg, Sinnott-Armstrong previously authored a book exploring how AI could serve as a "moral GPS" to help humans make tougher, fairer calls. In prior research, their teams developed a "morally aligned" algorithm to decide kidney donation recipients and studied which dilemmas people would actually prefer an AI to resolve.

Per the release, this OpenAI-backed project aims to train algorithms to "predict human moral judgements" in complex scenarios spanning medicine, law, and business.

The ghost of Ask Delphi

Teaching a probability machine to understand ethics is historically treacherous territory.

Back in 2021, the Allen Institute for AI rolled out Ask Delphi, an ethics bot designed to provide moral guidance. While Delphi handled simple baselines like flagging exam cheating, clever rephrasing easily manipulated it into approving horrifying acts, including smothering infants. It also baked in glaring demographic biases, such as rating heterosexuality as more "morally acceptable" than being gay.

Modern large language models learn word associations from web text, not ethical reasoning. Even top models carry subtle, unstated philosophies: Claude often leans Kantian by prioritizing rigid duties, while ChatGPT skews slightly utilitarian by weighing aggregate outcomes. Predicting human morality when humans cannot agree on moral philosophy is an exceptionally tall order.

Why it matters

As AI shifts from generating text to executing autonomous actions, moral reasoning becomes an unavoidable bottleneck. If agents are soon drafting legal settlements or triaging clinical workflows, subtle ethical biases will have real-world consequences.

We think OpenAI’s move to fund outside academics is a calculated gamble. By backing Duke’s team, OpenAI gets frontier insights into high-stakes alignment while keeping controversial "moral machine" experiments at an arm's length from its commercial core.

However, the concept of a "moral GPS" carries severe pitfalls. The web heavily overrepresents Western, educated views, and treating majority statistical consensus as moral truth risks codifying systematic bias into high-stakes decisions like healthcare. Predicting what humans might judge is one thing; relying on an algorithm to arbitrate morality is quite another.