OpenAI researchers warned board of AI advance before CEO ouster, sources say

Nov. 22 (Reuters) – In front of the CEO of OpenAI Sam Altman’s four days in exileseveral staff researchers wrote a letter to the board warning of a powerful artificial intelligence discovery that they say could threaten humanity, two people familiar with the matter told Reuters.

Both sources said the unreported letter and the AI ​​algorithm were key developments before the board’s ouster of Altman, the poster child for generative AI. Before his triumphant return As of Tuesday afternoon, more than 700 employees had threatened to quit and join sponsor Microsoft (MSFT.O) in solidarity with their sacked leader.

Sources cited the letter as one factor in a longer list of board grievances that led to Altman’s firing, including concerns about marketing advances before understanding the consequences. Reuters was unable to review a copy of the letter. The staff who wrote the letter did not respond to requests for comment.

After being contacted by Reuters, OpenAI, which declined to comment, acknowledged in an internal message to employees a project called Q* and a letter to the board before the weekend’s events, one of the people said. An OpenAI spokesman said the message, sent by longtime executive Mira Murati, alerted staff to certain media stories without commenting on their accuracy.

Some at OpenAI believe Q* (pronounced Q-Star) could be a breakthrough in the startup’s pursuit of what’s known as artificial general intelligence (AGI), one of the people told Reuters. OpenAI defines AGI as autonomous systems that outperform humans in the most economically valuable tasks.

Given the massive computing resources, the new model was able to solve certain mathematical problems, the person said on condition of anonymity because the individual was not authorized to speak for the company. Although they were only performing math at the level of elementary school students, performing these tests made the researchers very optimistic about Q*’s future success, the source said.

Reuters was unable to independently verify Q*’s capabilities claimed by the researchers.

“VEIL OF IGNORANCE”

Researchers see mathematics as a frontier of generative AI development. Currently, generative AI is good at typing and translating language by statistically predicting the next word, and answers to the same question can vary widely. But conquering the ability to do math, where there is only one right answer, implies that AI would have greater reasoning abilities that resemble human intelligence. This could be applied to new scientific research, for example, AI researchers believe.

Unlike a calculator that can solve a limited number of operations, AGI can generalize, learn and understand.

In their letter to the board, the researchers noted the AI’s prowess and potential danger, the sources said without specifying the exact security concerns noted in the letter. There has long been a debate among computer scientists about the danger posed by highly intelligent machines, for example, if they might decide that the destruction of humanity was in their best interest.

The researchers also flagged the work of a team of “AI scientists,” the existence of which was confirmed by multiple sources. The group, made up of the combination of previous “Code Gen” and “Math Gen” teams, was exploring how to optimize existing AI models to improve their reasoning and ultimately perform scientific work, one of the people said .

Altman led efforts to make ChatGPT one of the fastest-growing software applications in history and attracted the necessary investment and IT resources from Microsoft to approach AGI.

In addition to announcing a slew of new tools at a rally this month, Altman told a summit of world leaders in San Francisco last week that he believed major breakthroughs were in sight.

“Now four times in the history of OpenAI, the most recent being in the last two weeks, I’ve gotten to be in the room, when we pull back the veil of ignorance and the frontier of discovery forward, and achieve- it is the professional honor of a lifetime,” he told the Asia-Pacific Economic Cooperation summit.

A day later, the board fired Altman.

Our standards: The Thomson Reuters Trust Principles.

Anna Tong is a Reuters correspondent based in San Francisco, where she reports on the technology industry. She joined Reuters in 2023 after working at the San Francisco Standard as a data editor. Tong previously worked at tech startups as a product manager and at Google, where she worked on user insights and helped manage a call center. Tong graduated from Harvard University. Contact: 4152373211

Jeffrey Dastin is a Reuters correspondent based in San Francisco, where he reports on the technology industry and artificial intelligence. He joined Reuters in 2014, originally writing about airlines and travel from the New York bureau. Dustin graduated from Yale University with a BA in History. He was part of a team that examined Amazon.com’s lobbying worldwide, for which he won a 2022 SOPA award.

Krystal reports on venture capital and startups for Reuters. Covers Silicon Valley and beyond through the lens of money and characters, focusing on growth-stage startups, tech investments and AI. He has previously covered M&A for Reuters, breaking stories on Trump’s SPAC and Elon Musk’s Twitter funding. He previously reported on Amazon for Yahoo Finance, and lawmakers in Congress cited his investigation into the company’s retail practices. Krystal began a journalism career writing about technology and politics in China. He has a master’s degree from New York University and enjoys a scoop of Matcha ice cream as much as a scoop at work.