OpenAI keeps bulldozing mathematicians | The Verge

In a chaotic few months, OpenAI has demonstrated it can do two things with remarkable consistency: make impressive breakthroughs in mathematics, then colossally screw up announcing them. OpenAI is now trying to do better. Somehow, it has botched that too.

OpenAI’s latest attempt to repair fractured relations with a mathematical community it has repeatedly alienated is to consult a new independent advisory group of elite practitioners. But mathematicians speaking to The Verge, including one of the group’s members, describe a messy and confusing affair bearing many of the hallmarks of OpenAI’s previous rushed forays into mathematics, suggesting the company has learned little from its mistakes. And then there’s the daunting task the group has been handed first: helping coordinate the release of scores more results OpenAI says its unreleased model has produced, the prospect of which is already stirring dread among researchers over what this looming tidal wave of breakthroughs could do to their field.

On September 21st, an assortment of eminent mathematicians announced the formation of the Advisory Group on Mathematics and Artificial Intelligence (AGMAI) in a guest post on the blog of UCLA mathematics professor Terence Tao, a Fields Medalist and outspoken critic of AI companies’ conduct in mathematics. The group was to be an independent body of nine elite mathematicians at the very top of their field. The post said the group will operate independently from OpenAI and advise it and other frontier AI labs “on the review and communication of emerging results.” In its own announcement, OpenAI repeatedly stressed the group’s independence, saying members would be free to challenge the company publicly, publish their advice, and offer guidance it had not requested. It also stressed the limits of the group’s influence, noting that it “will not be responsible for advising us on how to pace our internal progress on mathematics.”

But many mathematicians, and certainly most people outside of that community, learned about that group only through OpenAI’s much louder rollout.

The result was widespread confusion over whether AGMAI was truly independent or if it was, as several mathematicians The Verge spoke to in the days after the announcement assumed, some kind of OpenAI-appointed body. That impression wasn’t entirely unreasonable, given AGMAI’s website says it formed after OpenAI approached some of its eventual members about establishing an advisory board, before they decided to strike out independently and invite others to join. It remains unclear which of the nine members OpenAI initially approached.

Speaking to The Verge on a video call from a lecture theater, Martin Hairer, an AGMAI member and mathematics professor at Imperial College London and the Swiss Federal Institute of Technology in Lausanne (EPFL), was keen to stress the group’s independence from OpenAI. While acknowledging that OpenAI approaching some mathematicians was the original impetus for its formation, he said the group receives no financial, technical, or other support from the company, and that none of its members have signed agreements restricting what they can say or do beyond standard confidentiality requirements needed to give them advanced access to research.

It’s clearly been a hectic time for Hairer, who described the preceding few days as a “very frustrating” and “intense” experience. “We don’t work for OpenAI, are not paid by them, and it’s totally independent,” he said. Hairer said the group is equally open to working with other frontier AI labs and had already begun conversations with some, though he declined to say which.

Nevertheless, Hairer seemed exasperated by the way OpenAI had announced the relationship. In a blog post published amid the ensuing uproar, Hairer acknowledged it would be naive to think AI companies wouldn’t try to spin the group’s involvement to their advantage. Given the group had emerged partly in light of OpenAI’s atrocious handling of previous mathematical breakthroughs, that risk was front and center. OpenAI’s solution to a prestigious Millennium Prize problem rapidly descended into fights over credit, scooping, and its treatment of mathematicians, so the community was naturally suspicious.

Hairer acknowledged that OpenAI’s announcement had done little to help the fledgling group establish its independence. AGMAI needs to win the trust of mathematicians in order to help them navigate an increasingly fraught relationship with AI companies. “The way they phrased their announcement didn’t really help,” Hairer said. “If you read it exactly in detail, there’s nothing wrong in what they say,” he said, noting that the company is “careful” in its wording. “They obviously are very good PR people,” he said, laughing.

The whole endeavor — the announcement, the group’s formation, its communications with the mathematical community — feels as if it came together in a hurry. Asked last Wednesday about AGMAI’s plans, Hairer laughed and said, “So we, well, you know, started two days ago. So it’s not like we have a big master plan.”

That urgency might be down to the fact that OpenAI is sitting on a tranche of results it is clearly eager to release as soon as possible. Since solving the Navier-Stokes Millennium Prize problem, OpenAI says its unreleased internal “model has now resolved more than 100 long-standing open problems across most areas of mathematics.”

Avoiding another PR disaster is why OpenAI went to the trouble of trying to assemble a team of some of the world’s best mathematicians to consult. But trumpeting the sheer number of results it has waiting in the wings while promising to handle them responsibly only underscores how much of a mathematical outsider OpenAI is.

“This is not how any academic behaves,” said Álvaro Lozano-Robledo, a professor of mathematics at University of Connecticut. “I don’t go around saying, like, ‘Oh, I’ve proved all these things, but I don’t know what to do with them.’”

He sees a similar disconnect in other OpenAI pronouncements, such as when OpenAI’s Laurance Fauconnet told The Verge the company had “made substantial progress” on another Millennium Prize problem. “No mathematician would go out and say that,” Lozano-Robledo said. “You either have solved it, or you’re still trying.”

Lozano-Robledo said he hopes engaging with AGMAI is a serious attempt by OpenAI to address mathematicians’ concerns, but stressed that doing so means it should be a responsible member of the research community. That requires more than simply dropping results into the world and moving on, as many mathematicians believe it has done with its most recent findings. Researchers The Verge spoke to complained of poorly written manuscripts and scant engagement with the literature surrounding a finding, all of which makes it difficult to understand a result’s significance and leaves key work without the appropriate credit. At times, the company had quietly changed documents to shore up shortcomings after they were released, but didn’t announce changes or leave a clear record of what was altered. The Verge noticed this when the company announced “Ten advances in mathematics and theoretical computer science” in early August, and Hairer separately described times when it felt like the company altered manuscripts “sneakily” in response to criticism. “That also makes people paranoid, right?” Hairer said, describing it as “shoddy” and “really bad and sloppy scholarship.”

As with AI slop elsewhere, the people creating it are rarely the ones paying the costs for dealing with it. Lozano-Robledo described the deluge of AI-generated math solutions as a “burden” on the community — one he thinks AI companies fundamentally misunderstand. “The burden is that they are producing a solution,” he said, but solutions are often less important than the understanding that comes with them. In human mathematics, the two have traditionally gone hand in hand, and AI seems to be changing that.

AI companies may be producing impressive results, but they still need human mathematicians to work out how important a result actually is. “They need our expertise,” Lozano-Robledo said. “They need us to celebrate that solution.”

For other mathematicians The Verge spoke to, that burden is much more personal. For weeks now, OpenAI has been hinting at another Millennium Prize problem and says it has more than 100 results to apparently major problems ready to go, while revealing little about what any of these actually are. After the upheaval surrounding past announcements, Navier-Stokes in particular, that uncertainty has created a tense atmosphere. Many researchers expressed fear that a problem or area of work to which they have devoted years, even decades, of their professional lives could be next on the chopping block, abruptly dispatched by a company they suspect is doing it as a publicity stunt on the way to an IPO.

“They are just feeding the paranoia of what they could possibly have proved,” said Lozano-Robledo, accusing the company of “feeding the frenzy” and increasing dread among many mathematicians that their area of work could be disrupted next. “Like, what other Millennium problem are they talking about when they say ‘We have almost solved another Millennium problem’? That sentence makes no sense in mathematics.” The researcher stressed that such grandstanding is simply not how mathematics is usually done.

For Colva Roney-Dougal, a mathematics professor at the University of St Andrews in Scotland, the uncertainty surrounding OpenAI is already having a profound effect on how she thinks about her work. “It’s somewhat agonizing to know that a ‘large number’ of results are likely to be announced soon,” she said. “The more there are, the more likely it becomes that my ongoing work, or that of my students, suddenly becomes irrelevant.”

The rapid pace of development has left Roney-Dougal at a peculiar sort of scholarly crossroads: “I am therefore unclear whether I should be trying to rush out as many papers as possible, or passively waiting to see what these results are, or carrying on as normal,” she said.

Hairer is well aware of this kind of anxiety. He said the potential disruption to researchers was one of the reasons why he added his name to a roster of Fields Medalists trying to address what they described as the “severely misaligned” goals of AI companies and the mathematical community. He is also aware that OpenAI may spin the involvement of him and other prominent AGMAI members to its advantage, particularly with the general public.

Let them, he said. “They’re not going to sort of damage me within the math community,” Hairer said. “In some sense, I don’t really care about what they say. But what I do worry about is the math community as a whole.”