Tech things: OpenAI is Still an Unaligned Agent (?)
A walk through memory lane as we dissect OpenAI's history of weird corporate structures
One of my favorite kinds of Tech Things posts is making fun of OpenAI's attempts to get out of its weird corporate structure. I have done this four times now. Well, OpenAI is back in the news again, so here's another one.
For folks who aren't quite up to speed, we'll start with a quick recap. OpenAI originally started in 2015 with the lofty goal of making the world's first super intelligence, and doing so in a way that prioritizes safety and transparency for the betterment of humanity. Altman, Brockman, Sutskever, and other members of the initial team worked hard to make sure that OpenAI would never be corrupted from this mission. One way to do that was to structure OpenAI as a non-profit.
Non-profits have been in the news recently, in part because Trump and team MAGA seem to think that they are hell pits of corruption and evil (unlike the much more clean and virtuous $Trump coin). As a result you may have a vague sense that non-profits are sorta sketchy. But basically every charity in the world is structured as a non-profit. The reason is straightforward. Most of US corporate law is structured around a simple premise: the organization has a fiduciary duty to its shareholders. That is the business can incur legal liability if it is doing things that are demonstrably bad for the financial outcome of the shareholder; it's capitalism baked into our legal code. By contrast, the non-profit designation shields the organization from that liability. There is no requirement for a non-profit to make money. But in exchange, any money that the non-profit makes must go back into the non-profit.
Unsafe, non-transparent AI is, like, pretty obviously more financially lucrative than safe, transparent AI. For one, "transparent" is another way of saying "competitors can steal your work", which is rarely good for making large buckets of cash. And why try to prioritize 'safety' when you can just externalize those losses on society? By making OpenAI a non-profit, Sam and the board were trying to ensure that they would get rid of that financial incentive, forever. Anyone giving money to OpenAI wasn't an investor, they were donors. And you can always tell a donor to go away if they try to use their money as leverage to do things.
Fast forward a few years. In 2019, it became obvious1 that OpenAI needed to raise a lot of money to compete with other players in the space and really unlock AGI. But who was going to give OpenAI lots of money? The rest of the world is still mostly capitalist! Anyone who is going to plop down a few hundred billion dollars is going to want to see some return on that capital! This led to a 'deal with the devil'. Sam convinced the board to spin off a "capped for-profit" arm of OpenAI. The basic idea was that anyone who invested in OpenAI would be able to get returns up to a certain 'profit cap'; but the whole thing was still (on paper) controlled by the non-profit board.
This is…a weird structure. It's an accounting nightmare. You get no obvious influence over the direction of the company. And you have a hard cap on how much upside you can get. It's hard to imagine anyone really springing for this deal unless the investment was too good to pass up. But, well, it's OpenAI. Microsoft took the deal. This pissed a lot of people off, and a solid chunk of OpenAI left to go found Anthropic in 2021.
Fast forward a few more years. In 2023, the non-profit board looked around and saw an unrecognizable company. OpenAI was off the rails, Sam had dollar signs in his eyes (and, it was later revealed, was lying to the board constantly), and other executives (Murati, Sutskever) were deeply concerned. And so the non-profit board did the only thing a board is actually allowed to do: it fired the CEO.
Sam Altman's firing was probably the most exciting and drama filled week Silicon Valley has had in the last 15 years. I mean, everyone was talking about it, and everyone was trying to get information about what was going on.
But, jokes aside, Matt Levine had my favorite reporting, in large part because he managed to cut directly to the important question at hand:
In the first diagram, the word “controls” appears four times, and if you trace it through, you will see that the board of directors of OpenAI ultimately controls each entity in the organization. All of OpenAI answers to its ultimate decision-making body, an independent nonprofit board of directors who do not own any equity in the OpenAI entities and who, broadly speaking, appoint themselves. They answer to their own consciences, not to any investors. “The Nonprofit’s principal beneficiary is humanity, not OpenAI investors,” explains OpenAI.
In the second diagram, I have written the word “MONEY” in large green letters.
The question is: Is control of OpenAI indicated by the word “controls,” or by the word “MONEY”?
Money! Obviously money!
Even though OpenAI’s non-profit structure theoretically insulates it from financial concerns and fiduciary duties, the funny thing about money is that you need it to run things. It keeps the server racks on. It pays the office rent. It buys the team sandwiches for lunch. So even though there is a legal sense in which OpenAI is not beholden to its financiers, there is a much more real sense by which sources of capital will always be able to exert some kind of pressure on even the most principled of non-profits.
While we'll never really be sure what exactly happened in those 48 hours when the board fired Sam, one thing is clear: by the time the smoke cleared, Sam was still standing and the non-profit board was, if not gone, significantly weaker. From here, the inevitable outcome was inevitable. OpenAI announced in 2024 that it was restructuring to remove the non-profit board entirely. It shut down its AI safety arm, leading to another exodus of safety-oriented employees. And it took money from SoftBank, conditioned on becoming a for-profit by the end of 2025.
What does all this have to do with AI alignment? Well, OpenAI is behaving exactly like an unaligned agent. Quoting from my first post on the subject, aptly named "OpenAI is an Unaligned Agent":
I don't personally take a side on whether OpenAI should or shouldn't be for profit or not. But I do think this is a fantastic and ironic example of exactly the kinds of things AI alignment researchers are so morose about. When OpenAI was created there were all these clever guard rails set up to make sure it would always 'be aligned' and 'do the right thing'. There's a controlling non profit board! There's a super alignment team! There's a ton of brand risk of people making fun of you and your company with snarky names like NopenAI or ClosedAI if you ever go back on your founding mission! Surely the organization will remain focused on safety over profit? And yet one by one, those guard rails have fallen. I think if you asked 2015 Sam Altman about why OpenAI isn't for profit, he might say something like:
"Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return."
But OpenAI-the-entity was too smart for 2015 Sam Altman. It managed to break free of the best shackles that humanity could possibly build at the time.
You may think I get a lot of joy out of these posts, and in some sense I do — sardonic writing is probably even more fun to write than it is to read, and it's just so easy to poke fun at OpenAI. But honestly, I write these OpenAI posts as a coping mechanism, a sort of morbid dark-humor attempt to make light of the fact that super intelligent AI is 99% going to be unaligned and we will not be able to constrain it. Back in 2015 Sam Altman saw the future. He knew that he would be tempted by boatloads of cash, and tried to create a system where no matter what happened, he would be able to avoid the siren’s call. A week ago, it seemed like he failed. If humanity can’t figure out how to restrain organizations run by people that we can talk to and empathize with, how could we possibly restrain a completely alien super-intelligence?
But perhaps I was too quick to impose judgement.
The news of the day is that despite its best efforts, OpenAI is struggling to transition away from the non-profit structure. It turns out that if you promise a whole bunch of people that you will do things for charitable purposes, and then build a bunch of valuable things ostensibly for charitable purposes, you can't just, like, sell all those valuable things and pocket the cash. Your original donors might get really ticked off, and may start throwing around words like 'fraud' and 'breach of contract'. And when your original mega-donor is Elon Musk, a man who is fond of ridiculous lawsuits and has money to spare, well, the writing is just on the wall isn’t it?
Elon Musk is asking a federal court to stop OpenAI from converting into a fully for-profit business.
Attorneys representing Musk, his AI startup xAI, and former OpenAI board member Shivon Zilis filed for a preliminary injunction against OpenAI on Friday. The injunction would also stop OpenAI from allegedly requiring its investors to refrain from funding competitors, including xAI and others.
The latest court filings represent an escalation in the legal feud between Musk, OpenAI and its CEO Sam Altman, as well as other long-involved parties and backers including tech investor Reid Hoffman and Microsoft
…
In their motion for preliminary injunction, attorneys for Musk argue that OpenAI should be prohibited from “benefitting from wrongfully obtained competitively sensitive information or coordination via the Microsoft-OpenAI board interlocks.”
“Elon’s fourth attempt, which again recycles the same baseless complaints, continues to be utterly without merit,” an OpenAI spokesperson said in a statement.
I mean, the OpenAI spokesperson is in some sense obviously right. Elon doesn’t have principles, are you kidding? Have you seen what he’s been up to since he bought Twitter? No, Elon is obviously doing this to put a stick in Altman's eye and screw with a competitor. But also, as much as I hate to say it, Elon clearly does have a point — enough that a few AGs in a few states have indicated that they would not look at OpenAI favorably.
So we get to the latest entry in OpenAI's ongoing saga to try and rid itself of its creator's chains. From the OpenAI blog:
We made the decision for the nonprofit to stay in control after hearing from civic leaders and having discussions with the offices of the Attorneys General of California and Delaware. We look forward to advancing the details of this plan in continued conversation with them, Microsoft, and our newly appointed nonprofit commissioners.
OpenAI was founded as a nonprofit, is today a nonprofit that oversees and controls the for-profit, and going forward will remain a nonprofit that oversees and controls the for-profit. That will not change.
The for-profit LLC under the nonprofit will transition to a Public Benefit Corporation (PBC) with the same mission. PBCs have become the standard for-profit structure for other AGI labs like Anthropic and X.ai, as well as many purpose driven companies like Patagonia. We think it makes sense for us, too.
Instead of our current complex capped-profit structure—which made sense when it looked like there might be one dominant AGI effort but doesn’t in a world of many great AGI companies—we are moving to a normal capital structure where everyone has stock. This is not a sale, but a change of structure to something simpler.
I wonder how those discussions with the AGs of California and Delaware went.
Reading between the lines, this was definitely a setback for OpenAI. The LLM race requires compute; OpenAI needs money for compute; that money is tied to SoftBank's StarGate project; and SoftBank is demanding OpenAI become a for-profit. I strongly suspect that this maneuver is enough to unlock SoftBank's funds (or why go through the trouble?) but it is probably not exactly the outcome that either Sam or his investors really wanted. The non-profit remains in control; the alignment worked!
…Right?
The pessimist in me says that all of this is still just on paper. In practice, OpenAI managed to neuter the controls set on it the moment it lost its independent board. I mean, yes, in theory, the non-profit still legally has control over the company through its leverage over Sam. But what are they going to do, fire him again? Sam already got rid of the people most likely to do that! For all practical purposes, the board is controlled by Sam, and Sam is clearly in profit mode, so the company is in profit mode and all the paperwork is just a distraction.
Right now, the non-profit likely has super-majority voting rights and a minority of the regular shares. It would not surprise me at all if, over the next months, we discover that the non-profit's voting rights have been diluted. At that point, OpenAI will have finally completed the transformation it started nearly 6 years ago. As an analogy for super intelligent AI agents, OpenAI's restructuring does not exactly bode well. So I am still more or less hoping that the legal structure holds and they never realize their goal of becoming a complete for-profit — just to show that when we really do put our smartest minds to the task of reigning in the incomprehensible optimization machine, we can actually stop it before it gets to escape velocity.
At least, internally. I think this was around the time that Ilya realized that transformers could scale, a lot. GPT-1 came out in 2018 and GPT-2 in 2019. To 10x GPT-2, OpenAI's chief scientist simply needed more chips than what they had. I don't think it was as obvious to the rest of the world that GPT-3 would work. Frankly I'm not sure even Ilya knew that it would work. But that was the trajectory they were on, and to continue in that direction they needed $$$.