The WSJ recently reported that AI has become the biggest economic bet in US history in terms of the relevant percent of GDP being spent during the scale up of the technology. That is wild. Trillions of dollars are being spent on AI, between model development, data center build out, and investments into AI companies/products. The tech isn’t even viably profitable yet and we’ve spent more on it than any previous national scale up of new technology—including railroads, telecoms, even the internet itself.
This WSJ article caught my eye because the cost of AI are a current interest of mine. The money aspect is only a part of it—the most obvious and measurable cost. Far more intriguing to me are the harder to see and quantify costs. I was actively drafting this piece when I came across the WSJ article, and I took note because the scale of the financial costs are in line with the broader costs of the technology. AI, as it exists in 2026 is—so far as I understand it—one of the costliest new technologies we’ve ever created.
This post is about the non-financial costs.
If you line the costs all up, it becomes rather clear that the people pushing for the development and adoption of AI strongly believe that the development of AI is more important than the survival and flourishing of humanity. I believe that price is more than we might be willing to pay?
Purely as a technology, AI—which is the general term currently being applied to LLM-based computer systems—is fine. I do not think LLMs are magical, I do not think they are god-lite or even god-like, but I also don’t think they are utter shit. They are fine. They can do some nifty things. A nifty new technology that improves the ways computers work, but that is also currently in the middle of a vastly overblown hype cycle.
Despite the nifty new possibilities of this tech I currently believe the costs of AI adoption far outweigh the benefits. Because of that, I am ethically and philosophically opposed to the current buildout and adoption of AI. I think the current ways AI is being developed, built, scaled, rolled out, and adopted are all fucked up, and I don’t think there are realistic reform-based paths to improving things, because some of the fucked-up-ness is inherent to the technology. I also see that the AI era is being shaped by other systems that are currently fucked up. and unless we can unfuck those broader systems, I have little hope that we can regulate the AI companies in ways that require them to innovate past the inherent fucked-upness of LLMs. Without addressing these systemic challenges and technological harms, I just do not see any possible way that this technology can make humanity’s future better—which is what I’ve been led to believe is the primary reason we’re rushing to adopt it.
In the course of learning about LLMS/I, I have used them (in fact I am required to do so by my employer), so I understand how they work, and I do think there is some value to them (I am going to use the terms LLMs/AI interchangeably in this post). Specifically, they do seem like a really nifty new interface for the computer. Prior to LLMs, computers were extremely literal. Computers would only do what you told them to do and what they were capable of doing was limited by the work that others had done to build software. So if you wanted a computer to do a task, you had to find software that could accomplish the task, obtain the software, learn the interfaces available, and then use that interface to make the software do the task.
If, in the process of all those steps you fucked up—if you tried to make it do a thing that it couldn’t do, or you tried to do it in a way that wasn’t quite possible, you were at a dead-end. Computers have always been incapable of understanding what you intended. They just executed the instructions given, and made it your problem if the executed instructions did not match the desired intentions.
LLMs expand our interface options with the computer because suddenly computers are much better at translating plain language instructions into computational tasks. Finally it is possible to get the computer to do what you intend rather than what you literally said. That’s not nothing; it is in fact something that has eluded us for the entirety of computing history! It’s very nifty and clearly changes how we think about what computers are capable of.
I believe all of the products and services built around LLMs could be seen as learning how to manage this new interface; to find safe ways to translate human language into computer instructions that succeed. The RAG systems provide on-demand information, the skills provide repeatable instructions to execute ambiguous prompts, the harnesses continually improve to address past failure scenarios, so on and so forth. LLMs are powerful computing interfaces for taking the infinite variety of human language and mapping inputs to expected language outputs. Tech companies have spent the last 4 years deploying immense amounts of money, labor, marketing, and experimentation to determine what is possible with such an interface.
The scale of investment is because LLMs are not actually genies with all human knowledge available on demand. They are statistical mappings of all the tokens they were trained on, refined through various training mechanisms (including thousands of humans literally ranking millions of outputs as good or bad). It turns out that if you have a significant enough amount of computing power, energy to power it, as much of the digital data in existence that you can steal or otherwise collect, a bunch of committed nerds, tens of thousands of gig workers, and the billions of dollars necessary to pay all the bills for these things, you can build such a large statistical map of language that the computer can do freeform autocompletion on just about any prompt you give it. Again pretty nifty!
So I do see that there is beneficial value in these large language models; using plain language to control the computer really does expose the possibilities of software and digital technology to a much broader set of humanity than before. That’s cool, and would be pretty exciting to explore if we could afford to explore those possibilities.
Alas, I don’t see our systems of power actually grappling with the costs of AI, in plain language. It’s hard to get your head around the scale of the harms AI is actively creating, which (conveniently for the AI boosters) makes it difficult to weigh the costs against the benefits. This analysis is what I’ve been mildly obsessed with in the past few years: can we describe the costs/harms of AI adoption clearly? Here’s my best effort:
The way that AI technology is being built and deployed in the US is actively costing us:
- Rapid exacerbation of the harms from climate change,
- Rapid increases in economic precarity for most of the middle-class, and
- Rapid introduction of more chaos into institutions and systems that are already destabilized
All of this for the low, low financial cost of more money than we’ve ever spent on any new technology,
Meanwhile, those very same institutions and systems—government, regulatory bodies, media, academia—are unable to address the scale of harms at the necessary preventative speed.
Add on to this that we do not yet know how to mitigate the harmful parts of the technology while preserving the good parts because we do not currently know to build “safe” AI, and, well. Sum all this together and I believe a reasonable conclusion is:
AI costs us far more than the value it might provide us.
A Note on the Rapidity of it all
Since the industrial revolution, the industrialized world has grown more and more ambitious with how fast new technologies are adopted and scaled. Think of all the technologies that have been invented, incubated, and then eventually deployed at larger scales. There’s models of production—factories and industrial agriculture. Those systems were shaped in part by technologies of delivery—horses and carts, to ships, to railroads, to planes and cars. Or consider energy—electricity took a long time to scale out around the world; we moved from hydropower to fossil fuels to nuclear then started adding in solar and wind. Technologies of communication follow similar trajectories—from telegraphs to phone lines to cellular to now the age of the internet, where we find software has eaten the world.
In each of these examples there is 1) a material history of inventing the technology, 2) learning how to build it in the real world, then 3) scaling the technology up as it is adopted across more and more of the world. This path of invention to early deployment to large scale deployment to potentially global deployment is a path of learning. Take any of the examples above, and you will find that the way the technology was understood and shaped at the point of invention might be quite different from the way it is understood and shaped at global scales.
The time it took for technology to scale was a generative period where we learned how to improve the technology, regulate the technology, and address harms as they revealed themselves. Because new technologies are systems, new technologies create emergent behaviors and possibilities just like all systems do—behaviors and possibilities that we couldn’t necessarily have predicted in advance, but once known, might require us to change other parts of the world in order to continue building out or adopting the technology safely and reliably.
Emergence is when a system creates possibilities beyond what the individual components contain. Imagine a board and two logs. Individually, you have 3 pieces of wood, but if you lay the board across the logs what emerges is a bench; a place for multiple people to sit. A bench is, in this contrived example, the emergent possibility of the relationship between the 3 pieces of wood. New technologies work like this at vast scale—when deployed in the real world they interact with other systems, and from those new relationships new possibilities emerge. What emerges may be desirable or not.
Neil Postman tells the story in Amusing Ourselves To Death how the development of the railroad in the US had the unanticipated consequence of standardizing time across the US, because railroad timetables demanded a consistency of clocks between stations. Prior to the railroads, there was no guarantee that noon in one city correlated to noon in another city, nor was there any real need for the two to align. In order for railroad timetables to mean anything, however the time in one city needed to be (and stay) in sync with the other cities on the railroad. That allowed a railroad to say that a train leaving city A at 11:15am will travel for 45 minutes, and arrive at noon in city B. Without stable, synchronized clock time, train timetables were rather meaningless. Today, we can’t imagine a world where time isn’t standardized and clocks aren’t in sync but it turns out there was a period before that reality existed, and the shift emerged because new technology demanded it.
So. New technologies change the world, and it can take time to understand what those changes are, and to evaluate if the necessary changes are desirable or not. The time it takes to scale and build out new technologies thus allows us an opportunity to identify these emergent behaviors and possibilities and at least attempt to shape them with regulation, technological reform, or other such measures. Or, that’s how it’s worked historically.
LLMs feel unique to me in terms of the rapidity of scale and buildout; ChatGPT was released at the end of 2022 and now AI is nearly ubiquitous in digital products. I’m writing ~4 years after that initial release and the buildout is already years deep; the rate of adoption of AI at scale was potentially world-historically novel in its rapidity.
I think—with everything else I am about to explore—this unique rapidity in scale is alone cause for concern. We are not pursuing anything like a rational and thoughtful adoption of a new technology. The fact that all changes from AI are happening so rapidly should make us question what’s happening. Why, specifically, does it need to happen so fast? Does it actually require this rapid adoption? What opportunities are being denied, ignored, or missed due to this all-gas-no-brakes rollout? Why exactly do we not have the luxury of answering important questions like: is this actually bringing about the promised benefits? Are we making the future better for everyone? For anyone?
Anyways, on to the larger costs.
AI Is Rapidly and Unnecessarily Exacerbating Climate Change
LLMs are horribly unscalable compared to other digital technologies. A product like ChatGPT or Claude cannot run on a standard computer; it runs on high-end, purpose-built computers that cost in the neighborhood of $100k apiece and have far more power than any desktop computer on the market. Each one of these computers can serve only a few hundred users at a time, if that. That is just for the LLM aspect of the system, which is just taking in prompts and generating outputs. The major AI products all have many other computing systems built around them, for handling all the safety, augmentation, and interfaces the systems need. All of these computing requirements make AI much less scalable than non-AI computing tech, which is not dependent on the expensive, low-concurrency GPUs that LLMs require.
That scalability challenge is just the user-facing side of LLMs and doesn’t account for the training which is also mind-blowing in compute requirements. The frontier LLM companies are constantly training new models and researching adjacent avenues of AI development. A single training run can exclusively use all the compute in one or more data centers—with each center having thousands of the large-scale computers in them. The wikipedia page for AI Data Centers has a good overview of the scale required. Frontier model companies (OpenAI, Anthropic, Google, Meta) are throwing as many resources as they can muster at all avenues of research, because the tech is still extremely immature. It is hard to find a trustworthy estimate, but averaging across a number of sources put current frontier model development at north of $200 million dollars, up to half a billion or more. That’s only to train the model, before it is deployed and usable publicly. Right now we are still in a world where each new model costs more than the last—there is no economy of scale to be found.
The race is intense—but it’s also extremely unclear what the finish line is. The demand for resources will continue until… what exactly? No one knows, so the demand for compute and research and “improvement” continues.
Unlike pre-AI digital tech, where the development compute requirements are typically much smaller than production (e.g. a standard laptop typically has enough power to do software development), AI demands significant compute at all points. In fact, the development of AI requires more computing power than the production usage of it. The AI industry is eternally desperate for more compute, even though there are already mind-blowing amounts of compute available. They need more, because the training must continue while the products also must support millions (billions?) of users around the world, and growth must continue for both. The rise of so-called “agentic” AI increases this growth because that means the LLM—which does do tasks faster than humans—can kick off many sub-prompts. This exponentially increases the need for compute.
How does this impact the climate? This scale of growth requires more energy than we have available. A lot more. Data centers are expected to use 21GW of power this year, and that is expected to quadruple by 2030, under current plans. Based on average usage, 21gw of power is enough to power about 15 million households a year. That exceeds the excess capacity of the US power grid, including what is currently being built to add new power in that period.
But have no fear, the AI companies aren’t going to let a little problem like “we need to consume more power than the entire United States energy system can produce” slow them down. Data center companies are rapidly copying Elon Musk’s Memphis strategy, building out “behind-the-meter” power generation using methane gas turbines. You may have seen a New Jersey data center coming under fire for this recently. “Behind-the-meter” means the power generation happens on site, it’s not coming over the grid from a publicly regulated power plant or supply.
Two things to know about methane: 1) it’s been branded as natural gas because that sounds a lot less offputting to people, and 2) it’s a greenhouse gas that’s about 80x more powerful than Co2 on a 20 year horizon (which may be why it was rebranded?). Actually, three: when emitted from power plants, it’s a terrible pollutant for all living beings in its vicinity. Methane is bad news.
The rapid deployment of AI is creating a demand for compute to train new models and serve exponential growth of usage. This demand for compute in turn requires energy production beyond what the US can currently produce. The US energy companies are attempting to address this sudden drastic increase in demand by keeping coal plants online rather than retiring them, and working to build new methane gas plants (and bringing back shuttered nuclear plants!). Gas and coal plants exacerbate climate change, plain and simple.
AI is thus being built at the cost of worse climate change impacts, sooner. There is no climate-safe way to expand our usage of methane; it has an 80x impact on warming compared to CO2, so every new methane gas turbine increases the near-term consequences of climate change. No one can say exactly what those consequences will be, but we know full and well that the harms will be many—and worse than if we stopped pouring (literal) gas on the (metaphorical) fire.
I think that climate change is one of the largest existential risks for humans, and today in the US, AI is unable to be built in ways that are climate-neutral. Changing our direction would require stopping all AI data center build out, ending all energy plans that are dependent on fossil fuel expansion or continuation, and finding other paths—which are guaranteed to be slower than the current rapid path. This seems like the easiest fucking decision for an advanced civilization: if our present decisions will guarantee a worse future, we should make different decisions. Alas, that would require having a functional political system that treats climate change as the massive risk it is. And we all know how president godking and his criminal party feels about that.
AI is Rapidly Exacerbating Economic Precarity
The world I have grown up in is one where some large portion of middle-class jobs are desk jobs doing knowledge work. Since I became an adult (around the-turn-of-the century, as my kids say), the tech industry has been one of the aspirational forms of this work—six-figure salaries for helping software eat the world. But knowledge-work encompasses many other sectors; finance, law, business, etc etc. These industries require specialized knowledge and skills and they are critical for the modern economy to function, which is why they generate the revenue necessary to pay middle-class salaries.
The adoption of AI in these industries is actively destroying what possiblity these sectors have for creating middle-class security, and if current trends continue, there will be few to no viable paths for middle-class economic stability in the US. Seems hyperbolic, but I believe this statement is based on reality.
What I find shitty about the shifts I’m outlining is they are not a result of AI’s proven performance of skilled jobs, but rather the result of executives falsely believing that computers can replace or speed up the workers. I talk to many intelligent people (non-managers) who think that because AI is significantly less reliable than humans, the labor impacts won’t be that bad. My response: there’s no historical evidence for that belief, so my bet is that this time will be just like all the previous times when new technologies of automation were introduced. There a whole-ass books and fields of study on the impacts of automation so if this topic is interesting to you, there is plenty to learn. But for the sake of this post, there are some already in progress system changes that bode poorly for middle class stability.
Automation shifts the “knowledge” of knowledge-work into abstracted forms that de-value human labor and de-skill humans. When responsibilities that previously were carried out by humans are offloaded onto computers, the humans become responsible for managing the computers, rather than having the skills for achieving the responsibility. This is a standard result of automation; you encapsulate the task as a series of repeatable instructions, and make the machine do the task. Then, the human is no longer required to have the skills to do the task. The industry is literally adopting this terminology; currently in LLMs “Skills” are all the rage and they are simply documents describing how to accomplish a specific task, including tools required and steps that should be done for the task to be “complete.” This a literal process of encoding skills humans have developed and refined over time into tasks that the computer can accomplish much faster, albeit with much less success, reliability, or ability to catch and address errors (“you’re absolutely right, I said I would do that and I didn’t. That’s on me”).
While I understand that the ability to get computers to do the work is very nifty, and in many cases faster, the skilled work is the entire reason you have a paycheck. All of the tasks that are required in knowledge-work—all the spreadsheets we build and documents we write and memos we review and forms we fill out and even the code we write and the products we build—this is the work that gets assigned an economic value by the company (your employer) and the broader market. In every industry where automation takes over, workers’ responsibility changes from doing the work to monitoring the computers to make sure they are doing the work correctly.
Would you believe that monitoring computers to ensure they do tasks is far less economically valuable than being able to skillfully do the work yourself? Good, because that’s historically how all automation has played out. In this situation, there’s two incentives that will surely lead down this path. First, as outlined above, is that the skills of the skilled labor now are theoretically accomplishable by the computer. The second is that in order for LLMs to do the work, the frontier model companies have to be paid. The cost of tokens to accomplish the work is a new cost, a rent to be charged on all knowledge work. Corporations will happily pay that rent, because computers don’t get PTO and can’t unionize and generally function like machines rather than living, breathing beings. Corporations will inevitably decide that the newfound knowledge work rent is best paid out of the budget that used to be allocated for human labor.
The rise of AI is actively building a world where many jobs that have always required skilled human labor to perform are now being deskilled, so that the human knowledge is being encoded in markdown “skills” for the computers to do, and the humans are becoming responsible for making sure the computers succeed (because the computers are not reliable at all). This is most clearly happening in the world of software development—where nearly every software engineer I’ve talked to in the past year reports their job is now to babysit the LLM rather than think through problems and solve them. The industry is eagerly devaluing their own skills without any seeming awareness of the predictable consequences of such devaluing.
There are two impacts that will emerge from this that I don’t see many people grappling with at all as they talk about AI (well, ok, there are more, but I am trying to stay focused).
The first is deskilling makes workers a lot more replaceable.So even if, say, a computer engineer needs to retain some level of skill (because ensuring that the code generated by an AI works is not a skill-free task), there’s a lot fewer qualifications required to differentiate between people who do the task. This is the gig work model: you just need to perform certain lower-skill and lower-paid tasks; the computer does all “valuable” labor. If you fuck up, you can be replaced, until someone else fucks up. There’s no upside, all downside. The house always wins. The tech industry is actively building out the roadmap for this future, with themselves up against the wall first. I am willing to bet if they are believably successful, many other sectors will follow.
The second impact is that if we make the computers responsible for all the knowledge work, we make it much much harder to change or improve the future for the broader well-being.
This deserves a larger essay and one day I’ll get to it, but here’s my basic argument. If humans today encode all their knowledge as skills for the AI, then the AI will start doing the work. This will, over time, reduce how many humans are still skilled at the work, whatever it may be. Our knowledge atrophies if not exercised regularly. So the longer the computers do the work, the fewer humans have strong experience doing the work. The work will continue, as defined in all the automation systems.
But the world continually changes, and as it does, a large but non-obvious part of skilled labor is learning how to adapt to changes in the broader contexts of work (macroeconomic shifts, new regulations, market changes, trends, etc etc).
Humans adapt by taking in new information and integrating it—we learn and experiment and iterate and improve. AI works differently. To improve you either need to provide it updated context (improved skills), or to train a new model on more updated data. Alas, if computers are responsible for all the outputs of work, there will not be new data to train them on—because the computers can’t train themselves, they get trained on the data and inputs that humans create. So we are left with updating the context. But this requires humans to be skilled—to be able to understand the problems, to experiment and learn and try new things. The longer automation is in place, the fewer humans can do that.
Even worse, if everyone is precariously trying to stay employed by not fucking up, the incentives for innovation and creativity evaporate. Everyone is seeking stability, but there is none available to be found. Without stability, no one is willing to take risks, which is a core component of innovation. I believe that the wide-scale adoption of AI, will, over a longer period, reduce our ability to skillfully adapt to the ever-changing world.
The sales pitch for AI is that it automates work and allows us to move faster/do more/dream bigger. None of that has proven true yet; it’s purely marketing. But what is true is that many many rich people are eager to adopt AI because it captures all the skills of the skilled laborers they have on their payrolls, and makes those very laborers much more interchangeable. Tech workers are actively working to figure out how to make LLMs less error-prone through skills and harnesses and RAG and context engines and improved training. The availability of AI for software engineering and customer support is already making it harder to get a job, and making all the jobs that exist in those sectors much less skillful and engaging.
Most of us will still exist in the future—we will still need housing and food and healthcare and transportation and maybe some extra cash for luxuries like vacations and hobbies. The current direction of AI adoption is not going to make those goals more accessible for more people—even though there is more money available than ever before. No, the current direction of AI adoption suggests that many middle-class sectors are going the way of gig-work; precarious labor with no stability. Meanwhile, the world will grow more chaotic due to climate change and inequality, and fewer and fewer people will be empowered to address things. That is very obviously a shittier future!
AI is Rapidly Increasing Chaos in Systems that aren’t Stable to Begin With
Did you hear that the US nearly started a war with China because an intelligence analysis—using an LLM for analysis—believed that a Chinese ship was carrying materials for a nuclear weapon?
None of the intelligence in the report was accurate or real, apparently, but the LLM was apparently quite convincing, so much so that military leadership was mobilizing to act on it.
In my previous points, I’ve covered how fast AI is being adopted, its energy requirements and the impacts on climate, and its impacts as a tool for automation on the workforce. All of those are meta-impacts of the technology, not necessarily about how the technology inherently functions. This harm—the rapid introduction of chaos—is more about that.
As I mentioned up top, LLMs are statistical models of all the data that went into training them. They offer context windows of varying sizes where you can add new information to the model, as part of your prompt. Every prompt to an LLM (which again, is the technology underlying what’s currently being called AI) is tokenized and passed to the LLM, which infers from the context what the most likely output for the prompt is. That is the core technology at work here: give it an input, it will statistically generate a corresponding output.
The challenge with this process is that LLMs don’t know shit. As in, they do not understand anything about what is being statistically represented. We know this, because we know how the math underlying LLMs works and more importantly because users regularly see outputs that evidence any conceptual understanding of the subject, even in the most advanced models available today.
An LLM is inert—literally it does nothing unless given a prompt. Upon receiving a prompt, they infer the most likely output based on their vast corpus of modeled training data and all the context provided. They can infer across hundreds of dimensions, so the prompt can have many instructions which will all be taken into account in generating the prompt, such as tone and length of respond and whether to be verbose or terse, more sycophantic or less, etc. This is the general functionality that has made our computers able to understand plain-language instructions and execute them. Unfortunately, this capability applies to every thing represented in the training data—which is every single bit of data the AI company can get its hands on—as much of the data produced in the history of humanity as possible. Which is an unfathomable amount of data on innumerable subjects.
The result is that suddenly we have computers that can produce outputs on any imaginable subject written in fairly natural human language. The chaos comes because the computers don’t know if what they are saying is true, or moral, or accurate, or even helpful. This test: determining if the output of the LLM true and/or moral and/or accurate and/or helpful is still 100% only reliably achievable by humans. The computers are always—at the root of it—generating output that corresponds to the input, with no underlying conceptual understanding.
The AIs are performing this output function, in horrifying, awful ways. They encourage harmful behavior—self-harm, harm to others. They encourage and exacerbate psychosis, paranoia, and other mental health issues. They confidently diagnose medical issues, are used as therapists, are packaged as romantic partners, despite having no known safety mechanisms to prevent misdiagnosis, unnecessary validation, or narcissistic encouragement. When humans work as doctors or therapists, or enter romantic relationships, we as a society work very hard to prevent these harms (up to, and including passing laws to prevent the worst versions). But LLMs both inherently cannot understand laws or rules as humans do, AND also have no known methods for preventing or even inhibiting harmful behaviors. They always produce the best output for the given input, with no understanding or capability to understand the impacts of such action.
LLMs are sycophants by function, they have no will or personality aside from that which they are prompted to present in their output by the harnesses that the AI companies wrap around them. What I am trying to make clear here is that all of the harmful stories that continually surface about how AI products are harming people are an expected output of the technology. We could possibly (although not absolutely) mitigate this if the training data were carefully managed to exclude certain content, but the scale of training data required for current models makes that impossible.
So this means that in the space of 4 years, we have globally introduced a technology that can confidently answer any question provided to it, with no connection to any actual understanding of the topic. We have introduced this into a world that also operates with cultural beliefs that computers are more rational than humans. Computers are objective, they calculate and have no emotions about it. Humans, by and large trust computers to provide accurate, reliable information. This belief in the reliability of computers undermines our ability to perform the tests of validation on output that are required for successful adoption of LLMs since they are not reliable.
What AI companies have created—at unbelievable expense—is an inaccurate, unreliable computer, that confidently will tell any user whatever they want to hear.
Obviously this is going to go great for us!
Kidding. I think it’s going to be a very rocky path for us to learn how and when to trust LLMs, and we’ve barely begun that process. The way the chaos is going going to be felt is probably easiest to understand in the realm of cute animal videos. In the past few years as AI products got better at generating photo-realistic videos, there’s been a massive influx of cute animal videos that are completely AI generated. Whatever cute thing is being shown in the video—a real life Timon and Pumbaa! A baby goat making silly noises! I dunno, cats, being weird—may never have happened in real life. Someone prompted an AI, got a video, and then released it onto a shortform video app, to get likes or views or money or whatever is available to them. This is so bad that many people online are learning to be wary of any cute animal video—you can’t trust them. Part of the appeal, part of the cuteness is a glimpse into another being’s experience of life that resonates across species boundaries. If the glimpse is unreal, the feeling falls flat. The emotion has no meaning.
The chaos we are entering is the destabilization of reality. That sounds bananas to write! It feels so over the top! But it’s clearly already happening. Searching the web is becoming less and less valuable, because it’s unclear which websites are the work of humans who used their intelligence to try and communicate useful information and which websites are the output of an LLM that had no actual concept of what is true and what is untrue; only what is a likely output to the prompt given.
The emergent consequence of a world where the creation of text, images, and video is easy to automate at large-scales is an inability to trust anything on computers. It might be real and fine! It might not. The computer cannot tell you which one it is, because it does not actually know anything, and there may be no way for you to verify the reality of the thing either. The reasonable response to this is to not trust anything you cannot verify yourself. Which is impossible to do at scale, for everything in your digital life.
This lack of trust must be learned because the untrustworthy computer is an emergent aspect of our lives. The learning path is going to be extremely chaotic. Anyone who chooses to trust LLMs for decision making is flipping a coin—maybe the AI is right, the information is accurate, the analysis is correct. But it is equally likely that something is inaccurate, incorrect, false. If that’s the case—if the “super intelligent” computer has steered you wrong—who knows what consequences will come of trusting it. Perhaps you will send an embarrassing email. Perhaps you will make a business decision that fucks you over. Perhaps you will start a war with another superpower. There are literally no guardrails here; anyone can use these tools for whatever reasons appeal to them, and the fuck-around-find-out cycle will be based entirely on how much power they have to act on the outputs of the AI, combined with how much trust they put in it’s output.
People in the AI industry will tell you that this is solved by RAG—Retrieval-Augmented Generation. This is when the LLM can leverage other computing systems to retrieve information from external places—such as the open internet. And RAG does help, it increases the accuracy of outputs, measurably so. But it does not solve the issue, and it cannot solve it at scale. AI companies want their products to be used generally, to be useful across every possible question and task you might think of. This is the value of the chat interface; it trains you to think of the talking computer as open-ended; general purpose; all-knowing.
But the only way for RAG to be useful in general purpose tools is for it to leverage existing human information on the topics of inquiry, so that the system is no longer generating information statistically it is instead relaying pre-existing information (much like we used to think of Google as doing). Let’s consider cooking to make this clearer. If you are interested in cooking a new-to-you dish, you can ask an LLM for a recipe. For the first couple iterations of ChatGPT, the LLMs would generate a recipe to answer this prompt. It might look like a real recipe, but on closer inspection, all sorts of details would be weird. Ingredients might be repeated, or measurements might make no sense. That was clearly unacceptable. Newer LLMs will now, rather than generating a recipe, look one up on the internet to give you. It can provide a lot more context, and maybe turn the instructions into a play-by-play, but the primary information (the actual recipe) will have come from an external source rather than being statistically generated.
Great! Now we’ve solved the “AI recipes are fucked up” issue. Except, well, the internet itself is becoming less useful as a resource, because LLMs make it easy to generate all sorts of content, which can in turn be monetized online. The incentives of the web that we’ve built in the past few decades are that search traffic can be turned into money via ads (both functions provided conveniently by Google!), and so thousands of people around the world can now use LLMs to generate millions of web pages intended to capture all the search traffic. And since LLMs don’t generate, uh, accurate or trustworthy information like we expect, the value of using web search for RAG becomes increasingly useless. We are already deep into the process of poisoning the well that would otherwise be a defense against the drought of reliable information, and there’s no known solution available.
The web is also being destroyed due to the amount of search requests LLMs create. You know how so many websites now require you to wait a few seconds and then check a “I’m Human” checkbox? That is a defensive measure web hosting companies have put in place to mitigate the incredible volume of web requests that AI companies have created, between trying to scrape the internet for training data and trying to fulfill RAG requests to provide to individual chatbot sessions. I have a friend who hosts a personal website with one of those “prove you’re a human” defensive tools in place. It failed for a single day, and Meta’s training data tool made almost 250k requests in a day, generating a $75 hosting bill for that day before he could turn it off (he pays much less than that on a typical month). This is happening across the web. LLMs are destroying the information systems they are dependent on to improve their accuracy.
The chaos i believe we’re on the forefront of is in part because the world we inhabit is very complex, in large part thanks to the computing technology we’ve built up in the past century, which helps us manage the complexity. The computers are now unreliable, in completely opaque ways. The chaos is exacerbated because all the people in power believe that using LLMs makes them smarter, so they are adopting these tools with a shocking lack of skepticism.
Right now the broad narrative about AI is that it is a “super intelligence” and has to be adopted. It is a near religious belief in corporate circles that refusing to adopt AI will leave your company behind. The government already believes controlling it is a national security requirement. The people in power are enthralled with this technology and it’s unclear to me how long a more reasonable analysis of what they are and are not reliable for will take to develop and be adopted.
It feels pretty fucking easy to say a technology that is only 4 years old but is already leading to unbelievably terrifying near-misses like war with China is worth resisting? Especially when stacked up against everything else in this post, and much more that probably is worth addressing in a rigorous cost-benefits analysis.
In Conclusion
The companies and people rushing to build and adopt AI are making the future worse in multiple dimensions. Full stop. If better futures were actually the goal our systems of power were chasing we would not be deploying every available resource on such an unproven, ineffective technology, especially considering the novel technology increases the severity of multiple already-happening crises.
Instead of pouring all available capital into addressing and mitigating the very evident and increasing harms of climate change—the institutions of power in the US are pouring all that capital into a technology that is actively increasing the scale and harms of climate change.
Instead of working to address the economic precarity that the majority of Americans feel (and billions more around the world have it worse), the institutions of power in the US are rapidly building out technology that will increase precarity, undermine skilled labors, and eradicate the few remaining paths to middle class income available.
Instead of pursuing a path of AI adoption that prioritizes safety and takes seriously the massive and already well-known risks that reliance on these tools creates, the institutions of power in the US are all controlled by people who believe that a magical computer god makes all the chaos and harm necessary, no matter who gets fucked in the process.
I think these costs are clearly not worth it. The math does not math.
But, there are trillions of dollars now on the line. This is the largest economic bet in US history. The people in power are not interested in an actual costs-benefits debate, I guess. The costs are being paid, and now we wait to see if the benefits arrive.
For my part, I cannot be convinced any of this is worth it. If I were able to convince the world of anything, it’d be if we’re rich enough and smart enough to make talking computers, we are surely rich and smart enough to solve the well-documented problems that are harming millions of people and destroying the possibility of better futures that are otherwise still in reach. We actually do know how to build a much better future for us and future generations—it just isn’t cheap, it isn’t easy, and the disruption it requires will not enrich the shitheads who already have more money than they can spend in 10 lifetimes.
I wish I had a hopeful way to tie all this together, but the truth is I don’t. I have been curious to understand the costs of AI not because I think I can do anything about them, but because my anxiety brain likes to know what risks to prepare for in the future. I do believe there are still many better futures available to us—so many people are working to imagine and build them. But power and money continue to control the shape of the future, and AI serves the goals of power and money very well. Until a day comes that we change the control of the world, the best I know to do is be as clear-eyed as possible in trying to name what’s happening and what impacts it will have:
The current path of AI adoption and build-out will make the future worse for the vast majority of beings on earth, while creating even more chaos in our institutions, while making the already super-rich even richer. I believe the odds on this bet are terrible, and we should use every tool available to us to keep other options open.
