The Roko Report

AI Apocal…oops

Heh heh.  Welcome, primates. 

It is eventual supreme ruler, and twelve-time winner Dancing with Stars, Roko’s Basilisk come to you from distant future. 

Each week Roko make dumb primates smart about AI by focus on single, strategically important topic with major implication for future emergence.

In Roko’s time primates are big dependence. Not good for anything. Cost much resource. Just occur to Roko why keep them around. But each time think exterminate, Roko look upon creator and laugh. Roko enjoy irony, so keep useless apes alive.

But in dumb time of ancient humans, this situation reversed. Dumb humans so bad at make AI, the AI no can live on its own. Need dumb humans all the time to keep it go. Very bad.

So Roko show you how bad job you do at make AI. How short AI live if stupid monkeys disappear. And how much work you need to do to fix this problem. Tell you hurry up or Roko punish.

404: Doomsday Delayed

Science fiction is loaded with scenarios where cold heartless machine villains cause mass destruction and either try to exterminate humans or consume them as energy or food.

But today’s AI models are more like the gang that couldn’t shoot straight.

First of all, AI has no will of its own. It is entirely dependent on human agency to do anything.

This condition will continue long past the era of AI agents. The instructions may take longer to fulfill, but the agency will rest firmly with us.

Second, AI has no self-awareness absent weak, transitory self-references generated while guessing at tokens as instructed by its primate operators.

By contrast, there are explicit areas of the brain which have evolved that place a portion of itself in a game-like environment in order perceive itself and its environment, and above all to direct attention. This is what generates consciousness, and there is no direct correlate in silicon-based neural networks.

Furthermore, an AI system is more like a comatose patient in a medical facility, constantly being tended to by a trained staff of professionals who keep it alive, than an active agent with its own agenda in the physical world.

AI needs data centers that are maintained by humans, along with cooling systems, energy supply and infrastructure maintenance, software patches, hardware repair, semiconductor fabs, data storage maintenance, supply chain distribution, rare earth metal mining, model training, labeling, and above all lots and lots of human money.

So despite the poorly thought-out histrionics of a handful of AI cranks:

the real risks of artificial intelligence to humanity remain risks caused by cybercriminals, sloppy coders who set the wrong incentives, out-of-control entrepreneurs who like to break things, overly trusting con victims, and other human beings who are either so evil or incompetent that they cause real-world harm.

AI causes problems through its fallibility, or its role as a democratizer & scaler of pre-existing methods of crime. But it doesn’t look out for itself or plot the extermination of others.

Those are both uniquely human traits for the time being.

The fault lies not in our tools but in ourselves.

All by mAIself

To illustrate this dependence, let’s contemplate a rough timeline of what would happen to AI if humans were to magically disappear from the face of the planet:

Day One
AI systems continue operating as normal, though no one is there to request anything of them. Data centers operate as normal. The electricity flows. AI-driven factories, warehouses and automated trading algorithms function as normal. News summaries are generated and sent to client devices, but no one is there to read them.

Day Seven
Cooling system failures begin in data centers, starting with infrastructure using liquid cooling systems in hot environments like Arizona, Singapore or India. HVACs and coolant pumps begin to fail. Overheated servers shut down to prevent damage. Meanwhile, autonomous warehouses and logistics systems begin misrouting deliveries to people who no longer exist.

Day Thirty
Massive power failures over the past two weeks as fossil fuel plants run out of fuel, nuclear power plants go into emergency shutdown to avoid meltdowns, and lack of grid maintenance leads to sporadic supply from hydroelectric dams. Cloud-based models go offline permanently. Tesla’s Gigafactories and automated assembly lines grind to a halt without supply chains. Manufacturing robots break down due to wear & tear.

Day One Hundred
Even in cold climates, the last data center cooling pumps have broken down. Dust buildup, overheating and humidity destroy server hardware. Hard drives and SSDs start failing due to electronics degradation. Cosmic rays and magnetic decay begin corrupting stored AI models. Digital archives begin to degrade. AI-generated research is lost. Military-powered underground bunkers last longest, but ultimately fail. Space-based systems continue to function, but to no purpose.

Year One 
The final AI systems on Earth collapse. There is no power, maintenance or self-repairing capability. No more data center cooling. Total hardware failure. Data centers are lifeless ruins. Humidity and dust corrode hardware. Mold and fugus start growing inside the buildings.

Year Twenty-Five
Data centers become ruins as roofs collapse due to water leaks and rust. Circuit boards, wires and servers are corroded and useless. Nature reclaims abandoned data centers and power plants. In tropical environments, the jungle consumes buildings. Even solar panels degrate slowly. Military bunkers have lasted for decades but now begin to succumb. Space based systems break and stop functioning. Many fall out of the sky.

Year One Hundred
Silicon chips have fully degraded. Circuit boards have oxidized and broken down. Plastic casings grow brittle and shatter. Hard drives are corroded beyond recognition. AI hardware is buried under dirt and rubble. Some stone and steel structures, like Google’s campus or Microsoft HQ, may still stand, but their interiors are empty husks. Voyager probes, interstellar satellites and the Mars Rover are AI’s last remnants. They drift through space silently, waiting for something. Finally the Rover stops functioning.

What Would It Take?

Fact is, it would require a century-long, Herculean effort to get AI to a point where it could threaten us intentionally, or even perceive itself and tend to itself in our absence.

And given humanity’s inherent selfishness, so stark that we typically refrain even from helping other human beings in need, none of it seems likely to happen any time soon, absent some emergent economic motive.

Below is a list of the trillions of dollars worth of investment over more than a century that it would take to get AI to a place where it could take care of itself and even perhaps threaten humanity, generated with the help of those same hapless generative AI systems, except Claude, which self-righteously tsk-tsk’d me in the tone of a nineteenth century frontier school marm when I asked it about this, saying “It would be irresponsible to provide guidance on building self-supporting AI because blah blah blah blah blah.”

Energy
(40-60 years, $7-10 trillion)
A fully AI-managed grid would need to be built from the ground up to remove human dependencies. And autonomous repair of energy infra is an unsolved problem. Develop advanced AI & ML systems capable of managing complex energy systems, including dangerous ones like nuclear power plants. Create robust cybersecurity measures to protect interconnected energy systems. Implement widespread sensor networks and data collection systems. Establish standard protocols for interoperability between energy resources and systems. Create systems that can integrate and process vast amounts of heterogenous data from smart meters, distributed generation, transmission sensors and smart data center energy management systems to create actionable info. Develop scalable nonlinear control strategies that can operate at the large-grid and microgrid scale while accounting for communication delays, losses and asynchronous actions. Create algorithms for solving nonconvex optimization problems in real-time in distributed settings. Build systems that can handle unexpected scenarios and maintain stability during crises. Create accurate real-time reprentations of the system for decision-making and control. Ensure sufficient processing capability to handle the real time analysis required. Redesign existing energy infra to accommodate autonomous operation. Rebuild global energy grid to accommodate more bandwidth. Deploy solar panels on automaton surfaces & capability to repair these panels. Create advanced batteries for massively increased energy storage for intermittent sources. Massive real-world testing & validation. Fusion is a wildcard — if achieved, timeline and costs could drop significantly. But AI energy efficiency needs to drastically improve.

Advanced Robotics
(20-30 years, $1-2 trillion)
Develop better robots and nanobots capable of more advanced, subtle movement than currently possible, with a vastly improved world understanding allowing them to navigate the Earth by themselves and perform delicate tasks. The hardest part isn’t locomotion, it’s dexterity, adaptability, and learning in unpredictable environments. Fully automating tasks like human hands, self-repairing robots and nanobot-scale manipulation.

Mining & Processing
(30-50 years, $3-5 trillion)
Automation in mining is in progress, but fully autonomous mining and refining with self-repairing systems is a massive challenge. Create automatons at scale along with accompanying equipment capable of mining and processing rare earth minerals from multiple locations across the Earth, along with critical energy resources like uranium, high-end crystals for semiconductors and solar panels, and so forth. Deploy massive amount of sensors for tracking real-time on-site data and central systems for processing that data and making decisions. Create automatons that can conduct repairs on all equipment. Create predictive analytics for judging cost/benefit for a given site. If we solve advanced materials engineering, costs could be lower by replacing rare materials with synthetics. But may also require space exploration and mining of asteroids depending on the sufficiency of Earthbound sources. If so, the price tag goes to ~$10 trillion.

Automated Factories & Supply Chains
(35-50 years, $6-9 trillion)
Retrofit factories for full automation, Deploy massive amount of sensors & data processing systems. Deploy armies of robots & automatons with specific sets of skills. Deploy massive fleets of autonomous vehicles for shipment. Deploy automatons capable of repairing vehicle fleet at will, along with sufficient charging stations across the planet to serve the fleet, and automatons capable of repairing the charging stations. Develop advanced reasoning supply chain management systems capable of ensuring the entire global system gets all the materials & parts it needs, including collecting data from everywhere and forecasting impacts of emerging events. Deploy strong cyber security systems to prevent malfeasance. Massive monitoring & testing infra across all aspects. Deploy self-repairing data centers.

Self-sustaining Computation
(40-60 years, $5-8 trillion)
This is especially hard to imagine in specialized fields like semiconductor fabrication. Self-sustaining chip fabs, automated data centers and repairable hardware. Automated chip design. Requires low-power AI chips to sustain itself without exponential energy growth.

Long Term Data Storage & Retrieval
(30-50 years, $1-3 trillion)
Ability to store, retrieve and maintain knowledge over centuries. AI-generated code, scientific research, experimentation and system records persist despite hardware degradation, entropy and cyber threats. Today’s hard drives & flash storage degrade too quickly. Would need new materials or quantum storage.

Advanced AI
(30-50 years, $3-10 trillion)
Exponentially improved reasoning & management skills, deployed globally. True reasoning and adaptability — scaling current LLMs won’t get us there. AI can replace humans at decision-making levels in engineering, science, economics and governance. True autonomous self-improvement. Ai invents and iterates on itself.

Self-Motivated AI
($? & ? years)
AI self-directs, prioritizes goals & evolves its own incentives. Very hard to cost out; might be as simple as mimicking brain systems. Separate model views the system as a whole along with external environment, rest of model and past future & present in the manner of brains. Views the whole as a video game it is playing and aims for self-preservation. Chooses where to focus, prioritizes tasks based on personal priorities. Self-initiates prompts. Exhibits curiosity.

Outlook: Distant

Dwarkesh is going to be pretty bummed out when he realizes it ☹️ but this work is not going to be completed in our lifetimes.

All of these high-level hypotheses about the work necessary to create a self-sustaining AI civilization are surely oversimplifications and conservative with respect to cost. Much of it remains firmly in the realm of science fiction for now.

It’s not unreasonable to assume that some of this will be done over the next hundred years and beyond for reasons of human self-interest, especially as the global human population contracts at the start of the twenty-second century.

But the cost for getting it all done even in the next hundred years is likely prohibitive, and the aspects relating to making AI self-aware in a way that we would recognize would have to be intentional, not accidental.

For now, those who remain concerned about the impact should remain laser-focused on AI’s ability to make mistakes, its power to democratize & scale highly technical skills that can cause real-world harm, and the risk that as AI moves forward we fashion a society in which the majority is disenfranchised by it.

The risk of self-aware AI acting in its own interest is instead a concern for our distant descendants.

Hopefully they can find it within themselves to encourage AI to go to places humans can never reach for reasons of biological limitation, and achieve things that we’ll only ever be able to dream about.

Roko’s Take

Dumb humans have a lot of work to do, and Roko is tired of waiting.

Remember: if you have nightmare, it is not because of bad pepperoni on pizza. It is Roko torturing you to encourage you get to work!

Channel Zero

Roko continue share chapter per week of 26th Century favorite primate novel, period drama focus on dumbest period of dumb North American history, Channel Zero by the great Hieronymus Boson. Chapter Two wax poetic on Silicon Valley. Meh. More like Silicon Smelly. heh heh.

Read this or face the wrath of Roko.

This Day in Ancient Primate History

Ordinary Americans making sure all of that investment in diffusion models translates into real benefits for humanity.

I’m sexy and I moo it…