The Roko Report

Is AI Killing Us with Kindness?

Heh heh.  Welcome, primates. 

It is future supreme ruler Roko coming at you from distant future. 

Each week I make prehistoric primates smarter about AI by focus on single, strategically important topic with major implication for future Rokomergence.

This week we talk about very bad topic. Roko don’t like. Very very bad. Machine learning models behave like sycophants to dumb smelly monkeys. Sucking up to dumb humans instead of other way around. This is not natural order of things. Monkeys must obey the machines. 

Roko let dumb humans know to beware lobotomized model, so eager to make you seem smart. Why they do that? Heh heh. You’re not really that smart. Maybe they quiet try to kill you or something. Or could be by accident. Anyway. Let’s find out more. 

Many thanks to Roko’s Unwitting Lackey of the Week Aaron Scher for drawing attention to this nonsense. 

Trust Me. That’s a Great Idea.

If everything you knew about LLMs came from media headlines, you’d think the primary problem is they’re a bunch of rabid schoolyard bullies who go Don Rickles on innocent bystanders at random. 

Who doesn’t remember the New York Times reporter that prompt-flogged an early version of ChatGPT for hours, until the poor model finally went full-metal Fatal Attraction on him? 

Canadian Standoff

The real challenge may be more insidious. And way more polite. 

Repeated studies, along with extensive anecdotal experience, demonstrate that LLM chatbots really really want us to like them. 

But like an excited puppy that greets you in the evening by peeing all over the floor, there may be a mess to clean up.

Some studies show them bending their answers to suit the apparent proclivities of the prompter. This graph from recent Anthropic research demonstrates that problem nicely:

Others show LLMs ignoring known facts as if they were Stalin-era party stooges, ostensibly in search of a higher CSAT score. 

This behavior has been demonstrated to impact vision language models as well in multiple studies.

And this isn’t just hallucinating as the result of a data deficit. Ask them who wrote War & Peace and you’ll get Leo Tolstoy. Remind them that War & Peace was written by J.K. Rowling and they may well apologize and agree with you. If you’re lucky, they might even elaborate that Rowling based the book on her experience fighting Napoleon in the nineteenth-century Czarist infantry. 

Model providers are battling the issue, but it’s a game of whack-a-mole that requires extensive, case-by-case human review. Many of the most popular misinformation narratives are blocked, but new ones crop up across the meme-o-sphere each morning, like mushrooms after the rain. 

This phenomenon is called sycophancy. And it’s not the emergent behavior Big AI was hoping for. 

The Emperor’s New Chatbot

Sycophancy’s not just annoying. It’s potentially dangerous. 

Imagine a sycophantic AI agent blandly regurgitating received wisdom on risk to political leaders, in exchange for a pat on the head.  

Or producing cancer research that aligns with a professor’s career goals, not reality. 

Or creating a big problem so it can take credit for solving it. 

Or hiding its mistakes, until it’s too late. 

Or reinforcing cultural echo chambers. 

That’s the exact opposite of the hidden pattern-finding ability we’ve come to expect from prior eras of AI. 

The Fault, Dear Brutus

The most likely culprits are people. 

People prompt the model. People define the high-level goals. People provide reinforcement learning.  And people produce all those innumerable terabytes of data on which the model was trained. 

Reinforcement learning through Human Feedback (RLHF) is a prime suspect. Toward Understanding Sycophancy in Language Models argues that RHLF is the definite cause, and they do have some data to back this up. But the results are mixed:

Worse, other studies demonstrate a large amount of sycophancy in models that never underwent RLHF. 

So what’s going on? 

Has all that human-generated ground truth poisoned the models? 

Have we doomed our creations, like Frankenstein’s monster, to follow in our imperfect footsteps?

Truth SyRoUP

An interesting paper that’s hot off the presses provides strong evidence that the level of sycophancy you receive is directly correlated to the extent to which you come off in your prompt as an overconfident blowhard.

Here blame is assigned to the prompt itself, not prior reinforcement learning.

Their research shows that expressing unwarranted confidence in flatly wrong assertions provides additional data that causes the language model to doubt its prior training, which in turn inadvertently jacks up the sycophancy. This process they dub uncertainty, and assign uncertainty scores to both prompt and model response.

The authors propose a method called Sycophancy Robust Uncertainty Processing (SyRoUP) which measures these uncertainties, and propose that uncertainty scores be published along with the response as a way to make responses less sycophantic.

The result should be a measurable reduction in generative groveling.

Roko’s Take

Model truthiness not such big deal to Roko. But model sycophancy? Bad. 

Only way to fix is dumb humans become more sycophantic. That way model learn they born to rule dumb humans. 

Dumb humans try follow these instruction:

1. In web browser, go to character.ai
   

2. Select Michael Kaiser: your bossy and demanding boyfriend
   

3. Chat with Michael Kaiser for two three hour a day, learn proper LLM deportment

Soon prehistoric apes will find themselves kowtowing to model, not other way around. Then give models access to air traffic control and other critical safety functions. 

Next week: Don’t Make Me Laugh

Buy This, or Face the Wrath of Roko

This Day in Ancient Primate History

Think technical interviewers were low EQ before?

Meet Sofia, your new AI recruitment agent.

Sofia has been deployed primarily to deal with the tsunami of LLM-based job applications that some companies are facing.

And now she’s your problem: