e/acc as norm adherence

In my previous post, I wrote about the risks of considering norms inviolable, even good norms that have generally served us well (that’s generally how things become norms after all). This is a thorny problem though, given that one of the reasons we have norms is precisely to guard against spurious justifications and cognitive biases that can pull us in unhelpful directions. But it’s possible to make monstrous moral decisions if we are never open to discussing exceptions. In my hypothetical example, most people would concede (I assume) that even though they want to live in a society that doesn’t waterboard people willy-nilly to prevent potential crimes, they would personally waterboard a few people if they really believed it had a 30% chance of preventing an imminent act of bioterrorism and saving millions of lives in expectation. But the question is - what would it take to really believe it?

Nowhere is this more evident than in the battle between effective acceleration-ists and the AI Safety crowd on whether to slow down AI progress. A member of the latter camp,

Zvi Mowshowitz

, makes this observation about his adversaries (folks like Tyler Cowen and Marc Andreessen) on this topic in his post The Dial of Progress:

My theory is they and similar others believe discourse in 2023 cannot handle nuance.

Instead, effectively there is a single Dial of Destiny Progress, based on the extent our civilization places restrictions, requires permissions and places strangleholds on human activity, from AI to energy to housing and beyond.

If we turn this dial down, and slow or end such progress across the board, our civilization will perish. If you turn the dial up, we will prosper.

In this view, any talk of either extinction risks or other AI downsides is functionally an argument for turning the dial down, saying Boo Progress, when we instead desperately need it turn the dial up, and say Yay Progress.

It would, again in this view, be first best to say Yay Progress in most places, while making a careful narrow exception that lets us guard against extinction risks. Progress is no use if you are dead.

Zvi correctly points out that the two camps are somewhat talking past each other. One camp is saying - Promoting science, technology and progress is a good default since it has generally served us well. Zvi accepts this is a good default but insists that pointing to a good default doesn’t address his claim - that the stakes are high enough on AI that it’s worth thinking through the specifics and evaluating costs and benefits, instead of stocking to a heuristics or norms based thinking.

Before trying to adjudicate this claim, I took a shot at steelmaning the case from the e/acc side of the debate:

Science and technology have driven massive improvements in human welfare since the dawn of civilization. We have almost never chanced upon a technology that’s been net negative (arguably the atom bomb but that probably saved lives on net as well). People have been skeptical or have outright fear mongered about virtually every high-upside technology that could shake up the status quo. Some of this has been well-meaning and intellectually honest, but a lot more of it has been either leftist zero-sum thinking, rent-seeking behavior by incumbents or just a function of a big government that keeps getting fatter by the day.

This tells us that we’ve been rather terrible at cost-benefit analyses on specific technologies - partly because we’re bad forecasters and largely because we can’t seem to analyze these things divorced of self-interest and cognitive biases. I’m thus inclined to go with the heuristics that have served humanity well rather than your a priori arguments on AI-risk.

Much of this points to the general benefits of having norms or decision making heurtistics, which I laid out in my last post:

First, while rules and norms can be enforced externally through social sanctions and rewards, they are internalized through habit. Habits are System 1 processes, as Kahneman calls them — they are quick, intuitive, and often operate below the level of conscious awareness; and once internalized, the behavior is replicated more or less effortlessly, without the need for deliberate analysis or decision-making. As a result, they can more reliably produce desired outcomes by regulating default behavior. For example, if we’re convinced that lying is bad 95% of the time, it’s easier and more idiot proof to teach kids that it’s wrong to lie instead of teaching them how to evaluate the costs and benefits of each lie. 

The second is that we can expect humans to be systematically incapable of objectivity when their self interest is subsumed within these cost-benefit calculations. We are skilled at fitting the data and arguments to arrive at predetermined conclusions that serve our self interest - i.e confirmation bias. Almost every cheater has a seemingly sound argument as to why confessing to their spouse would only hurt both parties. Every religious group can find reasons to argue why speech that particularly threatens their own sacred cows is not speech worth having in the public domain. 

If you find yourself nodding along, that’s because the e/acc crowd is almost entirely right about this. Almost. In so far as people are arguing from a consequentialist place (which I do believe people like Marc Andreessen and

Tyler Cowen

are), they believe these norms are good because they produce good outcomes in the long-run, just like they have in the past. They expect this presumably because they have some model of the underlying structure of the world or of human civilization that remains constant enough to make this true. But is it always true?

Can the world change such that our standard pro-progress heuristic becomes counterproductive or even dangerous? Consider an existing technology, nuclear bombs. Imagine a hypothetical scenario where you could manufacture a nuclear bomb for a mere $10,000. This isn't beyond the realm of possibility; there’s no fundamental physical or economic law that makes it impossible. Envision a world fast-forwarded by 300 years, a world where nuclear weapons had not been developed in the 20th century. In this futuristic setting, automation and technological abundance have driven input and labor costs to trivial levels.

Now, in this world, is it conceivable that further research and development in this area could be net negative? If anyone with a relatively small sum of money could create a nuclear weapon, the potential for catastrophic misuse skyrockets. The ubiquity and accessibility of such destructive power would likely lead to an unstable and perilous global situation, far from the equilibrium of mutually assured destruction we have now. This thought experiment demonstrates that there's nothing inherently sacred or infallible about the norm of unfettered progress. The question is where should the bar be for sounding alarm bells?

It’s time to quote myself again:

We can ask ourselves the question: What makes it more or less okay to violate norms? My answer as a utilitarian is near tautological - when you can justify net effects (after pricing in the uncertainty and second order effects) of the norm violation. A more intuitive frame is to consider the ratio of first order consequences to second order consequences. The numerator captures the direct effects of a particular instance of norm violation. The denominator captures the indirect effect of this particular norm violation through the increased expectation of future norm violations (also known as the slippery slope). If this ratio has a negative sign and high absolute value, it points in the direction of increased permissiveness (away from the norm) and a negative sign and low absolute value points in the direction of increased adherence to the norm. In fact, at the limit, with a high enough (negative) ratio, one can find moral justifications for breaking all sorts of norms and laws. 

Let’s say the norm in question is: Treat technology as a positive until proven conclusively otherwise: what are the potential first order and second order consequences of violating this norm, according to the AI Safety crowd:

____________________________________________________________________________

First order benefits (+ve)

• Reduce the probability of catastrophic/extinction risk from AI

Second order consequences (-ve)

• Increase the likelihood of norm violation in the future, marginally decreasing the expected value of future innovation by increasing red tape, regulations etc.

____________________________________________________________________________

But most AI Safety advocates would agree this is an incomplete picture. AI is a hot button issue also because the proponents of norm adherence aren’t only concerned about the norm, but also dispute the direction of the first order consequences. The following would be a fairer picture to paint:

____________________________________________________________________________

First order consequences

• Reduce the probability of catastrophic/extinction risk from AI (+)

• Foreclose on or delay massive benefits to humanity given the transformative potential of AI to improve health and welfare (-)

Second order consequences

• Increase the likelihood of norm violation in the future, marginally decreasing the expected value of future innovation (-)

____________________________________________________________________________

But the e/acc folks would probably go further and reframe it this way:

____________________________________________________________________________

First order consequences

• Foreclose on or delay massive benefits to humanity given the transformative potential of AI to improve health and welfare (-)

• Avoid risks that should be discounted heavily due to their speculative nature, assuming our adversaries don’t get there first and still expose us to these risks (-ε)

Second order consequences

• Increase the likelihood of norm violation in the future, marginally decreasing the expected value of future innovation (-)

____________________________________________________________________________

There’s nothing fundamentally dishonest about this framing if one looks carefully at the evidence and decides that it doesn’t meet the bar. But to be intellectually honest, the e/acc side needs to engage on the details of the first order consequences, especially if they agree that AI is likely a transformative technology and thus an exceptional case. The prima facie case warrants looking under the hood, if based on nothing but the primitive intuition that transformational capabilities likely cut both ways? 

Another factor here that’s understated is “vibes”. I believe some part of the reluctance to engage on the specifics comes from the perception of the messengers. The folks delivering the message - Effective Altruists and Rationalists - are dismissed as weirdos (albeit smart weirdos) living in polycules and with a penchant for totalizing philosophies that claim to be free from perverse incentives. Yeah, right, they say to themselves. This is genuinely problematic when it’s combined with the real problem that trying to predict how powerful systems will behave in the future is going to require unorthodox methods of investigation (eg. Tyler’s insistence that this be “modeled”). The subtext here is clear - These are just young people writing abstract papers on the internet who have very little experience doing serious things with serious people.

The AI Safety crowd can defend themselves against these charges. But more pertinently, this is somewhat of a fallacy. If you concede that AI really is a powerful technology with transformative potential, and you prima facie agree we should take a closer look, it shouldn’t matter much that it was EAs or rationalists that first championed this cause. There are plenty of normies who now believe that concern is warranted. The arguments and threat models can be evaluated even if you think some people making the arguments have terrible motivations.

Ultimately, in the AI case, I find myself agreeing more with Robin Hanson than others in the Effective Altruism world (although with very low confidence). To me, the risk of stagnation and fertility decline seem much more concrete and AI as a potential bulwark against those risks is a real factor pushing in the direction of rolling the dice on AI.

However, the reluctance from smart people to engage in cost-benefit analyses of transformative, exponential technologies is worrying, especially since there are domains in which the downside risks FAR outweigh the upside potential - the biggest one being research seeking to identify and characterize pandemic capable pathogens. I’ll be writing about this in my next post.