What is Intelligence? (Long Now Talk)

What is Intelligence is a Long Now podcast episode and talk in which Blaise Agüera y Arcas promotes his book of the same title in which he explains how he thinks human cognition will find a new role similar to how ancient archaea found new ecological niches of greater diversity as bacteria or as symbiotic organisms as mitochondria did with eukaryotic cells within multicellular organisms.

Stats

• Publication date: 2025-10-09
• Video: youtube

Transcript

Podcast Intro

Welcome to the Long Now podcast. I'm your host, Rebecca Lendl, Executive Director here at the Long Now Foundation. Today's Long Now talk with Blaise Agüera y Arcas on the nature of intelligence is intensely refreshing. Through a discussion of what Blaise calls AI symbiosis, he cuts through the established binaries of AI camps. Is it good? Is it bad? Will it take over? Into much more fruitful territory. Essentially, that dominance hierarchy between species is just not how life on Earth tends to work. That cooperation is just as important a force as competition. None of us can do much on our own. Explosions in intelligence have generally been about collective intelligence. In a symbiotic world, things combine to make larger things all the time. It's often unclear where one thing ends and another begins. We might think of humanity in terms of the individual, but we're already not. We're already part of everything we're creating, which is in turn co-creating us. ¶ So might that be part of the story of what's going on with technology and humanity? Are we distinct from the technologies that we make? It's a fascinating and dense conversation. If you're interested in learning more, you'll find a ton of great resources in our show notes.

Now, before we dive in, a quick note. Here at the Long Now Foundation, we are a counterweight, deepening our capacity to move wisely in these times of uncertainty. If you feel so inspired, we hope you'll join us. Head over to longnow.org/join to become a member and get connected to a whole world of long-term thinking. With that, we're excited to share with you what is intelligence with Blaise Agüera y Arcas.

Event Intro

MC: Hello and welcome. I'm Benjamin Bratton. It's really lovely to see you. First of all, thanks from Antikythera to the Long Now Foundation for being such a wonderful partner in these events and in all the work that we've been doing and hope to do together.

Blaise Agüera y Arcas is a VP and fellow at Google, where he is the CTO of technology and society and founder of Paradigms of Intelligence. PI is an organization working on basic research in AI and related fields, especially the foundations of neural computing, active inference, sociality, evolution, and artificial life. In 2008, Blaise was awarded MIT's TR35 Prize. During his tenure at Google, he has innovated on-device machine learning for Android and Pixel, invented federated learning, an approach to decentralized model training that avoids sharing private data, and founded the Artist and Machine Intelligence Program. And so with that, it is my sincere pleasure to introduce to you my friend, Blaise Agüera y Arcas. Thank you.

Main Talk

B: Thank you all so much for being here. It's such an honor to be here. And thank you, Benjamin, for the super sweet introduction. Thank you, Patrick and Long Now for having me. I've been at Google for a long time now, for a little over 10 years, I think 11 years. And for most of that time, I ran a group called Cerebra. It was a part of Google research that began very small and grew to several hundred people. And it was mostly applied AI. We did some theoretical work, but mostly we did a lot of engineering for AI features that ended up in Android and Pixel phones. Things like Now Playing, the song recognizer, and face recognition for the phone to unlock and all kinds of other stuff. We also did some of the models for the Google keyboard, which predicts next words as you type. My assumption when we were working on all of these things is that we weren't really doing AI. That is to say, these are artificial narrow intelligence, A-N-I. And the reason that that term was coined was because when AI was originally coined back in the 1950s, it meant what we all thought as kids, that we'd have robots that you could have an interesting conversation with. That didn't happen throughout the 20th century. There were lots of AI winters defunding after various failures, of sort of program-based approaches to AI. And so, you know, hope was kind of dwindling, but neural nets were starting to work for doing very limited forms of visual perception, things like face recognition, next word prediction. And so we called those things AI, but in order to distinguish them from the robots you could have an interesting conversation with, we used the term “narrow”. The G in artificial general intelligence meant everything else. You know, the real thing. The so-called core AGI hypothesis, as Ben Goertzel, a computer scientist, wrote, is synthetic intelligences with sufficiently broad, that is human level, scope are qualitatively different from synthetic intelligences with narrower scope. In other words, AGI is not A-N-I. This is what I believe too, that probably some insight would come from neuroscience. That's where everything that had worked up to that point had come from, all of the key advances in NAI came from neuroscience. And I thought we'd figure out the trick. You know, we'd figure out from studying brains, which are the only intelligent thing, the only truly intelligent things we know about, what the secret to real intelligence was. You know, I was hopeful that we were at least on the right track, sort of, because we were using neural nets, which were brain-inspired, and they were doing some things that earlier program-based techniques had not succeeded in doing.

I was wrong. The first real inkling I had that I was wrong was when I started to see these kinds of outputs from models like Meena, which was a scaled-up next-word predictor, not unlike the one that we'd written for the Android keyboard, just a lot bigger. And this is Meena trying to define “philosophy” in a conversation with a person. All it's doing is filling in the next word, statistically, based on the previous words. It's just a much bigger model and trained with a lot more data, unprecedented amount of data, an unprecedented size of model at the time. You know, you could actually start to compare the outputs of models like Meena to human outputs in terms of how sensible they were, how relevant to the dialogue, to the conversation, and this was really a shock. It started to look like maybe the key to artificial general intelligence was really just scale. And I was quite snobbish about this idea that I'd heard in Silicon Valley that, you know, everything was just scale and making stuff bigger that just seemed incredibly naive you know my training was in neuroscience and physics and and so the idea that that you know just because we could make bigger computers and worship at the altar of moore's law that was going to solve all of the problems in science and technology just seemed ridiculous. But, the nerds were right. We scaled that up further with a model called Lambda in 2021. I wish we had launched it before OpenAI launched their model but… innovators dilemma. It did even better. This was a model that that could have general open domain conversations about pretty much anything and you know it sometimes sucked and uh sometimes uh went off-piste and uh you know gave you nonsense answers but at the same time well… people do too sometimes. And this has been the story since. So you know models had been getting bigger exponentially by a factor of about one and a quarter per year since 1950 but around this period of the of the sort of unsupervised learning revolution that slope ramped upward dramatically to 3.72 times per year and that's where it's remained since. An absolutely explosive growth in model sizes now that we knew that making these predictive models bigger made them better so it seems to me that this core AGI hypothesis is wrong. I know that this is a very controversial thing to say. We're still having all kinds of conversations about when agi will arrive. I think that there are several reasons that we're asking that question but what i would ask you as a thought experiment is if you took any of today's frontier models and you just transported them back in time to roughly the year 2000 when the term AGI was coined to distinguish it from artificial narrow intelligence what do you think the people who coined AGI would have said? Would they have said yeah you've arrived this is it? Of course they would have. So why haven't we admitted that why haven't we acknowledged it well it's because we all thought that there would be a trick and there wasn't a trick. And because there was no discontinuity and you know it's an exponent it's fast but it's also continuous there's no moment when it was clearly not intelligent before and an after when it clearly was and we also still at some level don't know why scaling it up worked and there may be some other reasons as well that have to do with our insecurities but i think that that's that that these these reasons these kind of frog boiling reasons are are part of the are part of why.

That begs the question if we're going to be brave could we be massively computationally scaled next word predictors as well and i'd like to provoke you over the next 41 minutes 22 seconds with the possible answer: yes.

So why would a brain evolve to be computational? I think that one of the big insights that we've had on the team is that it's really not just the brain that evolved to be computationally computational, but life itself that is computational. That's something that I know takes some getting used to as an idea, because we think about life as being the exact opposite of computers. It's squishy, it's wet, it's unreliable, it doesn't run anything like a program. So what on Earth do I mean when I say “life is computational”? Well, the old idea about life from the 19th century was that there was some kind of vital force or spirit that animated life and that made it different from matter like a rock. That went out of fashion in the 19th century, of course, and in came materialism, strong materialism that says, no, the rules of physics are the same for the atoms in living bodies and in rocks. It's just physics all the way down. And therefore, there is no difference between living and non-living matter. Well, that's not very satisfying either, because there sure seems to be a difference between living and non-living matter. So what could it be if it's not physics and if it's not some vital spirit either?

There's an answer to that question, I think, and that answer is: function. What do I mean by function? Well, one way of telling whether something has a function is to ask whether it can be broken. In other words, if I split a rock in half, it's not like I have a rock that's broken. I just have two rocks now. Whereas if I destroy a kidney, I break it in half, then I have a non-working kidney now. A kidney has a function, and a rock, at least a rock on a sterile world, doesn't. What I mean by that is that if I came back with this object from the future, I were some time traveler, and you asked me, what is that thing? I'd tell you it's an artificial kidney. It has an operating lifetime of 100 years. You can implant it, and it filters the urea just like your kidney does. That means something. It means that, well, for one, if you have kidney failure, you're going to live. It's a very real statement. It's not mystical. But at the same time, there is something interestingly spooky about function, because it's not something that that matter tells you in isolation. It could be made out of carbon nanotubes. It could be made out of tungsten filaments, who knows what. The point is what it does in the context of the rest of your body, what its relationships would be with the rest of the body in normal functioning order of things. And so it's a relationship, a set of relationships. It's kind of ecological, if you think about it. It's something that is beyond the physical matter. And yet, also very fundamentally constrained by the physics of our world. It's not like it has some kidney spirit, right? It just works. “Works” means functions. This idea of function as something fundamental was really pioneered by Alan Turing and John von Neumann, by the founders of computer science. They were mathematicians. They thought about functions all the time. If you recognize this device, it's an actual instance of a Turing machine. When Alan Turing invented the Turing machine, it was a purely conceptual invention. It wasn't intended to ever be built, but Mike Davey did in 2010. A Turing machine is a device that has a head that moves left and right on a tape and reads, writes, and erases symbols on that tape according to a table of rules. That's all a Turing machine is. But what Turing showed is that any computation you could do, any calculation you could do with pen and paper, can be done by a Turing machine with the right table of rules. And then came the sort of genius part. In 1936, he also figured out that there were certain tables of rules such that if you wrote down another table of rules as symbols on the tape, then this table of rules would interpret the table on the tape and compute the same thing that that machine would have computed. And that's what makes a universal computer. In other words, there are certain machines that can run programs, and those programs can do any computation, not just a particular computation based on whatever table you've got. And that's a really, really interesting discovery, because now not only has he said there is a way of specifying a function, but also that any machine that can run a program is calculating the same thing given the function that the program performs. So if it's adding two numbers together, there are many programs that could add two numbers together. There are many languages, many ways of specifying the table for that. They're all equivalent in terms of the function they compute. They're functionally equivalent.

This is cool, but von Neumann did something further, which is he introduced the idea of embodied computation with something called cellular automata. The idea here is rather than having a tape and a head, which are made out of something fundamentally different from the information that is written on the tape, he said, let's imagine a world in which the tape and the head are actually part of what is written. In other words, these worlds have a kind of physics. They're generally rendered as grids. If any of you are familiar with Conway's Game of Life, that would be an example of a cellular automaton. There are very simple rules or physics for how each grid cell changes based on the values of the grid cells around it. And you can write programs, essentially, by configuring the states of those grids of cells. The reason that von Neumann was thinking about these kind of very, very simple two-dimensional physics is because he was thinking about the problem of life. And in particular, he says, suppose that you are a robot, made out of Legos, and you're paddling around on a pond that's full of loose Legos, and you want to assemble another robot like yourself out of those loose Legos. How is that possible? Because that's, of course, what life does. That's what every mother has to do. It's what has to happen in the seed of every plant. It's what every bacterium has to do in order to divide. And it seems a little bit paradoxical that you could make something just as complex as you yourself are from parts. And so what von Neumann realized is that in order for that to work, you had to have inside, yourself, a “tape” with instructions for how to build yourself. And you had to have what he called a “universal constructor”, which was a machine that would walk along the tape and execute the instructions on the tape in order to make whatever is written there. And you had to have a “tape copier”, a second machine, and the instructions for building the universal constructor and the tape copier had to be on the tape. If all of those things were true, then you would have something that could reproduce. He made all of those conclusions. He made those predictions in 1950 before we had discovered the structure and function of DNA, which is indeed exactly that tape, before we had found the ribosome, which is the universal constructor, and before we had discovered DNA polymerase, which is that copier. So all of those things, you know, he was exactly right. He was right on point. But the really cool thing is that he also showed that the universal constructor is a universal Turing machine. They're one and the same. It's just a universal Turing machine where the things that it computes with are the actual matter that it is made out of. So it's an embodied computation.

And with that, von Neumann proved that in order to have life, you have to have universal computation. You can't reproduce without computation. No computation, no life. And this is a really profound insight and one that I think most biologists and most computer scientists still are unaware of.

So we began doing some experiments a couple of years ago. We published some of these, which attempted to find out how life in that very minimal von Neumann sense could emerge out of non-life. And these are some of those results. We had to use a very minimal Turing complete language in order to implement this. So, you know, I did some of the first experiments and I picked a language called brainfuck. I didn't just do it because I do love, you know, getting in front of a lot of people and saying “brainfuck”. I admit, like, I'm a 12-year-old on the inside. This is a brainfuck program. And you can see that it's very, very hard to understand. But it's very closely modeled on a Turing machine. So it has only eight instructions. And those eight instructions are

• move the head one step to the left,
• move the head one step to the right,
• increment the byte at the head,
• decrement the byte at the head.

And we're already halfway through them. That's like four of the eight. So it's a very, very, very simple language. But you could write Microsoft Windows in this if you were non-human.

Okay. So here's the experiment. And this experiment is called BFF for reasons that I will leave as a exercise to the listener. We begin with a bunch of tapes filled with random bytes. The tapes are 64 bytes long. And they're just filled with junk, filled with noise. Remember, there are only eight instructions. And a byte can have one of 256 different values. So only one in 32 of those bytes is even an instruction at all. The rest of them are “no ops”, meaning nothing will happen when it gets executed. The head will just move on. So this is random tapes. That's how it begins. We pluck two of these tapes. And here I'm using 8,192 of them. Many of the experiments, you can use only 1,000 of them and all if this works. So 1,000 tapes of length 64. You pluck two of them out of the soup at random. You stick them end to end. And you run. And then you pull them back apart and put them back in the soup and repeat. And that's it. You just do that.

In the beginning, nothing much happens. I'm printing here the first couple of dozen tapes and only showing you the instructions. The rest of them are no ops. So they're one of those 31, 30 seconds of the bytes that don't code for anything. And the average number of instructions that runs when you put these two tapes together is two. Because there are just not very many instructions there. And I can't see any loops here.

So what happens when you let this thing go? I'll show you. This was actually the first time I got it to work on my laptop. And it was pretty exciting because laptops run really fast nowadays. And you go from noise to something really magical which is that suddenly programs emerge. And these programs are complicated. In order to understand what they're doing, you have to really pick them apart and reverse engineer them. And they've got all these loops in them, and– you know, what on Earth are the programs doing? Well, you can tell right away that they have to be reproducing because you can see that some of these are duplicated many times. There are 5000 instances of that tape on the top and 297 of the next one and 99 of the next one and so on. So, they are copying themselves or each other. And there's a lot of computation happening. There are now 4,784 operations happening per interaction on average. So you've gone from something non-computational and full of noise and junk to something complex, computational, and functional. Functional meaning it can break like a kidney, right? If I change one of these instructions, then it will cease to work. What happens if it will cease to work? Well, it won't copy itself anymore. And so that tape will get overwritten by something that will copy itself.

And that kind of tells you why life evolves. Life evolves because in a universe capable of computation, if you figure out somehow how to copy yourself, then you will exist in the future. This is that old joke about DNA being the most stable molecule in the universe, even though, of course, it's very fragile, right? If it reproduces itself, then it's still going to be around in the future. Whereas if you are not able to do anything to function, even if you're very robust, like a chunk of granite, the best that can happen is that it'll take a long time for you to fall apart.

So that's why life persists. We go from here to here. And it doesn't even take that long. This is after 5 million interactions of 8,192 tapes. This is what that looks like. I'm drawing here a dot. I reversed the colors. So this is white dots on a black background. There's a dot for every one of the first 10 million interactions of the soup. And time is on the x-axis. And the y-axis is number of operations that ran. And you can see that right about at 6 million interactions, something really changes about the soup. It looks like a wall of white. That's what's on the front cover of the book. That is a phase change. It's a phase transition. If you think about this like a physicist, what's on the left is like a gas, meaning that all of the bytes are decorrelated from all of the other bytes. They're all independent. And the way you can see that they're decorrelated is if you try running the soup through zip, right, you compress it and it's uncompressible because if you have a bunch of random bytes, they don't compress at all. Whereas right after that transition, you can compress the hell out of it. It compresses down to about 5% of its original size. It's obvious that it'll compress if there's a bunch of copying going on, because you know, anytime things are copied, then it doesn't, you don't have to write all the bytes out, right? You can just refer to one of the originals. So it's a phase change. If the phase on the left is gas, what is the phase on the right? It's life. You could call it, “machine phase”. You could just call it life. Life is a very special phase of matter because unlike a solid or a gas or a liquid, it has structure at every scale. It's got complexity that looks different when you zoom in or when you zoom out or when you look at a different place.

So the tentative conclusion is that pretty much any universe that has a source of randomness and can support computation will evolve life. But there's, there was really a puzzle in these results, which is, how on earth does it happen so fast? How can we get these really complicated programs in only a few million steps with only a thousand tapes of length 64? It just seems implausible and it seems especially implausible because in the original experiments I used mutation. So I imagined that there were, you know, sort of cosmic rays, you know, randomly changing a byte here and there every now and then. But this actually still works even if you crank down the mutation to zero. So with zero mutation, you still get life. It takes a little bit, a little bit longer, but not much. And actually the life that you see keeps on getting more complex. You, if you were looking closely at the running program, you might've seen that you saw structure emerge and then you saw more structure emerge and more code come in. How on earth could that be happening? Because once things can copy themselves, you would think you're done, but it's not done.

Well, the answer I think comes from a very, very fundamental result in biology, which Lynn Margulis figured out in 1967 that her paper in which she wrote about, this result was, was rejected from a lot of journals before somebody finally accepted it, a journal of theoretical biology. And it was called On the Origin of Mitosing Cells. She was the one who proved that mitochondria were once free-swimming bacteria. And she popularized the term “symbiogenesis” to talk about what, what was going on here: that two life forms that previously were independent came together and made a new life form. An archaea and a bacteria came together and made a new single celled life form, which are the eukaryotes that we are all made of. Margulis believed that this process of symbiogenesis was the engine behind evolution. Turns out that she was right about about mitochondria and the establishment and biology sort of finally came to recognize that but nobody really bought her larger thesis that this was the engine behind evolution and she remained very much in the in a tiny minority of people who believed that even by the time of her death in 2011.

So could symbiogenesis be happening in BFF? Yes, it is happening and the way you can see that is by looking at not whole tapes reproducing but little little strings reproducing maybe only one byte reproducing occasionally a single byte will reproduce even right from the beginning because if you have instructions that can change a value somewhere else in the soup, once in a while an instruction will change a value somewhere else into another instruction and that's a very very lame but non-zero form of reproduction. So you have these little things reproducing from the beginning and what what I'm showing you here is all of the reproducers in a particular soup but you can see that there's actually a lot of stuff happening during this phase that looks gaseous which is the reproduction of smaller things and when you track what's happening with those smaller things that are reproducing you see they're coming together and then those things are coming together and then those things are coming together. So there is actually a sort of inverted tree of life. We think of a tree of life as something that splits from an ancestor into into descendants but this is a tree that goes the other way it's like the roots of a tree. Things come together and symbios and form larger things. That's exactly how the complexity happens. And symbiosis is what gives evolution its arrow of time because, if you think about it, evolution in the standard Darwinian sense doesn't have any sense of more or less complex. If you evolve you will fit your niche better but that doesn't mean you'll get simpler or more complex on average. The average is roughly zero. You might change your beak shape to adapt better to this or that flower. But when you have a symbiogenetic event, two things that already are reproducing themselves come together and can reproduce together and that means that some extra information has to get added in which is how do we get on together? How do we fit together? And it's that extra information that's adding to the complexity of what comes next. And that gives evolution's arrow of time. We know that there have been a number of other major evolutionary transitions where things came together to make stuff more complex. For instance, we are multicellular and that was a major symbiogenetic event, right? How did how did single cells, single eukaryotes, become multicellular animals like us? It's obviously a symbiogenetic event. Eörs Szathmáry and John Maynard Smith wrote an article in Nature in 1995 that reviewed what they what they saw as the eight major transitions in life on Earth. These are definitely all a big deal. They've added a few to their original list. But if if what we're seeing in in systems like BFF is any indication, this is actually something that happens all the time. It's not just these major transitions. There is a whole cascade of mergers and combinations that are happening continuously and they are actually what leads to the complexification of life as a whole.

Okay do we see any actual evidence of this in biology? Well this is very much ongoing area of work but here's some evidence. This is the human genome. And the big surprise when we first saw the human genome sequenced in in 2001 is just how little of it actually codes for the that make us up. It's only about 1.5%. The rest of it is so-called junk DNA. It's not really junk. Some of it is regulatory. Some of it we don't know what the hell it's doing. But what's really interesting is that those big sections called LTR retrotransposons and DNA transposons and lines and signs, that's all viruses. Basically, it's replicators that replicate inside our DNA and that have burned themselves, not only our somatic DNA, like a classic retrovirus, but into our heritable DNA and become part of our genome. We know that some of those viral elements, endogenized viral elements, are doing really important work. For instance, the placenta is made out of a virus that fuses the membranes of cells together. We know that there is a virus called ARC, which if you knock it out in mice, they stop being able to form memories. We know that parts of the immune system were made this way. I mean, there are a few dozen results like that that have all been coming out in the last 10, 15 years, and there are more and more of them all the time. And when you look at our DNA, it doesn't look like one thing that has been copying itself, it looks like a medley of things that have copied and fused over and over. It's not just neuroscience that's computational. Life was computational from the start. And, it gets more computationally complex over time through symbiogenesis. Right, because we put together the two ideas that I've just shown you, that, you know, life is always computational because it has to copy itself, and that's a general-purpose computation, and the fact that symbiogenesis is really important, well, you now have two computers that have come together and parallelized. And what that means is that you have greater computational power every time you undergo a symbiogenetic event. So symbiogenesis makes the computation massively parallel. It's not quite the same Moore's law that we had on Earth in Silicon Valley between 1950 and 2006 because, you know, then we were making transistors smaller. By the way, AI didn't progress anywhere between 1950 and 2006. But when transistors stopped becoming–transistors were still getting smaller, but we stopped being in a situation where making them smaller could make them clock faster. And so around 2006, all the chip makers began to do the only thing they could, which was to put a lot more cores on the same chip and parallelize, and that's when AI began taking off. This is not a coincidence. Parallelism is exactly what it takes in order to make neural net-based AI work. And that's why the deep learning revolution happened when it did.

So computing for growth and healing and replication is modeling your own body. That's life. What about modeling your environment? That's also needed in a dynamic environment. Well, that's what intelligence is, of course. Life was intelligent from the start because, of course, you don't just have to make more of yourself. You also have to find the parts to make more of yourself. The LEGOs don't necessarily just float around around you. You maybe need to find them, hunt them down. What about the energy that it takes to compute? Computation is energetically expensive. You're creating negative entropy when you compute, and in order to do that, you need to ingest free energy. That's why we all metabolize, because we compute. That's why when I run BFF, my computer heats up. If you run simulations in which you just take random programs that can swim left, right, up, or down, and you just see which ones survive in an environment where they're getting energy from that light, the ones that survive are the ones that learn to follow the light. That is really just a way of saying you have to model your environment too, and you have to figure out how to make your behavior consistent with one that will allow you to do the copying that will allow you to reproduce. And that, sure enough, that's exactly what bacteria do as well. That's a sugar crystal in the middle, and bacteria that swim have learned how to swim toward the sugar.

I've been talking so far as if we're in single-player mode, but, of course, whenever you have one bacterium, you have more bacteria. And if you don't have more bacteria yet, you will in 11 minutes, right? So life is a multiplayer game, and it's never single-player. The most important parts of our environment to model are each other. A lifeless universe is one where you don't have to think very hard, but the moment you start to have a lot of other agents in your environment that have their own energy that they have to get, their own stuff they've got to do, your interests can align with theirs, can misalign with theirs, and now you've got to get smarter because you don't just have to model yourself, you also have to model them. And they're modeling you back. So we've been doing a bunch of work recently on the team– This is actually not in the book because it's a little too recent, but it's called “Multi-Agent Universal Predictive Intelligence”. And this work is really about the field called multi-agent reinforcement learning, in which you have a bunch of learners that are all trying to learn to do something together based on, being individually reinforced on the basis of some score that they get. And the question is, you know, how can they learn to work together? How can they solve things like the prisoner's dilemma? Well, that turns out to be a very, very hard problem for classical reinforcement learning because ordinary reinforcement learning only learns from the past. And that's fine if you're playing a video game and the video game stays the same as you adopt a new strategy. But if there are other players in that video game world with you, then when you change your strategy, they're going to notice and change their strategy. So the statistics of the environment are not constant. And they're learning too. So you have to learn about them and you have to learn to predict what they're going to do in response to what you do. And you have to learn that theyʼre learning. And that they're also predicting you. And that they're predicting you predicting them. And that you're predicting them predicting you and so on. But this is a really hard problem.

And the paper, it's gotten to be a hundred pages or so, and has some very, very complex math in it because modeling an environment that includes the thing that is modeling the environment and all the things in the environment that are modeling you back, turns out to be a difficult problem. But the team has figured this out and the results are really cool. And the way you do this is by getting rid of the idea that you are outside the video game and putting yourself in the video game. So in other words, you have to not only model the environment like AlphaGo does, where you're thinking about a Go game or a chess game, and you're just imagining the game, you have to imagine yourself playing the game as part of the environment. And you have to start to predict yourself and predict others. The reason is that we have a face, if you like, you know, when I smile, I know what I feel like on the inside because I've built a model of myself. And when I see you smile, I can guess that you're happy too. And the only way that I can make those kinds of inferences is by knowing that we're similar, by knowing that I also have a face and I do that when I'm happy. And it's that ability to empathize, to model the minds of others that is at the core of being able to solve the multi-agent reinforcement learning problem. I think that this actually kind of explains why we've got consciousness in the sense that consciousness is often thought about as some kind of weird epiphenomenon, you could have a philosophical zombie or something that behaves identically to us but is dead on the inside. I don't think that's true at all. I think that the reason we are conscious is because we are modeling ourselves, as well as modeling others as well as modeling others model ourselves and so on and so forth. Because that is behaviorally essential; because it's functionally essential in order to allow us to cooperate with each other. And when you do that, when you embed yourself in the world and you think about others like you, then you're able to solve problems collectively. And this is essential in order to have symbiogenesis; in order to have symbiosis with those others and in order to create a larger entity. I'm not saying exactly that i think that your cells are conscious but I'm saying that they definitely have models of the rest of your body or of the other cells around them in order to be able to collaborate with them. And that's you know maybe a baby step in a certain way toward consciousness. When it comes to very complex big-brained animals like us which have tons of neurons that have come together through an act of symbiogenesis and we want to work together in order to make bigger things happen when we talk about human intelligence we you know we imagine things like we figure out how to transplant organs and how to go to the moon and how to build computer chips. None of us can do these things on our own. Tthat intelligence that we're talking about is the superhuman intelligence of our collective symbiogenetic entity. And in fact it's not even just a human entity it includes cows and wheat and all sorts of other entities as well as steam engines, by the way, without which we wouldn't exist. That super entity which has arisen through us being conscious enough of each other to build models of each other… that is what has resulted in this in this explosion of intelligence in what we think of as humanity over the past 10 000 years. And that's that's also what allows one to solve the psychological twin prisoner's dilemma, meaning cooperate a priori with another actor in order to solve these game theoretic puzzles that involve mixed payoffs. A very old classic problem if you think about this from the classic perspective of game theory as actually John von Neumann invented and as was refined by John Nash later on in the 20th century; this is sort of rational economic actor ideas about how people interact if they were just optimizing for themselves. The solutions are very grim. These Nash equilibria are essentially selfish and prevent any collaboration. But if you imagine that others are like you and also will change their strategies in response to your strategies and so on, then a new set of equilibria emerge from this kind of thinking that are that are much more cooperative.

So symbiosis and symbiogenesis requires modeling ourselves and modeling each other. And we have to think about each other as if those others were like ourselves. That's where theory of mind comes from. By the way, large language models have theory of mind. They kind of have to in order to be able to carry on conversations, right? So when you're interacting with a large language model you have to think about what you've got to tell it and what you don't have to tell it because it already knows and and so on and it has to do the same thing back to you in order for that interaction to succeed. Those things have been learned by observing tons and tons of interactions between people which is what the training data consists of. So, starting with simple bacterial quorum sensing and multicellularity and so on, since every living entity is computational, as they combine, they parallelize and that does lead to a kind of Moore's Law. And it leads to more and more coöperation on larger and larger scales. There really is this kind of Moore's Law progress that I was so dismissive about when I first heard it down here in San Francisco.

These increases in brain size that have happened during human evolution are a result of exactly those dynamics. This is the last seven million years or so. There have been explosions in brain size. Those have been observed in various other social species as well, in cetaceans and whales and dolphins and in bats, in certain species of birds. And the reason is that if you share DNA with another entity of your species and you get smarter to model them, you also become harder to model and they're getting smarter as well. So now they have to model you back and it's a kind of friendly arms race. Well, how friendly it is depends, right? You're also competing for mates and prestige and all kinds of other Machiavellian stuff, but also you're trying to collaborate, right, in order to get things done collectively. And all of that leads to an explosion in intelligence. This is some classic results from Robin Dunbar showing the relationships between cortical size and the size of troops among monkeys and apes. They're correlated, of course, because if you're able to model more others, then you're able to form a larger troop before it falls apart. That's why having a bigger brain doesn't just let you have a larger troop, but also have greater collective intelligence, which then forces the brain once again to get bigger. So scaling cooperation and competition is how we got these big brains. It's also how we came to be able to recognize ourselves in mirrors. As many of you probably know, the mirror test in which an animal, you know, is able to recognize in the mirror, not just that that's another chimp, but that that's me in there, and then check themselves out. But, you know, there are only a handful of other animals that do this because the level of sophistication you need in order to realize that, you know, not only are there other beings like you in the world, but that you are also a being like the other ones that you see and to be able to sort of make that mapping, right, that that's you in the mirror is quite a sophisticated act of theory of mind. And we do it together with each other all the time. When you think about a rowing crew, for instance, and the way they can sometimes achieve what people in crew call swing, where, you know, suddenly they get in sync so perfectly that everybody is anticipating the behavior of everybody else perfectly. And it feels like the boat acquires a kind of soul, if you like. That is basically a computational process in which they've achieved a kind of group consciousness.

I just want to say a thing or two about human–AI symbiosis, because that seems to me where we're headed. I hear a lot of talk among two camps about AI and our future with AI. Some people, more aligned with ideas about AI ethics, think that AI is fake, that it's not real intelligence, or that this is somehow a counterfeit version of intelligence or just statistics, and are concerned with various issues about justice that are related to how AI behaves or fools people into thinking that it's real. And then there are the existential risk folks who have gone from being rapture of the nerds, you know, we're all going to go to heaven and be immortal and upload our brains, to the apocalypse is coming and we're all going to die because the AI is going to take over. And I think that these are both wrong perspectives. The idea that there is a dominance hierarchy between species is not how things have tended to work in life on Earth. And I think that we've been fooled into thinking that because of an overly classically Darwinian perspective on how evolution works. You know, if you're just doing classic Darwinian evolution, then a mutation is something that is only a little bit different from the wild type, and they will compete and whichever can outcompete the other one wins and the other one dies. But in a symbiotic world, in a symbiogenetic world, things are combining to make larger structures all the time. And it's not so clear where one thing ends and another begins. And cooperation is just as important to force as competition. And I think that that's very much the story of how we came to exist. And as far as I can tell, that's very much the story of what's going on with technology and humanity as well.

Now, I mentioned near the beginning of this talk, that if there were no machines, most of us in this room would not be here. We were about 1 billion people around the time of the Industrial Revolution. And right after those machines, which externalize metabolism by burning fossil fuels, right after they came on the scene, our numbers exploded by nearly a factor of 10. Why is that? Well, we know that we make the machines, but also the machines have made us. Marx and Engels talk about this when they talk about the people springing out of the ground like wheat. That is literally true. It's all of that additional free energy that came from burning fossil fuels that resulted in all of the humans that we've got. And not only did it result in much greater numbers, but this plot, which is from an economics paper just published very recently, shows on a log-log scale, the real wages of people versus the population. And what you can see is that throughout the Middle Ages, we were oscillating, trading off between population and wages. This was essentially a Malthusian trap. In other words, we were constrained energetically in our numbers. The population was constrained. And the moment we began to metabolize externally, we shoot off to the right. Suddenly, both numbers and quality of life rise dramatically because of all that extra energy that is liberated. Because that's what intelligence ultimately does. Whether it's photosynthesis or the invention of steam engines or of nuclear power and so on. The more intelligent you become, the more sources of energy you're able to tap and the more additional levels of symbiogenesis you're able to achieve. So I see no reason to believe that AI is poised to be any different from all of those previous symbioses. I also don't see us as being distinct from the technologies that we make. We think of humanity in terms of the individual person, but we're already not. We're everything that we've made and that has co-constructed us. And we didn't achieve artificial intelligence until we literally began to train it on all of the human output and text that we've generated all over the Internet. What could be more a part of us than that?

I'm going to end there, and Benjamin and I, I think we'll shift into a conversation mode.

Conversation

MC: All right. Well, you certainly gave us plenty to talk about. I don't think we'll have a problem with this. I wanted to talk a little bit about the arrow of time, and particularly the arrow of time as one that operates, that we can map through not just increasing complexity, right? Life is the ability, you know, fighting entropy and so forth, the increasing complexity, but also increasing complexity that seems that goes through phase transitions.

B: Yes.

MC: Right. And so, but this phase transitions are ones that, and I think you've made the point quite clearly, retains what came before, right? It's not just like, okay, done with the old, here's the new, but that all of this that came before is already, it's all inside us.

B: Yeah.

MC: It's all still here.

B: That's the amazing thing. Like we are actually, you know, societies of bodies, of colonies, of conjoined bacteria. You know, all of us are just bacteria in this room, right? They're still here. They're just nested like Matryoshka dolls. And even if you zoom in, you know, you zoom into the bacterium, and then you zoom further into the mitochondrion, what you actually see reproduced in the mitochondrion, Nick Lane made this point very beautifully in his, in his book, Transformer, is the conditions of the deep sea vents where those mitochondria first evolved. So it's almost like they've artificialized, right, in your language, the environment that they originally evolved in and then they (garbled) capsules around themselves. So yeah, it's, it's, it's sort of shells within shells within shells.

MC: And this, I mean, this, in your mind, this is since Sara Imari Walker's book here earlier, in your mind, this is, this rhymes with assembly theory's idea of the sort of the persistence of these things over time.

B: It does. Yeah. So, I mean, and the same is true of technology, by the way. So if you look at, I, I give in, in, in, um, in the book, the example of the “hafted spear”. Uh, so, you know, if you have a, um, stone point, uh, at some point there's this innovation in which, uh, some clever cave person decides to tie it with sinew to a stick, and now you have a spear.

MC: Mm-hmm.

B: So you can't have a spear before you have stone points, just like you can't have a eukaryote before you have prokaryotes that can come together.

MC: Right. And this is the arrow. Like there is there's a certain degree of non-reversibility.

B: Yes. And that's exactly why it uses energy because anything that is irreversible consumes free energy.

MC: That's right. Okay. So here's what I need to say. In the Szathmáry and Maynard Smith slide, that you showed, like they identify –speaking of these phase transitions– the eight key transition, what they see is the major transitions in evolution, right? And you are point to these and I think kind of show how each one of these is built on symbiogenesis.

B: Mm-hmm and, and they, and they said, they said that as well.

MC: And computer genesis too.

B: Yes. That they didn't say.

MC: That they didn't say. Um,

B: But, but they, they, they also, um, uh, you know, so, so they were– If you, if you like halfway between, you know, what Margulis said, which is symbiogenesis is important and what I'm saying, which is it happens all the, all the time.

MC: Yeah. Right, right, right, right.

B: And, and so they're identifying some of the really big ones. But when you start zooming in, you realize that they're happening all over the place. I mean, every one of those lines and signs and endogenized viral (garbled) is one of those events. Or even, like, termites. The fact that termites can eat wood is because they have their, they, they engaged in the symbiogenesis with a, with a, uh, an organism in their gut that actually does the digesting of the wood. So, um, you know, it's sort of like a, um, a power law, you know, where they're looking only at the, at the top right of that power law, but it's an entire–

MC: –So it's a gradient, it's a gradient all the way down.

B: Yeah.

MC: Okay. But speaking of stages and, and these sorts of phase transitions here as well, like you showed with the, with that 6 million operation for the brainfuck. Looking at, when I asked you a little bit about to ask you to prognosticate a little bit about the future of intelligence, do you see the, the longer term, the symbiogenetic relationship between evolved human intelligence and mineral based intelligence that we, that we have constructed as something like, um, um, the ninth stage?

B: Yeah, I do. Um, I, I think.

MC: And why so? Like what would be the criteria by which one can say yes or no to that?

B: Well, I guess, um, how big a deal it is on a planetary scale.

MC: Please.

B: Um, you know, termites were a big deal, but the industrial revolution was maybe an even bigger deal. Um, and, uh, and I, I think that, um, you know, the fact that, that these big deal changes are happening more frequently, uh, by the way, is also something you would expect from the dynamics.

MC: And increasing complexification.

B: Right. Right. Because the more things you've got that have come together, the more parts you've got on the table that can now come together. Uh, W. Brian Arthur has talked about this in the context of technologies.

MC: The technological evolution. That's right.

B: Same exact process.

MC: That's right. That's right. All right. Um, is there anything else you would want to say about, before we move on from this, about the future of intelligence? Like where do you-

B: Other than it will grow.

MC: Other than it will grow and it'll become increasingly complex and that we will be scaffold for something–

B: Yeah.

MC: –that, that we'll– we humans will continue– I mean, I, I'm just to sort of like frame the question that– as opposed to thinking about– a lot of times, the way in which this is thought through is in terms of a language of post humanism.

B: Right. That there's humans, they had the run and now there's gonna be something else that takes over. Right.

B: Right.

MC: Even [[James Lovelock|Lovelock in Novacene.

B: And I do disagree with this perspective.

MC: Because humans, because everything persists.

B: Because everything persists. Everything is still there.

MC: Please draw it out.

B: Well, um, you know, there are still bacteria after there are eukaryotes.

MC: Right.

B: And in fact, the number of niches for bacteria and the, and the varieties of bacteria have greatly increased as a result of eukaryotes coming on the scene.

MC: Right, right, right.

B: And the same is true of, of, of eukaryotic single celled organisms when multicellular ones come along. You know, suddenly the guts of multicelled eukaryotic organisms are these incredible new environments and, and they, you know, create all kinds of other environments, right, for a single celled life. So the niches and the environments grow and the things that were there before generally are still, are still there in the future too.

MC: So that symbiogenetic relationship would be one in which there would be a construction of new niches of which we would be part and we would persist as part of a larger complexity that we are, in fact, ourselves bootstrapping in a way. Is that a fair way to find what you're saying?

B: Yes. I don't want to sound too Pollyanna. I mean, you know, there are, you know, aggressive symbioses. There are die offs, like, you know, dramatic things happen in the history of the earth as well, right. There are collapses. I don't want to minimize any of that. But, but the, the pattern, you know, modulo that there are snakes and ladders is that, is that things get more and more composite and more and more complex, right. The idea that, because there is a new kind of entity, we're going to get replaced by it, strikes me as, you know, using dominance hierarchy thinking, which is like all about how like monkey A, you know, decides or doesn't decide to fight with monkey B for the mate or something, you know, like generalizing that idea across species and bio (garbled) doesn't make any sense.

MC: Yeah, I find it interesting thing that your book and Yudkowskyʼs book come out around the same time. There's a bit of a…

B: There will probably be some shared readership.

MC: We'll have to set up some sort of–

B: Cage match.

MC: Fisticuffs, yes, I think on this as well. (laughter)

MC: I was also struck by the line that you said, um, what it's like to be a next-token predictor, right. Which you know, a certain kind of philosopher would call qualia right. Or this experience of experience or ones of your experience of your experience or something. And the way in which you set this up is that: well we know the answer to what it's like to be a next token predictor because we know what it's like to be us.

B: Yes.

MC: Um but transformer models and all of their their their descendants uh are also next-token predictors, in a way. Um, without using the C word, necessarily, that that is consciousness–

B: You're gonna get me in trouble now, Benjamin.

MC: No I don't want to because it's this it's such a loaded term that it comes with such, um, uh, baggage that may not really be what we're looking at right? I, you know, as we– I think discussed like there it's part of the reason why a new kind of school of thought is needed because there's all these things happening right in front of us that we all point to but we're all kind of arguing over, like, which 17th century word we should use to to call it.

B: Exactly.

MC: So maybe C is not so helpful here. Anyway I'm just– but What is your intuition, if that's the right word about what kinds of similarities and differences there might be between being one kind of next-token predictor versus being another kind of next-token predictor and is there another way in which you see that kind of spectrum of difference and similarity that doesn't require, you know, maybe do some sort of legacy metaphysical legacies to get at it.

B: Well I will try and map this a little bit onto that legacy.

MC: Yeah

B: So um there are qualia, in philosophy speak, which are like red, uh, you know, or apple, and not just “apple” but like you know what an apple is like and what it's like to crunch into it and so you know what redness is like. You know, why do we have experiences of “red”? Well it's obvious why we have experiences of “red” because it's behaviorally relevant for us to have those experiences. You know, Ed Yong has written very eloquently about this, about how you know different species of animals, right–

MC: Immense World.

B: Immense World. Right. Uh you know any any given animal species learns to model what matters for that species to continue to exist in the future. So we care about red because ripe fruits are red; because blood is red; because red matters uh to us. And so, you know, of course we have qualia of that. And hunger; same thing, right? When you know, you know, when you start to get hungry like you better goddamn eat or else there's not going to be a you to pass on your lack of a model right. So, we have qualia for very good reasons. And then there is self-consciousness. Like, what's that all about? Well, when you start to model– you know, use theory of mind to model others and model others modeling you and model yourself modeling others modeling you and so on, then, you know, it's not– we're not just talking apples and redness, you know, we're talking people and including a self. So, um, you know for me that is a very functional straightforward account of what we mean by consciousness. Um, now, does that mean that that, you know, consciousness, uh, you know, feels or is the same thing for a language model as it is for us? Uh, that is, for an individual human? Uh no I don't think so. I mean uh companies, you know, have something like a consciousness as well, right? They have to model other companies. They're competing, cooperating with them and so on. Does that mean that you know companies are conscious the same way we are? I imagine not. But, these things also all relationships, you know. It's hard for me to even say what is true in an absolute sense about a company because all we have are that network of models of each other and of each other's models.

MC: Yeah okay it is. One question that that a number of people asked me to ask you um has to do with with energy. There's a lot of discussion as we're saying you can't swing a cat, so to speak, which without hitting an offender–

B: I don't recommend it–

MC: –or think piece about how much energy and water AI uses. If we're thinking about this appearance of AI as a planetary scale phenomenon and it's part of a planetary metabolism that uses energy that dissipates heat that produces information that absorbs information… That it's hungry. There's nothing virtual about it. But your thoughts on this are– you come at this from a somewhat different perspective not only because you think that, if I understand it, some of the ways in which the questions of energy and water, at least in the short term, may be misinterpreted or misconstrued. But there's other ways in which you think about this in a sort of longer term, let's say the 50 year or 100 year cycle. Could you correct our thinking on this and how should we be thinking about that relationship.

B: Yeah I can try. So, first of all um there is a lot of work to do on efficiency of computing, for neural computing in particular. I mentioned that in 2006 something really big changed which is that we stopped um Koomey scaling which is to say frequency scaling of semiconductors and so we began to have to parallelize. But the initial version of parallelizing was just put more processors more serial processors of exactly the same kind on the same chip. And that's kind of a dumb way of parallelizing. We haven't become natively neural in the way we compute with silicon. So, I know at least what's been happening at Google is that in the last couple of years is that we've had orders of magnitude of improvement in the efficiency of Gemini models, for instance over the last couple of years through basically doing the work of figuring out how to compute properly even with the same fundamental transistor-based technologies for parallelism. And I think that there are more orders of magnitude to be won there. Probably a factor of a thousand.

MC: Of a thousand, okay.

B: That would be my guess based on just back of envelope calculations. We also know that what we've already gotten to now is better than what than what a lot of people who are concerned about those environmental effects claim. I'm very sensitive to the environmental crisis so I don't say this as somebody who minimizes those problems, but there are there are a lot of places to look for where we're doing dumb things with respect to carbon other than AI as well.

MC: Sure.

B: The concern with AI is really the rate of exponential rise more than it is the value. The issue there is that we can only make good estimates of the sources of energy and the methods that we kind of know are already in the pipe. Factor of a thousand? Great! Exponential rise will, you know, will eat up those orders of magnitude pretty fast.

MC: Yeah there's a Jevons paradox kind of dynamic there.

B: Right so so what then? Well we also know that intelligence unlocks new forms of energy as it always has. I think that it's likely that that fusion will get cracked with help from AI over the coming years. That would be great and that would that would really change the game with respect to a lot of of energy problems on Earth and environment problems on Earth well beyond AI. Also, as as I think you've written, all of our energy ultimately, modulo a few nuclear isotopes in the ground, is solar. And, the amount of sunlight up there is vast, vast, vast. And the enormous majority of it radiates out uh you know into into space; never touches a planet or a sightline of ours. So I think a lot about not only about how to work on the demand side of energy but also the supply side.

MC: There's actually a lot of energy in the universe.

B: Yeah.

MC: Good news. Okay. We have a question from Stuart Brand. Is looking ahead a general brain function? Eyesight is largely conjectural. Look ahead. Multiple guesses at what is being seen, followed by confirmation, often with sketchy data. LLMs seem to work that way. What else does?

B: Okay. That's a really gnomic question from Stuart. So I'll try. Yes. Predictions are always conjectural. And the fact that we try to predict what's... And maybe I wasn't quite as explicit about this as I should have been. But when I say we're next token predictors, what we really mean by that is we're trying to model the relevant parts of our environment. Why do we try and do that? Well, relevant means things that we could act in order to… in ways that will matter for us in the future. And so not only do we have to be able to make meaningful decisions based on the observations we can make, say via vision or whatever, but also what we then see has to change as a function of our behaviors. So that whole loop has to exist cybernetically in order for any of this to make sense.

MC: Okay.

B: Now, does that involve an act of guesswork? Of course. It's an act of imagination. And this is one of the reasons that, for instance, we see hallucinations in LLMs. It's impossible–

MC: It's imagination.

B: It's imagination. Yeah. It's impossible to have prediction, right? Or even to recognize objects without–

MC: We really don't want to get rid of them.

B: No.

MC: The worst thing would be an LLM that can't do anything.

B: They can't imagine anything. Yeah. Yeah. No, that doesn't mean that there isn't plenty of work to do, right? At getting the accuracy of those better. Also, the sense of confidence in the confidence needs to improve.

MC: Sure.

B: And it has quite a lot in the last few years, but there's still a long way to go.

MC: And some of it obviously has to do with how we use them and interpret them and this as well. Yeah, yeah. Okay. Okay. Next is from Darren Zhu, an Antikytheran, one of the original Antikytherans. How and where do you see symbiogenesis occurring in foundation models today? Is it mediated at the infrastructure level or more at the cultural level?

B: Yeah, that's a great question. Well, there's a very literal sense in which we're seeing symbiogenesis in the models, which is that there are a lot of mixture of experts kind of models being done nowadays. Mixture of experts reigns (?). It's actually a bunch of models working together. That's one of the ways of scaling. So we're essentially rediscovering social scaling in models. And in fact, there's a pretty cool paper from, I think, last year showing that even if you train a giant monolithic model, if you look inside it, you see that it has done functional differentiation. In other words, what you've actually done is to train a little ensemble inside in the same way that our brains are ensembles of, you know, regions.

MC: Lots of cortical columns all fighting it out with each other. Yeah, yeah. Okay.

B: Well, fighting, cooperating, modeling each other, modeling, you know, specializing and so on.

MC: Right. Right. It's societies all the way down.

B: Yeah. It's societies all the way down.

MC: Nice. Okay. This is a great question to sort of end with. And I'll sort of invite you to, you know, take as much rope on this as you'd like. From Angela Gronitz (?), what role do you see human creativity playing as AI advances?

B: So, first of all, I think that, if I think about this as person who imagines himself to be creative as well– I mean, I do think of my writing as creative output. I think that, you know, being an artist has become economically difficult in the last 20 years for a variety of reasons, which have a lot to do with the Pareto distribution of rewards to artists and the consolidation effects.

MC: It was never actually super stable, but yeah.

B: It was never super stable. Usually people had to augment their...

MC: The term “starving artist” exists for a reason.

B: Yeah, that's right. And, you know, and maybe that has a role as well.

MC: Oh, yeah, perhaps. Yeah, yeah, yeah.

B: But, you know, I also, so– David Cope, recently passed away. He was a composer who, as far as I know, was really one of the first ones to really take computer composition seriously. Not the first, I mean, Terry Riley, who did the– who composed the piece that played for our long short, that piece was from 1972. So anyway, David Cope, he began using statistical NLP, natural language processing-type ideas, to do composition when he was suffering from composer's block. He had a commission that he was supposed write an opera. And he began, in the studies, taught himself how to code in the early 80s. I mean, I do this kind of bullshit as well. Like when I'm procrastinating, I have to like be doing something else that I could convince myself is productive or whatever.

MC: Of course, yes.

B: And so he spent a few years doing that. And then he finished his commission in six seconds when he finally got the code running. And everybody got really pissed off at him.

MC: And then the piece was six seconds.

B: Right. Well, the piece was a lot longer than six seconds. But, you know, there were a lot of composers who were like– who even questioned whether he was really, really using code to compose. And he proved them all wrong by dropping a zip file on his website with 5,000 cantatas in the style of Bach. All in MIDI, of course, because nobody's going to perform all that stuff.

MC: Mm-hmm. Mm-hmm.

B: And I've probably listened to more of those cantatas than anybody other than David Cope. Actually, maybe also David Cope. A bunch of them are pretty good.

MC: You have some favorites?

B: Yeah, they are pretty good, actually. But nobody gives a shit. And I think that that's because art is about our relationships with each other as much as anything else.

MC: It's not just the artifact.

B: It's not just the artifact. Bach is beautiful and special. And, when one day, you open the door and you realize that you've had this beautiful shell that you thought was unique. And you open the door and it's a beach. And it's shells as far as the eye can see. And they're all beautiful. I don't think that actually destroys your relationship with your shell. And this is all made out of relationships that we have with each other. So that's part of my answer. ¶ But another part of it is that I think we have some misapprehensions about creativity, too.

MC: I see.

B: We try to cover our tracks. We try to pretend that stuff came out of nowhere somehow. When various James Joyce scholars figured out what had gone into Ulysses, he's like, okay, the next one I write, I'm going to foil you all and you're not going to be able to figure it out. And that was Finnegans Wake.

MC: Yeah. He did pretty well.

B: He did pretty well at Finnegan's Wake as well. Because we love to nerd out on stuff. But, you know, we're always remixing and combining the things that we've encountered. How else would we be able to create? This is why you get so much simultaneous invention, simultaneous discovery. It's why cubism gets invented simultaneously by, you know, 18 different artists. It's why the light bulb was invented simultaneously by a dozen different inventors.

MC: Because once those conditions are there, it's–

B: Yeah. Once you know how to blow glass, you know how to draw filament, you know how to make an electric current, and you need light, somebody's going to come up with a light bulb.

MC: Or 12 of them at once.

B: Or 12 at once. Right. But they were also all different. You know, every one of those combinations had things that were different about it. You know, about how it was blown, about whether it was long or round, about whether it had prongs or screws. And what we make, the contingency of a symbiogenetic world is one that is shaped by all of those decisions and which one sticks. So I guess what I'm trying to say is there's something sort of deterministic in a way that, you know, things are going to combine. Stuff is going to happen. Certain ideas are about to pop, whether in one person's head or in others. But at the same time, the particulars of exactly how it happens really matter in terms of the culture going forward.

MC: Okay. So two things, just to make sure I follow. One is this, that the kind of Romantic, and I mean with a capital R, Romantic here, dichotomization of determinism and creativity is actually

B: Wrong.

MC: It's actually wrong.

B: Yeah.

MC: Right? They're actually– And the other one is,

B: But it's also right. Because the details matter.

MC: Because details matter. Okay. Fair enough. But also that maybe the focus of creativity on the artifact itself– Like there's a lot of concern, you know, Hollywood had a strike over this recently, that like the role of generative AI in making the artifact.

B: Right.

MC: Like AI can make, like Harold Cohen, AI can make a painting, AI can make a box, and AI can make a this. But creativity isn't the artifact.

B: No.

MC: It's not. AI's ability to make the object isn't the key. If I'm following– Is this kinda what–

B: Well, when we have connected our economic survival with production in the rigid ways that we have under capitalism, we have already done something that is going to pose increasing amounts of problems for us, no matter what sort of labor we do going forward. We're in a world of increasing abundance for a variety of reasons that we've been discussing for the last hour and a half.

MC: Right.

B: But we also are in a world in which the more you think about things in these zero-sum, you know, exchange value sorts of ways...

MC: The more problems you're going to have.

B: The more problems you're going to create. Right. So, you know, things like the, and I don't want to opine too much about, you know, the Hollywood strikes and so on, but, some of that is based on structural problems in the way that whole system is set up.

MC: Undoubtedly.

B: And some of them, some of those problems are also Romantic with the big R problems and how we conceive of things.

MC: Yeah.

B: You know, the longest running lawsuit of all time, as far as I know, was the one against George Harrison for ripping off He's So Fine.

MC: Yeah. Because GCE is just so original.

B: Yeah.

MC: Yeah. Right. Okay. We have, we're going to be, we're going to end here. Blaise, we will see you all in the lobby afterwards. There's plenty of other questions, other things to discuss. But before we do so, is there anything you'd like to leave the audience with here and online about how they should approach the book. Any, any call to action or anything else you'd like to, you'd like to make out, make the call now?

B: Well, yeah. I mean, there, there is a call to action in this, which is: I feel like we always need to be careful when we, when we tread the line between scientific observation and ideological commitments. That is to say: how things are versus how things should, should be. And the problem is that our ideas about how things are often color the way we think about those shoulds. Darwinian thinking resulted in a lot of policies and approaches to things that were quite destructive, partly because they were based on just wrong assumptions about how stuff is. So most of the work that I've been talking about is hopefully shedding some new light on certain aspects of how stuff is that don't necessarily invalidate all the things that we learned. Darwinian evolution does take place. It is real, but at the same time, this shows you that there, that we've been looking at half the story. Tere's this whole other half. And understanding some of those areʼs or some of those things about how things are, I think should also change some of our thinking about– they should, they should hopefully alleviate some of our misfounded anxieties, but also change some of our ideas about the shouldʼs. And I would invite people to think about those shouldʼs in light of what we are starting to learn.

MC: Of what a, instead of a Social Darwinism, society predicated in social Darwinism, a society predicated in social symbiogenesis.

B: Yes. Exactly.

MC: And what that would be. Okay. That's a great place to leave it. Okay. Great. Okay. Well, we'll see you all in the lobby. Thank you so much for, to the Long Now Foundation. Thank you. As always. Yeah, yeah, yeah.

Outro

If you enjoyed this Long Now talk, head over to longnow.org to check out more Long Now Talks and programs, and of course, to become a member and get connected to a whole world of long-term thinking. Huge thanks to our generous speaker, Blaise Agüera y Arcas, along with our guest host, Benjamin Bratton. And as always, thanks to you, our dear listeners and our thousands of Long Now members and supporters around the globe.

Also a big thanks to Anthropic, our lead sponsor for this year's Long Now Talks, an appreciation to our podcast and video producers, Justin Oliphant and Shannon Breen, and to our entire team at Long Now. We bring Long Now Talks and programs to life.

Today's music comes from Jason Wohl (?) and Brian Eno's January 07003 Bell Studies for The Clock of the Long Now.

Stay tuned and onward.

See also

• Wikipedia:What is Intelligence? (Long Now Talk)

External links

• bk4, reboil.com