Entropic Thoughts

Lessons From Evolution

Lessons From Evolution

I’ve stated before that I haven’t read Gould. I’m currently taking a break from reading Dune and extreme value theory and fixing that by reading Gould’s magnum opus, The Structure of Evolutionary Theory1 The Structure of Evolutionary Theory; Gould; Harvard University Press; 2002..

I know Gould is considered a bit of a black sheep in this area2 The major complaint I’ve found is that he misrepresents the ideas of others in ways that paint himself in a better light. But I’ve also heard much worse accusations. but I’m not reading him because I believe he’s right on everything. Rather, much like with Graeber, I’ve found that he directs my gaze in interesting directions. When reading Gould (and Graeber) I end up looking at things from new perspectives.

One of the inspiring things I’ve found in the book is the incredible change of perspective Darwin brought to the scientific community, in multiple ways. Let’s start with the power of selection.

The power of selection is underestimated

To put you in the right mindspace, imagine a human three-year old, that grows to become five years, nine years, sixteen years, and eventually 25 years. All along, they mature and their body undergoes changes. What causes these changes in a human? Not a trick question. Most people would say it’s an intrinsic drift toward the more mature. Humans are pre-programmed to grow and mature a particular way.

This was how a lot of people looked at evolution in the time of Darwin. The evolution of species was an intrinsic drift in some direction. Species were created – much like Pokémon – pre-programmed to evolve a particular way.

Then Darwin came around and said, no, you’ve got it all wrong. There are four elements that make up the evolution we observe:

  1. Species have large numbers of offspring, who inherit many of the traits of their parents;
  2. Except that the traits of the offspring are subtly different in all directions;
  3. Natural selection eliminates more of the individuals with bad changes; and
  4. This process has had a very long time to work its course.

Darwin was saying that the apparent drift toward the more adapted is entirely due to compounding selection bias.3 Note that Darwin may not have been entirely correct in this, but correct in broad enough strokes to lay the foundation for the field even today. He did this even without knowing about the gene! He never postulated a mechanism for inheritance, only offered the observation that a calf is consistently somewhat like its cow.

It’s not hard to see how this can be applied to other areas than biological evolution. Darwin himself was inspired by similar ideas in geology and economics. Richard Gabriel views innovation and product development through this lens. The second world war planes-with-bullet-holes story is another typical example of how the correct solution needs to account for selection bias.

Think of how natural it was for humans before Darwin to consider evolution as intrinsic to species, and how wrong they were. Humans quickly default to intrinsic explanations, that particular behaviours come from within the thing behaving. What are all the things where we default to an intrinsic explanation today, in our daily lives, when selection effects may be dominant?

Darwinianism has strict requirements

Aside from selection effects more generally, we can run Darwinianism both backwards and forwards, and in either case, we’ll find that the requirements are relatively strict. Let’s first run it in the same direction as Darwin, and then see what happens when we reverse it.

Darwinism is not a good metaphor for free market competition

I sometimes hear the competition between private companies in free markets being compared to Darwinism, in that it ensures the best adapted thrive or whatever. But if we look at the requirements of Darwinism above, we can see that it becomes a very strained analogy.

After all, what are the offspring of private companies? Is there really strong evidence of inheritance? Is variation of offspring really undirected, or is there some other level of agency in there? Has enough time passed to produce meaningful results?

This is a particularly sad mistake, because there are parts of Darwin’s evolutionary theory that fit free market competition: exactly the parts that Darwin incorporated into biological life from Adam Smith’s theories on free markets! So we already have an evolutionary-like theory of free markets in Adam Smith. I suspect the idea of bringing Darwin into it is to lend it an air of additional credibility, but the result is mainly confuddlement.

Creating artificial Darwinism is difficult

In the above analysis, we tried to go from existing system to explanation based on Darwinism. We might be interested in going in reverse. Maybe we have a company and we want to be successful – what if we set up a Darwinistic type environment within the company, and trust compounding selection effects to give us good ideas? Sounds like this would guarantee us success in the market!

But now we have four new problems:

  1. How do we make sure ideas have offspring and inherit from their parents?
  2. How do we create undirected variation in the offspring ideas?
  3. How can we select the good ideas if we don’t know what they are?
  4. Can we really afford to spend a lot of time on this system?

The compounding selection of Darwinism requires these four things, and without one of them you have something else – possibly something less effective.

Compounding selection is the key

I think the main thrust of both of the above examples is that Darwinianism is about compounding selection effects. That’s what makes it so powerful. When many people think of artificial evolutionary systems, they think of plain selection. In Darwinian evolution, the selection of today builds on the success of the selection yesterday, and that is a force multiplier that’s difficult to achieve, but very desirable if one tries to achieve success through selection alone.

Historical inference is possible but fuzzy

Another important contribution from Darwin4 At least the way Gould tells it. are his methods of historical inference. Think of a time when you stumbled over Chesterton’s fence at work; the way something is makes no sense to you, but you’re afraid to change it in case it performs an important function you just don’t know about. You’ve asked around, but nobody knew why it is the way it is.

What can you do? These four methods are listed from more concrete and directly applicable to fuzzier and less helpful.

Uniformity

Carefullly observe all the processes going on today, and try to extrapolate their results; into the past and future. Does this produce something that resembles the Chesterton fence you’re looking at? Then maybe that process – or something similar to it – produced the fence and you have more clues as to its purpose.

Example: I inherited a complex set of rules according to which people were allowed to spend a consultant’s hours, but I didn’t understand what problem that solved.

I allowed a few people to use consulting hours freely and carefully observed what happened: it turned out people were offloading more and more of their tough problems onto the consultant. This was a barely perceptible effect, but let’s extrapolate boldly: over the years, offloading tough problems will make people less confident5 Tough problems is how we build expertise, after all., which will make them ready to offload more tough problems, in a reinforcing feedback loop. At some point, it might become a problem big enough to come up with a complex rule set around.

Sequencing

If you take a collection of seemingly unrelated things and arrange them in sequences, does one sequence make them seem like stages in a single process? If yes, does the Chesterton fence you’re looking at fit into that sequence somewhere? Then maybe it’s a byproduct of a stopped process that started with the earlier things in the sequence, and would have produced the later things if it wasn’t interrupted.

Example: I stumbled over an inconsistent configuration file. I didn’t know what to make of it, so I let it be.

As I spent more time at that workplace, I observed how people went about editing configuration files, and learned what their edits looked like at various stages in the process. I eventually figured out the inconsistent configuration file was in an intermediary stage on an uncommon type of edit branch, which was why nobody knew why it was inconsistent in that way.6 What’s more interesting is that the person who left the configuration in an inconsistent state couldn’t even explain themselves why it looked the way it did, because they viewed their editing sequence as one smooth flow, and didn’t think of how things appeared when frozen mid-flow.

Consilience

Aim for a single parsimonious explanation that can account for many seemingly disparate things. That makes it more likely to be true, even in the absence of a large volume of evidence on any single one thing.

Example: I encountered a lot of weird policies and incentives and meetings and culture that seemed out of place in the company at the time.

I eventually figured out that the purpose of the company was not what they said it was – it must have shifted into generating short-term profits. That might sound like an obvious realisation, but management in the company worked really hard to prevent people from realising that7 And there may have been no small element of self-delusion involved..

Discordance

If something seems especially weird when you look at it today, imagine there must have been a good reason for it in the past, and that reason has gone away. If you can guess what that reason was, then you know something about what has changed in the environment.

Example: When I was hired at one of my past employers that had just three programmers aside from myself, there was a lot of process in place for onboarding new programmers. This is an oddity that, if you can guess the explanation, can clue you in to other ways this explanation has shaped the environment.

I didn't figure that puzzle out on my own, but after being there for a while, I learned that they had planned on doubling the number of programmers every six months for two years. However, just before I was hired, an investor withdrew which made that plan unrealistic.8 This was also why they were in the middle of a large project of decoupling software components and improving the deployment experience!

Market success is more than just luck

When I was talking to one of the founders of a previous company I worked for, I had to ask what he thought made the company successful. He went silent for a few seconds, then shrugged and suggested, “Luck?”

This was a refreshingly honest contrast to all the people who stand ready to list the steps in their recipes for success, and, of course, the mountains of people who followed that recipe to the letter but still didn’t end up successful.

The system theorists constant mantra is “It’s not intrinsic, it emerges from the interaction between individual and environment.”

But! We can choose whether we view those interactions as plain luck, or whether we – like Darwin – try to dig deeper into what drives those interactions and how they appear to go in some directions but not others.