Escaping
the Red Queen effect
|
||||||||||
|
The same laws may govern the evolution of organisms and organizations “Rugged fitness” landscapes determine the rate of innovation Breaking your company into “patches” to balance
order and chaos
— STUART A KAUFFMAN The McKinsey Quarterly, 1995 Number 1, pp. 118—129
At first glance, the adaptive evolution of living organisms and the development of human artifacts seem worlds apart. When a gene mutates, it does not do so intentionally. The mutation may be helpful, harmful, or neutral as far as the survival of the species is concerned. By contrast, human artifacts like tools, products, and even organizations are the fruits of a conscious struggle to invent and improve. What can biology and
technology possibly have in common? Perhaps nothing, perhaps a great deal.
THE CAMBRIAN EXPLOSION In the Cambrian period of the Paleozoic era, some 500 million years ago, a burst of biological creativity took place that finds an echo in the development of many technologies. Over a relatively brief period of time, a vast diversity of fundamentally different life forms appeared. What was extraordinary about the Cambrian explosion, as it is known, was that the emergence of new organisms happened “top down.” In the taxonomy of living creatures, the highest categories, kingdoms and phyla, capture the most general features of a very large group of organisms. The phylum of vertebrates, for instance, includes fish, birds, and mammals. There are 32 phyla today, all dating back to the post-Cambrian period. Evidence suggests, however, that as many as a hundred phyla may have existed during the Cambrian itself, most of which quickly became extinct. These phyla are thought
to have been established by the first species to emerge — hence “top down.”
These radically different creatures then branched into daughter species,
slightly more similar to one another but still distinctive enough to become
founders of the next category in the hierarchy, classes. The process replicated
itself to produce daughter species somewhat more similar to one another
which in their turn founded orders. Next were families, and finally genera.
The pattern is one of explosive differences among the species that branch
early in the process, with progressively less dramatic variation in successive
branchings.
Technological innovation The way in which technology
evolves is strikingly similar to the Cambrian explosion. Diverse forms
branch out bushily at the beginning, then the rate of branching declines,
extinction sets in, and, like today’s phyla, only a few major alternative
forms persist at the end. Early on, the diversity of forms is more radical;
later, it shrinks to a fine-tuning of details. Development is “top down.”
In any basic innovation, it is common to find a wide range of early experiments with radically different designs that branch further, then settle down to a few dominant lineages. Take the bicycle. The nineteenth century saw a plethora of forms: bicycles with no handlebars, versions with small back and large front wheels, forms with more than two wheels in a line, and even bicycles with two wheels of equal size. An early leader, the pennyfarthing, branched further. The “class” of bicycles eventually settled to today’s dominant genera: street, racing, and mountain bikes. Equally
apt illustrations can be found in the development of the automobile early
in this century, from its highly diverse origins in primitive steam and
gasoline vehicles, or in the design of aircraft or motorcycles. After the
emergence of a fundamental innovation, people experiment with radical modifications
to find ways to improve it. As better designs are found, it becomes progressively
harder to make further improvements, so variations become more modest.
The process closely resembles the evolution of living creatures on what
biologists describe as a “rugged fitness landscape” (see the boxed
insert), in which the rate of finding improvements slows exponentially
as “peaks” of Conflicting constraints Why should developments
in technology so resemble the emergence of species in the living world?
One answer is that similar processes of evolution may operate in both spheres.
In both biology and technology, adaptation can be seen as the attempt to
optimize systems riddled with
A foraging creature, for instance, must spend time on its search for food. But its need to browse is in conflict with its need for speed over turf to evade its predators. These conflicting requirements must somehow be reconciled and optimized if the creature is to survive. Similarly, suppose
we are designing a supersonic airplane. The fuel tanks must go somewhere;
the plane must have strong but flexible wings to carry the load; wires
are needed to control the flight surface; the seating must be put in place;
we need room for the hydraulics, and so on. Unfortunately, the best solution
to one part of the design problem conflicts with optimal solutions to other
parts of the overall design. Thus we must find some compromise solution
that satisfies the conflicting constraints of the This common factor
is precisely what makes it possible to compare the evolution of organisms
and technologies. Both evolve on fitness landscapes made rugged LEARNING CURVES Evidence that technology, like biology, is susceptible to the process of evolution also manifests itself in the phenomenon of the learning curve. One kind of learning curve arises in production, where the greater the number of items that are produced, the more efficient production becomes. Economists have noted that at each doubling of unit output in a factory, the cost per unit falls by a constant fraction, often about 20 percent. The other kind of learning
curve arises on what are called technological trajectories. The rate of
improvement of a technology often slows exponentially with total industry
expenditure. Improvement in performance is rapid at first, then eases off.
The shapes of these learning curves are important factors in growth in techno-logical sectors of the economy. Investment in new technology, for instance, yields rapid improvement in performance at first. These increasing returns attract investment and drive further innovation. Then learning begins to slow exponentially, and less improvement occurs per investment dollar. The mature technology centers on a few dominant forms in a period of diminishing returns. Attracting capital for further innovation becomes difficult. Growth of that technology sector slows and markets become saturated. Further growth will depend on a burst of fundamental technological innovation in some other sector. Despite the importance
of these learning curves, there is no theory to account for their prevalence
in technological innovation and economic growth. Perhaps the analogy with
adaptation on fitness landscapes (see the boxed insert) can shed
some light.
The struggle uphill On a rugged fitness landscape, for every step you take uphill, the number of directions uphill declines by a constant fraction. Conversely, after each improvement is made, the number of attempts needed to find another improvement increases by a constant fraction. The rate of finding improvements thus shows exponential slowing. Technological development seems to follow a similar pattern. The first type of learning curve with a technological trajectory exhibits a power law slowing in cost reductions with units produced. According to rugged landscape theory, this power law slowing reflects both the exponential slowing in the rate of finding improvements, and an exponential decrease in the amount of improvements at each step as successive improvement steps are taken. The second type of technological learning curve features exponential slowing in output performance as improvements to the fundamental design become ever more modest. In both, the number of routes “uphill” toward further progress dwindles by a constant fraction following each improvement. These remarkable parallels
suggest that it is worth taking seriously the idea that biological evolution
and technological development may be governed by similar general laws.
Both are forms of adaptive evolution, exploring vast spaces of possibilities
on more or less rugged “fitness” landscapes. If the structures of the two
types of landscape are broadly similar, it follows that the branching processes
of adaptation should also be similar.
COEVOLUTION In real ecosystems, species change as they interact with one another. They coevolve, one species living in the niches afforded by others. Flowers coevolved with the insects that pollinated them and fed upon their nectar. Plants manufacture the oxygen we breathe; we produce the carbon dioxide they use to photosynthesize. And as species change, so too do their niches in the ecosystem. Host—parasite systems coevolve too. The agent of malaria alters its surface antigens to evade detection by the host, while in its turn the host immune system evolves to try to catch and destroy the malaria. A molecular game of hide and seek ensues. Coevolution also occurs
between predator and prey species. The shells of certain marine creatures
exhibit spur-like calcified fortifications, presumably an adaptive response
to the ability of starfish to capture and open their shells. Responding
in kind, starfish have gotten bigger, developed sharper beaks, and strengthened
the suction that grasps their prey.
Chaos and order This form of persistent
coevolution has been dubbed an “arms race,” or the “Red Queen effect,”
after her comment to Alice: “You have to run faster and faster just to
stay in the same place!” Where this effect applies, all species keep changing
in a never-ending race simply to sustain their current level of fitness.
Chaos prevails.
In contrast, an “evolutionary stable strategy” is one where each species is better off not changing its survival strategy so long as the other species with which it is coevolving continue to follow their current strategies. The result is an equilibrium, an ordered regime in which there is no incentive for any species to alter its strategy. Indeed, if a species were to deviate, its own fitness would be harmed. In real ecosystems,
some processes of coevolution may lie in the chaotic and some in the ordered
realm. An ecosystem deep in the ordered regime of an evolutionary stable
strategy will be too rigid, too frozen into place, to coevolve away from
low local peaks. Under the chaos of the Red Queen effect, on the other
hand, species climb and plummet on heaving fitness landscapes, never staying
at a peak. But between these two extremes of low fitness, in the transition
between chaos and order at the “edge of chaos,” peaks are high but can
be attained. Here, fitness can be optimized.
Niches in a web Coevolution can be seen at work in our economic and cultural systems too. An economy, like an ecosystem, is a web of coevolving agents. The mutualism of the biosphere finds its echo in the relationships within the vast web of goods and services, each “living” in its economic niche. When the car came in,
it drove out the horse. With the horse went the smithy, the stable, the
saddlery, the harness shop, buggies, and the Pony Express. But once cars
are around, it makes sense to expand the oil industry, build gas stations,
and pave the roads. Once the roads are paved, people drive everywhere,
so motels are useful. When cars get faster, traffic lights, traffic cops,
and parking fines make their way into the economy and our behavior patterns.1
Goods and services
exist only if they are useful — whether in their own right or as an “intermediate
good,” a component in the creation of other goods and services. An intermediate
good like the engine for a car may have inputs from a variety of sources,
from tool makers to iron mines to computer assisted design to the manufacturer
of the computer and the expert who wrote the CAD software.
Changing landscapes Novel goods create niches for still further new goods. The process is truly one of coevolution: the goods and services that emerge must always make sense in the context of those that already exist. But as economic activities alter, the coevolutionary fitness landscape is deformed, attacking the fitness of existing goods and services. This provides the opportunity for a family of new, “neighboring” technologies to proliferate, climbing uphill on the deformed landscape. As aircraft design
improved and engine power increased, the fixed-blade propeller became less
useful and a new innovation, the variable pitch propeller, took its place.
This invention opened a modest new era of learning how to create better
variable pitch propellers. So, as adjacent products and technologies are
called forth by the deforming landscape,
Some new goods and technologies have a particularly potent effect. The car, as we have seen, ushered out many old technologies, ushering in new ones in vast avalanches. So did the computer. Such avalanches produce enormous arenas of increasing returns, thanks both to the massive early improvements that are won through learning and to the major new markets that are created. Other new technologies,
on the other hand, emerge and disappear with scarcely a ripple. It may
be that the difference between the automobile and the hula hoop, say, has
to do with how central — or peripheral — the new product or technology
is in the economic web, both when it first emerges and subsequently.
PATCHES Human constructs share another property with evolving ecosystems. It can be explained by means of the image of patches in a quilt. Take a hard, conflict-laden task in which many parts interact, and divide it up to make a quilt of patches. Try to optimize within each patch. Although the patches do not overlap, there are couplings between parts of separate patches across patch boundaries. This means that finding a good solution in one patch will change the problem to be solved by the parts in the adjacent patches. These parts will themselves make adaptive moves that in turn alter the problems faced by yet other patches. The quilt is in effect
a model of a coevolving system. Each patch behaves like a species, climbing
toward fitness peaks on its own landscape. In doing so, it deforms the
fitness landscapes of the other species that share its ecosystem.
This process of coevolution may spin out of control, changing ceaselessly without ever converging on any good overall solution: the chaotic behavior typical of the Red Queen effect. As each patch selfishly optimizes, it deforms the landscapes of other patches so drastically that all patches climb toward ever shifting peaks forever. This is likely if the patches chosen are too small. Alternatively, the whole system of patches may freeze up on a low local peak under the ordered regime of the evolutionary stable strategy. This is likely if the patches are too big. Such “patch” systems undergo a phase transition. As patch size for a given quilt system is turned from small to large, the system suddenly switches from chaotic Red Queen behavior to settle on a mutually optimal solution in an ordered way. The best solution for the quilt, as for an ecosystem, is found at this point between these two extremes, poised in the transition between order and chaos. By analogy, a hierarchical company with too much control at the top is likely to freeze too rigidly into poor compromise solutions. A company broken into too many small selfishly optimizing departments may churn chaotically. If the entire conflict-laden
task is broken into patches of the right size, the coevolving system that
results will lie precisely at the phase transition position. As if by an
invisible hand it will rapidly find excellent solutions to the hard problems.
Breaking down large systems into patches, in short, may be a fundamental
approach we have evolved in our social systems and perhaps elsewhere to
Performance Consider the value of understanding optimal patching in the management of complex organizations. Manufacturing, for example, has traditionally used fixed facilities of interlinked production processes to make a single end product, like an assembly line for a car. Such fixed facilities are used for long production runs. Today, however, it is becoming important to shift to flexible manufacturing. Here the idea is to be able to specify a diverse range of end products, reconfigure production facilities quickly and cheaply, and carry out short production runs to yield small quantities of specialized products for niche markets. The output must, of course, be tested for quality and reliability. But how should this be done? At the level of each individual production stage? Of the entire system? Or of some intermediate chunk? I wonder if there is some optimal way to break the total production process into local patches, each with a modest number of linked production steps: keep partitioning the system into smaller patches. When overall performance degrades, break up to slightly larger patches. Then one could optimize within each patch, let the patches coevolve, and rapidly attain excellent overall performance. Patching systems so that they are poised on the “edge of chaos” may be valuable for two quite different reasons. Not only do such systems quickly achieve good compromise solutions under conflicting constraints, but they should also track the moving peaks on a changing fitness landscape very well if the existing environment changes. If external conditions alter, a rigidly ordered system will tend to cling stubbornly to its peaks. Poised systems, by contrast, should cope better with shifts in the landscape. These insights hint
at the reasons why flatter, decentralized organizations may function well.
Counterintuitively, breaking an organization into “patches,” where each
patch attempts to optimize solely for its own selfish benefit, can lead,
as if by an invisible hand, to the greater welfare of the whole organization.
The trick, as we have seen, lies in how the “patches” are chosen.
~~~ It seems clear that
at least an analogy exists between the unrolling panorama of interacting,
coevolving species — each coming into existence and making its living in
the niche afforded by other species, each becoming extinct in due course
— and the way that technological evolution drives the emergence and extinction
of technologies, goods, and services. This analogy can offer intriguing
and fruitful insights into the ways that products, organizations, and economies
develop, as I have tried to suggest. But I suspect that there is more than
an analogy. I suspect that biological coevolution and technological coevolution,
the increasing diversity of the biosphere and of the “technosphere,” may
be governed by the same or similar fundamental laws.
Notes
1. I am indebted to my Santa Fe colleague W. Brian Arthur for this example of the way in which economists think about technological evolution. View or right-click to download this article (select Save Target As) in Adobe Acrobat PDF file format. To obtain permission with McKinsey & Co. to make and distribute multiple copies of this article, please submit our permission form. You may also visit this library’s PDF index for a complete listing of all PDF files available for downloading. Requests
|
