 |
|
 |
|
|
|
|
|
|
Science of Diversity
|
Diversity: The key
to understanding innovative
and adaptive systems
How Diversity can lead to better system performance in Organizations, Economies and Ecosystems
|
Research on how loosely interacting groups can solve hard problems better than experts led to a general understanding of the above questions in a variety of systems: ecosystems, social groups, large organizations, free market economies, the stock market, our society - any system where there are individuals or groups make decisions or solve problems with minimal centralized control or planning.
In fact, the importance of self-organization and diversity is highly underestimated, particularly in systems that are often thought to be centralized.
The core theme is that “loosely cooperative relationships which form by chance” by being at the right place at the right time with the right capability/information can benefit one or both of the participants, but not at either's expense. These "random" interactions can solve problems beyond the awareness of the individuals involved - often more difficult problems than can often be solved by experts. Human behavior has evolved to create these interactions - but many aspects of our current life, society, organizational structure inhibit this natural process. By better understanding these processes, we will all benefit and solve problems that are a challenge to our centralized organizations.
Summary of the viewpoint:The role and function of diversity depends on the stage of development (or maturation) of an organization (or a system in general).
In our competitive view of the world, we often view organizations that are loosely cooperative as being competitive or cooperative. But excessive competition and cooperation are the bookends to the sweat spot of an innovative organization. Three stages of development are defined:
In Formative systems in rapid transition, competition between diverse groups occurs, leading to a reduction in diversity through selection "by survival of the fittest." Through the processes of selection, these system as a whole perform better, because the individuals that make up the system are selected to perform better.
In Co-operational systems, in direct contrast, achieve a higher systemwide performance from loosely cooperative interactions between diverse populations, without the need for selection. Performance of the sytem requires diversity!
In a stable environment, loosely cooperative relationships can become exclusive (as in symbiosis). These exclusive interactions may be more efficient and have higher performance, but they are at the expense of flexibility. This is the mechanism for the transition to a Senescent stage. Within this understanding, formalized cooperation is a form of exclusion, disallowing other relationships to form. Despite all the current emphasis on cooperation, it is a form of selection and exclusion and not advantageous to solving hard problems or to robustness.
If the rigid interactions are mutually beneficial to all, then these structures may be incorporated system-wide (for example, the common language of DNA in life). Future variation then occurs on top of these universal structures. This process is the origin of infrastructure and hierarchical systems.
While it is common thinking that most ecosystems and social systems are inherently competitive and selective, our view is that higher system functionality, incorrectly attributed to competition, is actually due to self-organizing, non-competitive processes (as in next item).
In Condensed systems, both diverse individuals and the system as a whole mutually benefit from the advantages of self-organizing, non-competitive processes - rather than competitive and selective processes.
Self-organizing, non-competitive processes are where a diversity of individuals or entities solve parts of a difficult problem in a common world. Their interactions (which may be chaotic and unintentional) combine in such a way that the system as a whole is performs better and is more robust, than what would be predicted by looking at the performance of the individuals in the system. The higher performance and robustness is an emergent property of the system, i.e., not directly attributable to or observable in the individuals.
The above conclusions can be applied to most decentralized, self-organizing systems found in economies, organizations, ecosystems and social systems. One focus of this research on diversity is to show that the processes are a natural aspect of these systems and, whether knowingly or not, are essential to the performance of the system.
In evolving systems, it is often argued that attributes of the individuals are selected to enhance these self-organizing processes. One example is the evolution of social attributes in human and animal systems: social processes enhance the self-organizing nature of our human systems, society or organizations. Social processes make the whole the organism, not the individual (social insects are an excellent example: the colony is the organism, not the individual insect). Once social attributes are part of the individuals, there is no need for further selective development of the individual: The higher performance needed for future challenges come from self-organizing non-competitive processes across the whole organism.
Because a common understanding of these self-organizing processes is argued for a variety of systems, a question arises if there are any differences in these processes or additional processes found in some and not in others?
One aspect that social systems have that is absent in other systems is the ability to have extensive systemwide reinforcement of ideas (paths or dynamics, in general). An example is mass-communication and might be viewed as a process that hastens the condensed stage. This coherence enables social and economics systems to "self-resonate" to a degree that can repress diversity and make the overall system less robust. The effect of extreme nationalism on the policies and social diversity of a country is a prime example. This effect can also occur as a positive feedback mechanism which can cause a larger impact of a change than would otherwise occur. Many stock market bubbles and bursts are argued to occur by this process.
Similarly, the unappreciated aspects of social networks in organizations provide problem solving capability and contingencies that directly come from diverse individuals. An obvious realization of this is that most organizations would quickly fail if all employees thought alike or had little social interactions (in contrast to business-only interactions).
Distributed economies also have self-regulation (the "invisible hand" of Adam Smith) and high performance, not because of the competitive nature of the individuals or even because of direct cooperation, but because of the interdependency of the diverse elements in an economy. By counterexample, it is easy to see that an economy or market that is absent of diversity is quite fragile (only one path for material/energy/monetary flow) and less efficient (serial instead of parallel performance).
Diversity in the workplace, organization, even nation, has recently become a much touted goal. Our understanding has evolved to recognize that diversity includes more than ethnic variety, but also cultural and educational variety, in fact, any aspect of an individual that leads to a variety of approaches to problems, even personality. Yet, the scientific basis behind the role of diversity in social systems is largely unknown. Research at Los Alamos is providing support for our intuitive understanding of the importance of diversity in social systems.
Much of the past understanding about social diversity is motivated by the analogies to biodiversity in nature. The prevailing research on the role of biological diversity focuses on improvement of system performance (e.g., individual or species survival) by the selection from a pool of genetic diversity, in the process of natural selection or survival of the fittest. This process is fundamentally competitive between individuals or groups, with explicit winners and losers. This selection reduces diversity in its application - diversity is reduced by selection and must be replenished by mutation or migration. While a similar selection process may occur in social systems, this is neither an compelling justification for diversity within organizations, nor the only mechanism by which diversity contributes to better organizations or society.
Current research at Los Alamos has identified an alternative process for higher system performance, which does not involve competition or cooperation between the individuals, but instead stresses the importance of non-competitive self-organization.
We wish to address the question: what is the most simple demonstration of increased global performance of a collective above that of the individual? The idealized system examined is the solution of a sequential problem (Insert in Figure 1), which has many optimal and non-optimal solutions, solved by agents that have identical capabilities. While this maze problem is quite simple, it serves as a representation of more complex problems encountered by individuals and organizations: the solution of a problem that has many decisions points and possible solutions and that has difficulty greater than solvable by one individual. Here, agents can represent individuals, groups or organizations within a greater system.
Because the agents have no global sense of the problem, they initially explore the problem until a solution is found. This "learned" information is then applied by the individual agent to solve the problem again, often with a shorter path as a consequence of eliminating unnecessary loops. Because the initial search is random, a collection of individuals shows a diversity of experience (regions of the maze), of preferences (preferred paths), and of performance (path lengths), even though they started with identical capability. (This source of diversity in performance by individuals of identical capability is a reminder how many of the advantages and disadvantages an individual has in social situations are a result of random events and not directly related to our "inherent" capability. Said another way, if we replayed or lived our own lives again, likely it would be quite different due to these random effects.)
Information for a collective of individuals is then constructed by a linear combination of the each individual's experiences. Then the same rules are used on this collective information to find a collective solution. As seen in Figure 1, the collective typically outperforms the average individual for larger collectives. In repeated solutions to a problem, we tend to remember only the information needed to solve a problem and forget extraneous information associated with unused paths. Here, the equivalent effect is for the individual to contribute to the collective only "established" information along paths used by individual, thereby "forgetting" unused paths. Both the learned and established information produce the same path for the individual agent. As seen in Figure 1, the solution using the established information performs better than the learned information. Furthermore, for collectives above 20, the optimal solution is found, even though nothing in the agent's rules seeks a minimal path length. Figure 2 shows one mechanism for the reason that the collective does better than any individual: individual information can be combined to indicate a shorter path for the collective.
Individual solutions Collective solution
To better understand the role and importance of diversity in this simple model and gain insight into social systems, quantitative measures of diversity were examined. The first choice of a measure, the breadth of experience of a collective over all possible paths, was found to be uncorrelated with performance.
The best measure found defines diversity as the degree of unique information in a collection of agents. If all agents contribute the same information, even if it is for the entire domain, then this measure of diversity is low. If each agent contributes unique information not shared by others, then this diversity measure is high. Consequently, collectives contributing "established" rather than "learned" information exhibit higher diversity, even though less information is contributed by each agent. So we conclude that it is not how much information is important, but how the information contributed fits in to the other information known.
Not only does this measure of diversity correlate best with collective performance, it also indicates the degree of insensitivity to noise. The performance of a collective with low diversity is poor when valid information is randomly replaced with false information, a measure of the robustness of a solution. False information can lead to unexplored paths in a less diverse collective, and then the solution degenerates to a random search (imagine driving and finding yourself on an unfamiliar road - the solution is either to backtrack or to search randomly). Diverse collectives have contingency information that makes the groups highly insensitive to noise. The stabilizing effect of diverse groups is critical in dealing with difficult problems where false information can lead to unexplored paths. In research which focuses on rational individuals with perfect information, an evaluation of performance often does not consider the robustness to noise, hence, the reason that diversity is rarely considered important. Yet, robustness is a critical aspect of any modern organization.
If the effect of information exchange between individuals (a form of cooperation) is included in the above simulations, such that the individual while learning the maze can benefit from other agent's experiences, we find that improved individual performance can be achieved. But if the outside information is used too strongly, then there is the ultimate loss of diversity in the collective as every individual has the identical experience. In this case, the robustness of the collective can be severely degraded if the positive feedback or coherence is too great. We conclude that cooperation is actually a form of selection and reduces the expression of diversity (if I cooperate with you consistenly, then I don't cooperate with someone else). It is easy to see that the random associations that are beneficial to the group and individual above can be reinforced to the point of being exclusive. While this leads to more optimal performace, it also causes the groups to be less robust. (To see how this transition is part of a bigger view of how systems develop, see the paper titled Developmental Insights into Evolving Systems: Roles of Diversity, Non-Selection, Self-Organization, Symbiosis on the documents page. )
Social and Organizational ImplicationsThe above study illustrates how diversity can arise with agents of identical capability from experiential differences within a system which contains multiple options. Just the existence of random options in the problem domain creates diversity. This is in contrast to the standard ecological arguments that diversity originates from competition to fill new niches. It is believed that this generation of random options and traits are a direct consequence of the system becoming more complex, a natural consequence of the development cycle of evolving systems (see the paper titled Developmental Insights into Evolving Systems: Roles of Diversity, Non-Selection, Self-Organization, Symbiosis on the documents page. )
Furthermore, higher system performance and robustness occurs by the simple combination of the experiences of individuals, even though each individual solves a problem from a limited perspective. Unlike the selectionist view of diversity in natural systems, this study indicates that even in the absence of direct competition between and the consequencial selection of individuals, a higher system performance can be achieved with an alternative mechanism: the non-competitive combination of information from a diversity of individuals. In fact, the idealized system exhibits lower performance or lower stability if any selection is made, either by eliminating participation or reducing their contribution. In some cases, it is even found that eliminating the contributions of the higher performers actually can improve the group solution!
The simulations also illustrate that improved collective performance can only be achieved with minimally skilled (or better) agents. If the agents gain experience randomly (have no "skill"), the collective shows no improvement over the average individual. We conclude that the performance of the whole is tied to the performance of the individual - as the individual learns, the non-competitive combination of diverse experiences increases the performance of the system as a whole.
These results argue for the importance of organizational environment that freely exchanges information for both the benefit of the individual, but also for the group or organization. Many economic and social models of human dynamics begin with the assumption of competitive agents seeking limited resources. Furthermore, they embody significant capability in their agents to explain higher global performance. In the simple model above, system performance is shown to be greater than the capability of the agents and to occur from essentially independent agents randomly sharing information. Mechanisms of competition or cooperation are not required. The results also show that if the collective dominates the learning of the individual, and thereby reduces the diversity, then group performance become less robust.
In modern times of organizations becoming more complex and
facing problems of greater difficulty, centrally directed management of expert
resources may not be an optimal approach to problem solving. For organizations
to take advantage of increased performance from diversity, these studies suggest
that, in addition to a skilled and diverse workforce, it is also necessary
to encourage the expression of diverse views and to enable mechanisms for the
exchange and processing of these views. The implications for all organizations
is to create a work environment in which all employees are willing and able
to contribute their knowledge and experience to solving the problems facing
these organizations.
To better understand the processes of selection in evolutionary systems (ecological to economic to social to artificial systems), the origins and role of diversity are examined in two systems that show increased group functionality (better performance, efficiency, robustness, adaptability, or stability, etc.). Diversity was chosen as a clarifying concept, because it appears to be largely ignored or misunderstood.
One system is a model of group selection within an ecosystem. The other is the group solution of a sequential problem using self-organizing dynamics in the absence of any selection. A comparison of the two systems show that while diversity is essential to both, the improvement by natural selection is derived from "consuming" diversity, while the improvement by non-competitive self-organization is decreased by any reduction in diversity.
The resulting perspective is that natural selection is a mechanism that increases the functionality of the individual (or groups within a larger system); non-competitive self-organization of the system, without need of selection, increases the functionality of the whole above that of the individual or group. The two extreme roles of diversity are reconciled if natural selection is not strongly expressed in these systems &endash; "survival of the fittest" becomes "survival of the adequate" &endash; so that non-competitive processes can occur. The resulting view of a mature ecosystem is an elastic web of interactions where natural selection is dormant or retains the status quo. The processes of natural selection for individual or group improvement are activated only if environment changes are sufficient to "break" the elastic interconnections, as might occur in punctuated equilibria.
INTRODUCTIONThis paper suggests that a significant revolution is taking place in the fields of ecology, economics and social sciences that is changing our understanding of the processes in these systems. The expression of this change is in many forms, from theoretical understanding to experimental studies. The character of the change is based on the growing observations that the traditional views of these systems (mature ecosystems, developed economies and interdependent social systems) have processes that have been overlooked:
Within the field of ecology, both observations are best captured by the sustained work of Salthe, summarized in his most recent book1 and from a different perspective by Kauffman.2 The latter observation is the best captured by the group of researchers involved with the study of Artificial Life3 in the general field of complex adaptive systems.
This paper focuses on the role of diversity in self-organizing systems, focusing primarily on ecosystems. Although the concept of diversity has been part of the lexicon of ecologists and social scientists since the beginning, a quantitative understanding of diversity has been limited until recently. The difficulty is that diversity is only a meaningful property in heterogeneous constituent systems and available analytical tools have been lacking. Recent studies of diversity have primarily been quantitatively advanced by simulations of genetic evolution 4 and knowledge systems,5 and by both analytical solutions and simulations in economics.6, 7 The latter two studies have observed initially counter-intuitive results in non-selective, problem-solving systems, such as how diverse groups solve problems better than individuals, without selection being present. The current study is an attempt to extend this new understanding of the role of diversity to ecosystems.
The following discussion begins with a summary of the traditional viewpoint of natural selection, taking as an example from the recent literature a careful simulation of group selection. As a contrasting example, a model problem is presented where the non-selective interaction of randomly-generated diversity leads to higher system performance. We then introduce a current ecological understanding that de-emphasize selective processes and speculate how the apparently contradictory processes of natural selection and non-competitive self-organization might be integrated into one understanding for ecosystems, with the role of diversity as the pivotal concept.
DEFINING CONCEPTSBecause the following text spans many areas of expertise, the following definitions, assumptions and restrictions establish a common perspective.
An agent or individual refers to any localized constituent or entity with a decision-making or problem-solving ability. It can be a single individual or a sub-group of individuals within a larger system. The decision making or problem solving can be as simple as a deterministic response of a physical subsystem given an initial state and external boundary conditions (because these systems are typically non-linear, deterministic chaos is still possible) or a conscious, premeditative act by a complex human problem solver. A sequence of decisions is a path through the problem domain, each step requiring that a previous problem be solved in order to proceed. For example, a path may be the sequence of events that are associated with decay of an organism or the sequence of decisions for an investment strategy. For example, in an ecosystem, nutrients can take many different paths from the initial creation by conversion of sunlight to use by a lower life form to more complex life forms to the final decay process and recycling.
A group is a collection of agents that solve a common problem, either knowingly or not, cooperating or not, but which share a common view and expectations within the system. Local and global extent describes the degree of proximity of a property to an agent or group of agents. Local extent is limited to the region of the agent; global extent encompasses the system as a whole. Note that local and global are applied to more than just spatial extent. These concepts apply to any system where the information of the agent is limited to their proximity, including more abstract domains of functional space or knowledge space.5 The systems of interest are ones that have no or little centralized control and are self-organizing; that is, their dynamics are such that the system as a whole exhibits self-regulating processes that are largely determined by the properties of the subsystems and the governing processes of the dynamical system. Global properties that cannot be determined from the properties of the constituents are called emergent.
Note that in the above definitions, the concept of decision making or problem solving is used outside of its normal context of solving a "posed" problem. Problem solving is extended to describe a change of state of a subsystem as a consequence of internal processes, which may not explicitly pose a "problem." This liberty in the definition of problem solving is taken in order to apply a common vocabulary to a variety of systems. This approach is similar to how concepts of cooperation and altruism are applied to both cognitive and non-cognitive systems in biology.8
PERSPECTIVE ON DIVERSITYBefore an analysis of the origin and role of diversity in these systems is made, a common understanding of meaning of diversity is needed. In the following, a working definition is given and the parameter space for diversity is discussed.
In the current context, diversity of a group is defined to be the degree of unique differences within a group in which its constituents have a common "world view" (see Johnson5 for a mathematical description). Applying this definition, if all the individuals within a group have identical qualities, then the group has zero diversity, although the qualities of the individuals may encompass all possible variations of the system. If each individual contributes a unique quality not shared by others, then the diversity of a group is a maximum. The restriction to a common construct of the world is necessary, because differences between individuals in a group can arise from different assumptions (world-views) about the system. While this source of differences may appear to be a source of diversity, we argue that comparisons between different world constructs are not advantageous within a self-organizing system. For example, the approaches to problem solving of a New Yorker and Australian bushman are likely mutually exclusive and therefore "unique," but because these approaches operate in very different environments, it is of questionable meaning to measure their diversity (as defined above) and ask how it correlates to system performance. This is equivalent to saying that meaningful expressions of diversity to the system dynamics require the unique contributions to be potentially coupled by the system dynamics. Implicit in the above definition is that diversity is a property of a group of individuals, not of a single individual. Hence, the common phrase, "she has diverse interests" is meaningful only in comparison to a group. Diversity can be a measure of any characteristic of the system at a given time, either in function, capability or information.
Because the systems of interest often have extent (as defined above), diversity can be evaluated either locally or globally. Global definitions of diversity have significance only if the system is coupled globally. For example, if one looks at the correlation between some measure of system performance and some measure of diversity over a greater and greater spatial extent, then at some extent, no correlation will be found as the diversity measure is including states that are no longer coupled by the system dynamics. An illustration of this would be the application of the concept of biodiversity of populations across uncoupled ecosystems, as commonly, possibly incorrectly, is done.
NATURAL SELECTION—COMPETITIVE / SELECTIVE PROCESSESThe dominant model for the advancement of individual functionality within biology is natural selection, often cited as the process of "the survival of the fittest." The role of natural selection in improving the individual fitness is not questioned here, but the exclusive role of natural selection on improving the fitness of the group or global system functionality is questioned. In this section a recent work on multilevel selection is reviewed to illustrate the basic relationships between diversity and selection.
As a beginning point, the generally accepted role of diversity within natural selection can be summarized as follows: 9
These statements are clear about the role of diversity within natural selection, but only for selection within one level and within one niche or closely related niches. Selection between multiple levels (individual, group, metagroups) introduces interdependencies that can lead to more complex behavior. Group selection is one common explanation of the origin of cooperative (altruistic) behavior or processes. The argument is that if selection operates between groups, then traits which are a disadvantage to the individual but advantageous to the group can be selected and propagated.
A definitive paper on multilevel selection was published recently by Pepper and Smuts8 which presents an agent-based ecological model. This work addresses the need for developing the simplest model that illustrates cooperative behavior from group selection. Peppers and Smuts examine the development of observed altruistic behavior in two separate simulations: alarm callers in predator-prey systems and feeding restraint in foraging systems. Mutation is not considered in this work, so they examine the amplification of pre-existing traits. The authors summarize that "…the model has shown that the groups emerging through the behavior of individual agents in patchy environments are sufficient to drive the evolution of group-beneficial traits, even in the absence of kin selection."8 They observe that "(b)ecause of its within-group disadvantage, cooperation can only spread though an advantage in founding new groups. Successful groups must be able to export their productivity from the local area, so that their reproductive success is not suppressed by local population regulation."8 In the absence of "patchiness" or spatial heterogeneity, individuals with the cooperative traits inevitably lose out to their selfish counterparts in this model.
The trait variation, synonymous here with diversity, of within- and between-groups was examined by Pepper and Smuts by looking at different patch sizes and separations of patches. They observed the following:
"Smaller trait groups in turn increased the strength of between-groups selection relative to within-group selection by changing the partitioning of genetic variance. Selection at any level requires that the units being selected vary genetically, and all else being equal, the strength of the selection increased with the genetic variance among units. In a subdivided population, all variance among individuals can be partitioned into within- and between group components, and the proportion of the total variance found at each level strongly affects the relative strength of the within- versus between-group selection. The smaller groups are, the more variance is shifted from within to between groups, and thus the stronger the between group component of selection becomes relative to the within group component. Because small isolated patches reduced trait group size, both small patches and large gaps facilitated the evolution of both forms of cooperation."8
The results of the simulations confirm the three roles of diversity listed in the beginning of this section. The expression of group selection in the simulations does not alter these roles, with the exception that diversity is shifted to between groups and reduced within groups.
ORIGINS OF DIVERSITY IN A COMPETITIVE SYSTEMMuch has been written to explain the source of the observed bio-, social- and economic-diversity, but little quantitative proof of the mechanisms involved or discussion of the role of diversity has been offered. The basic argument10 is that to minimize utilization of scarce resources, material or energy, an individual or group will fare better if it can avoid direct competition with other individuals by creating a new niche, whether spatially or functionally. By occupying and adapting to the new niches, the system as a whole expresses greater diversity. Furthermore, the occupation of new niches can create additional diversity by the further adaptation of individuals that are interdependent with the original re-locator. Therefore, according to this argument, the existence of unexploited niches is the driving force for increased diversity.
This argument is an explanation only for non-local diversity; it does not explain the observed diversity within a given niche, but only between niches. There appears not to be a satisfactory explanation of both local and global diversity in the simple application of natural selection. With more complex models, new mechanisms for diversity may arise. As an ecosystem becomes more complex, there exist mechanisms within group selection that might result in higher diversity within the group. As populations begin to specialize in function, and consequently become more interdependent, a population may function as a mutualistic entity better than as competing groups &endash; an example is that the best house builder is a group of specialists and not any one of the specialists (plumber, carpenter, roofer, etc.).11 Presumably, this diverse, mutualistic entity would have a selective advantage. In the discussion section, we revisit a model with more complex processes, along with the role that diversity plays within a broader perspective.
SIMULATIONS OF NON-COMPETITIVE SELF-ORGANIZATIONIn the following text, a quite different system, than the ecosystems considered above, is examined to investigate mechanisms for diversity creation and its importance to global functionality. The following is a summary of a detailed study.5 We wish to address the question: what is the simplest demonstration of increased global performance of a group above that of the individual? By most simple, we mean the least number of assumptions, processes or rules.
The idealized system examined is the solution of a sequential problem (Insert in Fig. 1), which has many optimal and non-optimal solutions, solved by agents that have identical capabilities and do not interact. While this maze problem is quite simple from a global perspective, it serves as a representation of more complex processes: the solution of a problem that has many decisions points and many possible solutions and that has difficulty greater than that solvable optimally by one individual. It is argued that a more realistic landscape would not change the underlying processes that are observed in this simple model.
The solution process for a single agent is divided into a Learning phase where simple rules of movement are used to explore and learn about the problem domain. Because the agents have no global sense of the problem, they initially explore the problem until the goal is found. The learning process can be thought of as an agent exploring the maze randomly and leaving "breadcrumbs" behind to aid in their search for the goal, thereby avoiding fruitless paths. Then in an Application phase, this "learned" information (the bread crumbs) is then used by the agent to solve the problem again, typically with a shorter path as a consequence of eliminating unnecessary loops. Essentially, the agent follows the path with the most breadcrumbs in the Application phase.
Because the initial search is random, a collection of individuals shows a diversity of experience (knowledge of different regions of the maze), diversity of preferences (different preferred paths at any one location in the maze), and diversity of performance (different numbers of steps), even though each agent has identical capability. This is the source of diversity in the population: by the domain having multiple optimal and non-optimal solutions, a diversity of experience, preferences and performance is created.
In the repeated solution to an unchanging problem domain, we tend to remember only the information that is needed to solve a problem and forget extraneous information associated with unused paths. Here, the equivalent effect is for the agent to remember only "established" information along paths used by individual, thereby "forgetting" unused paths. The process of "forgetting" unused information does not change the performance of an individual agent, because both the learned and established information produces the same path in the Application phase, discounting random choices between paths of equal preference. Therefore, an established individual experience is created from the learned experience by retaining information just used in an individual solution, and forgetting unused information.
The process of forgetting information has been argued as a form of selection in these simulations, but two points can be made. As we shall observe, there is only a quantitative effect of using established or learned information, so the basic conclusions of the simulations are not pivotal by the inclusion of this effect or not. The consideration of effect of forgetting is introduced, as we shall see, to clarify the correlation between diversity and performance: higher diversity leads to higher performance. Secondly, selection in a biological context means that the individual is removed from further contribution to the gene pool, not just part of their gene contribution. There is no equivalent removal of the agent in the current simulations. Indeed, the individual's performance is unchanged by the process of forgetting.
Information for a group of individuals is then constructed by a linear combination of the each individual's experiences at each node in the maze. That is, the breadcrumbs from each individual in a selected group are summed for each decision point (node) in the maze. Then the same Application rules as used for the individual are used on this group information to find a group solution. As seen in Figure 1, the group solution always outperforms the average individual for larger groups, and the solution using the established individual information performs better than the learned information. Furthermore, for groups above 20, the optimal solution is found, although nothing in the agent's rules seeks a minimal path length. Figure 2 shows one mechanism for the reason that the group does better than any agent: individual information is combined to indicate a shorter path for the group (follow the maximum bread crumbs at each intersection). The dynamics of the group solution are chaotic in detail. For example, the specific path of a group is sensitive to the addition of one individual, even for arbitrarily large groups. Nevertheless, the global solution for the group, any path of minimum number of steps, is stable. This illustrates the desired feature of chaotic dynamics that leads to a responsive and robust system, but not at the expense of the quality of the global solution.
Fig. 2. One mechanism for the better performance of the group. Note that the path length of the group is better than any agent.
To better understand the role and importance of diversity in this simple model, quantitative measures of diversity were examined. The best measure found defines diversity as the degree of unique information in a collection of agents, based on a node-by-node comparison of preferences, as defined in an earlier section. Groups contributing "established" rather than "learned" information exhibit higher diversity, although less information is available. Moreover, as observed in Fig. 1, the groups based on established information perform significantly better than those based on learned information. Furthermore, this measure of diversity also indicates the degree of insensitivity to noise. In the process of combining information for the group, if valid information of an agent is replaced by random information (breadcrumbs are randomly replaced with some amount), this is a test for the stability of the group solutions. It was found that groups with low diversity were very sensitive to noise, where groups with high diversity were not: up to 90% of valid information can be replaced before a group path degenerates to a random walk &endash; the worst solution of all methods.
All of the above studies assumed that the agents do not share information while learning or applying information; they are completely independent, except that they solve a problem with a common world view. If the effect of information exchange is included such that the individual can benefit from other agent's experiences while learning the maze, we find that improved individual performance is achieved. Not unexpectedly, groups made up of these shared-learning agents, converge with fewer agents to a minimum path length, much faster than observed in Fig. 1. But, the improved performance is not without a cost. Shared learning results in individuals with similar information and, therefore, the group exhibits low diversity, and, consequently, the stability of the group is degraded, often severely.
How does the performance of the group depend on the individual performance? Two studies were done, one in which the mazes were made more complex while the individual's capability was held constant, and the other in which the maze was held constant and the individual's capability was varied. From these studies the following conclusions were drawn. 1) A simple maze to a good individual solver is a trivial problem, and no improvement is obtained by a group solution. 2) More difficult global problems require larger groups. 3) An extremely difficult problem to an individual with fixed capability leads to a random individual solution that shows no group advantage. The last conclusion is significant; it suggests that harder and harder problems cannot be solved by larger and larger groups of individuals. Or, equivalently, the individual must have some capability (i.e., not random) which can be amplified in groups. This observation is also related to the assumption of having a common world view. A different world view in the above simulations is equivalent to solving a maze with different connectivity between the nodes (i.e., each agent sees a different set of possible paths at each node), while still having the same common goal. While not demonstrated in the simulations described above, it is expected that a group of agents with "capability" above a random walk would perform poorly as a group, because the information that each contributes does not correspond to a common world view and, hence, will not be compatible and cannot be amplified. Therefore, different world views and limited individual capability both lead to no better group performance than the average individual.
DISCUSSION: THE ROLE OF DIVERSITYThe main observations of the above non-competitive simulations are twofold. One, they illustrate how diversity can arise in groups of agents of identical capability when a system has options of equal likelihood or fitness. Secondly they illustrate how random creation of diversity can contribute directly to both global performance and robustness, above that of an individual and in the absence of any selection from the population. Both of these observations are in direct contrast to the processes observed in natural selection when applied to a single level. In this section, we try to reconcile the two extreme approaches to higher functionality, represented by the two models summarized above.
In both models, diversity is a prerequisite for improvement to occur: without diversity, there can be no improvement. For natural selection, this improvement is for the group by increasing the capability of the individuals in the group; for non-selective self-organization it is for the group, assuming shared learning is not present. However, this is where the similarity ends. Once the necessary functionality is achieved by natural selection, the immediate need for diversity is lost. Thereafter, having diversity at a current time becomes an investment for the future. When selection processes operate at a single level, diversity does not directly contribute to current system performance; only past diversity contributes to the current performance, and then only by the reduction of diversity in the selection process. Indeed, diversity can be argued to lower the group performance in a natural selection viewpoint by the inclusion of individuals with poorer fitness. By direct contrast, diversity in the non-competitive system directly contributes to performance and robustness at the current time.
The above differences between the two extremes of self-organizing systems can be reconciled if the predominance of natural selection and competition in ecosystems, and in general all self-organizing systems, is relaxed. A possible beginning point is to adopt a less competitive view of ecosystems, where "the survival of the fittest" is replaced by "the survival of the adequate,"12 also referred to as "soft-selection."13, 14 Essentially this is a statement that in mature ecosystems, significantly greater expression of random variation is likely and there is no need to select among this diversity. This is equivalent to the observations in the non-competitive simulations that the source of diversity originates from indeterminacy in the solution space; one path is, at an individual level, as successful as another. For ecosystems, this argument requires that there exist multiple paths of near equal fitness. Mature ecosystems in nature are observed to have rich interdependencies.15 These interdependencies create many alternative paths for energy, material and information. Consequently, global system function is not dependent on a single critical path as, for example, in the many alternatives for transforming sunlight.
The flexibility to express random diversity is not a sufficient argument for high diversity alone, because it begs the question why such flexibility exists. The non-competitive simulations provide insight into this question. The existence of multiple paths leads to a chaotic, but robust system. Ecosystems are chaotic in the same sense as the present study: a small change in initial conditions, or by the addition of noise, results in a different paths or different distribution of paths through the system. An example of the chaotic but robust nature of evolutionary systems is the recent theories of "frozen accidents"2, 16, in which the details are chaotic, e.g., the specific base-pairs in DNA, but where the global need is still satisfied - the need to develop an encoding system for passing on information to offspring. Another example is the material-energy path in an ecosystem: a slight difference in predator location can result in, say, a wolf consuming a dying deer, instead of bacterial decay, but in either scenario the global need for recycling nutrients is achieved. As observed in the non-competitive simulations, this chaotic property leads to the responsive nature of the system and prevents the failures due to senescence. 1 Nevertheless, the system is also robust in the sense that the global solution is stable, and not sensitive to random details or localized failures.
A final result in the non-competitive simulations that is not unexpected, but does complete the global view of an ecosystem: the global performance, including robustness, is directly coupled to the performance of the individual. The ecological example of this would be if all populations that convert sunlight to useful forms of energy did this only erratically or with lower efficiency, then the global system as a whole would be less productive and robust. This suggests that from a global perspective, natural selection is needed to make the individual adequate to the global task of survivability. But because of the complexity of typical evolving systems, particularly in the degree of redundancy, it is difficult to argue what is essential and what is adequate for survivability. We can conclude only that some degree of individual performance is necessary, and natural selection is the likely mechanism for providing the functionality.
The above speculations result in a powerful, multilevel perspective that can be simply stated. Natural selection is responsible for improvement or adjustment of the relative performance of the individual. But, once an interdependent, multiple-level system develops, then the need for selection is reduced, as the non-competitive processes for global performance become operative. Because both self-organizational processes require high diversity to be expressed, this creates an optimal system where either process can be operative as need be.
One aspect of diversity that has not been discussed yet is the tradeoff between diversity, and its associated advantages, and the development of a common, and sometimes universal, functionality or approach. In ecosystems, these "cohesions" across populations can reduce diversity by limiting the potential space for random variation. An extreme example of this standardization in living systems is the universal adaptation of the DNA/RNA coding system &endash; one of the few exceptions to the "rule" of biodiversity. In the non-competitive simulations the effect of information exchange during the Learning phase captures this concept. The reduced diversity of the system, and its associated decrease in robustness, is a tradeoff for the improved performance of the individual and small groups. This might be a mechanism for the beginning of an expression of specialization. Possibly, in a more complex simulation where groups could differentiate, this cohesion effect could actually lead to higher system diversity, and its accompanying advantages, as different populations define their own "uniqueness," an analog to speciation.
What remains unanswered in the above argument is by what processes do global system functionality, such as robustness, become operative. It appears that desirable emergent global properties occur in many ecological, economic and social systems, but it is not clear how the properties in the individuals which result in these emergent global properties come into existence. There are two views of their origin. One is the view that the system itself contains these "structural" aspects and organisms form and exist within this structure.1 The other is that there are mechanisms for the global coevolution of the necessary traits to propagate the global system. 2 The present understanding does not resolve these issues.
CONCLUSIONSThe role of diversity, defined as the degree of unique contributions of individuals to a group, is compared in two extremes of achieving higher system functionality, natural selection and non-competitive self-organization. While both processes require diversity to function, there is a fundamental difference in the mechanisms by which diversity is activated. Natural selection consumes diversity to yield improvements for a later time; non-competitive self-organization yields improvements at the current time without selection from the population. The two extremes are argued to be compatible in ecosystems if there exists many alternative paths (energy, mass or information flow) of near-equal fitness. This flexibility is argued to occur in mature ecosystems.
The resulting viewpoint of such an ecosystem is not static, but one which has "elastic" interconnections of many entities and processes. Interconnections are chaotic in detail, but globally robust. The response to moderate changes in the environment is the readjustment of these elastic interrelationships, but not their failure. The role of natural selection is largely dormant as a process of improving individual or system functionality. Where active, natural selection operates at the level of rejecting the least fit mutations and preserving the existing interconnections. If a sufficiently rapid environmental change occurs, then there is a catastrophic breakdown of this stable elastic web, which in turn precipitates a high degree of natural selection and a reformation of a new elastic system, based on significantly different interrelationships between new functional forms. We speculate that this process may be the true origin of the proposed punctuated equilibria. The advantage of this viewpoint is that it does not require the questionable assumption of reproductive isolation.17 The true test of the above speculations will require a sufficiently complex simulation that contains randomly mutualistic interdependence, combined with mutation and natural selection.
ACKNOWLEDGMENTSThe author gratefully acknowledges insightful conversations with Stanley Salthe, Dan Brooks, Mark Bedau and Shareen Joshi. This work is supported by the Department of Energy under contract W-7405-ENG-36
REFERENCESMany of the arguments and examples given above for social systems and ecosystems are directly analogous to economies and markets. Instead of developing this topic separately, some illustrative examples are provided here. First some general comments are made.
An excellent summary of economies as (self-organizing) complex systems is given in the introduction of "The Economy as an Evolving Complex System II," written by and edited by W. B. Arthur, S. N. Durlauf and D. Lane (Addison-Wesley, 1997). They cite the properties of economies that are a challenge for traditional models: dispersed interactions of possibly heterogeneous agents, no global controller, cross-cutting hierarchical organization, continual adaptation, perpetual novelty, and out-of-equilibrium dynamics. In the following text and papers, the points that they emphasize relevant to the current discussion are the importance of networks of agents (interactions are local, not global) and of informal social processes in information transfer. What is missing in the discussion is a consideration of non-selective processes in the evolution of the markets and the role of heterogeneous (diverse) agents in the system performance. The above introduction is supplemented by the following observations.
Similar arguments have been made as to the importance of a diversity of agents in an economic system - how stability and improved performance can occur in systems with diverse participants. This is a major conclusion of a recent analytical and computational study of the stock market (J. Doyne Farmer, Market Force, Ecology, and Evolution). What is missing from the discussion is how, by having agents with diverse market strategies, the performance and robustness of the market can be improved as a whole. Similarly across many studies of economies or markets, the starting assumption is that the agents in the systems are essential competitive and the processes are selective of only the fittest participants. At first reflection, this may be the most common view of the economic world, but on deeper observation of the relative effectiveness of the players, our experience is that many participants are far from the "fittest" competitors and, in fact, there is significant tolerance of all but the least "fit." Just as in the arguments presented for ecosystems, the lack of selection leads to more diversity and the higher diversity leads to higher system robustness.
A common argument that is made is that an efficient market is a source of diversity - because new strategies are selected to accommodate new market dynamics. But just the opposite is true, as in ecological arguments, an efficient market would ultimately reduce diversity by eliminating less effective strategies. In our experience, just the opposite appears to be true: diversity of strategies can exist within an efficient market. Again, the conclusion is that in mature economic systems, the selective forces resemble the "survival of the adequate" rather than "survival of the fittest." Parallel arguments can be made that once diverse agents are present, the system as a whole can form a web of loose interactions that enable the system to be more robust, and can possibly perform better.
The above discussion becomes even more cogent when applied to the current US stock market. Because of the influx of investors from a far wider representation of the public than ever before (50% of American family by current estimates), there is a much broader diversity of investment strategies. This single observation has many consequences that the arguments for social and ecosystems predict. Given the diversity of this system, it is not surprising that previous models of market prediction for homogenous investors are failing. Prior models assume some simple set of dominant strategies, but the current market is driven by possibly unpredictable combination of diverse strategies - some of which may seem random to a professional investor. This is supported by the observation that only a diversity of investors can predict the market, and not any one type of investor at any one time (See stockjungle.com, a company that uses players of a stock market game to manage an actual mutal fund. Also, a prior web page, investorsforecast.com, showed how predictions by diverse investors are consistently better than the predictions of any single cross-section of investors.) Similarly, the presence of diverse agents also is a compelling argument of the surprising stability of the current market, in contrast to many predictions otherwise by seasoned analysts. Because of the diversity of market strategies, there is significant insensitivity to prior observed sensitivities of the market. Indeed, the more random component of a novice investor may reduce market efficiency, but significantly increase market robustness.
The one caveat to the above observations is that human systems are highly sensitive to excessive coherence of actions (and thereby loss of diversity) because of mass communication . The increase in robustness argued above can be lost if there exist a condition where the diversity of investment strategy is lost due to widespread positive feedback, leading to a coherent action. Although this possibility always exists with human social systems, it is also balanced by the diversity of the components of the system. Likely destructive coherence is more probable in system with lower diversity than larger diversity. This may be the explanation for the continued stability of the market, despite significantly large influences in the market, such as the crisis of the Pacific-rim economies.
Publications | Talks | Interests:
Past & Present | Symbiotic Intelligence
Project
Contact | Science of Diversity | Home | Links | Diversity Documents | Norman Johnson New Topics: Infectious Idea Spread using Epidemiological Modeling | Collective Intelligence in the presence of bias and misinformation Community, Coexistence and Pre-conflict Minimization Projects | The CISS (Coexistence in Suggestive Simulations) Project ©2013 Norman L. Johnson |