The Science of Diversity and Its Relevance in a Fast Changing World

One would think that the role diversity plays in many components of science, from ecology to politics to the stock market, would long ago have been understood. Unfortunately, our social views of diversity are in conflict with our mainstream scientific views. The different roles of diversity are presented within the context of a familiar example (foraging among social insects), and understood within a theory of systems development. This viewpoint may resolve any confusion associated with diversity in organizations. With greater clarification, we may better understand the advantages and disadvantages of diversity in organizations: Does diversity lead to conflict or does it lead to synergy and better solutions? This analysis is timely because of the importance of diversity in decision-making in complex environments, particularly in times of rapid change and particularly post 9/11.  

Diversity in the workplace, organization, even nation, has recently become a much touted goal. Our understanding has evolved to recognize that diversity includes more than ethnic variety, but also cultural and educational variety, in fact, any aspect of an individual that leads to a variety of approaches to problems, even personality. Yet, the scientific basis behind the role of diversity in social systems is largely unknown. Research at Los Alamos is providing support for our intuitive understanding of the importance of diversity in social systems.

Much of the past understanding about social diversity is motivated by the analogies to biodiversity in nature. The prevailing research on the role of biological diversity focuses on improvement of system performance (e.g., individual or species survival) by the selection from a pool of genetic diversity, in the process of natural selection or survival of the fittest. This process is fundamentally competitive between individuals or groups, with explicit winners and losers. This selection reduces diversity in its application – diversity is reduced by selection and must be replenished by mutation or migration. While a similar selection process may occur in social systems, this is neither an compelling justification for diversity within organizations, nor the only mechanism by which diversity contributes to better organizations or society.

Current research at Los Alamos has identified an alternative process for higher system performance, which does not involve competition or cooperation between the individuals, but instead stresses the importance of non-competitive self-organization.

Simulating collective problem solving

We wish to address the question: what is the most simple demonstration of increased global performance of a collective above that of the individual? The idealized system examined is the solution of a sequential problem (Insert in Figure 1), which has many optimal and non-optimal solutions, solved by agents that have identical capabilities. While this maze problem is quite simple, it serves as a representation of more complex problems encountered by individuals and organizations: the solution of a problem that has many decisions points and possible solutions and that has difficulty greater than solvable by one individual. Here, agents can represent individuals, groups or organizations within a greater system.

Because the agents have no global sense of the problem, they initially explore the problem until a solution is found. This “learned” information is then applied by the individual agent to solve the problem again, often with a shorter path as a consequence of eliminating unnecessary loops. Because the initial search is random, a collection of individuals shows a diversity of experience (regions of the maze), of preferences (preferred paths), and of performance (path lengths), even though they started with identical capability. (This source of diversity in performance by individuals of identical capability is a reminder how many of the advantages and disadvantages an individual has in social situations are a result of random events and not directly related to our “inherent” capability. Said another way, if we replayed or lived our own lives again, likely it would be quite different due to these random effects.)

Information for a collective of individuals is then constructed by a linear combination of the each individual’s experiences. Then the same rules are used on this collective information to find a collective solution. As seen in Figure 1, the collective typically outperforms the average individual for larger collectives. In repeated solutions to a problem, we tend to remember only the information needed to solve a problem and forget extraneous information associated with unused paths. Here, the equivalent effect is for the individual to contribute to the collective only “established” information along paths used by individual, thereby “forgetting” unused paths. Both the learned and established information produce the same path for the individual agent. As seen in Figure 1, the solution using the established information performs better than the learned information. Furthermore, for collectives above 20, the optimal solution is found, even though nothing in the agent’s rules seeks a minimal path length. Figure 2 shows one mechanism for the reason that the collective does better than any individual: individual information can be combined to indicate a shorter path for the collective.

Importance of Diversity

To better understand the role and importance of diversity in this simple model and gain insight into social systems, quantitative measures of diversity were examined. The first choice of a measure, the breadth of experience of a collective over all possible paths, was found to be uncorrelated with performance.

The best measure found defines diversity as the degree of unique information in a collection of agents. If all agents contribute the same information, even if it is for the entire domain, then this measure of diversity is low. If each agent contributes unique information not shared by others, then this diversity measure is high. Consequently, collectives contributing “established” rather than “learned” information exhibit higher diversity, even though less information is contributed by each agent. So we conclude that it is not how much information is important, but how the information contributed fits in to the other information known.

Not only does this measure of diversity correlate best with collective performance, it also indicates the degree of insensitivity to noise. The performance of a collective with low diversity is poor when valid information is randomly replaced with false information, a measure of the robustness of a solution. False information can lead to unexplored paths in a less diverse collective, and then the solution degenerates to a random search (imagine driving and finding yourself on an unfamiliar road – the solution is either to backtrack or to search randomly). Diverse collectives have contingency information that makes the groups highly insensitive to noise. The stabilizing effect of diverse groups is critical in dealing with difficult problems where false information can lead to unexplored paths. In research which focuses on rational individuals with perfect information, an evaluation of performance often does not consider the robustness to noise, hence, the reason that diversity is rarely considered important. Yet, robustness is a critical aspect of any modern organization.

If the effect of information exchange between individuals (a form of cooperation) is included in the above simulations, such that the individual while learning the maze can benefit from other agent’s experiences, we find that improved individual performance can be achieved. But if the outside information is used too strongly, then there is the ultimate loss of diversity in the collective as every individual has the identical experience. In this case, the robustness of the collective can be severely degraded if the positive feedback or coherence is too great. We conclude that cooperation is actually a form of selection and reduces the expression of diversity (if I cooperate with you consistenly, then I don’t cooperate with someone else). It is easy to see that the random associations that are beneficial to the group and individual above can be reinforced to the point of being exclusive. While this leads to more optimal performace, it also causes the groups to be less robust. (To see how this transition is part of a bigger view of how systems develop, see the paper titled Developmental Insights into Evolving Systems: Roles of Diversity, Non-Selection, Self-Organization, Symbiosis on the documents page. )