Dr. Norman L. Johnson Contact norman@santafe.edu Self-Organizing Knowledge Systems: Enabling Diversity Norman L. Johnson (Ph.D.)   Theoretical Division, Los Alamos National Laboratory   4th Annual Desktop Collaboration Conference & Exposition 11-14 May 1999 Founders Inn, Virginia Beach, VA (LA-UR-99-1724) (Viewgraphs are also available)

Abstract

Within the last few years, major advances have been made on enabling collaborative work with intelligent networks. But despite the availability of new capability, there still remains both a limitation of the difficulty of problems that can be addressed and a lack of understanding of the needed processes to further evolve the capability.

A unified view of the future of the intelligent networks in both organizations and society is presented. In its most simple realization, individuals or groups just by solving problems of local interest contribute to a problem solving capability much greater than the individual or group. Motivating analogies are, for example, ecosystems, distributed capitalistic economies (Adam Smith's invisible hand of self-regulation) and some man-made distributed systems (information and power networks). These self-organizing systems appear chaotic in detail, but exhibit stable global properties. They are enabled by contributing members with diverse perspectives and capabilities.

The understanding for the increased power of self-organizing knowledge systems comes from an enhancement of the pre-existing, self-organizing dynamics of social processes, when combined with the unique properties of intelligent networks. The symbiotic combination of the ultimate problem solver &emdash; humans, not machines &emdash; with intelligent networks can result in the much sought-after capabilities of automatic knowledge capture and creation, problem identification and problem solving greater than any centralized solution &emdash; all within an adaptable, robust, noise-tolerant, conflict-resolving system. Research is presented that identifies the chaotic, but bounded dynamics of these self-organizing decision systems and the required components and institutional policies for enabling the system.

Overview

We envision an unprecedented capability in organizational and societal problem solving will result from increased human activity on smart distributed information systems, like the Internet. This heightened capability will be an enhancement of the self-organizing social dynamics that has guided human groups, organizations, and societies for millennia. While sufficient connectivity is developing on the Internet (and intranets) to activate this transformation, the enabling methodologies and theories, along with the data needed to test the new technologies, are lacking. A multi-disciplinary, multi-institutional effort, The Symbiotic Intelligence Project, is being developed at Los Alamos with other institutions to provide the needed methodologies and theory for this new science. Los Alamos is ideally positioned to coordinate and contribute to this effort as the key laboratory in the study of complexity, artificial life, distributed processing, and precompetative technology transfer.

Society and the Internet

We have evolved social structures, and the supporting dynamics, which enabled us to "solve" problems that threaten our existence. Social and biological evolution use the same dynamic processes, and exhibit the same characteristics, inherent to self-organizing systems:

• A "Solution" arises as a selection by the system dynamics from a diversity of potential solutions.
• Complex global behavior is driven by loosely connected, relatively simple, local processors.
• The global characteristics of these systems are: functionality greater than the individual subsystems, robustness, persistent nonequilibrium, the capability to find solutions in the presence of conflicting needs, and scalability without loss of viability.

In social dynamics, the "local processor" is the centralized problem solving of individuals, teams, organizations, and governments. This centralized approach relies on the premiere problem solving ability of humans to understand complex systems and to be able to predict their outcome.

As a species we have gone through many fundamental reorganizations without centralized coordination: from hunter-gatherer to agricultural to industrial societies. Technological advances have radically changed the time and size (length) scales of the social dynamics, the prime example being the increase in the maximum size of a social group. Through advances in communication, transportation and information storage, our largest social unit has grown from tribes, to cities, to city-states, to nations, to regional coalitions, to almost a global society. Not unexpectedly the next revolution will also occur because of a new technology, the Internet.

The Internet has three significant capabilities beyond prior human-technological systems: (1) The Net integrates the breadth of knowledge processes: information storage, communication of information, traditional computing of huge amounts of data, and human processing of limited but complex information. Until recently these were physically separated processes that required human action to integrate. (2) The Net captures the depth of knowledge processes. The detailed signatures of the use of information, such as the Amazon.com example below, are captured by the Net. This information was previously obtainable only at extreme cost. (3) The Net transmits knowledge processes accurately. Verbal human-to-human communication results in a rapid loss of information a bit removed from its creator. By contrast, information exchanged or related on the Net, whether in web pages or emails, suffers minimal loss of information during transmission or linking, in the same way that the contents of a book or journal are not altered when exchanged.

With the stronger presence of these three capabilities in human information processes, we propose that a knowledge "phase change" will occur: these processes, which were previously lost or diminished before, will contribute to an enhanced self-organizing social dynamics.

An example of collective problem solving: Have you ever clicked the "Show me related books" on Amazon.com? These lists are not constantly updated by staffs of librarians for 2 million books - an impossible task. Instead, these lists are created by listing books purchased by people that have also ordered the selected book. This is an ideal example of capturing information that was previously lost (#2 above) and of how individual behavior leads to useful collective knowledge. Other demonstrations have been done or are in progress that illustrate how humans on intelligent, distributed networks can achieve useful collective solutions or knowledge, but with minimal instruction and effort.

Simulations of collective problem solving:

Simulations are being done at Los Alamos that support the fundamental assumption that more people can effectively interact under "Internet-like" conditions than in traditional ways. These also illustrate the desirable features of a large and diverse self-organizing system. We wish to answer the questions: Can groups of independent individuals solve hard problems? Under what conditions does the collective advantage occur? What degrades it?

The idealized problem is a maze which represents a bounded system where individuals are faced with a sequence of decision points that lead to a final goal. In the Figure at the upper left below, a sample maze is shown with two of the 14 paths of minimal length highlighted. A simulation consists of two phases. In the Learning Phase the individuals randomly explore the maze until the goal is found. In the Application Phase they apply what they have learned to "solve" the problem. Because the rules in the two phases involve random choices, a population of individuals result which represent a diversity of performance and preferred paths. For now, these individuals are taken to be independent, because we are interested in isolating the effect of collective decision making from the processes of shared learning. To find a collective solution, the individual contributions are then combined in various ways and then the same Application Rules are applied. This collective solution is then compared to the average performance of the individuals in the collective. The effect of learning by the individual can be examined by excluding information on paths that are not used in the Application Phase.

In the sample maze, the average number of steps of an individual to "solve" the maze is 34.3 in the Learning Phase and 12.8 in the Application Phase for 100 individuals. If a collective experience is constructed by a simple average of the individual's experiences and then the same Application Rules are applied, the performance of the collective is better than the average individual when the learned experiences are used (Figure at the upper right). Also one of the minimum paths is found for collectives with more than 20 individuals. This optimal solution (nine) is found even though the rules have no global sense of the problem and do not select for a minimum path. This minimum solution is, therefore, an emergent property of the system. The system also exhibits classical chaotic behavior in that a small change in the collective path preference can result in an entirely different, but still optimal, path.

The collective solution is remarkably robust. Degradation of the individual's contribution, however implemented, generally had no effect on just postponed the collective convergence to the minimal path. For example, by appropriate modification of the individual's contribution to the collective, we find that the less dominant opinions of the individual have no effect on the performance of the collective decision. Also the collective solution is insensitive to moderate amounts of random noise.

A few effects were found to degrade the collective solution. The worst degradation (Figure at the bottom right) was found to be the random selection of a "leader" from the group in the collective during the solution, with a different individual selected at each node. Using this idealized problem, it is can be shown that a diversity of learning experiences leads to a better performance in the collective decisions and that the collective is significantly more robust than the average individual in adapting to unexpected changes in the final goal or the structure of the maze.

The Future

The implications of the ideas under investigation are far reaching to the future success of organizations, society, and economics. With the increased use of the Net in our economy and society, many of the problems that have challenged traditional forms of management and organizations may now find solutions. The importance of this alternative will become more significant as the complexity of our world increases and our traditional ways of solving problems fail. The main consequence will be for management and governments to facilitate the use of collective problem solving when problems arise in systems that are not understandable.

Figure for above text