Basic questions about self-organizing, distributed systems:
Concerns about the using a self-organizing system for problem solving
Concerns about the implementation of the system:
Natural systems: Social insects (termites, bees, ants), "Free" economies, Large companies (too large to be truly managed centrally, e.g., Microsoft), Immune system (unlike most people's perception that the immune system has a central coordinator, it is in fact highly distributed and takes a diverse approach to protection of the body), ecosystems, societies larger than the largest social unit (family, tribe, community,...), Modern battlefield "management", Crisis management. Physical systems are often overlooked as a distributed, self-organizing system, e.g., where atomic-scale atoms determine macroscopic properties. Possibly the dynamics of the human brain.
Human-made systems: Traffic systems and control, Power systems, Internet routing, TelCom system, Artificial Life studies.
We do not use the words "complex adaptive systems" in the present study because of its popular application to any emergent system, including idealize dynamical systems which may not be distributed in any sense. We choose instead to use distributed, collective, self-organizing systems (or simply self-organizing systems) to describe distributed systems with discrete subsystems (entities) that exhibit emergent properties.
The attributes are:
- 1. Distributed resources, processes and information.
- 2. Absent (or limited) centralized control, planning or prediction.
- 3. Diversity (often redundant) of dynamics, capabilities or "goals".
- 4. Dynamics of persistent disequilibrium, the absence of a stable steady state.
- 5. Mechanisms for information loss, filtering or condensation.
- 6. Limited connectivity of typically local extent.
And global features are:
- a. Global functions or solutions from local actions or rules.
- b. Robust, resilient, adaptable, fault-tolerant systems.
- c. Function of the whole not dependent on individual subsystems.
- d. Scalable to larger size without loss of function.
Let us examine the dynamical system of atoms in a gas as modeled by hard spheres (meaning that the atoms contain no internal structure and only interact on contact). On an atomic scale, the atoms are in constant motion (item 4 above) and over any small time interval interact with atoms nearby but not atoms at any distance (item 6). In the determination of their dynamics, there is no global control (item 2), only local interactions. All necessary information to characterize completely the system is contained in the atoms: their position and velocity (1). Their velocities and local number density will not be single valued, but a distribution (3) that varies with specific atoms and the location in the domain. On a macroscopic scale, the hard sphere gas exhibits global properties (item a). But if we look at a single atom, we cannot determine its global properties, e.g., viscosity. It is only through the collective dynamics that the property of viscosity is measurable and defined. Hence, viscosity is an emergent property of this system. And this global property is not dependent on any single atom; if one were removed, the viscosity would still be defined and unchanged (5, b, c). And finally, the system functions at any macroscopic size (d).
Kinetic or statistical mechanics theory of gases has long been established. These theories assume the system is comprised of collections (a collection of just one type, hard spheres, in the above example) of identical entities with dynamics governed by the same set of rules or equations (Hamiltonian equations in our hard sphere model). At the level of atoms, the system is deterministic (paths of the atoms given the same starting conditions are identically reproduced). But at a practical level, the motions are stochastic (initial conditions cannot be sufficiently controlled nor the equations accurately solved to reproduce the identical paths). Because of the stochastic variations and the need to eliminate information that is not necessary to properties of interest, global properties are defined from ensemble averages over many realizations, where a realization is the time evolving state resulting from an identical set of entities starting from different initial conditions. Each realization results from one selection of initial conditions. The process of applying the ensemble average to determine a global property selectively uses information from part of the entire system (such as the dynamics of the atoms in the above example) and discards other information that is not needed (such as the history of a single atom).
We argue that this is the identical approach that is needed for other self-organizing systems of interest, such as social systems.
Artificial Intelligence efforts try to duplicate, reproduce or capture the capabilities and functioning of natural intelligent systems, such as in the translation of languages or the creation of an expert system. Although the promise has been great, the progress has been disappointing. The Symbiotic Intelligence Project states on the outset that humans are the essential processors of complex information and premiere problem solvers. The Project takes advantage of the best of human capabilities, ones which will likely not be duplicated for a long time, and enhances them with an improved medium for communication, storage and knowledge creation. Hence, this is not an effort in Artificial Intelligence.
The Artificial Life studies attempt to create and study systems that demonstrate many of the attributes we associate with life. Often this work was motivated by the frustration that biological systems are difficult to control and to obtain essential information as needed for quantitative analysis. And in the case of evolution, much of the early formation of life happened billions of years ago and the processes by which life first evolved is not available for study. To address these frustrations and shortcomings, researchers created artificial systems that duplicate the processes of life, primarily in the data rich environment of simulations. Much of the understanding that has developed by Artificial Life studies is directly applicable to the present work, and the current project is best seen as an extension of many of these earlier efforts.
As for the applications of Artificial Intelligence, Artificial Life efforts also suffered by the exclusion of the natural participants in the processes.
Because the processes of interest in the present study typically involve sequential problem solving, this is a new area of investigation from prior studies in Artificial Life, e.g., in genetic algorithms or neural nets. In particular, there appears to be a central difference in the approach of modeling these systems: the use of selection and competition versus cooperation and symbiosis.
The Global Brain study group involves a diverse group of people, brought together by the common belief that the Internet will someday become an intelligent, if not sentient, being. Because of the absence of an expressed common understanding of the group, it is difficult to make a comparison to the diverse viewpoints of the participants. If there is a difference, it is one of emphasis on the role of humans in the global brain. The current project has humans as the primary processors of complex information within a larger social organism, enabled by a new technology. This viewpoint is close to that expressed by Heylighen in From World-Wide Web to Super-Brain. In contrast, Goertzel has given a perspective that the hardware of the Internet will develop as a global brain, in which the human component will be secondary. The differences in these viewpoints may be less substantiative and more a question of time scale (next century versus next millennia).
This question is answered in brief in the Overview and is discussed in detail in the alternative overview from the perspective of problem solving as a societal need. The essential point is the our species (and others) have biologically evolved a dynamical process to solve problems at a global level without the necessity of genetic alteration. This results is a social equivalent of the much desired Lamarckian evolution, establishment of "inheritable" solutions within the lifetime of an individual.
The fundamental premise of the Symbiotic Intelligence Project is that the pre-existing social evolution will be greatly enhanced by the Net. Unfortunately, in contrast to biological evolution, the role of social evolution in our development as a species is just beginning to be appreciated. And, the examination of the biological co-evolution (e.g., mobility, larger brain, speech, personality, emotions) that enables the social evolution to function optimally has barely been undertaken. Hence, within the project, there is a need to understand the processes in the pre-existing social evolution, while at the same time extending its function to a new technology, the Internet.What are the characteristics of social evolution?
These are listed in the Introduction in a study of collective problem solving, in the context of distributed, self-organizing systems. Social evolution posses many of the same characteristics as biological evolution, the least appreciated being (1) persistent disequilibrium (the absence of a stable state), (2) the absence (or of low extent) of centralized control and (3) the lack of unique solutions. These attributes are emphasized over the others, because they offer unique insights or challenge our common understanding. These are briefly discussed below; a more detail discussion can be found elsewhere. Without discussion, we also note that diversity is essential to both social and biological systems.
(1) The disequilibrium of social systems has the consequence of an apparent chaotic dynamic. As individuals within this dynamical system we often see more chaos than stability. But on a global level our social system is observably stable, although we would have difficulty to say why.
(2) As discussed in the alternative overview, our perception is that our organizations and governments are "in control" and are able to guide the course of society, in all its components: technological, economical or political. Certainly these forms of centralized control can influence the overall dynamics. But centralized efforts, particularly on a large scale, often lead to consequences that were not originally intended, resulting in the need for repeated corrections. This inability to control can either occur from lack of understanding in the system as a whole or from the inability to implement the understanding of the system. What often is not appreciated is that the system has a self-organizing dynamics.
(3) The lack of a unique solution in biological and social evolution may be the most challenging concept. What is meant by this is the inability of the system to return to the same point in detail if it is "restarted" from an earlier point. This is an essential consequence of the chaotic nature of the system. Suppose one could wind the clock back to some earlier time in our history and "restart" the social dynamics again. Just as has been argued for biological systems, the new outcome of the experiment will be different in detail from the previous experiment. We qualify the above by saying that the outcome will differ "in detail" because the system as a whole may find alternative solutions for global problems which differ in detail, but not in function. A simple example of this is given in the study on collective problem solving.What proof is there that social evolution exists?
Because we can't turn back the clock (see the last question) and because we lack the data necessary to test the concept, little proof can be given. The situation is similar to the early days of the theories of biological evolution. It was not until fossil records substantiated the history of evolution, that the theories became a scientific certainty. The current project, and others like it, may be able to provide the essential data to validate the idea of social evolution, due to the ability of the Net to capture detailed signatures on the dynamics of knowledge creation, the core component of social evolution.How does social evolution differ from biological evolution?
(Two alternative perspectives on this question are of interest. See "What is wrong with the concept of human (social) evolution?" This contains an excellent review of the history of the question. Also see the working paper for the CLEA study group "Evolution and Progress.")
This is an unresolved question, largely due to the absence of hard data to understand the dynamics in either system. But one point of difference might be suggested, the idea of progress, and is illustrative of some important concepts.
The concept of progress in biological evolution has been a controversial from the beginning (see the link above). We like to think that humans are the culmination of 5 billion years of evolution, the consequence of constant improvement of each organism along the way. But actually if you "ran the solution" again, the current state of species development would be very different and possibly with humans not even present. It is now generally accepted that the concept of progress, at least as defined as improvement, is not applicable to biological evolution. This point suggests that an alternative definition of progress would facilitate the comparison.
The popular connotation of progress is the betterment of the system, which results from a subjective judgment. Alternatively, progress could be defined as the establishment of a "universal" structure, on which later developments are built upon. The creation of these structures enables the system to function at a higher level, and arguably often results in a "better" state both for the subsystems and the system as a whole. (For the broadest perspective on this viewpoint, see MetaSystem Transition Theory.) In the distributed systems of interest, a universal structure applies to all or large numbers of the subsystems.
Examples in biological systems are the universal structure of DNA coding, differentiated nucleus and multicellular organism. We do not include in this idea of progress the variations between species, which all use the same universal structures. With a change of driving forces, species diversity can be altered and even return to prior states. But no amount of external changes will eliminate the universal structures. By this definition of progress, there has been little "progress" in biological evolution in recent times.
In contrast to this, social evolution is constantly establishing new underlying structures, and then using these to evolve "higher" systems. Knowledge, which is one essential aspect of social evolution, is a premiere example of this. The knowledge within the human race is constantly being developed and expanded. At the interface of knowledge and the unknown, there is great flux and uncertainty. But once the uncertainty is resolved, new knowledge is added to a commonly held base of prior knowledge (the universal structures). We do not include subsystem diversity in these underlying structures, which can be momentarily transient. Just as in biological systems, universal structures in knowledge, once established, are fixed for all later times.
Using this definition of progress for comparison between biological and social evolution, what we conclude is that there is a quantitative difference of progress in the two systems, social being much greater than biological. This difference is so great that the quantitative difference leads to a qualitative difference.
How does this relate to the current interests in the Symbiotic Intelligence project? What we expect to find is that it will be essential to capture of underlying structures of knowledge as they develop and to differentiate these from the transient structures associated with diversity of the subsystems.
A central premise of the Symbiotic Intelligence Project is that, in order to activate the self-organizing functionality, we need to use the Internet as much a possible for all aspects of human activity. What is currently being experience on the Net is an unprecedented growth of information, e.g., the number of hosts on the Net is increasing by a factor of 10 every one and a half years and will be 100 million by 2000 if the current rate of increase continues). Unfortunately the tools for dealing with this huge amount of information are woefully lacking. But what we continually forget is that the Internet is essentially an infant technology and still very much in the early stages of evolution. In the same way the telephone, automobile and television has undergone significant improvements, so will the Internet.
The problem with the Internet as it now exists is that it essentially is a flat database, meaning that all information is present at the top most layer; there is little hierarchical structure or organization. The second problem is that the Net appears the same for any individual and is insensitive to the purpose or needs of the user. The third problem is that there is no mechanism for filtering out false from true information, useful from useless information. All of these problems are being addresses by new technologies, and there is every expectation that significant progress will be made. In fact, the current project specifically addresses how these changes must occur for the Net to become a medium for future social evolution.The problem with the Net is that much of the information out there is "worthless", at least to me. Won't the dominance of this "low quality" information just make the system worse?
What makes some information more valuable than other information? How is the value of information established? Certainly there is some information that is sufficiently unique and generally recognized as significant that there is a general consensus that it valuable. But this is the exception rather than the rule. Information is valued by those who use it, and is not valued by those who do not. Just as in nature it is difficult to establish what are the important components of the ecosystem for its continuation, the same is true for information in our society. To the user of the information, whether it is family photos or financial records of a large business, all are valuable within the context of its use. But to others, the same information is of no value. From a greater perspective, all information is valuable. What is essential for the process of self-organization is that as much diversity of information as possible is present. In a self-organizing system, no one knows what contributions are necessary to solve a problem. But we do know that the greater the representation, the more likely the problem will be solved.There is a major push towards making visual entertainment (TV, movies, shows, etc.) available on the Net. Won't this degrade any future usefulness of the Net, if not by sheer overloading of the system making it unusable for others, then by the difficulty in capturing content of activity?
It is certainly true that the most difficult characterization (or content) to capture automatically in a transmission is visual and verbal information, as compared to textual information. The more that non-textual information is used, the more difficult it will be to associate patterns of similar activity and thereby enable the self-organizing system. But as long as people are choosing the transmissions and thereby processing the content and making other decisions based on the content, the system will still work. The symbiotic system will always require humans to do the complex processing and the network to do make use of the processing. For example, for any photo on the Net, one can tell much about it by who and how it is accessed, without ever knowing the content of the photo itself. The same will be true for other complex media.
The other answer to this question is also the answer to the last question. What is important is not the specific content of any transmission, but the diversity of the types of transmissions. From this viewpoint, entertainment is equally as important as scientific discourse.
While some people may never use a computer, either by choice or circumstance, they will interact with people that are directly part of the network. This form of representation is much more direct than our accepted "representative" democracy.
Parallels can be made with the ancient democracies in Greece. While the common belief is that these were a direct democracy (excluding the issue of lack of representation of women or people without property), in fact on any decision made in the Senate only a fraction of the populous was involved. While any person could participate, typically only people with a direct interest took the time to educate themselves on the issue and travel to the Senate to contribute. And even of those that were directly involved in an issue, certainly there was representation of views of neighbors and members of the household that did not attend. So even in this rather idealize democracy, we see that universal representation was not likely and a mechanism existed to supplement direct representation. The same will be true for the Net. Those that are interested in issues will become more involved. Those that don't have access or wish not to be part of a technological solution (Luddites) will still have contributions through normal societal channels.
While in the future more human activities will happen on the Net, there will be significant aspects of society that will not. These "off-line" interactions will still be part of the overall dynamics, just as in the past.Most of the people and activities on the Net are not representative of the best of humankind, and may be representative of the worst. Won't these people just degrade any self-organization to the point of triviality or uselessness?
As argued in a prior question on therelative value of information, in a self-organizing system it is almost impossible to know what is or value and what is not. Abstracted studies of collective decision making show that for the collective solution to function at its best even a diversity of performance is needed. If teams made up of just the highest achiever are brought together to solve a problem, they will not function as well as a diverse group of individuals.
Admittedly it is difficult to understand how some negative aspects of society can usefully contribute to a collective solution, but we have the example that prior social evolution contained all aspects of society and has survived. Similarly the Net will be a reflection of the society within which it exists.What about underdeveloped or developing countries, won't they be left behind?
There is a point, in the answer to the first question above, where indirect representation cannot occur for those far enough removed from the Net, such as in a developing country. While the benefits of the Net in our future will impact underdeveloped and developing counties, their ability to contribute to solutions that have direct consequences on their future will be limited. What will be the challenge is convincing the developed countries that it is in their interest to help the developing countries to participate in the Net, again by applying the belief that it is the greatest diversity of contribution enables that system.
The use of self-organizing systems is needed when centralized control is not feasible, either because of the absence of central control or because the complexity of the system is beyond comprehension and therefore beyond traditional problem solving methods. The use of self-organizing systems is not intended to replace existing systems that are working, but rather to supplement these systems into working better.How can an approach that is inherently chaotic be useful for solving essential problems?
A self-organizing system is always in disequilibrium, in order that the system can readily adjust to differences in changes of state. Or said another way, if the system exhibited a stable solution, it would not be able to move to a new solution space when necessary. The consequence of this disequilibrium is twofold. First the level of randomness might lead to excursions in the state of the system that may not be desirable. And second, even in a "good" collective solution, there are often some subsystems that may perform very poorly. These are two different questions and are addressed separately. For physical systems that exhibit emergent properties, we do know that we get stable results either for an average over many realizations or if the system is contains many redundant subsystems. In real life it may not be possible to "run" a system many times in order to observe the reasonable solution. Nor might it be tolerable for individual subsystems to do poorly in a large collective. (Certainly the later has happened many times in history where the "average" population has fared well, but some minority has not.)
The partial answer to the first concern is that the redundancy in the system will reduce the failure or poor performance of a single realization. Redundancy is achieved by having large numbers of contributors to the collective solution. Then many "sub" realizations are being tested at one time, and the results of the collective system are not random but a stable, well-defined solution. Similar lessons are learned from studying natural systems. In these systems excursions occur that may be damaging to some of the subsystems, but on the whole the overall system is robust and survives. The answer to the second concern of the poor performance of some of the subsystems is much more difficult, and ultimately depends on the cooperative nature between the subsystems within the whole.If no one is in charge, who is going to rally the troops when an important problem arises?
With self-organizing systems, a solution to a problem can occur before the subsystems may even recognize a problem exists. This act of faith that the system will work is contrary too much of our past approach to problem solving, and hence or difficulty in accepting this alternative approach. But when no other alternatives exist (see the question at the top of this page), there is no other option. One can argue that when we are part of a system where the complexity of the system is beyond the understanding of central decision maker, we often wish to believe that the powers in control are really are in control, when they actually are not.Suppose the system really works and we begin to depend on it? Aren't we exceptionally vulnerable to someone pulling the plug on the Net?
This is a valid concern. The beauty of distributed, self-organizing systems is that the redundancy and lack of centralized control make them remarkably robust. For example, during the 1989 Earthquake in San Francisco the Net stayed up, while the phones and other services failed. Just as the function of the brain survives when significant damage occurs, the Net is expected to survive either natural disaster or intentional acts of terrorism.
This is probably the single greatest challenge of implementing a self-organizing system on the Net: How to capture and compare the content (structure and semantics) of information that is exchanged, stored or processed. This is particularly evident given the observation that the same raw piece of information, such as a photo, can have completely different meanings to different people. This was essentially the roadblock of Artificial Intelligence efforts that tried to capture the complexity of human activity, even in a restricted system. What gives us some hope in addressing this challenge is that people, the ultimate processors of complex information, will continue to be part of the process. Hence, the first goal is to capture the activity of humans processing complex information. This has been demonstrated in simple experiments. There is also progress in both characterizing and capturing (automating) the semantics in artificial environments.How can hundreds to millions contribute to a collective solution when a committee barely works with five people?
The processes by which large numbers of subsystems contribute to an overall solution are still not understood, but some progress is being made. We do know that many self-organizing biological systems regularly demonstrate this level of functionality, from the brain, to social insects, to the global ecological system. And, indeed, we as human in our social evolution demonstrate the same. What remains to be proven is that by using the Net useful solutions can be obtained from large numbers of people in a timely fashion.All past biological systems created themselves. The problem with artificial emergent systems is that no one knows how to create one. Many have tried and failed. How do you expect to do what others have not been able to?
Where human creations of self-organizing systems have been successful, these are systems that have a natural component which introduces a necessary level of functionality that enables the system to begin, grow and continue. Similarly the proposed system is really not different than the existing system of social evolution in any prior times. The human remains the central element. And with humans come the natural ability to form a self-organizing system. In fact, the development of the Net for the human social system will likely occur independently from the contributions of any centralized group, such as the current project.So much of problem solving depends on having the right information at the right time. Without some central coordination, how can a self-organizing system get the resources where they are needed?
Often in sequential problem solving, there is a challenge of having the right information at the right time - a synchronicity problem. If the right information is not present when it is needed, even a linear solution method will fail. This is often our experience in traditional problem solving. Most problems remain unsolved, not because critical information doesn't exist; but because we don't have access to it when we need it. To some degree, the redundancy and increased access to information of self-organizing systems lessens this problem. In addition, the ability to create agents that represent a person wishes and needs will enable a greater diversity of contributions to be present even when humans themselves are not. This added presence was only marginally possible within our current social evolution, by the use of communication that represents ourselves. The Net offers a significant additional representation of ourselves that was not possible before.
Publications | Talks | Interests: Past & Present | Symbiotic Intelligence Project
Contact | Science of Diversity | Home | Links | Diversity Documents | Norman Johnson
Infectious Idea Spread using Epidemiological Modeling | Collective Intelligence in the presence of bias and misinformation
Community, Coexistence and Pre-conflict Minimization Projects | The CISS (Coexistence in Suggestive Simulations) Project