Self-organization in stigmergic systems

by Peter Small


Most of what we do with information technology have their counterparts in biological systems. The difference is that we solve our problems through organization, planning and deliberate design. Nature takes a different approach and solves the same problems by getting systems to self-organize.

Nature's strategy is far more efficient and less costly than many of the methods we use. By understanding how Nature gets biological systems to self-organize, we can copy these strategies to get our own systems to self-organize.

The semantic problem

Although the term 'self-organizating' is used to describe organic structures that organize themselves, this expression gives a very inaccurate connotation. Organizing and organization are usually associated with planning ahead, but organic self-organization is a structuring that emerges without any pre-planning. It just happens. Probably a better expression to use is emergent-organization, this is more representative of what actually happens.

Another problem with the term 'self-organization' is that it seems to imply that it is an organization that occurs without human involvement. It is assumed that if people are involved in any decision making then a system cannot be self-organizing. This is not true.

Stigmergy - the strategy that Nature uses to get insects to self-organize - is totally dependent upon the insects being involved and having freedom of choice. When we use stigmergy for systems in information technology we are totally dependent on people being involved and having a free choice to make changes.

In fact, stigmergic sytems are specifically designed to maximize the opportunities for human to make decisions.

Note: self-organization is a subject that has many interpretations. To fully appreciate what self-organization is, it is worth reading Ethan Decker's excellent tutorial Self-Organizing Systems: A Tutorial in Complexity.

Stigmergy and self-organization in design

To understand stigmergy and stigmergic strategies you have to have a clear picture in your mind as to what stigmergy is. It is about people reacting to a situation and their reactions causing them to change the situation. The changes they make then cause them to react differently. The phenomenon of stigmergy is a result of this process continuing indefinitely - with the changes and reactions altering at every step of a continuous ongoing sequence.

In some ways, a stigmergic strategy is similar to strategies used to develop products for the consumer market. Consumer products are never put straight onto the market as soon as they are designed. They first go through a series of tests and trials.

Initially, a prototype of the product is made, with the design based upon intelligent guesses as to what the intended customer wants and how they might be expected to use it. This prototype is then given to a few people to try out and make suggestions as to how it might be improved.

The results from this trial are fed back to the design department, where appropriate changes are made. This process is continuously repeated until the product seems to fully satisfy the customers' requirements.

This is stigmergy at work. People react to what they see and make suggestions as to how it can be improved. When those changes are made their reaction is different because they now focus their attention on something else about the product they would like to see improved. With this process repeating, both the product and the reaction of the people change as a series of steps. And the product gradually 'self-organizes' to become optimally desirable.

This illustrates the counter intuitive aspect of stigmergy. It isn't about the self-organization of the individuals. It is about a strategy that causes a product to become optimally efficient without planning or rational design. This particular application of a stigmergic strategy effective arranges for the customers to design a product for themselves.

Stigmergy works its magic because users are responding to an ever improving product. Their feedback and suggestions become increasingly more detailed as the product gets nearer and nearer to perfection.

Of course, Nature's strategy is far more efficient - because she doesn't need a design department.

Using stigmergy to design a Web site

In a similar way, a stigmergic strategy can be employed to get users to design a Web site or a knowledge management system for themselves. You start off with a prototype, let visitors use it and, from the feedback they provide, the design is changed until it is working properly.

However, for this to work as efficiently as Nature's strategy, we have to be able to cut out the design department. For this, we'll need to create a program that will do all the changing for us - without having to get specialist help.

This is the way a stigmergic strategy is used in information technology. A design department isn't involved. The only human function needed is simply to observe what is happening and provide a guiding intelligence to keep the system on track.

So, what sort of program needs to be designed to allow us to use a stigmergic strategy without planning or a design department? For answers, we need to see how Nature does it.

Note: Nature has no use for planning, insights or design. This is because biological systems are in a state of continuous change. There is constant competition. Eco-systems are constantly in a state of flux. Weather and environmental events can be disruptive. Planning ahead is pointless if the future is totally unpredictable. But, aren't these similar to the problems encountered in information technology? There is competition and constant technological progress. Customer requirements can change. The way customers use the product can change.

To understand the problem this represents, think again about the stigmergic strategy used by companies when they send their products out to customers for suggestions for improvements. Imagine what the progress would be if in between cycles the customers changed their minds about what they wanted or a competitor came out with a completely new solution that made the product obsolete.

Nature has to solve for these kind of problems.

Order out of chaos

Nature solves for the problem of continuous change by allowing systems the freedom to change and adapt. This involves self-organization because the system has to be able to re-organize itself to adapt to changing situations. However, stigmergic self-organization cannot be used because this relies upon positive feedback, which can quickly lead to instability with the system rapidly going out of control.

The best way to visualize this kind of instability is to think of a social discussion group. This has many similarities with stigmergic systems if you think of people reacting to and adding to a conversation, rather than talking directly to each other.

One person makes a comment. Another person reacts to that comment and makes a comment of their own. This might prompt yet another person to comment as a reaction to the comments that have gone before. In this way, the conversation twists and turns in unpredictable ways as people continuously change their responses according to the continuously changing nature of the conversation.

This is what stigmergy is about, individuals reacting to an environment by changing it and the changes they make causing them to react differently. This sets up a positive feedback effect, whereby the reactions and the environment are continually changing due to their effects on each other.

As you will be aware, conversations can take unpredictable forms. They can be orderly and directional, leading to people gaining knowledge, or, they can be chaotic and disjointed with not much value coming out of them. They can peter out quickly or they can go on for a long time. The same thing can happen to self-organizing systems, unless there is a strategy in place to keep them under control.

Note: In academic descriptions of stigmergy, this situation would be described as a complex system acting chaotically.

You can find many examples of stigmergic systems on the Web that fall foul of this problem. Discussion forums and many sites that rely upon visitor interaction are prone to this tendency to become unstable. They often degrade or die out. Clearly, a stigmergic system has to be able to avoid this hazard, so once again we must look to Mother Nature for a solution.

Nature's ingenious solution

Most Web site designs prevent this kind of instability from occuring by limiting and carefully controlling visitor interaction with a site. Visitors may be allowed to enter information, but this activity is not allowed to change or disrupt the organization of the site.

This is where Nature is smarter. She doesn't prevent disruptive activity from taking place; in fact, she actively encourages it. However, she uses an ingenious trick to make this work to her advantage. Instead of her systems degrading and becoming disorderly, they become more organized and increasingly efficient.

Nature's trick - or way of creating order out of disorder - is to allow a system the maximum freedom to be widely erratic, but to control it by using a filtering mechanism. This filtering mechanism takes the form of rules that allow any erratic changes that improve the system to take effect, but to stifle any erratic changes that degrade the system.

This acts like a one way valve in a water system, were water movements are allowed to go forward but are prevented from going backwards. Movement goes in one direction only: forward. In biological systems it is order and organization that is allowed to go in one direction only.

Note: Biologists call this trick 'evolution'.

Using genes to control self-organization

Nature's way of getting insect colonies to self-organize is to provide every individual insect with tools and techniques in the form of genes. These genes determine the behavior of the insects, giving nature a simple way to control the direction of self-organization.

Genes that promote good organizing behavior are retained, those promoting disruptive behavior are removed. This is the one-way mechanism that ensures insect behavior becomes progressively more organized and efficient.

A similar strategy can be used with Web sites and knowledge management systems by giving visitors or users of the system the equivalent of genes. These can be varied and selected to ensure that the self-organization of the Web site or knowledge management system always proceeds in a positive direction.

Of course, you cannot transfer biological genes to the users of a system, but you can effectively transfer genes if it is arranged for the users to interact with the system via software agents. The programming of these agents can determine the kind of changes a user can make to a Web site. This programming takes the form of functions and procedures that are the software equivalent of genes.

It is by manipulating the "genes" in these agents that user interaction can be controlled in a way that always favors activity that improves the efficiency and effectiveness of a site. 'Genes' that lead to better organization and efficiency are retained, those that cause degradation are eliminated. This is identical to the one-way mechanism used by nature to direct self-orgaization and enable systems to adapt to changing condtions..

Structure of a stigmergic system

A stigmergic system consists of three main elements: a Web site, a program that creates a Web site and the agents. Users interact with this system via agents. They send information through their own agents and receive information from the agents of other users.

These agents also interact with the Web site building program, so that it can constantly monitor user activity. In this way the Web building program can keep making changes and modifications to the Web site and the "genes' of the agents. It is this program that puts into effect the one-way mechanism that ensures the Web site becomes progressively more organized and efficient.

Note: You can learn more about the Web site building program and the programming of the agents from the article "Organic prototyping" and the tutorial "Harnessing the power of stigmergy".

Making use of existing human genes

Nature's use of stigmergy is not confined to the insect world. Humans have many genes that give rise to both instinctive and learned emotions that influence our behaviour. Many of these genes motivate us to instinctively create order within our social frameworks.

A simple example is our automatic tendency to form orderly queues rather than try to push past each other. Another example of stigmergy in action can be seen in most online discussion forums. Unspoken rules keep discussions on subject and act quickly to stifle flame wars. These unspoken rules are the subtle influence of stigmergic emotions, which allow freedom of expression, yet exercise an implicit controlling influence.

It is stigmergic genes that prompt us to react to situations by trying to improve them. Although we see these genes as motivating us to compete with each other, the overall effect is to manifest increased order and organization. The Internet and the Web are good examples of this effect. People see what other are doing and are prompted to make improvements and create organizing links. This behavior may be motivated for reasons of self-interest, but it is adding to the overall order and increasing the efficiency of the system as a whole.

Such instinctive emotions can be encouraged in stigmergic systems. When people create agents or provide information there is a strong tendency for them to make improvements on what others are doing. A one-way mechanism built into a stigmergic system can make use of this natural tendency for people to add improvements to a system.

Note: You might test your own stigmergic genes by creating an agent for yourself in the 'example' (see menubar). Will you create an agent that is inferior to all the other agents you see there?