Intelligent use of Web logs for self-organizing Web sites
Content driven sites
Whatever the purpose of a site, it has to attract visitors. Good content attracts visitors - by word of mouth or by high placings in search engine results.
But, just having good content isn't enough. People can just read it and move on. Information content needs to do more than this. This calls for a self-organizing site and an evolutionary design strategy.
The difference is in the design strategy
Functionally, self-organizing web sites are the same as any other kind of Web site. Their purpose is to attract a particular type of visitor for a particular reason.
Content has to be appropriate, beneficial and optimized for search engine placements. Navigation has to make it easy for visitors to get round the site. Web logs are used to monitor visitor activity and are the main guide in the development process.
So, what is the difference?
The difference
The difference between conventional and self-organizing Web sites is the design and development strategy that is used.
A conventional Web site design approach is planned. Efforts are made to control how visitors interact with a site.
A self-organizing site is not planned. No attempt is made to control the way visitors interact with the site. The site is allowed to find its own best way of working by using an evolutionary strategy.
Why use a different strategy?
It may seem bizarre not to plan the design of a Web site, but the reason for this is that visitor traffic to and around a site is highly dynamic and can be affected by many different kinds of complex inter-dependencies.
Dynamic complexity
Changing the title or text in a document may influence the way a visitor responds to it. This may affect the way the visitor moves further around the site. It may add to or detract from their interest; perhaps influencing whether or not they bookmark the page, recommend it to others or link to it from their Web site.
The changes might also alter the way the document is categorized in the search listings and even affect the number of listings the document appears in.
Search engine listings, bookmarkings, page recommendations and linking effectively open up a multitude of doorways into a site, perhaps each bringing a different type of visitor. This can create a highly complex visitor flow pattern that is difficult to analyse, predict or control.
Changes to the navigation or structure of a site will affect the way visitors explore the site and the content they are likely to see or miss. Cross linking in pages and links to other sites can set up a multitude or different routes around a site that is impossible to analyse or predict. Search engine listings and placement are not constant and changes here can make vast differences to the visitor traffic flow.
The point is that a Web site is so complex that it isn't possible to work out what to do for the best. This is why a stigmergic strategy - a continuous sequence of changes and responses - can produce far better results than trying to rationalize the complete overall design.
Stigmergy
The essence of stigmergy is responding to an environment by making changes to it. Each change initiates a different response. The developer of a stigmergic Web site works this same way. The way a Web site is working is observed. Changes are made. It is rebuilt and observed again. This is continuously repeated with no overall plan to the changes. They are made only as direct responses to the feedback coming from the Web logs after each new rebuild.
The subtle difference here, compared to the way conventional sites are developed, is the attitude. You don't make changes to try to force a Web site performance to improve. You assume you don't know what effect changes might have and you make changes to see what happens. If the changes make an improvement you keep them. If not, you undo the changes at the next rebuild. In effect, development is treated as a series of intelligent trial and error exercise.
Evolutionary strategy
Changes responding to changes can lead to less rather than more order and efficiency, unless a mechanism is put in place to guide the development process. Such a mechanism is an evolutionary strategy, which can give directional control in the same way living organisms are developed - through a process of evolution.
Living organisms are not planned. They evolve over a series of generations. Their form and organization emerge to become increasingly effective and efficient over a series of trials.
Biological organisms try out genes, gene combinations and structural organization at every trial (generation). Those that work are retained. Those that don't work are removed. We can copy this strategy by trying out content, content combinations and navigational structures through a series of successive new builds of a Web site (Note: not by incremental changes).
This current site is an example. It is continuously being rebuilt in its entirety. Each time in a different way.
Feedback
Nature uses a "survival of the fittest" test, to get the feedback necessary to decide which genes and gene combinations go through to new generations. We have to use a different kind of feedback - the feedback we get from Web logs. These can give us the information necessary to decide what design features are left out and which go through to the next generation (rebuild of the site).
Testing content and organization
This is where the paradigm of a living organism comes into play. Instead of thinking about visitors coming to a site, we can think of a site reaching out into the Internet to bring selected people to it. This can be done by bringing attention to certain content pages in something like a newsletter, a news group or an on-line discussion forum.
Each page referred to, will act like a newly opened door, bringing visitors to a particular point on the site for a particular reason. A Web log analysis will be able to track the activities of these people as they move from the content page that has attracted them to look at other content that takes their interest.
In this way we can think of developing a Web site by gradually adding more and more doors to the outside world, each attracting a specific type of visitor.
A site acquires a life of its own
Visitor counts at a content page will spike at the time attention is drawn to it. The size of this spike is relatively unimportant. What is important is what happens to the visitor counts at these content pages later on.
The later counts will be determined by several important factors: book-marking, recommendations, being linked to from another site, search engine placements and visits from visitors exploring the site. This will apply to each content page individually, not to the site in general.
The subsequent activity, after initial spikes have died down, is the activity that tells you how the site is performing when it is attracting visitors in its own right. The flow patterns of traffic around the site from different entrances will merge into one, as all the different content pages combine together. This is the point where a site begins to acquire a life of its own. It is the results of this overall performance that determine how the site should be changed for the next generation (rebuild).
Metamorphosis
Although the patterns of traffic flow around the site will be due to visitor interests and preferences, you can influence the flow at new rebuilds of the site by changing content, making changes to the navigation system or introducing links between pages. Unproductive content can be removed and more popular content strategically relocated.
In this way, a site can be made to morph into quite a different flow pattern of visitor activity. The effects of this metamorphosis can be quickly determined by looking at the feedback provided by the Web logs after each new rebuild.
Here is where the advantages of viewing a Web site as an organic structure becomes truly valuable. Improving the performance of a site - by adding new content to the hierarchy, or tweaking the navigational system - isn't seen as incremental changes, but as changes that can have a more widespread effect, changing the overall dynamic performance of the site.
Dynamic complexity
If you've looked around the references section of this site, you'll probably have noticed articles and links to complexity theory. The reason these are included is because they provide insights into the kind of things that can happen when you have a dynamic Web site. Changes to content or its position in a hierarchy can have dramatic and unpredictable effects on traffic flow and the performance of the site as a whole.
Updating, adding or removing content is a regular part of the day to day maintenance of any Web site. But, switching content around in the hierarchy or making radical changes to the navigational system strikes terror into the minds of many Web site managers.
This needn't be so if a lead is taken from biological systems. They have no problem with this. Look how easy it is for a caterpillar to morph into a butterfly.
The trick is to think of the navigational structure as an element of the site that is just as easy to change as content; a separate component of the system that is easily replaced and just as easily put back again.
Fundamental stability
The immediate thought that might come to mind is that changes to the navigation system or a restructuring of the hierarchy linking the various content together will be massively disruptive. What about people who are recommending content to others? What about the references to content in the search engines? What about links to the various pages on the site from other Web sites? Will all this be lost?
A little thought will tell you that changes to the internal organization of a site need have no effect on the addresses of the content pages. The URLs do not have to be changed, so all the links between the site and the external world remain intact however much you play around with the navigational system or the organization of the content.
This is what provides a fundamental stability to a site, even though it may go through many different stages of evolution, involving any number of new generations or rebuilds. The function of the Web site building program, is to allow such changes to be made without affecting the URLs of the content pages.
Note: the Web site creator program - used to build and rebuild Web sites for new generations - is described in the link 'Organic Web site prototyping'.
Determining the point of success
With this approach, you don't have to spend time on extensive planning and consultations to develop a Web site. Instead you can gradually steer it to greater and greater effectiveness and efficiency by:
1) Continuously introducing and testing new content,
2) Changing or removing poorly performing content
3) Improving traffic flow around the site with navigational changes,
The effects of these changes will be picked up immediately by the Web logs. If total visitor counts are on the increase week on week, you know that you are on the right track. This will be dependent upon ranking in search engine results, recommendations and links from other sites.
However, real success is not just about attracting visitors to a site. Success can only measured by determining how effectively and efficiently a site is doing the job it was designed to do.
Action points
Once you get a Web site to where it is generating its own visitors, and the patterns of activity around the site have begun to stabilize, you can start to think about getting the Web site to fulfil its purpose, or purposes.
Purposes are fulfilled at special places on a Web site, known as action points. These have to be considered quite separately from other features of the site because they have to be placed strategically.
Placing an action point on a Web site doesn't necessarily mean placing it in the most prominent position. When people arrive at an action point they must be in the right frame of mind for them to take the required action. This may necessitate routing a navigation pathway that takes them though preparatory content first. This is where an evolutionary strategy is invaluable because you can keep making subtle changes to content and navigation to get the best results.
Maintaining efficiency
Many Web sites grow and expand much like a cancerous growth. As more and more content is added, it gets hidden away in obscure pockets of a gigantic maze. Visitors can't find it easily and updating becomes difficult and expensive. In this way, Web sites can degrade, becoming less efficient and less effective.
An evolutionary strategy can prevent this happening by allowing the hierarchy and the content to be continuously pruned. The Web logs will tell you which areas are under-performing, allowing the site to be kept compact and efficient - keeping the content to a minimum and the navigation optimal for achieving the purposes of the site.
Information is constantly evolving. Needs for information are continuously changing. An evolutionary strategy will allow a site to rapidly adapt with the minimum of effort.
Note: maintaining visitor growth
There is no short cut to getting visitor growth. The site has to provide good content that can be retrieved efficiently. Once you have this, word will spread through the Net by the Small World Clustering effect (see link). Good content will be linked to from other sites. An increasing number of visitors and an increasing number of links to the site will be picked up by the search engines, which will result in higher placings in their search results. This is the way self-organization manifests on the Web, popularity itself brings more popularity because this is how sites are judged.
Maintaining the information content of a site though, particularly in subject areas that are constantly evolving, can be very time consuming and costly. This is where it is possible to use stigmergy in a different way - the way ants use it to self-organize and collaborate to share sources of information.
To put this strategy into effect, we have had to use agent technology and devise a Web based structure - the Kempelen Box - that can be used to emulate the way in which this strategy is employed in nature. Look at the link 'Example (Kempelen Box)' and then read through the tutorial 'Harnessing the power of stigmergy' for explanations.