Application: HOTAIR
1. IntroductionWith the volume of information being published in digital form increasing rapidly, there is an increasing need for users to be capable of efficiently and accurately locating documents that satisfy their information need. The aim of the HOTAIR project is the development of a Multi Agent Information Retrieval (IR) System. An IR system is aimed at identifying documents that satisfy a user's information need, typically expressed in the form of a text query. Documents that are considered to be relevant to the given query are returned in the form of a list ranked from that document considered to be of most relevance. The use of a Multi Agent system in the development of such a system allows it to be:
2. Ongoing ResearchTo date, a functioning prototype of the HOTAIR architecture has been developed. This consists of two distinct subsystems, which represent the two key components of an IR system. These two subsystems are as follows:
These are discussed in more detail in the following sections. 2.1 Indexing SubsystemThe Indexing Subsystem is responsible for the identification and gathering of documents that are to be made searchable. In order for a docuement to be inserted into an index (where it is therefore available to the Querying Subsystem), it must undergo three levels of processing: Data Gathering, Translation and Indexing. Each of these three processing stages is encapsulated by a type of agent, of which many instances exist in the system. Data Gathering is the process of crawling web sites, FTP sites, shared directories and other data sources and downloading documents that are found there. As these documents may be stored in a variety of file formats, the next stage is to translate each of these into a common internal format, known as the HOTAIR Document Format (HDF), an XML representation of the contents of the original document, along with associated metadata (such as its original location and MIME type). The incorporation of the Translating stage contributes to the extensibility of the system, as support for additional file formats can be added by the introduction of a single new type of Translator. Once a document has undergone translation, the final step in the indexing process is for it to be stored in a searchable index. As HOTAIR uses multiple IR algorithms to perform its searches, documents may be added to a number of distinct indices. As each IR algorithm can use different data about each document in order to gauge a its relevance to the given query, each index contains data that is specific to the relevant algorithm. Again, separating this stage from the rest of the overall indexing process contributes to the extensibility of the system, as support for further IR algorithms necessitates only the addition of a new type of Indexer agent. Fig 1 illustrates the stages of processing a document must undergo in order to be included in an index. Fig 1. The stages a document must go through in order to be included in the index. 2.2 Querying SubsystemThe Querying Subsystem consists of two types of agent. Query Dispatcher agents accept queries from users and are responsible for returning the final set of results in response. In order to do this, the Query Dispatcher forwards the query to a number of Query Handler agents, each of which performs a search using a different IR algorithm. Once these results have been returned to the Query Dispatcher, they are combined using the probFuse data fusion algorithm (see the "Associated Research" below for further details) into a single result that is then returned to the user. 2.3 Management AgentsThe quantity of each type of agent outlined above is allowed to vary in accordance with system demand. Performance management agents are responsible for ensuring that scarce computing resources are being utilised in the most efficient manner. They can do this by taking such actions as creating new agents, terminating agents to free up resources, halting groups of agents that are less in demand and controling the agent platforms on which certain processing tasks are carried out. Additionally, other management agents are responsible for monitoring the overall health of the system by identifying failed agents and agent platforms and ensuring that the system is still capable of functioning when such a failure occurs. Fig 2 and Fig 3 are sample screenshots of the visualisation tool developed to monitor the functioning of the system. Fig 2. Visualisation tool demonstrating the three stages of processing to be undergone by each document. The blue lines illustrate which agents are in contact with one another in order to source jobs for processing Fig 3. Visualisation tool demonstrating system performace. Each line graph demonstrates the number of documents processed at each stage that have yet to be requested and processed by the subsequent stage, plotted against time. 3. Associated ResearchIn addition to work on Multi Agent Systems, the HOTAIR project has also been responsible for the devleopment of the probFuse algorithm, which is a "data fusion" algorithm aimed at combining the outputs of multiple IR algorithms in order to achieve superior results. The Indexing Subsystem maintains multiple indices which are usable by several IR algorithms. ProbFuse weights the results output by each algorithm based on the quality of that algorithm's past performance. 4. Future WorkFuture work on HOTAIR is intended to be largely in the domain of Autonomic Computing. This refers to the strain of research under which large-scale computer systems are created in such a way as to be capable of self-management. The aim of this initiative is to greatly reduce the amount of human interaction necessary to achieve optimal system performance. Key features of an Autonomic Computing system include self-awareness, self-configuration, self-optimisation, self-healing and self-protection. In particular, the area of self-optimisation is of key interest. Foremostly, instilling in the system the ability to exploit and balance the utilisation of scarce computing resources such as processing power, memory and storage as demand dictates. This will involve the management of the output levels of each stage of the production line-style Indexing Subsystem, in order to ensure that new documents and updated versions of existing documents are included in the index as efficiently and as speedily as possibly. This must be balanced against the users' need for accurate search results to be returned to them almost instantaneously. Overemphasis on either subsystem would lead to the system failing to achieve its goals, either by driving away users with slow response times, or by failing to maintain a sufficiently large document collection for relevant information to be available for users's queries. 5. Selected Publications
|
