Download Elephant User Manual - Common

82
Elephant User Manual
There are several interesting uses and extensions of Elephant in Conceptminer:
• Bulk storage of post-processed web data: Elephant was used to store hundreds of
thousands of processed web pages as strings, associate pages with queries and store
related metadata.
• Derived index: a custom string hash function over URLs was used to populate a class
derived index, allowing fast identification of pages from their URL without requiring
expensive eql comparisons.
• Inverted document index: a (not terribly efficient) data structure that efficiently maps
words to documents allowing pages to indexed by the words contained in them. Allowed
for phrase and conjunction searches.
• User association data structure: a data structure based on oids that supports general
one-to-many mappings between classes. Had a custom migrate method to support
migration of associations. Supplanted by persistent sets as of 0.9.
The most interesting use of Elephant was extending its transactional architecture to cover
in-memory lisp operations. PCOMP (Process Components) is a framework for constructing
and managing simple, dataflow-style multi-threaded applications in Common Lisp. The
goal is to simplify the process sufficiently so that the ordinary user can hide from many of
the details associated with aborting transactions. To this end, the model provides for safe,
asynchronous communications among a set of components which may be scheduled together
in a single process or communicate across separate threads (and potentially processes).
Components are packaged into a system inside a Container object which schedules execution
and mediates communications.
Communications between components can be in a dataflow style or using messages. Each
component has a single port for receiving incoming data items. These items, if access is
shared among components, should have the proper synchronization protections on mutating
accesses. There is also an asynchronous communications method allowing you to send
messages to components with particular names.
The basic building block is a component. Components are defined using the defcomponent form and contain several major elements, such as:
(defcomponent counter
(:vars (count 0) end (increment 1))
(:initialize (assert end))
(:body
(when (>= (incf count increment) end)
(terminate))))
The arguments to defcomponent behave as follows:
• :vars - Values that the body wants to retain between invocations
• :initialize - A reserved message handler called at the begining of time
• :body - A body expression that is executed whenever data has arrived
The body and messages are evaluated in a very specific environment. Within the body
certain variables and functions are bound:
• Variables:
• data - The current data item