The slides of the presentation held by Erik J. Groeneveld are now available for download.
Meresco Presentation June 25th 2010 (PDF)
These slides were presented on the Open Space MERESCO meeting. Topics presented were:
Download the agenda for the oncoming Meresco meeting in English or Dutch.
Agenda English: ODT or PDF format.
Agenda Nederlands: ODT of PDF formaat.
The Registration Form (dutch) is online. Please register here.
Using these links you can check MERESCO’s news and updates.
Follow the MERESCO Blog (use the RSS feed)
Join Google group MERESCO (join group and get e-mails)
Join the LinkedIn Group MERESCO (configure e-mail notifications)
Follow Erik J. Groeneveld on Twitter or follow the hashtag #meresco
We are going to use these resources to supply you with important information and announcements for meetings.
Nederlandse versie staat hieronder.
On June 25th SURFfoundation and Seek You Too will organize a MERESCO meeting in Utrecht. It will be a meeting accessible for everyone who is interested.
Last year there was a conference wherein we explained the uses of MERESCO, and there was time to discuss certain matters concerning MERESCO. There also were some requests for subjects for the next meeting, and those subjects will be treated during the oncoming meeting.
This meeting will be an Open Space meeting so it will be possible to about new subjects with the other visitors, or with the experts who will also attend the meeting.
We will finish the meeting with some refreshments.
Nederlandse versie
SURFfoundation en Seek You Too zullen op 25 junieen MERESCO bijeenkomst organiseren bij SURFfoundation in Utrecht. Het zal een open bijeenkomst zijn; alle geinteresseerden zijn welkom.
MERESCO staat voor ‘MEtadata based REpository Search Components in Open source’. Het is een open source platform ontwikkeld in verschillende projecten van SURFfoundation, SURFnet en Kennisnet door Seek You Too (CQ2).
Tijdens de conferentie van vorig jaar zijn de toepassingen van MERESCO toegelicht en is er tijd geweest om onderling kennis uit te wisselen. Er zijn toen tevens verzoeken gedaan voor onderwerpen voor de volgende bijeenkomst. Deze onderwerpen zullen tijdens deze bijeenkomst besproken worden.
De bijeenkomst van dit jaar zal een Open Space bijeenkomst zijn waarin u zelf ook weer nieuwe onderwerpen kunt aansnijden. Er zullen experts aanwezig zijn die actief aan de gesprekken zullen deelnemen en eventuele vragen kunnen beantwoorden.
We zullen afsluiten met een hapje en een drankje.
Recently Kennisnet asked me how to scale up Edurep with regard to:
- queries per second
- record updates per second
- total number of records
I suspect that this is of broader interest, so below are two approaches for scaling CPUs, memory or bandwidth.
Queries per second
A single machine Meresco system runs between 10 and 100 queries per second. Scaling this requires adding more machines so load can be distributed over CPUs and networks. There are two approaches.
Approach A
Replicate the entire server process and feed updates to them simultaneously.
Approach B
Extract the most demanding components from the server’s configuration and put these on separate machines. Reconnect them using the Inbox component.
 |
 |
|
| Before | After |
Both approaches are based on standard Meresco functionality and therefore easily configured.
Record updates per second
Meresco is able to process 1 to 10 updates per second concurrently with querying. Scaling this up requires adding machines that can share the load of processing the records using approach B. These machines can feed into one or more query processing machines, effectively enabling scaling along both axes.
The main idea is to decompose a system into subsystems which can be distributed and replicated. This analysis must be done before a system can scale up using cloud-like environments. How Meresco’s configuration supports this will be outlined in a future blog.
Total number of records
Meresco can host 10 – 100 million records on one machine, mostly limited by what its indexes can do. Scaling up requires a closer look at these indexes to see how additional resources must be allocated. In this area Lucene, BerkeleyDB and OWLIM have earned great reputations. Meresco’s architecture helps to get the most out of these.
Meresco’s homegrown Facet Index and Sorted Dictionary Index (used for auto-complete) can be scaled following approach B. However, with a single-node limit of roughly one billion records most applications would not need more than one node.
Conclusion
I realize that I only scratched the surface of how to scale Meresco. There are many details to discuss and you probably wonder how your situation could be dealt with. I’d love to hear your responses!
MERESCO combines components written in various programming languages. It uses Python to tie these components together. It integrates Java using JTool.
It began with Lucene
Lucene is a well-known Java library for full-text search. MERESCO used PyLucene which compiled Lucene to native machine code. PyLucene was unstable and did not cover all of Lucene. In 2008 it changed strategy and the performance dropped significantly. We decided to try a completely different approach and that turned out to work very well.
What did we try?
We quickly discovered that compiling Lucene with GCJ was easy and that it resulted in robust, fast and reliable programs. Then we created a Python extension called JTool which mirrors the complete Java API in Python.
How to use?
Here is how you use it in Python:
$ python
>>> import jtool
>>> jtool.load('liblucene-core.so') # compiled lucene-core.jar
>>> from org.apache.lucene.index import IndexReader
>>> reader = IndexReader.open("/indexdir")
This is how all of Lucene is accessed in MERESCO. It runs fast, reliable and with low memory footprint. The code base of JTool is only 1500 lines, there is no code generation and it is completely generic. So the next question is:
Will JTool work for other Java libraries?
In February 2010 we started looking for a more scalable Triple Store for MERESCO. Our choice was OWLIM…. written in Java. While Lucene is quite a large library, OWLIM is even larger. The latter depends on 22 other Java projects, including the Sesame RDF Framework.
Compilation of OWLIM took a bit more effort as we needed to gather all needed jar files and make sure some factories did not get duplicated in the final library. Then we tried to load this library in Python using JTool:
>>> import jtool
>>> jtool.load("libowlim-core.so")
>>> from org.openrdf.repository.sail import SailRepository
>>> from org.openrdf.query import QueryLanguage
>>> ...
This enabled us to insert RDF and execute SPARQL queries on the triple store. Yes it works!
Future of JTool
JTool can not yet call methods with NULL-parameter or Java 5 varargs. It also does not support callbacks in Python yet. We have solutions for these omissions which we will implement this year. Meanwhile, it is easy enough to create a Java wrapper and use this via JTool. So JTool allows us to quickly integrate any Java libraries in MERESCO.
Avaliability
Sources for JTool up to version 4 are available JTool Sources.
JTool version 5 and up are available in binary form JTool Binaries.
Solr focusses to get the most out of one index type: Lucene. Meresco supports a number of different index types, each specialized for a specific task. Queries are split, each part processed by the most appropriate index, and the results are integrated. This ensures that all types of queries are processed within tens or at most hundreds of milliseconds.
Each type of index has distinct and unique properties such as specific query algorithms, optimized access patterns and scalability. We will introduce each index type below together with a short characterization.
Fulltext Index
This index is optimized for queries for a combination of words, literal phrases, words nearby other words etc. It is implemented with Lucene which is known to scale very well. Meresco helps scaling it by keeping it small, this post about Storage versus Index.
Facet Index
The facet index specializes in drilldown (faceting) queries, dynamic clustering and tag clouds. It produces exact results even on large data sets, which is one of Merescos unique selling points. Meresco uses custom data structures and algorithms which scale to billions of postings on a single node.
Dictionary Index
This index supports fast lookup of arbitrary textual information related to keys. It supports simple lookup ‘queries’ only. It is implemented using Berkeley DB, which is known for its good scalability and performance. It is being used to scale up set and metadataPrefix queries in OAI-PMH to tens of millions of records. This post describes the process: Dependable OAI Repositories.
Sorted Dictionary Index
This index supports extremely fast lookup of simple numeric information attached to alphabetically ordered terms. It supports prefix queries such as needed for auto-complete. It is implemented using a Burst Trie.
Triple Store
This index supports queries about arbitrary relationships between objects (graph-inference) typically through SPARQL or extensions to CQL. It is implemented with rdflib and OWLIM, the former being simple, the latter being one of the most scalable and fast triple stores around. An application is relating traditional records to social metadata such as tagging, ratings and reviews. A lot is going to happen around here.
Range Index
This index supports ultra fast retrieval of data contained in numerical ranges. It supports range queries such as 20090101 < date <= 20101231. Meresco has its own optimized implementation. This index is so small, it scales to billions of documents even on a single node.
N-gram
The n-gram index is capable of performing approximate matches and hence used for suggestions in ‘Did you mean?’-like solutions. More generally it allows for language neutral queries. This index lays on top of the Lucene index, but is nominated to be replaced by a faster and more specialized one in 2010
Meresco can maintain these index types in sync both during batch and real-time updates. Together, these indexes deliver fast results to queries, even if those queries are complicated and demanding such as tag clouds, auto-complete, clustering, term suggestions, did-you-mean and relationship queries.
The high concurrent performance of Meresco is not achieved by deploying an army of processes and threads but by the asynchronous power of Weightless.
Server processes are either synchronous or asynchronous.
Synchronous servers accept a connection and wait for the whole request to be received before processing. Every connection is handled by a single thread or process. The program flow in synchronous servers seems conceptionally easier, but often gets complicated in practice by all kinds of locking issues.
Asynchronous servers read when there is data available and send responses when there is something to send, all in one single-threaded process. Code that runs within a asynchronous server needs to be crafted with special care to allow for fair resource sharing between requests. This is known as cooperative scheduling. Because there are no threads that interrupt and lock each other (simplifying the software when compared to the alternative) the server is able to handle a large number of concurrent connections without a noticable speed penalty.
Weightless was developed to bring the advantages of asynchronous I/O to Meresco. Weightless is a lightweight framework that provides the infrastructure for asynchronous servers. Weightless comes with HTTP and HTTPS server functionality. By making use of Python generators (co-routines) to facilitate input and output, Weightless provides an easy to use mechanism to read and write data.
More information on weightless can be found at: http://weightless.io
In a lot of search engines the data and indices are stored together, creating a single huge entity. This approach potentionally leads to a number of problems, ranging from backup problems to performance issues. Also, with these systems access to the data is limited by what is offered through their respective APIs.
Index
Meresco works differently, using an index for what it is designed to do best: to return the identifiers of documents that match a query. The index of a book gives you the number of the page that covers a certain topic.
Similarly, the identifiers returned by a Meresco index point to documents in a separate storage. This leads to a simple index that even for millions of documents typically stays small enough to fit in memory entirely. This yields an obvious speed advantage.
Storage
The data is stored in a Meresco Storage. The storage is basically a well defined directory structure. Identifiers are used to pinpoint a directory in which the data is stored. This means that it can be stored on basically any filesystem (although e.g. the ext2/3 filesystems impose a limit on the number of subdirectories in a directory).
Native
Having all data in native format on disk makes it easier to control and maintain. Data can be read immediately without having to be decoded or transformed in any other way. Data enrichment tools, for example to get metadata from PDF files or digital images, can do their work in the background directly on the data files.
Caching
Many systems come with their own caching mechanisms. Meresco Storage however takes advantage of the disk caching capabilities of modern unix systems. This results in fast data lookups with no added complexity.
Conclusion
By keeping only identifiers, a Meresco index stays simple, small and fast. The accompanying storage offers fast retrieval of stored documents in their native formats.
The web is moving towards linked data. Many data collections are available as Linked Data, including Dutch scientific libraries, museums and archives. What can we do with all this data? What tools do we need? The good news is that Linked Data can be adopted incrementally.
What problem does Linked Data address?
Objects in libraries, museums and archives are increasingly described by experts not related to these institutions. The resulting descriptions relate to persons, places and other concepts, of which none of the experts or institutions can claim authority. Institutions are no longer the only authority on specific data collections and certainly not authoritative on all the concepts their collection relates to. Maintaining collections of authoritative information is becoming increasingly difficult. Life cycle management of metadata records [possibly even maintaining different versions for different communities] become major challenges. Failing to maintain a clear authoritative, and not isolated, collection undermines the existence of museums, archives and libraries and any other (broker) service that is to add value somehow.
How does Linked Data solve the problem?
Linked data allows everyone to make statements about everything [RDF Concepts]. It does not encapsulate knowledge about objects in a record, but represents the knowledge as a set of statements about the object. The record becomes a set of statements. This is simple, but fundamental. Consider the following record about a certain piece of art:
Record 5832:
identifier = http://institute.org/987639
title = "A true work of Art"
creator = "V. van Gogh"
This could be represented with (at least) two statements:
http://institute.org/5832 has a title whose value is “A true work of Art”
http://institute.org/5832 has a creator whose value is “V. van Gogh”
The fundamental change here is that record 5832 no longer plays a role when exchanging data. The record has been artificially created to describe an object, but the record itself is not important, only the statements it introduces are. (Linked Data can transparently introduce intermediate objects to group statements, however these are not manageable items ‘to worry about’ as records were). Only these sets of statements are exchanged. Maintaining an authoritative collection comes down to carefully selecting sets of statements to join.
Resolving Statements
Subject and Predicate are always URIs, while Object can be an URI or a value. In the example above the Creator statement could have been:
http://institute.org/5832 has a creator whose URI is info:eu-repo/dai/nl/071792279
For the actual name of this person, we will have to look for statements saying something about info:eu-repo/dai/nl/071792279 as the subject. This again might resolve to a URI so we have to repeat the process until we find a value.
How can institutions take advantage of Linked Data?
If the world around an institution is a cloud of Linked Data sources, the center of the cloud is where the institution has most of its authority. Surrounding this authority center are the related data sources on which the institution has less authority. Together we call this the authority cloud.
With this as a reference, do the following little steps:
What tools are needed to deal with Linked Data?
Keep your tools! Unless you are dissatisfied of course, retain your investment. You will need a scalable triple store in your own data center however. Since this Triple Store contains all the statements you decided need resolving before offering your service, it must be fast and readily available.
In the next installment of this blog, we will outline how MERESCO can be used to implement Linked Data.
