Op woensdag 16 november namen Karin de Bruin; informatiemanager van de Openbare Bibliotheek Amsterdam (OBA) en ikzelf in Utrecht deel aan de Meresco Open bijeenkomst georganiseerd door Seecr en SURFfoundation met als thema ‘Metadata in the cloud’. Behalve de OBA waren bij de bijeenkomst vertegenwoordigd DANS, het Gemeente Museum (Den Haag), R5, Emid Consult, Deltares, Kennisnet en de UvTilburg.

Vragen
Voorafgaand aan deze bijeenkomst stelden Karin en ik onszelf de vragen:

1. Wat gaat de cloud betekenen voor (openbare) bibliotheken?
2. Als de open index de aquabrowser opvolgt, hoe gaat dat er dan in de praktijk uitzien?

Language-model-based ranking
Karin en ik vielen qua onderwerpen met onze neus in de boter, want Erik Groeneveld van Seecr had juist die morgen een bezoek gebracht aan Bibliotheek.nl en schetste de laatste stand van zaken van hun proof of concept voor ons. Gebruikers zoeken en vinden veel informatie in Wikipedia, Youtube en op Library Thing. Daarbij formuleren zij hun zoekvraag in ‘gewone mensentaal’ en zoeken doorgaans niet in bibliotheekcatalogi. De uitdaging is deze zoektaal in de Nationale Bibliotheek Catalogus (NBC) te matchen met ‘nette bibliotheekconcepten’. Op dit moment is verfijning van de hits (‘facetten’) mogelijk op type (materiaal), stijl, auteur, componist en taal. Als testbronnen zullen Wikipedia, kranten, youtube, CDR en MUI worden geladen. Deze worden opgeslagen in RDF-formaat. Aan de hand van een test op een – nu nog – kleine database gaf Erik de meerwaarde aan van dit algoritme. Inspiratie putte Seecr onder meer uit het artikel “Language-model-based ranking for queries on rdf-graphs” over communities als Wikipedia.

Meresco software en kennisdeling
Met de methodologie van Open Space wil Seecr kennis delen stimuleren tussen geïnteresseerden binnen én buiten de Meresco community. Iedere innovatieve toepassing die voor één gebruiker gebouwd wordt komt via de Meresco software ten goede aan de hele community.

Bibliothecaris van de toekomst
Vergaderen aan de hand van deze formule levert bijeenkomsten op waarin de deelnemers zich gelijkwaardig voelen en gedragen, waardoor veel creatief denken en kennisdeling plaats kan vinden. Dit gebeurde ook tijdens de bijeenkomst van 16 november bij Seecr. Toepassing van deze formule binnen de OBA leverde reeds een geïnspireerde bijeenkomst over de bibliothecaris van de toekomst op.

Community
Fragmenten van die vragen hebben we in de Open Space sessies uitgediept. Ik heb gemerkt dat de ruimte van het speelveld groot is. De vraag is hoe je die ruimte comfortabel maakt met relevante wensen en antwoorden daarop. Een community? Het aanwezige gezelschap was gemêleerd en daarmee werd de belofte van een goede 360 graden kijk op het speelveld daadwerkelijk ingelost.

Wel of niet in de Cloud
Wat zijn de feiten, meningen en ingrediënten voor een optimaal aggregatieproces voor metadata van meerdere organisaties? Dat was mijn leerdoel voor de Open Space sessies. In die vraag ligt ook een belangrijke rol van toegevoegde waarde van de Cloud besloten. De Cloud kan het daadwerkelijke delen van metadata goed faciliteren. De Cloud wordt door menigeen ervaren als een onveilig gebied voor vertrouwelijke data. Misschien is het geruststellender om al te vertrouwelijke gegevens gewoonweg niet te delen in de Cloud. Wil je met de Cloud aan de slag voor het uitwisselen van gegevens en informatie, dan zul je dus binnen je organisatie al moeten bepalen welke metadata al dan niet vertrouwelijk is. Het werkproces waarin metadata tot stand komt moet daarop ingericht zijn.

Find out more at Meresco Open (Dutch).

Meta Index

One of the key elements of the Open Index is the Meta Index (one or more). It does not index documents, it indexes indexes. It records, among other things, which index indexes which vocabularies. Also, it indexes the vocabularies itself. Let me explain that.

Vocabulary indexing

When an index joins an Open Index it registers itself with the Meta Index by telling the Meta Index its own location. Also, it tells the Meta Index which vocabularies it uses to index its documents. Many indexes use the Dublin Core vocabulary for indexing, but more specialized indexes use more specialized vocabularies such as FOAF (for social relationships), Geo (longitude and latitude info), MusicBrainz (to describe music, performances, concerts). So now the Meta Index knows exactly which indexes specialize on certain vocabularies.

For every vocabulary reference the Meta Index receives, it retrieves the descriptions of the vocabularies itself (does this make it a meta-meta index then?). That means that it now also knows which fields each vocabulary contains. For example, the MusicBrainz vocabulary contains fields like Album, Artist, Track and so on. For some vocabularies, it might even know about possible values too. The GTAA (an extensive vocabulary for TV etc.) for example contains approximately 97.000 Persons, 27.000 Names, 14.000 Locations and 18.000 (TV) Makers. The Meta Index knows all these names, and it knows if a name representes a person or a location.

Querying for Vocabularies

For any given query, the Meta Index can tell which indexes that are part of the Open Index will give meaningful results. How this is done? In two ways. Suppose someone enters the query “artist=lennon”. If you send this query to the Meta Index, it will lookup which vocabularies have a field named ‘artist’ (ignoring a few problems that arise during matching here), then it will look up the indexes registered for these vocabularies and it will send you this list of indexes. The next step is to send the same query to these indexes and integrate the results.

Now suppose you would enter a simpler query such as “yvon jaspers”. The Meta Index could lookup the word “yvon jaspers” and find it to be in a list of names for television Makers in the GTAA. So it gives you the list of GTAA indexes, and you could take this as a suggestion to include these in your query.

Automating versus User controlled
The examples above assume you being in control of querying the Meta Index and deciding what to do with the hits it gives you. In practise however, the interaction with the Meta Index will be invisible to users. A search portal might show the suggestions from the Meta Index and let the user free to direct his query to one or more of the suggested indexes. For example by saying “did you mean to search for TV maker ‘yvon jaspers’?”. Another search portal might simply take the hints from the Meta Index directly, carry out the users search query on all of them and just show the results.

Freedom of Design Choices

It all comes down to decoupling design choices: creating flexibility because we can not see into the future. With current technology (big integrated indexes), the choice for a particular search engine often implies many other choices you will often only become aware of later. One such decision is the way the search engine deals with multiple indexes. The Open Index allows such decisions to be made separately. The way of working as outlined above is only our first take on how we will do it. It could be any other algorithm in the future.

Next

In a next installment of this blog I’ll cover efficiency and scalability of the Open Index.

This second installment about the Open Index deals with how search queries are processed. Each index contains RDF descriptions (metadata) for objects identified by URIs (identifiers). The objects are not in the index, the metadata and the URIs are.

Late Integration

Late integration means that integration happens late in the process: during query processing. This is quite different from integrating metadata databases. But it is even quite different from the current practise of leaving the databases in place and integrating the metadata into one monolithic index.

Wide or deep

In the Open Index, when an index contains few metadata for many objects, then we call this a wide index. An index containing much metadata for a small set of objects, we call a deep index. Current style indexes are often both wide and deep, as they try to encompass everything. Indexes may contain overlapping sets of URIs, such as, one index might enrich another.

Leading Index

The index that receives a query from a client (e.g. portal) is called the leading index. It determines which other indexes to involve and it takes care of the final ranking and faceting. It returns to the user a top list of matching records, along with facet data.

Integration

The leading index executes the query itself and sends it to one or more other indexes. It asks the other indexes to return only the URIs (conceptually). The leading index integrates these sets using set arithmetic resulting in one set of hits. This arithmetic is the same as is found deep inside search engines.

Facets

Based on the set of hits the leading index adds facets. These are correct and complete facets; not estimates. (How facets can be distributed is the subject of an advanced blog later).

Relevancy

The Open Index is technology agnostic and lends the solution to integrating relevancy from the way indexes are intended to be used. Assumptions:

1. The deeper the index, the more relevant its hits are. It is more specialized.
2. The more indexes yield hits on a URI, the more relevant it is.
3. Native relevancy of the leading index can be used when the width of this index determines the full scope of the query.

A later blog will go into more detail about relevancy.

End result

Once a top of hits has been determined, the leading index gathers the RDF documents for this top. It does so by asking the same indexes for their parts and then it merges the results into on one RDF description and sends this to the client.

What’s next?

In the following installment of this blog I will describe how indexes are found and how optimizations make the basic idea outlined so far scale up.

What does it do?
The Open Index makes maintaining search indexes easier while delivering more room for specialized metadata formats. It makes no choices regarding funtionality or technology unless it is essential for the concept to work properly. It is Open in a sense that:

1. Any metadata format can play.
2. Any technology can be used.

No more monolithic indexes
The main idea is to stop creating one big index for all metadata sets because of these disadvantages:

1. All metadata formats are unified to a common denominator. The richer metadata is ignored.
2. All queries are unified to what one specific type of index supports. Not all is full-text.
3. Update processing becomes a bottleneck, especially when trying to avoid 1 and work around 2.

This causes organisational and technical costs to rise while failing to deliver on specialization.

Specialization
The Open Index does not integrate indexes but it integrates search results. Integrate; not federate! This allows maintainers of specialized sets to make their own choices regarding metadata, technology and update processing when creating their own, independent indexes. These indexes then join in a bigger Open Index by providing unified identifiers and a standard search protocol.

Unified Resource Identifiers
Each index must use URI’s to identify what is in the index. This is just a good practise being applied widely and increasingly, but it is essential.

Standard Search Protocol
Each index should implement a standard protocol for searching. The query language is a variable (see 2), but standardizing on one or two does not hurt. What is essential is the two types of results that must be supported:

1. A top list of the complete records for the best ranked results.
2. A complete list of only the URI’s of all the results.

Peer to peer
Indexes are arranged in a peer to peer fashion. Any index may deal with user queries and will then be called the leading index for the duration of one particular query. User queries are handled by returning results of type A. The leading index uses type B results from other indexes to fullfill the request. The algorithms are packaged in reusable components and deserve a separate blog post.

What’s next?
To explain the concept clearly, I will write some more blogs about:

• Query resolving: about how the actual integration works.
• Finding and selecting indexes: how indexes find other indexes to work with.
• Efficiency optimizations: what is needed to make it work with large indexes and big query loads.

Or call or e-mail me if you don’t want to wait! I am happy to discuss this subject.

We nodigen je van harte uit op 1 juni a.s.

Kijk op http://cq2.nl/1juni voor de deelnemers, de onderwerpen en hoe aan te melden.

Tot ziens!

Multiple Implementations For an Interface
DNA makes a distinction between interfaces and implementation, which is a well-known practice.  However, sometimes there are multiple distinct implementations for the same interface.  For example, two Storage components might support the same interface, yet representing distinct storages.  They can have distinct performance properties or represent distinct data sets.  DNA already supports this by supporting multiple implementations for any interface. At configuration time, one branch might use one implementation while another branch uses another. For a step-by-step programmers guide too how this works see Component Configuration with DNA.

Components Choose Their Own Implementations
Labeled Interface Invocation allows components to choose distinctive implementations of the same interface at run-time. This is done by first labeling the implementations during configuration, and then use these labels during the invocation of interfaces.

Suppose we have two storages supporting the interface

    addNew(data)

Normally, a message to an arbitrary implementation of this interface would look like:

    self.any.addNew(someData)

Now suppose that there are two implementations, differing in the way they store and safeguard data. Lets label these implementations “reviews” and “audits”:

    reviewStore = Storage(name="reviews")
    auditStore = Storage(name="audits")

With Labeled Interface Invocation we can now target messages to any of these implementation as follows:

   self.any["reviews"].addNew(newReview)
   self.any["audits"].addNew(auditRecord)

An interesting aspect of the lines above is that the constant labels “reviews” and “audits” can in fact be variables. Thus, labels can also be computed and routing messages to different components can be done on the fly. For example:

    target = ... compute or lookup a label ...
    self.any[target].addNew(someData)

Conclusion
Meresco’s DNA has been stable for about two years. So new DNA features are rare, and we were pleasantly surprised to have found one. As usual, it took a long time, and a lot of thinking, but only a few lines of code; 3 actually. Most code changes are for tests and other components to used LLI. Here is the complete LLI changeset.

Have any questions about this feature or want support to use it your self? Please contact me.

Meresco Presentation June 25th 2010 (PDF)

These slides were presented during the Open Space MERESCO meeting. Topics presented were:

• What is Meresco?
• What it does?
• What’s new?

Agenda English: ODT or PDF format.

Agenda Dutch: ODT of PDF formaat.

The Registration Form (Dutch) is online. Please register here.