Comments for MERESCO http://blog.meresco.org What happened in the MERESCO community? Tue, 18 May 2010 22:04:18 +0000 hourly 1 http://wordpress.com/ Comment on Open Space MERESCO meeting June 25th 2010 by Webhamer Weblog: Search & ICT-related blogging » links for 2010-05-18 http://blog.meresco.org/2010/05/18/open-space-meresco-bijeenkomst/#comment-20 Webhamer Weblog: Search & ICT-related blogging » links for 2010-05-18 Tue, 18 May 2010 22:04:18 +0000 http://blog.meresco.com/?p=206#comment-20 [...] Open Space MERESCO bijeenkomst « MERESCO SURFfoundation en Seek You Too zullen in juni een MERESCO bijeenkomst organiseren bij SURFfoundation in Utrecht. Het zal een open bijeenkomst zijn; alle geinteresseerden zijn welkom. (tags: Meresco opensource searchengine) [...] [...] Open Space MERESCO bijeenkomst « MERESCO SURFfoundation en Seek You Too zullen in juni een MERESCO bijeenkomst organiseren bij SURFfoundation in Utrecht. Het zal een open bijeenkomst zijn; alle geinteresseerden zijn welkom. (tags: Meresco opensource searchengine) [...]

]]> Comment on Storage versus Index by How to scale up Meresco « MERESCO http://blog.meresco.org/2010/02/12/storage/#comment-15 How to scale up Meresco « MERESCO Fri, 02 Apr 2010 12:36:48 +0000 http://blog.meresco.com/?p=110#comment-15 [...] must be allocated. In this area Lucene, BerkeleyDB and OWLIM have earned great reputations. Meresco’s architecture helps to get the most out of [...] [...] must be allocated. In this area Lucene, BerkeleyDB and OWLIM have earned great reputations. Meresco’s architecture helps to get the most out of [...]

]]> Comment on Autocomplete with 5 million+ proper names by How to scale up Meresco « MERESCO http://blog.meresco.org/2009/12/18/autocomplete-with-5-million-proper-names/#comment-14 How to scale up Meresco « MERESCO Fri, 02 Apr 2010 12:35:11 +0000 http://meresco.wordpress.com/?p=19#comment-14 [...] homegrown Facet Index and Sorted Dictionary Index (used for auto-complete) can be scaled following approach B. However, with a single-node limit of roughly one billion [...] [...] homegrown Facet Index and Sorted Dictionary Index (used for auto-complete) can be scaled following approach B. However, with a single-node limit of roughly one billion [...]

]]> Comment on What makes Meresco different from Solr? by How to scale up Meresco « MERESCO http://blog.meresco.org/2010/03/01/what-makes-meresco-differ-from-solr/#comment-13 How to scale up Meresco « MERESCO Fri, 02 Apr 2010 12:35:04 +0000 http://blog.meresco.com/?p=139#comment-13 [...] of records Meresco can host 10 – 100 million records on one machine, mostly limited by what its indexes can do. Scaling up requires a closer look at these indexes to see how additional resources must be [...] [...] of records Meresco can host 10 – 100 million records on one machine, mostly limited by what its indexes can do. Scaling up requires a closer look at these indexes to see how additional resources must be [...]

]]> Comment on Inbox component by How to scale up Meresco « MERESCO http://blog.meresco.org/2010/01/29/inbox-component/#comment-12 How to scale up Meresco « MERESCO Fri, 02 Apr 2010 12:34:55 +0000 http://blog.meresco.com/?p=91#comment-12 [...] Approach B Extract the most demanding components from the server’s configuration and put these on separate machines. Reconnect them using the Inbox component. [...] [...] Approach B Extract the most demanding components from the server’s configuration and put these on separate machines. Reconnect them using the Inbox component. [...]

]]> Comment on What makes Meresco different from Solr? by Webhamer Weblog: Search & ICT-related blogging » links for 2010-03-02 http://blog.meresco.org/2010/03/01/what-makes-meresco-differ-from-solr/#comment-10 Webhamer Weblog: Search & ICT-related blogging » links for 2010-03-02 Tue, 02 Mar 2010 22:02:17 +0000 http://blog.meresco.com/?p=139#comment-10 [...] What makes Meresco differ from Solr? « MERESCO Solr focusses to get the most out of one index type: Lucene. Meresco supports a number of different index types, each specialized for a specific task. (tags: Meresco Solr) [...] [...] What makes Meresco differ from Solr? « MERESCO Solr focusses to get the most out of one index type: Lucene. Meresco supports a number of different index types, each specialized for a specific task. (tags: Meresco Solr) [...]

]]> Comment on Autocomplete with 5 million+ proper names by What makes Meresco differ from Solr? « MERESCO http://blog.meresco.org/2009/12/18/autocomplete-with-5-million-proper-names/#comment-9 What makes Meresco differ from Solr? « MERESCO Mon, 01 Mar 2010 10:25:01 +0000 http://meresco.wordpress.com/?p=19#comment-9 [...] information attached to alphabetically ordered terms. It supports prefix queries such as needed for auto-complete. It is implemented using a Burst [...] [...] information attached to alphabetically ordered terms. It supports prefix queries such as needed for auto-complete. It is implemented using a Burst [...]

]]> Comment on Dependable OAI Repositories by What makes Meresco differ from Solr? « MERESCO http://blog.meresco.org/2010/01/22/dependable-oai-repositories/#comment-8 What makes Meresco differ from Solr? « MERESCO Mon, 01 Mar 2010 10:24:57 +0000 http://blog.meresco.com/?p=75#comment-8 [...] Dictionary Index This index supports fast lookup of arbitrary textual information related to keys. It supports simple lookup ‘queries’ only. It is implemented using Berkeley DB, which is known for its good scalability and performance. It is being used to scale up set and metadataPrefix queries in OAI-PMH to tens of millions of records. This post describes the process: Dependable OAI Repositories. [...] [...] Dictionary Index This index supports fast lookup of arbitrary textual information related to keys. It supports simple lookup ‘queries’ only. It is implemented using Berkeley DB, which is known for its good scalability and performance. It is being used to scale up set and metadataPrefix queries in OAI-PMH to tens of millions of records. This post describes the process: Dependable OAI Repositories. [...]

]]> Comment on What to do with Linked Data? by Erik J. Groeneveld http://blog.meresco.org/2010/02/05/what-to-do-with-linked-data/#comment-7 Erik J. Groeneveld Fri, 12 Feb 2010 08:09:35 +0000 http://blog.meresco.com/?p=100#comment-7 What @jvanvuuren points out, could that be added as step 7 as: 7. Start using fine grained provenance information? I wasn't clear about the fact that I assume equal authority within your Authority Cloud. For example, when your records contain author URIs and you load author information from an institution you trust, you can just merge the triples. That is what happens today (using other technologies), and you can continue doing so. After some time you will definitely want to distinguish who said what, but by that time, you already have your (RDF) tools in place making making life easier. I got a reply by e-mail from Frits van Latum suggesting this type of graphs for dealing with tags being added to objects. I am posting it here assuming Frits agrees: <a href="http://meresco.files.wordpress.com/2010/02/rdfdiscover.jpg" rel="nofollow"><img class="alignright size-full wp-image-109" title="rdfDiscover" src="http://meresco.files.wordpress.com/2010/02/rdfdiscover.jpg" alt="" width="432" height="134" /></a> What @jvanvuuren points out, could that be added as step 7 as:

7. Start using fine grained provenance information?

I wasn’t clear about the fact that I assume equal authority within your Authority Cloud. For example, when your records contain author URIs and you load author information from an institution you trust, you can just merge the triples. That is what happens today (using other technologies), and you can continue doing so. After some time you will definitely want to distinguish who said what, but by that time, you already have your (RDF) tools in place making making life easier.

I got a reply by e-mail from Frits van Latum suggesting this type of graphs for dealing with tags being added to objects. I am posting it here assuming Frits agrees:

]]> Comment on What to do with Linked Data? by jvanvuuren http://blog.meresco.org/2010/02/05/what-to-do-with-linked-data/#comment-6 jvanvuuren Wed, 10 Feb 2010 14:10:37 +0000 http://blog.meresco.com/?p=100#comment-6 As stated a record describes a set of statements about everything (e.g a set of triplets). But apart from the content of a record (e.g. triplets about objects) a record is also implicitly used to define the “source” of the data. If you do not bring this information in the equation you will lose relevant data. This is especially prominent when the triplets are subjective, so in the querying of the data you would like to include characteristics of the “source” of these data (like experience, role, organization) This means that if the source of the record is not defined in the record itself, not only the triplets in a record should be stored, but also the triplets about the record needs to be stored, before you abandon the principle of a “record”. In theory a “record” is just any object as another. In triplet terms you can define “record 5832” contains “triplet 123”. You can also define “record 5832” is supplied by “organization ABC”. This implies that “triplet 123” is supplied by “organization ABC”. Only if the later is stored the object of “record” is not relevant anymore. As stated a record describes a set of statements about everything (e.g a set of triplets). But apart from the content of a record (e.g. triplets about objects) a record is also implicitly used to define the “source” of the data. If you do not bring this information in the equation you will lose relevant data. This is especially prominent when the triplets are subjective, so in the querying of the data you would like to include characteristics of the “source” of these data (like experience, role, organization)

This means that if the source of the record is not defined in the record itself, not only the triplets in a record should be stored, but also the triplets about the record needs to be stored, before you abandon the principle of a “record”.

In theory a “record” is just any object as another. In triplet terms you can define “record 5832” contains “triplet 123”. You can also define “record 5832” is supplied by “organization ABC”. This implies that “triplet 123” is supplied by “organization ABC”. Only if the later is stored the object of “record” is not relevant anymore.

]]>