RDFpro usage examples (SAC paper)

Processing

We assume the latest version of Freebase is available as file freebase_new.nt.gz, while a previous version is available as file freebase_old.nt.gz. The figures below show how to use RDFpro to extract TBox and VOID statistics (left figure) and to identify newly added triples (right figure) in the considered scenario:

We use the @tbox and @stats processors to extract TBox and VOID statistics. The two processors can be used separately by invoking RDFpro twice as follows (note the use of the -t 100 option: it require to emit statistics of classes and properties with at least 100 instances, which is necessary in order to load the generated statistics in tools such as Protégé - see below):

rdfpro @read freebase_new.nt.gz @tbox @write tbox.tql.gz
rdfpro @read freebase_new.nt.gz @stats -t 100 @write stats.tql.gz

The two processors can also be composed in a single pipeline where Freebase data is read once and fed to both processors in parallel, as in the figure above. The corresponding RDFpro command is:

rdfpro @read freebase_new.nt.gz { @tbox @write tbox.tql.gz , @stats -t 100 @write stats.tql.gz }

Extraction of newly added triples can be done exploiting the parallel composition with the difference set operator (flag d) to combine quads, using the command:

rdfpro { @read freebase_new.nt.gz , @read freebase_old.nt.gz }d @write new-triples.tql.gz

Results

The table below reports the tasks execution times, throughputs, input and output sizes (quads and gzipped bytes) we measured on our test machine, an Intel Core I7 860 workstation with 16 GB RAM and a 500GB 7200RPM hard disk, using pigz and pbzip2 as compressors/decompressors and sort -S 4096M --batch-size=128 --compress-program=pigz as the sort command; our ‘latest’ Freebase release (2789 Mquads) was obtained on 2015/01/18, while our old release (2623 Mquads) was downloaded on 2014/07/10. In addition to the table below, when running the comparison task we measured a disk usage of 92.8 GB for storing the temporary files produced by the sorting-based difference set operator (∼18 bytes per input triple).

Comparing the two Freebase releases results the most expensive task due to sorting and involved input size. When performed jointly, TBox and statistics extraction present performance figures close to statistics extraction alone, as data parsing is performed once and the cost of TBox extraction (excluded parsing) is negligible. This is an example of how the aggregation of multiple processing tasks in a single RDFpro computation can generally lead to better performances due to a reduction of I/O overhead.

To provide an idea of the how analysis results can be used to explore the dataset, the figure below shows the joint browsing of extracted TBox and statistics in Protégé, exploiting the specific concept annotations emitted by @stats. The class and property hierarchies are augmented with the number of entities and property triples (marked as 1 in the figure), as well as with the detected property usage (2), e.g., O for object property, I for inverse functional; each concept is annotated with an example instance and a VOID partition individual (3), which provides numeric statistics about the concept (4).

Processing

We assume that the Freebase dump is available as file freebase_new.nt.gz. We implement the filtering task with two invocations of RDFpro as shown in the figure below.

The first invocation (marked as 1 in the figure) generates an RDF file listing as subjects the URIs of the entities of interest. This is done with two parallel @groovy processors, extracting respectively musical groups and no more active musical entities, whose outputs are combined with the difference merge criterion using the following RDFpro command:

rdfpro @read freebase.nt.gz \
       { @groovy -p 'emitIf(t == fb:music.musical_group)' , \
         @groovy -p 'if(p == fb:music.artist.active_end) emit(s, rdf:type, fb:music.musical_group, null)' }d \
       @write entities.tql.gz

The second invocation (marked as 2 in the figure) uses another @groovy processor to extract the desired quads, testing predicates and requiring subjects to be contained in the previously extracted file (whose URIs are indexed in memory by a specific function in the @groovy expression). The corresponding RDFpro command is:

rdfpro @read freebase.nt.gz \
       @groovy 'def init(args) { instances = loadSet("./instances.tql", "s"); };
                emitIf((p == rdfs:label || p == fb:music.artist.genre || p == fb:music.artist.origin)
                       && instances.match(s) );' \
       @write output.tql.gz

Results

The table below reports the execution times, throughputs, input and output sizes of the two invocations of RDFpro on our test machine (see previous scenario for the specs), when applied to the Freebase dump (2789 Mquads) downloaded on 2015/01/18.

Although simple, the example shows how practical, large-scale filtering tasks are feasible with RDFpro. Streaming allows processing large amounts of data, while sorting enables the use of the intersection, union and difference set operators to implement, respectively, the conjunction, disjunction and negation of entity selection conditions.

More complex scenarios may be addressed with additional invocations that progressively augment the result (e.g., a third invocation can identify albums of selected artists, while a fourth invocation can extract the quads describing them). In cases where RDFpro model is insufficient or impractical (e.g., due to the need for recursive extraction of related entities, aggregation or join conditions), RDFpro can still be used to perform a first coarse-grained filtering that reduces the number of quads and eases their downstream processing.

Processing

The figure below shows the required processing steps (the most significant processor for each step is shown).

The five steps can be executed both individually (bash script process_single.sh) or by aggregating the first two and the last three (bash script process_aggregated.sh. The RDFpro commands for the single steps are reported below.

• Transform. Data transformation serves (i) to track provenance, by placing quads in different named graphs based on the source dataset; and (ii) to adopt optimal serialization format (Turtle Quads) and compression scheme (gzip) that speed up further processing.

  rdfpro { @read metadata.trig , \
           @read vocab/* @transform '=c <graph:vocab>' , \
           @read freebase/* @transform '-spo fb:type.object.name fb:type.object.type <http://rdf.freebase.com/key/*>
               <http://rdf.freebase.com/ns/user.*> <http://rdf.freebase.com/ns/base.*> =c <graph:freebase>' , \
           @read geonames/*.rdf .geonames:geonames/all-geonames-rdf.zip \
               @transform '-p gn:childrenFeatures gn:locationMap gn:nearbyFeatures gn:neighbouringFeatures
                   gn:countryCode gn:parentFeature gn:wikipediaArticle rdfs:isDefinedBy rdf:type =c <graph:geonames>' , \
           { @read dbp_en/* @transform '=c <graph:dbp_en>' , \
             @read dbp_es/* @transform '=c <graph:dbp_es>' , \
             @read dbp_it/* @transform '=c <graph:dbp_it>' , \
             @read dbp_nl/* @transform '=c <graph:dbp_nl>' } \
               @transform '-o bibo:* -p dc:rights dc:language foaf:primaryTopic' } \
         @transform '+o <*> _:* * *^^xsd:* *@en *@es *@it *@nl' \
         @transform '-o "" ""@en ""@es ""@it ""@nl' \
         @write filtered.tql.gz
  

• TBox extraction. This step extracts the TBox needed for RDFS inference.

  rdfpro @read filtered.tql.gz \
         @tbox \
         @transform '-o owl:Thing schema:Thing foaf:Document bibo:* con:* -p dc:subject foaf:page dct:relation bibo:* con:*' \
         @write tbox.tql.gz
  

• Smushing. Smushing identifies owl:sameAs equivalence classes and assigns a canonical URI to each of them.

  rdfpro @read filtered.tql.gz \
         @smush '<http://dbpedia>' '<http://it.dbpedia>' '<http://es.dbpedia>'
                '<http://nl.dbpedia>' '<http://rdf.freebase.com>' '<http://sws.geonames.org>' \
         @write smushed.tql.gz
  

• Inference. RDFS inference excludes rules rdfs4a, rdfs4b and rdfs8 to avoid materializing uninformative <X rdf:type rdfs:Resource> quads.

  rdfpro @read smushed.tql.gz \
         @rdfs -c '<graph:vocab>' -d tbox.tql.gz \
         @write inferred.tql.gz
  

• Deduplication. Deduplication takes quads with the same subject, predicate and object (possibly produced by previous steps) and merges them in a single quad inside a graph linked to all the original sources.

  rdfpro @read inferred.tql.gz \
         @unique -m \
         @write dataset.tql.gz
  

The RDFpro commands for the aggregated steps are reported below:

• Transform + TBox extraction

  rdfpro { @read metadata.trig , \
           @read vocab/* @transform '=c <graph:vocab>' , \
           @read freebase/* @transform '-spo fb:type.object.name fb:type.object.type <http://rdf.freebase.com/key/*>
               <http://rdf.freebase.com/ns/user.*> <http://rdf.freebase.com/ns/base.*> =c <graph:freebase>' , \
           @read geonames/*.rdf .geonames:geonames/all-geonames-rdf.zip \
               @transform '-p gn:childrenFeatures gn:locationMap gn:nearbyFeatures gn:neighbouringFeatures
                   gn:countryCode gn:parentFeature gn:wikipediaArticle rdfs:isDefinedBy rdf:type =c <graph:geonames>' , \
           { @read dbp_en/* @transform '=c <graph:dbp_en>' , \
             @read dbp_es/* @transform '=c <graph:dbp_es>' , \
             @read dbp_it/* @transform '=c <graph:dbp_it>' , \
             @read dbp_nl/* @transform '=c <graph:dbp_nl>' } \
               @transform '-o bibo:* -p dc:rights dc:language foaf:primaryTopic' } \
         @transform '+o <*> _:* * *^^xsd:* *@en *@es *@it *@nl' \
         @transform '-o "" ""@en ""@es ""@it ""@nl' \
         @write filtered.tql.gz \
         @tbox \
         @transform '-o owl:Thing schema:Thing foaf:Document bibo:* con:* -p dc:subject foaf:page dct:relation bibo:* con:*' \
         @write tbox.tql.gz
  

• Smushing + Inference + Deduplication

  rdfpro @read filtered.tql.gz \
         @smush '<http://dbpedia>' '<http://it.dbpedia>' '<http://es.dbpedia>' \
                '<http://nl.dbpedia>' '<http://rdf.freebase.com>' '<http://sws.geonames.org>' \
         @rdfs -c '<graph:vocab>' -d tbox.tql.gz \
         @unique -m \
         @write dataset.tql.gz
  

Results

The table below reports the execution times, throughputs and input and output sizes of each step, covering both the cases where steps are performed separately via intermediate files and multiple invocations of RDFpro (upper part of the table), or aggregated per processing phase using composition capabilities (lower part). RDFpro also reported the use of ∼2 GB of memory for smushing an owl:sameAs graph of ∼38M URIs and ∼8M equivalence classes (∼56 bytes/URI).

Also in this scenario, the aggregation of multiple processing tasks leads to a marked reduction of the total processing time from 45291 s to 26487 s (times were respectively 47803 s and 31981 s in September 2014 test) due to the elimination of the I/O overhead for intermediate files. While addressed separately, the three scenarios of dataset analysis, filtering and merging are often combined in practice, e.g., to remove unwanted ABox and TBox quads from input data, merge remaining quads and analyze the result producing statistics that describe and document it; an example of such combination is reported in the ISWC SemDev paper and further detailed in this page.