Wikidata indexing with qEndpoint

[ate47]

Hi all,

Recently we used qEndpoint to index a recent version of the Wikidata-all dataset, it was done on a Windows computer with a Ryzen 5 5600 CPU with 600GB SSD using 10 cores and 10GB of RAM.

The dataset was found here

dataset	size (gz)	triples	time to index
wikidata-all	148GB	17.5B	50h
wikidata-truthy	58GB	7.3B	14h

Wikidata-all

Here the times to compress the Wikidata all dataset with qEndpoint:

# dictionary compression + SPO index

## Sub chunks creation
section compression -- : 23.86 h
bit triple creation -- :  0.41 h
triple compression/map :  2.01 h
dictionary write ----- :  4.41 h
## Sub chunks merge
sub dict merge ------- :  4.57 h
sub triple merge ----- :  7.85 h

# OPS/PSO/POS indexes
OPS index ------------ :  3.33 h
P.. index ------------ :  1.50 h

Final size:

total 296G
183G index_dev.hdt
113G index_dev.hdt.index.v1-1

Try by yourself

You can try by yourself using the wikidata docker image or with the latest release of qEndpoint.

[D063520]

This is really cool!