Faster indexing

[raulcf]

• Check how to improve elasticsearch's performance
• Build a pre-indexer that filters out data that has been indexed for a given column. Basically this requires a count-min sketch per column, so that we can decide not to send certain data to the store if it's been already indexed (note that even though sending the data won't change the index, it requires processing anyway).
• throw more strategies here... (that don't involve building our own stuff, for now)

[jmftrindade]

Do any of these help? https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html

[raulcf]

Thanks!
Bulk requests should help.

In general, however, we need a more aggressive strategy here, as we are 1 order of magnitude lagging behind profiling. Ultimately, it is about reducing the amount of data we are indexing, and exploiting that we are loading columns of databases---repetition is common---filtering out data we have already seen (per column) should help a lot too.

[raulcf]

I just implemented bulk request. It helped a lot, actually. Indexing is now only 3x slower than profiling (although I haven't optimized profiling yet). In any case this is great news, the gaps is closing.

[jmftrindade]

Great, glad to hear that guide helped!