Increase CSV Ingest speed

[wmehling]

Building on benchmarking in #51, we can increase CSV ingest speed.

Especially lazy loading / Coupling of CSV Parsing and DB Ingest could be a bottleneck and could be parallelized.

This branch will explore options to increase speed and provide new benchmark results.

Ressources

linkedfactory-pod/bundles/io.github.linkedfactory.core/src/main/java/io/github/linkedfactory/core/kvin/memtable/KvinMemTable.java

Line 314 in c4186a0

var future = async ? executor.submit(() -> {

<< for how to parallelize

[kenwenzel]

A main bottleneck could be that CSV data is sorted row-wise. It could help to create "row groups" where columns are vectorized (e.g. with 1000 tuples each) and those columns are then loaded sequentially. This prevents repeated item, property and context lookups for each tuple and also improves data locality.

The row groups (or buckets if you want) can be created outside of the CsvParser before inserting the data. Combined with a multi-threading model (1 thread for parsing, 1 thread for storing the data) a speedup could be achieved.

[wmehling]

Okay, I will evaluate that as well and report how far we can push this 🚀