(Insert) order guarantee

[fungs]

Hi,

starting from Python 3.7, dict() objects guarantee insert order, much like collections.OrderedDict. Is there any order guarantee or possibilities for this library? I didn't find any statement in the docs.

Best

[Congyuwang]

In raw mode. Keys are ordered as rocksdb will order them (just ordered). If you are storing Python string / int / etc., each type of keys are group ordered.

However for other objects, since the keys are pickle serialized, the order is uncertain. There is no way to ensure for these keys since calling into Python for comparison would be too slow.

[Congyuwang]

I do recommend using raw mode for better performance though.

[fungs]

Thanks. I assume you mean byte order and not insert (time) order.

[fungs]

Seems like it is nothing one should rely on.

[Congyuwang]

Right. They are sorted like a treemap. Not like a queue.

[Congyuwang]

If you need to iterate them according to insertion order. Just make a new column family to map (insert sequence no) -> (key). And then iterate over that column family and fetch the value using the keys.

[fungs]

Great, thanks for all your input!

I was doing some research for a prototype with persistent storage and now settled with https://grantjenks.com/docs/diskcache/ which is really well tested and has many different interfaces, including dict-like (ordered), queue-like and mixed forms. The backend uses sqlite3 though. I wonder what the benefits would be to change the backend to rocksDB instead. DiskCache would be a good baseline for benchmark comparison.

[Congyuwang]

That's a good library obviously.

As of being battle tested, rocksdict relies on rocksdb (Facebook developed industrial strength key-value database developed in C++), which is only much more battle tested than many more libraries.

But you are right, rocksdict and/or rocksdb are more of a low-level library. And if you want some certain functionalities, you'll have to write a couple more lines of codes.

[fungs]

DiskCache has a dict-like interface named Index, which should be 90 % compatible to RocksDict or builtin dict typical usage. So it should be straightforward to generate some generic benchmarks.

from rocksdict import Rdict as Dict  # yours
from diskcache import Index as Dict  # baseline
import shelve  # even more baseline

It is just a suggestion since it seems there are plenty of file-based (kv) databases and it may be hard to decide.