diff --git a/api/index.html b/api/index.html index e6576ec..af87029 100644 --- a/api/index.html +++ b/api/index.html @@ -62,7 +62,7 @@
- + Skip to content @@ -277,16 +277,6 @@ - - - - - @@ -510,16 +495,6 @@ - - - - - @@ -670,13 +640,6 @@

Module

-

VeloxDB

-

Unit Tests

-

VeloxDB is a persistent key-value store database library. It designed to store -key-value pairs and allow efficient retrieval based on the key. This system is -inspired by modern databases like LevelDB -and RocksDB, and supports multiple data -types using C++ Templates and Protocol Buffers.

Database Operations

VeloxDB::Open(string db_name)

Initializes the database system, setting up the necessary files and directories (including SSTs and related data). Can be initialized with a custom Memtable size or default size of 1e3.

diff --git a/search/search_index.json b/search/search_index.json index 770d8be..abe7db7 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"VeloxDB","text":"

VeloxDB is a persistent key-value store database library. It designed to store key-value pairs and allow efficient retrieval based on the key. This system is inspired by modern databases like LevelDB and RocksDB, and supports multiple data types using C++ Templates and Protocol Buffers.

"},{"location":"#supported-data-types","title":"Supported Data Types","text":"

2024-09-12 Restructure with Protobuf Using Protocol Buffer for data serialization

syntax = \"proto3\";\n\nmessage KeyValue {\n\n  oneof key {\n    int32 int_key = 1;\n    int64 long_key = 2;\n    double double_key = 3;\n    string string_key = 4;\n    string char_key = 5;\n  }\n\n  oneof value {\n    int32 int_value = 6;\n    int64 long_value = 7;\n    double double_value = 8;\n    string string_value = 9;\n    string char_value = 10;\n  }\n\n  enum KeyValueType {\n    INT = 0;\n    LONG = 1;\n    DOUBLE = 2;\n    CHAR = 3;\n    STRING = 4;\n\n  }\n\n  KeyValueType key_type = 11;\n  KeyValueType value_type = 12;\n}\n

2024-09-09 Support Template 

enum KeyValueType { INT, LONG, DOUBLE, CHAR, STRING };\n

2024-08-28 Support"},{"location":"#supported-platforms","title":"Supported Platforms","text":"

The KV-Store system has been tested across multiple platforms and compilers. Below is the current support status:

Platform Compiler Status MacOS ARM64 GCC \u2705 Ubuntu ARM64 GCC \u2705 Ubuntu ARM64 Clang \u2705 Windows x86 MSVC (cl) \u2705"},{"location":"#legacy-repo","title":"Legacy Repo","text":"

KvDB

KV-Store

"},{"location":"api/","title":"Module","text":""},{"location":"api/#veloxdb","title":"VeloxDB","text":"

VeloxDB is a persistent key-value store database library. It designed to store key-value pairs and allow efficient retrieval based on the key. This system is inspired by modern databases like LevelDB and RocksDB, and supports multiple data types using C++ Templates and Protocol Buffers.

"},{"location":"api/#database-operations","title":"Database Operations","text":""},{"location":"api/#veloxdbopenstring-db_name","title":"VeloxDB::Open(string db_name)","text":"

Initializes the database system, setting up the necessary files and directories (including SSTs and related data). Can be initialized with a custom Memtable size or default size of 1e3.

#include \"VeloxDB/VeloxDB.h\"\n/*\n *  Initialize with default value : \n *       Memtable::size == 1e3\n *       SsTFileManager::DiskBTree::Degree == 3\n */ \nauto MyDBDefault = std::make_unique<VeloxDB>();\nauto MyDBDefault = std::make_unique<VeloxDB>(int memtableSize, int BTreeDegree);\n\nMyDBDefault->Open(\"database_name\");\n
"},{"location":"api/#veloxdbclose","title":"VeloxDB::Close()","text":"

Closes the database, flushing any data in memory (Memtable) to disk and storing it in SSTs.

#include \"VeloxDB/VeloxDB.h\"\n// Close the database and flush the Memtable to disk\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\nMyDB->Close();\n
"},{"location":"api/#data-operations","title":"Data Operations","text":""},{"location":"api/#template-veloxdbputk-key-v-value","title":"Template VeloxDB::Put(K key, V value)

Inserts a key-value pair into the database, where both the key and value can be of various types (int, double, string, etc.).

#include \"VeloxDB/VeloxDB.h\"\n// Example of inserting different data types\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\nMyDB->Put(1, 100);             // int -> int\nMyDB->Put(1.5, 'A');           // double -> char\nMyDB->Put(\"Hello\", 1e8LL);     // string -> long long\n
","text":""},{"location":"api/#veloxdbgetconst-keyvaluewrapper-key","title":"VeloxDB::Get(const KeyValueWrapper& key)

Retrieves a value from the database based on the key. Supports multiple data types.

#include \"VeloxDB/VeloxDB.h\"\n#include \"kv/KeyValue.h\"\n// Example of retrieving values\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\nMyDB->Put(1, 100);\nMyDB->Put(1.5, 'A');\nMyDB->Put(\"Hello\", 1e8LL);\n\n// Retrieve the value by key\nauto result1 = MyDB->Get(\"Hello\");\nlong long value1 = result1.kv.long_value(); // 1e8\nstring key1 = result1.kv.string_key(); // \"Hello\"\n\n// Retrieve the value by `KeyValueWrapper` instance\nauto result1 = MyDB->Get(KeyValueWrapper(\"Hello\", \"\")); \n// Expected result1: { key: \"Hello\", value: 1e8LL }\nlong long value1 = result1.kv.long_value(); // 1e8\nstring key1 = result1.kv.string_key(); // \"Hello\"\n\n// e.g.2\nauto result2 = MyDB->Get(1);\nint value2 = result2.kv.int_value();\nint key2 = results.kv.int_key();\n\n// check if not found using : bool KeyValueWrapper::isEmpty() const;\nif(result.isEmpty()){\n    // Key not found :-(\n} else {\n    // Found :-D\n}\n
","text":""},{"location":"api/#veloxdbscankeyvaluewrapper-smallestkey-keyvaluewrapper-largestkey","title":"VeloxDB::Scan(KeyValueWrapper smallestKey, KeyValueWrapper largestKey)

Scans the database for key-value pairs within a specified key range. The results are returned in sorted key order.

#include \"VeloxDB/VeloxDB.h\"\n// Scan for key-value pairs within a range\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\n// Scan by key\nstd::set<KeyValueWrapper> results = MyDB->Scan(1, 10);\n// Scan by `KeyValueWrapper` instance\nstd::set<KeyValueWrapper> results = MyDB->Scan(KeyValueWrapper(1, \"\"), KeyValueWrapper(10, \"\"));\n
","text":""},{"location":"api/#veloxdbupdatekeyvaluewrapper-keytoupdate","title":"VeloxDB::Update(KeyValueWrapper KeyToUpdate)

(TBA) This will allow the updating of key-value pairs within the database.

","text":""},{"location":"api/#veloxdbdeletekeyvaluewrapper-key","title":"VeloxDB::Delete(KeyValueWrapper Key)

(TBA) This will allow the deletion of key-value pairs from the database.

","text":""},{"location":"api/#buffer-pool-operation","title":"Buffer Pool Operation","text":""},{"location":"api/#setbufferpoolparameterssize_t-capacity-evictionpolicy-policy","title":"setBufferPoolParameters(size_t capacity, EvictionPolicy policy)

Set/reset buffer pool size_t:: capacity and EvictionPolicy:: policy (LRU, CLOCK, RANDOM)

EvictionPolicy newPolicy = EvictionPolicy::LRU;\nEvictionPolicy newPolicy = EvictionPolicy::CLOCK;\nEvictionPolicy newPolicy = EvictionPolicy::RANDOM;\n

This method will clear all the previous cache in the buffer pool.

#include \"VeloxDB/VeloxDB.h\"\n#include \"Memory/BufferPool/BufferPool.h\"\n// Open the database\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\n\n// Set buffer pool parameters\nsize_t Capacity = 1e3;\nEvictionPolicy Policy = EvictionPolicy::CLOCK;\nMyDB->SetBufferPoolParameters(Capacity, Policy);\n\n// Reset \nsize_t newCapacity = 1e4;\nEvictionPolicy newPolicy = EvictionPolicy::LRU;\nMyDB->SetBufferPoolParameters(newCapacity, newPolicy);\n\n// Perform database operations\nMyDB->Put(1, \"value1\");\nKeyValueWrapper value = MyDB->Get(1);\n\n// Close the database\nMyDB->Close();\n
","text":""},{"location":"api/#printcachehit","title":"printCacheHit()

print total number of cache hit in the buffer pool during the database operations.

#include \"VeloxDB/VeloxDB.h\"\n\nint memtableSize = 1e4; \nauto db = std::make_unique<VeloxDB>(memtableSize, 3);\ndb->Open(\"test_db\");\n\nconst int numEntries = 1e6;  // Insert 1e6 key-value pairs\n\n// Insert key-value pairs\nfor (int i = 0; i < numEntries; ++i) {\n    db->Put(i, \"value_\" + std::to_string(i));\n}\n\nstd::set<KeyValueWrapper> resultSet = db->Scan(KeyValueWrapper(1, \"\"), KeyValueWrapper(50000, \"\"));\n\ndb->printCacheHit(); // this will print the total number of cache hit in buffer pool\n\n// Clean up\ndb->Close();\n
","text":""},{"location":"benchmark/","title":"Benchmark","text":""},{"location":"benchmark/#benchmark","title":"Benchmark","text":""},{"location":"benchmark/#veloxdbput","title":"VeloxDB::Put","text":"

Put throughput with different Memtable size

    B Tree Degree = 3\n    page size = 4 kb\n

"},{"location":"benchmark/#veloxdbget","title":"VeloxDB::Get","text":"

Get latency with different Memtable size

    B Tree Degree = 3\n    page size = 4 kb\n

"},{"location":"benchmark/#veloxdbscan","title":"VeloxDB::Scan","text":"

Scan throughput with different Memtable size

    B Tree Degree = 3\n    page size = 4 kb\n

"},{"location":"design/","title":"Database Design","text":""},{"location":"design/#tba","title":"TBA","text":""},{"location":"layout/","title":"Layout","text":""},{"location":"layout/#sst-files-layout","title":"SST Files Layout","text":"
[Internal Node Page (Root)]\n[Internal Node Page 1]\n[Internal Node Page 2]\n...\n[Internal Node Page n]\n[Leaf Node Page 1]\n[Leaf Node Page 2]\n[Leaf Node Page 3]\n...\n[Leaf Node Page m]\n[* Clustered Index Page]\n[* Bloom Filter Page]\n[SST Metadata Page]\n
"},{"location":"layout/#pagepagesize","title":"Page::PageSize","text":"

Page with PageSize:: PageSize (4KB, 8KB)

"},{"location":"layout/#pagesst_metadata","title":"Page::SST_MetaData","text":"
LeafNode_Begin_Offset\nLeafNode_End_offset\nFileName\n
"},{"location":"layout/#pageleafnodes","title":"Page::LeafNodes","text":"
/*\n *  4kb / 8kb chunk\n *  sorted by key\n */\nserialized key-value pair 1 metadata (serialized by protobuf)\nserialized key-value pair 2 metadata (serialized by protobuf)\nserialized key-value pair 3 metadata (serialized by protobuf)\n...\n// with padding\n
"},{"location":"layout/#pageinternalnodes","title":"Page::InternalNodes","text":"
/*\n *  4kb / 8kb chunk\n *  sorted by level\n */\nlevel#0 key-value pair 0 metadata (serialized by protobuf), jump_offset_L1_K0, jump_offset_L1_K1\nlevel#1 key-value pair 1 metadata (serialized by protobuf), jump_offset_L2_K0, jump_offset_L2_K1\nlevel#1 key-value pair 2 metadata (serialized by protobuf), jump_offset_L2_K1, jump_offset_L2_K2\n...\n// with padding\n
"},{"location":"layout/#pagebloomfilter","title":"Page::BloomFilter","text":"

TBD

"},{"location":"layout/#pageclusteredindex","title":"Page::ClusteredIndex","text":"

TBD

"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"VeloxDB","text":"

VeloxDB is a persistent key-value store database library. It designed to store key-value pairs and allow efficient retrieval based on the key. This system is inspired by modern databases like LevelDB and RocksDB, and supports multiple data types using C++ Templates and Protocol Buffers.

"},{"location":"#supported-data-types","title":"Supported Data Types","text":"

2024-09-12 Restructure with Protobuf Using Protocol Buffer for data serialization

syntax = \"proto3\";\n\nmessage KeyValue {\n\n  oneof key {\n    int32 int_key = 1;\n    int64 long_key = 2;\n    double double_key = 3;\n    string string_key = 4;\n    string char_key = 5;\n  }\n\n  oneof value {\n    int32 int_value = 6;\n    int64 long_value = 7;\n    double double_value = 8;\n    string string_value = 9;\n    string char_value = 10;\n  }\n\n  enum KeyValueType {\n    INT = 0;\n    LONG = 1;\n    DOUBLE = 2;\n    CHAR = 3;\n    STRING = 4;\n\n  }\n\n  KeyValueType key_type = 11;\n  KeyValueType value_type = 12;\n}\n

2024-09-09 Support Template 

enum KeyValueType { INT, LONG, DOUBLE, CHAR, STRING };\n

2024-08-28 Support"},{"location":"#supported-platforms","title":"Supported Platforms","text":"

The KV-Store system has been tested across multiple platforms and compilers. Below is the current support status:

Platform Compiler Status MacOS ARM64 GCC \u2705 Ubuntu ARM64 GCC \u2705 Ubuntu ARM64 Clang \u2705 Windows x86 MSVC (cl) \u2705"},{"location":"#legacy-repo","title":"Legacy Repo","text":"

KvDB

KV-Store

"},{"location":"api/","title":"Module","text":""},{"location":"api/#database-operations","title":"Database Operations","text":""},{"location":"api/#veloxdbopenstring-db_name","title":"VeloxDB::Open(string db_name)","text":"

Initializes the database system, setting up the necessary files and directories (including SSTs and related data). Can be initialized with a custom Memtable size or default size of 1e3.

#include \"VeloxDB/VeloxDB.h\"\n/*\n *  Initialize with default value : \n *       Memtable::size == 1e3\n *       SsTFileManager::DiskBTree::Degree == 3\n */ \nauto MyDBDefault = std::make_unique<VeloxDB>();\nauto MyDBDefault = std::make_unique<VeloxDB>(int memtableSize, int BTreeDegree);\n\nMyDBDefault->Open(\"database_name\");\n
"},{"location":"api/#veloxdbclose","title":"VeloxDB::Close()","text":"

Closes the database, flushing any data in memory (Memtable) to disk and storing it in SSTs.

#include \"VeloxDB/VeloxDB.h\"\n// Close the database and flush the Memtable to disk\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\nMyDB->Close();\n
"},{"location":"api/#data-operations","title":"Data Operations","text":""},{"location":"api/#template-veloxdbputk-key-v-value","title":"Template VeloxDB::Put(K key, V value)

Inserts a key-value pair into the database, where both the key and value can be of various types (int, double, string, etc.).

#include \"VeloxDB/VeloxDB.h\"\n// Example of inserting different data types\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\nMyDB->Put(1, 100);             // int -> int\nMyDB->Put(1.5, 'A');           // double -> char\nMyDB->Put(\"Hello\", 1e8LL);     // string -> long long\n
","text":""},{"location":"api/#veloxdbgetconst-keyvaluewrapper-key","title":"VeloxDB::Get(const KeyValueWrapper& key)

Retrieves a value from the database based on the key. Supports multiple data types.

#include \"VeloxDB/VeloxDB.h\"\n#include \"kv/KeyValue.h\"\n// Example of retrieving values\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\nMyDB->Put(1, 100);\nMyDB->Put(1.5, 'A');\nMyDB->Put(\"Hello\", 1e8LL);\n\n// Retrieve the value by key\nauto result1 = MyDB->Get(\"Hello\");\nlong long value1 = result1.kv.long_value(); // 1e8\nstring key1 = result1.kv.string_key(); // \"Hello\"\n\n// Retrieve the value by `KeyValueWrapper` instance\nauto result1 = MyDB->Get(KeyValueWrapper(\"Hello\", \"\")); \n// Expected result1: { key: \"Hello\", value: 1e8LL }\nlong long value1 = result1.kv.long_value(); // 1e8\nstring key1 = result1.kv.string_key(); // \"Hello\"\n\n// e.g.2\nauto result2 = MyDB->Get(1);\nint value2 = result2.kv.int_value();\nint key2 = results.kv.int_key();\n\n// check if not found using : bool KeyValueWrapper::isEmpty() const;\nif(result.isEmpty()){\n    // Key not found :-(\n} else {\n    // Found :-D\n}\n
","text":""},{"location":"api/#veloxdbscankeyvaluewrapper-smallestkey-keyvaluewrapper-largestkey","title":"VeloxDB::Scan(KeyValueWrapper smallestKey, KeyValueWrapper largestKey)

Scans the database for key-value pairs within a specified key range. The results are returned in sorted key order.

#include \"VeloxDB/VeloxDB.h\"\n// Scan for key-value pairs within a range\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\n// Scan by key\nstd::set<KeyValueWrapper> results = MyDB->Scan(1, 10);\n// Scan by `KeyValueWrapper` instance\nstd::set<KeyValueWrapper> results = MyDB->Scan(KeyValueWrapper(1, \"\"), KeyValueWrapper(10, \"\"));\n
","text":""},{"location":"api/#veloxdbupdatekeyvaluewrapper-keytoupdate","title":"VeloxDB::Update(KeyValueWrapper KeyToUpdate)

(TBA) This will allow the updating of key-value pairs within the database.

","text":""},{"location":"api/#veloxdbdeletekeyvaluewrapper-key","title":"VeloxDB::Delete(KeyValueWrapper Key)

(TBA) This will allow the deletion of key-value pairs from the database.

","text":""},{"location":"api/#buffer-pool-operation","title":"Buffer Pool Operation","text":""},{"location":"api/#setbufferpoolparameterssize_t-capacity-evictionpolicy-policy","title":"setBufferPoolParameters(size_t capacity, EvictionPolicy policy)

Set/reset buffer pool size_t:: capacity and EvictionPolicy:: policy (LRU, CLOCK, RANDOM)

EvictionPolicy newPolicy = EvictionPolicy::LRU;\nEvictionPolicy newPolicy = EvictionPolicy::CLOCK;\nEvictionPolicy newPolicy = EvictionPolicy::RANDOM;\n

This method will clear all the previous cache in the buffer pool.

#include \"VeloxDB/VeloxDB.h\"\n#include \"Memory/BufferPool/BufferPool.h\"\n// Open the database\nauto MyDB = std::make_unique<VeloxDB>();\nMyDB->Open(\"database_name\");\n\n// Set buffer pool parameters\nsize_t Capacity = 1e3;\nEvictionPolicy Policy = EvictionPolicy::CLOCK;\nMyDB->SetBufferPoolParameters(Capacity, Policy);\n\n// Reset \nsize_t newCapacity = 1e4;\nEvictionPolicy newPolicy = EvictionPolicy::LRU;\nMyDB->SetBufferPoolParameters(newCapacity, newPolicy);\n\n// Perform database operations\nMyDB->Put(1, \"value1\");\nKeyValueWrapper value = MyDB->Get(1);\n\n// Close the database\nMyDB->Close();\n
","text":""},{"location":"api/#printcachehit","title":"printCacheHit()

print total number of cache hit in the buffer pool during the database operations.

#include \"VeloxDB/VeloxDB.h\"\n\nint memtableSize = 1e4; \nauto db = std::make_unique<VeloxDB>(memtableSize, 3);\ndb->Open(\"test_db\");\n\nconst int numEntries = 1e6;  // Insert 1e6 key-value pairs\n\n// Insert key-value pairs\nfor (int i = 0; i < numEntries; ++i) {\n    db->Put(i, \"value_\" + std::to_string(i));\n}\n\nstd::set<KeyValueWrapper> resultSet = db->Scan(KeyValueWrapper(1, \"\"), KeyValueWrapper(50000, \"\"));\n\ndb->printCacheHit(); // this will print the total number of cache hit in buffer pool\n\n// Clean up\ndb->Close();\n
","text":""},{"location":"benchmark/","title":"Benchmark","text":""},{"location":"benchmark/#benchmark","title":"Benchmark","text":""},{"location":"benchmark/#veloxdbput","title":"VeloxDB::Put","text":"

Put throughput with different Memtable size

    B Tree Degree = 3\n    page size = 4 kb\n

"},{"location":"benchmark/#veloxdbget","title":"VeloxDB::Get","text":"

Get latency with different Memtable size

    B Tree Degree = 3\n    page size = 4 kb\n

"},{"location":"benchmark/#veloxdbscan","title":"VeloxDB::Scan","text":"

Scan throughput with different Memtable size

    B Tree Degree = 3\n    page size = 4 kb\n

"},{"location":"design/","title":"Database Design","text":""},{"location":"design/#tba","title":"TBA","text":""},{"location":"layout/","title":"Layout","text":""},{"location":"layout/#sst-files-layout","title":"SST Files Layout","text":"
[Internal Node Page (Root)]\n[Internal Node Page 1]\n[Internal Node Page 2]\n...\n[Internal Node Page n]\n[Leaf Node Page 1]\n[Leaf Node Page 2]\n[Leaf Node Page 3]\n...\n[Leaf Node Page m]\n[* Clustered Index Page]\n[* Bloom Filter Page]\n[SST Metadata Page]\n
"},{"location":"layout/#pagepagesize","title":"Page::PageSize","text":"

Page with PageSize:: PageSize (4KB, 8KB)

"},{"location":"layout/#pagesst_metadata","title":"Page::SST_MetaData","text":"
LeafNode_Begin_Offset\nLeafNode_End_offset\nFileName\n
"},{"location":"layout/#pageleafnodes","title":"Page::LeafNodes","text":"
/*\n *  4kb / 8kb chunk\n *  sorted by key\n */\nserialized key-value pair 1 metadata (serialized by protobuf)\nserialized key-value pair 2 metadata (serialized by protobuf)\nserialized key-value pair 3 metadata (serialized by protobuf)\n...\n// with padding\n
"},{"location":"layout/#pageinternalnodes","title":"Page::InternalNodes","text":"
/*\n *  4kb / 8kb chunk\n *  sorted by level\n */\nlevel#0 key-value pair 0 metadata (serialized by protobuf), jump_offset_L1_K0, jump_offset_L1_K1\nlevel#1 key-value pair 1 metadata (serialized by protobuf), jump_offset_L2_K0, jump_offset_L2_K1\nlevel#1 key-value pair 2 metadata (serialized by protobuf), jump_offset_L2_K1, jump_offset_L2_K2\n...\n// with padding\n
"},{"location":"layout/#pagebloomfilter","title":"Page::BloomFilter","text":"

TBD

"},{"location":"layout/#pageclusteredindex","title":"Page::ClusteredIndex","text":"

TBD

"}]} \ No newline at end of file