diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml index e5744d83a6..468fa7c32f 100644 --- a/.github/workflows/rust.yml +++ b/.github/workflows/rust.yml @@ -24,8 +24,8 @@ jobs: fail-fast: false matrix: redis: - - 6.2.4 - - 7.0.0 + - 6.2.13 + - 7.0.12 steps: - uses: actions/checkout@v3 diff --git a/README.md b/README.md index 7ff6c1852b..6032419f5e 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,40 @@ # babushka -## developer pre-requirements: +Babushka (temporary name, final name pending) is a collection of open source Redis clients in various managed languages, based around shared logic written in Rust. We call the clients "wrappers", and the shared logic "core". - - GCC - - pkg-config - - protobuf-compiler (protoc) - - openssl - - libssl-dev +## Supported languages + +- [node](./node/README.md). +- [python](./python/README.md) + +## Folder structure + +- babushka-core - [the Rust core](./babushka-core/README.md). This code is compiled into binaries which are used by the various wrappers. +- benchmarks - [benchmarks for the wrappers](./benchmarks/README.md), and for the current leading Redis client in each language. +- csharp - [the C# wrapper](./csharp/README.md). currently not in a usable state. +- logger_core - [the logger we use](./logger_core/). +- node - [the NodeJS wrapper](./node/README.md). +- python - [the Python3 wrapper](./python/README.md) +- submodules - git submodules. +- utils - Utility scripts used in testing, building, and deploying. + +## development pre-requirements + +- GCC +- pkg-config +- protobuf-compiler (protoc) +- openssl +- libssl-dev +- python3 Installation for ubuntu: -`sudo apt install -y gcc pkg-config protobuf-compiler openssl libssl-dev` +`sudo apt install -y gcc pkg-config protobuf-compiler openssl libssl-dev python3` -### git submodule - +### git submodule run `git submodule update --init --recursive` on first usage, and on any update to the redis-rs fork. -### rustup - +### rustup https://rustup.rs/ @@ -27,6 +46,8 @@ after the instalation will show-up in the terminal steps to add rustup to the pa ### redis-server +This is required for running tests and local benchmarks. + https://redis.io/docs/getting-started/ ``` @@ -42,56 +63,19 @@ sudo mkdir /etc/redis sudo cp src/redis-server src/redis-cli /usr/local/bin ``` -change {0} to the version you want, e.g. 7.0.7. version names are available here: http://download.redis.io/releases/ -recommended version - 7.0.7 ATM +change {0} to the version you want, e.g. 7.0.12. version names are available here: http://download.redis.io/releases/ +recommended version - 7.0.12 ATM ### node 16 (or newer) +This is required for the NodeJS wrapper, and for running benchmarks. + ``` curl -s https://deb.nodesource.com/setup_16.x | sudo bash apt-get install nodejs npm npm i -g npm@8 ``` -## benchmarks: - -`benchmarks/install_and_test.sh` is the benchmark script we use to check performance. run `install_and_test.sh -h` to get the full list of available flags. - -If while running benchmarks your redis-server is killed every time the program runs the 4000 data-size benchmark, it might be because you don't have enough available storage on your machine. -To solve this issue, you have two options - - -1. Allocate more storage to your'e machine. for me the case was allocating from 500 gb to 1000 gb. -2. Go to benchmarks/install_and_test.sh and change the "dataSize="100 4000"" to a data-size that your machine can handle. try for example dataSize="100 1000". - -## Additional packages - -For python, run `sudo apt install python3.10-venv` (with the relevant python version), in order to be able to create a virtual environment. -In order to run rust-analyzer through remote VScode, you need to run `sudo apt install pkg-config libssl-dev`. - ## Recommended VSCode extensions -### General - [GitLens](https://marketplace.visualstudio.com/items?itemName=eamodio.gitlens) - -### Rust development - -[rust-analyzer](https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer) - Rust language server. -[CodeLLDB](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) - Debugger. -[Even Better TOML](https://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml) - TOML language support. - -### C# development - -[C#](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) - lightweight language server and in-editor test runner. - -### TypeScript development - -[Prettier - Code formatter](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) -JavaScript / TypeScript formatter. -[Jest Runner](https://marketplace.visualstudio.com/items?itemName=firsttris.vscode-jest-runner) - in-editor test runner. -[Jest Test Explorer](https://marketplace.visualstudio.com/items?itemName=kavod-io.vscode-jest-test-adapter) - adapter to the VSCode testing UI. -[ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) - linter. - -### Python development - -[Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) - Language server -[Black Formetter](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter) - Formatter diff --git a/babushka-core/README.md b/babushka-core/README.md new file mode 100644 index 0000000000..395377f908 --- /dev/null +++ b/babushka-core/README.md @@ -0,0 +1,7 @@ +# Rust core + +## Recommended VSCode extensions + +[rust-analyzer](https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer) - Rust language server. +[CodeLLDB](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) - Debugger. +[Even Better TOML](https://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml) - TOML language support. diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000000..67d7cdc11c --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,11 @@ +# Benchmarks + +[`install_and_test.sh`](./install_and_test.sh) is the benchmark script we use to check performance. run `install_and_test.sh -h` to get the full list of available flags. + +The results of the benchmark runs will be written into .csv files in the `./results` folder. + +If while running benchmarks your redis-server is killed every time the program runs the 4000 data-size benchmark, it might be because you don't have enough available storage on your machine. +To solve this issue, you have two options - + +1. Allocate more storage to your'e machine. for me the case was allocating from 500 gb to 1000 gb. +2. Go to benchmarks/install_and_test.sh and change the "dataSize="100 4000"" to a data-size that your machine can handle. try for example dataSize="100 1000". diff --git a/csharp/README.md b/csharp/README.md new file mode 100644 index 0000000000..1d3b34df64 --- /dev/null +++ b/csharp/README.md @@ -0,0 +1,7 @@ +# C# wrapper + +The C# wrapper is currently not in a usable state. + +## Recommended VSCode extensions + +[C#](https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.csharp) - lightweight language server and in-editor test runner. diff --git a/node/README.md b/node/README.md index 90ce83f722..a3aee4908d 100644 --- a/node/README.md +++ b/node/README.md @@ -1,23 +1,50 @@ -This package contains an internal package under the folder `rust-client`, which is just the Rust library and its auto-generated TS code. The external package, in this folder, contains all wrapping TS code and tests. +# NodeJS wrapper -### First time install +This package contains an internal package under the folder `rust-client`, which is just the Rust library and its auto-generated TypeSCript code. The external package, in this folder, contains all wrapping TypeSCript code and tests. + +## Building + +### dependency installation Homebrew is the easiest way to install node, but you can choose any way you want to install it. Npm should come as part as node. IMPORTANT - right now we support only npm major version 8. The default install will be of version 9, so you'll need to downgrade - run `npm i -g npm@8`. -### Build +### Adding the babushka package to your package + +Before you add the babushka package to your application, make sure to perform the stages in "[Building the wrapper](#building-the-wrapper)". +ATM Babushka isn't on npm, so you'll need to clone this repo to you device, and add the package using a folder path - `npm install /node`. + +### Building the wrapper + +Before you start, make sure that you have all dependencies mentioned [here](../README.md#development-pre-requirements), [rustup](../README.md#rustup), [node](../README.md#node-16-or-newer) installed, and all [submodules updated](../README.md#git-submodule). +Enter this folder in your terminal. +On the first install, run `npm i`, to install all dependencies. Also enter the `rust-client` folder, and run `npm i` there, too. -Run `npm run build-internal` to build the internal package and generates TS code. `npm run build-external` builds the external package without rebuilding the internal package. Run `npm run build-protobuf` to generate node's protobuf files. -Run `npm run build` runs a full build. +make sure that all dependencies, including rustup, have been installed. +Run `npm run build:release` runs a full build in release mode. This might take a while. + +For testing purposes, you can run `npm run build` runs a full, unoptimized build. +If you've only made changes to the Rust code, run `npm run build-internal` to build the internal package and generates TypeSCript code. +If you've only made changes to the TypeScript code, run `npm run build-external` to build the external package without rebuilding the internal package. +Run `npm run build-protobuf` to generate node's protobuf files. + +Once building completed, you'll find the compiled JavaScript code in the ./build-ts folder, ### Testing Run `npm test` after building. -### [Optional] Manually compile protobuf files +### [Optional] Manually compile protobuf files 1. Run `npm install protobufjs-cli` to install the JS protobuf command line utility 2. Generate static JS code `pbjs -t static-module -o ProtobufMessage.js ~/babushka/babushka-core/src/protobuf/*.proto` -3. Generates TS definitions from the compiled JS file `pbts -o ProtobufMessage.d.ts ProtobufMessage.js` +3. Generates TypeSCript definitions from the compiled JS file `pbts -o ProtobufMessage.d.ts ProtobufMessage.js` 4. Due to a bug in the autogenerated files (see https://github.com/protobufjs/protobuf.js/issues/1063), manually replace `return this.encode(message, writer).ldelim();` to `return this.encode(message, writer && writer.len ? writer.fork() : writer).ldelim();` in the encodeDelimited functions of Request and Response under babushka/node/src/ProtobufMessage.js. + +## Recommended VSCode extensions + +[Prettier - Code formatter](https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode) -JavaScript / TypeScript formatter. +[Jest Runner](https://marketplace.visualstudio.com/items?itemName=firsttris.vscode-jest-runner) - in-editor test runner. +[Jest Test Explorer](https://marketplace.visualstudio.com/items?itemName=kavod-io.vscode-jest-test-adapter) - adapter to the VSCode testing UI. +[ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) - linter. diff --git a/python/README.md b/python/README.md index a28fb3ea1e..e278e0bf0d 100644 --- a/python/README.md +++ b/python/README.md @@ -1,7 +1,11 @@ +# Python Wrapper + Using [Pyo3](https://github.com/PyO3/pyo3) and [Maturin](https://github.com/PyO3/maturin). ### Create venv +Install python-venv: run `sudo apt install python3.10-venv` (with the relevant python version), in order to be able to create a virtual environment. + `cd python` cd into babushka/python folder `python -m venv .env` in order to create a new virtual env. @@ -36,14 +40,14 @@ Autogenerate python's protobuf files with: Run `pytest --asyncio-mode=auto` from this folder, or from the `tests` folder. Make sure your shell uses your virtual environment. -#### Running linters: +#### Running linters Using VS code, install the following extensions: -- Python -- isort -- Black Formatter -- Flake8 +- [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python) +- [isort](https://marketplace.visualstudio.com/items?itemName=ms-python.isort) +- [Black Formetter](https://marketplace.visualstudio.com/items?itemName=ms-python.black-formatter) +- [Flake8](https://marketplace.visualstudio.com/items?itemName=ms-python.flake8) Or, run in a terminal: