Skip to content

Latest commit

 

History

History
232 lines (182 loc) · 8.32 KB

DEVELOPER.md

File metadata and controls

232 lines (182 loc) · 8.32 KB
Raw

Developer Guide

This document describes how to set up your development environment to build and test the Valkey GLIDE Python wrapper.

Development Overview

The Valkey GLIDE Python wrapper consists of both Python and Rust code. Rust bindings for Python are implemented using PyO3, and the Python package is built using maturin. The Python and Rust components communicate using the protobuf protocol.

Build from source

Prerequisites

Software Dependencies

  • python3 virtualenv
  • git
  • GCC
  • pkg-config
  • protoc (protobuf compiler) >= v3.20.0
  • openssl
  • openssl-dev
  • rustup

Dependencies installation for Ubuntu

sudo apt update -y
sudo apt install -y python3 python3-venv git gcc pkg-config openssl libssl-dev unzip
# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version
# Install protobuf compiler
PB_REL="https://github.com/protocolbuffers/protobuf/releases"
# For other arch type from x86 example below, the signature of the curl url should be protoc-<version>-<os>-<arch>.zip,
# e.g. protoc-3.20.3-linux-aarch_64.zip for ARM64.
curl -LO $PB_REL/download/v3.20.3/protoc-3.20.3-linux-x86_64.zip
unzip protoc-3.20.3-linux-x86_64.zip -d $HOME/.local
export PATH="$PATH:$HOME/.local/bin"
# Check that the protobuf compiler is installed
protoc --version

Dependencies installation for CentOS

sudo yum update -y
sudo yum install -y python3 git gcc pkgconfig openssl openssl-devel unzip
pip3 install virtualenv
# Install rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version
# Install protobuf compiler
PB_REL="https://github.com/protocolbuffers/protobuf/releases"
curl -LO $PB_REL/download/v3.20.3/protoc-3.20.3-linux-x86_64.zip
unzip protoc-3.20.3-linux-x86_64.zip -d $HOME/.local
export PATH="$PATH:$HOME/.local/bin"
# Check that the protobuf compiler is installed
protoc --version

Dependencies installation for MacOS

brew update
brew install python3 git gcc pkgconfig protobuf@3 openssl virtualenv
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"
# Check that the Rust compiler is installed
rustc --version
# Verify the Protobuf compiler installation
protoc --version

# If protoc is not found or does not work correctly, update the PATH
echo 'export PATH="/opt/homebrew/opt/protobuf@3/bin:$PATH"' >> /Users/$USER/.bash_profile
source /Users/$USER/.bash_profile
protoc --version

Building and installation steps

Before starting this step, make sure you've installed all software requirments.

  1. Clone the repository: 
    git clone https://github.com/valkey-io/valkey-glide.git
cd valkey-glide
  2. Initialize git submodule: 
    git submodule update --init --recursive
  3. Generate protobuf files: 
    GLIDE_ROOT_FOLDER_PATH=.
protoc -Iprotobuf=${GLIDE_ROOT_FOLDER_PATH}/glide-core/src/protobuf/ --python_out=${GLIDE_ROOT_FOLDER_PATH}/python/python/glide ${GLIDE_ROOT_FOLDER_PATH}/glide-core/src/protobuf/*.proto
  4. Create a virtual environment: 
    cd python
python3 -m venv .env
  5. Activate the virtual environment: 
    source .env/bin/activate
  6. Install requirements: 
    pip install -r requirements.txt
  7. Build the Python wrapper in release mode: 
    maturin develop --release --strip

    Note: To build the wrapper binary with debug symbols remove the --strip flag.

  8. Run tests:
    1. Ensure that you have installed redis-server or valkey-server and redis-cli or valkey-cli on your host. You can find the Redis installation guide at the following link: Redis Installation Guide. You can get Valkey from the following link: Valkey Download.
    2. Validate the activation of the virtual environment from step 4 by ensuring its name (.env) is displayed next to your command prompt.
    3. Execute the following command from the python folder: 
      pytest --asyncio-mode=auto

      Note: To run Valkey modules tests, add -k "test_server_modules.py".

  • Install Python development requirements with:

    pip install -r python/dev_requirements.txt

  • For a fast build, execute maturin develop without the release flag. This will perform an unoptimized build, which is suitable for developing tests. Keep in mind that performance is significantly affected in an unoptimized build, so it's required to include the "--release" flag when measuring performance.

Test

To run tests, use the following command:

pytest --asyncio-mode=auto

To execute a specific test, include the -k <test_name> option. For example:

pytest --asyncio-mode=auto -k test_socket_set_and_get

IT suite starts the server for testing - standalone and cluster installation using cluster_manager script. If you want IT to use already started servers, use the following command line from python/python dir:

pytest --asyncio-mode=auto --cluster-endpoints=localhost:7000 --standalone-endpoints=localhost:6379

Submodules

After pulling new changes, ensure that you update the submodules by running the following command:

git submodule update

Generate protobuf files

During the initial build, Python protobuf files were created in python/python/glide/protobuf. If modifications are made to the protobuf definition files (.proto files located in glide-core/src/protofuf), it becomes necessary to regenerate the Python protobuf files. To do so, run:

GLIDE_ROOT_FOLDER_PATH=. # e.g. /home/ubuntu/valkey-glide
protoc -Iprotobuf=${GLIDE_ROOT_FOLDER_PATH}/glide-core/src/protobuf/ --python_out=${GLIDE_ROOT_FOLDER_PATH}/python/python/glide ${GLIDE_ROOT_FOLDER_PATH}/glide-core/src/protobuf/*.proto

Protobuf interface files

To generate the protobuf files with Python Interface files (pyi) for type-checking purposes, ensure you have installed mypy-protobuf with pip, and then execute the following command:

GLIDE_ROOT_FOLDER_PATH=. # e.g. /home/ubuntu/valkey-glide
MYPY_PROTOC_PATH=`which protoc-gen-mypy`
protoc --plugin=protoc-gen-mypy=${MYPY_PROTOC_PATH} -Iprotobuf=${GLIDE_ROOT_FOLDER_PATH}/glide-core/src/protobuf/ --python_out=${GLIDE_ROOT_FOLDER_PATH}/python/python/glide --mypy_out=${GLIDE_ROOT_FOLDER_PATH}/python/python/glide ${GLIDE_ROOT_FOLDER_PATH}/glide-core/src/protobuf/*.proto

Linters

Development on the Python wrapper may involve changes in either the Python or Rust code. Each language has distinct linter tests that must be passed before committing changes.

Language-specific Linters

Python:

  • flake8
  • isort
  • black
  • mypy

Rust:

  • clippy
  • fmt

Running the linters

Run from the main /python folder

  1. Python

    Note: make sure to generate protobuf with interface files before running mypy linter

    pip install -r dev_requirements.txt
isort . --profile black --skip-glob python/glide/protobuf --skip-glob .env
black . --exclude python/glide/protobuf --exclude .env
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics --exclude=python/glide/protobuf,.env/* --extend-ignore=E230
flake8 . --count --exit-zero --max-complexity=12 --max-line-length=127 --statistics --exclude=python/glide/protobuf,.env/* --extend-ignore=E230
# run type check
mypy .

  2. Rust

    rustup component add clippy rustfmt
cargo clippy --all-features --all-targets -- -D warnings
cargo fmt --manifest-path ./Cargo.toml --all

Recommended extensions for VS Code