Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ahnlich python client #34

Merged
merged 101 commits into from
Jun 21, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
101 commits
Select commit Hold shift + click to select a range
81bb9df
Project init
Iamdavidonuh Jun 11, 2024
986afd7
WIP: trying to serialize a type
Iamdavidonuh Jun 12, 2024
b61cda0
WIP: Figuring out serialization
deven96 Jun 12, 2024
a97d191
Merge remote-tracking branch 'origin/david/types-gen' into david/pyth…
deven96 Jun 12, 2024
b18727e
WIP: Talking to Database
deven96 Jun 12, 2024
5bbbac5
Merge branch 'main' into david/python-client
Iamdavidonuh Jun 13, 2024
57798c7
refactor client folder names
Iamdavidonuh Jun 13, 2024
f0b1d8c
Refactor protocol and fix poetry dir issue
Iamdavidonuh Jun 13, 2024
a2dcc3a
check if server is disconnected
Iamdavidonuh Jun 15, 2024
3501557
WIP: create client wrapper for ahnlich db
Iamdavidonuh Jun 16, 2024
ea3ad51
Parse version from pyproject.toml file
Iamdavidonuh Jun 16, 2024
90eae7e
Merge branch 'main' into david/python-client
Iamdavidonuh Jun 16, 2024
c42ee40
fix old dep
Iamdavidonuh Jun 16, 2024
ec1ac67
add tests for unit commands(ping, server_info, list_clients)
Iamdavidonuh Jun 16, 2024
470c011
add a builder for executing mulitple or single requests
Iamdavidonuh Jun 16, 2024
e3d9780
move builder init into dbclient class
Iamdavidonuh Jun 16, 2024
8ac35e2
add a client builder to create all queries
Iamdavidonuh Jun 16, 2024
d22ae7d
move builder to internals
Iamdavidonuh Jun 16, 2024
838ecf0
setting up job to test python client
Iamdavidonuh Jun 17, 2024
07787d0
cleaning out test_serialize
Iamdavidonuh Jun 17, 2024
ae8a822
run ahnlich db in the background for tests
Iamdavidonuh Jun 17, 2024
63f981f
Run python job if build succeds
Iamdavidonuh Jun 17, 2024
cfd4ef8
fix job error in workflow
Iamdavidonuh Jun 17, 2024
1a8a1e9
job filtering fix
Iamdavidonuh Jun 17, 2024
4bd0f24
remove filtering
Iamdavidonuh Jun 17, 2024
d885037
add working dir
Iamdavidonuh Jun 17, 2024
01a643c
add working dir
Iamdavidonuh Jun 17, 2024
370df01
debug: output dir
Iamdavidonuh Jun 17, 2024
197f4e6
debug: output dir
Iamdavidonuh Jun 17, 2024
b2877c4
debug: output dir
Iamdavidonuh Jun 17, 2024
7e18b18
debug: print out host and port from env
Iamdavidonuh Jun 17, 2024
a940bee
sleep to allow ahnlich db startup
Iamdavidonuh Jun 17, 2024
da3e820
increase sleep time
Iamdavidonuh Jun 17, 2024
18491dd
merge both ahnlichdb startup and client test
Iamdavidonuh Jun 17, 2024
5232943
Add a wait for it to give ahnlich db time to startup
Iamdavidonuh Jun 17, 2024
289561b
parse out port in nc commands
Iamdavidonuh Jun 17, 2024
d7639e7
comment out exec cmd
Iamdavidonuh Jun 17, 2024
124e970
fix kill server step
Iamdavidonuh Jun 17, 2024
6d5394b
update actions versions
Iamdavidonuh Jun 17, 2024
e9fc8bf
Refactor builders and add ability to spin up ahnlichdb or a random port
Iamdavidonuh Jun 18, 2024
aadf718
format
Iamdavidonuh Jun 18, 2024
acbf378
add return annotations to commands
Iamdavidonuh Jun 18, 2024
3c437d5
add test for create stores and allow modular db spin up
Iamdavidonuh Jun 18, 2024
582b2c4
fix test on create store
Iamdavidonuh Jun 18, 2024
8fbb9fc
add tests for python client
Iamdavidonuh Jun 19, 2024
8f3ad58
add custom exception
Iamdavidonuh Jun 19, 2024
84d214d
add command to bump types crate version and python client
Iamdavidonuh Jun 19, 2024
2600aef
Add some info to main readme
Iamdavidonuh Jun 19, 2024
55587e8
change workflow name
Iamdavidonuh Jun 19, 2024
9ae73b9
fix typo and improve test db server provisioning
Iamdavidonuh Jun 19, 2024
25a7f34
Setup bump2version to update client versions
Iamdavidonuh Jun 20, 2024
072a814
Add makefile command to bumppython version
Iamdavidonuh Jun 20, 2024
85bf2c4
Add client version bump to readme
Iamdavidonuh Jun 20, 2024
059685c
Readme changelog
Iamdavidonuh Jun 20, 2024
fe9dd1d
WIP: workflow for python client deployments
Iamdavidonuh Jun 20, 2024
327dcd6
allow all branches for now
Iamdavidonuh Jun 20, 2024
957de73
comment out condition for pipeline to run
Iamdavidonuh Jun 20, 2024
5f4e7df
temporary set conditions to tag to true to test taging step
Iamdavidonuh Jun 20, 2024
27b0e56
remove deprecated set_version, fix tagging step failing
Iamdavidonuh Jun 20, 2024
780bdc6
change output name and specify checkout depth
Iamdavidonuh Jun 20, 2024
e403cf7
dump github context
Iamdavidonuh Jun 20, 2024
25402e9
dump github context
Iamdavidonuh Jun 20, 2024
956f03d
triggering actor is used to create tag
Iamdavidonuh Jun 20, 2024
611c8ea
Read tag message from file
Iamdavidonuh Jun 20, 2024
900f90b
use a deploy token
Iamdavidonuh Jun 20, 2024
fc29be1
use ssh socks
Iamdavidonuh Jun 20, 2024
0979f3f
change tag format
Iamdavidonuh Jun 20, 2024
b02ed03
remove deploykey
Iamdavidonuh Jun 20, 2024
8dff67b
WIP: work on deploying tagged repo
Iamdavidonuh Jun 20, 2024
a105db4
Fix error with env names
Iamdavidonuh Jun 20, 2024
9a5c17c
fix variable assignment
Iamdavidonuh Jun 20, 2024
ad5d0ca
fix variable assignment
Iamdavidonuh Jun 20, 2024
6cfc977
fix variable assignment
Iamdavidonuh Jun 20, 2024
d10117d
correct order of message taging
Iamdavidonuh Jun 20, 2024
12e16cc
change logic for packaging client
Iamdavidonuh Jun 20, 2024
21640f0
setup poetry in pipeline
Iamdavidonuh Jun 20, 2024
f2d90ce
cleanup tag names and uncomment poetry deploy
Iamdavidonuh Jun 20, 2024
e823892
sed escape version
Iamdavidonuh Jun 20, 2024
41c0549
add fetch depth for sed to work and remove duplicate sed command
Iamdavidonuh Jun 20, 2024
2b1cd45
increase fetch depth
Iamdavidonuh Jun 20, 2024
db832c7
use a default tag for now
Iamdavidonuh Jun 20, 2024
2f44f9b
still on sed
Iamdavidonuh Jun 20, 2024
8ba3b17
comment out sed command
Iamdavidonuh Jun 20, 2024
04631ea
clean up, no more sed for now
Iamdavidonuh Jun 20, 2024
6996546
add logic to run pipeline and allow only main
Iamdavidonuh Jun 20, 2024
4c60a4b
test deployment to test pypi
Iamdavidonuh Jun 20, 2024
5d6f926
use python 3.11
Iamdavidonuh Jun 20, 2024
7b6453c
add conditions for deploying tags
Iamdavidonuh Jun 20, 2024
fe95d2c
Refactor, reorganize folder and fix imports
Iamdavidonuh Jun 20, 2024
13944b8
create a subdir for create-client output type generation
Iamdavidonuh Jun 20, 2024
20d6260
Rename build for db to reflect in class name
Iamdavidonuh Jun 20, 2024
c5efd0c
Refactor: move generated files to internals
Iamdavidonuh Jun 20, 2024
eaeffbf
refactor files and fix imports
Iamdavidonuh Jun 21, 2024
3fae177
change name from get_predicate to get_by_predicate
Iamdavidonuh Jun 21, 2024
0caa10b
WIP: Add readme to serve as docs
Iamdavidonuh Jun 21, 2024
c01f2a7
create a sub output dir for only python
Iamdavidonuh Jun 21, 2024
4510e90
Add test for bulk request and update readme for that
Iamdavidonuh Jun 21, 2024
a93f948
Rename client to ahnlich_protocol as it's appropriate
Iamdavidonuh Jun 21, 2024
e5943cc
Format with black and isort
Iamdavidonuh Jun 21, 2024
7505f5a
Create protocol from dbclient, add a request pipeline to client to bu…
Iamdavidonuh Jun 21, 2024
78510d6
Change makefile commands name, and cleanup
Iamdavidonuh Jun 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ahnlich/db/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.1.0"
edition = "2021"

[[bin]]
name = "anlich-db"
name = "ahnlich-db"
path = "src/main.rs"

[lib]
Expand Down
2 changes: 2 additions & 0 deletions ahnlich/db/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -228,10 +228,12 @@ impl ServerTask {
self.reader.read_exact(&mut data).await?;
// TODO: Add trace here to catch whenever queries could not be deserialized at all
if let Ok(queries) = ServerQuery::deserialize(&data) {
tracing::debug!("Got Queries {:?}", queries);
// TODO: Pass in store_handler and use to respond to queries
let results = self.handle(queries.into_inner());
if let Ok(binary_results) = results.serialize() {
self.reader.get_mut().write_all(&binary_results).await?;
tracing::debug!("Sent Response of length {}, {:?}", binary_results.len(), binary_results);
}
} else {
tracing::error!("{}", self.prefix_log("Could not deserialize client message as server query"));
Expand Down
3 changes: 0 additions & 3 deletions ahnlich/typegen/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ fn main() -> Result<(), Box<dyn Error>> {
{
panic!("Cannot generate type specs into invalid directory")
}

let output_dir = &config
.output_dir
.to_owned()
Expand Down Expand Up @@ -61,8 +60,6 @@ fn main() -> Result<(), Box<dyn Error>> {
.to_owned()
.unwrap_or(std::path::PathBuf::from(SDK_PATH));

std::fs::create_dir_all(output_dir).expect("Failed to create sdk directory");
deven96 marked this conversation as resolved.
Show resolved Hide resolved

generate_language_definition(config.language, input_dir, output_dir);

println!("Language type definition generated");
Expand Down
2 changes: 1 addition & 1 deletion ahnlich/typegen/src/tracers/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ struct OutputFile<'a> {
impl<'a> OutputFile<'a> {
fn generate(&self, config: &CodeGeneratorConfig, registry: &Registry) {
let extension: &str = (&self.language).into();
let output_dir = self.output_dir.join(extension);
let output_dir = self.output_dir.join(format!("ahnlich-client-{extension}"));
let _ = std::fs::create_dir_all(&output_dir);
let output_file = output_dir.join(format!("{}.{extension}", self.output_file));

Expand Down
13 changes: 12 additions & 1 deletion ahnlich/typegen/src/tracers/query.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use types::similarity::Algorithm;
use types::{
keyval::{StoreKey, StoreName},
metadata::{MetadataKey, MetadataValue},
query::Query,
query::{Query, ServerQuery},
};

pub fn trace_query_enum() -> Registry {
Expand Down Expand Up @@ -78,6 +78,8 @@ pub fn trace_query_enum() -> Registry {
condition: test_predicate_condition.clone(),
};

let server_query = ServerQuery::from_queries(&[deletepred_variant.clone(), set_query.clone()]);

let _ = tracer
.trace_value(&mut samples, &create_store)
.expect("Error tracing the variant");
Expand All @@ -100,6 +102,10 @@ pub fn trace_query_enum() -> Registry {
.trace_value(&mut samples, &deletepred_variant)
.expect("Error tracing the deletepred variant");

let _ = tracer
.trace_value(&mut samples, &server_query)
.expect("Error tracing the server_query");

// trace enums to fix missing variants error
//
// Also trace each enum type separately to fix any `MissingVariants` error.
Expand All @@ -121,6 +127,11 @@ pub fn trace_query_enum() -> Registry {
.inspect_err(|err| println!("Failed to parse type {}", err.explanation()))
.unwrap();

let _ = tracer
.trace_type::<ServerQuery>(&samples)
.inspect_err(|err| println!("Failed to parse type {}", err.explanation()))
.unwrap();

tracer
.registry()
.expect("Failed to create registry for query")
Expand Down
12 changes: 11 additions & 1 deletion ahnlich/typegen/src/tracers/server_response.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use types::similarity::Similarity;
use types::{
keyval::{StoreKey, StoreName},
metadata::{MetadataKey, MetadataValue},
server::{ConnectedClient, ServerInfo, ServerResponse, ServerType, StoreInfo, StoreUpsert},
server::{ConnectedClient, ServerInfo, ServerResponse, ServerResult, ServerType, StoreInfo, StoreUpsert},
version::Version,
};

Expand Down Expand Up @@ -100,6 +100,16 @@ pub fn trace_server_response_enum() -> Registry {
.inspect_err(|err| println!("Failed to parse type {}", err.explanation()))
.unwrap();

let _ = tracer
.trace_type::<Result<ServerResponse, String>>(&samples)
.inspect_err(|err| println!("Failed to parse type {}", err.explanation()))
.unwrap();

let _ = tracer
.trace_type::<ServerResult>(&samples)
.inspect_err(|err| println!("Failed to parse type {}", err.explanation()))
.unwrap();

tracer
.registry()
.expect("Failed to create registry for server response")
Expand Down
216 changes: 216 additions & 0 deletions sdk/ahnlich-client-py/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
# Created by https://www.toptal.com/developers/gitignore/api/python,visualstudiocode,vim
# Edit at https://www.toptal.com/developers/gitignore?templates=python,visualstudiocode,vim

### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
.python-version
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

### Python Patch ###
# Poetry local configuration file - https://python-poetry.org/docs/configuration/#local-configuration
poetry.toml

# ruff
.ruff_cache/

# LSP config files
pyrightconfig.json

### Vim ###
# Swap
[._]*.s[a-v][a-z]
!*.svg # comment out if you don't need vector files
[._]*.sw[a-p]
[._]s[a-rt-v][a-z]
[._]ss[a-gi-z]
[._]sw[a-p]

# Session
Session.vim
Sessionx.vim

# Temporary
.netrwhist
*~
# Auto-generated tag files
tags
# Persistent undo
[._]*.un~

### VisualStudioCode ###
.vscode/*
!.vscode/settings.json
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json
!.vscode/*.code-snippets

# Local History for Visual Studio Code
.history/

# Built Visual Studio Code Extensions
*.vsix

### VisualStudioCode Patch ###
# Ignore all local history of files
.history
.ionide

# End of https://www.toptal.com/developers/gitignore/api/python,visualstudiocode,vim
3 changes: 3 additions & 0 deletions sdk/ahnlich-client-py/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## Ahnlich Client PY

A Python client that interacts with both ahnlich DB and AI
Empty file.
Loading