Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: initial version of backtest tool #282

Merged
merged 54 commits into from
Sep 14, 2023
Merged

feat: initial version of backtest tool #282

merged 54 commits into from
Sep 14, 2023

Conversation

ab93
Copy link
Member

@ab93 ab93 commented Sep 11, 2023 •
edited
Loading

Backtesting tool currently with the capability of:

  • Running univariate analysis on Prometheus data
  • Train models
  • Generate scores
  • Save plots
  • CLI tool
  • Prometheus data client now uses Threadpool for a 60% speedup on 8 days of data

ab93 and others added 30 commits August 28, 2023 20:32
@ab93
fix: tensor dimension swap instead of view change
61a5859 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
revert transformer changes
de813aa 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
rename udf module
d6f7a63 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: typo
ef738b5 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
add entities
e25c755 
Signed-off-by: Avik Basu <avik_basu@intuit.com>
@ab93
add inference udf
34018aa 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
typo fix
a8fce7f 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
add tests for inference
fbba76e 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
add stream conf (#251)
41d3458 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@s0nicboOm @ab93
feat: add preproc changes (#250)
a860b75 
Preprocess udf

---------
Signed-off-by: Kushal Batra <34571348+s0nicboOm@users.noreply.github.com>
@ab93
feat: add trainer udf (#257)
d3b1fbd 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@s0nicboOm @ab93
feat: add postproc and threshold as PostProcessUDF (#256)
198da3b 
Add PostProc

---------

Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>
@ab93
feat: add dockerfile (#258)
9b90116 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@s0nicboOm @ab93
fix: get config from constructor (#259)
7b2baa4 
Get config from constructor for Preproc and Postproc

Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>
Signed-off-by: Kushal Batra <34571348+s0nicboOm@users.noreply.github.com>
@ab93
fix: trainer conf (#260)
a5feecc 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
feat: load config from conf map (#268)
ab9288b 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: trainer udf tests
b17af23 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@s0nicboOm
fix: add publisher (#272)
181a2c0 
1. Add publisher
2. Add test cases for preproc and postproc

Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>
@ab93
fix test errors
dbb834c 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix lints
d056a02 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: ci errors
fb6915d 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: postprocess perf
6b62679 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: logging
f4ff3ac 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@s0nicboOm
feat: add docker image release workflow (#274)
62364c9 
Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>
@ab93
feat: log from file conf
9914b4c 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
feat: backtest tool
2529e96 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: add connector test cases
f20e075 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@s0nicboOm @ab93
fix: log config (#276)
f187125 
Signed-off-by: s0nicboOm <i.kushalbatra@gmail.com>
@ab93
chore: address comments
b01f874 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
feat: univariate cli
b87ee9f 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: redis type
89c6462 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
add druid dependency
7e53376 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix druid dependency
dfdd0d1 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
use factory
54bd305 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
remove set logger
9eab07f 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix tests
0b5cbb0 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
Merge branch 'main' into backtest-tool
8bc4218 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@codecov
Copy link

codecov bot commented Sep 11, 2023 •
edited
Loading

Codecov Report

Merging #282 (21ff828) into main (0cdc257) will decrease coverage by 0.96%.
The diff coverage is 86.73%.

@@            Coverage Diff             @@
##             main     #282      +/-   ##
==========================================
- Coverage   93.36%   92.41%   -0.96%     
==========================================
  Files          66       74       +8     
  Lines        2911     3349     +438     
  Branches      234      267      +33     
==========================================
+ Hits         2718     3095     +377     
- Misses        154      201      +47     
- Partials       39       53      +14
Files Changed Coverage Δ
numalogic/tools/types.py 92.85% <0.00%> (ø)
numalogic/udfs/__main__.py 0.00% <0.00%> (ø)
numalogic/backtest/__init__.py 60.00% <60.00%> (ø)
numalogic/connectors/__init__.py 80.00% <75.00%> (-20.00%) ⬇️
numalogic/config/factory.py 87.67% <76.47%> (-10.25%) ⬇️
numalogic/connectors/_base.py 77.27% <77.27%> (ø)
numalogic/backtest/__main__.py 78.57% <78.57%> (ø)
numalogic/connectors/druid.py 86.36% <85.71%> (+1.74%) ⬆️
numalogic/connectors/prometheus.py 88.69% <88.69%> (ø)
numalogic/backtest/_prom.py 89.36% <89.36%> (ø)
... and 11 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@ab93
add tests for prom fetcher
9ab7b9e 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: packaging as a non optional script
06edd6c 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93 ab93 self-assigned this Sep 12, 2023
@ab93 ab93 added the enhancement New feature or request label Sep 12, 2023
@ab93
chore: add tests
257ce0a 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
docs: add docstrings for backtester
008a3dc 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93 ab93 marked this pull request as ready for review September 12, 2023 23:03
@ab93 ab93 requested a review from vigith September 12, 2023 23:03
@ab93 ab93 changed the title feat: Backtest tool feat: initial version of backtest tool Sep 12, 2023
s0nicboOm
s0nicboOm approved these changes Sep 13, 2023
View reviewed changes
Copy link
Contributor

@s0nicboOm s0nicboOm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments

Dockerfile Outdated Show resolved Hide resolved
numalogic/connectors/prometheus.py Show resolved Hide resolved
pyproject.toml Show resolved Hide resolved
@ab93
chore: update docstrings
5d3caab 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
Merge branch 'main' into backtest-tool
e973276
@ab93
remove poetry lock from precommit
b3cd337 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix: druid opt dependency issue
8d674f9 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix import
fc54688 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
fix lint
4f038a3 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93
tmp: limit mlflow version to 2.6
21ff828 
Signed-off-by: Avik Basu <ab93@users.noreply.github.com>
@ab93 ab93 merged commit a364721 into main Sep 14, 2023
10 checks passed
@ab93 ab93 deleted the backtest-tool branch September 14, 2023 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@s0nicboOm s0nicboOm s0nicboOm approved these changes

@qhuai qhuai Awaiting requested review from qhuai qhuai is a code owner

@nkoppisetty nkoppisetty Awaiting requested review from nkoppisetty nkoppisetty is a code owner

@vigith vigith Awaiting requested review from vigith

Assignees

@ab93 ab93

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ab93 @s0nicboOm