Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI/CD overhaul #403

Merged
merged 141 commits into from
Jul 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
1b5766e
moved dependencies to requirements subdir
jtisbell4 Jan 18, 2024
42516b0
added delta helper func to configure spark session
jtisbell4 Jan 18, 2024
5133f9d
cleaned up comments
jtisbell4 Jan 18, 2024
002040f
added DBR 12.2 and 14.2
jtisbell4 Jan 19, 2024
e89df24
changed conditional to use isinstance
jtisbell4 Jan 19, 2024
fdaf6ec
added DBR 11.3 to tox.ini
jtisbell4 Jan 19, 2024
4812caf
added DBR 10.4 to tox.ini
jtisbell4 Jan 20, 2024
9b5b650
set ingore_metadata=True
jtisbell4 Jan 21, 2024
2a5c33d
simplified tox.ini and moved doc requirements to docs dir
jtisbell4 Jan 23, 2024
14d29ec
added some TODOs
jtisbell4 Jan 24, 2024
7e5a5de
added DBR 9.1
jtisbell4 Jan 24, 2024
cf1abe0
updated CONTRIBUTING.md
jtisbell4 Jan 25, 2024
f3297a9
fixed test to check for appropriate delta-spark version
jtisbell4 Jan 27, 2024
16d29ab
removed useDeltaOpt conditional in io.write
jtisbell4 Jan 27, 2024
fc0b357
formatting
jtisbell4 Jan 27, 2024
e11fbb2
linting
jtisbell4 Jan 27, 2024
c557fc1
removed event_time assumption and made zorder contingent on optimizat…
jtisbell4 Feb 22, 2024
e316127
moved requirements and removed DBR 7 testing
jtisbell4 Mar 4, 2024
d7fb52c
fixed tox.ini and added pyenv to test.yml
jtisbell4 Apr 16, 2024
506d088
manual trigger
jtisbell4 Apr 16, 2024
d465f3a
readded push trigger
jtisbell4 Apr 18, 2024
941855f
split into two jobs
jtisbell4 Apr 18, 2024
51399c3
typo
jtisbell4 Apr 18, 2024
c0a908e
modified pyenv install cmd
jtisbell4 Apr 18, 2024
4a465cc
restart shell cmd
jtisbell4 Apr 18, 2024
fedabd3
added cmds to .profile
jtisbell4 Apr 18, 2024
e752015
.bash_profile
jtisbell4 Apr 18, 2024
e1f606f
combined steps
jtisbell4 Apr 18, 2024
33d8b02
added print statement
jtisbell4 Apr 18, 2024
7865d37
removed shell restart
jtisbell4 Apr 18, 2024
545065f
pulled pyenv action from mlflow
jtisbell4 Apr 18, 2024
10ff339
renamed job
jtisbell4 Apr 18, 2024
15d0c02
forgot installation cmd
jtisbell4 Apr 18, 2024
fe89957
moved python installation into matrix
jtisbell4 Apr 18, 2024
b513b85
tox type checking now works
jtisbell4 Apr 18, 2024
de56c14
ignore imports for mypy
jtisbell4 Apr 18, 2024
5eb5ac7
added pyenv local cmd
jtisbell4 Apr 18, 2024
fab1343
trying gabrielfalcao/pyenv-action@v18
jtisbell4 Apr 18, 2024
2358319
removed default
jtisbell4 Apr 18, 2024
35d47bd
trying setup-python action
jtisbell4 Apr 18, 2024
9eab483
changed sequence to strings
jtisbell4 Apr 18, 2024
30d4436
removed pyenv cmd
jtisbell4 Apr 18, 2024
5e2554a
only trying two env
jtisbell4 Apr 18, 2024
f29e5b7
tox-gh-actions plugin
jtisbell4 Apr 18, 2024
124f93a
fetch tags true
jtisbell4 Apr 18, 2024
36f633c
testing git
jtisbell4 Apr 18, 2024
acba106
testing git describe
jtisbell4 Apr 18, 2024
d5e60cb
fetch-depth = 0
jtisbell4 Apr 18, 2024
13efc50
uncomment
jtisbell4 Apr 18, 2024
8ceb750
removed unneccessary tox envs
jtisbell4 Apr 18, 2024
ed756bb
removed dupe codecov step
jtisbell4 Apr 18, 2024
4b30e4e
fixed coverage cmds in tox.ini
jtisbell4 Apr 18, 2024
74352ea
removed coverage-init
jtisbell4 Apr 18, 2024
f1479e7
moved erase cmd
jtisbell4 Apr 18, 2024
14c3030
PR template
Apr 18, 2024
eeaea58
issue templates
Apr 18, 2024
8d8f95f
Merge branch 'lorin/tempo/cicd-updates' into tox-refactor
Apr 18, 2024
47870bb
dbr 14.3 support
jtisbell4 Apr 18, 2024
3c7f188
removed lint and type-check from default envlist
jtisbell4 Apr 18, 2024
39e49c4
made reusable ci action
jtisbell4 Apr 18, 2024
c7a509a
forgot version ref
jtisbell4 Apr 18, 2024
4939ac5
moved to workflows subdir
jtisbell4 Apr 18, 2024
12ef690
removed version ref
jtisbell4 Apr 18, 2024
2eafd0e
fixed job name mismatch
jtisbell4 Apr 18, 2024
9eb5fb0
convert int to str
jtisbell4 Apr 18, 2024
5dd768a
forgot -e flag
jtisbell4 Apr 18, 2024
3a5b4a2
split push and release actions
jtisbell4 Apr 23, 2024
96db776
testing new push action
jtisbell4 Apr 24, 2024
b5089e2
try again
jtisbell4 Apr 24, 2024
eff62de
added code cov token
jtisbell4 Apr 24, 2024
d24d0f4
fixed tox build env
jtisbell4 Apr 24, 2024
457dea8
testing release action
jtisbell4 Apr 24, 2024
97e860c
fixed checkout depth
jtisbell4 Apr 24, 2024
6a97cb9
removed requirements copy step
jtisbell4 Apr 24, 2024
59b2c65
fixed bad docstrings
jtisbell4 Apr 24, 2024
357abe6
got tox build-docs working
jtisbell4 Apr 24, 2024
6383c1b
toxified release
jtisbell4 Apr 24, 2024
5bcee97
formatting
jtisbell4 Apr 24, 2024
68e0c39
forgot checkout step
jtisbell4 Apr 24, 2024
706fdc9
black check and diff
jtisbell4 Apr 24, 2024
7703b7e
stupid black version mismatch
jtisbell4 Apr 25, 2024
837e806
updated doc requirements
jtisbell4 Apr 25, 2024
59e9fc1
downgraded to py3.9 on docs job
jtisbell4 Apr 25, 2024
bc837c0
relaxing all constraints
jtisbell4 Apr 25, 2024
09db68c
switched to newer tox gh plugin
jtisbell4 Apr 25, 2024
ee09bd5
tox gh extension is no good
jtisbell4 Apr 25, 2024
c47a48e
comma
jtisbell4 Apr 25, 2024
0f0144d
fixme reminders
jtisbell4 Apr 25, 2024
dd1407f
removed extra workflows
jtisbell4 Apr 25, 2024
3b50f8c
Merge branch 'master' into tox-refactor
R7L208 Apr 25, 2024
b4e3cfe
remove build requirements for dbr91 env
R7L208 Apr 25, 2024
3b4a825
changes to buld locally on arm64 arch
R7L208 Apr 27, 2024
4f18dc0
simplify shell for dbr104 setup
R7L208 Apr 27, 2024
e4e8f6a
simplify shell for dbr104 setup
R7L208 Apr 28, 2024
59171a5
no build deps for packages that need numpy
R7L208 Apr 28, 2024
a0cc65b
remove dbr104 env support
R7L208 Apr 28, 2024
550d4e0
removed support for python 3.8 DBRs
jtisbell4 Apr 28, 2024
333975a
updated contribution docs
jtisbell4 Apr 28, 2024
4d02b80
set fail-fast to false
jtisbell4 Apr 29, 2024
b5a9a0c
checkpoint commit of some updates to the code that converts nano-seco…
tnixon May 14, 2024
8a7eb5e
Merge branch 'fix_nanos_conversion' into tox-refactor
tnixon May 14, 2024
7469a50
refactored test code data format to allow for better separation of DF…
tnixon May 15, 2024
0e1c3ef
formatting
jtisbell4 May 15, 2024
4c90318
We should just check for style compliance with black, not apply them
tnixon May 15, 2024
81f0239
Revert "We should just check for style compliance with black, not app…
tnixon May 15, 2024
fc55de2
moving dbr version specifiers to use compatibility syntax
tnixon May 15, 2024
8e42fb4
updating 2 test cases
tnixon May 21, 2024
dbf08c0
renamed test action
jtisbell4 May 22, 2024
e543b69
testing json refactor
jtisbell4 May 22, 2024
8d5a583
found new error
jtisbell4 May 22, 2024
35e2b50
still broken
jtisbell4 May 22, 2024
afb1175
variable name refactoring
R7L208 Jul 8, 2024
c24db4c
remove re import as it was unused
R7L208 Jul 8, 2024
4ea9e22
remove get_data_as_tsdf
R7L208 Jul 8, 2024
9b0094f
remove get_data_as_sdf
R7L208 Jul 8, 2024
645114f
fix typo in docstring
R7L208 Jul 8, 2024
d6e48f6
refactor schema comparison for test helper
R7L208 Jul 8, 2024
2414027
general refactor for TSDFBaseTests
R7L208 Jul 8, 2024
f6f3520
remove schema check since that is performed by chispa.asssert_df_equa…
R7L208 Jul 8, 2024
a46dd39
existing tests refactored and passing
R7L208 Jul 8, 2024
1bb8383
interpol test case work
R7L208 Jul 8, 2024
582dac5
WIP for test_tsdf_interpolate
R7L208 Jul 9, 2024
3d33a68
add idf getter to test dataframe builder
R7L208 Jul 9, 2024
1a5add8
tests for tsdf refactored
R7L208 Jul 9, 2024
ab5210a
remove test_tsdf_interpolate
R7L208 Jul 9, 2024
09e6423
refactor interpol_tests complete
R7L208 Jul 9, 2024
981f1ab
additional checks for idf dataframe equality
R7L208 Jul 9, 2024
55e65ad
refactor intervals_tests
R7L208 Jul 9, 2024
f1958b2
refactor io_tests
R7L208 Jul 9, 2024
21405b8
reindent with 2 spaces for consistency
R7L208 Jul 9, 2024
79390e7
refactor resample_2_tests
R7L208 Jul 9, 2024
8558e90
refactor resample_tests
R7L208 Jul 10, 2024
fc60934
safe delete and refactor to remove resample_2_tests
R7L208 Jul 10, 2024
4f2e8f9
refactor utils_tests
R7L208 Jul 10, 2024
a095585
chore: tox lint
R7L208 Jul 10, 2024
3cb8cd3
fix missing ts_convert keys in json
R7L208 Jul 10, 2024
1ac3391
adjust build release to only trigger when tag is pushed to master
R7L208 Jul 11, 2024
8ffdce7
Revert "adjust build release to only trigger when tag is pushed to ma…
tnixon Jul 11, 2024
fb17a8b
Merge branch 'master' into tox-refactor
tnixon Jul 11, 2024
60f9fc8
need to update CodeQL action version
tnixon Jul 11, 2024
06f93ac
updating to latest actions versions
tnixon Jul 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions .github/ISSUE_TEMPLATE/bug.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# See https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms
# and https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-githubs-form-schema
name: Bug Report
description: Something is not working with Tempo
title: "[BUG]: "
labels: ["bug", "needs-triage"]
projects: ["databrickslabs/12"]
body:
- type: checkboxes
attributes:
label: Is there an existing issue for this?
description: Please search to see if an issue already exists for the bug you encountered.
options:
- label: I have searched the existing issues
required: true
- type: textarea
attributes:
label: Current Behavior
description: |
A concise description of what you're experiencing.
**Do not paste links to attachments with logs and/or images, as all issues will attachments will get deleted.**
Use the `Relevant log output` field to paste redacted log output without personal identifying information (PII).
You can Ctrl/Cmd+V the screenshot, which would appear as a rendered image if it doesn't contain any PII.
validations:
required: false
- type: textarea
attributes:
label: Expected Behavior
description: A concise description of what you expected to happen.
validations:
required: false
- type: textarea
attributes:
label: Steps To Reproduce
description: Steps to reproduce the behavior.
placeholder: |
1. In this environment...
1. With this config...
1. Run '...'
1. See error...
validations:
required: false
- type: dropdown
id: cloud
attributes:
label: Cloud
description: What cloud are you using?
options:
- AWS
- Azure
- GCP
validations:
required: true
- type: textarea
id: version
attributes:
label: Version
description: What version of our software are you running?
validations:
required: true
- type: textarea
id: logs
attributes:
label: Relevant log output
description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks.
render: shell
9 changes: 9 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
blank_issues_enabled: false
contact_links:
- name: General Databricks questions
url: https://help.databricks.com/
about: Issues related to Databricks and not related to UCX
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
about: Issues related to Databricks and not related to UCX
about: Issues related to Databricks and not related to Tempo


- name: UCX Documentation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- name: UCX Documentation
- name: Tempo Documentation

url: https://databrickslabs.github.io/tempo/
about: Documentation about Tempo
33 changes: 33 additions & 0 deletions .github/ISSUE_TEMPLATE/feature.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# See https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms
# and https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-githubs-form-schema
name: Feature Request
description: Something new needs to happen with Tempo
title: "[FEATURE]: "
labels: ["enhancement", "needs-triage"]
projects: ["databrickslabs/13"]
body:
- type: checkboxes
attributes:
label: Is there an existing issue for this?
description: Please search to see if an issue already exists for the feature request you're willing to submit
options:
- label: I have searched the existing issues
required: true
- type: textarea
attributes:
label: Problem statement
description: A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
validations:
required: true
- type: textarea
attributes:
label: Proposed Solution
description: A clear and concise description of what you want to happen.
validations:
required: true
- type: textarea
attributes:
label: Additional Context
description: Add any other context, references or screenshots about the feature request here.
validations:
required: false
29 changes: 29 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
## Changes
<!-- Summary of your changes that are easy to understand. Add screenshots when necessary -->

### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword -->

Resolves #..

### Functionality

- [ ] added relevant user documentation
- [ ] added a new Class method
- [ ] modified existing Class method: `...`
- [ ] added a new function
- [ ] modified existing function: `...`
- [ ] added a new test
- [ ] modified existing test: `...`
- [ ] added a new example
- [ ] modified existing example: `...`
- [ ] added a new utility
- [ ] modified existing utility: `...`

### Tests
<!-- How is this tested? Please see the checklist below and also describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] verified on staging environment (screenshot attached)
78 changes: 78 additions & 0 deletions .github/workflows/build-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: build-release

on:
tnixon marked this conversation as resolved.
Show resolved Hide resolved
pull_request:
types: [opened, synchronize]
push:
branches: ['master']

jobs:
release:
runs-on: ubuntu-latest
environment: release
permissions:
# Used to authenticate to PyPI via OIDC and sign the release's artifacts with sigstore-python.
id-token: write
# Used to attach signing artifacts to the published release.
contents: write

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox

- name: Build dist
working-directory: ./python
run: tox -e build-dist

- name: Publish a Python distribution to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
if: startsWith(github.ref, 'refs/tags/v')
with:
packages-dir: python/dist/

docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
fetch-tags: true

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.9'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install tox

- name: Build docs
working-directory: ./python
run: tox -e build-docs

- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: html-docs
path: docs/_build/html/

- name: Deploy 🚀
uses: peaceiris/actions-gh-pages@v3
if: startsWith(github.ref, 'refs/tags/v')
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: docs/_build/html
70 changes: 0 additions & 70 deletions .github/workflows/codeql-analysis.yml

This file was deleted.

42 changes: 0 additions & 42 deletions .github/workflows/docs.yml

This file was deleted.

48 changes: 0 additions & 48 deletions .github/workflows/release.yml

This file was deleted.

Loading