feat: New structure for 1.x.x (#69)

* chore: move checker -> checker_old * feat: move checker template from course-template * fix(deps): add python deps * test(plugins): add aggregate and regexp plugin * test(plugins): add simple script test * feat: collect plugin stage time * feat: add jinja2 templating and plugin output * style: apply black and isort * ref: move tester and pipeline to the pkg root * fix: strategy validation in aggregation plugin * fix: add env to context * test: add print basic tests * style: fix some mypy comments * chore: add makefile for testing and linting * ci: use make file in ci tests * ci: add docs-preview job template * docs: add checker documentation draft * docs: remove or move old docs * feat: add hidden schema command * ci: fix concurrency fallback * ci: fall back to use list in matrix defenition * fix: pydantic union for older versions * fix: typings for 3.8-3.9 * fix: pydantic typings in 3.8 and 3.9 * fix: pydantic typings in 3.8 and 3.9 * fix: type[] -> Type[] for 3.8 * fix: add missing future annotation * feat(docs): docs generation (#78) * feat(docs): docs generation with MkDocs * fix(make): correct indentation * fix(make): use 'python -m' for docs build * ci: test pipeline minor fix * ci: permission deployments: write for docs preview * docs: enable mermaid and admonition * docs: fixes to match mkdocs style * docs: add mkdocs-click working * fix: revery to basic types * feat: refactor course and add course tests * feat: add exporter to collect files for testing * feat(plugin): Add plugin manytask.py (#80) * implement plugin manytask.py * rename var with submit time * fix plugin class name * move default time calculation to Args * add args test * fix when origin is empty str * add test for files collection * fix style * fix url * change from unittest mock to pytest mocker * add test for _post_with_retries * fix output for plugin (now not using logging, but properly outputted in run) * add validation for time, extract time formatting to function * fix newline * cleanup prints * refactor tests * add test for plugin run * add tests for verbose and bad response * add upper boundary for requests-mock version * remove unused get_default_args * change time format * test on requests 2.0.0 * remove redundant requests in pyproject.toml dependencies * fix mypy errors in test_manytask.py * fix redundant newlines in method declaration * simplify test_bad_response, make better assert * fix: typo in TODO * fix: remove _output in manytask plugin * test: pipeline and fix some errors * test: add config utils tests * fix: remove unused exceptions * test: exporter test template * fix: rename tester_token to report_token * fix: small fixes with pipeline result percentage * style: apply stylefixes and typings fixes * docs: refactor pipelines doc + some fixes * style: apply black * fix: pony in make file * fix: ```mermaid * ci: try to all dev docs * style: fix ruff * chore: delete old checker * feat: add checker docker image * style: apply black * chore: remove python 3.8 * style: fix with isort * style: fix add black line length configuration --------- Co-authored-by: Arman Tovmasyan <86357987+lilpuzeen@users.noreply.github.com> Co-authored-by: fortrest-jr <30597575+fortrest-jr@users.noreply.github.com>
manytask · Dec 27, 2023 · b8dae8d · b8dae8d
1 parent b33cb4f
commit b8dae8d
Show file tree

Hide file tree

Showing 107 changed files with 5,251 additions and 7,702 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -0,0 +1,89 @@
+# Git
+.git
+.gitignore
+.gitattributes
+
+
+# CI
+.codeclimate.yml
+.travis.yml
+.taskcluster.yml
+
+# Docker
+docker-compose.yml
+Dockerfile
+.docker
+.dockerignore
+
+# Byte-compiled / optimized / DLL files
+**/__pycache__/
+**/*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.cache
+nosetests.xml
+coverage.xml
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Virtual environment
+.env
+.venv/
+venv/
+
+# PyCharm
+.idea
+
+# Python mode for VIM
+.ropeproject
+**/.ropeproject
+
+# Vim swap files
+**/*.swp
+
+# VS Code
+.vscode/
diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml
@@ -32,6 +32,7 @@ jobs:
       name: pypi
       url: https://pypi.org/p/manytask-checker
     permissions:
+      deployments: write
       id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
     steps:
     - uses: actions/checkout@v4
@@ -49,27 +50,35 @@ jobs:
     - name: Publish to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
 
-#  release-github-pages:
-#    runs-on: ubuntu-latest
-#    steps:
-#      - uses: actions/checkout@v4
-#      - name: Setup Python 3.9
-#        uses: actions/setup-python@v3
-#        with:
-#          python-version: 3.9
-#      - uses: actions/cache@v3
-#        with:
-#          path: ${{ env.pythonLocation }}
-#          key: ${{ runner.os }}-python-3.9-${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml', 'setup.cfg') }}-docs
-#          restore-keys: |
-#            ${{ runner.os }}-python-3.9-
-#            ${{ runner.os }}-python-
-#            ${{ runner.os }}-
-#      - name: Install dependencies
-#        run: python -m pip install -e .[docs]
-#      - name: Build and publish docs
-#        run: |
-#          git fetch --all
-#          # lazydocs
-#          python -m mkdocs build --config-file docs/mkdocs.yml
-#          python -m mkdocs gh-deploy --config-file docs/mkdocs.yml --force
+  release-github-pages:
+    permissions:
+      contents: write
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+          cache: 'pip' # caching pip dependencies
+      - name: Install dependencies
+        run: |
+          python -m pip install -e .[docs]
+      - name: Docs deploy
+        run: |
+          echo -n "${{github.ref_name}}" > VERSION
+          make docs-deploy
+
+  release-docker:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.9', '3.10', '3.11', '3.12']
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build docker image
+        run: |
+          docker build --build-arg PYTHON_VERSION=${{ matrix.python-version }} -t manytask-checker:${{ github.sha }} .
+      - name: Test run --help in docker image
+        run: |
+            docker run --rm manytask-checker:${{ github.sha }} --help
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -1,17 +1,23 @@
 name: Lint and Test
 
 on:
-#  push:
-  pull_request:
+  push:
     branches: [ main ]
+  pull_request:
+    branches: [ main, new-1.x.x ]
+
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref_name || github.ref }}
+  cancel-in-progress: true
 
 
 jobs:
-  lint-python:
+  package-lint:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.9', '3.10', '3.11', '3.12']
       fail-fast: false
     steps:
       - uses: actions/checkout@v4
@@ -22,20 +28,15 @@ jobs:
           cache: 'pip' # caching pip dependencies
       - name: Install dependencies
         run: |
-          python -m pip install wheel
           python -m pip install -e .[test]
-      - name: Run isort
-        run: python -m isort checker --check
-      - name: Run ruff
-        run: python -m ruff checker
-      - name: Run mypy
-        run: python -m mypy checker
+      - name: Run linters
+        run: make lint
 
-  test-python:
+  package-test:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.9', '3.10', '3.11', '3.12']
       fail-fast: false
     steps:
       - uses: actions/checkout@v4
@@ -46,12 +47,9 @@ jobs:
           cache: 'pip' # caching pip dependencies
       - name: Install dependencies
         run: |
-          python -m pip install wheel
           python -m pip install -e .[test]
-          sudo apt-get update
-          sudo apt-get install -y iputils-ping cmake ninja-build clang-tidy clang-format
-      - name: Run tests
-        run: python -m pytest --python --cpp --cov-report=xml tests/
+      - name: Run all tests
+        run: make test
       - name: Upload coverage
         uses: codecov/codecov-action@v3
         with:
@@ -61,11 +59,11 @@ jobs:
           fail_ci_if_error: true  # optional (default = false)
           verbose: true  # optional (default = false)
 
-  build-python-package:
+  package-build:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.8', '3.9', '3.10', '3.11', '3.12']
+        python-version: ['3.9', '3.10', '3.11', '3.12']
       fail-fast: false
     steps:
       - uses: actions/checkout@v4
@@ -79,4 +77,85 @@ jobs:
           python -m pip install wheel
       - name: Build wheel
         run: |
+          # TODO: replace with make build
           python -m pip wheel . --no-deps --wheel-dir dist
+
+  docker-build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ['3.9', '3.10', '3.11', '3.12']
+      fail-fast: false
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build docker image
+        run: |
+          docker build --build-arg PYTHON_VERSION=${{ matrix.python-version }} -t manytask-checker:${{ github.sha }} .
+      - name: Test run --help in docker image
+        run: |
+            docker run --rm manytask-checker:${{ github.sha }} --help
+  
+  docs-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+          cache: 'pip' # caching pip dependencies
+      - name: Install dependencies
+        run: |
+          python -m pip install -e .[docs]
+      - name: Build docs
+        run: |
+          make docs-build
+
+  # publish dev docs on push to main
+  docs-deploy-dev:
+    permissions:
+      contents: write
+      deployments: write
+    runs-on: ubuntu-latest
+    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+          cache: 'pip' # caching pip dependencies
+      - name: Install dependencies
+        run: |
+          python -m pip install -e .[docs]
+      - name: Docs deploy
+        run: |
+          make docs-deploy-dev
+
+  docs-preview:
+    permissions:
+      deployments: write
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: 3.12
+          cache: 'pip' # caching pip dependencies
+      - name: Install dependencies
+        run: |
+          python -m pip install -e .[docs]
+      - name: Docs preview
+        run: |
+          make docs-build
+      - name: Publish to Cloudflare Pages
+        id: deploy
+        uses: cloudflare/pages-action@v1
+        with:
+          apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
+          accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+          projectName: manytask-checker
+          directory: ./site
+          # Optional: Enable this if you want to have GitHub Deployments triggered
+          gitHubToken: ${{ secrets.GITHUB_TOKEN }}
+          # Optional: Switch what branch you are publishing to.
+          # By default this will be the branch which triggered this workflow
+          branch: ${{ ( github.event.workflow_run.head_repository.full_name == github.repository && github.event.workflow_run.head_branch == 'main' && 'main' ) || ( github.event.workflow_run.head_sha ) }}
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,27 @@
+ARG PYTHON_VERSION=3.12
+
+# Stage 1: Build stage
+FROM python:${PYTHON_VERSION}-alpine as builder
+
+WORKDIR /usr/src/app
+
+COPY pyproject.toml VERSION Makefile setup.py README.md ./
+COPY checker ./checker
+
+RUN python -m venv /opt/checker-venv
+RUN /opt/checker-venv/bin/python -m pip install --no-cache-dir --require-virtualenv .
+RUN find /opt/checker-venv -type d -name '__pycache__' -exec rm -r {} + && \
+    find /opt/checker-venv -type d -name 'tests' -exec rm -rf {} + && \
+    find /opt/checker-venv -name '*.pyc' -delete && \
+    find /opt/checker-venv -name '*.pyo' -delete
+
+
+# Stage 2: Runtime stage
+FROM python:${PYTHON_VERSION}-alpine
+
+WORKDIR /usr/src/app
+
+COPY --from=builder /opt/checker-venv /opt/checker-venv
+
+ENTRYPOINT [ "/opt/checker-venv/bin/python", "-m", "checker" ]
+CMD [ "--help" ]