Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tasking manager fastapi #6649

Open
wants to merge 183 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
183 commits
Select commit Hold shift + click to select a range
94ae0ac
feat: initial setup for fastapi
kaditya97 Mar 25, 2024
a7d7dd4
fix: basic setup
kaditya97 Apr 11, 2024
c74ee92
feat: added pyinstrument for api performance testing
kaditya97 Apr 17, 2024
e9b4798
feat: asyncio implementation
kaditya97 Apr 25, 2024
a549e0c
fix: minor import issues
kaditya97 Apr 25, 2024
31e9338
feat: new docker files with pdm integration
kaditya97 May 7, 2024
9f2936c
fix: projects all api
kaditya97 May 8, 2024
93fb6a1
fix: countries api working
kaditya97 May 15, 2024
6a50637
Fix backend container name on ECS
eternaltyro May 16, 2024
f4dcce9
Fix image name
eternaltyro May 16, 2024
de3e23e
Make deploy job depend on completion of build job
eternaltyro May 16, 2024
65f36c0
Test image output from build job
eternaltyro May 16, 2024
7e43443
Fix image link
eternaltyro May 16, 2024
bd97588
Fix interpolation in workflow
eternaltyro May 16, 2024
723f3bf
Move to new Task Definition family
eternaltyro May 16, 2024
b11ec40
Fix github actions output step
eternaltyro May 17, 2024
36d7422
Better naming and targeted build
eternaltyro May 17, 2024
db01309
Fix output format
eternaltyro May 17, 2024
4e54b2c
Fix syntax error in workflow
eternaltyro May 17, 2024
b976deb
Fix metadata parse
eternaltyro May 17, 2024
174ab6d
Build metadata fixes
eternaltyro May 17, 2024
323d6c6
Move shell variables to workflow step env
eternaltyro May 17, 2024
7edf11f
Fix image_name for push and cleanup Dockerfile
eternaltyro May 17, 2024
8bc532f
Suppress apt output properly
eternaltyro May 17, 2024
a31fe32
Fix output ref
eternaltyro May 17, 2024
c70658f
Bug fixes
eternaltyro May 17, 2024
f60d811
fix: add json to ignored types in config
kaditya97 May 17, 2024
08cedf8
fix: frontend update
kaditya97 May 22, 2024
f498125
hot-fix: migrations error fix for multiple entry
kaditya97 May 28, 2024
fcb699c
feat: Campaigns get api for fastapi app
prabinoid Jun 3, 2024
4ce3838
feat: Async post for campaigns api
prabinoid Jun 6, 2024
7782e88
feat: Campaign retrieve and delete api now works asynchronously
prabinoid Jun 11, 2024
d8dfc49
Refac: Refactored project retrieve api to work asynchronously
prabinoid Jun 13, 2024
27c0ceb
feat: Organisations get apis with statistics
prabinoid Jun 18, 2024
9a63f4e
feat: Organisations apis updated for managers, teams statistics and d…
prabinoid Jun 25, 2024
83b6819
feat: Organisation campaigns and project campaign get apis
prabinoid Jun 25, 2024
2cdc407
feat: Organisation post and patch patch apis
prabinoid Jun 27, 2024
80d94b7
refac: Campaign, organisations and projects association delete apis a…
prabinoid Jul 2, 2024
2ab0cec
Implement query execution using encode/databases for organisations li…
prabinoid Jul 31, 2024
bb5695e
Organisation retrieve api refactored to work asynchronously using dat…
prabinoid Aug 5, 2024
fd10851
Refactor: Get organisation via organisation slug
prabinoid Aug 5, 2024
3a9fdc1
Organisation delete apis refactored and migrations env database uri f…
prabinoid Aug 6, 2024
4a26042
Organisation create, update, statistics refactored and field validato…
prabinoid Aug 8, 2024
dc98999
Campaign apis refactored and updated organisations list query to retr…
prabinoid Aug 9, 2024
e02e8b7
Update campaign refactored
prabinoid Aug 12, 2024
b2fa655
Campaign linkage apis with organisations refactored
prabinoid Aug 13, 2024
d594456
Countries api refactored
prabinoid Aug 13, 2024
54c88c2
Project retrieve and countries api refactored
prabinoid Aug 15, 2024
605c436
Project list api and pagination util refactored
prabinoid Aug 21, 2024
038dab0
refactor: licenses
Anuj-Gupta4 Aug 22, 2024
79bc4fc
Refactor: Project campaigns apis
prabinoid Aug 22, 2024
973c65f
refactor: interests
Anuj-Gupta4 Aug 22, 2024
c941a1d
Teams module added to routers and teams get apis refactored.
prabinoid Aug 27, 2024
ace25cc
Project teams api and fixes in project get api queries
prabinoid Aug 27, 2024
ae4192e
Project queries bbox api refactored
prabinoid Aug 29, 2024
d58a295
Project admin service, owner projects api and validators refactored
prabinoid Aug 31, 2024
3d9c76e
Project detail summary and user touched projects api refactored
prabinoid Sep 3, 2024
75ec750
fix: Campaigns list and allowed usernames list
prabinoid Sep 3, 2024
c3c34d4
Project user activities apis refactored
prabinoid Sep 3, 2024
61dfc11
Project contributions apis and other geom queries apis refactored
prabinoid Sep 4, 2024
a7add10
refactor: comment
Anuj-Gupta4 Sep 5, 2024
8a9cf50
Project statistics and User project statistics refactored
prabinoid Sep 9, 2024
097f7da
refactor: user's license acceptance
Anuj-Gupta4 Sep 5, 2024
9eb2d50
refactor: user rate contributions
Anuj-Gupta4 Sep 6, 2024
c42d930
refactor: get user interests
Anuj-Gupta4 Sep 6, 2024
63847ac
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
20de1fb
Heartbeat api refactored and ohsome api environment import handled
prabinoid Sep 9, 2024
4ad3bcc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 9, 2024
166223c
Pre commit hooks refactor and db connection param json postgres port …
prabinoid Sep 9, 2024
9d16c9c
Similar and Popular projects apis refactored and heartbeat api fixed
prabinoid Sep 10, 2024
bf98849
fix: field validator for OrganisationDTO and NewCampaignDTO
kaditya97 Sep 11, 2024
0241fb2
fix: black formatting
kaditya97 Sep 11, 2024
6e88839
Project and user favorites apis refactored
prabinoid Sep 10, 2024
214d58f
refactor: set user interests
Anuj-Gupta4 Sep 9, 2024
1b640c9
refactor: get message
Anuj-Gupta4 Sep 9, 2024
2bf5714
refactor: get all notifications with filters
Anuj-Gupta4 Sep 10, 2024
04c437c
refactor: unread message count
Anuj-Gupta4 Sep 11, 2024
e0a375f
refactor: mark messages as read
Anuj-Gupta4 Sep 11, 2024
b42e094
refactor: delete messages
Anuj-Gupta4 Sep 11, 2024
ac34c0b
refactor: update notification time
Anuj-Gupta4 Sep 11, 2024
a2751d4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 12, 2024
33d126b
refactor: get user(s)
Anuj-Gupta4 Sep 12, 2024
37ed3d3
refactor: query users
Anuj-Gupta4 Sep 13, 2024
757d669
Tasks meta data api refactored
prabinoid Sep 12, 2024
6bfd42b
Lock task for mapping and its functions refactored
prabinoid Sep 13, 2024
c8dfd1b
fix: system and related api's migration
kaditya97 Sep 16, 2024
c393908
fix: black errors
kaditya97 Sep 16, 2024
a48dc8a
fix: black erros
kaditya97 Sep 16, 2024
07856ac
fix: issues api's
kaditya97 Sep 23, 2024
c9652c6
fix: field validator for OrganisationDTO and NewCampaignDTO
kaditya97 Sep 11, 2024
2fd8a6d
fix: black formatting
kaditya97 Sep 11, 2024
7896b5d
refactor: set user interests
Anuj-Gupta4 Sep 9, 2024
64b49c0
refactor: get message
Anuj-Gupta4 Sep 9, 2024
0b5788a
refactor: get all notifications with filters
Anuj-Gupta4 Sep 10, 2024
ce4b0bf
refactor: mark messages as read
Anuj-Gupta4 Sep 11, 2024
293c98f
refactor: delete messages
Anuj-Gupta4 Sep 11, 2024
3f3b80d
refactor: update notification time
Anuj-Gupta4 Sep 11, 2024
d054a15
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 12, 2024
3d5cea6
refactor: get user(s)
Anuj-Gupta4 Sep 12, 2024
96d17bf
refactor: query users
Anuj-Gupta4 Sep 13, 2024
4875211
Tasks meta data api refactored
prabinoid Sep 12, 2024
d84f624
Lock task for mapping and its functions refactored
prabinoid Sep 13, 2024
f284fae
fix: system and related api's migration
kaditya97 Sep 16, 2024
b67a063
fix: black errors
kaditya97 Sep 16, 2024
4f15895
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 16, 2024
b94d0a7
fix: team api's
kaditya97 Sep 25, 2024
bc4b824
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 23, 2024
324c5bc
Invalidate all and lock a task for mapping refactored
prabinoid Sep 17, 2024
49ed76b
Unlock task for mapping and related stats utils refactored
prabinoid Sep 23, 2024
7bfb5fe
Lock task for validation refactored
prabinoid Sep 24, 2024
ae6c1ea
Unlock task after validation refactored
prabinoid Sep 24, 2024
16a824a
fix: precommit changes
kaditya97 Sep 25, 2024
30a2b5b
fix: sonarcloud suggestion
kaditya97 Sep 26, 2024
36a8b38
Stop task validation refactored
prabinoid Sep 26, 2024
245b113
Stop mapping task refactored
prabinoid Sep 26, 2024
89f4210
Task undo last action refactored
prabinoid Sep 27, 2024
46f7962
Map all tasks, Validate all tasks and Reset all tasks refactored
prabinoid Sep 27, 2024
68c2906
fix: project post requests
kaditya97 Oct 1, 2024
9ceb89c
fix: user action and related api's
kaditya97 Oct 1, 2024
de60da9
Extend task lock duration and Revert user action from task refactored
prabinoid Sep 30, 2024
8ac40b0
Split task api and related methods refactored
prabinoid Oct 1, 2024
ba79d6e
fix: user tasks api
kaditya97 Oct 1, 2024
6dbffdc
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 1, 2024
cbe838a
Project tasks gpx and xml export
prabinoid Oct 2, 2024
a1c07bf
fix: project post api
kaditya97 Oct 7, 2024
2bfe370
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Oct 7, 2024
6575c13
fix: Project action filter and user stats
prabinoid Oct 8, 2024
a7869f4
JSONResponse handling and cleanups
prabinoid Oct 9, 2024
21d41ab
fix project filters
prabinoid Oct 15, 2024
6b73668
Fix project responses and filters
prabinoid Oct 16, 2024
d1d16bb
fix project info dto
prabinoid Oct 16, 2024
aa14ae8
fix project teams patch
prabinoid Oct 16, 2024
217cd91
Project patch fixed and dtos refactored
prabinoid Oct 22, 2024
b7b0b87
Bulk action being applied for all tasks fixed, due date exception han…
prabinoid Oct 22, 2024
b481747
All messages, organisation retrieve and task activities fixes
prabinoid Oct 23, 2024
6e9609e
Project and notification duplication handled, organisation teams and …
prabinoid Oct 25, 2024
fb7d0bd
Messages, teams, comments, notification modules and project filters a…
prabinoid Oct 29, 2024
14a3b96
Teams managers and members assignment and retrieve query fixed
prabinoid Nov 4, 2024
91771c1
Task dto last updated attribute made timezone aware and add teams by …
prabinoid Nov 4, 2024
f0fd183
Project clone and transfer project ownership refactored
prabinoid Nov 5, 2024
637a216
Added root validator for project info dto
prabinoid Nov 5, 2024
eba8a2c
Split task url and geom fixed
prabinoid Nov 6, 2024
5e5ae60
Project, organisation teams deletion and other fixes
prabinoid Nov 6, 2024
28488aa
Http exception handler in the app for token expiration handling
prabinoid Nov 11, 2024
6b13f3a
Register user, retrieve org by id and slug and user interests refactored
prabinoid Nov 12, 2024
ca22a59
feat(backend): Timestamps made utcnow and serialization done with utc…
prabinoid Nov 13, 2024
cbeafb7
fix: Join team by request
prabinoid Nov 14, 2024
d05f3fe
Task and project card last contributions fixed and list dto for proje…
prabinoid Nov 15, 2024
4250fd4
fix: Project search and deletion.
prabinoid Nov 15, 2024
3f2c2bb
fix: clear priority areas, send message after validation, project com…
prabinoid Nov 18, 2024
8992504
fix: Update teams and formatting
prabinoid Nov 18, 2024
d1b33bf
* Team messaging background task,
prabinoid Nov 19, 2024
982d1ac
fix: Invalid token exception in TokenAuthBackend
prabinoid Nov 19, 2024
8f26578
* Send message to contributors and background task.
prabinoid Nov 20, 2024
3ecfc1b
Revert user task fixed and get latest task activity now returns disti…
prabinoid Nov 20, 2024
439ec94
* Send message after comment, parse usernames and mentions.
prabinoid Nov 26, 2024
761db17
* Rebased against develop.
prabinoid Dec 2, 2024
ad4011f
Remove db connection dependency in routers
prabinoid Dec 3, 2024
9b7e089
* Concurrent task validation using asyncio gather
prabinoid Dec 4, 2024
fbacdc5
fix: Recommended projects, my tasks in my contrib section and cleanups
prabinoid Dec 4, 2024
b7259cc
* Asyncio cron job scheduler set up.
prabinoid Dec 11, 2024
280522a
Aoi, tasks and project non geometries exports and cleanups
prabinoid Dec 11, 2024
0bf2ace
Remove atomic transaction from concurrent task execution
prabinoid Dec 13, 2024
105129d
fix: Tasks added to cloned projects
prabinoid Dec 13, 2024
fd9292b
* Export projects as csv
prabinoid Dec 17, 2024
9bb5f86
Private projects filters fixed and author names in list view and csv …
prabinoid Dec 20, 2024
4191fc5
Task statistics and filters refactored
prabinoid Dec 20, 2024
0315800
init: locustfile added
mahesh-naxa Dec 17, 2024
58c4851
Locust and compose files for performance bench marking and its setup …
prabinoid Dec 24, 2024
1d1c353
* Organisation statistics.
prabinoid Dec 26, 2024
fd4a8c8
User statistics refactored.
prabinoid Dec 30, 2024
7579887
refactor: cron job for auto unlock task.
prabinoid Jan 2, 2025
2989a7d
User statistics query results cast to int
prabinoid Jan 2, 2025
07ca6b7
Remove sqlalchemy db configs
prabinoid Jan 2, 2025
3bd0363
feat: Cron jobs updated for stats update and validator service except…
prabinoid Jan 16, 2025
fdcab7e
fix: update aws-actions/amazon-ecs-deploy-task-definition to v2
nischalstha9 Jan 16, 2025
3ba8cd0
* Team membership and permission fixed for projects.
prabinoid Jan 17, 2025
6fcc416
project to map and validate sorting zero division case handled
prabinoid Jan 17, 2025
618eae8
* Rebase against develop
prabinoid Jan 20, 2025
1197601
pm only dependency function and injection in the required functions
prabinoid Jan 21, 2025
98b2bb3
Merge pull request #6701 from hotosm/fastapi-refactor
prabinoid Jan 21, 2025
1921714
interests and license deletion message and cron service
prabinoid Jan 24, 2025
6eb2efb
Merge pull request #6713 from hotosm/fastapi-refactor
prabinoid Jan 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions .github/workflows/ecs-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,13 @@ env:
REGISTRY: ghcr.io
AWS_REGION: us-east-1
ECS_CLUSTER: tasking-manager
ECS_SERVICE: tasking-manager-fastAPI
ECS_SERVICE: backend-fastAPI
CONTAINER_NAME: backend
IMAGE_NAME: hotosm/tasking-manager-backend # was ${{ github.repository }}

jobs:
build-push-image:
name: Build Images
image-build-and-push:
name: Build Container Images
runs-on: ubuntu-latest
environment: production

Expand All @@ -24,14 +24,11 @@ jobs:
packages: write

outputs:
imageid: steps.build-push-image.imageid
image_tags: ${{ steps.meta.outputs.tags }}

steps:
- name: Setup QEMU
uses: docker/setup-qemu-action@v3

- name: Setup Buildx
uses: docker/setup-buildx-action@v3
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3

- name: Log in to the Container registry
uses: docker/login-action@v3
Expand All @@ -45,31 +42,34 @@ jobs:
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

tags: |
type=ref,event=branch
type=raw,value=fastapi
- name: Build and push container image
id: build-push-image
uses: docker/build-push-action@v5
with:
context: "{{defaultContext}}"
target: prod
platforms: linux/amd64,linux/arm64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}


deploy:
name: Deploy
name: Deploy to ECS
runs-on: ubuntu-latest
environment: production

needs: image-build-and-push

permissions:
contents: read
id-token: write

steps:
- name: Checkout
uses: actions/checkout@v4
- uses: actions/checkout@v4

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
Expand All @@ -80,18 +80,18 @@ jobs:

- name: Download task definition
run: |
aws ecs describe-task-definition --task-definition tasking-manager --query taskDefinition > task-definition.json
aws ecs describe-task-definition --task-definition tasking-manager-fastAPI --query taskDefinition > task-definition.json
- name: Task definition rendition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@v1
with:
task-definition: task-definition.json
container-name: ${{ env.CONTAINER_NAME }}
image: ${{ needs.build-push-image.outputs.imageid }}
image: ${{ needs.image-build-and-push.outputs.image_tags }}

- name: Deploy task definition
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
uses: aws-actions/amazon-ecs-deploy-task-definition@v2
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: ${{ env.ECS_SERVICE }}
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,4 @@ htmlcov/

# Docker & Docker compose
docker-compose.override.yml
postgres_data/
41 changes: 17 additions & 24 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,13 +1,10 @@
ARG DEBIAN_IMG_TAG=slim-bookworm
ARG PYTHON_IMG_TAG=3.10



FROM docker.io/python:${PYTHON_IMG_TAG}-${DEBIAN_IMG_TAG} as base
ARG APP_VERSION=0.1.0
ARG DOCKERFILE_VERSION=0.5.0
ARG ALPINE_IMG_TAG
ARG DEBIAN_IMG_TAG
ARG PYTHON_IMG_TAG
ARG MAINTAINER=sysadmin@hotosm.org
LABEL org.hotosm.tasks.app-version="${APP_VERSION}" \
Expand All @@ -28,7 +25,7 @@ FROM base as extract-deps
RUN pip install --no-cache-dir --upgrade pip
WORKDIR /opt/python
COPY pyproject.toml pdm.lock README.md /opt/python/
RUN pip install --no-cache-dir pdm==2.18.1
RUN pip install --no-cache-dir pdm==2.8.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small thing, but it might be worth updating pdm to 2.18.1 here to avoid downgrading on merge

RUN pdm export --prod --without-hashes > requirements.txt


Expand All @@ -37,13 +34,14 @@ FROM base as build
RUN pip install --no-cache-dir --upgrade pip
WORKDIR /opt/python
# Setup backend build-time dependencies
RUN apt-get update && apt-get install --no-install-recommends -y \
build-essential \
libffi-dev \
libgeos-dev \
postgresql-server-dev-15 \
python3-dev \
&& rm -rf /var/lib/apt/lists/*
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive \
apt-get -q install --no-install-recommends -y \
build-essential \
postgresql-server-dev-15 \
python3-dev \
libffi-dev \
libgeos-dev
# Setup backend Python dependencies
COPY --from=extract-deps \
/opt/python/requirements.txt /opt/python/
Expand All @@ -66,9 +64,10 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
# Setup backend runtime dependencies
RUN apt-get update && \
apt-get install --no-install-recommends -y \
libgeos3.11.1 postgresql-client proj-bin && \
rm -rf /var/lib/apt/lists/*
DEBIAN_FRONTEND=noninteractive \
apt-get -q install --no-install-recommends -y \
postgresql-client libgeos3.11.1 proj-bin curl && \
apt-get clean && rm -rf /var/lib/apt/lists/*
COPY --from=build \
/home/appuser/.local \
/home/appuser/.local
Expand All @@ -83,25 +82,19 @@ COPY manage.py .

FROM runtime as debug
RUN pip install --user --no-warn-script-location \
--no-cache-dir debugpy==1.8.5
--no-cache-dir debugpy==1.8.1
EXPOSE 5678/tcp
CMD ["python", "-m", "debugpy", "--wait-for-client", "--listen", "0.0.0.0:5678", \
"-m", "gunicorn", "-c", "python:backend.gunicorn", "manage:application", \
"-m", "uvicorn", "backend.main:api", "--host", "0.0.0.0", "--port", "5000", \
"--reload", "--log-level", "error"]



FROM runtime as prod
USER root
RUN apt-get update && \
apt-get install -y curl && \
rm -rf /var/lib/apt/lists/*
# Pre-compile packages to .pyc (init speed gains)
RUN python -c "import compileall; compileall.compile_path(maxlevels=10, quiet=1)"
RUN python -m compileall .
EXPOSE 5000/tcp
USER appuser:appuser
# Default gunicorn worker count is 1
# For prod the WEB_CONCURRENCY env var can be used to set this
CMD ["gunicorn", "-c", "python:backend.gunicorn", "manage:application", \
"--log-level", "error"]
CMD ["uvicorn", "backend.main:api", "--host", "0.0.0.0", "--port", "5000", \
"--log-level", "error","--reload"]
Loading
Loading