DEV > EPH-2 test migration script TP>PP (#4559)

* uv * fixes / mypy dont pass * fix compose paths * fix e2e * 228219_fincacial_service_provider_admin_fix * fix flaky test failing due to beginning of new year * fix another flaky test * fix also last_Registration_date * 217655_dedup_statistics_rdi_population_preview (#4545) * Adjust PaymentPlan model to contain informations coming from TP and HS (#4372) * init * add tp list rest * init changes * add TP_MIGRATION_MAPPING * modify create target population add collector group * style 🎯 * add collector filters blocks * upd few more fields and properties * move full_rebuild & refactoring * add collector rule groups * add CheckConstraint & migrations * update validation & migrations * upd tests * upd migrations * fe cleanup & add more ut 🌟 * coverage 💯 * init script * upd * script optimization * refactor: create/upd mutation * add updated_tc_rules * refactor mutations * refactor mutations * upd migrations * more fixes * migrations // fix initdemo * migration * add more statuses transitions * fixes * fixes // upd unit tests * more refactoring and upd unit tests * unit tests // fixes * conflicts & remove unused TPnode * migrations conflict * migration * add TODOs & skip * add more tests // fixes * remove unused code * fixes // add more tests * upd snapshot * more tests * more tests * add more 💯 🌟 * upd migration script * change mutations part 1 * add more test & refactoring migration script * add test_migrate_tp_into_pp * more adjustments * upd tests & fix .delete() * remove imports * upd test * delete, copy * accountability * add filters isTargetPopulation & isPaymentPlan * more changes * generate schema * add migrate_message_and_survey * fix update targeting_criteria * upd filter * migration script upd * more changes * more * add more unit tests ⭐ * fix migration script & upd unit tests 🌟 * next part * more * more changes * upd filters * add total_households_count_with_valid_phone_no * upd filter * fix * another part * add noStatus filter * statusNot * add GraphQLError when no ba header for allPrograms query * frontend compiles * fixes * fix payments lists TP * fix pp list * lint * fix snapshots * minor fixes * fix buttons and status * fix pp queries * fix generated * fix schema * fix e2e fixtures * upd statuses * fixes 🌟 * unit tests ⭐ * unit tests & coverage 🌟 * fix e2e * fix e2e * fix labels + lock unlock * id undefined * fix mutation * bring back verify manual * bug fixes * targeting hh fix * upd fe test * fix fe fixtures * fix more e2e * again fix more e2e * fix e2e * schema * upd filtering * test with e2e * test with test_edit_targeting * add xfail * small fixes * update snapshots * fixes * upd test * add open pp mutation * fix upd currency ⭐ * review * migration * ba fixtures * fe schema upd * remove isPaymentPlanApplicable and Draft status * clean up * upd filters * fix status * change steficon mutation * steficon pp * upd mutations * upd schema * fixes for SetSteficonRuleOnPaymentPlanPaymentListMutation * fe lint * remove none option in vulnerability component * fix ut * fe fixes * more fixes * upd query 🌟 * review * one more fix * revert filtering back --------- Co-authored-by: Maciej Szewczyk <maciej.szewczyk@tivix.com> Co-authored-by: Maciej Szewczyk <34482854+mmaciekk@users.noreply.github.com> Co-authored-by: marekbiczysko <marek.biczysko@tivix.com> * fixing path * cert download develop * exclude hope on dev install * fix docker file one more time python path for dev image * New API endpoints aurora (#4548) * WIP: add OrganizationListView * WIP: upd api & add simple unit tests * ut * add cache & upd unit tests * remove pdm.lock * fixes for selenium test localy * DEV update migration script TP>PP (#4553) * upd migrations script * fix 🌟 * upd comment --------- Co-authored-by: Jan Romaniak <jan.romaniak@kellton.com> Co-authored-by: marekbiczysko <marek.biczysko@tivix.com> Co-authored-by: Domenico <dom.dinicola@gmail.com> Co-authored-by: Paulina Kujawa <p.kujawa2f@gmail.com> Co-authored-by: Paulina Kujawa <42150286+pkujawa@users.noreply.github.com> Co-authored-by: Marek Biczysko <34810846+MarekBiczysko@users.noreply.github.com> Co-authored-by: Maciej Szewczyk <maciej.szewczyk@tivix.com> Co-authored-by: Maciej Szewczyk <34482854+mmaciekk@users.noreply.github.com> Co-authored-by: Jan Romaniak <jan.romaniak@tivix.com>
unicef · Jan 15, 2025 · 7f0c734 · 7f0c734
1 parent acd4be1
commit 7f0c734
Show file tree

Hide file tree

Showing 312 changed files with 13,665 additions and 18,901 deletions.
diff --git a/.github/helpers/docker-compose.selenium.yml b/.github/helpers/docker-compose.selenium.yml
@@ -4,11 +4,11 @@ volumes:
 services:
   backend:
     volumes:
-      - ../../tests/test-coverage:/code/test-coverage
-      - ../../tests/report/:/code/tests/selenium/output_data/
+      - ../../tests/test-coverage:/app/test-coverage
+      - ../../tests/report/:/app/tests/selenium/output_data/
       - type: volume
         source: backend-web-app
-        target: /code/src/hct_mis_api/apps/web
+        target: /app/src/hct_mis_api/apps/web
         volume:
           nocopy: false
     depends_on:
@@ -31,7 +31,7 @@ services:
       - backend-web-app:/tmp/
     command: |
       sh -c "
-        cp -r /packages/__pypackages__/3.12/lib/hct_mis_api/apps/web/* /tmp/
+        cp -r /app/.venv/lib/python3.12/site-packages/hct_mis_api/apps/web/* /tmp/
       "
     restart: "no"
 

diff --git a/.github/helpers/docker-compose.tst.yml b/.github/helpers/docker-compose.tst.yml
@@ -3,8 +3,8 @@ services:
   backend:
     image: ${dev_backend_image}
     volumes:
-      - ../../tests/test-coverage:/code/test-coverage
-      - ./dev.sh:/code/dev.sh
+      - ../../tests/test-coverage:/app/test-coverage
+      - ./dev.sh:/app/dev.sh
     depends_on:
       - db
       - redis

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1,19 @@
+exclude *
+exclude __pycache__
+exclude node_modules
+
+include README.md
+include MANIFEST.in
+include AUTHORS
+include CHANGES.md
+include LICENSE.md
+include pyproject.toml
+include *.py
+include uv.lock
+
+
+recursive-include src/hct_mis_api *
+recursive-include src/hct_mis_api *.html
+recursive-include src/hct_mis_api *.json
+
+recursive-exclude src/**/node_modules *.*
diff --git a/development_tools/compose.yml b/development_tools/compose.yml
@@ -24,10 +24,10 @@ services:
     ports:
       - "8080:8000"
     volumes:
-      - ../.:/code/
+      - ../src:/app/src
       - backend-data:/data
-      - ../pyproject.toml:/packages/pyproject.toml
-      - ../pdm.lock:/packages/pdm.lock
+      - ../pyproject.toml:/app/pyproject.toml
+      - ../uv.lock:/packages/uv.lock
       - ipython_data_local:/root/.ipython
     command: "dev"
     depends_on:
@@ -61,7 +61,7 @@ services:
     env_file:
       - .env
     volumes:
-      - ../src:/code/
+      - ../src:/app/src
       - backend-data:/data
     command: "celery-worker"
     depends_on:
@@ -88,7 +88,7 @@ services:
     env_file:
       - .env
     volumes:
-      - ../src:/code/
+      - ../src:/app/src/
       - backend-data:/data
     command: "celery-beat"
     depends_on:
@@ -179,7 +179,7 @@ services:
       - .env
     command: "celery-flower"
     volumes:
-      - ../src:/code/
+      - ../src:/app/src
       - backend-data:/data
     ports:
       - "5555:5555"

diff --git a/development_tools/local_selenium_init.sh b/development_tools/local_selenium_init.sh
@@ -41,6 +41,7 @@ export CELERY_TASK_ALWAYS_EAGER=true
 export LIBRARY_PATHS=true
 SCRIPT_DIR=$(realpath "$(dirname $0)")
 MAIN_DIR=$(realpath $SCRIPT_DIR/..)
+export ELASTICSEARCH_SYNONYMS_FILE=$MAIN_DIR/src/data/synonyms.txt
 echo "SCRIPT_DIR: $SCRIPT_DIR"
 export PYTHONPATH=$MAIN_DIR/src:$PYTHONPATH
 export OUTPUT_DATA_ROOT=$MAIN_DIR/tests/selenium/output_data

diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -1,16 +1,15 @@
 FROM alpine:3.19.1 as curl
 RUN apk add curl && mkdir /data
-
 FROM curl as waitforit
 RUN curl -o /data/waitforit -sSL https://github.com/maxcnunes/waitforit/releases/download/v2.4.1/waitforit-linux_amd64 \
     && chmod +x /data/waitforit
 
 FROM curl as certs
-RUN curl -o /data/psql-cert.crt -L https://dl.cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem
+RUN curl -o /data/psql-cert.crt -L https://cacerts.digicert.com/DigiCertGlobalRootCA.crt.pem
 
 # Base image
 FROM python:3.12.7-slim-bookworm as base
-
+ENV APP_PATH=/app
 ARG UID=82
 
 RUN apt-get update \
@@ -35,42 +34,30 @@ RUN apt-get update \
   && rm -rf /var/lib/apt/lists/* \
   && addgroup --system --gid "${UID}" hope \
   && adduser --system --uid "${UID}" --disabled-password --home /home/hope --shell /sbin.nologin --group hope --gecos hope \
-  && mkdir -p /code /tmp /data \
+  && mkdir -p $APP_PATH /tmp /data \
   && mkdir -p -m 0700 /run/user/"${UID}" \
-  && chown -R hope:hope /code /tmp /data /run/user/"${UID}"
-
-ENV PDM_PACKAGES=/packages
-ENV CODE=/code
-ENV PDM_PROJECT=$PDM_PACKAGES
-ENV PYPACKAGES=$PDM_PACKAGES/__pypackages__/3.12
-ENV PYTHONPYCACHEPREFIX=/tmp/pycache \
-    PYTHONPATH=$PYPACKAGES/lib:$PYTHONPATH \
-    PATH=$PYPACKAGES/bin:$PATH \
-    XDG_RUNTIME_DIR=/run/user/"${UID}"
+  && chown -R hope:hope $APP_PATH /tmp /data /run/user/"${UID}"
+
+
+ENV XDG_RUNTIME_DIR=/run/user/"${UID}"
 ENV DJANGO_SETTINGS_MODULE=hct_mis_api.config.settings
-WORKDIR $CODE
+WORKDIR $APP_PATH
 
 COPY --from=waitforit /data/waitforit /usr/local/bin/waitforit
 
 # Dist builder image
-FROM base as pdm
+FROM base as uv
 RUN pip install --upgrade pip &&\
-    pip install pdm==2.19.2 &&\
-    pip install pdm-backend &&\
-    pip install setuptools==71.1.0 &&\
-    pdm config cache_dir /var/cache/pdm &&\
-    pdm config python.use_venv false &&\
-    pdm config venv.in_project true &&\
-    pdm config check_update false
-WORKDIR $PDM_PACKAGES
-COPY README.md LICENSE pyproject.toml pdm.lock ./
+    pip install uv &&\
+    pip install setuptools==71.1.0
+COPY README.md LICENSE pyproject.toml uv.lock MANIFEST.in ./
 
 
 
 
 # Dev image
-FROM pdm AS dev
-
+FROM uv AS dev
+ENV PYTHONPATH=$APP_PATH/src:$APP_PATH/test/:$PYTHONPATH
 RUN apt-get update \
   && apt-get install -y --no-install-recommends \
     python3-dev \
@@ -82,15 +69,17 @@ RUN apt-get update \
     dbus \
   && apt-get clean \
   && rm -rf /var/lib/apt/lists/*
-
-ENV PYTHONPATH=$CODE/src:$CODE/test/:$PYTHONPATH
-RUN pdm sync --no-editable --no-self --no-isolation
-
-WORKDIR $CODE
+WORKDIR $APP_PATH
 COPY ./src/ ./src/
+RUN --mount=type=cache,target=/root/.uv-cache \
+    uv sync --cache-dir=/root/.uv-cache \
+        --python=/usr/local/bin/python \
+        --python-preference=system \
+        --no-editable --frozen --extra distribution --no-install-package hope
 COPY ./tests ./tests
+COPY ./src/data /app/data
 COPY ./manage.py ./manage.py
-COPY .flake8 pyproject.toml pdm.lock ./
+COPY .flake8 pyproject.toml uv.lock MANIFEST.in ./
 COPY ./docker/entrypoint.sh /bin/
 ENTRYPOINT ["entrypoint.sh"]
 
@@ -106,20 +95,23 @@ COPY ./src/frontend ./
 RUN NODE_ENV="production" NODE_OPTIONS="--max-old-space-size=4096" yarn build
 
 # Dist builder image
-FROM pdm as dist-builder
+FROM uv as dist-builder
+WORKDIR $APP_PATH
 COPY ./src/ ./src/
-COPY --chown=hope:hope --from=frontend-builder /fe-build/build $PDM_PACKAGES/src/hct_mis_api/apps/web/static/web
-RUN pdm sync --prod --no-editable --no-isolation
+COPY --chown=hope:hope --from=frontend-builder /fe-build/build $APP_PATH/src/hct_mis_api/apps/web/static/web
+RUN uv sync --python=/usr/local/bin/python \
+        --python-preference=system \
+        --no-dev --no-editable --frozen --extra distribution
 
 
 ## Dist (backend only) image
 FROM base AS dist
 
 
 COPY ./src/gunicorn_config.py /conf/gunicorn_config.py
-COPY --chown=hope:hope --from=dist-builder $PDM_PACKAGES $PDM_PACKAGES
+COPY --chown=hope:hope --from=dist-builder $APP_PATH/.venv $APP_PATH/.venv
 COPY --chown=hope:hope --from=certs /data/psql-cert.crt /certs/psql-cert.crt
-
+COPY ./src/data /app/data
 USER hope
 
 COPY ./docker/entrypoint.sh /bin/

diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
@@ -1,6 +1,6 @@
 #!/bin/bash
 set -e
-
+source .venv/bin/activate
 wait_for_db() {
   until pg_isready -h $1 -p 5432;
   do echo "waiting for database ${1}"; sleep 2; done;