Skip to content

Latest commit

 

History

History

ba-torment-batch

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
parse
parse
 
 
utils
utils
 
 
README.md
README.md
 
 
constants.py
constants.py
 
 
logo.png
logo.png
 
 
main.py
main.py
 
 

README.md

Logo

Data processing code for BA Torment

Bug Report | Request

Python Poetry Polars Supabase


Overview

Data processing code for BA Torment.
It is processed on my local machine in schedule.
Most of the processed data is updated to the Oracle Object Storage, and the rest is updated to the PostgreSQL database powered by Supabase.


History

Period Description
2023.07 ~ 2024.03 Used python script to process data
2024.03 ~ 2024.10 Extended the data schema to serve more information,
configured auto-migration via GitHub Actions
2024.10 ~ Migrated data to Oracle Object Storage & Supabase,
deployed the backend

Major data reference

Data source Description
Schale DB To extract character data
info.herdatasam.me To extract party data
Arona.AI 1 To validate the data + get additional information

Prerequisites

  1. You need to install Python 3.12 and Poetry to manage the packages.

  2. You need to set environment variables to run the script.

    POSTGRES_HOST=
POSTGRES_PORT=
POSTGRES_USER=
POSTGRES_PASSWORD=
POSTGRES_DBNAME=
BATORMENT_UPLOAD_URL="presigned-url-to-upload-images"

Run the script locally

Activate the virtual environment

poetry shell

Install packages
This will install the required packages in the whole repository.

poetry install

Run the script
If you want to run the script manually, run ba-torment-batch/main.py.


Contributing

See the CONTRIBUTING.md.



Footnotes

  1. The developer of Arona.AI is same as the developer of info.herdatasam.me. ↩