Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor product/ingredient import #1666

Merged
merged 39 commits into from
May 29, 2024
Merged

Commits on Jan 1, 2024

  1. Configuration menu
    Copy the full SHA
    feb4540 View commit details
    Browse the repository at this point in the history
  2. update authors

    strawpants committed Jan 1, 2024
    Configuration menu
    Copy the full SHA
    fff7c86 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2024

  1. Configuration menu
    Copy the full SHA
    f5fcae9 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2024

  1. Configuration menu
    Copy the full SHA
    193e6ca View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    42d2add View commit details
    Browse the repository at this point in the history
  3. Add remote_id to ingredients.

    This allows to properly update them during imports. Currently, we were using the
    code field for OFF products but e.g. USDA products have their own id
    rolandgeider committed May 18, 2024
    Configuration menu
    Copy the full SHA
    ebd9534 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    741a3c7 View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. Configuration menu
    Copy the full SHA
    d5a796e View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. Configuration menu
    Copy the full SHA
    1a0ab64 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    248f28b View commit details
    Browse the repository at this point in the history
  3. Download the dataset to a temporary folder by default

    Also, add some error handling for missing folders, urls, etc.
    rolandgeider committed May 20, 2024
    Configuration menu
    Copy the full SHA
    53230ac View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    5aebcd2 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    aa3f49b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    8c3ab81 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Configuration menu
    Copy the full SHA
    5eb3783 View commit details
    Browse the repository at this point in the history
  2. Fix migration order

    rolandgeider committed May 21, 2024
    Configuration menu
    Copy the full SHA
    e4c75c4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bff088e View commit details
    Browse the repository at this point in the history
  4. fibre/fibres/fibers -> fiber

    Dieterbe committed May 21, 2024
    Configuration menu
    Copy the full SHA
    b234b2e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    dc1a4d1 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9aa837f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e4a6f7e View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. Merge branch 'refs/heads/fork/import_off_jsonl' into feature/import-u…

    …sda-products
    
    # Conflicts:
    #	wger/nutrition/management/commands/import-off-products.py
    rolandgeider committed May 22, 2024
    Configuration menu
    Copy the full SHA
    a6733be View commit details
    Browse the repository at this point in the history
  2. Refactor import scripts

    Now it is possible to import the OFF dumps or delta files directly from the
    JSONL files, without needing to import them to mongoDB
    rolandgeider committed May 22, 2024
    Configuration menu
    Copy the full SHA
    3dbeb62 View commit details
    Browse the repository at this point in the history
  3. Make sync of OFF daily delta files a celery task

    This can run automatically once per day if configured
    rolandgeider committed May 22, 2024
    Configuration menu
    Copy the full SHA
    fdca08d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    69b53e5 View commit details
    Browse the repository at this point in the history
  5. Move some common data checks to the IngredientData class

    This is specially important if we regularly import the delta files, since these
    can have bad quality. We will probably need to add a bit more before putting
    this live
    rolandgeider committed May 22, 2024
    Configuration menu
    Copy the full SHA
    e3aa521 View commit details
    Browse the repository at this point in the history
  6. Merge remote-tracking branch 'refs/remotes/origin/term-consistency' i…

    …nto feature/import-usda-products
    
    # Conflicts:
    #	wger/nutrition/off.py
    rolandgeider committed May 22, 2024
    Configuration menu
    Copy the full SHA
    352c19b View commit details
    Browse the repository at this point in the history
  7. Merge the branch term-consistency

    Also, make sure just rename the field and don't accidentally nuke the fibre field ;)
    rolandgeider committed May 22, 2024
    Configuration menu
    Copy the full SHA
    234cef4 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Configuration menu
    Copy the full SHA
    91ebb8c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e3214eb View commit details
    Browse the repository at this point in the history
  3. Bump minimum app version

    Needed because the change of "fibres" to "fiber"
    rolandgeider committed May 23, 2024
    Configuration menu
    Copy the full SHA
    15a5bc0 View commit details
    Browse the repository at this point in the history
  4. Revert "Consolidate fiber related migrations into one"

    This reverts commit e3214eb.
    rolandgeider committed May 23, 2024
    Configuration menu
    Copy the full SHA
    88af124 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Remove the status for ingredients

    Ingredients / products will only be imported through automatic processes. We
    should just point users in case of missing products to Open Food Facts, and not
    handle this on our side. Admins can still add entries if they want.
    rolandgeider committed May 24, 2024
    Configuration menu
    Copy the full SHA
    5349446 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6381dbb View commit details
    Browse the repository at this point in the history
  3. Fix test

    rolandgeider committed May 24, 2024
    Configuration menu
    Copy the full SHA
    5e79936 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Remove unused imports

    rolandgeider committed May 28, 2024
    Configuration menu
    Copy the full SHA
    6ee150f View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Configuration menu
    Copy the full SHA
    0dfb912 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5b6b035 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    fde12ad View commit details
    Browse the repository at this point in the history