Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor product/ingredient import #1666

Merged
merged 39 commits into from
May 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
feb4540
Add option to download ingredients as JSONL from Open Food Facts (req…
strawpants Jan 1, 2024
fff7c86
update authors
strawpants Jan 1, 2024
f5fcae9
Merge branch 'master' into import_off_jsonl
rolandgeider Jan 23, 2024
193e6ca
Start working on import script for USDA products
rolandgeider Apr 21, 2024
42d2add
Fix path for OFF mongo dump
rolandgeider May 18, 2024
ebd9534
Add remote_id to ingredients.
rolandgeider May 18, 2024
741a3c7
If no ingredients were found the result is now None
rolandgeider May 18, 2024
d5a796e
Make USDA food import actually import the products
rolandgeider May 19, 2024
1a0ab64
Some polishing, add tests for the USDA extraction function
rolandgeider May 20, 2024
248f28b
Better handling branded products
rolandgeider May 20, 2024
53230ac
Download the dataset to a temporary folder by default
rolandgeider May 20, 2024
5aebcd2
Add method to try to fetch existing USDA entries
rolandgeider May 20, 2024
aa3f49b
Rename command line options
rolandgeider May 20, 2024
8c3ab81
Commit missing migration to also increase author field length in othe…
rolandgeider May 20, 2024
5eb3783
Merge branch 'refs/heads/master' into feature/import-usda-products
rolandgeider May 21, 2024
e4c75c4
Fix migration order
rolandgeider May 21, 2024
bff088e
Don't try to rename the index twice
rolandgeider May 21, 2024
b234b2e
fibre/fibres/fibers -> fiber
Dieterbe May 21, 2024
dc1a4d1
Some products have empty names, skip them
rolandgeider May 21, 2024
9aa837f
Further increase the author field length
rolandgeider May 21, 2024
e4a6f7e
Increase number of returned results in the API ingredient search
rolandgeider May 21, 2024
a6733be
Merge branch 'refs/heads/fork/import_off_jsonl' into feature/import-u…
rolandgeider May 22, 2024
3dbeb62
Refactor import scripts
rolandgeider May 22, 2024
fdca08d
Make sync of OFF daily delta files a celery task
rolandgeider May 22, 2024
69b53e5
Refactor and remove hard coded paths
rolandgeider May 22, 2024
e3aa521
Move some common data checks to the IngredientData class
rolandgeider May 22, 2024
352c19b
Merge remote-tracking branch 'refs/remotes/origin/term-consistency' i…
rolandgeider May 22, 2024
234cef4
Merge the branch term-consistency
rolandgeider May 22, 2024
91ebb8c
Merge branch 'master' into feature/import-usda-products
rolandgeider May 23, 2024
e3214eb
Consolidate fiber related migrations into one
rolandgeider May 23, 2024
15a5bc0
Bump minimum app version
rolandgeider May 23, 2024
88af124
Revert "Consolidate fiber related migrations into one"
rolandgeider May 23, 2024
5349446
Remove the status for ingredients
rolandgeider May 24, 2024
6381dbb
Update all remote IDs at once
rolandgeider May 24, 2024
5e79936
Fix test
rolandgeider May 24, 2024
6ee150f
Remove unused imports
rolandgeider May 28, 2024
0dfb912
Remove one-off method, this is not needed anymore
rolandgeider May 29, 2024
5b6b035
Seems it's possible for a product to not have "lang" key
rolandgeider May 29, 2024
fde12ad
Reformat and remove unused imports
rolandgeider May 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions AUTHORS.rst
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,11 @@ Developers
* Gabriel Liss - https://github.com/gabeliss
* Alexandra Rhodes - https://github.com/arhodes130
* Jayanth Bontha - https://github.com/JayanthBontha
* Roelof Rietbroek https://github.com/strawpants
* Ethan Winters - https://github.com/ebwinters
* Dieter Plaetinck - https://github.com/Dieterbe


Translators
-----------

Expand Down
1 change: 1 addition & 0 deletions extras/docker/development/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@
WGER_SETTINGS["SYNC_EXERCISE_IMAGES_CELERY"] = env.bool("SYNC_EXERCISE_IMAGES_CELERY", False)
WGER_SETTINGS["SYNC_EXERCISE_VIDEOS_CELERY"] = env.bool("SYNC_EXERCISE_VIDEOS_CELERY", False)
WGER_SETTINGS["SYNC_INGREDIENTS_CELERY"] = env.bool("SYNC_INGREDIENTS_CELERY", False)
WGER_SETTINGS["SYNC_OFF_DAILY_DELTA_CELERY"] = env.bool("SYNC_OFF_DAILY_DELTA_CELERY", False)
WGER_SETTINGS["USE_RECAPTCHA"] = env.bool("USE_RECAPTCHA", False)
WGER_SETTINGS["USE_CELERY"] = env.bool("USE_CELERY", False)

Expand Down
2 changes: 1 addition & 1 deletion extras/open-food-facts/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ services:
ports:
- "27017:27017"
volumes:
- $PWD/dump:/dump
- $PWD/dump/dump:/dump
environment:
MONGO_INITDB_ROOT_USERNAME: off
MONGO_INITDB_ROOT_PASSWORD: off-wger
Expand Down
4 changes: 2 additions & 2 deletions wger/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
from .celery_configuration import app


MIN_APP_VERSION = (1, 6, 0, 'final', 1)
MIN_APP_VERSION = (1, 7, 3, 'final', 1)

VERSION = (2, 3, 0, 'alpha', 1)
VERSION = (2, 3, 0, 'alpha', 2)
RELEASE = True


Expand Down
51 changes: 30 additions & 21 deletions wger/core/templates/navigation.html
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
<div class="container-fluid">
<div class="container-lg" style="display: flex;">
<a class="navbar-brand" href="{% url 'core:index' %}">
<img src="{% static 'images/logos/logo-bg-white.png' %}" style="width: 38px;" alt="wger logo">
<img src="{% static 'images/logos/logo-bg-white.png' %}" style="width: 38px;"
alt="wger logo">
</a>
<button class="navbar-toggler"
type="button"
Expand All @@ -24,18 +25,21 @@
<a href="{% url 'manager:workout:overview' %}"
rel="nofollow"
class="nav-link dropdown-toggle text-white {% if active_tab == 'workout' %}active{% endif %}"
data-bs-toggle="dropdown">{% translate "Training" %} <b class="caret"></b></a>
data-bs-toggle="dropdown">{% translate "Training" %} <b
class="caret"></b></a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="{% url 'manager:workout:overview' %}" rel="nofollow">
<a class="dropdown-item" href="{% url 'manager:workout:overview' %}"
rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
{% translate "Workouts" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'manager:schedule:overview' %}" rel="nofollow">
<a class="dropdown-item"
href="{% url 'manager:schedule:overview' %}" rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
Expand Down Expand Up @@ -63,7 +67,8 @@
{% endif %}

<li>
<a class="dropdown-item" href="{% url 'gallery:images:overview' %}" rel="nofollow">
<a class="dropdown-item" href="{% url 'gallery:images:overview' %}"
rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
Expand All @@ -75,15 +80,17 @@
<li class="dropdown-divider"></li>
<li class="dropdown-header">{% translate "Workout templates" %}</li>
<li>
<a class="dropdown-item" href="{% url 'manager:template:overview' %}" rel="nofollow">
<a class="dropdown-item"
href="{% url 'manager:template:overview' %}" rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
{% translate "Your templates" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'manager:template:public' %}" rel="nofollow">
<a class="dropdown-item" href="{% url 'manager:template:public' %}"
rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
Expand All @@ -93,7 +100,8 @@

<li class="dropdown-divider"></li>
<li>
<a class="dropdown-item" href="{% url 'exercise:exercise:overview' %}">
<a class="dropdown-item"
href="{% url 'exercise:exercise:overview' %}">
{% translate "Exercises" %}
</a>
</li>
Expand All @@ -102,22 +110,26 @@
<li class="dropdown-divider"></li>
<li class="dropdown-header">{% translate "Administration" %}</li>
<li>
<a class="dropdown-item" href="{% url 'exercise:history:overview' %}">
<a class="dropdown-item"
href="{% url 'exercise:history:overview' %}">
{% translate "Exercise edit history" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'exercise:equipment:list' %}">
<a class="dropdown-item"
href="{% url 'exercise:equipment:list' %}">
{% translate "Equipment" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'exercise:muscle:admin-list' %}">
<a class="dropdown-item"
href="{% url 'exercise:muscle:admin-list' %}">
{% translate "Muscles" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'exercise:category:list' %}">
<a class="dropdown-item"
href="{% url 'exercise:category:list' %}">
{% translate "Categories" %}
</a>
</li>
Expand All @@ -136,23 +148,26 @@
rel="nofollow">{% translate "Nutrition" %} <b class="caret"></b></a>
<ul class="dropdown-menu">
<li>
<a class="dropdown-item" href="{% url 'nutrition:plan:overview' %}" rel="nofollow">
<a class="dropdown-item" href="{% url 'nutrition:plan:overview' %}"
rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
{% translate "Nutrition plans" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'nutrition:bmi:view' %}" rel="nofollow">
<a class="dropdown-item" href="{% url 'nutrition:bmi:view' %}"
rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
{% translate "BMI calculator" %}
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'nutrition:calories:view' %}" rel="nofollow">
<a class="dropdown-item" href="{% url 'nutrition:calories:view' %}"
rel="nofollow">
{% if not user.is_authenticated %}
<span class="{% fa_class 'lock' %}"></span>
{% endif %}
Expand All @@ -171,12 +186,6 @@
<li><a class="dropdown-item"
href="{% url 'nutrition:weight_unit:list' %}">{% translate "Ingredient weight units" %}</a>
</li>
<li>
<a class="dropdown-item"
href="{% url 'nutrition:ingredient:pending' %}">
{% translate "Ingredients pending review" %}
</a>
</li>
{% endif %}
</ul>
</li>
Expand Down
2 changes: 1 addition & 1 deletion wger/core/tests/test_sitemap.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,4 @@ def test_sitemap_ingredients(self):
reverse('django.contrib.sitemaps.views.sitemap', kwargs={'section': 'nutrition'})
)
self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.context['urlset']), 13)
self.assertEqual(len(response.context['urlset']), 14)
100 changes: 100 additions & 0 deletions wger/exercises/migrations/0030_increase_author_field_length.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Generated by Django 4.2.6 on 2024-05-20 19:40

from django.db import migrations, models


class Migration(migrations.Migration):
dependencies = [
('exercises', '0029_full_text_search'),
]

operations = [
migrations.AlterField(
model_name='exercise',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='exercisebase',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='exerciseimage',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='exercisevideo',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='historicalexercise',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='historicalexercisebase',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='historicalexerciseimage',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
migrations.AlterField(
model_name='historicalexercisevideo',
name='license_author',
field=models.CharField(
blank=True,
help_text='If you are not the author, enter the name or source here.',
max_length=3500,
null=True,
verbose_name='Author(s)',
),
),
]
3 changes: 1 addition & 2 deletions wger/nutrition/api/filtersets.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,10 @@ class Meta:
'energy': ['exact'],
'fat': ['exact'],
'fat_saturated': ['exact'],
'fibres': ['exact'],
'fiber': ['exact'],
'name': ['exact'],
'protein': ['exact'],
'sodium': ['exact'],
'status': ['exact'],
'language': ['exact'],
'license': ['exact'],
'license_author': ['exact'],
Expand Down
11 changes: 6 additions & 5 deletions wger/nutrition/api/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ class Meta:
fields = [
'id',
'uuid',
'remote_id',
'code',
'name',
'created',
Expand All @@ -124,7 +125,7 @@ class Meta:
'carbohydrates_sugar',
'fat',
'fat_saturated',
'fibres',
'fiber',
'sodium',
'license',
'license_title',
Expand Down Expand Up @@ -160,7 +161,7 @@ class Meta:
'carbohydrates_sugar',
'fat',
'fat_saturated',
'fibres',
'fiber',
'sodium',
'weight_units',
'language',
Expand Down Expand Up @@ -265,7 +266,7 @@ class NutritionalValuesSerializer(serializers.Serializer):
carbohydrates_sugar = serializers.FloatField()
fat = serializers.FloatField()
fat_saturated = serializers.FloatField()
fibres = serializers.FloatField()
fiber = serializers.FloatField()
sodium = serializers.FloatField()


Expand Down Expand Up @@ -312,7 +313,7 @@ class Meta:
'goal_protein',
'goal_carbohydrates',
'goal_fat',
'goal_fibers',
'goal_fiber',
# 'nutritional_values',
]

Expand All @@ -336,6 +337,6 @@ class Meta:
'goal_protein',
'goal_carbohydrates',
'goal_fat',
'goal_fibers',
'goal_fiber',
'meals',
]
7 changes: 3 additions & 4 deletions wger/nutrition/api/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ def list(self, request, *args, **kwargs):

def get_queryset(self):
"""H"""
qs = Ingredient.objects.accepted()
qs = Ingredient.objects.all()

code = self.request.query_params.get('code')
if not code:
Expand Down Expand Up @@ -129,7 +129,7 @@ def get_values(self, request, pk):
'carbohydrates_sugar': 0,
'fat': 0,
'fat_saturated': 0,
'fibres': 0,
'fiber': 0,
'sodium': 0,
'errors': [],
}
Expand Down Expand Up @@ -219,7 +219,6 @@ def search(request):
languages = [load_language(l) for l in language_codes.split(',')]
query = Ingredient.objects.filter(
language__in=languages,
status=Ingredient.STATUS_ACCEPTED,
).only('name')

# Postgres uses a full-text search
Expand All @@ -232,7 +231,7 @@ def search(request):
else:
query = query.filter(name__icontains=term)

for ingredient in query[:100]:
for ingredient in query[:150]:
if hasattr(ingredient, 'image'):
image_obj = ingredient.image
image = image_obj.image.url
Expand Down
Loading
Loading