Research upgrading to Django 2.2 LTS version

[patphongs]

Last time it was attempted to upgrade from Django 1 to Django 2, stylesheets were not loading and grunt was not working. Parsing was working.

Need to consider these packages when upgrading: regulations-site, regulations-parser, regulations-core

• Django is updated to Version 2.2 with all functionality working

Related issue #626

[pkfec]

With django2.2.18:
python manage.py migrate command run only with django-mptt~=0.8.6. At runtime the templates fail to load the regulations page with the following error:

2021-10-26T21:47:59.79-0400 [APP/PROC/WEB/0] ERR django.template.library.InvalidTemplateLibrary: Invalid template library specified. ImportError raised when trying to load 'mptt.templatetags.mptt_admin': cannot import name 'RemovedInDjango20Warning' from 'django.utils.deprecation' (/home/vcap/deps/0/python/lib/python3.7/site-packages/django/utils/deprecation.py)

once the db migrations are run on local, switch the django-mptt==0.11.0 in requirements.txt file and install the new requirements, start the server. The templates load OK.

[pkfec]

[pkfec]

[johnnyporkchops]

As a workaround, we can add the {% spaceless %}...{% endspaceless %} tags the the parent templates or around certain blocks to remove the newlines (\n\n etc). This was tested on a branch and works almost completely except we were unable to completely remove newlines from the sidebar, but this could probably be figured out if necessary.

[pkfec]

python manage.py migrate errors:

(20211018-eregs-env) macadmins-mbp-5:fec-eregs pkasireddy$ python manage.py migrate
System check identified some issues:

WARNINGS:
regcore.Diff: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
regcore.Layer: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
regcore.NoticeCFRPart: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the AppConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.
Operations to perform:
Apply all migrations: auth, contenttypes, regcore, regulations, sessions
Running migrations:
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying auth.0012_alter_user_first_name_max_length... OK
Applying regcore.0001_initial... OK
Applying regcore.0002_mptt_add_fields... OK
Applying regcore.0003_mptt_copy_children...Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/core/management/init.py", line 419, in execute_from_command_line
utility.execute()
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/core/management/init.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/core/management/base.py", line 89, in wrapped
res = handle_func(*args, **kwargs)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 246, in handle
fake_initial=fake_initial,
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
state = migration.apply(state, schema_editor)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/db/migrations/migration.py", line 126, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
self.code(from_state.apps, schema_editor)
File "/Users/pkasireddy/.pyenv/versions/3.7.10/envs/20211018-eregs-env/src/regcore/regcore/migrations/0003_mptt_copy_children.py", line 17, in rebuild
mptt.register(Regulation)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/mptt/init.py", line 13, in register
return MPTTModelBase.register(*args, **kwargs)
File "/Users/pkasireddy/.pyenv/versions/20211018-eregs-env/lib/python3.7/site-packages/mptt/models.py", line 352, in register

    cls._meta.index_together += (index_together,)
TypeError: unsupported operand type(s) for +=: 'set' and 'tuple'

[pkfec]

To resolve the TypeError: unsupported operand type(s) for +=: 'set' and 'tuple', @johnnyporkchops added custom code to convert a set into a tuple inside a 3rd party package(site-packages/mptt/models.py)

    def convert(set):
        return tuple(i for i in set)

For testing purpose, forked django-mptt pkg to pkfec (later will fork to fecgov org)and monkey patched django-mptt/model.py

Here are the code changes:

1. https://github.com/pkfec/django-mptt/blob/7ce309569d08a174275d93da49fe36a94af0e14b/mptt/models.py#L317
2. https://github.com/pkfec/django-mptt/blob/7ce309569d08a174275d93da49fe36a94af0e14b/mptt/models.py#L384

[pkfec]

Instead of monkey patching django-mptt, removed alterindextogether and alteruniquetogether regcore/migrations that instantly fixed the TypeError: unsupported operand type(s) for +=: 'set' and 'tuple'

code changes here#
fecgov/regulations-core@4465793

[johnnyporkchops]

CF logs show that ,with the above fix, all migrations are applied without error, but gives Warning:

Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

cf logs eregs ouput on dev:

Creating a new migration(0016) by running makemigrations breaks the Circle build when deployed with relation exists error...
But tests on dev have shown that instead, just editing migration 0011 to add this operation under the commented code, then rebuilding the app passes Circle build and removes the warning
Around line 44, under commented out alterIndexTogethor, add:

        migrations.AlterUniqueTogether(
            name='document',
            unique_together={('doc_type', 'version', 'label_string')},
        ),
         migrations.AlterIndexTogether(
             name='document',
             index_together={('doc_type', 'version', 'label_string')},
         ),

Please see our issue in django-mptt repo issue for more info on final solution:
django-mptt/django-mptt#803

[pkfec]

Here is the eregs-django 2.2.24 WIP pr

[dorothyyeager]

We're going to go forward and implement this.

[pkfec]

The django upgrade implementation can be tracked here #632. Closing the research ticket.