Unable to upgrade from 16.1.5

[LewS]

I've successfully upgraded a 12.4.2 clone to 16.1.5 with an aim to upgrade beyond to the latest, however whenever I change the docker-gitlab version above 16.1.5 we encounter a previously described error in the db migration script in...
"6 is not a known status value
/home/git/gitlab/vendor/bundle/ruby/3.0.0/gems/state_machines-0.5.0/lib/state_machines/state_collection.rb:78:in `match!'"
This was previously described in #2938 which was closed with reference to it already being listed in #2944, however the later details the inability to upgrade from 15.7.0 to 17.0.0 with reference to postgres, whereas #2938 details gitlab 16.1.6 to 16.3.7; my problem is to upgrade from 16.1.5 to anything higher (aside from the missing 16.1.6).
16.1.5 is running with a migrated pg 15 running via aws RDS.
Are we able to get some clarity on how we are getting a status value that is not known, and what we can do to move past it?

[FluffyDiscord]

Same issue here

[FluffyDiscord]

@LewS it took some time experimenting , but I successfully migrated from ancient source-installed Gitlab to Docker latest - 1. Posting here what I did as a reference:

From	Target	#checkout_source_tag	Note	Time
13.5.3	13.12.15	sameersbn/gitlab:13.12.4		30m
13.12.15	14.1.8	sameersbn/gitlab:14.1.1	#14_2_5	2h
14.1.8	14.2.7	sameersbn/gitlab:14.2.5	#14_2_5	30m
14.2.7	14.3.6	sameersbn/gitlab:14.3.3		30m
14.3.6	14.5.4	sameersbn/gitlab:14.5.2		30m
14.5.4	14.6.7	sameersbn/gitlab:14.6.3		30m
14.6.7	14.8.6	sameersbn/gitlab:14.8.4		30m
14.8.6	14.9.5	sameersbn/gitlab:14.9.3		30m
14.9.5	15.0.5	sameersbn/gitlab:15.0.3		1h
15.0.5	15.4.6		#15_4_6 16 migrations	1h*
15.4.6	15.11.13		24 migrations 🤦	2h
15.11.13	16.3.7	sameersbn/gitlab:16.3.6	Before updating Gitlab image version, first update database image from 12-20200524 to 14-20230628 and wait for psql to upgrade.	1h
16.3.7	16.7.8	sameersbn/gitlab:16.7.7		2h
16.7.8	16.11.5	sameersbn/gitlab:16.11.4		1h
16.11.5	17.1.1			1h

#wait-until-background-migrations-are-done after each update.

#checkout_source_tag

The target image is not available at sameersbn/gitlab. The workaround is rather easy, but time consuming. You need to build the Docker images locally on your machine, where you have your Gitlab instance. Each build took me around 2000 seconds, YMMV.

Steps:

1. mkdir sammeersbn_git
2. cd sammeersbn_git
3. git clone git@github.com:sameersbn/docker-gitlab.git . (or use https if required)
4. git restore .
5. git checkout 13.12.4 (the 13.12.4 is the source tag you want to use)
6. nano Dockerfile
7. change ARG VERSION=13.12.4 (at the top) and GITALY_SERVER_VERSION=13.12.4 (slightly lower from the version arg) to the target image version, in this case to 13.12.15
8. apply #psql_repo_fix if needed
9. save and exit
10. docker build -t sameersbn/gitlab:13.12.15 . where 13.12.15 is the target image version
11. take a ~2000 seconds break

Now repeat steps 4 to 11 whenever image is missing.

#wait-until-background-migrations-are-done

After bringing the Gitlab Docker container up, go to Admin -> Monitoring -> Background Migrations. Wait until you no longer have error 502, and then until every migration is done.

#psql_repo_fix

Older Gitlab images use removed PostgreSQL Ubuntu repository. Not a problem. You need to replace it in Dockerfile - change deb http://apt.postgresql.org/pub/repos/apt/ focal-pgdg main to deb https://apt-archive.postgresql.org/pub/repos/apt bionic-pgdg main .

#14_2_5

After Background Migrations are done:

1. Enter the container docker compose exec -it gitlab bash
2. Run each one, wait for green Done
3. bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
4. bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds,id,'[["id"\, "stage_id"]\, ["id_convert_to_bigint"\, "stage_id_convert_to_bigint"]]']
5. bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_builds_metadata,id,'[["id"]\, ["id_convert_to_bigint"]]']
6. bundle exec rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,taggings,id,'[["id"\, "taggable_id"]\, ["id_convert_to_bigint"\, "taggable_id_convert_to_bigint"]]']
   If you miss this, upgrading further will be impossible. You will break your Gitlab instance.

#15_4_6

Wait until said migration is in process, then:

• BackfillProjectNamespaceOnIssues bundle exec rake gitlab:background_migrations:finalize[BackfillProjectNamespaceOnIssues,issues,id,'[]']
• RemoveBackfilledJobArtifactsExpireAt bundle exec rake gitlab:background_migrations:finalize[RemoveBackfilledJobArtifactsExpireAt,ci_job_artifacts,id,'[]']
• BackfillProjectImportLevel bundle exec rake gitlab:background_migrations:finalize[BackfillProjectImportLevel,namespaces,id,'[]']

References

https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/?current=13.4.7&distro=docker&edition=ce&n1=true
#2409 (comment)
https://stackoverflow.com/a/77443882

[vegarberget]

I found out why I got "6 is not a known status value" for me there was two migrations that had something wrong in the status field. I changed status to 5 on both in postgresql and then I could upgrade past 16.1.5.

[oboote]

#2954 (comment)

OMG I could kiss you - THIS! - exactly what we needed.

Thank you.

Note that the steps shared above don't quite match recommended upgrade path from https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/

Succeeded with the following:

These require #psql_repo_fix:
sameersbn/gitlab:14.0.12 < (need to build from 14.0.6 with #psql_repo_fix)
sameersbn/gitlab:14.3.6 < (need to build from 14.3.3 with #psql_repo_fix)
sameersbn/gitlab:14.9.5 < exists in dockerhub
sameersbn/gitlab:14.10.5 < exists in dockerhub
sameersbn/gitlab:15.0.5 < (need to build from 15.0.3 with #psql_repo_fix)
sameersbn/gitlab:15.4.6 < exists in dockerhub
sameersbn/gitlab:15.11.13 < exists in dockerhub

Here onwards no longer need #psql_repo_fix (build will fail)
#upgrade postgreSQL to >=13.x (can jump straight to 15)
sameersbn/gitlab:16.3.9 < (needs building from 16.3.6)
sameersbn/gitlab:16.7.10 < (needs building from 16.7.7)
sameersbn/gitlab:16.11.10 < (needs building from 16.11.6)
#upgrade postgreSQL to >=14.x (can jump straight to 15)
sameersbn/gitlab:17.3.6 < exists in dockerhub
sameersbn/gitlab:17.5.1 < exists in dockerhub