Skip to content
Search

GitLab:Upgrade

lastmod 2023-03-06created 2021-02-0327 edits

Upgrading GitLab

Upgrade path

Version upgrade path so far:

  • GitLab 8: 8.11.Z > 8.12.0 > 8.17.7
  • GitLab 9: 9.0.13 > 9.5.10
  • GitLab 10: 10.0.7 > 10.8.7
  • GitLab 11: 11.0.6 > 11.11.8
  • GitLab 12: 12.0.12 > 12.1.17 > 12.10.14
  • GitLab 13: 13.0.14 > 13.1.11 > 13.8.8 > 13.12.15
  • GitLab 14: 14.0.12 > 14.3.6 > 14.9.5 > 14.10.5
  • GitLab 15: 15.0.5 > 15.1.6 (for GitLab instances with multiple web nodes) > 15.4.6 > 15.6.x, 15.7.x or 15.8.x > latest 15.Y.Z (on hold for now!)

When moving up to the next major version, upgrade to the first version of that next major version.

Docker swarm upgrade

Upgrade following the order in #Upgrade path above. For example, if you are on 11.1.4, upgrade in the following order. 

docker service update --image gitlab/gitlab-ce:11.11.8-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:12.0.12-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:12.1.17-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:12.10.14-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:13.0.14-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:13.1.11-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:13.5.4-ce.0 gitlab_web

## 2023-03-05 진행:
docker service update --image gitlab/gitlab-ce:13.8.8-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:13.12.15-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:14.0.12-ce.0 gitlab_web

Important here !!!!

At each step, check that the Background Migrations have finished.

  1. Click the Admin button in the main menu at the top.
  2. In the left sidebar, select Monitoring > Background Migrations.

Wait for all running migrations to finish. (If you missed this check and something goes wrong, see the Troubleshooting section below.) 

docker service update --image gitlab/gitlab-ce:14.3.6-ce.0 gitlab_web

From here on, keep checking the Background Migrations at every step: 

docker service update --image gitlab/gitlab-ce:14.9.5-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:14.10.5-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:15.0.5-ce.0 gitlab_web

At this step, if the GitLab instance has multiple web nodes, you must go through the following update. Otherwise you can skip it. 

docker service update --image gitlab/gitlab-ce:15.1.6-ce.0 gitlab_web

Remaining upgrades: 

docker service update --image gitlab/gitlab-ce:15.4.6-ce.0 gitlab_web
docker service update --image gitlab/gitlab-ce:15.6.8-ce.0 gitlab_web

WARNING

Hold off on any further upgrades.

When each step upgrades successfully, a message similar to the following is printed. 

gitlab_web
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

Note that the Runner must be upgraded separately.

Troubleshooting

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active'

While upgrading from 14.0.12-ce.0 to 14.3.6-ce.0, the following error may occur. 

$ docker service update --image gitlab/gitlab-ce:14.3.6-ce.0 gitlab_web
gitlab_web
overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]
service update paused: update paused due to failure or early termination of task u2na696ul8f9iips268mcjgif

The container log looks like this: 

...

Running handlers:
There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20230305-29-1bjfrpx" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':   {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Finalize it manualy by running

    sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation

    https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':   {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}

Finalize it manualy by running

    sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']

For more information, check the documentation

    https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210622045705 FinalizeEventsBigintConversion: migrating ===================
STDERR:
---- End output of "bash"  "/tmp/chef-script20230305-29-1bjfrpx" ----
Ran "bash"  "/tmp/chef-script20230305-29-1bjfrpx" returned 1

Running handlers complete
Chef Infra Client failed. 131 resources updated in 59 seconds

Roll back again with docker service rollback gitlab_web. 

$ docker ps
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                    PORTS                      NAMES
b74968f619bf        gitlab/gitlab-ce:14.0.12-ce.0      "/assets/wrapper"        37 minutes ago      Up 37 minutes (healthy)   22/tcp, 80/tcp, 443/tcp    gitlab_web.1.yrce0pprgrinfsrnf381jiu71
...

After going back to version 14.0.12-ce.0, a 500 error page suddenly appears. To fix the error, check the Upgrading GitLab | GitLab # 14.0.0 page: 

14.0.5 또는 이후 14.0 패치 버전으로 업데이트한 후 이후 버전으로 업데이트하기 전에 [https://docs.gitlab.com/ee/update/background_migrations.html#batched-background-migrations 일괄 백그라운드 마이그레이션]을 완료해야 합니다.

If you try to update to the latest version before the migrations have finished, the following error is shown. 

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':

See how to resolve this error.

First, make sure you followed the version-specific upgrade instructions for 14.2. If you did, you can manually finish the batched background migration. <- I don't want to upgrade manually. -_- Too much hassle.

Otherwise, choose one of the following methods.

  1. Roll back and upgrade through one of the required versions before updating to 14.2+.
  2. Roll forward: stay on the current version and manually make sure the batched migrations complete successfully.

Roll back and follow the required upgrade path

  1. Roll back and restore the previously installed version (current state)
  2. Update to 14.0.5 or 14.1 before updating to 14.2+
  3. Check the status of the batched background migrations and make sure they are all marked as finished before attempting the upgrade again.

Roll forward and finish the migrations on the upgraded version.

My case

The current state is 14.0.12-ce.0, so update to 14.1+.

Checking [https://hub.docker.com/r/gitlab/gitlab-ce/tags?page=1&name=14.1. the 14.1.x versions available on Docker],

  • 14.1.8-ce.0
  • 14.1.7-ce.0
  • 14.1.6-ce.0
  • 14.1.5-ce.0
  • 14.1.4-ce.0
  • 14.1.3-ce.0
  • 14.1.2-ce.0
  • 14.1.1-ce.0
  • 14.1.0-ce.0

any one of these works. (In case I make another mistake, the 14.1+ condition has to be met) Upgrade to 14.1.0-ce.0 

docker service update --image gitlab/gitlab-ce:14.1.0-ce.0 gitlab_web
Gitlab_-Admin_Area-Monitoring-_Background_Migrations.png 
docker service update --image gitlab/gitlab-ce:14.3.6-ce.0 gitlab_web

If the output looks like the following, it's OK. 

gitlab_web
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

See also

Favorite site

author Zer0 sitemap