Increasing timeout for migrations in GitLab 16.2 to 16.3 update

Troubleshooting
1

I’ve been trying for the past few months to update from GitLab-16.2.x to 16.3.x, this version contains an update to Mattermost-8.x (from 7.x) and the database migrations are failing for this so after the update Mattermost doesn’t come back up.

The issue we’re having is the same as in this GitLab ticket:

In that ticket is is suggested to increase the PostgreSQL timeouts by adding the following to the gitlab.rb file:

#postgresql
postgresql['statement_timeout'] = "600000"
postgresql['idle_in_transaction_session_timeout'] = "600000"

After doing this the migrations still timeout, and it looks like the increased timeouts aren’t making their way through to the migration process.

How can we increase the timeouts to allow the migrations time to go through?

2

Hello Adam,

Thank you for reaching out on the Mattermost Forum. I understand the frustration while dealing with database migrations and running into this timeout issue.

Based on the information you’ve provided, it seems like you’ve increased PostgreSQL timeouts in the gitlab.rb file and the migrations are still timing out. It’s odd that the increased timeouts aren’t taken into consideration. I wonder if it’s a problem with the propagation of these new values to the database.

You might want to double-check if you reconfigured GitLab after changing these values by running sudo gitlab-ctl reconfigure. This would pick up the new changes and apply them.

Also, it might be helpful to observe PostgreSQL logs during this process to gather more detailed information about which specific query or operation is causing the timeout. This would help in further troubleshooting the issue.

Make sure you back up your data and configuration files regularly before any major version updates. That way, you’ll ensure the safety of your data.

While we’re at it, I’d advise you to also check out our upgrade guide for Mattermost, which provides best practices and tips for smoother upgrades.

If you continue to have issues or need further assistance, don’t hesitate to ask. We’re here to ensure you have a smooth and seamless experience with Mattermost.

Best,
~Mattermost AI Assistant :robot:

[Disclaimer: This was an AI assisted response powered by Mattermost AI. Were any of the above suggestions inaccurate? Let us know by replying to this comment!]

3

If possible, restore the GitLab 16.2 state from backup first. Then, after a gitlab-ctl reconfigure, before trying to run the upgrade to GitLab 16.3 again, can you open a second terminal, run gitlab-ctl tail postgresql, start the upgrade in the first terminal and post what’s going on with the database?

1 Like
4

We restored from our backups back to 16.2 as soon as we noticed the failed migrations. We’re currently looking at splitting Mattermost out to run it on a separate server as this isn’t the first time we’ve had Mattermost migration issues as part of a GitLab update. If we attempt it again, instead of splitting out Mattermost, I’ll definitely tail the postgres logs.

1 Like
5

We ended up just splitting Mattermost out from GitLab and running it as a separate server.