Upgrading to GitLab Mattermost in GitLab 8.9

This is a thread started to help troubleshoot issues with upgrading to GitLab 8.9, which upgrades users to Mattermost 3.1.

If you’re having issues with the automated install, try running:

sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30

If this doesn’t work, please post a new topic to this forum with:

  1. A brief description of your issue as title
  2. The step on which you’re having issues
  3. The relevant portion of the Mattermost log file at /var/log/gitlab/mattermost/mattermost.log

To start a new topic, click this link to the Gitlab Mattermost forum category then click New Topic. See bottom right button in screenshot for an illustration:

Thanks for the topic!
How can i login with a renamed user/account when the login method is GitLab SSO?
I can’t login to the other teams anymore, because the renamed users/accounts have no credentials like username/passwort.

Hi. When upgrading gitlab to 8.9 mattermost was not started.
in /etc/gitlab/gitlab.rbi’ve added
mattermost[‘db2_backup_created’] = true
mattermost[‘db2_team_name’] = “TEAMNAME” with proper teamname.

/var/log/gitlab/mattermost/mattermost.log :

   [2016/06/22 09:24:03 MSK] [EROR] /havana/town-square: code=404 ip=
   !!!!Automatic database upgrade failed.!!!

    If you are upgrading from Mattermost v2 to v3
    make sure that you have backed up your database
    and then in /etc/gitlab/gitlab.rb set:

    mattermost['db2_backup_created'] = true
    mattermost['db2_team_name'] = TEAMNAME

    where TEAMNAME is the name of the default team.
    Run gitlab-ctl reconfigure again.


    for more information.

and that’s all.
I was trying to manual run commands above, but get

 $ /opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: interface conversion: interface is runtime.errorString, not string
[signal 0xb code=0x1 addr=0x0 pc=0x78090f]

goroutine 1 [running]:
panic(0xcd1aa0, 0xc820140180)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:73 +0x6f
panic(0xccbec0, 0xc8200100e0)
        /usr/local/go/src/runtime/panic.go:443 +0x4e9
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/utils/mail.go:82 +0x9f
github.com/mattermost/platform/utils.LoadConfig(0x7fff53bf3794, 0x26)
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:181 +0x10aa
main.doLoadConfig(0x7fff53bf3794, 0x26, 0x0, 0x0)
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:76 +0x6a
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:84 +0x43

Please help.

1 Like

How exactly do I back up the database? I can’t find any documentation other than “just do what you normally do with postgres databases,” but I don’t normally do that kind of thing :confused:

Hey guys!

I’m having trouble upgrading mattermost.

I added the config parameters, ran apt-get upgrade, it worked fine but now mattermost won’t start.

The log says:

Error creating database tables: pq: trigger "truncate_trigger" for relation "teams" already exists

So I ran

sudo -u mattermost -i bash cd /opt/gitlab/embedded/service/mattermost /opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30

Which returned …nothing and did not help.

Any ideas?

Hi @manuel, duplicate accounts are switched to email login, you can use “reset password” to get credentials sent to your email, and you can switch to GitLab SSO under Account Settings > Security Sign In Method.

Hi @revxx14, try pg_dump dbname > outfile, you can just web search for your database type and version to find documentation for backup, for example: https://www.postgresql.org/docs/9.4/static/backup-dump.html

Hi @Korni22, Mattermost doesn’t use database triggers, did you or someone on your team manually modify the Mattermost database to add triggers?

I’ve just update my gitlab installation using apt-get upgrade. My gitlab install work well but the mattermost embedded doesn’t work any more.
I check log, and mattermost ask me to upgrade the mattermost database.

[2016/06/22 10:01:12 EDT] [CRIT] The database version of 2.2.0 cannot be automatically upgraded to 3.0 schema
[2016/06/22 10:01:12 EDT] [CRIT] You will need to run the command line tool ‘./platform -upgrade_db_30’

So i launch this command :

mattermost@Kalay:/opt/gitlab/embedded/bin$ /opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30

Please see http://www.mattermost.org/upgrade-to-3-0/
WARNING This upgrade process will be irreversible.
Have you performed a database backup? (YES/NO): YES
We found 14 teams.

Please pick a primary team from the list above: datexim
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x4a6653]

goroutine 1 [running]:
panic(0xccbec0, 0xc82000e090)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
html/template.(*Template).lookupAndEscapeTemplate(0x0, 0xddb800, 0xf, 0x0, 0x0, 0x0)
/usr/local/go/src/html/template/template.go:125 +0x63
html/template.(*Template).ExecuteTemplate(0x0, 0x7f90facb4790, 0xc820149f80, 0xddb800, 0xf, 0xcfda40, 0xc820437680, 0x0, 0x0)
/usr/local/go/src/html/template/template.go:114 +0x4b
github.com/mattermost/platform/utils.(*HTMLTemplate).Render(0xc820437680, 0x0, 0x0)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/utils/html.go:98 +0x103
main.convertTeamTo30(0xc820407af0, 0x7, 0xc82040e020, 0xc82015f5e8, 0xc82015f5b8, 0xc82015f588, 0x0, 0x0, 0x0)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:686 +0x2bae
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:462 +0x2a75
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:102 +0x9c4

Each time i try to update the mattermost database, the mattermost cli crash … do you have any idea to re-up my mattermost installation ?
Thanks a lot in advance. best regards, Vianney

Please try running:

sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30

Let us know what happens?


I was trying to upgrade mattermost and encounterd the same error as previous comment 3.

Following commands were worked on my system (centos 6).

sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30

Thank you.

Ok, I’ve found the reason.

It’s all becouse of enabled mattermost['email_send_email_notifications'] = true setting in /etc/gitlab/gitlab.rb

@it33, thank you. But…
The automatic generated email addresses does not exists in the real world. So I can’t reset my password through the UI.
All migrated email addresses are in this format: ‘[real-email-name]+[mattermost-team-name]@[real-email-domain]’.

All I get after picking the primary team:

panic: Failed to create index pq: could not create unique index "idx_users_email_unique"

goroutine 1 [running]:
panic(0xb8dd80, 0xc820252880)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/mattermost/platform/store.SqlStore.createIndexIfNotExists(0xc8205870e0, 0xc8200211f8, 0x1, 0x1, 0x7f1ef69bd480, 0xc820021218, 0x0, 0x0, 0x0, 0x0, ...)
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:511 +0x6df
github.com/mattermost/platform/store.SqlStore.CreateUniqueIndexIfNotExists(0xc8205870e0, 0xc8200211f8, 0x1, 0x1, 0x7f1ef69bd480, 0xc820021218, 0x0, 0x0, 0x0, 0x0, ...)
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:475 +0xba
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:516 +0x1fc7
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:102 +0x9c4

Hey @it33,

Thanks for replying, it was BDRs fault, but I resolved it!

Thanks for your help!

Thanks @it33, but that doesn’t much help. I was able to make some progress with the following command:

sudo /opt/gitlab/embedded/bin/pg_dump -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production > mattermost_production_backup

But that returns the error

pg_dump: [archiver (db)] connection to database "mattermost_production" failed: FATAL: Peer authentication failed for user "gitlab_mattermost"

See my standalone post: [Solved] How do I back up a GitLab Mattermost database?

Tried some more stuff, I found the pg_hba.conf, and it looks like there’s a peer map, whatever that means. I think that’s what’s causing the error but I don’t know what to do.

OMG! I got it. Here’s what I did:

sudo -u mattermost -i bash
/opt/gitlab/embedded/bin/pg_dump -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production > mattermost_production_backup

And now I have a backup! Jeez, that was overly difficult.

Same thing here. Panic looks like this:

2016-06-23_15:11:53.39476 main.main()
2016-06-23_15:11:53.39476       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:104 +0x9c9
2016-06-23_15:11:55.05863 panic: Failed to open sql connection pq: the database system is starting up
2016-06-23_15:11:55.05868 goroutine 1 [running]:
2016-06-23_15:11:55.05868 panic(0xb8dd80, 0xc8202dd300)
2016-06-23_15:11:55.05869       /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_15:11:55.05869 github.com/mattermost/platform/store.setupConnection(0xdb97c0, 0x6, 0xc820135d80, 0x8, 0xc82004efc0, 0x5d, 0xa, 0xa, 0xc8202dcf00, 0x10)
2016-06-23_15:11:55.05870       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:218 +0x894
2016-06-23_15:11:55.05871 github.com/mattermost/platform/store.initConnection(0x41da0f)
2016-06-23_15:11:55.05871       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:63 +0x11b
2016-06-23_15:11:55.05872 github.com/mattermost/platform/store.NewSqlStore(0x0, 0x0)
2016-06-23_15:11:55.05872       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:85 +0x49
2016-06-23_15:11:55.05873 github.com/mattermost/platform/api.NewServer()
2016-06-23_15:11:55.05874       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/api/server.go:36 +0xe8
2016-06-23_15:11:55.05874 main.main()
2016-06-23_15:11:55.05875       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:104 +0x9c9
2016-06-23_15:19:52.39029 panic: Failed to open sql connection pq: the database system is starting up
2016-06-23_15:19:52.39129 goroutine 1 [running]:
2016-06-23_15:19:52.39129 panic(0xb8dd80, 0xc82040b480)
2016-06-23_15:19:52.39129       /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_15:19:52.39130 github.com/mattermost/platform/store.setupConnection(0xdb97c0, 0x6, 0xc82012dd78, 0x8, 0xc82004f020, 0x5d, 0xa, 0xa, 0xc82040ae00, 0x10)
2016-06-23_15:19:52.39130       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:218 +0x894

Is this due to the failed upgrade?

I also tried a clean install and that doesn’t work at all either.


In my opinion, the whole point of the “omnibus” is that you should be able to turn on Mattermost on a clean installed new system (that has never run a pre-8.9 gitlab) and have it WORK.

After upgrading to 3.1, I no longer have access to the system console. I have tried running the ./platform command that will add my access back, but I get an error when attempting that.

I was able to get mine fixed, by copying the en.json to the directory that the mattermost process runs out of. Hopefully this will help someone.

What directory did you find en.json in and where did you copy it to?