Upgrading to GitLab Mattermost in GitLab 8.9

Hi @Korni22, Mattermost doesn’t use database triggers, did you or someone on your team manually modify the Mattermost database to add triggers?

I’ve just update my gitlab installation using apt-get upgrade. My gitlab install work well but the mattermost embedded doesn’t work any more.
I check log, and mattermost ask me to upgrade the mattermost database.

[2016/06/22 10:01:12 EDT] [CRIT] The database version of 2.2.0 cannot be automatically upgraded to 3.0 schema
[2016/06/22 10:01:12 EDT] [CRIT] You will need to run the command line tool ‘./platform -upgrade_db_30’

So i launch this command :

mattermost@Kalay:/opt/gitlab/embedded/bin$ /opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30

Please see http://www.mattermost.org/upgrade-to-3-0/
WARNING This upgrade process will be irreversible.
Have you performed a database backup? (YES/NO): YES
We found 14 teams.

Please pick a primary team from the list above: datexim
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x4a6653]

goroutine 1 [running]:
panic(0xccbec0, 0xc82000e090)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
html/template.(*Template).lookupAndEscapeTemplate(0x0, 0xddb800, 0xf, 0x0, 0x0, 0x0)
/usr/local/go/src/html/template/template.go:125 +0x63
html/template.(*Template).ExecuteTemplate(0x0, 0x7f90facb4790, 0xc820149f80, 0xddb800, 0xf, 0xcfda40, 0xc820437680, 0x0, 0x0)
/usr/local/go/src/html/template/template.go:114 +0x4b
github.com/mattermost/platform/utils.(*HTMLTemplate).Render(0xc820437680, 0x0, 0x0)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/utils/html.go:98 +0x103
main.convertTeamTo30(0xc820407af0, 0x7, 0xc82040e020, 0xc82015f5e8, 0xc82015f5b8, 0xc82015f588, 0x0, 0x0, 0x0)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:686 +0x2bae
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:462 +0x2a75
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:102 +0x9c4

Each time i try to update the mattermost database, the mattermost cli crash … do you have any idea to re-up my mattermost installation ?
Thanks a lot in advance. best regards, Vianney

Please try running:

sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30

Let us know what happens?


I was trying to upgrade mattermost and encounterd the same error as previous comment 3.

Following commands were worked on my system (centos 6).

sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30

Thank you.

Ok, I’ve found the reason.

It’s all becouse of enabled mattermost['email_send_email_notifications'] = true setting in /etc/gitlab/gitlab.rb

@it33, thank you. But…
The automatic generated email addresses does not exists in the real world. So I can’t reset my password through the UI.
All migrated email addresses are in this format: ‘[real-email-name]+[mattermost-team-name]@[real-email-domain]’.

All I get after picking the primary team:

panic: Failed to create index pq: could not create unique index "idx_users_email_unique"

goroutine 1 [running]:
panic(0xb8dd80, 0xc820252880)
        /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/mattermost/platform/store.SqlStore.createIndexIfNotExists(0xc8205870e0, 0xc8200211f8, 0x1, 0x1, 0x7f1ef69bd480, 0xc820021218, 0x0, 0x0, 0x0, 0x0, ...)
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:511 +0x6df
github.com/mattermost/platform/store.SqlStore.CreateUniqueIndexIfNotExists(0xc8205870e0, 0xc8200211f8, 0x1, 0x1, 0x7f1ef69bd480, 0xc820021218, 0x0, 0x0, 0x0, 0x0, ...)
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:475 +0xba
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:516 +0x1fc7
        /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:102 +0x9c4

Hey @it33,

Thanks for replying, it was BDRs fault, but I resolved it!

Thanks for your help!

Thanks @it33, but that doesn’t much help. I was able to make some progress with the following command:

sudo /opt/gitlab/embedded/bin/pg_dump -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production > mattermost_production_backup

But that returns the error

pg_dump: [archiver (db)] connection to database "mattermost_production" failed: FATAL: Peer authentication failed for user "gitlab_mattermost"

See my standalone post: [Solved] How do I back up a GitLab Mattermost database?

Tried some more stuff, I found the pg_hba.conf, and it looks like there’s a peer map, whatever that means. I think that’s what’s causing the error but I don’t know what to do.

OMG! I got it. Here’s what I did:

sudo -u mattermost -i bash
/opt/gitlab/embedded/bin/pg_dump -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production > mattermost_production_backup

And now I have a backup! Jeez, that was overly difficult.

Same thing here. Panic looks like this:

2016-06-23_15:11:53.39476 main.main()
2016-06-23_15:11:53.39476       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:104 +0x9c9
2016-06-23_15:11:55.05863 panic: Failed to open sql connection pq: the database system is starting up
2016-06-23_15:11:55.05868 goroutine 1 [running]:
2016-06-23_15:11:55.05868 panic(0xb8dd80, 0xc8202dd300)
2016-06-23_15:11:55.05869       /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_15:11:55.05869 github.com/mattermost/platform/store.setupConnection(0xdb97c0, 0x6, 0xc820135d80, 0x8, 0xc82004efc0, 0x5d, 0xa, 0xa, 0xc8202dcf00, 0x10)
2016-06-23_15:11:55.05870       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:218 +0x894
2016-06-23_15:11:55.05871 github.com/mattermost/platform/store.initConnection(0x41da0f)
2016-06-23_15:11:55.05871       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:63 +0x11b
2016-06-23_15:11:55.05872 github.com/mattermost/platform/store.NewSqlStore(0x0, 0x0)
2016-06-23_15:11:55.05872       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:85 +0x49
2016-06-23_15:11:55.05873 github.com/mattermost/platform/api.NewServer()
2016-06-23_15:11:55.05874       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/api/server.go:36 +0xe8
2016-06-23_15:11:55.05874 main.main()
2016-06-23_15:11:55.05875       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:104 +0x9c9
2016-06-23_15:19:52.39029 panic: Failed to open sql connection pq: the database system is starting up
2016-06-23_15:19:52.39129 goroutine 1 [running]:
2016-06-23_15:19:52.39129 panic(0xb8dd80, 0xc82040b480)
2016-06-23_15:19:52.39129       /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_15:19:52.39130 github.com/mattermost/platform/store.setupConnection(0xdb97c0, 0x6, 0xc82012dd78, 0x8, 0xc82004f020, 0x5d, 0xa, 0xa, 0xc82040ae00, 0x10)
2016-06-23_15:19:52.39130       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:218 +0x894

Is this due to the failed upgrade?

I also tried a clean install and that doesn’t work at all either.


In my opinion, the whole point of the “omnibus” is that you should be able to turn on Mattermost on a clean installed new system (that has never run a pre-8.9 gitlab) and have it WORK.

After upgrading to 3.1, I no longer have access to the system console. I have tried running the ./platform command that will add my access back, but I get an error when attempting that.

I was able to get mine fixed, by copying the en.json to the directory that the mattermost process runs out of. Hopefully this will help someone.

What directory did you find en.json in and where did you copy it to?

I used the following en.json > https://github.com/mattermost/platform/blob/master/i18n/en.json

I put the en.json INSIDE the directory where the mattermost executable is located (in my case: /opt/gitlab/embedded/bin/)

Then I was able to successfully restore my access to the system console.

1 Like

I may have a particular issue.
I have an instance of gitlab which when I installed, I set up mattermost but didn’t use it at the time.
Now that I’ve upgraded gitlab and plan to use it, I run the interactive upgrade but it says I have 0 teams (expected), but I can’t do the upgrade because I can’t choose any team for the script to keep running. Any help, thoughts?

I having exact same issue as @samypr100
It says I have 0 teams.

gitlab@gitlab:~/mattermost/bin$ sudo -u mattermost -i bash
mattermost@gitlab:~$ cd /opt/gitlab/embedded/service/mattermost
mattermost@gitlab:/opt/gitlab/embedded/service/mattermost$ /opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30

Please see http://www.mattermost.org/upgrade-to-3-0/
**WARNING** This upgrade process will be irreversible.
Have you performed a database backup? (YES/NO): YES
We found 0 teams.
Please pick a primary team from the list above:

I highly recommend that you start a separate thread so that your issue gets more exposure.

Hi @GhostLyrics
I thought this was the thread for upgrade issues with Gitlab. Where should I create the appropriate thread and how should I title it so that it get’s the context of this thread? Thanks!

Hi everyone,

If you haven’t found the answer to your issue by web searching the error message plus “mattermost”, then please use this link to post a new topic in the GitLab Mattermost forum for your troubleshooting issue.

This was recommended in the first message on this thread.

We’ve also been trying to support people who report issues on replies, and realize that it’s really confusing for anyone to follow, so we’re going to close this thread so that people create new topics that can be more easily supported.