it33
June 21, 2016, 10:53pm
1
This is a thread started to help troubleshoot issues with upgrading to GitLab 8.9, which upgrades users to Mattermost 3.1.
If you’re having issues with the automated install, try running:
sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30
If this doesn’t work, please post a new topic to this forum with:
A brief description of your issue as title
The step on which you’re having issues
The relevant portion of the Mattermost log file at /var/log/gitlab/mattermost/mattermost.log
To start a new topic, click this link to the Gitlab Mattermost forum category then click New Topic . See bottom right button in screenshot for an illustration:
manuel
June 22, 2016, 11:27am
2
Thanks for the topic!
therox
June 22, 2016, 2:46pm
3
Hi. When upgrading gitlab to 8.9 mattermost was not started.
/var/log/gitlab/mattermost/mattermost.log :
[2016/06/22 09:24:03 MSK] [EROR] /havana/town-square: code=404 ip=10.46.2.183
!!!!Automatic database upgrade failed.!!!
If you are upgrading from Mattermost v2 to v3
make sure that you have backed up your database
and then in /etc/gitlab/gitlab.rb set:
mattermost['db2_backup_created'] = true
mattermost['db2_team_name'] = TEAMNAME
where TEAMNAME is the name of the default team.
Run gitlab-ctl reconfigure again.
See:
http://docs.gitlab.com/omnibus/gitlab-mattermost/#upgrading-gitlab-mattermost-in-gitlab-8.9
for more information.
and that’s all.
$ /opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: interface conversion: interface is runtime.errorString, not string
[signal 0xb code=0x1 addr=0x0 pc=0x78090f]
goroutine 1 [running]:
panic(0xcd1aa0, 0xc820140180)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.doLoadConfig.func1(0xc820167d48)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:73 +0x6f
panic(0xccbec0, 0xc8200100e0)
/usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/mattermost/platform/utils.TestConnection(0xc820080b00)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/utils/mail.go:82 +0x9f
github.com/mattermost/platform/utils.LoadConfig(0x7fff53bf3794, 0x26)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/utils/config.go:181 +0x10aa
main.doLoadConfig(0x7fff53bf3794, 0x26, 0x0, 0x0)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:76 +0x6a
main.main()
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:84 +0x43
Please help.
1 Like
revxx14
June 22, 2016, 6:31pm
4
How exactly do I back up the database? I can’t find any documentation other than “just do what you normally do with postgres databases,” but I don’t normally do that kind of thing
Hey guys!
I’m having trouble upgrading mattermost.
I added the config parameters, ran apt-get upgrade, it worked fine but now mattermost won’t start.
The log says:
Error creating database tables: pq: trigger "truncate_trigger" for relation "teams" already exists
So I ran
sudo -u mattermost -i bash cd /opt/gitlab/embedded/service/mattermost /opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30
Which returned …nothing and did not help.
Any ideas?
it33
June 23, 2016, 1:53am
6
Hi @manuel , duplicate accounts are switched to email login, you can use “reset password” to get credentials sent to your email, and you can switch to GitLab SSO under Account Settings > Security Sign In Method.
it33
June 23, 2016, 1:57am
7
Hi @revxx14 , try pg_dump dbname > outfile, you can just web search for your database type and version to find documentation for backup, for example: https://www.postgresql.org/docs/9.4/static/backup-dump.html
it33
June 23, 2016, 1:58am
8
Hi @Korni22 , Mattermost doesn’t use database triggers, did you or someone on your team manually modify the Mattermost database to add triggers?
Blist
June 23, 2016, 1:58am
9
Hi,
[2016/06/22 10:01:12 EDT] [CRIT] The database version of 2.2.0 cannot be automatically upgraded to 3.0 schema
So i launch this command :
mattermost@Kalay:/opt/gitlab/embedded/bin$ /opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30
Please see Mattermost | Secure Collaboration for Technical Teams WARNING This upgrade process will be irreversible.
goroutine 1 [running]:github.com/mattermost/platform/utils.(*HTMLTemplate).Render (0xc820437680, 0x0, 0x0)
Each time i try to update the mattermost database, the mattermost cli crash … do you have any idea to re-up my mattermost installation ?
it33
June 23, 2016, 2:27am
10
Please try running:
sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -upgrade_db_30
Let us know what happens?
kgyk
June 23, 2016, 5:28am
11
Hi.
I was trying to upgrade mattermost and encounterd the same error as previous comment 3 .
Following commands were worked on my system (centos 6).
sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30
Thank you.
therox
June 23, 2016, 5:56am
12
Ok, I’ve found the reason.
It’s all becouse of enabled mattermost['email_send_email_notifications'] = true setting in /etc/gitlab/gitlab.rb
manuel
June 23, 2016, 9:27am
13
@it33 , thank you. But…
rsokoll
June 23, 2016, 9:56am
14
All I get after picking the primary team:
panic: Failed to create index pq: could not create unique index "idx_users_email_unique"
goroutine 1 [running]:
panic(0xb8dd80, 0xc820252880)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/mattermost/platform/store.SqlStore.createIndexIfNotExists(0xc8205870e0, 0xc8200211f8, 0x1, 0x1, 0x7f1ef69bd480, 0xc820021218, 0x0, 0x0, 0x0, 0x0, ...)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:511 +0x6df
github.com/mattermost/platform/store.SqlStore.CreateUniqueIndexIfNotExists(0xc8205870e0, 0xc8200211f8, 0x1, 0x1, 0x7f1ef69bd480, 0xc820021218, 0x0, 0x0, 0x0, 0x0, ...)
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:475 +0xba
main.cmdUpdateDb30()
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:516 +0x1fc7
main.main()
/var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:102 +0x9c4
Hey @it33 ,
Thanks for replying, it was BDRs fault, but I resolved it!
Thanks for your help!
revxx14
June 23, 2016, 1:14pm
16
Thanks @it33 , but that doesn’t much help. I was able to make some progress with the following command:
sudo /opt/gitlab/embedded/bin/pg_dump -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production > mattermost_production_backup
But that returns the error
pg_dump: [archiver (db)] connection to database "mattermost_production" failed: FATAL: Peer authentication failed for user "gitlab_mattermost"
See my standalone post: [Solved] How do I back up a GitLab Mattermost database?
Tried some more stuff, I found the pg_hba.conf, and it looks like there’s a peer map, whatever that means. I think that’s what’s causing the error but I don’t know what to do.
–
OMG! I got it. Here’s what I did:
sudo -u mattermost -i bash
/opt/gitlab/embedded/bin/pg_dump -U gitlab_mattermost -h /var/opt/gitlab/postgresql -p 5432 mattermost_production > mattermost_production_backup
And now I have a backup! Jeez, that was overly difficult.
wpostma
June 23, 2016, 3:35pm
17
Same thing here. Panic looks like this:
2016-06-23_15:11:53.39476 main.main()
2016-06-23_15:11:53.39476 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:104 +0x9c9
2016-06-23_15:11:55.05863 panic: Failed to open sql connection pq: the database system is starting up
2016-06-23_15:11:55.05867
2016-06-23_15:11:55.05868 goroutine 1 [running]:
2016-06-23_15:11:55.05868 panic(0xb8dd80, 0xc8202dd300)
2016-06-23_15:11:55.05869 /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_15:11:55.05869 github.com/mattermost/platform/store.setupConnection(0xdb97c0, 0x6, 0xc820135d80, 0x8, 0xc82004efc0, 0x5d, 0xa, 0xa, 0xc8202dcf00, 0x10)
2016-06-23_15:11:55.05870 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:218 +0x894
2016-06-23_15:11:55.05871 github.com/mattermost/platform/store.initConnection(0x41da0f)
2016-06-23_15:11:55.05871 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:63 +0x11b
2016-06-23_15:11:55.05872 github.com/mattermost/platform/store.NewSqlStore(0x0, 0x0)
2016-06-23_15:11:55.05872 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:85 +0x49
2016-06-23_15:11:55.05873 github.com/mattermost/platform/api.NewServer()
2016-06-23_15:11:55.05874 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/api/server.go:36 +0xe8
2016-06-23_15:11:55.05874 main.main()
2016-06-23_15:11:55.05875 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:104 +0x9c9
2016-06-23_15:19:52.39029 panic: Failed to open sql connection pq: the database system is starting up
2016-06-23_15:19:52.39128
2016-06-23_15:19:52.39129 goroutine 1 [running]:
2016-06-23_15:19:52.39129 panic(0xb8dd80, 0xc82040b480)
2016-06-23_15:19:52.39129 /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_15:19:52.39130 github.com/mattermost/platform/store.setupConnection(0xdb97c0, 0x6, 0xc82012dd78, 0x8, 0xc82004f020, 0x5d, 0xa, 0xa, 0xc82040ae00, 0x10)
2016-06-23_15:19:52.39130 /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/store/sql_store.go:218 +0x894
Is this due to the failed upgrade?
I also tried a clean install and that doesn’t work at all either.
http://forum.mattermost.com/t/clean-install-of-gitlab-8-9-not-upgrade-enable-mattermost-bad-gateway/1760
In my opinion, the whole point of the “omnibus” is that you should be able to turn on Mattermost on a clean installed new system (that has never run a pre-8.9 gitlab) and have it WORK.
dnoe
June 23, 2016, 4:42pm
18
After upgrading to 3.1, I no longer have access to the system console. I have tried running the ./platform command that will add my access back, but I get an error when attempting that.
dnoe
June 23, 2016, 4:57pm
19
I was able to get mine fixed, by copying the en.json to the directory that the mattermost process runs out of. Hopefully this will help someone.
wpostma
June 23, 2016, 5:05pm
20
What directory did you find en.json in and where did you copy it to?
