Clean install of Gitlab 8.9 (not upgrade), enable Mattermost, Bad Gateway

Troubleshooting
1

What do you have to do in Gitlab 8.9 OMNIBUS to be able to access your mattermost.yourcompany.com/admin/ url?

  1. I set up a clean Ubuntu 16 LTS vm. Completely clean and fresh.
  2. I installed the Gitlab 8.9 omnibus edition.
  3. I enabled the mattermost integration as per its docs:
    Files · master · GitLab.org / omnibus-gitlab · GitLab

I expect that if I open http://mattermost.mycompany.com/ I can see the mattermost page. Instead I get a nginx 502 bad gateway.

Even though this is a clean install, I see the following database upgrade failure message in /var/log/gitlab/mattermost/mattermost.log:

root@cleanvm:/var/log/gitlab/mattermost# cat mattermost.log
!!!!Automatic database upgrade failed.!!!

    If you are upgrading from Mattermost v2 to v3
    make sure that you have backed up your database
    and then in /etc/gitlab/gitlab.rb set:

    mattermost['db2_backup_created'] = true
    mattermost['db2_team_name'] = TEAMNAME


    where TEAMNAME is the name of the default team.
    Run gitlab-ctl reconfigure again.

Should I follow the notes above anyways? Ie, adding:

mattermost['db2_backup_created'] = true
mattermost['db2_team_name'] = TEAMNAME

You know, it would be friendlier to non-ruby-coders to point out that really they would write:

mattermost['db2_backup_created'] = true
mattermost['db2_team_name'] = "TEAMNAME"

I did that, and still I have 502 bad gateway.

Did anybody else do a clean 8.9 install and enable mattermost? Does it Just Work or is it Just Broken?

A rather kludgy workaround for me is to manually launch mattermost from a little shell script, in my case I called it run-mattermost.sh, it looks like this:

#!/bin/sh
cd /opt/gitlab/embedded/service/mattermost/
sudo -u mattermost /opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json

That runs it, and now my bad gateway error is gone. Now I can create my initial team (companyname).

The mattermost logs contain an ugly panic like this:

2016-06-23_16:54:26.85623 panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2016-06-23_16:54:26.85624       panic: interface conversion: interface is runtime.errorString, not string
2016-06-23_16:54:26.85624 [signal 0xb code=0x1 addr=0x0 pc=0x775569]
2016-06-23_16:54:26.85624
2016-06-23_16:54:26.85624 goroutine 1 [running]:
2016-06-23_16:54:26.85624 panic(0xcd1aa0, 0xc820045b00)
2016-06-23_16:54:26.85625       /usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-06-23_16:54:26.85625 main.doLoadConfig.func1(0xc820145d48)
2016-06-23_16:54:26.85625       /var/lib/jenkins/jobs/mattermost-platform-release/workspace/src/github.com/mattermost/platform/mattermost.go:73 +0x6f
2016-06-23_16:54:26.85625 panic(0xccbec0, 0xc820012070)
2016-06-23_16:54:26.85625       /usr/local/go/src/runtime/panic.go:443 +0x4e9
2016-06-23_16:54:26.85625 github.com/mattermost/platform/utils.configureLog(0xc820070c70)

I think to fix that ugly logging error I have to manually change unix file owners:

chown -R mattermost:mattermost /var/log/gitlab/mattermost/

Alternatively I could add mattermost to the git group in /etc/group.

Warren

2

Yep I just had the same the exact same experience on a clean gitlab install on a vanilla Ubuntu 16.04 VM (newly setup). Your workaround works for my case as well.

3

did you manually create a Team and then the manual hack is no longer needed?

4

New in 8.9.3, you must as root

cd /opt/gitlab/embedded/service/mattermost
sudo -u mattermost -i bash

And now continuing as mattermost, change back to the same mattermost folder:

cd /opt/gitlab/embedded/service/mattermost

Now see if the “platform” tool (a historical name for mattermost binary) which was renamed to the much more sensible “mattermost” name in the context of the mattermost-bundled-in-gitlab) actually runs by running this and seeing if it panics, like Go code seems to love to do at the least excuse:

$ /opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -version

If you get the version output then the piece of Go code is not feeling shy and may then be usable to do other stuff like:

 /opt/gitlab/embedded/bin/mattermost -config=/var/opt/gitlab/mattermost/config.json -upgrade_db_30

However that’s still not working. The database is both upgraded and not upgraded. We have tea and no tea. We are caught in between worlds.

The mattermost logs says we can’t run because the database is at schema level 2.2.
The upgrade tool says the database appears to be upgraded.

AAAAUGH.

5

It looks like a permissions issue on the mattermost log file. Steps to make it work

Gitlab on a clean machine

  1. sudo vi /etc/gitlab/gitlab.rb
  2. setup and uncomment mattermost_external_url
  3. Notice: I didn’t need to add the following for a clean install
mattermost['db2_backup_created'] = true
mattermost['db2_team_name'] = 'TEAMNAME'
  1. sudo gitlab-ctl reconfigure
  2. notice all the files in /var/log/gitlab/mattermost/ are the root owner
  3. sudo chown -R mattermost:mattermost /var/log/gitlab/mattermost/
  4. comment out mattermost_external_url
  5. sudo gitlab-ctl reconfigure
  6. uncomment out mattermost_external_url
  7. sudo gitlab-ctl reconfigure

Service should be running now.

You can verify mattermost is installed correctly with

sudo -u mattermost -i bash
cd /opt/gitlab/embedded/service/mattermost
/opt/gitlab/embedded/bin/mattermost -config='/var/opt/gitlab/mattermost/config.json' -version
2 Likes
6

Thank you for the hint… Mattermost was driving me crazy, it always crashed instantly. The only issue it had , was that it cannot write to the logfile… fixing that solved the startupcrash instantly.

Thank you ! :slight_smile:

7

That was helpful to me as well.

I also found that in a test deployment, that has been upgraded several times, it was easier to drop my postgres mattermost db and let the chef (omnibus) recipe rebuild my mattermost db, since it contained no real production history that I cared about. That sorted out a lot of weird things that I did to myself while trying to upgrade manually from older 7.x versions to the latest july 2016 version of mattermost in gitlab 8.9

2 Likes