Mattermost does not start after upgrade (Server v4.10.0)

Troubleshooting
1

Summary

mattermost-docker_app does not start

Steps to reproduce

I my case i upgraded from 4.8.1 to 4.10.0 using h ttps://github.com/mattermost/mattermost-docker#update-mattermost-to-latest-version and h ttps://github.com/mattermost/mattermost-docker#upgrading-mattermost-to-49

Expected behavior

mattermost-docker_app does start

Observed behavior

Everthing is set-up according to h ttps://docs.mattermost.com/install/prod-docker.html#production-docker-setup-on-ubuntu

$ docker container ls
CONTAINER ID        IMAGE                   COMMAND                  CREATED             STATUS                          PORTS                                      NAMES
2788625bedb5        mattermost-docker_db    "/entrypoint.sh post…"   15 minutes ago      Up 15 minutes (healthy)         5432/tcp                                   mattermost-docker_db_1
96f2e1a4d708        mattermost-docker_web   "/entrypoint.sh"         15 minutes ago      Up 15 minutes (unhealthy)       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   mattermost-docker_web_1
61e4b2568079        mattermost-docker_app   "/entrypoint.sh plat…"   15 minutes ago      Restarting (100) 1 second ago                                              mattermost-docker_app_1

The container “mattermost-docker_app” restarts itself over and over.

$ curl http://localhost
<html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

$ docker logs -f mattermost-docker_web_1
[snip]
2018/05/18 14:00:03 [error] 11#11: *81 connect() failed (113: Host is unreachable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET / HTTP/1.1", upstream: "h ttp://172.18.0.2:8000/", host: "localhost"
127.0.0.1 - - [18/May/2018:14:00:03 +0200] "GET / HTTP/1.1" 502 166 "-" "curl/7.59.0" "-"

$ docker logs -f mattermost-docker_app_1
Using existing config file /mattermost/config/config.json
Configure database connection...OK
Wait until database db:5432 is ready...
Starting platform
[repeates over and over]

$ docker logs -f mattermost-docker_db_1
AWS_ACCESS_KEY_ID is required for Wal-E but not set. Skipping Wal-E setup.
AWS_SECRET_ACCESS_KEY is required for Wal-E but not set. Skipping Wal-E setup.
WALE_S3_PREFIX is required for Wal-E but not set. Skipping Wal-E setup.
AWS_REGION is required for Wal-E but not set. Skipping Wal-E setup.
LOG:  database system was shut down at 2018-05-18 11:36:38 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  autovacuum launcher started
LOG:  database system is ready to accept connections
LOG:  incomplete startup packet
ERROR:  type "teams" already exists
HINT:  A relation has an associated type of the same name, so you must use a name that doesn't conflict with any existing type.
STATEMENT:  create table if not exists "teams" ("id" varchar(26) not null primary key, "createat" bigint, "updateat" bigint, "deleteat" bigint, "displayname" varchar(64), "name" varchar(64) unique, "description" varchar(255), "email" varchar(128), "type" text, "companyname" varchar(64), "alloweddomains" varchar(500), "inviteid" varchar(32), "allowopeninvite" boolean, "lastteamiconupdate" bigint) ;

The last four lines from “docker logs -f mattermost-docker_db_1” repeates over and over.

Can someone help me with that?

2

Sorry for breaking the links, as a new user i’m not allowed to post more than two links

3

Hi @compilenix, thank you for reporting this!

There is a note in our upgrade notes related to docker when you look at the section for 4.9.0: https://docs.mattermost.com/administration/important-upgrade-notes.html?highlight=important%20upgrade%20notes. “To improve the production use of Mattermost with Docker, the docker image is now running a as non-root user and listening on port 8000. Please read the upgrade instructions for important changes to existing installations.”

Does that sound like something that could be related to the issue you are seeing?

4

Thanks,
i’ve followed that instructions.

I think the issue is not related to the port change of the mattermost app container from 80 to 8000 nor the filesystem permissions. The mattermost app container does not start and my guess is that it has to do with the app conatiner trying to create a “teams” type / table on the db container which fails.

5

Thank you for the additional information!

Since this seems to be an issue with Docker, I will check with @pichouk if this is something he could help troubleshoot.

6

Hmm this is weird. I don’t know why the Mattermost application is trying to create a table in the database. Is your existing database in a coherent state ?

@amy.blais Thanks for the ping :slight_smile: Do you know someone with some skills about Mattermost database initialization ? Someone that may have some clues on what ERROR: type "teams" already exists mean ?

7

@pichouk do you know how can i check if the existing Database is in a coherent state?

8

I don’t really know but I guess if you are able to rollback and use your database in previous MM version it should confirm that your database is OK.

10

Hi @compilenix, quick update on this - I’ve asked our engineers to take a look at this and I’m hoping to have feedback soon.

11

@amy.blais many thanks :slight_smile:

12

Hi @compilenix,

This is weird, for some reason your postgresql database have a type “teams” defined, which makes no sense.

Could you enter in your database server and run \dt and \dT and paste the results here?

13

hi @jespino ,
do you know how i can do that, given that the postgresql is running within the docker container?

14

An easy option would be execute a shell in the container (you can follow this arcticle for that “https://hackernoon.com/run-bash-or-any-command-in-a-docker-container-9a1e7f0ec204”), and once you are inside the container, you can execute psql to connect to the postgres server.

15

@jespino many thanks, here is the output:

psql (9.4.18)
Type "help" for help.

mattermost=# \dt
               List of relations
 Schema |         Name         | Type  | Owner
--------+----------------------+-------+--------
 public | audits               | table | mattermost
 public | channelmemberhistory | table | mattermost
 public | channelmembers       | table | mattermost
 public | channels             | table | mattermost
 public | clusterdiscovery     | table | mattermost
 public | commands             | table | mattermost
 public | commandwebhooks      | table | mattermost
 public | compliances          | table | mattermost
 public | emoji                | table | mattermost
 public | fileinfo             | table | mattermost
 public | incomingwebhooks     | table | mattermost
 public | jobs                 | table | mattermost
 public | licenses             | table | mattermost
 public | oauthaccessdata      | table | mattermost
 public | oauthapps            | table | mattermost
 public | oauthauthdata        | table | mattermost
 public | outgoingwebhooks     | table | mattermost
 public | pluginkeyvaluestore  | table | mattermost
 public | posts                | table | mattermost
 public | preferences          | table | mattermost
 public | reactions            | table | mattermost
 public | roles                | table | mattermost
 public | sessions             | table | mattermost
 public | status               | table | mattermost
 public | systems              | table | mattermost
 public | teammembers          | table | mattermost
 public | teams                | table | mattermost
 public | tokens               | table | mattermost
 public | useraccesstokens     | table | mattermost
 public | users                | table | mattermost
(30 rows)

mattermost=# \dT
     List of data types
 Schema | Name | Description 
--------+------+-------------
(0 rows)
16

That is weird, the error from the server is that the type “teams” already exists, and exists, but as teams table, so make no sense that the error is happening because we execute “CREATE TABLE IF NOT EXISTS” so it mustn’t fail.

17

Exactly about that i was wondering about, too. Since the log explicitly states:

create table if not exists "teams" ("id" varchar(26) not...
18

Yes, exactly. Could you try to execute the SQL directly in the psql client?

create table if not exists "teams" ("id" varchar(26) not null primary key, "createat" bigint, "updateat" bigint, "deleteat" bigint, "displayname" varchar(64), "name" varchar(64) unique, "description" varchar(255), "email" varchar(128), "type" text, "companyname" varchar(64), "alloweddomains" varchar(500), "inviteid" varchar(32), "allowopeninvite" boolean, "lastteamiconupdate" bigint) ;

19 
mattermost=# create table if not exists "teams" ("id" varchar(26) not null primary key, "createat" bigint, "updateat" bigint, "deleteat" bigint, "displayname" varchar(64), "name" varchar(64) unique, "description" varchar(255), "email" varchar(128), "type" text, "companyname" varchar(64), "alloweddomains" varchar(500), "inviteid" varchar(32), "allowopeninvite" boolean, "lastteamiconupdate" bigint) ;
ERROR:  type "teams" already exists
HINT:  A relation has an associated type of the same name, so you must use a name that doesn't conflict with any existing type.
20

I’ve also tried to see into that table and got this:

mattermost=# select * from mattermost.teams;
ERROR:  relation "mattermost.teams" does not exist
LINE 1: select * from mattermost.teams;
                      ^

The result is the same using select * from teams, very strange…

21

Yes, that database looks like broken, could you try to create a dump of the database, and load again into another database (if you can, in another postgresql instance) and try to execute again the CREATE TABLE IF NOTE EXISTS query in the new created database?