Replica Setting prevents Mattermost start, pq: relation “users” does not exist

Troubleshooting
1

Summary

Mattermost server does not start as soon as I add a database replica.

Steps to reproduce

Mattermost 4.5.0
PostgreSQL 9.5.10

Expected behavior

When adding a second database server as read replica the mattermost server isn’t started properly. The second PostgreSQL server is installed, configured and can be accessed like the first one.

Observed behavior

Logs show an issue with the “users” relation on the replica.

[2018/01/12 00:21:44 UTC] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’
[2018/01/12 00:21:44 UTC] [INFO] Current version is 4.5.0 (4.5.0/Fri Dec 15 23:15:33 UTC 2017/03f5c939deb4a6ab2fd01639b5570799e4118bab/dc1a5c62b18d87b9effd4ec249f29c77d1ffdde0)
[2018/01/12 00:21:44 UTC] [INFO] Enterprise Enabled: true
[2018/01/12 00:21:44 UTC] [INFO] Current working directory is /opt/mattermost
[2018/01/12 00:21:44 UTC] [INFO] Loaded config file from /opt/mattermost/config/config.json
[2018/01/12 00:21:44 UTC] [INFO] Loaded system translations for ‘en’ from ‘/opt/mattermost/i18n/en.json’
[2018/01/12 00:21:44 UTC] [INFO] Server is initializing…
[2018/01/12 00:21:44 UTC] [INFO] Pinging SQL master database
[2018/01/12 00:21:44 UTC] [INFO] Pinging SQL replica-0 database
[2018/01/12 00:21:44 UTC] [INFO] The database schema has been set to version 4.5.0
[2018/01/12 00:21:45 UTC] [INFO] Initializing plugin: ldapextras
[2018/01/12 00:21:45 UTC] [INFO] Able to write files to local storage.
[2018/01/12 00:21:45 UTC] [INFO] License key has not been uploaded. Loading license key from disk at /tmp/christians_demo__trial-82190.mattermost-license
[2018/01/12 00:21:45 UTC] [INFO] Failed to save license key loaded from disk err=addLicense: api.license.add_license.invalid_count.app_error, SqlUserStore.AnalyticsUniqueUserCount: store.sql_user.analytics_unique_user_count.app_error, pq: relation “users” does not exist
[2018/01/12 00:21:45 UTC] [INFO] License key from https://mattermost.com required to unlock enterprise features.
[2018/01/12 00:21:45 UTC] [INFO] Starting up plugins
[2018/01/12 00:21:46 UTC] [INFO] Activated jira plugin
[2018/01/12 00:21:46 UTC] [INFO] Starting Server…
[2018/01/12 00:21:46 UTC] [INFO] Server is listening on [::]:8065
[2018/01/12 00:21:46 UTC] [INFO] Initializing job API routes
[2018/01/12 00:21:46 UTC] [INFO] API version 3 is scheduled for deprecation. Please see https://api.mattermost.com for details.
[2018/01/12 00:21:46 UTC] [INFO] Starting 2 websocket hubs
[2018/01/12 00:21:46 UTC] [INFO] Starting workers
[2018/01/12 00:21:46 UTC] [INFO] Starting schedulers.
[2018/01/12 00:22:09 UTC] [EROR] Error occured getting all pending statuses: SqlJobStore.GetAllByStatus: store.sql_job.get_all.app_error, Status=pending, pq: relation “jobs” does not exist

Replica setting looks like:

"SqlSettings": {
    "DriverName": "postgres",
    "DataSource": "postgres://mmuser:mmuser_password@172.31.28.153:5432/mattermost?sslmode=disable\u0026connect_timeout=10",
    "DataSourceReplicas": ["postgres://mmuser:mmuser_password@172.31.25.97:5432/mattermost?sslmode=disable\u0026connect_timeout=10"],
    "DataSourceSearchReplicas": [],
    "MaxIdleConns": 20,
    "MaxOpenConns": 300,
    "Trace": false,
    "AtRestEncryptKey": "8cbfs9aku18tnyppgzsgqjtfapz1xwn1",
    "QueryTimeout": 30
},

It seems that the tables are not created on the replica node (checked it with \d). I restarted the mattermost server a few times.

2

Hi @cjohannsen,

I’m not absolutely certain but I’m wondering whether the issue could be that APIv3 has been deprecated?

Here is the latest changelog where you can read more about this.