[solved] Restoring config.json fails


Restoring config.json after migrating to another server fails
Issue in one concise sentence

Steps to reproduce

  1. Backup MM following https://docs.mattermost.com/administration/backup.html
  2. Restore MM
    • restore DB dump
    • copy data dir
    • copy config.json

Expected behavior

MM starts

Observed behavior

When copying config.json MM fails to start with a very unspecific error (/var/log/gitlab/mattermost/mattermost.log):

{"level":"info","ts":1545783114.786322,"caller":"utils/config.go:211","msg":"Config file watcher detected a change reloading /var/opt/gitlab/mattermost/config.json"}
{"level":"error","ts":1545783114.7864351,"caller":"utils/config.go:216","msg":"Failed to read while watching config file at /var/opt/gitlab/mattermost/config.json with err=parsing error at line 1, character 1: unexpected end of JSON input"}
{"level":"info","ts":1545783114.8663583,"caller":"utils/config.go:211","msg":"Config file watcher detected a change reloading /var/opt/gitlab/mattermost/config.json"}

MM versions on both servers should be the same. (Using gitlab omnibus.)

When I remove config.json, MM starts with all channels and attachments, but the system config is missing obviously. So it seems that there is some incompatibility that prevents me to move my old config to the new server. I can’t figure out what though as the log messages is basically useless.

Did anyone encounter this previously?

Mattermost Version: 5.5.0
Database Schema Version: 5.5.0
Database: postgres

PS: config file is not corrupted, correct JSON, hash of file is same as on old servers
PPS: servers differ in underlying Ubuntu version (16.04 vs. 18.04)

Good question. I’ve touched base with our team to see if they have insight on this.

Edit: I posted it here in case you’re interested joining our community server :slight_smile: https://community.mattermost.com/core/pl/jcdh6s19oj839gs169tbqpjzsw

Hi @arnuschky,

That’s interesting. I’m not aware of any reason that a config should work on one server but not the other.

It looks like you’re copying the new config over while the MM server is still running. Is that the case? If so, there might be some issue with the server trying to read the file as it’s still being populated, so you could try stopping the server with gitlab-ctl stop mattermost, replacing the config, and then restarting it with gitlab-ctl start mattermost.

No, that’s what I first suspected but wasn’t the cause in the end.

I guess I have to go over the config line-by-line and see what section is the issue. Is there anything that’s OS dependent in there?

@arnuschky, can you clarify this point? How is your server successfully starting up (and connecting to the database) if the config.json is removed altogether? May I assume some kind of environment variables at play?

Sorry that was nonsense. What I did (I think) was: I removed config.json and then did a gitlab-reconfigure, which re-writes config.json with the correct base values (like db connection etc). I suspect gitlab does this only on the first install and doesn’t touch the config file afterwards. Or maybe it sets some env vars, I am not 100% how MM is integrated into gitlab.

What remains is that copying a config file from the same MM version, same gitlab version, from another install, doesn’t work.

I can test this again if necessary.

Got it: it might be useful to diff the copied config file with the auto-generated one.

There shouldn’t be anything OS dependent in the configuration file. The error you’re seeing is supposed to signal that the file is not valid JSON and that the error occurred at the start of the file (or perhaps that the file was completely empty).

It wasn’t, I checked that. It looked completely sane. I also ran it through a JSON linter; it passed without errors. So the error MM is showing is either bogus or it fails to read the file for some unknown reason.

What if you use the config.json generated by Mattermost and then paste contents of the old file into the new one instead of replacing the file entirely? It may be a permissions error even though I don’t know it would produce that specific error message.

I was unable to reproduce the problem when testing again. I suspect that the whole report was bogus, for example caused by myself doing the migration while being dead tired. Sorry about that.

In any case, the problem is resolved, I managed to restore my original config.json without any issue.

Thank you all for your help.

No worries. Just glad to hear the problem is fixed. :slight_smile:

Faced this problem again. Traced it back to a bug that the server fails to start after enabling email notifications. For further details see: Enabling email notifications caues mattermost not to start