[SOLVED] 504 nginx errors

Troubleshooting
1

For feature requests, please see: http://www.mattermost.org/feature-requests/.

For troubleshooting questions, please post in the following format:

Summary

Many 504 logs from nginx as the front-end to Mattermost.

Steps to reproduce

Mattermost 3.4.0
nginx 1.10.1
Postgres 9.5

All are hosted on separate VM’s running RHEL7.2

Expected behavior

MM responds as expected

Observed behavior

Attempting to use our POC Mattermost environment in a large chat we experienced 504 errors from the system. There doesn’t seem to be anything relevant in the MM logs, or the Postgres logs but the web server constantly saw these.

504:60.000:60.000:.:183 1674861 mattermost. - [28/Oct/2016:22:59:57 +0100] “POST /api/v3/users/status/set_active_channel HTTP/1.1” “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0” “-” 10.34.176.24:8065 - "in: _ga=GA1.2.1558470232.1473435083; MMAUTHTOKEN=hjejbnx8qpfs3dyqjqzymuiu6r"
504:60.000:60.000:.:183 1703753 mattermost. - [29/Oct/2016:01:22:38 +0100] “POST /api/v3/teams/9t198y14rbf45k6xtaijjcngna/channels/9b7j54at7jgbjew3zabydj564c/update_last_viewed_at HTTP/1.1” “-” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0” “-” 10.34.176.24:8065 - "in: _ga=GA1.2.1558470232.1473435083; MMAUTHTOKEN=hjejbnx8qpfs3dyqjqzymuiu6r"
504:60.002:60.002:.:585 1703750 mattermost… - [29/Oct/2016:01:22:46 +0100] “POST /api/v3/teams/9t198y14rbf45k6xtaijjcngna/channels/9b7j54at7jgbjew3zabydj564c/update_last_viewed_at HTTP/1.1” “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=tqwzyjf6mir73fwmshhwaemfgc"
504:60.002:60.002:.:585 1703747 mattermost… - [29/Oct/2016:01:22:50 +0100] “POST /api/v3/users/status/set_active_channel HTTP/1.1” “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=tqwzyjf6mir73fwmshhwaemfgc"
504:60.001:60.001:.:183 1722359 mattermost… - [29/Oct/2016:03:22:24 +0100] “POST /api/v3/teams/9t198y14rbf45k6xtaijjcngna/channels/9b7j54at7jgbjew3zabydj564c/update_last_viewed_at HTTP/1.1” “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=e9g611986fbzfqh3hqeq1xizsw; _ga=GA1.2.1132465454.1473346131"
504:60.000:60.000:.:183 1722563 mattermost… - [29/Oct/2016:03:22:24 +0100] “GET /static/files/3812df668bb0144f7ae51853b202315b.mp3 HTTP/1.1” “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=e9g611986fbzfqh3hqeq1xizsw; _ga=GA1.2.1132465454.1473346131"
504:60.001:60.001:.:183 1722564 mattermost… - [29/Oct/2016:03:22:24 +0100] “GET /static/files/3812df668bb0144f7ae51853b202315b.mp3 HTTP/1.1” “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=e9g611986fbzfqh3hqeq1xizsw; _ga=GA1.2.1132465454.1473346131"
504:60.002:60.002:.:585 1722403 mattermost… - [29/Oct/2016:03:22:25 +0100] “POST /api/v3/teams/9t198y14rbf45k6xtaijjcngna/channels/9b7j54at7jgbjew3zabydj564c/update_last_viewed_at HTTP/1.1” “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=ucwwc7ux8tfmpe5rmr498q7kka"
504:60.001:60.001:.:585 1722568 mattermost. - [29/Oct/2016:03:22:28 +0100] “POST /api/v3/users/status/set_active_channel HTTP/1.1” “-” “Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=ucwwc7ux8tfmpe5rmr498q7kka"
504:60.009:60.008:.:183 1722443 mattermost. - [29/Oct/2016:03:22:28 +0100] “POST /api/v3/teams/9t198y14rbf45k6xtaijjcngna/channels/9b7j54at7jgbjew3zabydj564c/posts/create HTTP/1.1” “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14” “-” 10.34.176.24:8065 - "in: MMAUTHTOKEN=e9g611986fbzfqh3hqeq1xizsw; _ga=GA1.2.1132465454.1473346131"
504:60.001:60.001:.:183 1722571 mattermost. - [29/Oct/2016:03:22:28 +0100] “GET /api/v3/users/egr5ucx1ftdrfd3sn5fhsfogja/image?time=1477559367554 HTTP/1.1” “-” “Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14” “-” 10.34.176.24:8065 - “in: MMAUTHTOKEN=e9g611986fbzfqh3hqeq1xizsw; _ga=GA1.2.1132465454.1473346131”

Is there anything I should be setting either on the MM end, or the nginx end to try and prevent this ?

Thanks

2

Humm. This might have been memory constraints on the MM server side. Have bumped it and it may help we shall see.

Still getting these on the nginx end:-
“an upstream response is buffered to a temporary file”

I know I can tweak these params. Anyone got any good ideas for this ?

Thanks

3

Hi @seymourp you can try increasing the ulimit on both the nginx and MM box.

You can try the settings to inprive performance found here https://docs.mattermost.com/install/prod-ubuntu.html

1 Like
4

Hi @elias

Many thanks yeah I have done that now. Seems to have done the trick.

Cheers