Gateway Timeout Error while runnning the Load Test

Troubleshooting
1

I have been working on performing the load test on Mattermost for 1000 users per second. I am using JMeter for the same. While running the load test I am continuously getting the error on create_direct API.
Below is the request and response which I am sending and receiving

Sampler Result
Connect Time: 0
Latency: 60023
Size in bytes: 361
Headers size in bytes: 170
Body size in bytes: 191
Sample Count: 1
Error Count: 1
Data type (“text”|“bin”|“”): text
Response code: 504
Response message: Gateway Time-out
Response headers:
HTTP/1.1 504 Gateway Time-out
Server: nginx/1.4.6 (Ubuntu)
Date: Mon, 31 Oct 2016 11:43:03 GMT
Content-Type: text/html
Content-Length: 191
Connection: keep-alive

HTTPSampleResult fields:
ContentType: text/html
DataEncoding: null

Request
POST http://xxxxxxxxxxx/api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/create_direct

POST data:
{“user_id”:“x75wnjsumffjudj3ch1ht6ykqh”}

Cookie Data:
MMAUTHTOKEN=k3ebx3pue7nw5nniipcmawrdar

Request Headers:
Connection: keep-alive
Cache-Control: no-cache
Content-Type: application/json; charset=UTF-8
Accept-Language: en-US,en;q=0.5
Accept: application/json
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:41.0) Gecko/20100101 Firefox/41.0
X-Requested-With: XMLHttpRequest
Accept-Encoding: gzip, deflate
Pragma: no-cache
Content-Length: 40
Host: xxxxxxxxxxxx

Response:
504 Gateway Time-out Error

Log generated in mattermost.log file:
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/: code=401 rid=s8wzeqaw8jgttjc757whe38pfh uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/s5remb7akjbm8x6ompbuco7rmy/: code=401 rid=jj86xfbkutny3bgc1pq1ztkgbo uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/s5remb7akjbm8x6ompbuco7rmy/: code=401 rid=eab8w51o7ifej8ktwzfxprdq5a uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/ag75ztxx3jn5brc44srtmjsgkh/posts/page/0/60: code=401 rid=aper319uopdotcygdud5pa13tc uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/c7xizs983jgdbgprg3z1frwi7c/posts/create: code=401 rid=q3d5ohkho3n33xntaxei4ycd7h uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/: code=401 rid=mk4akq4hhf86fxm8j998fizige uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/teams/6g7nwdnk678tjbk38buaogqy6o/channels/ag75ztxx3jn5brc44srtmjsgkh/posts/page/0/60: code=401 rid=49infgixp3bz7c5gji8abzqd9y uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]
[2016/10/31 11:10:19 UTC] [EROR] /api/v3/users/profiles/6g7nwdnk678tjbk38buaogqy6o: code=401 rid=sotshebg9pbujdhq8j3711no4e uid= ip=52.8.35.172 Invalid or expired session, please login again. [details: UserRequired]

2

Hi @usmanarif can you please share your mattermost config and if you are using nginx can you please share the config as well?

By looking at the logs you posted it seems that you are using an invalid or expired token.

Also we have a load test written in go if you want to give it a try https://github.com/mattermost/mattermost-load-test

3

Hi @elias I am sending you the files which you have asked for. Take a look at it and tell me if I am missing anything in my config file.

{
"ServiceSettings": {
"ListenAddress": ":8065",
"MaximumLoginAttempts": 10,
"SegmentDeveloperKey": "",
"GoogleDeveloperKey": "",
"EnableOAuthServiceProvider": false,
"EnableIncomingWebhooks": true,
"EnableOutgoingWebhooks": true,
"EnableCommands": false,
"EnableOnlyAdminIntegrations": true,
"EnablePostUsernameOverride": false,
"EnablePostIconOverride": false,
"EnableTesting": false,
"EnableDeveloper": false,
"EnableSecurityFixAlert": true,
"EnableInsecureOutgoingConnections": false,
"EnableMultifactorAuthentication": false,
"AllowCorsFrom": "",
"SessionLengthWebInDays": 30,
"SessionLengthMobileInDays": 30,
"SessionLengthSSOInDays": 30,
"SessionCacheInMinutes": 10,
"WebsocketSecurePort": 443,
"WebsocketPort": 80,
"WebserverMode": "gzip"
},
"TeamSettings": {
"SiteName": "XXXXXXXXXXXXXXX",
"MaxUsersPerTeam": 50,
"EnableTeamCreation": true,
"EnableUserCreation": true,
"EnableOpenServer": true,
"RestrictCreationToDomains": "",
"RestrictTeamNames": true,
"EnableCustomBrand": false,
"CustomBrandText": "",
"GetMessagesSinceMonth": "1"
},
"SqlSettings": {
"DriverName": "mysql",
"DataSource": "mmuser:mostest@tcp(dockerhost:3306)/mattermost_test?charset=utf8mb4,utf8",
"DataSourceReplicas": [],
"MaxIdleConns": 10,
"MaxOpenConns": 10,
"Trace": false,
"AtRestEncryptKey": "7rAh6iwQCkV4cA1Gsg3fgGOXJAQ43QVg"
},
"LogSettings": {
"EnableConsole": true,
"ConsoleLevel": "DEBUG",
"EnableFile": true,
"FileLevel": "INFO",
"FileFormat": "",
"FileLocation": ""
},
"FileSettings": {
"DriverName": "local",
"Directory": "./data/",
"EnablePublicLink": false,
"PublicLinkSalt": "A705AklYF8MFDOfcwh3I488G8vtLlVip",
"ThumbnailWidth": 120,
"ThumbnailHeight": 100,
"PreviewWidth": 1024,
"PreviewHeight": 0,
"ProfileWidth": 128,
"ProfileHeight": 128,
"InitialFont": "luximbi.ttf",
"AmazonS3AccessKeyId": "",
"AmazonS3SecretAccessKey": "",
"AmazonS3Bucket": "",
"AmazonS3Region": "",
"AmazonS3Endpoint": "",
"AmazonS3BucketEndpoint": "",
"AmazonS3LocationConstraint": false,
"AmazonS3LowercaseBucket": false
},
"EmailSettings": {
"EnableSignUpWithEmail": true,
"EnableSignInWithEmail": true,
"EnableSignInWithUsername": false,
"SendEmailNotifications": true,
"RequireEmailVerification": false,
"FeedbackName": "XXXXXXXXXXXXXXX",
"FeedbackEmail": "XXXXXXXXXXXXXXX",
"SMTPUsername": "XXXXXXXXXXXXXXX",
"SMTPPassword": "XXXXXXXXXXXXXXX",
"SMTPServer": "XXXXXXXXXXXXXXX",
"SMTPPort": "XXXXXXXXXXXXXXX",
"ConnectionSecurity": "XXXXXXXXXXXXXXX",
"InviteSalt": "XXXXXXXXXXXXXXX",
"PasswordResetSalt": "XXXXXXXXXXXXXXX",
"SendPushNotifications": false,
"PushNotificationServer": "",
"PushNotificationContents": "generic"
},
"RateLimitSettings": {
"EnableRateLimiter": true,
"PerSec": 10,
"MemoryStoreSize": 10000,
"VaryByRemoteAddr": true,
"VaryByHeader": ""
},
"PrivacySettings": {
"ShowEmailAddress": true,
"ShowFullName": true
},
"SupportSettings": {
"TermsOfServiceLink": "/static/help/terms.html",
"PrivacyPolicyLink": "/static/help/privacy.html",
"AboutLink": "/static/help/about.html",
"HelpLink": "/static/help/help.html",
"ReportAProblemLink": "/static/help/report_problem.html",
"SupportEmail": "XXXXXXXXXXXXXXX"
},
"GitLabSettings": {
"Enable": false,
"Secret": "",
"Id": "",
"Scope": "",
"AuthEndpoint": "",
"TokenEndpoint": "",
"UserApiEndpoint": ""
},
"GoogleSettings": {
"Enable": false,
"Secret": "",
"Id": "",
"Scope": "",
"AuthEndpoint": "",
"TokenEndpoint": "",
"UserApiEndpoint": ""
},
"LdapSettings": {
"Enable": false,
"LdapServer": "",
"LdapPort": 389,
"ConnectionSecurity": "",
"BaseDN": "",
"BindUsername": "",
"BindPassword": "",
"UserFilter": "",
"FirstNameAttribute": "",
"LastNameAttribute": "",
"EmailAttribute": "",
"UsernameAttribute": "",
"NicknameAttribute": "",
"IdAttribute": "",
"SkipCertificateVerification": false,
"QueryTimeout": 60,
"LoginFieldName": "",
"PasswordFieldName": ""
},
"ComplianceSettings": {
"Enable": false,
"Directory": "./data/",
"EnableDaily": false
}
}

My nginx setting looks like this:

# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

events {
worker_connections 20240;
}

http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;

sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;

include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html
# include
# for more information.
include /etc/nginx/conf.d/*.conf;

server {
server_name xxxxxxxxxxxxxxxx;
location / {
client_max_body_size 1024M;
proxy_read_timeout 300;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_pass http://127.0.0.1:8065;
}
}
}
worker_rlimit_nofile 30000;
4

Ok 2 things

  1. Can you disable RateLimiting in the config.json file
  2. Can you try the nginx config found here
5

@elias I am also facing the same issue can you help me what should be nginx settings?

6

Hi @smartbothub,

Have you tried the nginx config found here?