Hello there!
I have been here before, and back with what are obviously nginx connections errors (301).
paule@kame:~/ifb_sandbox/help$ curl https://mattermost.france-bioinformatique.fr:443
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.23.1</center>
</body>
</html>
Now:
ubuntu 22/04
mattemost 2.4
and running with the docker-compose.nginx.yml config
there are no errors I can see in the log files, the containers seem ok to me:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f8a67b74bb4e nginx:alpine "/docker-entrypoint.…" 4 minutes ago Up 4 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp nginx_mattermost
3076c43ad296 mattermost/mattermost-enterprise-edition:7.1 "/entrypoint.sh matt…" 4 minutes ago Up 4 minutes (healthy) 8065/tcp, 8067/tcp, 8074-8075/tcp matterm-mattermost-1
9fe4b2808175 postgres:13-alpine "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 5432/tcp matterm-postgres-1
What am I supposed to look into to get a handle here?
Many thanks!!
agriesser
(Alexander Griesser)
November 2, 2022, 1:26pm
2
froggypaule:
mattemost 2.4
I doubt that this is the correct version According to your docker ps
output you seem to be using 7.1 - is this correct?
I’d like to know where it tries to redirect you, can you please run the following curl
commands and post the output?
curl -I -X GET https://mattermost.france-bioinformatique.fr:443
curl -I -X GET https://mattermost.france-bioinformatique.fr
Thanks @agriesser ! no, it isn’t 2.4 (I picked this from the yml file )
So:
paule@kame:~$ curl -I -X GET https://mattermost.france-bioinformatique.fr:443
HTTP/1.1 301 Moved Permanently
server: nginx/1.23.1
date: Wed, 02 Nov 2022 13:40:26 GMT
content-type: text/html
content-length: 169
location: https://mattermost.france-bioinformatique.fr/
strict-transport-security: max-age=16000000; includeSubDomains; preload;
paule@kame:~$ curl -I -X GET https://mattermost.france-bioinformatique.fr
HTTP/1.1 301 Moved Permanently
server: nginx/1.23.1
date: Wed, 02 Nov 2022 13:40:42 GMT
content-type: text/html
content-length: 169
location: https://mattermost.france-bioinformatique.fr/
strict-transport-security: max-age=16000000; includeSubDomains; preload;
thanks for this, I am not at ease with involved connection matters.
Note: there is no nginx service running, since we run the nginx container.
to be complete, this is the log for the nginx:
ifb-user@mattermost:~/matterm$ docker logs -ft nginx_mattermost
2022-11-02T12:54:58.685472056Z /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
2022-11-02T12:54:58.685500478Z /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
2022-11-02T12:54:58.686472312Z /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
2022-11-02T12:54:58.698739858Z 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?)
2022-11-02T12:54:58.698751629Z /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
2022-11-02T12:54:58.713902506Z /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
2022-11-02T12:54:58.724297341Z /docker-entrypoint.sh: Configuration complete; ready for start up
What I fail to understand is why it says ‘ready for startup’?
agriesser
(Alexander Griesser)
November 2, 2022, 2:15pm
5
OK, something is wrong with your nginx configuration here. It’s redirecting in an endless loop as you can see here.
Please run the following commands in the directory where your .env
file is (I’m providing example output here too, you only need to run the commands in the lines starting with #
):
# egrep ^DOMAIN\|^MM_SERVICESETTINGS_SITEURL .env
DOMAIN=yourdomain.com
MM_SERVICESETTINGS_SITEURL=https://${DOMAIN}
# head -20 volumes/app/mattermost/config/config.json
{
"ServiceSettings": {
"SiteURL": "",
"WebsocketURL": "",
"LicenseFileLocation": "",
"ListenAddress": ":8065",
"ConnectionSecurity": "",
"TLSCertFile": "",
"TLSKeyFile": "",
"TLSMinVer": "1.2",
"TLSStrictTransport": false,
"TLSStrictTransportMaxAge": 63072000,
"TLSOverwriteCiphers": [],
"UseLetsEncrypt": false,
"LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
"Forward80To443": false,
"TrustedProxyIPHeader": [],
"ReadTimeout": 300,
"WriteTimeout": 300,
"IdleTimeout": 60,
This is just nginx
’s way of telling you that the configuration is OK and that it’s about to start up the daemons.
Ok thanks for
ready for startup’?
for the rest:
ifb-user@mattermost:~/matterm$ egrep ^DOMAIN\|^MM_SERVICESETTINGS_SITEURL .env
DOMAIN=mattermost.france-bioinformatique.fr
MM_SERVICESETTINGS_SITEURL=https://${DOMAIN}
ifb-user@mattermost:~/matterm$ head -20 volumes/app/mattermost/config/config.json
{
"ServiceSettings": {
"SiteURL": "",
"WebsocketURL": "",
"LicenseFileLocation": "",
"ListenAddress": ":8065",
"ConnectionSecurity": "",
"TLSCertFile": "",
"TLSKeyFile": "",
"TLSMinVer": "1.2",
"TLSStrictTransport": false,
"TLSStrictTransportMaxAge": 63072000,
"TLSOverwriteCiphers": [],
"UseLetsEncrypt": false,
"LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
"Forward80To443": false,
"TrustedProxyIPHeader": [],
"ReadTimeout": 300,
"WriteTimeout": 300,
"IdleTimeout": 60,
agriesser
(Alexander Griesser)
November 2, 2022, 2:30pm
7
OK, interesting.
Please run (and post the output):
docker exec -ti nginx_mattermost nginx -T
ifb-user@mattermost:~/matterm$ docker exec -ti nginx_mattermost nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/avif avif;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
font/woff woff;
font/woff2 woff2;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
docx;
application/vnd.wap.wmlc wmlc;
application/wasm wasm;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf.d/default.conf:
# mattermost
# config can be tested on https://www.ssllabs.com/ssltest/ and a good nginx config generator
# can be found at https://ssl-config.mozilla.org/
# proxy cache
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=mattermost_cache:10m max_size=3g inactive=120m use_temp_path=off;
# upstream used in proxy_pass below
upstream backend {
# ip where Mattermost is running; this relies on a working DNS inside the Docker network
# and uses the hostname of the mattermost container (see service name in docker-compose.yml)
server mattermost:8065;
keepalive 64;
}
# vhosts definitions
server {
server_name _;
listen 80 default_server;
listen [::]:80 default_server;
# redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
return 301 https://$host$request_uri;
}
server {
server_name _;
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
# logging
access_log /var/log/nginx/mm.access.log;
error_log /var/log/nginx/mm.error.log warn;
# gzip for performance
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
## ssl
ssl_dhparam /dhparams4096.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m;
ssl_session_tickets off;
# intermediate configuration
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
# certs sent to the client in SERVER HELLO are concatenated in ssl_certificate
ssl_certificate /cert.pem;
ssl_certificate_key /key.pem;
# enable TLSv1.3's 0-RTT. Use $ssl_early_data when reverse proxying to prevent replay attacks.
# https://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_early_data
ssl_early_data on;
# OCSP stapling
ssl_stapling on;
ssl_stapling_verify on;
#resolver 1.1.1.1;
# verify chain of trust of OCSP response using Root CA and Intermediate certs
#ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
## security headers
# https://securityheaders.com/
# https://scotthelme.co.uk/tag/security-headers/
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy no-referrer;
add_header Strict-Transport-Security "max-age=63072000" always;
add_header Permissions-Policy "interest-cohort=()";
## locations
# ACME-challenge
location ^~ /.well-known {
default_type "text/plain";
root /usr/share/nginx/html;
allow all;
}
# disable Google bots from indexing this site
location = /robots.txt {
add_header Content-Type text/plain;
return 200 "User-agent: *\nDisallow: /\n";
}
location ~ /api/v[0-9]+/(users/)?websocket$ {
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
client_max_body_size 50M;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_set_header Early-Data $ssl_early_data;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
client_body_timeout 60;
send_timeout 300;
lingering_timeout 5;
proxy_connect_timeout 90;
proxy_send_timeout 300;
proxy_read_timeout 90s;
proxy_http_version 1.1;
proxy_pass http://backend;
}
location / {
client_max_body_size 50M;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_set_header Early-Data $ssl_early_data;
proxy_buffers 256 16k;
proxy_buffer_size 16k;
proxy_read_timeout 600s;
proxy_cache mattermost_cache;
proxy_cache_revalidate on;
proxy_cache_min_uses 2;
proxy_cache_use_stale timeout;
proxy_cache_lock on;
proxy_http_version 1.1;
proxy_pass http://backend;
}
}
agriesser
(Alexander Griesser)
November 2, 2022, 2:42pm
10
Can you please post the output in quote blocks? It’s hard to read and I’m not sure if I overlook something.
Just type the ` three times in a row on a line by its own, then post the output and close it off with another row just containing ```, then it should be formated better.
Also please verify the nginx version of your container by running:
docker exec -ti nginx_mattermost nginx -v
(Should be 1.23.1)
nginx version: nginx/1.23.1
I am sorry: I thought I did that
do you want me to redo it/post it properly?
Hello! I am leaving for the day. If we can’t make sense of all this, I’ll follow your suggestion re omnibus, but on my side I’ll have to go back to ubuntu 20.04 first.
agriesser
(Alexander Griesser)
November 2, 2022, 3:54pm
13
You did use the “Quote” feature, not the “Code” feature - but I just recognized that I can edit your post and I did that now, hope that’s OK for you
There’s just a single configuration line in your nginx
config which does a rewrite and this is this block:
# vhosts definitions
server {
server_name _;
listen 80 default_server;
listen [::]:80 default_server;
# redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
return 301 https://$host$request_uri;
}
This would only happen if you try to access the server via HTTP, which you aren’t and this is what puzzles me here.
Although I’m not sure if this really makes a different, but your mattermost
application container does not have the default name - it could well be that your nginx
and mattermost
application container are in a different namespace.
Can you please post the output of the following commands to verify that?
docker ps --format "{{.ID}}" | xargs docker inspect | jq .[].NetworkSettings.Networks
Ta! I’ll look into as soon as I can
Yes I wondered about naming, but since the config says ‘service : mattermost’ I thought that was ok.
Hello I am back;
we have
{
"mattermost": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"nginx_mattermost",
"nginx",
"f8a67b74bb4e"
],
"NetworkID": "448cce786787f07b91417ec2680acb1b2e1dcf6d5abed021b437ea53786b6ae0",
"EndpointID": "64010fa69e6754c04fdcf48115c1016cc5a09603de8477dab47ecca86c7a3b20",
"Gateway": "172.21.0.1",
"IPAddress": "172.21.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:15:00:04",
"DriverOpts": null
}
}
{
"mattermost": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"matterm-mattermost-1",
"mattermost",
"3076c43ad296"
],
"NetworkID": "448cce786787f07b91417ec2680acb1b2e1dcf6d5abed021b437ea53786b6ae0",
"EndpointID": "3d618f3a46d1ad1bfbf2bd8a3f0dc5a70f19a063919a8c58860d3767b2091487",
"Gateway": "172.21.0.1",
"IPAddress": "172.21.0.3",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:15:00:03",
"DriverOpts": null
}
}
{
"mattermost": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"matterm-postgres-1",
"postgres",
"9fe4b2808175"
],
"NetworkID": "448cce786787f07b91417ec2680acb1b2e1dcf6d5abed021b437ea53786b6ae0",
"EndpointID": "d0c10e4b9d2ce8460944ddbfcf609db155ee572c0042be602e5d1e6694286c79",
"Gateway": "172.21.0.1",
"IPAddress": "172.21.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:15:00:02",
"DriverOpts": null
}
}
ach! I think I know why: in docker-compose.nginx.yml we have a ‘container_name’ declaration, hence the name of nginx_mattermost.
Let me fix this and have a look at the result.
Apparently that wasn’t the answer, but I have now
{
"mattermost": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"matterm-nginx-1",
"nginx",
"0ba39fea23b3"
],
"NetworkID": "d252d5f7ba6a8d975581bf25fefe078e88d2641954eda0f28de3f14c5e776aac",
"EndpointID": "db9de16e4846807a5d1556de66dc2d0b57e2d3228dc889584805f8ce85c71c8e",
"Gateway": "172.22.0.1",
"IPAddress": "172.22.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:16:00:04",
"DriverOpts": null
}
}
agriesser
(Alexander Griesser)
November 3, 2022, 1:54pm
19
The nginx container name is not the problem - I was looking for insights as to why the nginx container does the endless redirect and thought that this could be the case because it’s unable to resolve your backend, but the backendname mattermost
is available as an alias, so this should not have any impact.
Can you please run the following command (it will not finish but show the nginx logs to you) and try to access your Mattermost server with a client then?
You should see new log lines coming in as you try to access the website - please share the logs then with us.
docker exec -ti nginx_mattermost /bin/sh -c 'tail -f /var/log/nginx/*.log'
yes you are right, there were aliases…
for the cmd:
==> /var/log/nginx/error.log <==
2022/11/03 13:38:03 [notice] 1#1: OS: Linux 5.15.0-52-generic
2022/11/03 13:38:03 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/11/03 13:38:03 [notice] 1#1: start worker processes
2022/11/03 13:38:03 [notice] 1#1: start worker process 23
2022/11/03 13:38:03 [notice] 1#1: start cache manager process 24
2022/11/03 13:38:03 [notice] 1#1: start cache loader process 25
2022/11/03 13:39:03 [notice] 25#25: http file cache: /var/cache/nginx 0.000M, bsize: 4096
2022/11/03 13:39:03 [notice] 1#1: signal 17 (SIGCHLD) received from 25
2022/11/03 13:39:03 [notice] 1#1: cache loader process 25 exited with code 0
2022/11/03 13:39:03 [notice] 1#1: signal 29 (SIGIO) received
==> /var/log/nginx/mm.access.log <==
==> /var/log/nginx/mm.error.log <==
==> /var/log/nginx/access.log <==
172.22.0.1 - - [03/Nov/2022:13:58:15 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0" "77.130.249.205"
172.22.0.1 - - [03/Nov/2022:13:58:15 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0" "77.130.249.205"
172.22.0.1 - - [03/Nov/2022:13:58:15 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0" "77.130.249.205"
172.22.0.1 - - [03/Nov/2022:13:58:15 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0" "77.130.249.205"
172.22.0.1 - - [03/Nov/2022:13:58:15 +0000] "GET / HTTP/1.1" 301 169 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:106.0) Gecko/20100101 Firefox/106.0" "77.130.249.205"
etc...
agriesser
(Alexander Griesser)
November 3, 2022, 2:35pm
21
You should see log lines in mm.access.log
, but in fact, your requests are not hitting this target but are being logged in the default nginx access.log
, which is strange and the requests are also all coming from your local system 172.22.0.1
.
Are you sure you didn’t tinker with the nginx configuration or settings and that this nginx is the foremost proxy server listening on port 443 on this system? To me it looks like the requests on port 443 are not even reaching this system which is very strange.
Please check what process ID is running on port 443 on your host and verify that it’s actually pointing to the internal container IP of your nginx container (which should be 172.22.0.4 at the moment):
root@host:~# lsof -i :443
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 2147651 root 4u IPv4 25458799 0t0 TCP *:https (LISTEN)
docker-pr 2147657 root 4u IPv6 25458067 0t0 TCP *:https (LISTEN)
root@host:~# ps ax | grep 2147651
2147651 ? Sl 0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 192.168.176.4 -container-port 443
2148576 pts/0 S+ 0:00 grep 2147651