Outbound proxy with certificate

Troubleshooting
1

Summary

Unable to configure outbound proxy which need a certificate

Steps to reproduce

We are using Mattermost 5.21 on Red Hat 7 and have a outbound proxy which need a certificate to pass. I followed the configuration here https://docs.mattermost.com/install/outbound-proxy.html?highlight=outbound%20proxy and we see the communication to our proxy but because of missing certificate the proxy forbid the communication to MM Marketplace. Correct cer-files are placed at /etc/ssl/certs.

Expected behavior

How can we define to use certificates for outbound proxy?

Observed behavior

With curl -x proxy:port https://api.integrations.mattermost.com -v we can connect to Marketplace and receive message “Missing Authentication Token” which seems ok for us.

Did a lot of search but didn’t find any answer yet. Please help!

Thanks

2

@paulrothrock Would support team be familiar with this?

3

Can see following message in log file:

{"level":"error","ts":1586155294.3572779,"caller":"mlog/log.go:175","msg":"Failed to get plugins from the marketplace server.","path":"/api/v4/plugins/marketplace","request_id":"3k5pgtfk3bgfj8yqpbxr351djc","ip_addr":"xxxxxxx","user_id":"aac6f7s8t7nuzdtqm4s48x5koh","method":"GET","err_where":"getRemotePlugins","http_code":500,"err_details":"Get https://api.integrations.mattermost.com/api/v1/plugins?filter=&local_only=false&page=0&server_version=5.21.0: proxyconnect tcp: address forbidden, you may need to set AllowedUntrustedInternalConnections to allow an integration access to your internal network"}

When curl this URL on the command line I get full response from marketplace:
curl -x proxy:port "https://api.integrations.mattermost.com/api/v1/plugins?filter=&local_only=false&page=0&server_version=5.21.0" -v

Also tried mentioned setting “AllowedUntrustedInternalConnections” but without success. Guess this setting is not relevant.

Is there a way to install plugins manually? Didn’t find any docs for this. Can someone please point me there?

Many thanks

4

Any idea, at least where to look at?

5

@ahmaddanial @joewai.tye Would anyone on support team be familiar with this issue?

6

Hello, @wbaeck

A couple of questions to ask to help me understand the issue better:

  • What type of proxy are you configuring for you Mattermost?

  • Can you share the current configuration of the mattermost.service based on the Sample Configuration with the sensitive information redacted if any?

  • Can you also share the ServiceSettings configuration with any sensitive information removed here?

cat /opt/mattermost/config/config.json | grep -A91 "ServiceSettings"

If you are looking at installing plugins manually, you need to ensure that Enable and EnableUploads are set to true. For example:

ahmaddanial@mattermost:~$ cat /opt/mattermost/config/config.json | grep -A19 "PluginSettings"
    "PluginSettings": {
        "Enable": true,
        "EnableUploads": true,
        "AllowInsecureDownloadUrl": true,
        "EnableHealthCheck": true,
        "Directory": "./plugins",
        "ClientDirectory": "./client/plugins",
        "Plugins": {},
        "PluginStates": {
            "com.mattermost.nps": {
                "Enable": true
            }
        },
        "EnableMarketplace": true,
        "EnableRemoteMarketplace": true,
        "AutomaticPrepackagedPlugins": true,
        "RequirePluginSignature": false,
        "MarketplaceUrl": "https://api.integrations.mattermost.com",
        "SignaturePublicKeyFiles": []
    },

Once done, run the following command to ensure that the right ownership is granted to all directories in the /opt/mattermost directory:

sudo chown -R mattermost:mattermost /opt/mattermost/

Then, you can download the .tar.gz of a plugin (for example - Mattermost Jira Plugin ) and upload it through System Console > Plugins (Beta) > Plugin Management. For example:

image
image2398×1616 304 KB

image
image2398×1614 312 KB

image
image2400×1080 175 KB

Can you please let me know if that works for you?

7

Hi @ahmaddanial

Thank you very much for your reply.

  1. What type of proxy are you configuring for you Mattermost?
    In our environment every access to the internet (for user and server) is done via a central proxy. You don’t need any authentication but you need a proxy certificate to be able to pass the proxy. This is due to SSL-security reasons. So every client needs this certificate to pass the proxy. For me the question is where to put our proxy certificates so mattermost/golang will pick them up.
  2. “mattermost.service”
  1. “ServiceSettings”
  1. “PluginSettings”

When I open the Marketplace inside of Mattermost I get the following log entry:

{"level":"error","ts":1587156881.5682132,"caller":"mlog/log.go:175","msg":"Failed to get plugins from the marketplace server.","path":"/api/v4/plugins/marketplace","request_id":"yc9g8qxnn3rqdcdkkspqstr6wy","ip_addr":"10.179.9.81","user_id":"roeyih4u1p8g5y83o8irtfn4no","method":"GET","err_where":"getRemotePlugins","http_code":500,"err_details":"Get https://api.integrations.mattermost.com/api/v1/plugins?filter=&local_only=false&page=0&server_version=5.21.0: proxyconnect tcp: EOF"}

And in the UI it says:

For me the strange thing is that there is no error message at all. It just says “proxyconnect tcp: EOF”. There are no other entries in the log. Are there any other places to search or enable more logging?

I was reading that golang picks up certificates from defined places as documented here: https://golang.org/src/crypto/x509/root_linux.go

So for RedHat 7 (we are using) it must be at /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem? I verified this file and the proxy cert is already there.

Hope that helps for narrow the problem. Please let me know if you need additional information.

Thanks

8

@ahmaddanial

Just found this in the documentation of Mattermost (Redirect):

I think this is my issue but it’s not explained how to configure. Does someone knows how to?

Thanks

9

@ahmaddanial

Any ideas on this?

Many thanks