# Outbound proxy with certificate

**URL:** <https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390>\
**Category:** Troubleshooting\
**Created:** [April 4, 2020, 10:19am UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390 "2020-04-04T10:19:18Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![wbaeck](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wbaeck](https://forum.mattermost.com/u/wbaeck)\
**Post date:** [April 4, 2020, 10:19am UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/1 "2020-04-04T10:19:18Z")

</div>

#### Summary

Unable to configure outbound proxy which need a certificate

#### Steps to reproduce

We are using Mattermost 5.21 on Red Hat 7 and have a outbound proxy which need a certificate to pass. I followed the configuration here [https://docs.mattermost.com/install/outbound-proxy.html?highlight=outbound%20proxy](https://docs.mattermost.com/install/outbound-proxy.html?highlight=outbound%20proxy) and we see the communication to our proxy but because of missing certificate the proxy forbid the communication to MM Marketplace. Correct cer-files are placed at /etc/ssl/certs.

#### Expected behavior

How can we define to use certificates for outbound proxy?

#### Observed behavior

With curl -x proxy:port [https://api.integrations.mattermost.com](https://api.integrations.mattermost.com) -v we can connect to Marketplace and receive message “Missing Authentication Token” which seems ok for us.

Did a lot of search but didn’t find any answer yet. Please help!

Thanks

---

<div class="post-metadata">

**Author:** ![amy.blais](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/amy.blais/32/7481_2.png) [@amy.blais](https://forum.mattermost.com/u/amy.blais)\
**Post date:** [April 6, 2020, 12:28am UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/2 "2020-04-06T00:28:59Z")

</div>

@paulrothrock Would support team be familiar with this?

---

<div class="post-metadata">

**Author:** ![wbaeck](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wbaeck](https://forum.mattermost.com/u/wbaeck)\
**Post date:** [April 6, 2020, 7:03am UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/3 "2020-04-06T07:03:04Z")

</div>

Can see following message in log file:

`{"level":"error","ts":1586155294.3572779,"caller":"mlog/log.go:175","msg":"Failed to get plugins from the marketplace server.","path":"/api/v4/plugins/marketplace","request_id":"3k5pgtfk3bgfj8yqpbxr351djc","ip_addr":"xxxxxxx","user_id":"aac6f7s8t7nuzdtqm4s48x5koh","method":"GET","err_where":"getRemotePlugins","http_code":500,"err_details":"Get https://api.integrations.mattermost.com/api/v1/plugins?filter=&local_only=false&page=0&server_version=5.21.0: proxyconnect tcp: address forbidden, you may need to set AllowedUntrustedInternalConnections to allow an integration access to your internal network"}`

When curl this URL on the command line I get full response from marketplace:  
`curl -x proxy:port "https://api.integrations.mattermost.com/api/v1/plugins?filter=&local_only=false&page=0&server_version=5.21.0" -v`

Also tried mentioned setting “AllowedUntrustedInternalConnections” but without success. Guess this setting is not relevant.

Is there a way to install plugins manually? Didn’t find any docs for this. Can someone please point me there?

Many thanks

---

<div class="post-metadata">

**Author:** ![wbaeck](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wbaeck](https://forum.mattermost.com/u/wbaeck)\
**Post date:** [April 16, 2020, 3:47pm UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/4 "2020-04-16T15:47:31Z")

</div>

Any idea, at least where to look at?

---

<div class="post-metadata">

**Author:** ![amy.blais](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/amy.blais/32/7481_2.png) [@amy.blais](https://forum.mattermost.com/u/amy.blais)\
**Post date:** [April 17, 2020, 1:16pm UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/5 "2020-04-17T13:16:14Z")

</div>

@ahmaddanial @joewai.tye Would anyone on support team be familiar with this issue?

---

<div class="post-metadata">

**Author:** ![ahmaddanial](https://sea2.discourse-cdn.com/flex020/user_avatar/forum.mattermost.com/ahmaddanial/32/2938_2.png) [@ahmaddanial](https://forum.mattermost.com/u/ahmaddanial)\
**Post date:** [April 17, 2020, 4:01pm UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/6 "2020-04-17T16:01:51Z")

</div>

Hello, @wbaeck

A couple of questions to ask to help me understand the issue better:

- What type of proxy are you configuring for you Mattermost?

- Can you share the current configuration of the `mattermost.service` based on the [Sample Configuration](https://docs.mattermost.com/install/outbound-proxy.html#sample-configuration) with the sensitive information redacted if any?

- Can you also share the `ServiceSettings` configuration with any sensitive information removed here?

```auto
cat /opt/mattermost/config/config.json | grep -A91 "ServiceSettings"

```

If you are looking at installing plugins manually, you need to ensure that `Enable` and `EnableUploads` are set to true. For example:

```auto
ahmaddanial@mattermost:~$ cat /opt/mattermost/config/config.json | grep -A19 "PluginSettings"
    "PluginSettings": {
        "Enable": true,
        "EnableUploads": true,
        "AllowInsecureDownloadUrl": true,
        "EnableHealthCheck": true,
        "Directory": "./plugins",
        "ClientDirectory": "./client/plugins",
        "Plugins": {},
        "PluginStates": {
            "com.mattermost.nps": {
                "Enable": true
            }
        },
        "EnableMarketplace": true,
        "EnableRemoteMarketplace": true,
        "AutomaticPrepackagedPlugins": true,
        "RequirePluginSignature": false,
        "MarketplaceUrl": "https://api.integrations.mattermost.com",
        "SignaturePublicKeyFiles": []
    },

```

Once done, run the following command to ensure that the right ownership is granted to all directories in the `/opt/mattermost` directory:

```auto
sudo chown -R mattermost:mattermost /opt/mattermost/

```

Then, you can download the `.tar.gz` of a plugin (for example - [Mattermost Jira Plugin](https://github.com/mattermost/mattermost-plugin-jira/releases/tag/v2.3.2) ) and upload it through **System Console \> Plugins (Beta) \> Plugin Management**. For example:

 ![image](https://us1.discourse-cdn.com/flex020/uploads/mattermost/original/2X/e/e2f3843dd5938ecfe2643d828ff4b60a3b82e041.jpeg)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/mattermost/original/2X/a/a75ee4e6003ac7e5150b5cb4449d4a76b1194606.jpeg)

 ![image](https://us1.discourse-cdn.com/flex020/uploads/mattermost/original/2X/f/f8e554424e55acc3fc6162d4ca331a47c0cff9fa.jpeg)

Can you please let me know if that works for you?

---

<div class="post-metadata">

**Author:** ![wbaeck](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wbaeck](https://forum.mattermost.com/u/wbaeck)\
**Post date:** [April 17, 2020, 9:25pm UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/7 "2020-04-17T21:25:15Z")

</div>

Hi @ahmaddanial

Thank you very much for your reply.

1. What type of proxy are you configuring for you Mattermost?  
In our environment every access to the internet (for user and server) is done via a central proxy. You don’t need any authentication but you need a proxy certificate to be able to pass the proxy. This is due to SSL-security reasons. So every client needs this certificate to pass the proxy. For me the question is where to put our proxy certificates so mattermost/golang will pick them up.
2. “mattermost.service”

> [@](#):
>
> [Unit]  
> Description=Mattermost  
> After=syslog.target network.target mysqld.service
> 
> [Service]  
> Type=notify  
> WorkingDirectory=/app/mattermost  
> User=mattermost  
> ExecStart=/app/mattermost/bin/mattermost  
> PIDFile=/var/spool/mattermost/pid/master.pid  
> TimeoutStartSec=3600  
> LimitNOFILE=49152  
> Environment=HTTPS\_PROXY=[https://proxy:3113](https://proxy:3113)
> 
> [Install]  
> WantedBy=multi-user.target

1. “ServiceSettings”

> [@](#):
>
> ```
> "ServiceSettings": {
> "SiteURL": "https://chat.abc.org",
> "WebsocketURL": "",
> "LicenseFileLocation": "",
> "ListenAddress": ":8065",
> "ConnectionSecurity": "",
> "TLSCertFile": "",
> "TLSKeyFile": "",
> "TLSMinVer": "1.2",
> "TLSStrictTransport": false,
> "TLSStrictTransportMaxAge": 63072000,
> "TLSOverwriteCiphers": [],
> "UseLetsEncrypt": false,
> "LetsEncryptCertificateCacheFile": "./config/letsencrypt.cache",
> "Forward80To443": false,
> "TrustedProxyIPHeader": [
> "X-Forwarded-For",
> "X-Real-IP"
> ],
> "ReadTimeout": 300,
> "WriteTimeout": 300,
> "MaximumLoginAttempts": 10,
> "GoroutineHealthThreshold": -1,
> "GoogleDeveloperKey": "",
> "EnableOAuthServiceProvider": false,
> "EnableIncomingWebhooks": true,
> "EnableOutgoingWebhooks": true,
> "EnableCommands": true,
> "EnableOnlyAdminIntegrations": true,
> "EnablePostUsernameOverride": false,
> "EnablePostIconOverride": false,
> "EnableLinkPreviews": false,
> "EnableTesting": false,
> "EnableDeveloper": false,
> "EnableSecurityFixAlert": true,
> "EnableInsecureOutgoingConnections": false,
> "AllowedUntrustedInternalConnections": "chat.abc.org gitlab-internal.abc.org 10.8.0.0/16 10.9.0.0/16",
> "EnableMultifactorAuthentication": false,
> "EnforceMultifactorAuthentication": false,
> "EnableUserAccessTokens": false,
> "AllowCorsFrom": "",
> "CorsExposedHeaders": "",
> "CorsAllowCredentials": false,
> "CorsDebug": false,
> "AllowCookiesForSubdomains": false,
> "SessionLengthWebInDays": 180,
> "SessionLengthMobileInDays": 180,
> "SessionLengthSSOInDays": 30,
> "SessionCacheInMinutes": 10,
> "SessionIdleTimeoutInMinutes": 43200,
> "WebsocketSecurePort": 443,
> "WebsocketPort": 80,
> "WebserverMode": "gzip",
> "EnableCustomEmoji": false,
> "EnableEmojiPicker": true,
> "EnableGifPicker": false,
> "GfycatApiKey": "xxxxxxxxxx",
> "GfycatApiSecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
> "RestrictCustomEmojiCreation": "all",
> "RestrictPostDelete": "all",
> "AllowEditPost": "always",
> "PostEditTimeLimit": -1,
> "TimeBetweenUserTypingUpdatesMilliseconds": 5000,
> "EnablePostSearch": true,
> "MinimumHashtagLength": 3,
> "EnableUserTypingMessages": true,
> "EnableChannelViewedMessages": true,
> "EnableUserStatuses": true,
> "ExperimentalEnableAuthenticationTransfer": true,
> "ClusterLogTimeoutMilliseconds": 2000,
> "CloseUnusedDirectMessages": false,
> "EnablePreviewFeatures": true,
> "EnableTutorial": true,
> "ExperimentalEnableDefaultChannelLeaveJoinMessages": true,
> "ExperimentalGroupUnreadChannels": "disabled",
> "ExperimentalChannelOrganization": false,
> "ImageProxyType": "",
> "ImageProxyURL": "",
> "ImageProxyOptions": "",
> "EnableAPITeamDeletion": false,
> "ExperimentalEnableHardenedMode": false,
> "DisableLegacyMFA": true,
> "ExperimentalStrictCSRFEnforcement": false,
> "EnableEmailInvitations": true,
> "DisableBotsWhenOwnerIsDeactivated": true,
> "EnableBotAccountCreation": false,
> "EnableSVGs": false,
> "EnableLatex": false
> },
> "TeamSettings": {
> "SiteName": "Internal Chat",
> "MaxUsersPerTeam": 100,
> 
> ```

1. “PluginSettings”

> [@](#):
>
> ```
> "PluginSettings": {
> "Enable": true,
> "EnableUploads": true,
> "AllowInsecureDownloadUrl": false,
> "EnableHealthCheck": true,
> "Directory": "./plugins",
> "ClientDirectory": "./client/plugins",
> "Plugins": {
> "com.mattermost.nps": {
> "enablesurvey": false
> }
> },
> "PluginStates": {
> "com.mattermost.nps": {
> "Enable": false
> }
> },
> "EnableMarketplace": true,
> "EnableRemoteMarketplace": true,
> "AutomaticPrepackagedPlugins": true,
> "RequirePluginSignature": false,
> "MarketplaceUrl": "https://api.integrations.mattermost.com",
> "SignaturePublicKeyFiles": []
> },
> 
> ```

* * *

When I open the Marketplace inside of Mattermost I get the following log entry:

```
{"level":"error","ts":1587156881.5682132,"caller":"mlog/log.go:175","msg":"Failed to get plugins from the marketplace server.","path":"/api/v4/plugins/marketplace","request_id":"yc9g8qxnn3rqdcdkkspqstr6wy","ip_addr":"10.179.9.81","user_id":"roeyih4u1p8g5y83o8irtfn4no","method":"GET","err_where":"getRemotePlugins","http_code":500,"err_details":"Get https://api.integrations.mattermost.com/api/v1/plugins?filter=&local_only=false&page=0&server_version=5.21.0: proxyconnect tcp: EOF"}

```

And in the UI it says:

> [@](#):
>
> Error connecting to the marketplace server. Please check your settings in the [System Console](https://xxxxx).
> 
> There are no plugins available at this time.

For me the strange thing is that there is no error message at all. It just says “proxyconnect tcp: EOF”. There are no other entries in the log. Are there any other places to search or enable more logging?

* * *

I was reading that golang picks up certificates from defined places as documented here: [https://golang.org/src/crypto/x509/root\_linux.go](https://golang.org/src/crypto/x509/root_linux.go)

So for RedHat 7 (we are using) it must be at /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem? I verified this file and the proxy cert is already there.

Hope that helps for narrow the problem. Please let me know if you need additional information.

Thanks

---

<div class="post-metadata">

**Author:** ![wbaeck](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wbaeck](https://forum.mattermost.com/u/wbaeck)\
**Post date:** [April 17, 2020, 11:34pm UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/8 "2020-04-17T23:34:28Z")

</div>

@ahmaddanial

Just found this in the documentation of Mattermost ([Redirect](https://docs.mattermost.com/install/outbound-proxy.html)):

> [@](#):
>
> Note that when proxying HTTPS resources, you will need to configure your Mattermost server with the root certificate of the proxy. Otherwise, Mattermost will refuse any response from the proxy as it will detect that its connection has been intercepted.

I think this is my issue but it’s not explained how to configure. Does someone knows how to?

Thanks

---

<div class="post-metadata">

**Author:** ![wbaeck](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wbaeck](https://forum.mattermost.com/u/wbaeck)\
**Post date:** [April 22, 2020, 1:07pm UTC](https://forum.mattermost.com/t/outbound-proxy-with-certificate/9390/9 "2020-04-22T13:07:43Z")

</div>

@ahmaddanial

Any ideas on this?

Many thanks