I am part of the team building OpenZiti, and our new ‘browZer’ capabilities. We run Mattermost Enterprise 7.0.1 internally at our company (protected by OpenZiti).
I am testing some new browZer software by accessing our Mattermost instance over it. One thing I am running into during testing is that after I view a channel, if I click into another channel, and then click back to the original channel, that channel does not re-render, it just stays in the ‘Loading…’ phase.
I believe this behavior is caused by something in my browZer software that is upsetting the MM web app somehow, and this is not a MM bug, so…
I was wondering if you could give me some clues as to what REST calls and/or WebSocket calls the MM app is doing when a channel is re-clicked so that I can focus on those to determine if they might not be completing correctly under browZer, and thus I am causing the channel re-render stalls.
No errors appear in the dev tools console, btw.
Hi rentallect and welcome to the Mattermost forums!
What happens if you reload the page while you see the “Loading…” screen, does it then work again or what do you need to do to get it fixed and displaying correctly?
Yes, the reload button (complete web app reload) does render the channel again.
The OpenZiti BrowZer software that I mentioned is not a “proxy”, but for the purposes of this conversation, it can probably be considered one, because we intercept (some) network traffic originating in the web app, and route it over a Ziti network.
Have there been any issues or gotcha’s using MM in environments where a proxy was involved? If so, I’d love to hear the details to see if they relate to what I’m seeing.
Examples/possibilities could involve mistakenly altering the HTTP req/resp content, headers, etc., instead of remaining transparent to the web app. Or maybe introducing some delayed responses, that might confuse the web app?
Ah, gotcha. Well, the client applications use websockets for several purposes and update streaming for events between the server and the clients. Does this re-routing or proxynetwork maybe have multiple source IPs that can act as the connected client to the mattermost server for the same connection then? I recall having issues like that where multiple involved subdomains for a remote management console screen session where routed via different paths on an outbound loadbalancer which also caused the tunneled connections to drop and one other problem were invalid JWTs which have been assigned to the control connection on IP 1 (when the data connection then came from IP 2).
The OpenZiti browZer software lives inside the browser (Chrome, Brave, etc). How it gets there in an unmodified MM web app is a story I’ll save for another day. But suffice it to say we get injected into the browser as the MM web app is loading, and then we intercept all fetch, XHR, WebSocket calls being done by the MM web app.
We then determine if the request targets a Ziti network resource (e.g. the protected MM server itself) or if the request targets the raw internet (e.g. hits to https://pdat.matterlytics.com, or some 3rd-party image site).
Only requests targeting resources protected by Ziti will be routed over the Ziti network. The rest get tossed to the raw internet (like how any web app typically runs).
Regarding your question about multiple IPs: No, we do not involve that scenario. As fas as the MM web app is concerned, it is loaded from a single server, and all REST/WS calls go to that server.
My gut is telling me that some REST call is made when the UI gesture to re-view a channel is made, and that REST call is not completing as expected (as in, either it stalls and never completes, or it does complete, but the response contains data that the UI doesn’t like/expect, and thus, the ‘Loading…’ stays up because the normal data-load/render flow gets broken somehow.
I just need to know which of the REST calls might be of the type I describe above, if there are any.
I think I found a clue:
When running native (without browZer involved), when re-entering a channel, I see the web app issue the following REST call:
However, when running with browZer involved, I see the web app issue the following REST call:
Note the fact that the
since=<TIMESTAMP> parm isn’t there and was replaced by some other parms that seem to want to go back to the beginning of time (…and thus fetch a ton of data). This data starts coming back with chunked encoding, but I’m not interested in chasing down whether or not “the entire channel history” comes back… I’m more interested in the following:
I am wondering what logic in the web app is involved with making the choice of using the
since parm or not.
Can you help me track down which prior REST calls might have been involved in informing the decision to subsequently use the
since parm? (and that I might have corrupted the response data to the extent necessary that the web app simply chooses not to use timestamps)
I think that if I understand that, we will solve the ‘Loading…’ mystery.
Looking at the code, it seems to be related to disconnected websockets. If the socket stays intact, only newer messages will be fetched, if it disconnects, all messages will be read.