All clients cannot upload files

Greetings all!
I have the following problem, and I hope you can help me .

Summary
Cannot upload files via all apps. Win/Mac/Web/iOS/Android.

Steps to reproduce
Drag&drop or click upload button to choose a file.

Expected behavior
Load bar to finish, and “send” arrow to be active. (It’s not during “uploading”)

Observed behavior
File starts icon for uploading, but never stops.
ONLY Twice I was able to see and error, but it never saw it againg. Just twice.
urlsessiontask failed with error: the request timed out

Mattermost_rabvK16fcp

Mattermost Version: 7.8.0
Database Schema Version: 101
Database: mysql
Build Hash: f14c24391ca84740d90a29ec8b67579326eaa236
EE Build Hash: 63eff41e111e2636bb79bd27049e254ddcdb2d6f
Webapp Build Hash: ba673fb7226097a165d561ae88f2f9059807c166
Checking for update
Update for version 5.4.0 is not available (latest version: 5.4.0, downgrade is disallowed)
All other apps like iOS, android etc are tested with latest version
Selfhosted on a separate VM server- Debian 10.

“send” arrow is not active during trying to upload.
Web Devtools show nothing interesting, as errors in console or network response

“FileSettings”: {
“EnableFileAttachments”: true,
“EnableMobileUpload”: true,
“EnableMobileDownload”: true,
“MaxFileSize”: 12428800,
“MaxImageResolution”: 33177600,
“MaxImageDecoderConcurrency”: -1,
“DriverName”: “local”,
“Directory”: “./data/”,
“EnablePublicLink”: false,
“ExtractContent”: true,
“ArchiveRecursion”: false,
“PublicLinkSalt”: “AndPepper”,
“InitialFont”: “nunito-bold.ttf”,
“AmazonS3AccessKeyId”: “”,
“AmazonS3SecretAccessKey”: “”,
“AmazonS3Bucket”: “”,
“AmazonS3PathPrefix”: “”,
“AmazonS3Region”: “”,
“AmazonS3Endpoint”: “s3.amazonaws.com”,
“AmazonS3SSL”: true,
“AmazonS3SignV2”: false,
“AmazonS3SSE”: false,
“AmazonS3Trace”: false,
“AmazonS3RequestTimeoutMilliseconds”: 30000
},

Run Diagnostic

Only this error is available, but I think I’ve seen it before and doubt to couse the issue

{
message: ‘Step-3 failed’,
succeeded: false,
payload: { code: ‘ERR_INVALID_PROTOCOL’ },
duration: 2,
name: ‘diagnostic-Step-3: serverConnectivity’,
step: 3
},

[2023-06-27 13:30:19.815] [info] | index | name | succeeded |
[2023-06-27 13:30:19.816] [info] | — | — | — |
[2023-06-27 13:30:19.816] [info] | 0 | diagnostic-Step-0: logConfig | true |
[2023-06-27 13:30:19.816] [info] | 1 | diagnostic-Step-1: internetConnection | true |
[2023-06-27 13:30:19.816] [info] | 2 | diagnostic-Step-2: configValidation | true |
[2023-06-27 13:30:19.817] [info] | 3 | diagnostic-Step-3: serverConnectivity | false |
[2023-06-27 13:30:19.817] [info] | 4 | diagnostic-Step-4: sessionDataValidation | true |
[2023-06-27 13:30:19.817] [info] | 5 | diagnostic-Step-5: BrowserWindowsChecks | true |
[2023-06-27 13:30:19.817] [info] | 6 | diagnostic-Step-6: PermissionsCheck | true |
[2023-06-27 13:30:19.818] [info] | 7 | diagnostic-Step-7: PerformanceAndMemory | true |
[2023-06-27 13:30:19.818] [info] | 8 | diagnostic-Step-8: LogHeuristics | true |
[2023-06-27 13:30:19.818] [info] | 9 | diagnostic-Step-9: Config | true |
[2023-06-27 13:30:19.818] [info] | 10 | diagnostic-Step-10: CrashReports | true |
[2023-06-27 13:30:19.818] [info] | 11 | diagnostic-Step-11: AuthSSO | true |
[2023-06-27 13:30:19.819] [info] | — | — | — |
[2023-06-27 13:30:19.819] [info] 11 out of 12 steps succeeded

The most interesting from the log files "
============================ LOG FILES ======================================

Line 6: {"timestamp":"2023-06-27 07:29:28.693 -04:00","level":"debug","msg":"CSRF Header check failed for request - Please upgrade your web application or custom app to set a CSRF Header","caller":"web/handlers.go:454","path":"/api/v4/notifications/ack","ip":"exampleIP:34501","session_id":"3p9jc4c6zinwibagion6ss3b9o","user_id":"u7rwcubb5bbibdwwzt7iqrqyuh"}
	Line 13: {"timestamp":"2023-06-27 07:29:29.526 -04:00","level":"debug","msg":"failed to fetch app bindings","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","plugin_caller":"server/proxy/bindings.go:67","path":"/api/v1/bindings","error":"1 error occurred:\n\t* received app error: The system setting `Enable OAuth 2.0 Service Provider` needs to be enabled in order for the Apps plugin to work. Please go to http://matrix.pcloud.com/admin_console/integrations/integration_management and enable it.\n\n","request_id":"o9jeszuj9f8ztpjfxh4mqded8c","from_user_id":"rb3ixy5oypnh3eadrppuqn669a"}
	Line 48: {"timestamp":"2023-06-27 07:29:31.536 -04:00","level":"debug","msg":"CSRF Header check failed for request - Please upgrade your web application or custom app to set a CSRF Header","caller":"web/handlers.go:454","path":"/api/v4/files","ip":"exampleIP:34380","session_id":"r7476bdjrjyu7e4dhos38ngiqc","user_id":"jbunaam4ci853mig3himyixa7a"}
	
	Line 421: [2022-07-11 20:03:37.293] [error] Error: Error: net::ERR_NETWORK_IO_SUSPENDED
	Line 424: [2022-07-11 20:03:37.293] [error] [Mattermost] There was an error while trying to update: Error: net::ERR_NETWORK_IO_SUSPENDED
	Line 425: [2022-07-11 20:03:37.294] [error] [Mattermost] Failed to check for updates: Error: net::ERR_NETWORK_IO_SUSPENDED
	Line 427: [2022-07-11 21:41:28.497] [error] Error: Error: net::ERR_NETWORK_CHANGED
	
	Line   2: {"timestamp":"2023-06-27 05:53:38.011 -04:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/groups","request_id":"1234","ip_addr":"999.99.99.72","user_id":"7qbm31e","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
	Line  36: {"timestamp":"2023-06-27 05:53:47.865 -04:00","level":"info","msg":"api endpoint requires a license","caller":"web/context.go:115","path":"/api/v4/users/7qbm31e/groups","request_id":"zacnxnxos7ym5fyh13sjbkyp3e","ip_addr":"999.99.99.72","user_id":"7qbm31e","method":"GET","err_where":"","http_code":501,"error":": api endpoint requires a license"}
	Line  38: {"timestamp":"2023-06-27 05:53:47.872 -04:00","level":"debug","msg":"failed to fetch app bindings","caller":"app/plugin_api.go:970","plugin_id":"com.mattermost.apps","request_id":"1znjrie1jjyizr8dqj4cyegpew","from_user_id":"7qbm31e","plugin_caller":"server/proxy/bindings.go:67","path":"/api/v1/bindings","error":"1 error occurred:\n\t* received app error: The system setting `Enable OAuth 2.0 Service Provider` needs to be enabled in order for the Apps plugin to work. Please go to http://matrix.pcloud.com/admin_console/integrations/integration_management and enable it.\n\n"}

============================ LOG FILES ======================================

I checked some other similar issues, but most of them were suggesting testing newer versions. Currently up to date, so I doubt that’s the problem.

Thanks in advance.

Hi Ghosty and welcome to the Mattermost forums!

7.8.0 is supposed to have the fix for this issue, but it’s not the latest version, so upgrading to 7.10.3 should still be an option for you to rule out that there’s some other bug that prevents you from uploading files successfully.
Do you have an antivirus plugin running in Mattermost that could be the reason for the stuck uploads?

Thanks, Alexander.

I couldn’t wait any more and this morning I restarted the service. This seems to have fixed the issue. Sadly still do not know what caused it.

Thanks for the advice.

Yes, the restart is exactly what fixes it (temporarily), but the underlying issue has been fixed in a more recent release, so please upgrade ASAP.