Import From Slack: Error Uploading Data

The mmetl transform slack command is used to work around that when you specify the correct --team command line parameter. I really need to see the full output of the commands I’m asking for and also see the commands you entered in order to help you, it’s hard enough to diagnose such problems over the internet without access to the system…

What did your mmetl transform slack command look like and is this team already created in Mattermost? It needs to be there, the importer will not create the team.

I’m really not sure how i would show you the full output of the command it is HUGE and i tried copying and pasting and for some reason it is not doing that correctly.

to be honest i dont recall what the mmetl transform slack command looked like that was way too many commands ago and yest the team is already created in mattermost.

Give me a minute and i’ll attempt to somehow copy the full results of that last command

import validation error in “mattermost-bulk-import.zip”->“mattermost_import.jsonl”:1191 field “post.team”: reference to unknown team “slackimport”
[… repeated errors on different lines …]

import validation error in “mattermost-bulk-import.zip”->“mattermost_import.jsonl”:1291 field “post.team”: reference to unknown team “slackimport”
[… repeated errors on different lines …]

import validation error in “mattermost-bulk-import.zip”->“mattermost_import.jsonl”:1391 field “post.team”: reference to unknown team “slackimport”
[… repeated errors on different lines …]

this goes on for over 2,000 lines i can post them all if you want but i’ll try and point out anything unique:
import validation error in “mattermost-bulk-import.zip”->“mattermost_import.jsonl”:2142 field “post”: BulkImport: app.import.validate_post_import_data.message_length.error,

Schemes (0):  []                                                                                                                                                                              
Teams (0):    []                                                                                                                                                                              
Channels (35): [bankaccount, condo, deal-analysis-and-offer-sheets, ...]                                                                                               
Users (3):    [user1, user2, slack_advanced_export]                                                                                                                                    
Emojis (0):   []                                                                                                                                                                              
Posts (3141)                                                                                                                                                                                  
Direct Channels (0)                                                                                                                                                                           
Attachments (557): [data/bulk-export-attachments/F01C6SPK1KR_Screenshot_20201014-193625_Chrome.jpg, data/bulk-export-attachments/F01CP5J5JTY_image.png, ...]                                  
Unused Attachments (47):                                                                                                                                                                      
  data/bulk-export-attachments/F01KBV8JB26_image.png                                                                                                                                          
  data/bulk-export-attachments/F01KJ4QDQ4U_image.png                                                                                                                                          
  data/bulk-export-attachments/F01L1QEK8M6_Screenshot_20210120-155143_Chrome.jpg                                                                                                              
  data/bulk-export-attachments/F01L9A44MPY_image.png                                                                                                                                          
  data/bulk-export-attachments/F01LG9KUG03_image.png                                                                                                                                          
  data/bulk-export-attachments/F01LN8DAFJQ_image.png                                                                                                                                          
  data/bulk-export-attachments/F01M5UA4VA4_image.png                                                                                                                                          
  data/bulk-export-attachments/F01PER9JNNA_Screenshot_20210228-113035_Outlook.jpg                                                                                                             
  data/bulk-export-attachments/F020Z2KHBTQ_image.png                                                                                                                                          
  data/bulk-export-attachments/F0212P8PBTP_image.png                                                                                                                                          
  data/bulk-export-attachments/F02161Z6GE7_image.png                                                                                                                                          
  data/bulk-export-attachments/F023H3YTPC6_Wasabi2019.pdf                                                                                                                                     
  data/bulk-export-attachments/F023PSVH09Y_Revenue and Receipts Report for 560 2nd Ave. from 01-01-2021 to 06-01-2021.pdf                                                                     
  data/bulk-export-attachments/F023Q3RNQ67_Revenue and Receipts Report for 560 2nd Ave. from 01-01-2020 to 01-01-2021.pdf                                                                     
  data/bulk-export-attachments/F029FKPCTKQ_image.png                                                                                                                                          
  data/bulk-export-attachments/F02EZ13PXD3_JB-30169376.pdf                                                                                                                                    
  data/bulk-export-attachments/F02F21L65FC_JB-30169376 (1).pdf                                                                                                                                
  data/bulk-export-attachments/F02F22B2M7D_image.png                                                                                                                                          
  data/bulk-export-attachments/F02F79A0L3H_Trinity Episcopal.docx                                                                                                                             
  data/bulk-export-attachments/F02FK0PFHGV_Garry Sampson 144-96-34.docx                                                                                                                       
  data/bulk-export-attachments/F02FN0X15KL_Pisenti.docx                                                                                                                                       
  data/bulk-export-attachments/F02GB0ZEHHD_Dunn County - 144-95-13.pdf                                                                                                                        
  data/bulk-export-attachments/F02GBKEEX08_Garry Sampson 145-97-27,34,35.docx                                                                                                                 
  data/bulk-export-attachments/F02GNMJR6TX_Dunn County - 144-96-34.pdf                                                                                                                        
  data/bulk-export-attachments/F02H49S1H17_Dunn County - 144-97-26.pdf                                                                                                                        
  data/bulk-export-attachments/F02J0HTBF0S_3.1 Deed                                                                                 
  data/bulk-export-attachments/F02JHB1MC1E_30418957 - 1001005030.csv                                                                                                                          
  data/bulk-export-attachments/F02U1DTNHN2_taFyqO-FTcTw-1-8.csv                                                                                                                               
  data/bulk-export-attachments/F02U857EYN7_taFyqO-FTcTw-1-10.csv                                                                                                                              
  data/bulk-export-attachments/F02U857GJD9_taFyqO-FTcTw-1-1.csv                                                                                                                               
  data/bulk-export-attachments/F02UDFD2RD2_taFyqO-FTcTw-1-44.csv                                                                                                                              
  data/bulk-export-attachments/F02ULP6NA3B_taFyqO-FTcTw-1-7.csv                                                                                                                               
  data/bulk-export-attachments/F02ULP6Q1EV_taFyqO-FTcTw-1-9.csv                                                                                                                               
  data/bulk-export-attachments/F02UPRCEMBK_taFyqO-FTcTw-1-56.csv                                                                                                                              
  data/bulk-export-attachments/F033K3X927R_runs_1_13_combined.csv                                                                                                                             
  data/bulk-export-attachments/F033T1AMZ4N_Chevron 2.zip                                                                                                                                      
  data/bulk-export-attachments/F033ZL9Q60K_Chevron 1.zip                                                                                                                                      
  data/bulk-export-attachments/F034C9LHS2D_Chevron 3.zip                                                                                                                                      
  data/bulk-export-attachments/F034PE7JA4Q_t-VwGqLwrOz--1-10.pdf                                                                                                                              
  data/bulk-export-attachments/F0361S6K293_image.png                                                                                                                                          
  data/bulk-export-attachments/F0363N940P4_image.png                                                                                                                                          
  data/bulk-export-attachments/F036TEPR00G_image.png                                                                                                                                          
  data/bulk-export-attachments/F03A64DC934_tBzHuQGusUxW-2-300.png                                                                                                                             
  data/bulk-export-attachments/F03A64DK0TU_tBzHuQGusUxW-4-292.png                                                                                                                             
  data/bulk-export-attachments/F03A8EL4MHA_tBzHuQGusUxW-3-290.png                                                                                                                             
  data/bulk-export-attachments/F03AVSHA8FJ_tBzHuQGusUxW-2-301.png                                                                                                                             
  data/bulk-export-attachments/F03AVSHHADN_tBzHuQGusUxW-4-293.png                                                                                                                             
Validation complete                                                                                                                                                                           
root@mattermost ~/mmctl-beta#                                                                                                                                                                 

from my notes here was my mmetl transform command for some reason i’m thinking i had slackimport in quotes

sudo ./mmetl transform slack --team slackimport --file export-with-emails-and-attachments.zip --output mattermost_import.jsonl

Thanks, this is helpful!
Do you still have the mattermost_import.jsonl file lying around somewhere? If so, could you please post the output of the following command:

head -3 /path/to/mattermost_import.jsonl

It should show references to the teamname. In my test export it looks like this:

{"type":"version","version":1}
{"type":"channel","channel":{"team":"slackimport","name":"testchannel1","display_name":"testchannel1","type":"O","header":"","purpose":""}}
{"type":"channel","channel":{"team":"slackimport","name":"testchannel2","display_name":"testchannel2","type":"O","header":"","purpose":""}}

Does the number of channels, users, attachments and posts sound reasonable?

root@mattermost ~# head -3 /opt/mattermost/data/mattermost_import.jsonl                                                                                                                       
{"type":"version","version":1}                                                                                                                                                                
{"type":"channel","channel":{"team":"slackimport","name":"general","display_name":"general","type":"O","header":"Company-wide announcements and work-based matters","purpose":"This channel is
 for workspace-wide communication and announcements. All members are in this channel."}}                                                                                                      
{"type":"channel","channel":{"team":"slackimport","name":"database","display_name":"database","type":"O","header":"","purpose":""}}                                                           
root@mattermost ~#                                                                                                                                                                            

In my slack instance i have 35ish channels in the slackimport team in mattermost i only have two channels Off-Topic and Town Square

OK, so the quotes have not been carried over to the file, otherwise it would have been mentioned there.

I forgot to add the --team parameter for the mmctl validation, sorry.
Please run:

cd /root/mmctl-beta
./mmctl import validate /opt/mattermost/data/import/mattermost-bulk-import.zip --team slackimport

Does that help with reducing the error messages?

root@mattermost ~# cd root/mmctl-beta
-bash: cd: root/mmctl-beta: No such file or directory

root@mattermost ~# mkdir /root/mmctl-beta
mkdir: cannot create directory ‘/root/mmctl-beta’: File exists
root@mattermost ~# cd/root/mmctl-beta
-bash: cd/root/mmctl-beta: No such file or directory
root@mattermost ~#

nevermind typo on my end got it

root@mattermost ~/mmctl-beta# ./mmctl import validate /opt/mattermost/data/import/mattermost-bulk-import.zip --team slackimport
Predefined teams: slackimport
The .jsonl file has 3180 lines
import validation error in “mattermost-bulk-import.zip”->“mattermost_import.jsonl”:498 field “post”: BulkImport: app.import.validate_post_import_data.message_length.error,
import validation error in “mattermost-bulk-import.zip”->“mattermost_import.jsonl”:2142 field “post”: BulkImport: app.import.validate_post_import_data.message_length.error,
Schemes (0):
Teams (1): [slackimport]
Channels (35): [bankaccount, condo, database, …]
Users (3): [user1, user2, slack_advanced_export]
Emojis (0):
Posts (3141)
Direct Channels (0)
Attachments (557): [data/bulk-export-attachments/F01C6SPK1KR_Screenshot_20201014-193625_Chrome.jpg, data/bulk-export-attachments/F01CP5J5JTY_image.png, …]
Unused Attachments (47):
data/bulk-export-attachments/F01KBV8JB26_image.png
data/bulk-export-attachments/F01KJ4QDQ4U_image.png
data/bulk-export-attachments/F01L1QEK8M6_Screenshot_20210120-155143_Chrome.jpg
data/bulk-export-attachments/F01L9A44MPY_image.png
data/bulk-export-attachments/F01LG9KUG03_image.png
data/bulk-export-attachments/F01LN8DAFJQ_image.png
data/bulk-export-attachments/F01M5UA4VA4_image.png
data/bulk-export-attachments/F01PER9JNNA_Screenshot_20210228-113035_Outlook.jpg
data/bulk-export-attachments/F020Z2KHBTQ_image.png
data/bulk-export-attachments/F0212P8PBTP_image.png
data/bulk-export-attachments/F02161Z6GE7_image.png
data/bulk-export-attachments/F023H3YTPC6_Wasabi2019.pdf
data/bulk-export-attachments/F023PSVH09Y_Revenue and Receipts Report for 560 2nd Ave. from 01-01-2021 to 06-01-2021.pdf
data/bulk-export-attachments/F023Q3RNQ67_Revenue and Receipts Report for 560 2nd Ave. from 01-01-2020 to 01-01-2021.pdf
data/bulk-export-attachments/F029FKPCTKQ_image.png
data/bulk-export-attachments/F02EZ13PXD3_JB-30169376.pdf
data/bulk-export-attachments/F02F21L65FC_JB-30169376 (1).pdf
data/bulk-export-attachments/F02F22B2M7D_image.png
data/bulk-export-attachments/F02F79A0L3H_.docx
data/bulk-export-attachments/F02FK0PFHGV_ 144-96-34.docx
data/bulk-export-attachments/F02FN0X15KL_.docx
data/bulk-export-attachments/F02GB0ZEHHD_Dunn County - 144-95-13.pdf
data/bulk-export-attachments/F02GBKEEX08_ 145-97-27,34,35.docx
data/bulk-export-attachments/F02GNMJR6TX_Dunn County - 144-96-34.pdf
data/bulk-export-attachments/F02H49S1H17_Dunn County - 144-97-26.pdf
data/bulk-export-attachments/F02J0HTBF0S_3.1 Deed
data/bulk-export-attachments/F02JHB1MC1E_30418957 - 1001005030.csv
data/bulk-export-attachments/F02U1DTNHN2_taFyqO-FTcTw-1-8.csv
data/bulk-export-attachments/F02U857EYN7_taFyqO-FTcTw-1-10.csv
data/bulk-export-attachments/F02U857GJD9_taFyqO-FTcTw-1-1.csv
data/bulk-export-attachments/F02UDFD2RD2_taFyqO-FTcTw-1-44.csv
data/bulk-export-attachments/F02ULP6NA3B_taFyqO-FTcTw-1-7.csv
data/bulk-export-attachments/F02ULP6Q1EV_taFyqO-FTcTw-1-9.csv
data/bulk-export-attachments/F02UPRCEMBK_taFyqO-FTcTw-1-56.csv
data/bulk-export-attachments/F033K3X927R_runs_1_13_combined.csv
data/bulk-export-attachments/F033T1AMZ4N_Chevron 2.zip
data/bulk-export-attachments/F033ZL9Q60K_Chevron 1.zip
data/bulk-export-attachments/F034C9LHS2D_Chevron 3.zip
data/bulk-export-attachments/F034PE7JA4Q_t-VwGqLwrOz–1-10.pdf
data/bulk-export-attachments/F0361S6K293_image.png
data/bulk-export-attachments/F0363N940P4_image.png
data/bulk-export-attachments/F036TEPR00G_image.png
data/bulk-export-attachments/F03A64DC934_tBzHuQGusUxW-2-300.png
data/bulk-export-attachments/F03A64DK0TU_tBzHuQGusUxW-4-292.png
data/bulk-export-attachments/F03A8EL4MHA_tBzHuQGusUxW-3-290.png
data/bulk-export-attachments/F03AVSHA8FJ_tBzHuQGusUxW-2-301.png
data/bulk-export-attachments/F03AVSHHADN_tBzHuQGusUxW-4-293.png
Validation complete
root@mattermost ~/mmctl-beta#

Looks good except for these two lines. They are too long. You could check the jsonl file for these lines and manually truncate them or split them into two lines. You‘d have to recreate the zip then and try again. All other things look good now!

OK I went in and fixed those two lines within the jsonl and I understand i need to recreate the zip and go from there. But what I’m wondering is do i need to go in and delete all of those folders that were created in opt/mattermost/data/ there seems to be about 190-200 new folders that are named with a date.

I thought i had it this time for sure. here is the log - Also the folders that i had deleted prior are now back in the /opt/mattermost/data folder.

root@mattermost .../mattermost/data# cd /root/mmctl-beta                                                                                                                                      
root@mattermost ~/mmctl-beta# sudo ./mmctl import validate /opt/mattermost/data/import/mattermost-bulk-import.zip --team slackimport                                                          
Predefined teams: slackimport                                                                                                                                                                 
The .jsonl file has 3180 lines                                                                                                                                                                
Schemes (0):  []                                                                                                                                                                              
Teams (1):    [slackimport]                                                                                                                                                                   
Channels (35): [bankaccount, condo, channelname, database, deal-analysis-and-offer-sheets, ...]                                                                                               
Users (3):    [natwick, rperleberg, slack_advanced_export]                                                                                                                                    
Emojis (0):   []                                                                                                                                                                              
Posts (3141)                                                                                                                                                                                  
Direct Channels (0)                                                                                                                                                                           
Attachments (557): [data/bulk-export-attachments/F01C6SPK1KR_Screenshot_20201014-193625_Chrome.jpg, ...]                                  
Unused Attachments (47):                                                                                                                                                                      
  data/bulk-export-attachments/F01KBV8JB26_image.png                                                                                                                                          
  data/----truncated this list as its just 47 total file names                                                                                                                           
Validation complete                                                                                                                                                                                                                                             
root@mattermost .../data/import# cd /opt/mattermost/data/                                                                                                                                                                                                                                                                                                         
root@mattermost .../mattermost/data# ./mmctl import list available                                                                                                                            
mattermost-bulk-import.zip                                                                                                                                                                    
root@mattermost .../mattermost/data# ./mmctl import process mattermost-bulk-import.zip                                                                                                        
Import process job successfully created, ID: 8o7hkt3dai8fjyt95c6z7mxrqc                                                                                                                                                                                                                             
root@mattermost .../mattermost/data# ./mmctl import job show 8o7hkt3dai8fjyt95c6z7mxrqc --json                                                                                           
[                                                                                                                                                                                             
  {                                                                                                                                                                                           
    "id": "8o7hkt3dai8fjyt95c6z7mxrqc",                                                                                                                                                       
    "type": "import_process",                                                                                                                                                                 
    "priority": 0,                                                                                                                                                                            
    "create_at": 1663765903004,                                                                                                                                                               
    "start_at": 1663765906903,                                                                                                                                                                
    "last_activity_at": 1663765993804,                                                                                                                                                        
    "status": "success",                                                                                                                                                                      
    "progress": 0,                                                                                                                                                                            
    "data": {                                                                                                                                                                                 
      "import_file": "mattermost-bulk-import.zip"                                                                                                                                             
    }                                                                                                                                                                                         
  }                                                                                                                                                                                           
]                                                                                                                                                                                             
root@mattermost .../mattermost/data# systemctl restart mattermost  

This looks very good, actually.

Can you please run the following commands:

mmctl team list
mmctl channel list slackimport
mmctl user list

Can you see your team, your channels and your users when running these commands? Any superfluous information (like an additional team), or any other distorting entries?

root@mattermost .../mattermost/data# ./mmctl team list
                                                                                                                                                                                      
slackimport                                                                                                                                                                                   
                                                                                                                                                                                              
personal data removed --- all channels and etc did show up here

OK, if you cannot see the channels then you’re probably not in the right team with your user account.
Please make sure your user is member of the slackimport team and if it’s not, you can either add it to this team using the system console or using mmctl:

mmctl team users add slackimport <yourusername>

You should then (after refreshing the app) see the teams switcher bar in the leftmost pane and can switch between the teams.