Migration from MySQL to Postgres fails

For feature requests, please see: https://mattermost.com/suggestions/.

For troubleshooting questions, please post in the following format:

Summary
I tried to do migration from SQL to Postgress wtih help Migration guidelines from MySQL to PostgreSQL — Mattermost documentation

Steps to reproduce
Maybe if you generate same database and try to migrate?

Expected behavior
I expect, that data migration will work without errors and the postgres database can replace mysql database without loosing information

Observed behavior
I got some errors during migration with pgloader

Hi @all,

my old server is broken :frowning: but I did a backup with mysqldump :slight_smile:

Because I bought new Hardware I set up Mattermost and Postgress in a docker container. First I tried to migrate directly into the container, which failed…b

…now I installed MySQL, Postgress and pgloader without docker.

But Im still not able to perform migration.
Here is my migration.load file for pgloader. I did already some changes because there were some errors in the manual:

LOAD DATABASE
FROM mysql://mmuser:MyDatabasePassword@localhost:3306/mattermost
INTO pgsql://mmuser:MyDatabasePassword@localhost:5432/mattermost

WITH data only,
workers = 8, concurrency = 1,
multiple readers per thread, rows per range = 50000,
create no tables, create no indexes,
preserve index names

SET PostgreSQL PARAMETERS
maintenance_work_mem to ‘128MB’,
work_mem to ‘12MB’

SET MySQL PARAMETERS
net_read_timeout = ‘120’,
net_write_timeout = ‘120’

CAST column Channels.Type to “channel_type” drop typemod,
column Teams.Type to “team_type” drop typemod,
column UploadSessions.Type to “upload_session_type” drop typemod,
column Drafts.Priority to text,
type int when (= precision 11) to integer drop typemod,
type bigint when (= precision 20) to bigint drop typemod,
type text to varchar drop typemod,
type tinyint when (<= precision 4) to boolean using tinyint-to-boolean,
type json to jsonb drop typemod

EXCLUDING TABLE NAMES MATCHING ~‘IR_’, ~‘focalboard’, ‘schema_migrations’

BEFORE LOAD DO
$$ ALTER SCHEMA public RENAME TO mattermost; $$,
$$ DROP INDEX IF EXISTS idx_posts_message_txt; $$,
$$ DROP INDEX IF EXISTS idx_fileinfo_content_txt; $$

AFTER LOAD DO
$$ UPDATE mattermost.db_migrations set name=‘add_createat_to_teamembers’ where version=92; $$,
$$ CREATE INDEX IF NOT EXISTS idx_posts_message_txt ON mattermost.posts USING gin(to_tsvector(‘english’, message)); $$,
$$ CREATE INDEX IF NOT EXISTS idx_fileinfo_content_txt ON mattermost.fileinfo USING gin(to_tsvector(‘english’, content)); $$,
$$ ALTER SCHEMA mattermost RENAME TO public; $$,
$$ SELECT pg_catalog.set_config(‘search_path’, ‘“$user”, public’, false); $$,
$$ ALTER USER mmuser SET SEARCH_PATH TO ‘public’; $$;

This is how my mySQL-Database loooks like:

SHOW TABLES;
±------------------------------+
| Tables_in_mattermost |
±------------------------------+
| Audits |
| Bots |
| ChannelMemberHistory |
| ChannelMembers |
| Channels |
| ClusterDiscovery |
| CommandWebhooks |
| Commands |
| Compliances |
| DesktopTokens |
| Drafts |
| Emoji |
| FileInfo |
| GroupChannels |
| GroupMembers |
| GroupTeams |
| IR_Incident |
| IR_Playbook |
| IR_PlaybookAutoFollow |
| IR_PlaybookMember |
| IR_Run_Participants |
| IR_StatusPosts |
| IR_System |
| IR_TimelineEvent |
| IR_UserInfo |
| IR_ViewedChannel |
| IncomingWebhooks |
| Jobs |
| Licenses |
| LinkMetadata |
| NotifyAdmin |
| OAuthAccessData |
| OAuthApps |
| OAuthAuthData |
| OutgoingOAuthConnections |
| OutgoingWebhooks |
| PersistentNotifications |
| PluginKeyValueStore |
| PostAcknowledgements |
| PostReminders |
| Posts |
| PostsPriority |
| Preferences |
| ProductNoticeViewState |
| PublicChannels |
| Reactions |
| RecentSearches |
| RemoteClusters |
| RetentionIdsForDeletion |
| RetentionPolicies |
| RetentionPoliciesChannels |
| RetentionPoliciesTeams |
| Roles |
| Schemes |
| Sessions |
| SharedChannelAttachments |
| SharedChannelRemotes |
| SharedChannelUsers |
| SharedChannels |
| SidebarCategories |
| SidebarChannels |
| Status |
| Systems |
| TeamMembers |
| Teams |
| TermsOfService |
| ThreadMemberships |
| Threads |
| Tokens |
| TrueUpReviewHistory |
| UploadSessions |
| UserAccessTokens |
| UserGroups |
| UserTermsOfService |
| Users |
| db_lock |
| db_migrations |
| focalboard_blocks |
| focalboard_blocks_history |
| focalboard_board_members |
| focalboard_boards |
| focalboard_boards_history |
| focalboard_file_info |
| focalboard_notification_hints |
| focalboard_schema_migrations |
| focalboard_sessions |
| focalboard_sharing |
| focalboard_subscriptions |
| focalboard_system_settings |
| focalboard_teams |
| focalboard_users |
±------------------------------+
91 rows in set (0.01 sec)

But I all the time get the follwoing error, when running pgloader:

KABOOM!
FATAL error: Database error 3F000: schema “public” does not exist
QUERY: ALTER SCHEMA public RENAME TO mattermost;
An unhandled error condition has been signalled:
Database error 3F000: schema “public” does not exist
QUERY: ALTER SCHEMA public RENAME TO mattermost;

What I am doing here?

Database error 3F000: schema “public” does not exist
QUERY: ALTER SCHEMA public RENAME TO mattermost;

Can you please help me to migrate my mysql data into postgress? There are very important Posts

Thanks a lot in advance!
Zugang

The error seems clear here. The code is trying to rename the “public” schema to “mattermost” and failing. It seems like either you don’t have access to public schema, or it is probably named something else. Check your search_path variable.

1 Like

Hi,

yes, seems to be quite clear, but schema is there and I think I have access.

How can I debug it?
→ How can I check “search_path”?
→ How can I check access?
→ How can I get more detailed information what the problem is?

Thanks a lot!

Here are some tests I did:

psql -h localhost -d mattermost -U mmuser

→ I can log in as mmuser!

mattermost=# SELECT schema_name FROM information_schema.schemata;
schema_name

pg_toast
pg_catalog
information_schema
mattermost
(4 rows)

Database mattermost is ther and information_schema is there…but not public.

Now I deleted schema mattermost and created schema public…

…then I run again pgloader and get errror:

Database error 42P01: relation “mattermost.db_migrations” does not exist
QUERY: UPDATE mattermost.db_migrations set name=‘add_createat_to_teamembers’ where version=92;

But mySQL Shows:

mysql> SELECT * FROM mattermost.db_migrations WHERE version = 92;
±--------±----------------------------+
| Version | Name |
±--------±----------------------------+
| 92 | add_createat_to_teammembers |
±--------±----------------------------+
1 row in set (0.00 sec)

While PostgreSQL says.

SELECT * FROM information_schema.tables WHERE table_name = ‘mattermost.db_migrations’;
table_catalog | table_schema | table_name | table_type | self_referencing_column_name | reference_generation | user_defined_type_catalog | user_defined_type_schema | user_defined_type_name | is_insertable_into | is_typed | commit_action
---------------±-------------±-----------±-----------±-----------------------------±---------------------±--------------------------±-------------------------±-----------------------±-------------------±---------±--------------
(0 rows)

From pgloader Log-File I got this info:

2024-01-29T20:28:27.104000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {10083FC9F3}>
2024-01-29T20:28:27.104000Z DEBUG SET client_encoding TO ‘utf8’
2024-01-29T20:28:27.104000Z DEBUG SET maintenance_work_mem TO ‘128MB’
2024-01-29T20:28:27.108000Z DEBUG SET work_mem TO ‘12MB’
2024-01-29T20:28:27.108000Z DEBUG SET application_name TO ‘pgloader’
2024-01-29T20:28:27.116000Z NOTICE Executing SQL block for before load
2024-01-29T20:28:27.140000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {10083FC9F3}>
2024-01-29T20:28:27.140000Z DEBUG SET client_encoding TO ‘utf8’
2024-01-29T20:28:27.140000Z DEBUG SET maintenance_work_mem TO ‘128MB’
2024-01-29T20:28:27.140000Z DEBUG SET work_mem TO ‘12MB’
2024-01-29T20:28:27.140000Z DEBUG SET application_name TO ‘pgloader’
2024-01-29T20:28:27.144000Z DEBUG BEGIN
2024-01-29T20:28:27.144000Z SQL ALTER SCHEMA public RENAME TO mattermost;
2024-01-29T20:28:27.148000Z SQL DROP INDEX IF EXISTS idx_posts_message_txt;
2024-01-29T20:28:27.148000Z SQL DROP INDEX IF EXISTS idx_fileinfo_content_txt;
2024-01-29T20:28:27.152000Z LOG Migrating from #<MYSQL-CONNECTION mysql://mmuser@localhost:3306/mattermost {10083FBC63}>
2024-01-29T20:28:27.152000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {10083FC9F3}>
2024-01-29T20:28:27.184000Z ERROR mysql: Failed to connect to mysql at “localhost” (port 3306) as user “mmuser”: Condition QMYND:MYSQL-UNSUPPORTED-AUTHENTICATION was signalled.
2024-01-29T20:28:27.184000Z NOTICE Executing SQL block for after load
2024-01-29T20:28:27.203999Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {10083FC9F3}>
2024-01-29T20:28:27.203999Z DEBUG SET client_encoding TO ‘utf8’
2024-01-29T20:28:27.203999Z DEBUG SET maintenance_work_mem TO ‘128MB’
2024-01-29T20:28:27.203999Z DEBUG SET work_mem TO ‘12MB’
2024-01-29T20:28:27.203999Z DEBUG SET application_name TO ‘pgloader’
2024-01-29T20:28:27.207999Z DEBUG BEGIN
2024-01-29T20:28:27.207999Z SQL UPDATE mattermost.db_migrations set name=‘add_createat_to_teamembers’ where version=92;
2024-01-29T20:28:27.207999Z ERROR Database error 42P01: relation “mattermost.db_migrations” does not exist
QUERY: UPDATE mattermost.db_migrations set name=‘add_createat_to_teamembers’ where version=92;
2024-01-29T20:28:27.207999Z INFO Stopping monitor

With small steps Im going forward.
So first problem was loging from mysql8…pgloader was not able. This I fixed.

With this guideline:

If you are facing an error due to authentication with MySQL v8, it may be related to a known issue with the pgLoader. The fix is to set default authentication method to mysql_native_password in your MySQL configuration. To do so, add the default-authentication-plugin=mysql_native_password value to your mysql.cnf file. Also do not forget to update your user to use this authentication method.

ALTER USER ‘<mysql_user>’@‘%’ IDENTIFIED WITH mysql_native_password BY ‘<mysql_password>’;

When I now start transfering i get the following error:

2024-01-29T20:50:41.104000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {1008403503}>
2024-01-29T20:50:41.104000Z DEBUG SET client_encoding TO ‘utf8’
2024-01-29T20:50:41.104000Z DEBUG SET maintenance_work_mem TO ‘128MB’
2024-01-29T20:50:41.104000Z DEBUG SET work_mem TO ‘12MB’
2024-01-29T20:50:41.104000Z DEBUG SET application_name TO ‘pgloader’
2024-01-29T20:50:41.116000Z NOTICE Executing SQL block for before load
2024-01-29T20:50:41.140000Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {1008403503}>
2024-01-29T20:50:41.140000Z DEBUG SET client_encoding TO ‘utf8’
2024-01-29T20:50:41.140000Z DEBUG SET maintenance_work_mem TO ‘128MB’
2024-01-29T20:50:41.140000Z DEBUG SET work_mem TO ‘12MB’
2024-01-29T20:50:41.140000Z DEBUG SET application_name TO ‘pgloader’
2024-01-29T20:50:41.144000Z DEBUG BEGIN
2024-01-29T20:50:41.144000Z SQL ALTER SCHEMA public RENAME TO mattermost;
2024-01-29T20:50:41.144000Z SQL DROP INDEX IF EXISTS idx_posts_message_txt;
2024-01-29T20:50:41.144000Z SQL DROP INDEX IF EXISTS idx_fileinfo_content_txt;
2024-01-29T20:50:41.152000Z LOG Migrating from #<MYSQL-CONNECTION mysql://gruenderSql@localhost:3306/mattermost {1008402773}>
2024-01-29T20:50:41.152000Z LOG Migrating into #<PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {1008403503}>
2024-01-29T20:50:41.187999Z DEBUG CONNECTED TO #<MYSQL-CONNECTION mysql://mmuser@localhost:3306/mattermost {1008402773}>
2024-01-29T20:50:41.191999Z SQL MySQL: sending query: set net_read_timeout = 120;
2024-01-29T20:50:41.191999Z SQL MySQL: sending query: set net_write_timeout = 120;
2024-01-29T20:50:41.195999Z SQL MySQL: sending query: – params: db-name
– table-type-name
– only-tables
– only-tables
– including
– filter-list-to-where-clause incuding
– excluding
– filter-list-to-where-clause excluding
select c.table_name, t.table_comment,
c.column_name, c.column_comment,
c.data_type, c.column_type, c.column_default,
c.is_nullable, c.extra
from information_schema.columns c
join information_schema.tables t using(table_schema, table_name)
where c.table_schema = ‘mattermost’ and t.table_type = ‘BASE TABLE’

     and (table_name NOT REGEXP 'IR_' and table_name NOT REGEXP 'focalboard' and table_name != 'schema_migrations')

order by table_name, ordinal_position;
> 2024-01-29T20:50:41.219999Z ERROR mysql: 76 fell through ECASE expression.
Wanted one of (2 3 4 5 6 8 9 10 11 14 15 17 20 21 23 27 28 30 31 32 33
35 41 42 45 46 47 48 49 50 51 52 54 55 56 60 61 62 63 64
65 69 72 77 78 79 82 83 87 90 92 93 94 95 96 97 98 101
102 103 104 105 106 107 108 109 110 111 112 113 114 115
116 117 118 119 120 121 122 123 124 128 129 130 131 132
133 134 135 136 137 138 139 140 141 142 143 144 145 146
147 148 149 150 151 159 160 161 162 163 164 165 166 167
168 169 170 171 172 173 174 175 176 177 178 179 180 181
182 183 192 193 194 195 196 197 198 199 200 201 202 203
204 205 206 207 208 209 210 211 212 213 214 215 223 224
225 226 227 228 229 230 231 232 233 234 235 236 237 238
239 240 241 242 243 244 245 246 247 254).
2024-01-29T20:50:41.219999Z NOTICE Executing SQL block for after load
2024-01-29T20:50:41.239999Z DEBUG CONNECTED TO #<PGLOADER.PGSQL:PGSQL-CONNECTION pgsql://mmuser@localhost:5432/mattermost {1008403503}>
2024-01-29T20:50:41.239999Z DEBUG SET client_encoding TO ‘utf8’
2024-01-29T20:50:41.239999Z DEBUG SET maintenance_work_mem TO ‘128MB’
2024-01-29T20:50:41.239999Z DEBUG SET work_mem TO ‘12MB’
2024-01-29T20:50:41.239999Z DEBUG SET application_name TO ‘pgloader’
2024-01-29T20:50:41.243999Z DEBUG BEGIN
2024-01-29T20:50:41.243999Z SQL UPDATE mattermost.db_migrations set name=‘add_createat_to_teamembers’ where version=92;
2024-01-29T20:50:41.243999Z ERROR Database error 42P01: relation “mattermost.db_migrations” does not exist
QUERY: UPDATE mattermost.db_migrations set name=‘add_createat_to_teamembers’ where version=92;
2024-01-29T20:50:41.243999Z INFO Stopping monitor

No I really dont know how to proceed :frowning:

Hi@all,

so I have some news. I used newest pgloader soureces and then I was able to copy whole database to postgress via the follwoing pgloader config:

LOAD DATABASE
FROM mysql://mmuser:password@localhost:3306/mattermost
INTO pgsql://mmuser:password@localhost:5432/mattermost

 WITH include drop, create tables, create indexes, reset sequences

 SET maintenance_work_mem to '128MB', work_mem to '12MB', search_path to 'mattermost'

 CAST type datetime to timestamptz drop default drop not null using zero-dates-to-null,
      type date drop not null drop default using zero-dates-to-null

 BEFORE LOAD DO

$$ SET search_path TO mattermost; $$,
$$ DROP SCHEMA IF EXISTS mattermost CASCADE; $$,
$$ CREATE SCHEMA mattermost; $$;

Afterwards I removed by hand schema public and renamed Schema mattermost to public with the following commands:

DROP SCHEMA IF EXISTS public CASCADE;
ALTER SCHEMA mattermost RENAME TO public;

Then I can see old posts and users.

What do you think will it work with that approach? Or do I have to do some more manual work?
Im not able to test it until next week…