Merge pull request #52 from skysqlinc/DEV-894-Improve-migration-proc

Dev 894 improve migration proc

docs/Data loading, Migration/(WIP)Migrate-your-database-to-SkySQL.md

@@ -0,0 +1,69 @@
+# (WIP) Migrate your database to SkySQL
+
+SkySQL provides a range of options to suit different migration scenarios.
+<ul>
+   <li> Databases can be migrated to SkySQL from many different database platforms, including Oracle, MySQL, PostgreSQL, Microsoft SQL Server, IBM DB2, and more. </li>
+   <li> SkySQL supports migraion from both on-premise and cloud-based infrastructure and provides a range of options to suit different migration scenarios. </li>
+</ul>
+
+Below are the most common scenarios for database migration to SkySQL.
+
+---
+## Prerequisites
+
+1. An active SkySQL account. 
+2. An existing source database with the IP added to your SkySQL allowlist.
+
+<details>
+<summary>Considerations</summary>
+<br>
+
+Ensure that your SkySQL servce deploymned configuration is compatible with your existing source database one, including:
+<ul>
+   <li><b>Deployment region</b> - Ensure that the SkySQL deployment region is the same as the source database region.</li>
+   <li><b>Topology</b> - Enterprise Server Single node or with Replica(s)</li>
+   <li> <b>Server version</b> - Ensure that the SkySQL server version is compatible with the source database version. </li>
+   <li><b>Instance size</b> - Ensure that the SkySQL instance is compatible with the source database instance type and size</li>
+   <li><b>Storage</b> - Ensure that the SkySQL storage type and size is compatible with the source database</li>
+</br>
+</details>
+---
+
+## SkyDBA Assisted Migration 
+
+ - Existing customers can submit a [support case](https://support.skysql.com) to request assistance with a migration.
+ - New customers can [contact us](mailto:support@skysql.com) to begin the migration planning process.
+
+Our [SkyDBA team](https://skysqlinc.github.io/skysql-docs/FractionalDBA/) can help design a migration plan to suit your needs.
+
+<details>
+<summary>SkyDBA Assisted Migration Approach</summary>
+<br>
+ We use a multi-step process to assist customers with migrations:
+<ul>
+   <li><b>Assessment</b> of application requirements, inventory, and identified challenges</li>
+   <li><b>Schema Migration</b> including tables, constraints, indexes, and views</li>
+   <li><b>Application Code Migration</b> by porting and testing SQL and application code</li>
+   <li><b>Data Migration and Replication</b> with import of data, with conversion to the new schema, and ongoing inbound replication of new data</li>
+   <li><b>Quality Assurance</b> to assess data validity, data integrity, performance, accuracy of query results, stored code, and running code such as client applications, APIs, and batch jobs</li>
+   <li><b>Cutover</b> including final database preparation, fallback planning, switchover, and decommissioning of old databases</li>
+</br>
+</details>
+---
+
+## Self-Service Migration to SkySQL
+
+SkySQL provides two diffeent options for self-service migration 
+
+### Option 1 Migrate using the SkySQL REST API
+SkySQL Managed Migration is a REST-based service that handles the migration process, including data migration, schema migration, and user migration. It provides a follow us steps to set up a live replication of your database to SkySQL and various insights to monitor the migration process.
+
+- [Sky SQL Managed Migration Tutorial](./SkySQL-managed-migration.md)
+
+### Option 2: Custom Migration
+
+For most small, mid-size and large migrations SkySQL Managed Migration is the quickest and safest option. However, for large migrations or migrations with specific requirements, you and your team may require more flexibility and control over the migration process. In these cases, you can opt for a custom migration where you and your team can design a custoemmigration plan considering the steps suggested below.
+
+- [Sky SQL Custom Migration Tutorial](./Migrating your existing Production DB.md)
+
+

docs/Data loading, Migration/SkySQL-custom-migration.md

@@ -0,0 +1,114 @@
+
+//TODO Add a  section on physical backup and restore 
+
+## Live Replication for Minimal Downtime
+
+To minimize downtime during migration, set up live binary-loggd based replication from your source database to the SkySQL database. 
+Click [here](<./Replicating data from external DB.md>) for a detailed walk through of the steps involved. 
+
+
+
+
+
+
+
+Follow these steps:
+# Replicating data from an External DB
+
+
+
+
+
+2. **Create the Users and Grants Separately**: To avoid conflicts with the existing SkySQL users, use `SELECT CONCAT` on your source database to create users and grants in separate files. Note that you may need to create the schema and table grants separately as well.
+
+    ```bash
+    mysql -u [username] -p -h [hostname] --silent --skip-column-names -e "SELECT CONCAT('CREATE USER \'', user, '\'@\'', host, '\' IDENTIFIED BY PASSWORD \'', authentication_string, '\';') FROM mysql.user;" > users.sql
+
+    mysql -h [hostname] -u [username] -p --silent --skip-column-names -e "SELECT CONCAT('GRANT ', privilege_type, ' ON ', table_schema, '.* TO \'', grantee, '\';') FROM information_schema.schema_privileges;" > grants.sql
+
+    mysql -h [hostname] -u [username] -p --silent --skip-column-names -e "SELECT CONCAT('GRANT ', privilege_type, ' ON ', table_schema, '.', table_name, ' TO \'', grantee, '\';') FROM information_schema.table_privileges;" >> grants.sql
+    ```
+
+3. **Import the Dumps into SkySQL**: Import the logical dumps (SQL files) into your SkySQL database, ensuring to load the user and grant dumps after the main dump.
+
+    ```bash
+    mariadb -u [SkySQL username] -p -h [SkySQL hostname] --port 3306 --ssl-verify-server-cert < dump.sql
+    mariadb -u [SkySQL username] -p -h [SkySQL hostname] --port 3306 --ssl-verify-server-cert < users.sql
+    mariadb -u [SkySQL username] -p -h [SkySQL hostname] --port 3306 --ssl-verify-server-cert < grants.sql
+    ```
+
+If you encounter an error while importing your users, you may need to uninstall the `simple_password_check` plugin on your SkySQL instance.
+
+    ```sql
+    UNINSTALL PLUGIN simple_password_check;
+    ```
+
+4. **Start Replication**: Turn on replication using SkySQL stored procedures. There are procedures allowing you to set and start replication. See our [documentation](<../Reference Guide/Sky Stored Procedures.md>) for details. The `dump.sql` file you created in step 1 will contain the GTID and binary log information needed for the `change_external_primary` procedure.
+
+    ```sql
+    CALL sky.change_external_primary(
+   host VARCHAR(255),
+   port INT,
+   logfile TEXT,
+   logpos LONG ,
+   use_ssl_encryption BOOLEAN
+    );
+    
+    CALL sky.replication_grants();
+    CALL sky.start_replication();
+    ```
+
+### Performance Optimization During Migration
+
+- **Disable Foreign Key Checks**: Temporarily disable foreign key checks during import to speed up the process.
+
+    ```sql
+    SET foreign_key_checks = 0;
+    ```
+
+- **Disable Binary Logging**: If binary logging is not required during the import process, and you are using a standalone instance, it can potentially be disabled to improve performance. SkyDBA Services can assist with this as part of a detailed migration plan.
+
+### Data Integrity and Validation
+
+- **Consistency Checks**: Perform consistency checks on the source database before migration. Use a [supported SQL client](<../../Connecting to Sky DBs/>) to connect to your SkySQL instance and run the following.
+
+    ```sql
+    CHECK TABLE [table_name] FOR UPGRADE;
+    ```
+
+- **Post-Import Validation**: Validate the data integrity and consistency after the import.
+
+    ```sql
+    CHECKSUM TABLE [table_name];
+    ```
+
+### Advanced Migration Techniques
+
+- **Adjust Buffer Sizes**: Temporarily increase buffer sizes to optimize the import performance. This can be done via the Configuration Manager in the portal.
+
+    ```sql
+    innodb_buffer_pool_size = 2G
+    innodb_log_file_size = 512M
+    ```
+
+- **Parallel Dump and Import**: Use tools that support parallel processing for dumping and importing data.
+
+    ```bash
+    mysqlpump -u [username] -p --default-parallelism=4 --add-drop-database \
+        --databases [database_name] > dump.sql
+    ```
+
+- **Incremental Backups**: For large datasets, incremental backups can be used to minimize the amount of data to be transferred. SkyDBA Services can assist you with setting these up as part of a custom migration plan.
+
+### Monitoring and Logging
+
+- **Enable Detailed Logging**: Enable detailed logging while testing the migration process to monitor and troubleshoot effectively. The slow_log can be enabled in the SkySQL configuration manager.
+
+- **Resource Monitoring**: Use monitoring tools to track resource usage (CPU, memory, I/O) during the migration to ensure system stability. See our [monitoring documentation](<../Portal features/Service Monitoring Panels.md>) for details.
+
+### Additional Resources
+
+- [Backup with mariadb-dump](https://mariadb.com/kb/en/mariadb-dump/)
+- [MariaDB Backup Documentation](https://mariadb.com/kb/en/mariadb-backup-overview/)
+- [Advanced Backup Techniques](https://mariadb.com/kb/en/backup-and-restore-overview/)
+- [Migrate RDS MySQL to SkySQL using the AWS Data Migration Service (DMS)](<./migrate-rds-mysql-to-skysql-using-amazon-data-migration-service_whitepaper_1109.pdf>)

docs/Data loading, Migration/SkySQL-managed-migration.md

@@ -0,0 +1,12 @@
+//TODO 
+//Steps to follow
+0. Prerequisites
+1. Dump using logical or physical backup using the steps mentioned in the [Backup and Restore](../Backup and Restore/README.md) restore from your own bucket section.
+2. Upload the dump to S3/GCS bucket under your control
+3. Call the migration API - ref - 
+[Sky SQL Managed Migration Tutorial](../Backup and Restore/Restore From Your Own Bucket.md) 
+
+4. To minimize downtime during migration, set up live binary-loggd based replication from your source database to the SkySQL database. 
+Click [here](<./Replicating data from external DB.md>) for a detailed walk through of the steps involved. 
+
+5 Add a section with migration insights

mkdocs.yml

@@ -107,6 +107,7 @@ nav:
       - 'Data loading, Migration/Install Mariadb-dump.md'
       - 'Data loading, Migration/Install-mariadb-import.md'
       - 'Data loading, Migration/Migrating your existing Production DB.md'
+      - 'Data loading, Migration/(WIP)Migrate-your-database-to-SkySQL.md'
       - 'Data loading, Migration/Replicating data from external DB.md'
   - 'Data Offloading' :
       - 'Data offloading/README.md'