Merge pull request #276 from lukaszbudnik/updating-performance-tests

Updating performance benchmark and its documentation
lukaszbudnik · Aug 31, 2021 · eec6f1a · eec6f1a
2 parents 91d7307 + e6f080f
commit eec6f1a
Show file tree

Hide file tree

Showing 8 changed files with 221 additions and 40 deletions.
diff --git a/PERFORMANCE.md b/PERFORMANCE.md
@@ -0,0 +1,94 @@
+# migrator performance
+
+Back in 2018 I wrote a small benchmark to compare 3 DB migrations frameworks:
+
+- migrator
+- proprietary Ruby framework - used internally at my company
+- flyway - leading market feature rich DB migration framework: https://flywaydb.org
+
+In 2021 I decided to refresh the results (the original results for historical purposes are at the bottom).
+
+You can play around with performance benchmark yourself. See `test/performance/test.sh` for migrator test and `test/performance/flywaydb-test.sh` for flyway.
+
+# 2021 results
+
+In 2021 I compared migrator (`v2021.1.0`) and flyway (`Flyway Teams Edition 7.14.0 by Redgate`).
+
+_Note: I didn't use Ruby framework as it became deprecated. Also, the previous Ruby's results were dramatic and not worth investing time in re-doing them._
+
+Compared to 2018 the tests are single tenant with 10k migrations. This is because somewhere between 2018 and 2021 [multiple schemas](https://flywaydb.org/documentation/learnmore/faq.html#multiple-schemas) stopped working in flyway.
+
+| framework | number of migrations | time (s) | memory consumption (MB) |
+| --------- | -------------------- | -------- | ----------------------- |
+| migrator  | 10000                | 5        | 23                      |
+| flyway    | 10000                | 49       | 265                     |
+
+migrator is still orders of magnitude better than flyway.
+
+Because multiple schemas stopped working in flyway there wasn't too much sense in doing more comparison benchmarks.
+
+Instead, I ran additional multi-tenant/multiple-schema migrator benchmarks.
+
+# migrator performance showcase
+
+You can use `test/performance/test.sh` to run any simulation you want. You can also use it to simulate adding new migrations (so called append mode) - scroll to the bottom of that script to see a comment showing you how to do this.
+
+I prepared 2 multi-tenant simulations:
+
+1. 1000 tenants, 10 versions, 20 SQL files in each version - 20k migrations to apply in each version
+2. 500 tenants, 5 versions, 100 SQL files in each version - 50k migrations to apply in each version
+
+## 1000 tenants
+
+Execution time is growing slightly with every new version. The memory consumption grows proportionally to how many migrations are in the database. This is because migrator fetches all migrations from database to compute which migrations were already applied and which are to be applied.
+
+| version | number of migrations (before - after) | time (s) | memory consumption (MB) |
+| ------- | ------------------------------------- | -------- | ----------------------- |
+| 1       | 0 - 21001                             | 57       | 66                      |
+| 2       | 21001 - 41001                         | 58       | 86                      |
+| 3       | 41001 - 61001                         | 56       | 101                     |
+| 4       | 61001 - 81001                         | 62       | 165                     |
+| 5       | 81001 - 101001                        | 62       | 175                     |
+| 6       | 101001 - 121001                       | 59       | 242                     |
+| 7       | 121001 - 141001                       | 71       | 280                     |
+| 8       | 141001 - 161001                       | 68       | 300                     |
+| 9       | 161001 - 181001                       | 70       | 324                     |
+| 10      | 181001 - 201001                       | 69       | 380                     |
+
+## 500 tenants
+
+Similarly to 1000 tenants, 500 tenants simulation execution time is growing slightly with every new version. The memory consumption grows proportionally to how many migrations are in the database.
+
+Based on both simulations we can see that migrator under any load is stable and behaves very predictable:
+
+- applied ~300 migrations a second (creating tables, inserting multiple rows, database running on the same machine = my MacBook)
+- consumed ~2.5MB memory for every 1k migrations
+
+| version | number of migrations (before - after) | time (s) | memory consumption (MB) |
+| ------- | ------------------------------------- | -------- | ----------------------- |
+| 1       | 0 - 50501                             | 167      | 126                     |
+| 2       | 50501 - 100501                        | 170      | 140                     |
+| 3       | 100501 - 150501                       | 167      | 218                     |
+| 4       | 150501 - 200501                       | 181      | 292                     |
+| 5       | 200501 - 250501                       | 178      | 396                     |
+
+# 2018
+
+Keeping 2018 results for historical purposes.
+
+_Note: For all 3 DB frameworks I used multiple schemas benchmark. Unfortunately, back in 2018, I didn't commit flyway tests and its impossible now to recreate how multiple schemas were actually set up._
+
+Execution times were the following:
+
+| # Tenants | # Existing Migrations | # Migrations to apply | migrator | Ruby | Flyway |
+| --------- | --------------------- | --------------------- | -------- | ---- | ------ |
+| 10        | 0                     | 10001                 | 154s     | 670s | 2360s  |
+| 10        | 10001                 | 20                    | 2s       | 455s | 340s   |
+
+migrator was the undisputed winner.
+
+The Ruby framework had the undesired functionality of making a DB call each time to check if given migration was already applied. migrator fetched all applied migrations at once and compared them in memory. That was the primary reason why migrator was so much better in the second test.
+
+flyway results were... very surprising. I was so shocked that I had to re-run flyway as well as all other tests. Yes, flyway was 15 times slower than migrator in the first test. In the second test flyway was faster than Ruby. Still a couple orders of magnitude slower than migrator.
+
+The other thing to consider is the fact that migrator is written in go which is known to be much faster than Ruby and Java.
diff --git a/README.md b/README.md
@@ -1,6 +1,6 @@
 # migrator ![Build and Test](https://github.com/lukaszbudnik/migrator/workflows/Build%20and%20Test/badge.svg) ![Docker](https://github.com/lukaszbudnik/migrator/workflows/Docker%20Image%20CI/badge.svg) [![Go Report Card](https://goreportcard.com/badge/github.com/lukaszbudnik/migrator)](https://goreportcard.com/report/github.com/lukaszbudnik/migrator) [![codecov](https://codecov.io/gh/lukaszbudnik/migrator/branch/master/graph/badge.svg)](https://codecov.io/gh/lukaszbudnik/migrator)
 
-Super fast and lightweight DB migration tool written in go. migrator outperforms other DB migration/evolution frameworks by a few orders of magnitude.
+Super fast and lightweight DB migration tool written in go. migrator outperforms other DB migration/evolution frameworks by a few orders of magnitude when comparing both execution time and memory consumption.
 
 migrator manages and versions all the DB changes for you and completely eliminates manual and error-prone administrative tasks. migrator versions can be used for auditing and compliance purposes. migrator not only supports single schemas, but also comes with a multi-schema support (ideal for multi-schema multi-tenant SaaS products).
 
@@ -18,7 +18,11 @@ migrator support the following multi-tenant databases:
 - MySQL 5.6+ (and all its flavours)
 - Microsoft SQL Server 2017+
 
-The official docker image is available on docker hub at [lukasz/migrator](https://hub.docker.com/r/lukasz/migrator) or on the alternative mirror at [ghcr.io/lukaszbudnik/migrator](https://github.com/lukaszbudnik/migrator/pkgs/container/migrator).
+The official docker image is available on:
+
+- docker hub at: [lukasz/migrator](https://hub.docker.com/r/lukasz/migrator)
+- alternative mirror at: [ghcr.io/lukaszbudnik/migrator](https://github.com/lukaszbudnik/migrator/pkgs/container/migrator)
+
 It is ultra lightweight and has a size of 30MB. Ideal for micro-services deployments!
 
 # API
@@ -648,27 +652,7 @@ You can find it in [tutorials/oauth2-proxy-oidc-haproxy](tutorials/oauth2-proxy-
 
 # Performance
 
-As a benchmarks I used 2 migrations frameworks:
-
-- proprietary Ruby framework - used at my company
-- flyway - leading market feature rich DB migration framework: https://flywaydb.org
-
-There is a performance test generator shipped with migrator (`test/performance/generate-test-migrations.sh`). In order to generate flyway-compatible migrations you need to pass `-f` param (see script for details).
-
-Execution times are following:
-
-| # Tenants | # Existing Migrations | # Migrations to apply | migrator | Ruby | Flyway |
-| --------- | --------------------- | --------------------- | -------- | ---- | ------ |
-| 10        | 0                     | 10001                 | 154s     | 670s | 2360s  |
-| 10        | 10001                 | 20                    | 2s       | 455s | 340s   |
-
-migrator is the undisputed winner.
-
-The Ruby framework has an undesired functionality of making a DB call each time to check if given migration was already applied. migrator fetches all applied migrations at once and compares them in memory. This is the primary reason why migrator is so much better in the second test.
-
-flyway results are... very surprising. I was so shocked that I had to re-run flyway as well as all other tests. Yes, flyway is 15 times slower than migrator in the first test. In the second test flyway was faster than Ruby. Still a couple orders of magnitude slower than migrator.
-
-The other thing to consider is the fact that migrator is written in go which is known to be much faster than Ruby and Java.
+Performance benchmarks were moved to a dedicated [PERFORMANCE.md](PERFORMANCE.md) document.
 
 # Change log
 

diff --git a/test/performance/create-test-tenants.sh b/test/performance/create-test-tenants.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# not the most fancy script in the world
+
+if [[ $# -eq 0 ]]; then
+  echo "This script expects a number of test tenants to create as its argument"
+  exit
+fi
+
+i=1
+let end=$1+1
+
+while [[ $i -lt $end ]]; do
+  if [[ $i%10 -eq 0 ]]; then
+    echo "creating new tenant $i"
+  fi
+  name="tenant_${RANDOM}_${RANDOM}"
+  psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "create schema $name; insert into migrator.migrator_tenants (name) values ('$name');"
+  let i+=1
+done
diff --git a/test/performance/flyway.conf b/test/performance/flyway.conf
@@ -0,0 +1,6 @@
+flyway.driver=org.postgresql.Driver
+flyway.url=jdbc:postgresql://127.0.0.1:5432/migrator?ssl=false
+flyway.user=postgres
+flyway.password=supersecret
+flyway.locations=test/performance/migrations/tenants
+flyway.schemas=set dynamically by test script
diff --git a/test/performance/flywaydb-test.sh b/test/performance/flywaydb-test.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+NO_OF_MIGRATIONS=1000
+NO_OF_TENANTS=10
+
+export PGPASSWORD=supersecret
+# remove all existing tenants
+psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "delete from migrator.migrator_tenants"
+# create test tenants (connects to psql and creates them)
+./test/performance/create-test-tenants.sh $NO_OF_TENANTS
+
+./test/performance/generate-test-migrations.sh -f -n $NO_OF_MIGRATIONS
+
+# flyway doesn't support both single and multi-tenant migrations, delete public ones
+rm -rf ./test/performance/migrations/public
+
+# remove existing flyway.schemas config property
+gsed -i '/flyway.schemas/d' ./test/performance/flyway.conf
+
+output=$(psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "select string_agg(name, ',') from migrator.migrator_tenants")
+
+echo "flyway.schemas=$output" >> ./test/performance/flyway.conf
+
+start=`date +%s`
+flyway -configFiles=./test/performance/flyway.conf baseline migrate > /dev/null
+end=`date +%s`
+
+echo "Test took $((end-start)) seconds"
+
+rm -rf test/performance/migrations/
+
+# append test
+# 1. comment out above rm command
+# 2. RUN TEST
+# 3. generate new migrations:
+# ./test/performance/generate-test-migrations.sh -a -f -n $NO_OF_MIGRATIONS
+# 4. execute flyway migrate command, measure start and end times:
+# start=`date +%s` && flyway -configFiles=./test/performance/flyway.conf migrate > /dev/null && end=`date +%s`
diff --git a/test/performance/generate-test-migrations.sh b/test/performance/generate-test-migrations.sh
@@ -44,7 +44,7 @@ function generate_table {
   i=$1
   counter=$2
   timestamp=$(date +'%Y%m%d%H%M%S%N')
-  file="migrations/tenants/V${timestamp}.1__${i}.sql"
+  file="migrations/tenants/V${timestamp}.${counter}_${i}.sql"
   cat > $file <<EOL
 create table ${tenantprefixplaceholder}table_${counter} (
 a int,
@@ -59,7 +59,7 @@ function generate_public_table {
   timestamp=$(date +'%Y%m%d%H%M%S%N')
   file="migrations/public/V${timestamp}.sql"
   cat > $file <<EOL
-create table public.table_for_inserts (
+create table if not exists public.table_for_inserts (
 a int,
 b float,
 c varchar(100)
@@ -72,7 +72,7 @@ function generate_alter_drop_inserts {
   i=$1
   counter=$2
   timestamp=$(date +'%Y%m%d%H%M%S%N')
-  file="migrations/tenants/V${timestamp}.1__${i}.sql"
+  file="migrations/tenants/V${timestamp}.${counter}_${i}.sql"
   # if [[ $i%2 -eq 1 ]]; then
   #   echo "alter table ${tenantprefixplaceholder}table_${counter} add column d int, add column e varchar, add column f int;" > $file
   # else
@@ -98,14 +98,18 @@ if [[ $append -eq 0 ]]; then
   i=0
   counter=0
 else
-  i=$(ls -t migrations/tenants | head -1 | cut -d '_' -f 3 | cut -d '.' -f 1)
+  i=$(ls -t migrations/tenants | head -1 | cut -d '_' -f 2 | cut -d '.' -f 1)
   counter=$((i/10+1))
   i=$((i+1))
 fi
 
 end=$((i+no_of_migrations))
 
 echo "About to generate $no_of_migrations migrations"
+echo "is append? $append"
+echo "is without tenant prefix? $remove_tenant_placeholder"
+echo "counter = $counter"
+echo "i = $i"
 
 while [[ $i -lt $end ]]; do
   if [[ $i%10 -eq 0 ]]; then

diff --git a/test/performance/migrator-performance.yaml b/test/performance/migrator-performance.yaml
@@ -6,3 +6,4 @@ singleMigrations:
 tenantMigrations:
   - tenants
 schemaPlaceHolder: ":tenant"
+port: 8888
diff --git a/test/performance/test.sh b/test/performance/test.sh
@@ -4,17 +4,31 @@
 # it uses PostgreSQL and requires psql tool to be installed
 
 NO_OF_MIGRATIONS=100
+NO_OF_TENANTS=500
 
-go build
+# test from scratch
+EXISTING_TABLES=0
+EXISTING_INSERTS=0
+# in append test
+# EXISTING_TABLES=10
+# EXISTING_INSERTS=100
 
-# generate test migrtations
-./test/performance/generate-test-migrations.sh -n $NO_OF_MIGRATIONS
+go build
 
 export PGPASSWORD=supersecret
+# remove all existing versions and migrations
 psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "delete from migrator.migrator_versions"
+# remove all existing tenants
+psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "delete from migrator.migrator_tenants"
+# create test tenants (connects to psql and creates them)
+./test/performance/create-test-tenants.sh $NO_OF_TENANTS
 
-./migrator -configFile=test/performance/migrator-performance.yaml  &
-PID=$!
+# generate test migrtations
+./test/performance/generate-test-migrations.sh -n $NO_OF_MIGRATIONS
+# generate test migrtations in append test
+# ./test/performance/generate-test-migrations.sh -n $NO_OF_MIGRATIONS -a
+
+./migrator -configFile=test/performance/migrator-performance.yaml &> /dev/null &
 sleep 5
 
 COMMIT_SHA="performance-tests"
@@ -44,24 +58,25 @@ cat <<EOF | tr -d "\n" > create_version.txt
   }
 }
 EOF
-# and now execute the above query
 start=`date +%s`
-curl -d @create_version.txt http://localhost:8888/v2/service
+curl -d @create_version.txt http://localhost:8888/v2/service | jq -r '.data.createVersion.summary'
 end=`date +%s`
 
+echo "Done, checking if all migrations were applied correctly..."
+
 output=$(psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "select name from migrator.migrator_tenants")
 
 IFS=$'\n'; tenants=($output); unset IFS;
 
 for tenant in "${tenants[@]}"; do
     count=$(psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "select count(distinct col) from $tenant.table_for_inserts")
-    tables_count=$(psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "select count(*) from information_schema.tables where table_schema = '$tenant' and table_name like 'table_%';")
+    tables_count=$(psql -U postgres -h 127.0.0.1 -p 5432 -d migrator -tAq -c "select count(*) from information_schema.tables where table_schema = '$tenant' and table_name like 'table_%'")
 
-    if [[ $count -ne $NO_OF_MIGRATIONS+1 ]]; then
-      echo "[migrations inserts] error for $tenant, got $count, expected: $((NO_OF_MIGRATIONS+1))"
+    if [[ $count -ne $NO_OF_MIGRATIONS+$EXISTING_INSERTS+1 ]]; then
+      echo "[migrations inserts] error for $tenant, got $count, expected: $((NO_OF_MIGRATIONS+$EXISTING_INSERTS+1))"
     fi
-    if [[ $tables_count -ne $NO_OF_MIGRATIONS/10+1 ]]; then
-      echo "[migrations tables] error for $tenant, got $tables_count, expected: $((NO_OF_MIGRATIONS/10+1))"
+    if [[ $tables_count -ne $NO_OF_MIGRATIONS/10+$EXISTING_TABLES+1 ]]; then
+      echo "[migrations tables] error for $tenant, got $tables_count, expected: $((NO_OF_MIGRATIONS/10+$EXISTING_TABLES+1))"
     fi
 
 done
@@ -71,4 +86,23 @@ echo "Test took $((end-start)) seconds"
 # remove generated migrations and test request and kill migrator
 rm -rf test/performance/migrations
 rm create_version.txt
-kill $PID
+sleep 5
+killall migrator
+
+
+# prepare for append test
+# 1. comment out lines:
+# 87
+#
+# 2. RUN TEST
+#
+# 3. comment out lines:
+# 10, 11
+# 20, 22, 24
+# 27
+#
+# 4. uncomment lines:
+# 13, 14
+# 29
+#
+# 5. RUN TEST