pgloader v3.5.1

This release is mainly about lots of bug fixes thanks to user reports in GitHub issues, and also contains an heavily optimised code for preparing the COPY buffers. This optimisation comes with the realisation that when using pgloader to migrate from a source database to PostgreSQL, it's best to fail early. As a result, we don't keep batches in memory at all (by default) in those cases, allowing clear benefits of CPU and memory usage in pgloader.

As usual, for a full list of changes you may have a look at the git history here: v3.4.1...v3.5.1.

Guessing CSV formats

In this version of pgloader, when you load a CSV file into a target table, pgloader considers that if you don't specify the CSV format then it's expected to match with the target table definition. Which means that pgloader can guess the separator and quoting rules used in your source file!

Casting Rules

It's now possible in User Defined Casting Rules to specify new guards and actions, as seen in the documentation.

Other improvements

pgloader now supports loading data into PostgreSQL Foreign Tables and Partitioned Tables.

Lots of bug where fixed in the PostgreSQL support, in the SQLite support, in the MySQL support and in the MS SQL support.

SQLite improvements

The multiple ways SQLite can represent primary keys and indexes can be confusing, and pgloader got smarter about that. Pgloader now deal with more default values for SQLite too.

MySQL improvements

When loading a huge MySQL table it's now possible to have pgloader work with more than one reader in parallel, each reader querying a range of primary key values from the source database. This technique might improve reading times in some cases.

MySQL connection string can now use the useSSL parameter, or a sslmode parameter like when using PostgreSQL URIs.

In the previous release, pgloader changed to target in PostgreSQL a schema created with the name of the MySQL database. In this release, pgloader also automatically adds that schema to the database search_path.

Load file templates

In pgloader v3.5.1 it's now possible to use the https://mustache.github.io templating engine. Values can be given via the new command line parameter --context or from the process environment. This allows using the same load file with different source files, for instance, and has been a long asked-for feature for pgloader.