The value NIL is not of type PGLOADER.CATALOG:COLUM

[robert-leroy]

• pgloader --version

  Docker container dimitri/pgloader:latest
  3.6.7~devel
  

• did you test a fresh compile from the source tree?
  No

• did you search for other similar issues?
  Yes. It was originally failing due to heap exhaustion. Now I'm past that but I can't seem to figure out how to get around this issue.

• I've changed MySQL database collation to utf8mb4_unicode_ci

• Adjusted multiple options for pgloader.

• how can I reproduce the bug?

If you want to the full log, I have it. I can also share the MySQL Dump if it would help. MySQL v8.0.40 and PostgreSQL v17.2 are both running in containers too.

docker run --rm -it dimitri/pgloader:latest pgloader --verbose --debug  --with "batch rows = 1000" --with "batch size = 3MB" --with "batch concurrency = 1" --with "workers = 2" mysql://bob:password@10.211.55.2/metabasenew pgsql://postgres:password@10.211.55.2/metabase

• pgloader output you obtain

The last few lines are:

"7082"
"{\"query\":{\"source-table\":838,\"filter\":[\">\",[\"field-id\",6800],\"2019-12-31\"],\"aggregation\":[[\"sum\",[\"field-id\",6793]]],\"breakout\":[[\"datetime-field\",[\"field-id\",6800],\"month\"]]},\"type\":\"query\",\"database\":3,\"parameters\":[],\"async?\":true,\"middleware\":{\"add-default-userland-constraints?\":true,\"userland-query?\":true},\"constraints\":{\"max-results\":10000,\"max-results-bare-rows\":2000}}")
2024-12-26T18:49:47.980875Z DATA < #("\\x01885fcf4662ead8bb0941beb274f92538bcfd4250ea63d9e087f58b508ec798" "103"
"{\"query\":{\"source-table\":894,\"filter\":[\"and\",[\"=\",[\"field-id\",7601],\"Evacuation Traps & Screens\"],[\"time-interval\",[\"field-id\",7605],-2,\"year\",{\"include-current\":true}]],\"aggregation\":[[\"sum\",[\"field-id\",7599]]],\"breakout\":[[\"datetime-field\",[\"field-id\",7605],\"month\"],[\"field-id\",7601],[\"field-id\",9623],[\"field-id\",7614],[\"field-id\",7604]]},\"type\":\"query\",\"database\":3,\"parameters\":[],\"async?\":true,\"middleware\":{\"userland-query?\":true,\"skip-results-metadata?\":true}}")
2024-12-26T18:49:47.980875Z DATA < #("\\x018891890b92f3f274d06650fba334b7f276256e659f249ad4bc376da8ce0239" "139"
"{\"query\":{\"source-table\":895,\"filter\":[\"and\",[\"time-interval\",[\"field-id\",7627],-1,\"year\",{\"include-current\":true}],[\"=\",[\"field-id\",7620],\"Impression\"],[\"=\",[\"field-id\",7626],\"Silicone\"]],\"aggregation\":[[\"sum\",[\"field-id\",7639]],[\"sum\",[\"field-id\",7625]],[\"metric\",1114]],\"breakout\":[[\"datetime-field\",[\"field-id\",7627],\"quarter\"]]},\"type\":\"query\",\"database\":3,\"parameters\":[],\"async?\":true,\"middleware\":{\"add-default-userland-constraints?\":true,\"userland-query?\":true},\"constraints\":{\"max-results\":10000,\"max-results-bare-rows\":2000}}")
2024-12-26T18:49:47.980875Z DATA < #("\\x0188dae6095c3077b0cf63d98daa1a420f61d26ea5c8f9ea4e9ba5480c592c52"
"6084"
"{\"database\":13371338,\"type\":\"query\",\"query\":{\"source-table\":19295,\"aggregation\":[[\"sum\",[\"field\",28096,{\"base-type\":\"type/Number\"}]]],\"breakout\":[[\"field\",28111,{\"base-type\":\"type/Text\"}],[\"expression\",\"pivot-grouping\"]],\"filter\":[\"and\",[\"time-interval\",[\"field\",28107,{\"base-type\":\"type/Date\"}],-3,\"month\",{\"include-current\":false}],[\"=\",[\"field\",28094,{\"base-type\":\"type/Text\"}],\"Restoratives\"]],\"expressions\":{\"pivot-grouping\":[\"abs\",11]}},\"pivot-rows\":[2,3],\"pivot-cols\":[0,1],\"async?\":true,\"constraints\":{\"max-results\":10000,\"max-results-bare-rows\":2000}}")
2024-12-26T18:49:47.980875Z DATA < #("\\x01895d195bce62425c9064a55e45f065d1f0e92d48fd113411ab8396d930b9b5"
"8151"
"{\"constraints\":{\"max-results\":10000,\"max-results-bare-rows\":2000},\"type\":\"query\",\"middleware\":{\"userland-query?\":true},\"database\":3,\"query\":{\"source-table\":839,\"aggregation\":[[\"metric\",308],[\"metric\",310]],\"breakout\":[[\"datetime-field\",[\"field-id\",6822],\"month\"]]},\"parameters\":[{\"type\":\"category\",\"target\":[\"dimension\",[\"field-id\",6818]],\"value\":[\"Fluoride\"]},{\"type\":\"category\",\"target\":[\"dimension\",[\"field-id\",6817]],\"value\":[\"DENTSPLY CAULK\"]}],\"async?\":true,\"cache-ttl\":null}")
2024-12-26T18:49:47.980875Z INFO Stopping monitor

What I am doing here?

The value
  NIL
is not of type
  PGLOADER.CATALOG:COLUMN