1333 - add new AddressDetail table for TxOut

[Cmdv]

Description

Adding a new AddressDetail table for TxOut this functionality lives behind a config flag use_address_table.

This fixes #1333 + #1697

Also restructured the Db directory as it wasn't in the best state

.
├── Db
│   ├── Error.hs
│   ├── Git
│   │   ├── RevFromGit.hs
│   │   └── Version.hs
│   ├── Migration
│   │   ├── Haskell.hs
│   │   └── Version.hs
│   ├── Migration.hs
│   ├── Operations
│   │   ├── Core
│   │   │   ├── AlterTable.hs
│   │   │   ├── Delete.hs
│   │   │   ├── Insert.hs
│   │   │   ├── MinId.hs
│   │   │   ├── Query.hs
│   │   │   └── QueryHelper.hs
│   │   └── Variant
│   │       ├── ConsumedTxOut.hs
│   │       ├── JsonbInSchemaQueries.hs
│   │       ├── Multiplex.hs
│   │       └── TxOut.hs
│   ├── PGConfig.hs
│   ├── Run.hs
│   ├── Schema
│   │   ├── Core
│   │   │   ├── Schema.hs
│   │   │   └── TxOut.hs
│   │   ├── Orphans.hs
│   │   ├── Types.hs
│   │   └── Variant
│   │       ├── ConsumedTxOut.hs
│   │       └── TxOut.hs
│   ├── Types.hs
│   └── Version
│       ├── V13_0
│       │   ├── Query.hs
│       │   └── Schema.hs
│       └── V13_0.hs
└── Db.hs

Checklist

• Commit sequence broadly makes sense
• Commits have useful messages
• New tests are added if needed and existing tests are updated
• Any changes are noted in the changelog
• Code is formatted with fourmolu on version 0.10.1.0 (which can be run with scripts/fourmolize.sh)
• Self-reviewed the diff

Migrations

• The pr causes a breaking change of type a,b or c
• If there is a breaking change, the pr includes a database migration and/or a fix process for old values, so that upgrade is possible
• Resyncing and running the migrations provided will result in the same database semantically

If there is a breaking change, especially a big one, please add a justification here. Please elaborate
more what the migration achieves, what it cannot achieve or why a migration is not possible.

[kderme]

The FOREIGN KEY ma_tx_out_tx_out_id_fkey in function createConsumedTxOut is only necessary if prune is enabled.
Also it needs to run only once, unless the "IF NOT EXIST" clause can be used. The wPruneTxOutPreviouslySet in runExtraMigrations can be used to make sure this is run only once.

[kderme]

Some indexes are still required in address table

[ArturWieczorek]

I had set "prune" level and then change it to "bootstrap:

    "tx_out": {
      "value": "prune",
      "use_address_table": true,
    },

At the end of sync process on sanchonet I ran into issue:

[db-sync-node:Info:81] [2024-09-23 11:15:55.57 UTC] Starting epoch 466
[db-sync-node:Info:81] [2024-09-23 11:15:55.57 UTC] Insert Conway Block: epoch 466, slot 40262417, block 1990094, hash d1e4b009dbff7b18883e09744c1f6eeaf2e7b67c1b9f12f0d45cb75b4b6deef5
[db-sync-node:Info:90] [2024-09-23 11:15:55.73 UTC] Asynchronously wrote a ledger snapshot to ledger-state/sancho/40262388-9a208b3651-465.lstate in 0.164323499s.
[db-sync-node:Info:81] [2024-09-23 11:15:56.04 UTC] Inserted 97 EpochStake for EpochNo 467
[db-sync-node:Info:81] [2024-09-23 11:15:56.75 UTC] Deleted 0 tx_out
[db-sync-node:Info:81] [2024-09-23 11:15:56.92 UTC] Voting Anchor Offchain metadata fetch: 0 results, 3 fetch errors
[db-sync-node:Info:81] [2024-09-23 11:15:56.92 UTC] Starting UTxO bootstrap migration
[db-sync-node:Warning:81] [2024-09-23 11:15:56.97 UTC] Found and deleted 61224 tx_out.
[db-sync-node:Info:81] [2024-09-23 11:15:56.97 UTC] Inserting 87766 tx_out as pages of 100000
[db-sync-node:Info:81] [2024-09-23 11:15:56.97 UTC] Bootstrap in progress 0.0%
[db-sync-node:Info:81] [2024-09-23 11:16:13.68 UTC] UTxO bootstrap migration done
[db-sync-node:Info:81] [2024-09-23 11:16:13.69 UTC] The table epoch_stake was given a new unique constraint called unique_epoch_stake
[db-sync-node:Info:81] [2024-09-23 11:16:13.69 UTC] The table reward was given a new unique constraint called unique_reward
[db-sync-node:Info:81] [2024-09-23 11:16:13.71 UTC] Running database migrations in mode Indexes
[db-sync-node:Info:81] [2024-09-23 11:16:13.71 UTC] Found maintenance_work_mem=1GB, max_parallel_maintenance_workers=6
[db-sync-node:Warning:81] [2024-09-23 11:16:13.71 UTC] Creating Indexes. This may require an extended period of time to perform. Setting a higher maintenance_work_mem from Postgres usually speeds up this process. These indexes are not used by db-sync but are meant for clients. If you want to skip some of these indexes, you can stop db-sync, delete or modify any migration-4-* files in the schema directory and restart it.
ExitFailure 3

Errors in file: /tmp/migrate-2024-09-23T11:16:13.713492769Z.log

[db-sync-node:Error:81] [2024-09-23 11:16:14.78 UTC] runDBThread: ExitFailure 1
[db-sync-node:Error:105] [2024-09-23 11:16:14.78 UTC] ChainSyncWithBlocksPtcl: AsyncCancelled
[db-sync-node.Subscription:Error:101] [2024-09-23 11:16:14.78 UTC] Identity Application Exception: LocalAddress "../sancho/node.socket" SubscriberError {seType = SubscriberWorkerCancelled, seMessage = "SubscriptionWorker exiting", seStack = []}

cat /tmp/migrate-2024-09-23T11:16:13.713492769Z.log
Running : migration-1-0000-20190730.sql
WARNING: password file "config/pgpass-sancho" has group or world access; permissions should be u=rw (0600) or less
 init 
------
 
(1 row)

Running : migration-4-0002-20200810.sql
WARNING: password file "config/pgpass-sancho" has group or world access; permissions should be u=rw (0600) or less
Running : migration-4-0003-20210116.sql
WARNING: password file "config/pgpass-sancho" has group or world access; permissions should be u=rw (0600) or less
psql:schema/migration-4-0003-20210116.sql:12: NOTICE:  relation "idx_block_previous_id" already exists, skipping
psql:schema/migration-4-0003-20210116.sql:20: NOTICE:  relation "idx_pool_update_registered_tx_id" already exists, skipping
psql:schema/migration-4-0003-20210116.sql:37: ERROR:  column "stake_address_id" does not exist
ExitFailure 3

[kderme]

At the end of sync process on sanchonet I ran into issue:

I think this index also needs to move to file stage 4, version 1, since it's about tx_out.

[Cmdv]

Yeah I fixed that yesterday we were talking via dms 👍