[mtg-836] Separate fungible token accounts from non-fungible ones #305

kstepanovdev · 2024-11-05T15:42:58Z

No description provided.

…res for different types of assets

…ate-fungible-tokens-updates

StanChe

Let's catch up and sync on those 2 synchronizers running. The approach looks good, the separation should be thought through. I'd really love to merge this change asap

migrations/9_insert_fungible_idx.sql

rocks-db/src/batch_client.rs

rocks-db/src/asset.rs

nft_ingester/src/processors/accounts_processor.rs

postgre-client/src/asset_index_client.rs

postgre-client/src/temp_index_client.rs

nft_ingester/src/index_syncronizer.rs

entities/src/models.rs

nft_ingester/src/index_syncronizer.rs

StanChe

As an idea (not a suggestion to implement right now) - wdyt of running 2 separate binaries - synchronizer_fungible, synchronizer_nft. Maybe it'll make separation easier. Or it could be a single binary with run mode - fungible/non-fungible

nft_ingester/src/api/synchronization_state_consistency.rs

rocks-db/src/batch_client.rs

nft_ingester/src/processors/account_based/token_updates_processor.rs

postgre-client/src/asset_index_client.rs

postgre-client/src/temp_index_client.rs

rocks-db/src/batch_client.rs

also renamed methods due to their different nature

&& updated tests && refacrored ASSET_TYPE constant

StanChe

I still need to go over the synchronizer

nft_ingester/src/api/backfilling_state_consistency.rs

nft_ingester/src/bin/synchronizer/main.rs

postgre-client/src/load_client.rs

postgre-client/src/asset_index_client.rs

nft_ingester/src/index_syncronizer.rs

StanChe · 2024-11-25T18:44:58Z

postgre-client/src/asset_index_client.rs

+                };
+
+                self.copy_fungibles(fungible_tokens_path, &mut transaction)
+                    .await?;


The explicit rollback will not be called here, if the copy_fungible errors, so we'll still rely on rust to rollback the tx

StanChe

Great job, thank you! Those transaction rollbacks should be called explicitely on any errors though

Write a fungible token-related events into a dedicated RocksDb column

3ff2e46

kstepanovdev changed the title ~~Write a fungible token-related events into a dedicated RocksDb column~~ Separate fungible token accounts from non-fungible ones Nov 5, 2024

kstepanovdev and others added 15 commits November 6, 2024 13:07

Added writing fungible assets into RocksDb

15f7b20

add multithread operations for writing/reading from rocksDb and Postg…

c8afedd

…res for different types of assets

Fixed migration

8d4b5d8

turn the fork cleaner on/off from configuration

35265ab

nit the comment

b0bc8d7

updated dependencies and imports to support other SVMs

354734c

Split methods for fungible and non-fungible assets

61d09a6

Fix non-integration tests

1651dc3

chore: naming in env (#308)

ae88520

chore: env example changes

4328c4a

Merge branch 'main' of github.com:metaplex-foundation/aura into separ…

23f726f

…ate-fungible-tokens-updates

Add FungibleAssets to discriptors and update the RocksDB data model

a8f2132

Await for syncronizer to finish its work

a7fa0c7

Await for synchronizer execution

b4866af

Satisfy clippy

0cf3ab9

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from 079a666 to 0cf3ab9 Compare November 11, 2024 16:20

Even further separation of fungible assets updates from non-fungible

fe3ccd8

kstepanovdev changed the title ~~Separate fungible token accounts from non-fungible ones~~ [mtg-836] Separate fungible token accounts from non-fungible ones Nov 12, 2024

kstepanovdev requested a review from RequescoS November 12, 2024 16:53

kstepanovdev marked this pull request as ready for review November 12, 2024 17:26

kstepanovdev requested a review from StanChe November 14, 2024 18:25

kstepanovdev self-assigned this Nov 14, 2024

StanChe requested changes Nov 14, 2024

View reviewed changes

RequescoS reviewed Nov 15, 2024

View reviewed changes

entities/src/models.rs Show resolved Hide resolved

RequescoS reviewed Nov 15, 2024

View reviewed changes

nft_ingester/src/index_syncronizer.rs Show resolved Hide resolved

Fix comments from the review

2d9046d

kstepanovdev requested review from RequescoS and StanChe November 15, 2024 15:19

Satisfy clippy

0622c99

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from a8ccae6 to 0622c99 Compare November 15, 2024 16:16

StanChe reviewed Nov 19, 2024

View reviewed changes

kstepanovdev added 5 commits November 19, 2024 17:39

Add transactions for some update assets indexes batch operations

290175a

Removed redundant call for fungible token index, \

13bef3e

also renamed methods due to their different nature

Add a possibility to store fungible assets into the temporary db

1ac335f

Fixed tests

db7d5e9

Fixed overwhelm_seq_gap

555d596

&& updated tests && refacrored ASSET_TYPE constant

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from dcf47db to 555d596 Compare November 19, 2024 19:13

kstepanovdev added 2 commits November 19, 2024 22:09

Splitted load_from_dump() for fungible and non-fungible parts

024dab9

Fixed dump tests && unparallel dump test

0746bc1

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from 9f903ad to 0746bc1 Compare November 20, 2024 08:54

improve SQL readability

5745429

StanChe reviewed Nov 22, 2024

View reviewed changes

Change naming for non fungible tokens in methods

63a1c4a

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from 18d6a4a to f97a3e8 Compare November 22, 2024 12:42

Fix tests

08f705a

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from f97a3e8 to 08f705a Compare November 22, 2024 12:43

StanChe reviewed Nov 22, 2024

View reviewed changes

postgre-client/src/asset_index_client.rs Show resolved Hide resolved

StanChe reviewed Nov 22, 2024

View reviewed changes

postgre-client/src/asset_index_client.rs Outdated Show resolved Hide resolved

StanChe reviewed Nov 22, 2024

View reviewed changes

nft_ingester/src/index_syncronizer.rs Show resolved Hide resolved

kstepanovdev added 2 commits November 22, 2024 18:01

Added explicit rollbacks

5067244

Separate dumping of fungible tokens from NFTs

86b4587

kstepanovdev requested a review from StanChe November 22, 2024 17:10

StanChe reviewed Nov 25, 2024

View reviewed changes

StanChe approved these changes Nov 25, 2024

View reviewed changes

Fix explicit rollback transaction on error && add metrics

4a9b34f

kstepanovdev force-pushed the separate-fungible-tokens-updates branch from d5f3d6d to 4a9b34f Compare November 26, 2024 02:29

kstepanovdev merged commit 74cd2ee into main Nov 26, 2024
2 checks passed

kstepanovdev deleted the separate-fungible-tokens-updates branch November 26, 2024 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[mtg-836] Separate fungible token accounts from non-fungible ones #305

[mtg-836] Separate fungible token accounts from non-fungible ones #305

kstepanovdev commented Nov 5, 2024

StanChe left a comment

StanChe left a comment

StanChe left a comment

StanChe Nov 25, 2024

StanChe left a comment

[mtg-836] Separate fungible token accounts from non-fungible ones #305

[mtg-836] Separate fungible token accounts from non-fungible ones #305

Conversation

kstepanovdev commented Nov 5, 2024

StanChe left a comment

Choose a reason for hiding this comment

StanChe left a comment

Choose a reason for hiding this comment

StanChe left a comment

Choose a reason for hiding this comment

StanChe Nov 25, 2024

Choose a reason for hiding this comment

StanChe left a comment

Choose a reason for hiding this comment