Merge latest out-of-order indexing PR from metaplex #106
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix GetAssetsByCreator query
Handle and log http errors separately.
Partially addresses metaplex-foundation#50.
Enable configurable retries for the bgtask.
* acc forwarder * version to .0.1
Cleaning out unused imports.
Account forwarder
fix(das-api): fix getAssetProof
Handle HTTP errors
Fixing unused imports
Configurable bgtask runner
* add dto parameter * add dao condition * upd asset_data fetching to ensure single presence of asset_data in query * use asset.asset_data key for linking between asset/asset_data * fix: breaking arg order for array queries * chore: consistent arg order in `SearchAssets` destructuring * fix: breaking arg count for array params in searchAssets
* Hotfix for asset panic * Fixes some minor details * Update digital_asset_types/src/dapi/common/asset.rs Co-authored-by: Kirill Fomichev <fanatid@ya.ru> * Adding needed parameter. * Updating to kirill suggested change. * Hotfixes for files bugs Hotfixes to attempt to address some files bugs. * Remove commented code Clean out commented code. --------- Co-authored-by: Kirill Fomichev <fanatid@ya.ru> Co-authored-by: juanito87 <juancruz.f87@gmail.com>
Fixes CI build using latest rustc.
* Fix docker preparation script to build SPL * Update owner and delegate in asset table when collection or creator verification occurs * Modify program transformers to upsert in asset table * This allows out-of-order Bubblegum transactions to create and update the asset table. * Upsert leaf schema, owner, delegate, and seq separately since those are updated by all instructions and gated by sequence number to ensure freshest value. * Mint, burn, and decompress happen without regard to sequence number because they operate on unique fields. * Mint and burn have been updated but Decompress still needs to be fixed to handle out of order transactions. * Also remove unused 'filling' variable. * Update mint and decompress to be able to upsert asset info out of order * Add second sequence number for compression status fields * Reduce logging in docker * Comment out compressed_seq before regenerating Sea ORM objects * Add migration for asset specification * Update README * Rename PNFT and regenerate Sea ORM types * Apply usage of compressed_seq after regenerating Sea ORM types * Add owner delegate sequence number for owner and delegate fields. Also remove not null constraints for asset fields without defaults. * Regenerating database types * Update handling for non null constrained asset table * Update tests to use new Sea ORM types * Use owner_and_delegate_seq to separate upserts Also update redeem and decompress to not use leaf schema events. * Adding was_decompressed flag to replace compressed_seq compressed_seq won't work because decompression doesn't create a cl_event. * Regenerating Sea ORM types * Update code to use was_decompressed flag * Fix new boolean SQL conditions * Update comment * Remove column updates in asset table during mint for items not in model * Clippy fixes in ingester main * Cleanup debug comment * Allow for sequence number to be NULL (needed after decompress now) * Add leaf specific sequence number to protect that field in asset table * Revert "Allow for sequence number to be NULL (needed after decompress now)" This reverts commit 2713a18. * Update nft_ingester/src/program_transformers/bubblegum/redeem.rs Co-authored-by: Nicolas Pennie <Nicolas.Pennie@gmail.com> --------- Co-authored-by: Nicolas Pennie <Nicolas.Pennie@gmail.com>
…out of order (#87) * Fix docker preparation script to build SPL * Update owner and delegate in asset table when collection or creator verification occurs * Modify program transformers to upsert in asset table * This allows out-of-order Bubblegum transactions to create and update the asset table. * Upsert leaf schema, owner, delegate, and seq separately since those are updated by all instructions and gated by sequence number to ensure freshest value. * Mint, burn, and decompress happen without regard to sequence number because they operate on unique fields. * Mint and burn have been updated but Decompress still needs to be fixed to handle out of order transactions. * Also remove unused 'filling' variable. * Update mint and decompress to be able to upsert asset info out of order * Add second sequence number for compression status fields * Reduce logging in docker * Comment out compressed_seq before regenerating Sea ORM objects * Add migration for asset specification * Update README * Rename PNFT and regenerate Sea ORM types * Apply usage of compressed_seq after regenerating Sea ORM types * Add owner delegate sequence number for owner and delegate fields. Also remove not null constraints for asset fields without defaults. * Regenerating database types * Update handling for non null constrained asset table * Update tests to use new Sea ORM types * Use owner_and_delegate_seq to separate upserts Also update redeem and decompress to not use leaf schema events. * Adding was_decompressed flag to replace compressed_seq compressed_seq won't work because decompression doesn't create a cl_event. * Regenerating Sea ORM types * Update code to use was_decompressed flag * Fix new boolean SQL conditions * Update comment * Remove column updates in asset table during mint for items not in model * Clippy fixes in ingester main * Cleanup debug comment * Allow for sequence number to be NULL (needed after decompress now) * Add leaf specific sequence number to protect that field in asset table * Revert "Allow for sequence number to be NULL (needed after decompress now)" This reverts commit 2713a18. * Update nft_ingester/src/program_transformers/bubblegum/redeem.rs Co-authored-by: Nicolas Pennie <Nicolas.Pennie@gmail.com> * Change creator verification to use upserts to support out of order * Remove null constraints on asset_creators table * Add null clause to upsert during mint * Rename creator vecs and add comments * Removing comment * Fix typo in migration down function * Fix collection verification and change to use upserts to support out of order processing (#90) * Adding verified flag to asset_grouping table * Regenerate Sea ORM types * Remove null constraints on asset_grouping table * Regenerate Sea ORM types * Update digital asset types and ingester based on new Sea ORM objects * Setting new verified flag in asset_grouping table to be non null with default Also regenerating Sea ORM types * Separate out collection insert in mintV1 into separate upserts * Fix error message * Separate update collection base info from collection verified * Add group info seq to asset_grouping table * Regenerate Sea ORM types * Add group_info_seq checks to collection base info upsert * Add check for verified = true in grouping for Read API * Fix conditions for asset grouping updates * Require grouping to verified to be returned from API in all cases --------- Co-authored-by: Nicolas Pennie <Nicolas.Pennie@gmail.com>
NicolasPennie
pushed a commit
that referenced
this pull request
Aug 23, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Testing
All test cases were run "forward and backwards", to ensure out of order updates work as anticipated.
Test cases: