-
Notifications
You must be signed in to change notification settings - Fork 91
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
DA compression for fuel-tx types (#670)
* WIP * Migrate compact-derive from fuel-core * Add doc comments and cleanup * Add changelog * Cargo.toml fmt * Wrap Compaction return types to anyhow::Result * Update readme * Fix wording Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh> * Combine some bounds Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh> * Fix issues after merge * Fix compression for latest types * Keep used gas price * Cargo sort * Introduce ContractId table to da compression registry * Reference table types directly in #[da_compress(registry = _)] * WIP: migrate domain logic to fuel-core * Re-add dummy registry for testing * Add some missing docs * Move block section data for fuel-core * Minor change for fuel-core integration * cargo sort * Fix incorrect feature cfg on default_test_tx * Implement compacting for blobs as well * Fix no_std deps * WIP: working towards adapting Green's trait-based approach * Complete migration to Green's trait-heavy approach * Polish: docs, naming, argument order * Clean up proc macro a bit * Line count reduction :\ * More polish * Move readme from fuel-compression to fuel-core-compression * Enable more lints for fuel-derive, remove unnecessary regex dependency * Remove debug writing to /tmp, as we're hopefully done here * Get rid of anyhow * Add roundtrip tests * Use type-based registry keyspaces * Omit Message amount from DA compressed data * Remove extra "Compressible" bound from "RegistrySubstitutableBy" * Implement da compression for TxId -> TxPointer * Use the type instead of field annotations for determining compressed fields * Make da compression async * Also make compression context async * unused_crate_dependencies fix * More unused_crate_dependencies * cargo sort * Cosmetic changes: - Renamed derive macro `Compressed` -> `CompressibleBy` - Renamed `da_compress` -> `compressible_by` - Removed default constrains from `Compressible::Compressed` - Fixed compilation after removing constrains - Removed `da_compress(bound)` - Added new `TxId` type * Cleanup small nits after review * Revert commits af39dcf and bf75dbc * Split derive macro "Compressed" into "Compress" and "Decompress" * Re-apply Green's da_compress(bound) removal and bound simplification * Re-introduce Green's TxId type, do some cleanup related to these types * Re-apply bf75dbc * Compress the whole UtxoId instead of just TxId * Compress UtxoId: test fixes * Introduce PredicateCode to allow compressing it * Make fields for Compressed* public * Remove Decompress derive from Message and Coin Also, rename {C,De}omprssibleBy methods to {de}compress_with * Rename da_compress attribute to just compress * Add tests for TxId * Rename remaining instances of compaction into compression * Add explicit CompressedUtxoId type * Rename De/CompressionContext methods Fix tests * Clippy * Ignore incorrect unused crate warning * Fix no-default-features * Remove De/CompressionContext traits, they seem unnecessary * Simplifying tests for DA compression (#816) * Address PR review comments * Fix , => ; * Fix a memory leak in unsafe array de/compress code * Fix typo Co-authored-by: Rafał Chabowski <88321181+rafal-ch@users.noreply.github.com> * Fix typo Co-authored-by: Rafał Chabowski <88321181+rafal-ch@users.noreply.github.com> * Add a test cases to cover skipped fields and nested structs * Remove unneeded change of field ref for struct that impls Deref * Remove useless .scollect()s * Clarify variable namings around serialization in a test case * Add a test case showing that skipped fields are not part of the compressed output --------- Co-authored-by: Brandon Vrooman <brandon.vrooman@fuel.sh> Co-authored-by: green <xgreenx9999@gmail.com> Co-authored-by: Rafał Chabowski <88321181+rafal-ch@users.noreply.github.com>
- Loading branch information
4 people
authored
Sep 10, 2024
1 parent
b6defbc
commit d28a143
Showing
53 changed files
with
1,923 additions
and
81 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| [package] | ||
| name = "fuel-compression" | ||
| version = { workspace = true } | ||
| authors = { workspace = true } | ||
| categories = ["cryptography::cryptocurrencies"] | ||
| edition = { workspace = true } | ||
| homepage = { workspace = true } | ||
| keywords = ["blockchain", "cryptocurrencies", "fuel-compression"] | ||
| license = { workspace = true } | ||
| repository = { workspace = true } | ||
| description = "Compression and decompression of Fuel blocks for DA storage." | ||
|
|
||
| [dependencies] | ||
| fuel-derive = { workspace = true } | ||
| fuel-types = { workspace = true } | ||
| serde = { version = "1.0", features = ["derive"] } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| //! Trait impls for Rust types | ||
| use super::traits::*; | ||
| use crate::RegistryKey; | ||
| use core::mem::MaybeUninit; | ||
| use fuel_types::{ | ||
| Address, | ||
| AssetId, | ||
| BlobId, | ||
| BlockHeight, | ||
| Bytes32, | ||
| ContractId, | ||
| Nonce, | ||
| Salt, | ||
| }; | ||
|
|
||
| macro_rules! identity_compression { | ||
| ($t:ty) => { | ||
| impl Compressible for $t { | ||
| type Compressed = Self; | ||
| } | ||
|
|
||
| impl<Ctx> CompressibleBy<Ctx> for $t | ||
| where | ||
| Ctx: ContextError, | ||
| { | ||
| async fn compress_with(&self, _: &mut Ctx) -> Result<Self, Ctx::Error> { | ||
| Ok(*self) | ||
| } | ||
| } | ||
|
|
||
| impl<Ctx> DecompressibleBy<Ctx> for $t | ||
| where | ||
| Ctx: ContextError, | ||
| { | ||
| async fn decompress_with( | ||
| c: &Self::Compressed, | ||
| _: &Ctx, | ||
| ) -> Result<Self, Ctx::Error> { | ||
| Ok(*c) | ||
| } | ||
| } | ||
| }; | ||
| } | ||
|
|
||
| identity_compression!(u8); | ||
| identity_compression!(u16); | ||
| identity_compression!(u32); | ||
| identity_compression!(u64); | ||
| identity_compression!(u128); | ||
|
|
||
| identity_compression!(BlockHeight); | ||
| identity_compression!(BlobId); | ||
| identity_compression!(Bytes32); | ||
| identity_compression!(Salt); | ||
| identity_compression!(Nonce); | ||
|
|
||
| impl Compressible for Address { | ||
| type Compressed = RegistryKey; | ||
| } | ||
|
|
||
| impl Compressible for ContractId { | ||
| type Compressed = RegistryKey; | ||
| } | ||
|
|
||
| impl Compressible for AssetId { | ||
| type Compressed = RegistryKey; | ||
| } | ||
|
|
||
| impl<const S: usize, T> Compressible for [T; S] | ||
| where | ||
| T: Compressible, | ||
| { | ||
| type Compressed = [T::Compressed; S]; | ||
| } | ||
|
|
||
| impl<const S: usize, T, Ctx> CompressibleBy<Ctx> for [T; S] | ||
| where | ||
| T: CompressibleBy<Ctx>, | ||
| Ctx: ContextError, | ||
| { | ||
| #[allow(unsafe_code)] | ||
| async fn compress_with(&self, ctx: &mut Ctx) -> Result<Self::Compressed, Ctx::Error> { | ||
| // SAFETY: we are claiming to have initialized an array of `MaybeUninit`s, | ||
| // which do not require initialization. | ||
| let mut tmp: [MaybeUninit<T::Compressed>; S] = | ||
| unsafe { MaybeUninit::uninit().assume_init() }; | ||
|
|
||
| let mut i = 0; | ||
| while i < self.len() { | ||
| match self[i].compress_with(ctx).await { | ||
| Ok(value) => { | ||
| // SAFETY: MaybeUninit can be safely overwritten. | ||
| tmp[i].write(value); | ||
| } | ||
| Err(e) => { | ||
| // Drop the already initialized elements, so we don't leak the memory | ||
| for initialized_item in tmp.iter_mut().take(i) { | ||
| // Safety: First i elements have been initialized successfully. | ||
| unsafe { | ||
| initialized_item.assume_init_drop(); | ||
| } | ||
| } | ||
| return Err(e); | ||
| } | ||
| } | ||
| i += 1; | ||
| } | ||
|
|
||
| // SAFETY: Every element is initialized. In case of error, we have returned | ||
| // instead. | ||
| let result = tmp.map(|v| unsafe { v.assume_init() }); | ||
| Ok(result) | ||
| } | ||
| } | ||
|
|
||
| impl<const S: usize, T, Ctx> DecompressibleBy<Ctx> for [T; S] | ||
| where | ||
| T: DecompressibleBy<Ctx>, | ||
| Ctx: ContextError, | ||
| { | ||
| #[allow(unsafe_code)] | ||
| async fn decompress_with( | ||
| c: &Self::Compressed, | ||
| ctx: &Ctx, | ||
| ) -> Result<Self, Ctx::Error> { | ||
| // SAFETY: we are claiming to have initialized an array of `MaybeUninit`s, | ||
| // which do not require initialization. | ||
| let mut tmp: [MaybeUninit<T>; S] = unsafe { MaybeUninit::uninit().assume_init() }; | ||
|
|
||
| let mut i = 0; | ||
| while i < c.len() { | ||
| match T::decompress_with(&c[i], ctx).await { | ||
| Ok(value) => { | ||
| // SAFETY: MaybeUninit can be safely overwritten. | ||
| tmp[i].write(value); | ||
| } | ||
| Err(e) => { | ||
| // Drop the already initialized elements, so we don't leak the memory | ||
| for initialized_item in tmp.iter_mut().take(i) { | ||
| // Safety: First i elements have been initialized successfully. | ||
| unsafe { | ||
| initialized_item.assume_init_drop(); | ||
| } | ||
| } | ||
| return Err(e); | ||
| } | ||
| } | ||
| i += 1; | ||
| } | ||
|
|
||
| // SAFETY: Every element is initialized. | ||
| let result = tmp.map(|v| unsafe { v.assume_init() }); | ||
| Ok(result) | ||
| } | ||
| } | ||
|
|
||
| impl<T> Compressible for Vec<T> | ||
| where | ||
| T: Compressible, | ||
| { | ||
| type Compressed = Vec<T::Compressed>; | ||
| } | ||
|
|
||
| impl<T, Ctx> CompressibleBy<Ctx> for Vec<T> | ||
| where | ||
| T: CompressibleBy<Ctx>, | ||
| Ctx: ContextError, | ||
| { | ||
| async fn compress_with(&self, ctx: &mut Ctx) -> Result<Self::Compressed, Ctx::Error> { | ||
| let mut result = Vec::with_capacity(self.len()); | ||
| for item in self { | ||
| result.push(item.compress_with(ctx).await?); | ||
| } | ||
| Ok(result) | ||
| } | ||
| } | ||
|
|
||
| impl<T, Ctx> DecompressibleBy<Ctx> for Vec<T> | ||
| where | ||
| T: DecompressibleBy<Ctx>, | ||
| Ctx: ContextError, | ||
| { | ||
| async fn decompress_with( | ||
| c: &Self::Compressed, | ||
| ctx: &Ctx, | ||
| ) -> Result<Self, Ctx::Error> { | ||
| let mut result = Vec::with_capacity(c.len()); | ||
| for item in c { | ||
| result.push(T::decompress_with(item, ctx).await?); | ||
| } | ||
| Ok(result) | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| use serde::{ | ||
| Deserialize, | ||
| Serialize, | ||
| }; | ||
|
|
||
| /// Untyped key pointing to a registry table entry. | ||
| /// The last key (all bits set) is reserved for the default value and cannot be written | ||
| /// to. | ||
| #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] | ||
| pub struct RegistryKey([u8; Self::SIZE]); | ||
| impl RegistryKey { | ||
| /// Key mapping to default value for the table type. | ||
| pub const DEFAULT_VALUE: Self = Self([u8::MAX; Self::SIZE]); | ||
| /// Maximum writable key. | ||
| pub const MAX_WRITABLE: Self = Self([u8::MAX, u8::MAX, u8::MAX - 1]); | ||
| /// Size of the key, in bytes. | ||
| pub const SIZE: usize = 3; | ||
| /// Zero key. | ||
| pub const ZERO: Self = Self([0; Self::SIZE]); | ||
|
|
||
| /// Convert to u32, big-endian. | ||
| pub fn as_u32(self) -> u32 { | ||
| u32::from_be_bytes([0, self.0[0], self.0[1], self.0[2]]) | ||
| } | ||
|
|
||
| /// Wraps around just below max/default value. | ||
| /// Panics for max/default value. | ||
| pub fn next(self) -> Self { | ||
| if self == Self::DEFAULT_VALUE { | ||
| panic!("Max/default value has no next key"); | ||
| } | ||
| let next_raw = self.as_u32() + 1u32; | ||
| if next_raw == Self::DEFAULT_VALUE.as_u32() { | ||
| Self::ZERO | ||
| } else { | ||
| Self::try_from(next_raw) | ||
| .expect("The procedure above always produces a valid key") | ||
| } | ||
| } | ||
| } | ||
| impl TryFrom<u32> for RegistryKey { | ||
| type Error = &'static str; | ||
|
|
||
| fn try_from(value: u32) -> Result<Self, Self::Error> { | ||
| let v = value.to_be_bytes(); | ||
| if v[0] != 0 { | ||
| return Err("RegistryKey must be less than 2^24"); | ||
| } | ||
|
|
||
| let mut bytes = [0u8; 3]; | ||
| bytes.copy_from_slice(&v[1..]); | ||
| Ok(Self(bytes)) | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::RegistryKey; | ||
|
|
||
| #[test] | ||
| fn key_next() { | ||
| assert_eq!(RegistryKey::ZERO.next(), RegistryKey([0, 0, 1])); | ||
| assert_eq!(RegistryKey::ZERO.next().next(), RegistryKey([0, 0, 2])); | ||
| assert_eq!(RegistryKey([0, 0, 255]).next(), RegistryKey([0, 1, 0])); | ||
| assert_eq!(RegistryKey([0, 1, 255]).next(), RegistryKey([0, 2, 0])); | ||
| assert_eq!(RegistryKey([0, 255, 255]).next(), RegistryKey([1, 0, 0])); | ||
| assert_eq!(RegistryKey::MAX_WRITABLE.next(), RegistryKey::ZERO); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| //! Compression and decompression of fuel-types for the DA layer | ||
| #![cfg_attr(docsrs, feature(doc_auto_cfg))] | ||
| #![warn(missing_docs)] | ||
| #![deny(unsafe_code)] | ||
| #![deny(unused_crate_dependencies)] | ||
| #![deny(clippy::cast_possible_truncation)] | ||
|
|
||
| mod impls; | ||
| mod key; | ||
| mod traits; | ||
|
|
||
| pub use key::RegistryKey; | ||
| pub use traits::*; | ||
|
|
||
| pub use fuel_derive::{ | ||
| Compress, | ||
| Decompress, | ||
| }; |
Oops, something went wrong.