Have all the special storage keys in a single place

[xlc]

I am working on a feature of Chopsticks to decode all the storage keys. There are some keys requires special handling as they are not included in metadata.

Currently we have

polkadot-sdk/substrate/primitives/storage/src/lib.rs

Lines 192 to 216 in 86228fa

	/// Wasm code of the runtime.
	///
	/// Stored as a raw byte vector. Required by substrate.
	pub const CODE: &[u8] = b":code";
	/// Number of wasm linear memory pages required for execution of the runtime.
	///
	/// The type of this value is encoded `u64`.
	pub const HEAP_PAGES: &[u8] = b":heappages";
	/// Current extrinsic index (u32) is stored under this key.
	///
	/// Encodes to `0x3a65787472696e7369635f696e646578`.
	pub const EXTRINSIC_INDEX: &[u8] = b":extrinsic_index";
	/// Current intra-block entropy (a universally unique `[u8; 32]` value) is stored here.
	///
	/// Encodes to `0x3a696e747261626c6f636b5f656e74726f7079`.
	pub const INTRABLOCK_ENTROPY: &[u8] = b":intrablock_entropy";
	/// Prefix of child storage keys.
	pub const CHILD_STORAGE_KEY_PREFIX: &[u8] = b":child_storage:";
	/// Prefix of the default child storage keys in the top trie.
	pub const DEFAULT_CHILD_STORAGE_KEY_PREFIX: &[u8] = b":child_storage:default:";

polkadot-sdk/substrate/frame/support/src/traits/metadata.rs

Lines 163 to 167 in 86228fa

	/// The storage key postfix that is used to store the [`StorageVersion`] per pallet.
	///
	/// The full storage key is built by using:
	/// Twox128([`PalletInfo::name`]) ++ Twox128([`STORAGE_VERSION_STORAGE_KEY_POSTFIX`])
	pub const STORAGE_VERSION_STORAGE_KEY_POSTFIX: &[u8] = b":__STORAGE_VERSION__:";

polkadot-sdk/substrate/frame/support/src/storage/transactional.rs

Lines 34 to 37 in 86228fa

	/// The key that is holds the current number of active layers.
	///
	/// Encodes to `0x3a7472616e73616374696f6e5f6c6576656c3a`.
	pub const TRANSACTION_LEVEL_KEY: &[u8] = b":transaction_level:";

polkadot-sdk/substrate/primitives/consensus/grandpa/src/lib.rs

Lines 65 to 67 in 86228fa

	/// The storage key for the current set of weighted Grandpa authorities.
	/// The value stored is an encoded VersionedAuthorityList.
	pub const GRANDPA_AUTHORITIES_KEY: &[u8] = b":grandpa_authorities";

polkadot-sdk/polkadot/primitives/src/v6/mod.rs

Lines 242 to 251 in 86228fa

	/// The upward message dispatch queue remaining capacity for the given para id.
	///
	/// The storage entry stores a tuple of two values:
	///
	/// - `count: u32`, the number of additional messages which may be enqueued for the given para,
	/// - `total_size: u32`, the total size of additional messages which may be enqueued for the
	/// given para.
	pub fn relay_dispatch_queue_remaining_capacity(para_id: Id) -> WellKnownKey<(u32, u32)> {
	(b":relay_dispatch_queue_remaining_capacity", para_id).encode().into()
	}

Please let me know if I missed anything

This makes it really hard to implement a generic and all-catching storage decoder. Can we move all the Substrate well known keys to sp_core::storage::well_known_keys? The polkadot one are all in a single place so that's good.

[xlc]

The key format is also inconsistent. Some of them are :xxx and some of them are :xxx: but I guess we are already too late to change them.

We should also update the docs to include encoded hex and the expected value type.

[bkchr]

The polkadot one are all in a single place so that's good.

The Polkadot ones are not special keys, these are just the normal FRAME storage keys.

[xlc]

:relay_dispatch_queue_remaining_capacity is definitely a special key but yeah everything else seems just storage keys encoded. Maybe we could refactor them in such why we code the string and have macro to convert them to hex in compile time? So it is more obvious where the hex is coming from.

[bkchr]

:relay_dispatch_queue_remaining_capacity

Ahh you got me :P

Maybe we could refactor them in such why we code the string and have macro to convert them to hex in compile time?

In a perfect way we would not need them at all. Or better abstracted. But we already have the hash macros, so it should actually work.

[ggwpez]

I think we should use storage_alias to have type safety.

[dharjeezy]

is this issue just all about creating a cratewell_known_keys in sp_storage and moving the keys specified into it and then refractoring? is there anything else to be done asides that? @ggwpez

[ggwpez]

It seems a bit odd to me that the keys for runtime specific things are specified in sp_storage and not a frame crate.
But since sp-storage cannot depend on frame, we could re-export all well known keys somewhere in frame-support.

Not sure if that is ideal, better ideas?

and then refractoring

Please first add a deprecation warning. If you want to remove something.

[dharjeezy]

@ggwpez can we then put all these keys in the frame support storage crate?

[ggwpez]

We can re-export them there yes, but they are also needed in lower level crates, so they need to stay in there as well.

I guess a common re-export module is the best we can do now.

[dharjeezy]

We can re-export them there yes, but they are also needed in lower level crates, so they need to stay in there as well.

I guess a common re-export module is the best we can do now.

i don't seem to quite get what you mean by a common re-export module? what do you mean? @ggwpez

[ggwpez]

mod well_known_keye {
    pub use sp_storage::...
    pub use sp_consensus_grandpa::...
}

and so on. Although it would probably not cover the ones from polkadot primitives, since what would create a dependency cycle again.

Not sure what @bkchr was thinking.

[bkchr]

It seems a bit odd to me that the keys for runtime specific things are specified in sp_storage and not a frame crate.

Because these keys are not FRAME specific. These are Substrate well known keys for communicating data to the node. Things like the code should not require to first call into the runtime.

[bkchr]

Generally, maybe just adding a page in the polkadot sdk docs is a better idea. I don't see any reason for adding them somewhere to re-export them, while this ticket is just about being able to find these keys/having a place to look them up.

[dharjeezy]

so, no need to get this done anymore? @bkchr

[bkchr]

As I said above, I think there should be a guide/reference doc added to the sdk docs.

[aurexav]

Recently, I create this https://github.com/subscan-explorer/well-known-address-registry.

I'm interesting to add the well know storage keys there. It might be an option.

And also #324. Subscan team is struggling with the custom pallet account across different chains. 🤷🏻‍♂️