From 96d3ae0d4cf6e0a348f20af898afbc38c53be616 Mon Sep 17 00:00:00 2001 From: Ankan Date: Sun, 7 Jan 2024 02:07:45 +0530 Subject: [PATCH 01/15] wasm case --- syllabus/5-Substrate/1-Intro-to-Substrate_Slides.md | 2 +- ...M-Meta-Protocol-Slides.md => 2-Wasm-Meta-Protocol-Slides.md} | 0 syllabus/5-Substrate/README.md | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) rename syllabus/5-Substrate/{2-WASM-Meta-Protocol-Slides.md => 2-Wasm-Meta-Protocol-Slides.md} (100%) diff --git a/syllabus/5-Substrate/1-Intro-to-Substrate_Slides.md b/syllabus/5-Substrate/1-Intro-to-Substrate_Slides.md index 41846b51f..7568e9fd7 100644 --- a/syllabus/5-Substrate/1-Intro-to-Substrate_Slides.md +++ b/syllabus/5-Substrate/1-Intro-to-Substrate_Slides.md @@ -351,7 +351,7 @@ _The way to make a protocol truly upgradeable is to design a meta-protocol that Note: In this figure, the meta-protocol, the substrate client, is not forklessly upgrade-able. It can only -be upgraded with a fork. The WASM protocol, though, can be upgraded forklessly. +be upgraded with a fork. The Wasm protocol, though, can be upgraded forklessly. ---v diff --git a/syllabus/5-Substrate/2-WASM-Meta-Protocol-Slides.md b/syllabus/5-Substrate/2-Wasm-Meta-Protocol-Slides.md similarity index 100% rename from syllabus/5-Substrate/2-WASM-Meta-Protocol-Slides.md rename to syllabus/5-Substrate/2-Wasm-Meta-Protocol-Slides.md diff --git a/syllabus/5-Substrate/README.md b/syllabus/5-Substrate/README.md index 4f78b9904..8492c93f8 100644 --- a/syllabus/5-Substrate/README.md +++ b/syllabus/5-Substrate/README.md @@ -31,7 +31,7 @@ Ensure the `main` branch is write protected, by required a PR first`- no one sho #### Morning 1. [Introduction](./1-Intro-to-Substrate_Slides.md) (60m) -1. [WASM Meta Protocol](./2-WASM-Meta-Protocol-Slides.md) (90m) +1. [Wasm Meta Protocol](./2-Wasm-Meta-Protocol-Slides) (90m) 1. Activity: Finding Runtime APIs and Host Functions in Substrate From 7752dd00f03af77406756db17ee0635f32d84510 Mon Sep 17 00:00:00 2001 From: Ankan Date: Thu, 11 Jan 2024 19:27:24 +0530 Subject: [PATCH 02/15] update to merklized storage --- .../5-Substrate/3-Merklized-Storage_Slides.md | 457 ++++++++++-------- syllabus/5-Substrate/9-SCALE_Slides.md | 30 +- .../9-Substrate-Interactions_Slides.md | 5 +- 3 files changed, 277 insertions(+), 215 deletions(-) diff --git a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md index d97dbbfa3..2c4196df6 100644 --- a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md +++ b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md @@ -13,49 +13,58 @@ duration: 60mins ---v -### What We Know So Far - -- Recall that at the `sp_io` layer, you have **opaque keys and values**. - -- `sp_io::storage::get(vec![8, 2])`; - - `vec![8, 2]` is a "storage key". -- `sp_io::storage::set(vec![2, 9], vec![42, 33])`; +#### What We Know So Far ----v - -### What We Know So Far - -Nomenclature (with some simplification): +**Externalities** > Environment providing host functions, namely storage ones: "`Externalities` Environment". Notes: - In Substrate, a type needs to provide the environment in which host functions are provided, and can be executed. -- We call this an "externality environment", represented by [`trait Externalities`](https://paritytech.github.io/substrate/master/sp_externalities/trait.Externalities.html). +- We call this an "externality environment", represented + by [`trait Externalities`](https://paritytech.github.io/substrate/master/sp_externalities/trait.Externalities.html). - By convention, an externality has a "**backend**" that is in charge of dealing with storage. +- Externality is a trait that provides functionality to interact with storage and other extensions registered in the + node. ---v ### What We Know So Far + + +---v + +### What We Know So Far + +- Recall that at the `sp_io` layer, you have **opaque keys and values**. + ```rust -sp_io::TestExternalities::new_empty().execute_with(|| { - sp_io::storage::get(..); -}); + let storage_key = vec![8, 2]; + sp_io::storage::get(storage_key); + sp_io::storage::set(storage_key, vec![42, 33]); + ``` ---v ### What We Know So Far - +```rust + sp_io::TestExternalities::new_empty().execute_with(|| { + sp_io::storage::get(..); + }); +``` + +Notes: +- TestExternalities mimic a client. --- ## Key Value -- How about a key-value storage externality? why not? 🙈 +> All this seems to indicate our storage externality is a simple key value database. ---v @@ -67,22 +76,27 @@ sp_io::TestExternalities::new_empty().execute_with(|| { ### Key Value -- "_Storage keys_" (whatever you pass to `sp_io::storage`) directly maps to "_database keys_". +- Concatenate all data and hash to get the root. - O(1) read and write. -- Hash all the data to get the root. -Notes: +> Spoiler, that is not how data is stored internally in the database. + -Good time to hammer down what you mean by storage key and what you mean by database key. +Notes: +- "_Storage keys_" (whatever you pass to `sp_io::storage`) directly maps to "_database keys_". +- Probably don't wanna introduce storage key and db key right now. + Good time to hammer down what you mean by storage key and what you mean by database key. literally imagine that in the implementation of `sp_io::storage::set`, we write it to a key-value database. ---v -### Key Value +### Key Value: Proof sizes -- If alice only has this root, how can I prove to her how much balance she has? +- Suppose there is a large database. +- Alice has the state root of this database, wants to lookup her balance from this database. +- How can Alice verify the balance she receives from a full node is correct? SEND HER THE WHOLE DATABASE 😱. @@ -94,24 +108,23 @@ Alice is representing a light client, I represent a full node. ---v -### Key Value +### Key Value: State Root -- Moreover, if you change a single key-value, we need to re-hash the whole thing again to get the updated state root 🤦. +- If you change a single key-value, we need to re-hash the whole thing again to get the updated state root 🤦. --- ## Substrate Storage: Merklized -- This brings us again to why blockchain based systems tend to "merkelize" their storage. - ----v +> Substrate uses a base-16, (patricia) radix merkle tree. -### Merklized - -> Substrate uses a base-16, (patricia) radix merkle trie. +Notes: +- Find the code at [paritytech/trie](https://github.com/paritytech/trie). ---v +### Recap + @@ -125,8 +138,6 @@ flowchart TD C --> F["F \n value: 0x34"] - - @@ -138,36 +149,36 @@ flowchart TD ---v +### Recap + %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart TD - A --b--> C["C \n Hash(F) \n"] - A["A \n value: Hash(B|C)"] -- a --> B["B \n value: Hash(B|E)"] - B --c--> D["D \n value: 0x12"] - B --d--> E["E \n value: 0x23"] - C --e--> F["F \n value: 0x34"] + A["A \n value: Hash(B|C)"] -- v --> B["B \n value: Hash(B|E)"] + A --w--> C["C \n Hash(F) \n"] + B --"x"--> D["D \n value: 0x12"] + B --y--> E["E \n value: 0x23"] + C --z--> F["F \n value: 0x34"] - - -- Trie. +- Trie - Assuming only leafs have data, this is encoding: - + - + - +
"ac" => 0x12 "vx" => 0x12
"ad" => 0x23 "vy" => 0x23
"be" => 0x34 "wz" => 0x34
@@ -176,24 +187,25 @@ flowchart TD Notes: -this is how we encode key value based data in a trie. +- this is how we encode key value based data in a trie. +- Optimization of simple trie, ---v +### Recap + %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart TD - A["A \n Hash(B|C)"] -- a --> B["B \n Hash(B|E)"] - A --be--> F["F \n value: 0x34"] - B --c--> D["D \n value: 0x12"] - B --d--> E["E \n value: 0x23"] + A["A \n Hash(B|C)"] -- v --> B["B \n Hash(B|E)"] + A --wz--> F["F \n value: 0x34"] + B --"x"--> D["D \n value: 0x12"] + B --y--> E["E \n value: 0x23"] - - @@ -202,13 +214,13 @@ flowchart TD - + - + - +
"ac" => 0x1234 "vx" => 0x12
"ad" => 0x1234 "vy" => 0x23
"be" => 0x1234 "wz" => 0x34
@@ -217,11 +229,11 @@ flowchart TD Notes: -more resources: +More resources: -- https://en.wikipedia.org/wiki/Merkle_tree -- https://en.wikipedia.org/wiki/Radix_tree -- https://en.wikipedia.org/wiki/Trie +- +- +- Namely: @@ -230,10 +242,12 @@ Namely: > Donald Knuth, pages 498-500 in Volume III of The Art of Computer Programming, calls these > "Patricia's trees", presumably after the acronym in the title of Morrison's paper: "PATRICIA - -> Practical Algorithm to Retrieve Information Coded in Alphanumeric". Today, Patricia tries are seen +> Practical Algorithm to Retrieve Information Coded in Alphanumeric". +> Today, Patricia tries are seen > as radix trees with radix equals 2, which means that each bit of the key is compared individually > and each node is a two-way (i.e., left versus right) branch. -> ---v + +---v ### Merklized @@ -243,10 +257,9 @@ Namely: ### Merklized -- Substrate does in fact use a key-value based database under the hood.. -- In order to store trie nodes, not direct storage keys! - -
+- Substrate does in fact use a key-value based database under the hood. +- But this KV based DB is used to store the trie nodes, not directly the storage keys. +
@@ -287,20 +300,29 @@ simplification. --- -## Trie Walking Example +## Traversing the Trie - We know the state-root at a given block `n`. -- assume this is a base-26, patricia trie. English alphabet is the key-scope. +- Assume this is a base-26, patricia trie. + English alphabet is the key-scope. - Let's see the steps needed to read `balances_alice` from the storage. ---v +Notes: +- We start with the state root node. +- Read its children. + ---v +Notes: +- We are interested in "balances_" so we read that node from database. +- Did you notice the mistake in the slide? "_" technically would not be allowed in base-26, so it really is base-27. + ---v @@ -313,11 +335,22 @@ simplification. +---v + +## Traversing the Trie + +// TODO(ank4n) fix link + + +Try inserting (and deleting) bunch of keys and see how you fill up the trie in +the [radix tree visualization](https://www.cs.usfca.edu/~galles/visualization/RadixTree.html). + --- ## Merklized: Proofs -- If alice only has this root, how can I prove to her how much balance she has? +Back to our question +> If alice only has this state root, how can she verify her balance is correct? ---v @@ -333,7 +366,8 @@ Receiver will hash the root node, and check it against a publicly known storage This differs slightly from how actual proof generation might work in the code. -In general, you have a tradeoff: send more data, but require less hashing on Alice, or opposite (this is what we call "compact proof"). +In general, you have a tradeoff: send more data, but require less hashing on Alice, or opposite (this is what we call " +compact proof"). ---v @@ -347,7 +381,7 @@ In general, you have a tradeoff: send more data, but require less hashing on Ali
-- Storage key (whatever you pass to `sp_io`) is the path on the trie. +- Storage key (`balances_alice`) is the path on the trie.
@@ -355,13 +389,14 @@ In general, you have a tradeoff: send more data, but require less hashing on Ali
- Storage key is arbitrary length. +
- Intermediary (branch) nodes could contain values. - - `:code` contains some value, `:code:more` can also contain value. + - `:code` contains some value, `:code:more` can also contain value.
@@ -384,64 +419,54 @@ it will be `O(LOG_n)`. --- -## Base 2, Base 16, Base-26? - -- Instead of alphabet, we use the base-16 representation of everything. - -> Base-16 (Patricia) Merkle Trie. - -- `System` -> `73797374656d` -- `:code` -> `3a636f646500` - ----v - -### Base 2, Base 16, Base-26? - - - -Tradeoff: "_IO count vs. Node size_" - - - -Between a light clint and a full node, which one cares more about which? +## Substrate Storage: The Updated Picture - + -Notes: +--- -TODO: update figure to represent node size. +## Large data nodes 🤔 -Light client cares about node size. When proof is being sent, there is no IO. +- Two common problems that merkle proofs have: + - If the one of the parent nodes has some large data. + - If you want to prove the deletion/non-existence of a leaf node. -First glance, the radix-8 seems better: you will typically have less DB access to reach a key. -For example, with binary, with 3 IO, we can reach only 8 items, but with radix-8 512. +---v -So why should not chose a very wide tree? because the wider you make the tree, the bigger each node -gets, because it has to store more hashes. At some point, this start to screw with both the proof -size and the cost of reading/writing/encoding/decoding all these nodes. + ---v -### Base 2, Base 16, Base-26? +## Large data nodes 🤔 - +New "trie format" 🌈: -Note: +- All data containing more than 32 bytes are replaced with their hash (pointer to the actual value). +- The (larger than 32 bytes) value itself stored in the database under this hash. -Here's a different way to represent it; the nodes are bigger on the base-16 trie. +```rust +struct RuntimeVersion { + ... + state_version: 0, +} +``` ----v + -### Base 2, Base 16, Base-26? +---v -- base-2: Small proofs, more nodes. -- base-8: Bigger proofs, less nodes. + + -✅ 16 has been benchmarked and studies years ago as a good middle-ground. +What is the ramification of this for full nodes, and light clients? Notes: -Anyone interested in blockchain and research stuff should look into this. +Both read and write have an extra step now, but proof are easier. + +Note from emeric: the green node is not really a "real" node, it is just `{ value: BIG_STUFF }` stored in the database. +I will skip this detail for the sake of simplicity. +One can assume that the green node is like any other node in the trie. --- @@ -453,7 +478,8 @@ Anyone interested in blockchain and research stuff should look into this. ### Unbalanced Tree -- Unbalanced tree means unbalanced performance. An attack vector, if done right. +- Unbalanced tree means unbalanced performance. + An attack vector, if done right. - More about this in FRAME storage, and how it is prevented there. Notes: @@ -465,59 +491,7 @@ Notes: --- -## WAIT A MINUTE... 🤔 - -- Two common scenarios that merkle proofs are kinda unfair: - - If the one of the parent nodes has some large data. - - If you want to prove the deletion/non-existence of a leaf node. - ----v - - - ----v - -## WAIT A MINUTE... 🤔 - -New "tie format" 🌈: - -- All data containing more than 32 bytes are replaced with their hash. -- The (larger than 32 bytes) value itself stored in the database under this hash. - -```rust -struct RuntimeVersion { - ... - state_version: 0, -} -``` - - - ----v - - - -What is the ramification of this for full nodes, and light clients? - -Notes: - -TODO: update figure. - -Both read and write have an extra step now, but proof are easier. - -Note from emeric: the green node is not really a "real" node, it is just `{ value: BIG_STUFF }` -stored in the database. I will skip this detail for the sake of simplicity. One can assume that the -green node is like any other node in the trie. - ---- - -## Substrate Storage: The Updated Picture - - - ---- - -## WAIT A MINUTE... 🤔 +## Trie Caching 🤔 - We rarely care about state root and all the trie shenanigans before the end of the block... @@ -527,10 +501,11 @@ green node is like any other node in the trie. Notes: -in other words, one should one care too much about updating a "trie" and all of its hashing details -while the block is still being executed? all of that can be delayed. +In other words, one should care too much about updating a "trie" and all of its hashing details while the block is still +being executed? +All of that can be delayed. ---- +---v ## Overlay @@ -542,10 +517,10 @@ while the block is still being executed? all of that can be delayed. ### Overlay - Almost identical semantic to your CPU cache: - - Once you read a value, it stays here, and can be re-read for cheap. - - Once you write a value, it will only be written here. - - It can be read for cheap. - - All writes are flushed at the end of the runtime api call. + - Once you read a value, it stays here, and can be re-read for cheap. + - Once you write a value, it will only be written here. + - It can be read for cheap. + - All writes are flushed at the end of the runtime api call. - No race conditions as runtime is single-threaded. ---v @@ -585,7 +560,7 @@ while the block is still being executed? all of that can be delayed. - + @@ -593,16 +568,16 @@ while the block is still being executed? all of that can be delayed. Notes: -- In your code, you often have an option to either pass stack variables around, or re-read code from - `sp-io`. Most often, this is a micro-optimization that won't matter too much, but in general you - should know that the former is more performant, as won't go the the host at all. +- In your code, you often have an option to either pass stack variables around, or re-read code from `sp-io`. + Most often, this is a micro-optimization that won't matter too much, but in general you should know that the former is + more performant, as won't go the the host at all. - A deletion is basically a write to `null`. ---v ### Overlay -- The overlay is also able to spawn child-overlays, know as "_storage layer_". +- The overlay is also able to spawn child-overlays, known as "_storage layer_". - Useful for having a _transactional_ block of code. ```rust @@ -668,15 +643,16 @@ with_storage_layer(|| { Notes: -NO! overlay works on the level on key-values, ot knows nothing of trie nodes, and to compute the -root we have to go to the trie layer and pull a whole lot of data back from the disk and build all -the nodes etc. etc. +NO! +The overlay works on the level on key-values, it knows nothing of trie nodes, and to compute the root we have to go to +the trie layer and pull a whole lot of data back from the disk and build all the nodes etc. ---v ### Overlay: More Caches -- There are more caches in the trie layer as well. But outside of the scope of this lecture. +- There are more caches in the trie layer as well. + But outside of the scope of this lecture. ```bash ./substrate --help | grep cache @@ -684,13 +660,17 @@ the nodes etc. etc. Notes: -https://www.youtube.com/embed/OoMPlJKUULY + --- ### Substrate Storage: Final Figure - +// TODO(ankan): Fix link + + +Notes: +Should be Runtime on the top.. ---v @@ -699,11 +679,11 @@ https://www.youtube.com/embed/OoMPlJKUULY There are multiple implementations of `Externalities`: - [`TestExternalities`](https://paritytech.github.io/substrate/master/sp_state_machine/struct.TestExternalities.html): - - `Overlay` - - `TrieDb` with `InMemoryBackend` + - `Overlay` + - `TrieDb` with `InMemoryBackend` - [`Ext`](https://paritytech.github.io/substrate/master/sp_state_machine/struct.Ext.html) (the real thing 🫡) - - `Overlay` - - `TrieDb` with a real database being the backend + - `Overlay` + - `TrieDb` with a real database being the backend ---v @@ -723,7 +703,7 @@ let x = sp_io::storage::get(b"foo"); ```rust // ✅ SomeExternalities.execute_with(|| { - let x = sp_io::storage::get(b"foo"); + let x = sp_io::storage::get(b"foo"); }); ``` @@ -799,6 +779,68 @@ Meaning, if another client wants to sync polkadot, it should know the details of --- +## Base 2, Base 16, Base-26? + +- Instead of alphabet, we use the base-16 representation of everything. + +> Base-16 (Patricia) Merkle Trie. + +- `System` -> `73797374656d` +- `:code` -> `3a636f646500` + +---v + +### Base 2, Base 16, Base-26? + + + + +Tradeoff: "_IO count vs. Node size_" + + + +Between a light client and a full node, who cares more about which? + + + +Notes: + +Light client cares about node size. +When proof is being sent, there is no IO. + +First glance, the radix-8 seems better: you will typically have less DB access to reach a key. +For example, with binary, with 3 IO, we can reach only 8 items, but with radix-8 512. + +So why should not choose a very wide tree? +Because the wider you make the tree, the bigger each node gets, because it has to store more hashes. +At some point, this start to screw with both the proof size and the cost of reading/writing/encoding/decoding all these +nodes. + +---v + +### Base 2, Base 16, Base-26? + + + +Note: + +Here's a different way to represent it; the nodes are bigger on the base-8 trie. + +---v + +### Base 2, Base 16, Base-26? + +- base-2: Small proofs, more nodes. +- base-26: Bigger proofs, less nodes. + +✅ 16 has been benchmarked and studies years ago as a good middle-ground. + +Notes: + +Anyone interested in blockchain and research stuff should look into this. + +--- + #### Lecture Summary/Recap: @@ -809,7 +851,7 @@ Meaning, if another client wants to sync polkadot, it should know the details of - Merklized storage, and proofs - Large nodes - Radix order consequences -- Unbalanced trie +- Unbalanced tree - State pruning @@ -826,37 +868,42 @@ Meaning, if another client wants to sync polkadot, it should know the details of > Check speaker notes (click "s" 😉) - + Notes: -- Shawn's deep dive: https://www.shawntabrizi.com/blog/substrate/substrate-storage-deep-dive/ +- Shawn's deep dive: -- Basti's talk on Trie caching: https://www.youtube.com/watch?v=OoMPlJKUULY +- Basti's talk on Trie caching: - About state version: - - https://github.com/paritytech/substrate/pull/9732 - - https://github.com/paritytech/substrate/discussions/11824 + - + - -- An "old but gold" read about trie in Ethereum: https://medium.com/shyft-network/understanding-trie-databases-in-ethereum-9f03d2c3325d +- An "old but gold" read about trie in + Ethereum: -- On optimizing substrate storage proofs: https://github.com/paritytech/substrate/issues/3782 -- Underlying trie library maintained by Parity: https://github.com/paritytech/trie +- On optimizing substrate storage proofs: +- Underlying trie library maintained by Parity: -- https://github.com/paritytech/trie/ +- -- https://spec.polkadot.network/chap-state#sect-state-storage +- -- https://research.polytope.technology/state-(machine)-proofs +- -- An interesting, but heretical idea: can the runtime of block N, access state of block N-1? HELL - NO. It might sound like a "but why nooooot" type of situation, but it breaks down all assumptions - about what a state transition is. The runtime is the state transition function. Recall the formula - of that, and then you will know why this is not allowed. +- An interesting, but heretical idea: can the runtime of block N, access state of block N-1? + HELL. + NO. + It might sound like a "but why nooooot" type of situation, but it breaks down all assumptions + about what a state transition is. + The runtime is the state transition function. + Recall the formula of that, and then you will know why this is not allowed. ### Post Lecture Feedback -Double check the narrative and example of the `BIG_STUFF` node. An example/exercise of some sort +Double check the narrative and example of the `BIG_STUFF` node. +An example/exercise of some sort would be great, where students call a bunch of `sp_io` functions, visualize the trie, and invoke -proof recorder, and see which pars of the trie is exactly part of the proof. +proof recorder, and see which parts of the trie is exactly part of the proof. diff --git a/syllabus/5-Substrate/9-SCALE_Slides.md b/syllabus/5-Substrate/9-SCALE_Slides.md index dc8128cb1..fbb333d3d 100644 --- a/syllabus/5-Substrate/9-SCALE_Slides.md +++ b/syllabus/5-Substrate/9-SCALE_Slides.md @@ -20,6 +20,10 @@ Simple Concatenated Aggregate Little-Endian SCALE is a light-weight format which allows encoding (and decoding) which makes it highly suitable for resource-constrained execution environments like blockchain runtimes and low-power, low-memory devices. +Notes: +- It is a encoding format used to communicate over the wire. Similar to json, protobuf. +- Extremely light weight, we will see how. + --- ### Little-Endian @@ -42,6 +46,11 @@ Wasm is a little endian system, which makes SCALE very performant. +Notes: +- Endianness is the order of bytes. +- Big Endian => Most significant byte at the smallest address. Similar to English. Generally used in network protocols. +- Little Endian => Least significant byte at the smallest address. + --- ### Why SCALE? Why not X? @@ -55,6 +64,11 @@ Wasm is a little endian system, which makes SCALE very performant. - Supports a copy-free decode for basic types on LE architectures. - It is about as thin and lightweight as can be. +Notes: +- MaxEncodedLen: Maximum encoded size to make some runtime guarantees about computation. +- TypeInfo: Used to generate metadata. +- Bijective exception later. + --- ### SCALE is NOT Self-Descriptive @@ -155,14 +169,14 @@ The order of bytes in the vector follow endianness, but the hex and binary repre ```rust fn main() { println!("{:b}", 69i8); - println!("{:02x?}", 69i8.to_le_bytes()); - println!("{:02x?}", 69i8.to_be_bytes()); + println!("{:02x?}", 69i8.to_be_bytes()); + println!("{:02x?}", 69i8.to_le_bytes()); println!("{:b}", 42u16); - println!("{:02x?}", 42u16.to_le_bytes()); - println!("{:02x?}", 42u16.to_be_bytes()); + println!("{:02x?}", 42u16.to_be_bytes()); + println!("{:02x?}", 42u16.to_le_bytes()); println!("{:b}", 16777215u32); - println!("{:02x?}", 16777215u32.to_le_bytes()); - println!("{:02x?}", 16777215u32.to_be_bytes()); + println!("{:02x?}", 16777215u32.to_be_bytes()); + println!("{:02x?}", 16777215u32.to_le_bytes()); } ``` @@ -176,11 +190,11 @@ fn main() { [45] [45] 101010 -[2a, 00] [00, 2a] +[2a, 00] 111111111111111111111111 -[ff, ff, ff, 00] [00, ff, ff, ff] +[ff, ff, ff, 00] ``` diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index 5fe6c4edd..2067b9e86 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -17,7 +17,7 @@ Many of these interactions land in a wasm blob. So what question you need to ask yourself there? which runtime blob. -almost all external communication happens over JSPN-RPC, so let's take a closer look. +almost all external communication happens over JSON-RPC, so let's take a closer look. --- @@ -54,6 +54,7 @@ almost all external communication happens over JSPN-RPC, so let's take a closer - Entirely transport agnostic. - Substrate based chains expose both `websocket` and `http` (or `wss` and `https`, if desired). +// TODO(ank4n): Is this correct? I think both are now same port. > with `--ws-port` and `--rpc-port`, 9944 and 9934 respectively. ---v @@ -120,7 +121,7 @@ wscat \ ### JSON-RPC: Safety -- Some PRC methods are unsafe 😱. +- Some RPC methods are unsafe 😱. ---v From b8ec109e680d2695afe9d04b15a5d2a0116eb3ee Mon Sep 17 00:00:00 2001 From: Ankan Date: Thu, 11 Jan 2024 19:50:28 +0530 Subject: [PATCH 03/15] fix images --- .../dev-4-8-qr-radix-tree-visualization.png | Bin 0 -> 6608 bytes .../dev-storage-externalities-full.svg | 320 ++++++++++++++++++ .../5-Substrate/3-Merklized-Storage_Slides.md | 4 +- 3 files changed, 322 insertions(+), 2 deletions(-) create mode 100644 assets/img/5-Substrate/dev-4-8-qr-radix-tree-visualization.png create mode 100644 assets/img/5-Substrate/dev-storage-externalities-full.svg diff --git a/assets/img/5-Substrate/dev-4-8-qr-radix-tree-visualization.png b/assets/img/5-Substrate/dev-4-8-qr-radix-tree-visualization.png new file mode 100644 index 0000000000000000000000000000000000000000..fd4bcfea2f6b06668d8c1b852e848889fef8d6d6 GIT binary patch literal 6608 zcmd5>dtB1j`^PTJY%9~QYgrbEm%?%@O+|}QiYQii$qMPBD{V;^k-UJlutlrXk0O-{ zWnNM=%XAhbl&j2aGa<87BrP>mL=(k8^n1{?)%Uyo`+NO*z53^z^Ev16oacRc9$A|S z9$%S!Z32VAzQTE8x58jXm4^SuUx1Og1=Sm2u!S5Pc7xxs45h3oYSB7q>gMnlBV+al zECg%XOG2m5NoJYPj&(12c^lzya@9jAMFsxNf-UHZC(MGu2wRPf7Wi0B+4Wt5>G8Tb z7}zG_|FeP8F*N#k;N6OfOpD5#7P;JameI=x1GWuKvG?m}^3CEVDvLob7nj9$(fN|9 z@`4(zLUU(SXt!-^_}t2{MP0Pwodx0|nd%@nMqZ%qFX}2`Xz3!AjR`x=Sw5Q}_e-4r z;x&IvRy?MBt_#RXr1*0NrK(yjH%sz$stpr%D8Ve@h6MQwKWN6|4hy{pyUcovvC(VW z{)8J(kY`-bLMt*oOy3>vGAQy_yxCD7s#oifwS0H~Z7IW;#}+)&j3}aOvNwZf|vbS^73jrp&1= zuW=RNGWTf*!hcSa&lZWrRS^Yb#QA|+zcZ>12$j9%HF3hHS5qEz-rHAjb^zHpxAM$< z+t8;{Xh&2z$4!nJcn5E9iQn{ab$lMla?0_2rM67Ad%mBAOW)lmnz;6x@wxtn;W8WM zVz(<9+~58_*7@Bk-pr%J&vH#ms{S6`mAAIqp;fwV?LpDA>QWsze#hA5(V*f{R+RSP z7uvqNjUJZG!;5kfqY5ZDh^!UDrf%pg#9fdg`a_{>58?*u&W)b$Q!16vlCI$T-o*q& zNl!#E-oYzokGIx2PsQ1HESR+KM@v`gEh`uH;}T&7s@ht?TCM2L5Rtj9(bS@w38pX$ z;*8czUYu3@lS#py(bS4$Fy?mFj)`>Z_xq4awM`DC#gd>c67#*1hIMptVc%Ur)osk< zhW|cozQgch%lMaim{WaA{&3JMw`}R8T?rVcd-?G5C>By`wl^TBct+vppsFh}TL?}eD*VO2uG zp_{$m?Bh&H02AeYf1@4+U*GW3zma_~o<2&U*c( zIhfJq=;@nc8@r!dP)kS7C%NY!@9+1G&1}4yHjH^ke8uCEQq=GT%CKU5EH^N|;D%K{ zNWJTRS*8KQ*u(P=7`T;Dwll}uvhaO%o0mK3*-{BkatXVF|nIRq}Kxe@snm=Ve3< zx+o1)H%*xOVei7=TbDBG5I6T~;)Hj9wWY`1NCDXxeq_^;Jymn{gjfUR-H!k_xeBQ4 zY_2KD=!yA+2j?^Yo^Lhfp$`Q|AEb$eN-l-d0C;d#)ccb$uuVsoP1#ktwc0^ZJ%p_NU254S$D%Bxp zP}LSvvyGAz3gKefs#0NvTBA%8@d8t=0l&`us62LCjHM|y0(h)9eh$u8v@Ude<+I)) ziM)J=mJz%q%hb&TPK?A2T}ZQrxN%3HlrpaXb`<{#W=GG5EQD>c19v@cm+F-k#O6Lv z;zr)w(f)u%&QWpH^c%M5WQ5+W%R%l^<}y1Mxh5M)>Ipy}Zu?%#z{#iaAxR+g!aghl zrwio-B&9q<9qvyOm@d?{XRzW zFk-P8(qifmUhB4v&3z0pmQXLMrWcvCUR^&iU2l$b%bvK24IL4thdR~*BaDFi(_f710Gp`irkzf4gPCat9*iR$Q{UX5R@Q0KFR zEI{cP{+#X?8(0A1fk3Io)xFd#;n4OH53OsICbpnLrq4~ZCvoWFf*NY~Wvpe~m)=93 zysBP?Z&|reuI|t2E>Q(i??lw&(u{VMIv_Ey+nAb=i;*NKARq@B+mq%-p8mbYlSk3I zRS++l@8(Q7g~HkKB`N4AJg=B}B{sX(k{XG8>JPap zsks+`_3bkl5J5v~vaOjUvjqZp5s359%awGI*{R-SH@TV~gJEHY_ox2j9>Py9mk%x{ zO!-=3zCL#_y(L(yBjCgqdEQOa$cL9ThtR+iOnO|(kT&7tm8t6P!%s@-eTx$gos{9e zQcPbe)%sBE_4K@Kaw0KTd2;UOV7Yz2H~W-&N>Z3Ux12L7B1(#@Y@|c?O%e{JeENCW z#|av|1_o3E{HikVG@P&XI`+M1p(3{V3TH2QEoq3X9mpw(LRPAo5OMmKq>dfv`&sE~ zeRJqojyt*FfWI+_4FRz#@*?HuDZdaTnxh;&XEaW6d0e4CLU=nLbAh}0i0z3hHE}NR zZIO_OC+rccJMq)TGY(Uo4<2Noh`X!KN6Y9f8I$nrC!R72Jms-R??NsTm6=`T_%`w!2uco^1jj$ex3ROOD2Ht+f3x> zR?;mL-GMOubXdK!~2FLTvpKE2MMxvWVfb~pQLC+{C* zzAqmmX(v{7UzV`|bgz60-T&IC8-zJ-j%UWpBK(mr)==-rC7sD4MtzxN5sDH9>q&;O5MGmw_su;d z3wLHemLTpR!it<*CHhbQ~zS@{>80teu*z0RrJxy8yR1hpNgG?eAptFnC1*C~k+eSIT2(*? z_9Y_7iWK9J&>bMn$EKmGSK)dxi`2F!H0U>?N|xv)!p8~9@VB)3jOj7?hSi<}pZ{L4pjPhZICp`gzg?|3oe z?EBq~akS(W-^a?t-yWUu+kp%~t3}Vr=eS|C7wg&T75DHXwj8@;TZyGOlwY!vu_GRCgCm0>s?;$8gDyZT6|=nfDRfn zUgxMFj#1uhOjC|2xgIgG?h5{y>s7|Js1w)DJBQi?dqeY;^QMS z1c~s0ksTM!EqmIq#v@=6FeyueayN*9lnW~wi^s-9-%1Q@Q{q>V=xlRznYZt0r+P|^ z!@~$7gc9Jy5qV}o%^nL43eG%D)aN3JJ!@?gSt;u7DBI+R^wg_ZP&V*WwdYN*Cyi#d zzEwrMo@ubH0a>PuLDujS-x`I!*Fv7p2+Zu_G-Q*@Wx5u+pD)N_>m<{|1#;jEWVvus zBnSYw#O*`}&$JViW_{+B9cI1$6S*)Lcoq6OxZ4qZVv}T}4{3_2JOv?QU(I z00l;`P5z2;n@eTY{4s(^tFNh-daG!fVW5%WdPZB9>Wo?Eh~vbeJP@DM=Y8A7is|1$ zsPL{5m9W`k#2Z96`_M9G@Wh@uP$Y)yJ0@EG5D-FOGZ-P)lY9^R=Hk?Y$( zKrIy8Y6p-nA8~)n7JVmlJM^G*C`yy=CXy+sK^(P#_-WyW07hI)gRd?ccQ1fTNiPt3 z2A)?X1Ina+n+C*5+SnwcXosW98$wAERk=Dk<5_{}q^%G3i{%BU1KK5nuXwnfd%e~ut$FY4sjV>oK2SdE2^zYLmN2ylJ75? zw*n6gTq)8W7)*oT>lWJDOhtrc3ZE_qF?3qFB?S29>RAfhe1#67*IxJX-J~d)1s*)} ze_Pe9c*4BZ@n5DAFvAII_yvRd7@h*u?bXXkLp7k5 z@X7k9pA!Hgd;0!_^Ud(20Jwqroa-tH6&HeT2i)Y}H#c~7Ls{`%wZm7DI1>eAEoqO* z)*Q$rr?`n+*3F*RfUwENXL)kElUs_Z3?J=qZ?5UQ7yms-*!)?isoLk;*> za16d2&H@4+hFhgjQlo+BcRMIWu2;q%?vpb=+1m4Ovz;HZF!z@;&du>lEaNm&3_D3aB z%2C}A11)KKvm|FUEYizpBs~ySFMrXScQ7i9qS1Z2LjGjr=2P{u;bXhMlB9s9$+I2{ zd4vI|G1=u+ye9_J4;=kFVy%1O)O*3>)(0UnbvDu~Uai6Tw!MSa--?YB8c+|O-Xi{t zJ+mj7`=AbS;s?OW20S(_uRxEk9PKnGFnKe?Kf*#&M>gnhNzR;Vo0)W#YUyove2Lp5 z|B9tiIKhRri73`uVUFsg8hiQ4c816B=xA_TR=~o}eQtJbb7$yEqcbj1IQ^L=v3f~o zY&mEq?Ae{awgZKmFYTy9)csJdrYCWYpKlNvEG{5ber6%VKe`9TE*oz?tWCJ_8>`^S zHv%}xJKckWO$X)fhL~lWryVr0)O10Ss}EX`%nnB3<P=|%7cCZ=oaiMA&lR}|zMvkis zqR7cyr^DfKitZ9yy&y9jfKD1bzG}Q(b~H2Tf2Lc?ZS|wuR{si!aBsNf-2tK(+ZPKX zo?8Km&fPmLT@Pbi=(hiIb|L5;<$3+jy+Aut>@NBi71Z%u@uOxK^nKPZH0Zun(5VIN z6dT4ev=JP%phD)bHd38tn_U5p;ua{)K)=d&Q}Q;yUoejkFNSQm$y^vvfEb4}53*uD zQuIV8I_Zrwoanmz?`=Ssu-CZW{e_?qQ!}I~Tj2BRmj!iL0Vq3+cj-bFKI(=xLjc4~ z$WUQ&umtAW7l_9B| z$q9Xbm|v1EUH|JuCvhc{mhn8}i(6EjHI7mYz5ty;4Zh@NLSMi9m*|I6D+)+MpP%XX z1`3T*vsaz!tNCK{sU^?HFo%sUcv!xzibDRFjXZpetm^(%D3`$}k@kT%9ck`mo>RkT z77vOdggTkevS($Db1G-f&wKLCpX=oZR87j~!;(&+5Dj#46_$_L_JQctesi+li*aEy zbb68e@VzaWHvmjp>BSY4a^f32Pt`jFubHD68Vbm|?FaJx&;5!t)FzoK!tBKLdaIxg zyD(yp6Y_Y$rjg5{uKXFtyU$rOM?7LM%qdB+S|tREgM|+n>x$dKO>NMYm`RV(`Y2q8 z0y?`#9CF&8$#(N^Za1LzbQ0`{ub5;OB!FLiX}_)wbgOm(eTuXl`MC`Cl0V@QZI}d% zJNcxVX*_+YM%{!+&cQpJ(L>8erw#J*-%ss92~k_LJjcATI}8p(AD5*O$z>D{b=0dG zGkhg>!-}{t*b{60$(V#m#{=0P&I78w{J9Q&>}~zoQTUt+KkI(|pRE|12DM;Q-V~6` R!Cz`%xQzttb + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md index 2c4196df6..69cc8822f 100644 --- a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md +++ b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md @@ -340,7 +340,7 @@ Notes: ## Traversing the Trie // TODO(ank4n) fix link - + Try inserting (and deleting) bunch of keys and see how you fill up the trie in the [radix tree visualization](https://www.cs.usfca.edu/~galles/visualization/RadixTree.html). @@ -868,7 +868,7 @@ Anyone interested in blockchain and research stuff should look into this. > Check speaker notes (click "s" 😉) - + Notes: From 28c45cbfd3ad2c495dfc8282d6fe92e66a9fd9c7 Mon Sep 17 00:00:00 2001 From: Ankan Date: Fri, 12 Jan 2024 13:13:50 +0530 Subject: [PATCH 04/15] updates to storage slides --- .../5-Substrate/dev-trie-backend-proof.svg | 2 +- .../5-Substrate/3-Merklized-Storage_Slides.md | 168 +++++++++--------- 2 files changed, 87 insertions(+), 83 deletions(-) diff --git a/assets/img/5-Substrate/dev-trie-backend-proof.svg b/assets/img/5-Substrate/dev-trie-backend-proof.svg index 5ad63fa43..c30f7d3b9 100644 --- a/assets/img/5-Substrate/dev-trie-backend-proof.svg +++ b/assets/img/5-Substrate/dev-trie-backend-proof.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file diff --git a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md index 69cc8822f..c7e5be511 100644 --- a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md +++ b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md @@ -11,13 +11,16 @@ duration: 60mins ----v +Notes: -#### What We Know So Far +- Runtime interacts with Client/Host using Host functions. +- sp_io::storage helps with saving runtime state in the client using these host functions. -**Externalities** +---v -> Environment providing host functions, namely storage ones: "`Externalities` Environment". +#### Externalities + +> Externalities: An environment in which the runtime can access host functions, namely storage ones. Notes: @@ -58,6 +61,7 @@ Notes: ``` Notes: + - TestExternalities mimic a client. --- @@ -83,6 +87,7 @@ Notes: Notes: + - "_Storage keys_" (whatever you pass to `sp_io::storage`) directly maps to "_database keys_". - Probably don't wanna introduce storage key and db key right now. Good time to hammer down what you mean by storage key and what you mean by database key. @@ -119,6 +124,7 @@ Alice is representing a light client, I represent a full node. > Substrate uses a base-16, (patricia) radix merkle tree. Notes: + - Find the code at [paritytech/trie](https://github.com/paritytech/trie). ---v @@ -131,7 +137,7 @@ Notes: %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart TD - A["A \n value: Hash(B|C)"] --> B["B \n value: Hash(B|E)"] + A["A \n value: Hash(B|C)"] --> B["B \n value: Hash(D|E)"] A --> C["C \n value: Hash(F) \n"] B --> D["D \n value: 0x12"] B --> E["E \n value: 0x23"] @@ -157,7 +163,7 @@ flowchart TD %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart TD - A["A \n value: Hash(B|C)"] -- v --> B["B \n value: Hash(B|E)"] + A["A \n value: Hash(B|C)"] -- v --> B["B \n value: Hash(D|E)"] A --w--> C["C \n Hash(F) \n"] B --"x"--> D["D \n value: 0x12"] B --y--> E["E \n value: 0x23"] @@ -200,7 +206,7 @@ Notes: %%{init: {'theme': 'dark', 'themeVariables': { 'darkMode': true }}}%% flowchart TD - A["A \n Hash(B|C)"] -- v --> B["B \n Hash(B|E)"] + A["A \n Hash(B|C)"] -- v --> B["B \n Hash(D|E)"] A --wz--> F["F \n value: 0x34"] B --"x"--> D["D \n value: 0x12"] B --y--> E["E \n value: 0x23"] @@ -283,9 +289,9 @@ Namely: Notes: -imagine: +- imagine: `sp_io::storage::get(b"ad")` +- We will traverse the path later. -sp_io::storage::get(b"ad") ---v @@ -303,8 +309,8 @@ simplification. ## Traversing the Trie - We know the state-root at a given block `n`. -- Assume this is a base-26, patricia trie. - English alphabet is the key-scope. +- Assume this is a base-27, patricia trie. + English alphabet along with '_' is the key-scope. - Let's see the steps needed to read `balances_alice` from the storage. ---v @@ -312,6 +318,7 @@ simplification. Notes: + - We start with the state root node. - Read its children. @@ -320,6 +327,7 @@ Notes: Notes: + - We are interested in "balances_" so we read that node from database. - Did you notice the mistake in the slide? "_" technically would not be allowed in base-26, so it really is base-27. @@ -337,9 +345,8 @@ Notes: ---v -## Traversing the Trie +## Q/A Break -// TODO(ank4n) fix link Try inserting (and deleting) bunch of keys and see how you fill up the trie in @@ -358,9 +365,10 @@ Back to our question Notes: -The important point is that for example the whole data under `_system` is not hidden away behind one hash. +Give 30 seconds to students to make sense of the image by themselves. -Dark blue are the proof, light blue's hashes are present. +// TODO(ankan) clarify +The important point is that for example the whole data under `_system` is not hidden away behind one hash. Receiver will hash the root node, and check it against a publicly known storage root. @@ -666,12 +674,8 @@ Notes: ### Substrate Storage: Final Figure -// TODO(ankan): Fix link -Notes: -Should be Runtime on the top.. - ---v ### Substrate Storage @@ -779,68 +783,6 @@ Meaning, if another client wants to sync polkadot, it should know the details of --- -## Base 2, Base 16, Base-26? - -- Instead of alphabet, we use the base-16 representation of everything. - -> Base-16 (Patricia) Merkle Trie. - -- `System` -> `73797374656d` -- `:code` -> `3a636f646500` - ----v - -### Base 2, Base 16, Base-26? - - - - -Tradeoff: "_IO count vs. Node size_" - - - -Between a light client and a full node, who cares more about which? - - - -Notes: - -Light client cares about node size. -When proof is being sent, there is no IO. - -First glance, the radix-8 seems better: you will typically have less DB access to reach a key. -For example, with binary, with 3 IO, we can reach only 8 items, but with radix-8 512. - -So why should not choose a very wide tree? -Because the wider you make the tree, the bigger each node gets, because it has to store more hashes. -At some point, this start to screw with both the proof size and the cost of reading/writing/encoding/decoding all these -nodes. - ----v - -### Base 2, Base 16, Base-26? - - - -Note: - -Here's a different way to represent it; the nodes are bigger on the base-8 trie. - ----v - -### Base 2, Base 16, Base-26? - -- base-2: Small proofs, more nodes. -- base-26: Bigger proofs, less nodes. - -✅ 16 has been benchmarked and studies years ago as a good middle-ground. - -Notes: - -Anyone interested in blockchain and research stuff should look into this. - ---- - #### Lecture Summary/Recap: @@ -866,7 +808,8 @@ Anyone interested in blockchain and research stuff should look into this. ## Additional Resources! 😋 -> Check speaker notes (click "s" 😉) +- Check speaker notes (click "s" 😉). +- Follows some additional content that is not covered. @@ -907,3 +850,64 @@ Double check the narrative and example of the `BIG_STUFF` node. An example/exercise of some sort would be great, where students call a bunch of `sp_io` functions, visualize the trie, and invoke proof recorder, and see which parts of the trie is exactly part of the proof. + +--- + +## Base 2, Base 16, Base-26? + +- Instead of alphabet, we use the base-16 representation of everything. + +> Base-16 (Patricia) Merkle Trie. + +- `System` -> `73797374656d` +- `:code` -> `3a636f646500` + +---v + +### Base 2, Base 16, Base-26? + + + + +Tradeoff: "_IO count vs. Node size_" + + + +Between a light client and a full node, who cares more about which? + + + +Notes: + +Light client cares about node size. +When proof is being sent, there is no IO. + +First glance, the radix-8 seems better: you will typically have less DB access to reach a key. +For example, with binary, with 3 IO, we can reach only 8 items, but with radix-8 512. + +So why should not choose a very wide tree? +Because the wider you make the tree, the bigger each node gets, because it has to store more hashes. +At some point, this start to screw with both the proof size and the cost of reading/writing/encoding/decoding all these +nodes. + +---v + +### Base 2, Base 16, Base-26? + + + +Note: + +Here's a different way to represent it; the nodes are bigger on the base-8 trie. + +---v + +### Base 2, Base 16, Base-26? + +- base-2: Small proofs, more nodes. +- base-26: Bigger proofs, less nodes. + +✅ 16 has been benchmarked and studies years ago as a good middle-ground. + +Notes: +Anyone interested in blockchain and research stuff should look into this. From 2fca7b224d4b4e12df0262fa1189dfc61b39f4fc Mon Sep 17 00:00:00 2001 From: Ankan Date: Fri, 12 Jan 2024 14:17:47 +0530 Subject: [PATCH 05/15] minor edits --- syllabus/5-Substrate/3-Merklized-Storage_Slides.md | 2 +- syllabus/5-Substrate/9-SCALE_Slides.md | 12 ++++++------ .../5-Substrate/9-Substrate-Interactions_Slides.md | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md index c7e5be511..56d83655e 100644 --- a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md +++ b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md @@ -367,7 +367,7 @@ Notes: Give 30 seconds to students to make sense of the image by themselves. -// TODO(ankan) clarify +// TODO(ank4n) clarify The important point is that for example the whole data under `_system` is not hidden away behind one hash. Receiver will hash the root node, and check it against a publicly known storage root. diff --git a/syllabus/5-Substrate/9-SCALE_Slides.md b/syllabus/5-Substrate/9-SCALE_Slides.md index fbb333d3d..3822055ab 100644 --- a/syllabus/5-Substrate/9-SCALE_Slides.md +++ b/syllabus/5-Substrate/9-SCALE_Slides.md @@ -169,14 +169,14 @@ The order of bytes in the vector follow endianness, but the hex and binary repre ```rust fn main() { println!("{:b}", 69i8); - println!("{:02x?}", 69i8.to_be_bytes()); - println!("{:02x?}", 69i8.to_le_bytes()); + println!("{:02x?}", 69i8.to_be_bytes()); + println!("{:02x?}", 69i8.to_le_bytes()); println!("{:b}", 42u16); - println!("{:02x?}", 42u16.to_be_bytes()); - println!("{:02x?}", 42u16.to_le_bytes()); + println!("{:02x?}", 42u16.to_be_bytes()); + println!("{:02x?}", 42u16.to_le_bytes()); println!("{:b}", 16777215u32); - println!("{:02x?}", 16777215u32.to_be_bytes()); - println!("{:02x?}", 16777215u32.to_le_bytes()); + println!("{:02x?}", 16777215u32.to_be_bytes()); + println!("{:02x?}", 16777215u32.to_le_bytes()); } ``` diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index 2067b9e86..75858a991 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -54,7 +54,7 @@ almost all external communication happens over JSON-RPC, so let's take a closer - Entirely transport agnostic. - Substrate based chains expose both `websocket` and `http` (or `wss` and `https`, if desired). -// TODO(ank4n): Is this correct? I think both are now same port. +// TODO(ank4n): Verify if this is correct. > with `--ws-port` and `--rpc-port`, 9944 and 9934 respectively. ---v From d9eda023ffd4636ed8301e7690b1c10142c6ed37 Mon Sep 17 00:00:00 2001 From: Ankan Date: Fri, 12 Jan 2024 14:20:45 +0530 Subject: [PATCH 06/15] fix formatting --- syllabus/5-Substrate/9-SCALE_Slides.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/syllabus/5-Substrate/9-SCALE_Slides.md b/syllabus/5-Substrate/9-SCALE_Slides.md index 3822055ab..30f373940 100644 --- a/syllabus/5-Substrate/9-SCALE_Slides.md +++ b/syllabus/5-Substrate/9-SCALE_Slides.md @@ -168,15 +168,15 @@ The order of bytes in the vector follow endianness, but the hex and binary repre ```rust fn main() { - println!("{:b}", 69i8); - println!("{:02x?}", 69i8.to_be_bytes()); - println!("{:02x?}", 69i8.to_le_bytes()); - println!("{:b}", 42u16); - println!("{:02x?}", 42u16.to_be_bytes()); - println!("{:02x?}", 42u16.to_le_bytes()); - println!("{:b}", 16777215u32); - println!("{:02x?}", 16777215u32.to_be_bytes()); - println!("{:02x?}", 16777215u32.to_le_bytes()); + println!("{:b}", 69i8); + println!("{:02x?}", 69i8.to_le_bytes()); + println!("{:02x?}", 69i8.to_be_bytes()); + println!("{:b}", 42u16); + println!("{:02x?}", 42u16.to_le_bytes()); + println!("{:02x?}", 42u16.to_be_bytes()); + println!("{:b}", 16777215u32); + println!("{:02x?}", 16777215u32.to_le_bytes()); + println!("{:02x?}", 16777215u32.to_be_bytes()); } ``` From 8d8fed1d1c9ef4efb83ab5f5e56aaad0ea497a78 Mon Sep 17 00:00:00 2001 From: Ankan Date: Fri, 12 Jan 2024 14:21:40 +0530 Subject: [PATCH 07/15] 4 space tab --- syllabus/5-Substrate/9-SCALE_Slides.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/syllabus/5-Substrate/9-SCALE_Slides.md b/syllabus/5-Substrate/9-SCALE_Slides.md index 30f373940..2ebd0374d 100644 --- a/syllabus/5-Substrate/9-SCALE_Slides.md +++ b/syllabus/5-Substrate/9-SCALE_Slides.md @@ -168,15 +168,15 @@ The order of bytes in the vector follow endianness, but the hex and binary repre ```rust fn main() { - println!("{:b}", 69i8); - println!("{:02x?}", 69i8.to_le_bytes()); - println!("{:02x?}", 69i8.to_be_bytes()); - println!("{:b}", 42u16); - println!("{:02x?}", 42u16.to_le_bytes()); - println!("{:02x?}", 42u16.to_be_bytes()); - println!("{:b}", 16777215u32); - println!("{:02x?}", 16777215u32.to_le_bytes()); - println!("{:02x?}", 16777215u32.to_be_bytes()); + println!("{:b}", 69i8); + println!("{:02x?}", 69i8.to_le_bytes()); + println!("{:02x?}", 69i8.to_be_bytes()); + println!("{:b}", 42u16); + println!("{:02x?}", 42u16.to_le_bytes()); + println!("{:02x?}", 42u16.to_be_bytes()); + println!("{:b}", 16777215u32); + println!("{:02x?}", 16777215u32.to_le_bytes()); + println!("{:02x?}", 16777215u32.to_be_bytes()); } ``` From 5623b7c948bd4357c2ae6d481e0d7a3d9cff99f3 Mon Sep 17 00:00:00 2001 From: Ankan Date: Fri, 12 Jan 2024 14:23:14 +0530 Subject: [PATCH 08/15] another try --- syllabus/5-Substrate/9-SCALE_Slides.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/syllabus/5-Substrate/9-SCALE_Slides.md b/syllabus/5-Substrate/9-SCALE_Slides.md index 2ebd0374d..ce038a9bd 100644 --- a/syllabus/5-Substrate/9-SCALE_Slides.md +++ b/syllabus/5-Substrate/9-SCALE_Slides.md @@ -168,15 +168,15 @@ The order of bytes in the vector follow endianness, but the hex and binary repre ```rust fn main() { - println!("{:b}", 69i8); - println!("{:02x?}", 69i8.to_le_bytes()); - println!("{:02x?}", 69i8.to_be_bytes()); - println!("{:b}", 42u16); - println!("{:02x?}", 42u16.to_le_bytes()); - println!("{:02x?}", 42u16.to_be_bytes()); - println!("{:b}", 16777215u32); - println!("{:02x?}", 16777215u32.to_le_bytes()); - println!("{:02x?}", 16777215u32.to_be_bytes()); + println!("{:b}", 69i8); + println!("{:02x?}", 69i8.to_le_bytes()); + println!("{:02x?}", 69i8.to_be_bytes()); + println!("{:b}", 42u16); + println!("{:02x?}", 42u16.to_le_bytes()); + println!("{:02x?}", 42u16.to_be_bytes()); + println!("{:b}", 16777215u32); + println!("{:02x?}", 16777215u32.to_le_bytes()); + println!("{:02x?}", 16777215u32.to_be_bytes()); } ``` From da2f8abf2a41a07dfe51c0fa21f33f66874bb9bd Mon Sep 17 00:00:00 2001 From: Ankan Date: Sun, 14 Jan 2024 10:54:42 +0530 Subject: [PATCH 09/15] update storage lecture --- .../5-Substrate/dev-trie-backend-proof.svg | 2 +- .../5-Substrate/3-Merklized-Storage_Slides.md | 165 +++++++++--------- .../9-Substrate-Interactions_Slides.md | 2 +- 3 files changed, 83 insertions(+), 86 deletions(-) diff --git a/assets/img/5-Substrate/dev-trie-backend-proof.svg b/assets/img/5-Substrate/dev-trie-backend-proof.svg index c30f7d3b9..07c985452 100644 --- a/assets/img/5-Substrate/dev-trie-backend-proof.svg +++ b/assets/img/5-Substrate/dev-trie-backend-proof.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file diff --git a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md index 56d83655e..897251871 100644 --- a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md +++ b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md @@ -1,6 +1,6 @@ --- title: Substrate Merklized Storage -duration: 60mins +duration: 90mins --- # Substrate Storage @@ -367,8 +367,7 @@ Notes: Give 30 seconds to students to make sense of the image by themselves. -// TODO(ank4n) clarify -The important point is that for example the whole data under `_system` is not hidden away behind one hash. +The important point is that for example the whole data under `_system` is hidden away behind one hash. Receiver will hash the root node, and check it against a publicly known storage root. @@ -381,7 +380,7 @@ compact proof"). ### Merklized: Proofs -- 🏆 Small proof size is a big win for light clients, _and_ **Polkadot**. +- 🏆 Small proof size is a big win for light clients. --- @@ -620,32 +619,6 @@ Notes: ### Overlay -- There is a limit to how many nested layers you can spawn -- It is not free, thus it is attack-able. - -```rust -with_storage_layer(|| { - let foo = sp_io::storage::read(b"foo"); - with_storage_layer(|| { - sp_io::storage::set(b"foo", b"foo"); - with_storage_layer(|| { - sp_io::storage::set(b"bar", foo); - with_storage_layer(|| { - sp_io::storage::set(b"foo", "damn"); - Err("damn") - }) - Ok("what") - }) - Err("the") - }); - Ok("hell") -}) -``` - ----v - -### Overlay - - What if I call `sp_io::storage::root()` in the middle of the block? - Can the overlay respond to this? @@ -770,16 +743,62 @@ Notes: --- -## Trie Format Matters! +## Base 2, Base 16, Base-26? -- Recall that in our "trie walking", we took the state root, and got the root node from the DB. -- The state root of any substrate-based chain, including Polkadot, is the hash of the "Trie Node". +- Instead of alphabet, we use the base-16 representation of everything. -> Trie format matters! and therefore it is part of [the polkadot spec](https://spec.polkadot.network). +> Base-16 (Patricia) Merkle Trie. + +- `System` -> `73797374656d` +- `:code` -> `3a636f646500` + +---v + +### Base 2, Base 16, Base-26? + + + + +Tradeoff: "_IO count vs. Node size_" + + + +Between a light client and a full node, who cares more about which? + + Notes: -Meaning, if another client wants to sync polkadot, it should know the details of the trie format. +Light client cares about node size. +When proof is being sent, there is no IO. + +First glance, the radix-8 seems better: you will typically have less DB access to reach a key. +For example, with binary, with 3 IO, we can reach only 8 items, but with radix-8 512. + +So why should not choose a very wide tree? +Because the wider you make the tree, the bigger each node gets, because it has to store more hashes. +At some point, this start to screw with both the proof size and the cost of reading/writing/encoding/decoding all these +nodes. + +---v + +### Base 2, Base 16, Base-26? + + + +Note: + +Here's a different way to represent it; the nodes are bigger on the base-8 trie. + +---v + +### Base 2, Base 16, Base-26? + +- base-2: Small proofs, more nodes. +- base-26: Bigger proofs, less nodes. + +Notes: +Anyone interested in blockchain and research stuff should look into this. --- @@ -853,61 +872,39 @@ proof recorder, and see which parts of the trie is exactly part of the proof. --- -## Base 2, Base 16, Base-26? - -- Instead of alphabet, we use the base-16 representation of everything. - -> Base-16 (Patricia) Merkle Trie. - -- `System` -> `73797374656d` -- `:code` -> `3a636f646500` - ----v +### Overlay -### Base 2, Base 16, Base-26? +- There is a limit to how many nested layers you can spawn +- It is not free, thus it is attack-able. - - +```rust +with_storage_layer(|| { + let foo = sp_io::storage::read(b"foo"); + with_storage_layer(|| { + sp_io::storage::set(b"foo", b"foo"); + with_storage_layer(|| { + sp_io::storage::set(b"bar", foo); + with_storage_layer(|| { + sp_io::storage::set(b"foo", "damn"); + Err("damn") + }) + Ok("what") + }) + Err("the") + }); + Ok("hell") +}) +``` -Tradeoff: "_IO count vs. Node size_" +--- - +## Trie Format Matters! -Between a light client and a full node, who cares more about which? +- Recall that in our "trie walking", we took the state root, and got the root node from the DB. +- The state root of any substrate-based chain, including Polkadot, is the hash of the "Trie Node". - +> Trie format matters! and therefore it is part of [the polkadot spec](https://spec.polkadot.network). Notes: -Light client cares about node size. -When proof is being sent, there is no IO. - -First glance, the radix-8 seems better: you will typically have less DB access to reach a key. -For example, with binary, with 3 IO, we can reach only 8 items, but with radix-8 512. - -So why should not choose a very wide tree? -Because the wider you make the tree, the bigger each node gets, because it has to store more hashes. -At some point, this start to screw with both the proof size and the cost of reading/writing/encoding/decoding all these -nodes. - ----v - -### Base 2, Base 16, Base-26? - - - -Note: - -Here's a different way to represent it; the nodes are bigger on the base-8 trie. - ----v - -### Base 2, Base 16, Base-26? - -- base-2: Small proofs, more nodes. -- base-26: Bigger proofs, less nodes. - -✅ 16 has been benchmarked and studies years ago as a good middle-ground. - -Notes: -Anyone interested in blockchain and research stuff should look into this. +Meaning, if another client wants to sync polkadot, it should know the details of the trie format. diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index 75858a991..dc653dd2f 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -54,7 +54,7 @@ almost all external communication happens over JSON-RPC, so let's take a closer - Entirely transport agnostic. - Substrate based chains expose both `websocket` and `http` (or `wss` and `https`, if desired). -// TODO(ank4n): Verify if this is correct. + > with `--ws-port` and `--rpc-port`, 9944 and 9934 respectively. ---v From 443fa9302c418a005d643804559f45b377f92100 Mon Sep 17 00:00:00 2001 From: Ankan Date: Sun, 14 Jan 2024 16:40:28 +0530 Subject: [PATCH 10/15] fix link --- syllabus/5-Substrate/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/syllabus/5-Substrate/README.md b/syllabus/5-Substrate/README.md index 8492c93f8..1f5abe36f 100644 --- a/syllabus/5-Substrate/README.md +++ b/syllabus/5-Substrate/README.md @@ -31,7 +31,7 @@ Ensure the `main` branch is write protected, by required a PR first`- no one sho #### Morning 1. [Introduction](./1-Intro-to-Substrate_Slides.md) (60m) -1. [Wasm Meta Protocol](./2-Wasm-Meta-Protocol-Slides) (90m) +1. [Wasm Meta Protocol](./2-Wasm-Meta-Protocol-Slides.md) (90m) 1. Activity: Finding Runtime APIs and Host Functions in Substrate From b812abfabb8316812175585f6964f0402275afed Mon Sep 17 00:00:00 2001 From: Ankan Date: Sun, 14 Jan 2024 19:06:56 +0530 Subject: [PATCH 11/15] update port --- syllabus/5-Substrate/9-Substrate-Interactions_Slides.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index dc653dd2f..d0661b4ff 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -54,8 +54,9 @@ almost all external communication happens over JSON-RPC, so let's take a closer - Entirely transport agnostic. - Substrate based chains expose both `websocket` and `http` (or `wss` and `https`, if desired). - -> with `--ws-port` and `--rpc-port`, 9944 and 9934 respectively. +Notes: +- You could change which port to run the ws or http server on by using the flags `--ws-port` and `--rpc-port` + respectively. By default, port 9944 is used. ---v From 7399f9fced4dd18c7df9fcb21d3ef535c1cf4b0b Mon Sep 17 00:00:00 2001 From: Ankan Date: Tue, 16 Jan 2024 07:27:09 +0800 Subject: [PATCH 12/15] pr comment fixes --- .../5-Substrate/3-Merklized-Storage_Slides.md | 28 +++++++++---------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md index 897251871..2eaa3f689 100644 --- a/syllabus/5-Substrate/3-Merklized-Storage_Slides.md +++ b/syllabus/5-Substrate/3-Merklized-Storage_Slides.md @@ -237,9 +237,9 @@ Notes: More resources: -- -- -- +- https://en.wikipedia.org/wiki/Merkle_tree +- https://en.wikipedia.org/wiki/Radix_tree +- https://en.wikipedia.org/wiki/Trie Namely: @@ -641,7 +641,7 @@ the trie layer and pull a whole lot of data back from the disk and build all the Notes: - +https://www.youtube.com/embed/OoMPlJKUULY --- @@ -834,26 +834,26 @@ Anyone interested in blockchain and research stuff should look into this. Notes: -- Shawn's deep dive: +- Shawn's deep dive: https://www.shawntabrizi.com/blog/substrate/substrate-storage-deep-dive/ -- Basti's talk on Trie caching: +- Basti's talk on Trie caching: https://www.youtube.com/watch?v=OoMPlJKUULY - About state version: - - - - + - https://github.com/paritytech/substrate/pull/9732 + - https://github.com/paritytech/substrate/discussions/11824 - An "old but gold" read about trie in - Ethereum: + Ethereum: https://medium.com/shyft-network/understanding-trie-databases-in-ethereum-9f03d2c3325d -- On optimizing substrate storage proofs: -- Underlying trie library maintained by Parity: +- On optimizing substrate storage proofs: https://github.com/paritytech/substrate/issues/3782 +- Underlying trie library maintained by Parity: https://github.com/paritytech/trie -- +- https://github.com/paritytech/trie/ -- +- https://spec.polkadot.network/chap-state#sect-state-storage -- +- https://research.polytope.technology/state-(machine)-proofs - An interesting, but heretical idea: can the runtime of block N, access state of block N-1? HELL. From dd338f5ea0c214ccf924abc1640bf4f52d0ea88d Mon Sep 17 00:00:00 2001 From: Ankan Date: Tue, 16 Jan 2024 07:33:22 +0800 Subject: [PATCH 13/15] updates to interaction slide --- .../9-Substrate-Interactions_Slides.md | 316 +++++++++++------- 1 file changed, 204 insertions(+), 112 deletions(-) diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index d0661b4ff..dbfd751fe 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -7,17 +7,45 @@ duration: 60 minutes --- +## Before we start + +- Clone substrate node template + +```sh +git clone https://github.com/substrate-developer-hub/substrate-node-template` +``` + +
+ +- Compile your node + +```sh +cargo build --release +``` + +--- + ## Interacting With a Substrate Blockchain - +> How does a user or an application interact with a blockchain? -Notes: +---v + +## Interacting With a Substrate Blockchain -Many of these interactions land in a wasm blob. +- Usually they connect to a public RPC server, i.e. a substrate node that exposes its RPC interface publicly. -So what question you need to ask yourself there? which runtime blob. +
+ +- Run your own node. -almost all external communication happens over JSON-RPC, so let's take a closer look. + + +---v + +## Interacting With a Substrate Blockchain + + --- @@ -34,7 +62,10 @@ almost all external communication happens over JSON-RPC, so let's take a closer { "jsonrpc": "2.0", "method": "subtract", - "params": { "minuend": 42, "subtrahend": 23 }, + "params": { + "minuend": 42, + "subtrahend": 23 + }, "id": 3 } ``` @@ -42,7 +73,11 @@ almost all external communication happens over JSON-RPC, so let's take a closer
```json -{ "jsonrpc": "2.0", "result": 19, "id": 3 } +{ + "jsonrpc": "2.0", + "result": 19, + "id": 3 +} ``` @@ -55,25 +90,34 @@ almost all external communication happens over JSON-RPC, so let's take a closer - Substrate based chains expose both `websocket` and `http` (or `wss` and `https`, if desired). Notes: -- You could change which port to run the ws or http server on by using the flags `--ws-port` and `--rpc-port` + +- You could choose which port to run the ws or http server on by using the flags `--ws-port` and `--rpc-port` respectively. By default, port 9944 is used. ---v ### JSON-RPC -- JSON-RPC methods are conventionally written as `scope_method` +The RPC methods that a substrate node exposes are scoped and has the pattern `"_"`. + +```sh + wscat \ + -c ws://localhost:9944 \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"rpc_methods" }' \ + | jq +``` + +---v - - e.g. `rpc_methods`, `state_call` +### JSON-RPC: Scopes -- ­ `author`: for submitting stuff to the chain. +- ­ `author`: for submitting extrinsic to the chain. - ­ `chain`: for retrieving information about the _blockchain_ data. - ­ `state`: for retrieving information about the _state_ data. - ­ `system`: information about the chain. - ­ `rpc`: information about the RPC endpoints. Notes: - recall: https://paritytech.github.io/substrate/master/sc_rpc_api/index.html @@ -81,54 +125,179 @@ https://paritytech.github.io/substrate/master/sc_rpc/index.html The full list can also be seen here: https://polkadot.js.org/docs/substrate/rpc/ +--- + +### Workshop: Intro + +- Transfer some tokens from Alice to Charlie. + +
+ +- We will cheat a bit and take help sometimes from [Polkadot.js app](https://polkadot.js.org/apps/#/explorer). + + + +Notes: + +- When we start up a dev chain, some well known accounts are already minted some balance at genesis. Alice and Charlie + are well known accounts. +- The parts we cheat is because we will need to know more about FRAME to be able to calculate some storage keys. + ---v -### JSON-RPC +### Workshop: Spin up your node + +- Check out cli docs + +```sh +./target/release/node-template --help +``` + +
+ +- Spin up your dev node. + +```sh +./target/release/node-template --chain=dev --force-authoring --alice --tmp +``` + + + +Notes: + +- What does each flag do? What other flag can you use? + +---v + +### Workshop: Check balance + +- Query current balance of Charlie. +- Storage key where Charlie's balance is stored -- Let's look at a few examples: +``` +0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9b0edae20838083f2cde1c4080db8cf8090b5ab205c6974c9ea841be688864633dc9ca8a357843eeacf2314649965fe22 +``` -- `system_name`, `system_chain`, `system_chainType`, `system_health`, `system_version`, `system_nodeRoles`, `rpc_methods`, `state_getRuntimeVersion`, `state_getMetadata` +- We will use `state_getStorage` to query balance. ```sh wscat \ - -c wss://kusama-rpc.polkadot.io \ - -x '{"jsonrpc":"2.0", "id": 42, "method":"rpc_methods" }' \ + -c ws://localhost:9944 \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"state_getStorage", "params": [""] }' \ | jq ``` +Notes: +You will learn how the storage key is calculated in FRAME based substrate chains in the FRAME module. + ---v -### JSON-RPC: Runtime Agnostic +### Workshop: Transfer some tokens -- Needless to say, RPC methods are runtime agnostic. Nothing in the above tells you if FRAME is - being used or not. -- Except... metadata, to some extent. +- Take PJS help to get the signed extrinsic. +- Use the following command to submit the extrinsic. ----v +```sh +wscat \ + -c ws://localhost:9944 \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"author_submitExtrinsic", "params": [""] }' \ + | jq +``` -### JSON-RPC: Runtime API +
-- While agnostic, many RPC calls land in a runtime API. -- ­ RPC Endpoints have an `at: Option`, runtime APIs do too, what a coincidence! 🌈 - - ­ Recall the scope `state`? +- Check balance now. +- What happens if you execute the same payload again? + + + +Notes: +Let students do the second part themselves. ---v -### JSON-RPC: Extending +### Workshop: Metadata + +- Substrate exposes type information using metadata. + +```sh +wscat \ + -c ws://localhost:9944 \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"state_getMetadata" }' \ + | jq +``` + +
+ +- This itself is Scale Encoded. See [frame-metadata](https://github.com/paritytech/frame-metadata). +- Derive type of Balance using this metadata. + + + +Notes: -- The runtime can extend more custom RPC methods, but the new trend is to move toward using `state_call`. +- Metadata help users and apps understand the type information. Remember with SCALE, type information is lost. +- [Metadata](https://hackmd.io/@ak0n/rJUhmXmK6) with most details stripped off. +- Read more about + metadata: https://docs.substrate.io/build/application-development/#exposing-runtime-information-as-metadata. +- Can also use https://www.shawntabrizi.com/substrate-js-utilities/codec/ to decode balance with the following custom + type + +```json +{ + "Balance": { + "nonce": "u32", + "consumers": "u32", + "providers": "u32", + "sufficients": "u32", + "data": { + "free": "u128", + "reserved": "u128", + "frozen": "u128", + "flags": "u128" + } + } +} +``` ---v -### JSON-RPC: Safety +### Workshop: Decoding balance + +- Use https://www.shawntabrizi.com/substrate-js-utilities/codec/ to decode balance. +- Use the following balance struct -- Some RPC methods are unsafe 😱. +```json +{ + "Balance": { + "nonce": "u32", + "consumers": "u32", + "providers": "u32", + "sufficients": "u32", + "data": { + "free": "u128", + "reserved": "u128", + "frozen": "u128", + "flags": "u128" + } + } +} +``` ---v -### JSON-RPC: Resilience +### Workshop: Versions + +- Find runtime version of the polkadot and westend chain `state_getRuntimeVersion`. +- Find node version of the polkadot and westend chain `system_version`. +- Change RPC provider and see if any of the above value changes? + +Notes: -RPC-Server vs. Light Client +- `wscat -c wss://polkadot-rpc.dwellir.com -x '{"jsonrpc":"2.0", "id":1, "method":"state_getRuntimeVersion"}' | jq`. +- `wscat -c wss://polkadot-rpc.dwellir.com -x '{"jsonrpc":"2.0", "id":1, "method":"system_version"}' | jq`. +- For runtime version, you read `specVersion` : `1,005,000` as `1.5.0`. +- Polkadot Telemetry: https://telemetry.polkadot.io/. --- @@ -137,7 +306,6 @@ RPC-Server vs. Light Client - On top of `SCALE` and `JSON-RPC`, a large array of libraries have been built. - ­ `PJS-API` / `PJS-APPS` -- ­ `capi` - ­ `subxt` - ­ Any many more! @@ -156,7 +324,7 @@ In Kusama: - Find the genesis hash.. - Number of extrinsics at block 10,000,000. - The block number is stored under `twox128("System") ++ twox128("Number")`. - - Find it now, and at block 10,000,000. + - Find it now, and at block 10,000,000.
@@ -185,84 +353,6 @@ Notice that this number that we get back is the little endian (SCALE) encoded va --- -## Polkadot JS API - -A brief introduction. - -Excellent tutorial at: http://polkadot.js.org/docs - ----v - -## Polkadot JS API - - - ----v - -### PJS: Overview - -- `api.registry` -- `api.rpc` - ----v - -### PJS: Overview - -Almost everything else basically builds on top of `api.rpc`. - -- `api.tx` -- `api.query` -- `api.consts` -- `api.derive` - -Please revise this while you learn FRAME, and they will make perfect sense! - ----v - -### PJS: Workshop 🧑‍💻 - -Notes: - -```ts - -import { ApiPromise, WsProvider } from "@polkadot/api"; -const provider = new WsProvider("wss://rpc.polkadot.io"); -const api = await ApiPromise.create({ provider }); -api.stats; -api.isConnected; - // where doe this come from? -api.runtimeVersion; -// where does this come from? -api.registry.chainDecimals; -api.registry.chainTokens; -api.registry.chainSS58; -// where does this come from? -api.registry.metadata; -api.registry.metadata.pallets.map(p => p.toHuman()); -api.registry.createType(); -api.rpc.chain.getBlock() -api.rpc.system.health() -await api.rpc.system.version() -await api.rpc.state.getRuntimeVersion() -await api.rpc.state.getPairs("0x") -await api.rpc.state.getKeysPaged("0x", 100) -await api.rpc.state.getStorage() -https://polkadot.js.org/docs/substrate/rpc#getstoragekey-storagekey-at-blockhash-storagedata -await api.rpc.state.getStorageSize("0x3A636F6465"), -``` - -A few random other things: - -```ts -api.createType("Balance", new Uint8Array([1, 2, 3, 4])); - -import { blake2AsHex, xxHashAsHex } from "@polkadot/util-crypto"; -blake2AsHex("Foo"); -xxHashAsHex("Foo"); -``` - ---- - ## `subxt` - Something analogous to `PJS` for Rust. @@ -270,6 +360,8 @@ xxHashAsHex("Foo"); it statically. - ..It might need manual updates when the code, and therefore the metadata changes. +Notes: +Listen to James Wilson introducing subxt: https://www.youtube.com/watch?v=aFk6We_Ke1I --- ## Additional Resources! 😋 From e5ee3fabc1db394601c0ad6514514ad3161da69a Mon Sep 17 00:00:00 2001 From: Ankan Date: Wed, 17 Jan 2024 03:12:12 +0800 Subject: [PATCH 14/15] update to interaction slides --- .../9-Substrate-Interactions_Slides.md | 185 ++++++++++-------- 1 file changed, 99 insertions(+), 86 deletions(-) diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index dbfd751fe..1a6234672 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -9,10 +9,17 @@ duration: 60 minutes ## Before we start -- Clone substrate node template +Find all the commands that will be used in this workshop: +[hackmd.io/@ak0n/hk24-substrate-interaction](https://hackmd.io/@ak0n/hk24-substrate-interaction) + +--- + +## Before we start + +- Clone polkadot-sdk ```sh -git clone https://github.com/substrate-developer-hub/substrate-node-template` +git clone https://github.com/paritytech/polkadot-sdk.git ```
@@ -20,7 +27,7 @@ git clone https://github.com/substrate-developer-hub/substrate-node-template` - Compile your node ```sh -cargo build --release +cargo build --release -p minimal-node ``` --- @@ -29,6 +36,10 @@ cargo build --release > How does a user or an application interact with a blockchain? +Notes: + +- Wait for 1 answer from students or at least for 10 seconds. + ---v ## Interacting With a Substrate Blockchain @@ -37,7 +48,7 @@ cargo build --release
-- Run your own node. +- Run their own node. @@ -123,13 +134,15 @@ recall: https://paritytech.github.io/substrate/master/sc_rpc_api/index.html https://paritytech.github.io/substrate/master/sc_rpc/index.html -The full list can also be seen here: https://polkadot.js.org/docs/substrate/rpc/ +- The full list can also be seen here: https://polkadot.js.org/docs/substrate/rpc/ +- Specs: https://paritytech.github.io/json-rpc-interface-spec/introduction.html +- Upcoming changes to JSON-RPC api: https://forum.polkadot.network/t/new-json-rpc-api-mega-q-a/3048 --- ### Workshop: Intro -- Transfer some tokens from Alice to Charlie. +- Transfer tokens from one account to another.
@@ -139,8 +152,8 @@ The full list can also be seen here: https://polkadot.js.org/docs/substrate/rpc/ Notes: -- When we start up a dev chain, some well known accounts are already minted some balance at genesis. Alice and Charlie - are well known accounts. +- When we start up a dev chain, some well known accounts are already minted some balance at genesis. We will use Alice + and Bob which are well known accounts. - The parts we cheat is because we will need to know more about FRAME to be able to calculate some storage keys. ---v @@ -150,7 +163,7 @@ Notes: - Check out cli docs ```sh -./target/release/node-template --help +./target/release/minimal-node --help ```
@@ -158,27 +171,20 @@ Notes: - Spin up your dev node. ```sh -./target/release/node-template --chain=dev --force-authoring --alice --tmp +./target/release/minimal-node --chain=dev --tmp ``` Notes: -- What does each flag do? What other flag can you use? +- What does --chain=dev and --tmp do? What other flag can you use? ---v ### Workshop: Check balance -- Query current balance of Charlie. -- Storage key where Charlie's balance is stored - -``` -0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9b0edae20838083f2cde1c4080db8cf8090b5ab205c6974c9ea841be688864633dc9ca8a357843eeacf2314649965fe22 -``` - -- We will use `state_getStorage` to query balance. +- Query current balance of Alice and Bob. ```sh wscat \ @@ -188,96 +194,98 @@ wscat \ ``` Notes: -You will learn how the storage key is calculated in FRAME based substrate chains in the FRAME module. + +- You will learn how the storage key is calculated in FRAME based substrate chains in the FRAME module. +- What do you get? ---v -### Workshop: Transfer some tokens +### Workshop: Metadata -- Take PJS help to get the signed extrinsic. -- Use the following command to submit the extrinsic. +- Recall type information is lost in SCALE encoded data. +- Substrate exposes type information using metadata. ```sh wscat \ -c ws://localhost:9944 \ - -x '{"jsonrpc":"2.0", "id": 42, "method":"author_submitExtrinsic", "params": [""] }' \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"state_getMetadata" }' \ | jq ```
-- Check balance now. -- What happens if you execute the same payload again? +- This itself is Scale Encoded. See [frame-metadata](https://github.com/paritytech/frame-metadata). +- Derive type of Balance using this metadata. Notes: -Let students do the second part themselves. + +- Use PJS app to get frame-metadata: Developer > RPC Calls > state > getMetadata. +- [Metadata](https://hackmd.io/@ak0n/rJUhmXmK6) with most details not relevant stripped off. +- Read more about + metadata: https://docs.substrate.io/build/application-development/#exposing-runtime-information-as-metadata. ---v -### Workshop: Metadata +### Workshop: Transfer some tokens -- Substrate exposes type information using metadata. +- Take PJS help to get the signed extrinsic. +- Use the following command to submit the extrinsic. ```sh wscat \ -c ws://localhost:9944 \ - -x '{"jsonrpc":"2.0", "id": 42, "method":"state_getMetadata" }' \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"author_submitExtrinsic", "params": [""] }' \ | jq ```
-- This itself is Scale Encoded. See [frame-metadata](https://github.com/paritytech/frame-metadata). -- Derive type of Balance using this metadata. +- Check balance again for both accounts. +- What happens to nonce of Alice? Notes: -- Metadata help users and apps understand the type information. Remember with SCALE, type information is lost. -- [Metadata](https://hackmd.io/@ak0n/rJUhmXmK6) with most details stripped off. -- Read more about - metadata: https://docs.substrate.io/build/application-development/#exposing-runtime-information-as-metadata. -- Can also use https://www.shawntabrizi.com/substrate-js-utilities/codec/ to decode balance with the following custom - type +- Students will learn how to build the signed extrinsic themselves in their assignment. +- Let students do the second part themselves. + +---v + +### Workshop: Decoding balance + +- Use https://www.shawntabrizi.com/substrate-js-utilities/codec/ to decode balance. +- Use the following balance struct ```json { - "Balance": { + "info": { "nonce": "u32", - "consumers": "u32", - "providers": "u32", - "sufficients": "u32", - "data": { - "free": "u128", - "reserved": "u128", - "frozen": "u128", - "flags": "u128" + "ignore": "(u32, u32, u32)", + "balance": { + "free": "u64", + "ignore": "(u64, u64, u128)" } } } ``` ----v - -### Workshop: Decoding balance - -- Use https://www.shawntabrizi.com/substrate-js-utilities/codec/ to decode balance. -- Use the following balance struct +Notes: +The actual type is: ```json { - "Balance": { + "info": { "nonce": "u32", - "consumers": "u32", + "ignore": "u32", "providers": "u32", "sufficients": "u32", - "data": { - "free": "u128", - "reserved": "u128", - "frozen": "u128", + "balance": { + "free": "u64", + "reserved": "u64", + "frozen": "u64", "flags": "u128" } } @@ -292,12 +300,17 @@ Notes: - Find node version of the polkadot and westend chain `system_version`. - Change RPC provider and see if any of the above value changes? +
+ +- For runtime version, you read `specVersion` : `1,005,000` as `1.5.0`. + + + Notes: - `wscat -c wss://polkadot-rpc.dwellir.com -x '{"jsonrpc":"2.0", "id":1, "method":"state_getRuntimeVersion"}' | jq`. - `wscat -c wss://polkadot-rpc.dwellir.com -x '{"jsonrpc":"2.0", "id":1, "method":"system_version"}' | jq`. -- For runtime version, you read `specVersion` : `1,005,000` as `1.5.0`. -- Polkadot Telemetry: https://telemetry.polkadot.io/. +- Show polkadot telemetry: https://telemetry.polkadot.io/. --- @@ -317,6 +330,29 @@ more here: https://project-awesome.org/substrate-developer-hub/awesome-substrate --- +## `subxt` + +- Something analogous to `PJS` for Rust. +- The real magic is that it generates the types by fetching the metadata at compile time, or linking + it statically. +- ..It might need manual updates when the code, and therefore the metadata changes. + +Notes: +Listen to James Wilson introducing subxt: https://www.youtube.com/watch?v=aFk6We_Ke1I +--- + +## Additional Resources! 😋 + +> Check speaker notes (click "s" 😉) + +Notes: + +- see "Client Libraries" here: https://project-awesome.org/substrate-developer-hub/awesome-substrate +- https://paritytech.github.io/json-rpc-interface-spec/introduction.html +- Full subxt guide: https://docs.rs/subxt/latest/subxt/book/index.html + +--- + ### JSON-RPC: Mini Activity In Kusama: @@ -324,7 +360,7 @@ In Kusama: - Find the genesis hash.. - Number of extrinsics at block 10,000,000. - The block number is stored under `twox128("System") ++ twox128("Number")`. - - Find it now, and at block 10,000,000. +- Find it now, and at block 10,000,000.
@@ -349,27 +385,4 @@ wscat -c wss://kusama-rpc.polkadot.io -x '{"jsonrpc":"2.0", "id":72, "method":"s wscat -c wss://kusama-rpc.polkadot.io -x '{"jsonrpc":"2.0", "id":72, "method":"state_getStorage", "params": ["0x26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac", "0xdcbaa224ab080f2fbf3dfc85f3387ab21019355c392d79a143d7e50afba3c6e9"] }' | jq ``` -Notice that this number that we get back is the little endian (SCALE) encoded value that we passed in at first. - ---- - -## `subxt` - -- Something analogous to `PJS` for Rust. -- The real magic is that it generates the types by fetching the metadata at compile time, or linking - it statically. -- ..It might need manual updates when the code, and therefore the metadata changes. - -Notes: -Listen to James Wilson introducing subxt: https://www.youtube.com/watch?v=aFk6We_Ke1I ---- - -## Additional Resources! 😋 - -> Check speaker notes (click "s" 😉) - -Notes: - -- see "Client Libraries" here: https://project-awesome.org/substrate-developer-hub/awesome-substrate -- https://paritytech.github.io/json-rpc-interface-spec/introduction.html -- Full subxt guide: https://docs.rs/subxt/latest/subxt/book/index.html +Notice that this number that we get back is the little endian (SCALE) encoded value that we passed in at first. \ No newline at end of file From 354764d5aa42afabb909f69f08e034e506b2ae46 Mon Sep 17 00:00:00 2001 From: Ankan Date: Wed, 17 Jan 2024 03:39:33 +0800 Subject: [PATCH 15/15] updates to interaction slide --- .../9-Substrate-Interactions_Slides.md | 81 +++++++++---------- 1 file changed, 36 insertions(+), 45 deletions(-) diff --git a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md index 1a6234672..fb9a915f1 100644 --- a/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md +++ b/syllabus/5-Substrate/9-Substrate-Interactions_Slides.md @@ -10,7 +10,7 @@ duration: 60 minutes ## Before we start Find all the commands that will be used in this workshop: -[hackmd.io/@ak0n/hk24-substrate-interaction](https://hackmd.io/@ak0n/hk24-substrate-interaction) +[tinyurl.com/hk24-substrate](https://hackmd.io/@ak0n/hk24-substrate-interaction) --- @@ -142,7 +142,7 @@ https://paritytech.github.io/substrate/master/sc_rpc/index.html ### Workshop: Intro -- Transfer tokens from one account to another. +- Transfer tokens from Alice to Bob.
@@ -161,7 +161,6 @@ Notes: ### Workshop: Spin up your node - Check out cli docs - ```sh ./target/release/minimal-node --help ``` @@ -169,7 +168,6 @@ Notes:
- Spin up your dev node. - ```sh ./target/release/minimal-node --chain=dev --tmp ``` @@ -215,7 +213,7 @@ wscat \
- This itself is Scale Encoded. See [frame-metadata](https://github.com/paritytech/frame-metadata). -- Derive type of Balance using this metadata. +- Derive type of AccountInfo using this metadata. @@ -228,36 +226,10 @@ Notes: ---v -### Workshop: Transfer some tokens - -- Take PJS help to get the signed extrinsic. -- Use the following command to submit the extrinsic. - -```sh -wscat \ - -c ws://localhost:9944 \ - -x '{"jsonrpc":"2.0", "id": 42, "method":"author_submitExtrinsic", "params": [""] }' \ - | jq -``` - -
- -- Check balance again for both accounts. -- What happens to nonce of Alice? - - - -Notes: - -- Students will learn how to build the signed extrinsic themselves in their assignment. -- Let students do the second part themselves. - ----v - ### Workshop: Decoding balance -- Use https://www.shawntabrizi.com/substrate-js-utilities/codec/ to decode balance. -- Use the following balance struct +- Use [scale decoder](https://www.shawntabrizi.com/substrate-js-utilities/codec/) to decode balance. +- Use the following type information for AccountInfo. ```json { @@ -294,6 +266,26 @@ The actual type is: ---v +### Workshop: Transfer some tokens + +- ­ Take PJS help to get the signed extrinsic. +- ­ Use the following command to submit the extrinsic. +```sh +wscat \ + -c ws://localhost:9944 \ + -x '{"jsonrpc":"2.0", "id": 42, "method":"author_submitExtrinsic", "params": [""] }' \ + | jq +``` +- ­ Check balance again for both accounts. +- ­ What happens to nonce of Alice? + +Notes: + +- Students will learn how to build the signed extrinsic themselves in their assignment. +- Let students do the second part themselves. + +---v + ### Workshop: Versions - Find runtime version of the polkadot and westend chain `state_getRuntimeVersion`. @@ -327,18 +319,8 @@ Notes: https://github.com/JFJun/go-substrate-rpc-client https://github.com/polkascan/py-substrate-interface more here: https://project-awesome.org/substrate-developer-hub/awesome-substrate - ---- - -## `subxt` - -- Something analogous to `PJS` for Rust. -- The real magic is that it generates the types by fetching the metadata at compile time, or linking - it statically. -- ..It might need manual updates when the code, and therefore the metadata changes. - -Notes: Listen to James Wilson introducing subxt: https://www.youtube.com/watch?v=aFk6We_Ke1I + --- ## Additional Resources! 😋 @@ -385,4 +367,13 @@ wscat -c wss://kusama-rpc.polkadot.io -x '{"jsonrpc":"2.0", "id":72, "method":"s wscat -c wss://kusama-rpc.polkadot.io -x '{"jsonrpc":"2.0", "id":72, "method":"state_getStorage", "params": ["0x26aa394eea5630e07c48ae0c9558cef702a5c1b19ab7a04f536c519aca4983ac", "0xdcbaa224ab080f2fbf3dfc85f3387ab21019355c392d79a143d7e50afba3c6e9"] }' | jq ``` -Notice that this number that we get back is the little endian (SCALE) encoded value that we passed in at first. \ No newline at end of file +Notice that this number that we get back is the little endian (SCALE) encoded value that we passed in at first. + +--- + +## `subxt` + +- Something analogous to `PJS api` for Rust. +- The real magic is that it generates the types by fetching the metadata at compile time, or linking + it statically. +- ..It might need manual updates when the code, and therefore the metadata changes.