diff --git a/assets/img/5-Substrate/dev-4-1-substrate-meta-version.svg b/assets/img/5-Substrate/dev-4-1-substrate-meta-version.svg index dee986dc7..b38cab1f9 100644 --- a/assets/img/5-Substrate/dev-4-1-substrate-meta-version.svg +++ b/assets/img/5-Substrate/dev-4-1-substrate-meta-version.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file diff --git a/assets/styles/PBA-theme.css b/assets/styles/PBA-theme.css index 015d9942e..4f94db0aa 100644 --- a/assets/styles/PBA-theme.css +++ b/assets/styles/PBA-theme.css @@ -160,6 +160,11 @@ --r-hub-pink: #E6007A; --r-hub-lime: #D3FF33; --r-hub-green: #56F39A; + + /* Colors used in the Polkadot-SDK Lectures */ + --substrate-green: #25cc85ff; + --substrate-runtime: #bb8ba6; + --substrate-host: #8d3aed; } /* box sizing; diff --git a/syllabus/6-Polkadot-SDK/Polkadot-SDK-slides.md b/syllabus/6-Polkadot-SDK/Polkadot-SDK-slides.md index 2599ea04f..9b67b2dec 100644 --- a/syllabus/6-Polkadot-SDK/Polkadot-SDK-slides.md +++ b/syllabus/6-Polkadot-SDK/Polkadot-SDK-slides.md @@ -60,7 +60,9 @@ The holy grain of what ETH 2.0 was meant to be. -- Have we compromised security in any way in this model? +Note: + +Have we compromised security in any way in this model? ---v @@ -161,8 +163,8 @@ All proudly™️ built with Substrate. Substrate 🤝 FRAME -Substrate: Un-opinionated primitives for building blockchains based on a WASM meta-protocol. -FRAME: An opinionated way to build that WASM protocol/runtime/STF. +- Substrate: Un-opinionated primitives for building blockchains based on a WASM meta-protocol. +- FRAME: An opinionated way to build that WASM protocol/runtime/STF. Note: @@ -205,7 +207,7 @@ but it can 100% ensure that the parachain will do exactly as stated in its WASM ### Interoperability -Polkadot's services to developers: +Polkadot-SDK's services to developers: - Transport protocols for parachains/cores to send payloads to one another. @@ -213,11 +215,16 @@ Polkadot's services to developers: - As a developer you won't deal with these! 😮‍💨 - Language to compose messages and programs and send them over. - XCM + - Shared ideas between sovereign consensus systems. + - Asynchronous. Note: -Consensus System: Moreover, the sender and recipient of these messages can be contracts, accounts, or any other -abstract entities within chains, not just the chain itself. +On "XCMP, HRMP, VMP, DMP": It is similar to building a website. You know HTTP and TCP and such +exist, and you use them via an interface, but you don't need to know them by heart. + +Consensus System: Moreover, the sender and recipient of these messages can be contracts, accounts, or any +other abstract entities within chains, not just the chain itself. Async and Sync: Disjoint consensus systems are in principle[^1] asynchronous by nature. This is an important concept to keep in mind. diff --git a/syllabus/6-Polkadot-SDK/Substrate/1-Intro-to-Substrate-slides.md b/syllabus/6-Polkadot-SDK/Substrate/1-Intro-to-Substrate-slides.md index 7e7486d38..e4cd2bd06 100644 --- a/syllabus/6-Polkadot-SDK/Substrate/1-Intro-to-Substrate-slides.md +++ b/syllabus/6-Polkadot-SDK/Substrate/1-Intro-to-Substrate-slides.md @@ -61,6 +61,28 @@ this when you learn about Substrate's role in the Polkadot ecosystem. Figure: https://www.youtube.com/watch?v=0IoUZdDi5Is +---v + +### Why Substrate? + +#### 💭 An update in 2024 💭 + +🧐 Multi-chain vs. Multi-shard 🧐 + +Note: + +Polkadot led the way since 2016 to convince the world that the future is multi-chain. + +Cosmos tried to replicate the same without shared security. + +ETH is now doing something similar, with a Rollup-Centric roadmap. + +Interestingly, Polkadot is now arguing that the future is not necessarily multi-chain, but rather +multi-shard. In that, there needs to be a sharded system, with a high degree of flexibility to +schedule those shards. "Multi-chain" is merely one manifestation of a multi-shard system, which is +actually not so flexible. We are now striving to make the multi-chain Polkadot more flexible, which +is what JAM is going to be. + --- ## Core Philosophies of Substrate 💭 @@ -335,7 +357,11 @@ whole thing to be updated. ### 🏦 Governance + Upgradeability -_The way to make a protocol truly upgradeable is to design a meta-protocol._ + + +Note: + +This is very similar to treating your own runtime as a smart contract. ---v @@ -352,11 +378,7 @@ be upgraded with a fork. ### 🏦 Governance + Upgradeability - - -Note: - -This is very similar to treating your own runtime as a smart contract. +_The way to make a protocol truly upgradeable is to design a (fixed) meta-protocol._ ---v @@ -382,15 +404,27 @@ This is very similar to treating your own runtime as a smart contract.

Runtime (Protocol)

-- Application logic -- Wasm (maybe **FRAME**) -- Stored as a part of your chain state +- Application logic aka. How we execute block +- Stored as a part of your chain state as a WASM Blob +- Upgradeable - Also known as: STF
-

Node (Meta-protocol)

+ + + +
+ + +---v + +#### Substrate (simplified) Architecture + + + +

Fixed Node (Meta-protocol)

- Native Binary - Executes the Wasm runtime @@ -399,6 +433,12 @@ This is very similar to treating your own runtime as a smart contract.
+ + + + + +
--- @@ -514,47 +554,6 @@ https://forum.polkadot.network/t/announcing-polkavm-a-new-risc-v-based-vm-for-sm --- -## State of Light Client/Nodes - -- Node that follows headers, therefore knows state roots and can ask for state-proofs to do more. - -Note: - -Establish that it is merely an alternative node implementation. - ----v - -### State of Light Clients - - - -Notes: - -What was a light client? follows only block headers, therefore knows state roots, and a few other -pieces of information, others send it state proofs if it wishes to do more. - -SMOLDOT is not exactly a substrate client. It is mainly designed to work with Polkadot. But with -minimal tweaks, you could make it work for more substrate based chains. - -This has to do with the fact that consensus and a few other bits of the client and runtime are not -100% independent. For example, GRANDPA has a pallet on the runtime side, but is mostly in the -client. Now, a client that is configured with GRANDPA can only work with runtimes that are also -configured with GRANDPA. - ----v - -### State of Light Clients - - - -Note: - -Similarly it is dead simple to embed smoldot in mobile etc. - -Separate lecture on this later. - ---- - ## Communication Paths @@ -674,34 +673,6 @@ fn main() { ### Substrate and Smart Contracts -> SMOLDOT is syncing a chain who's runtime is executing wasm contracts. - -Question: How many nested Wasm blobs are executing one another? - ----v - -### Substrate and Smart Contracts - - - - - - - - - -- The browser is executing: -- a Wasm blob (SMOLDOT in extension) -- that executes a Wasm blob (runtime) -- that executes a Wasm blob (contract) - - - - ----v - -### Substrate and Smart Contracts - - So when should you write with a smart contract, and when a Runtime (FRAME)? Notes: @@ -722,7 +693,7 @@ fee-less. --- -## Technical Freedom vs Ease +## Development Options @@ -975,8 +946,7 @@ state is sometimes called "storage" asd well. - **Rust**, **Generic Design**, **Upgradeability/Governance** - Positive and negative consequences of Wasm - Substrate next to Polkadot and other chains. -- Substrate for Smart Contracts. -- Light Nodes/Clients. +- Substrate and Smart Contracts. - Node / Runtime architecture - State Transition Deep Dive -> Forkless Upgrade @@ -1018,7 +988,7 @@ hardcoded, but the protocol itself is flexible. #### Track: Aux Lecture -- Tx Pool (Kian) +- Tx Pool - Interacting with Substrate - SCALE Codec @@ -1086,7 +1056,13 @@ exception](https://www.gnu.org/software/classpath/license.html). --- -## Appendix: More Diagrams of Substrate and Polkadot +## Appendix + +Content that is not covered, but is relevant. + +--- + +### More Diagrams of Substrate and Polkadot Notes: @@ -1156,3 +1132,72 @@ People actually tried sticking things like JVM into the browser (_Java Applets_) - Any language that can compile to Wasm and exposes a fixed set of functions, to be used by the client. - ... But, of course, Substrate comes with a framework to make this developer-friendly, **FRAME™️**. + +--- + +## State of Light Client/Nodes + +- Node that follows headers, therefore knows state roots and can ask for state-proofs to do more. + +Note: + +Establish that it is merely an alternative node implementation. + +---v + +### State of Light Clients + + + +Notes: + +What was a light client? follows only block headers, therefore knows state roots, and a few other +pieces of information, others send it state proofs if it wishes to do more. + +SMOLDOT is not exactly a substrate client. It is mainly designed to work with Polkadot. But with +minimal tweaks, you could make it work for more substrate based chains. + +This has to do with the fact that consensus and a few other bits of the client and runtime are not +100% independent. For example, GRANDPA has a pallet on the runtime side, but is mostly in the +client. Now, a client that is configured with GRANDPA can only work with runtimes that are also +configured with GRANDPA. + +---v + +### State of Light Clients + + + +Note: + +Similarly it is dead simple to embed smoldot in mobile etc. + +Separate lecture on this later. + +---v + +### Substrate and Smart Contracts + +> SMOLDOT is syncing a chain who's runtime is executing wasm contracts. + +Question: How many nested Wasm blobs are executing one another? + +---v + +### Substrate and Smart Contracts + + + + + + + + + +- The browser is executing: +- a Wasm blob (SMOLDOT in extension) +- that executes a Wasm blob (runtime) +- that executes a Wasm blob (contract) + + + diff --git a/syllabus/6-Polkadot-SDK/Substrate/2-Wasm-Meta-Protocol-Slides.md b/syllabus/6-Polkadot-SDK/Substrate/2-Wasm-Meta-Protocol-Slides.md index c1e1b166b..072a207c6 100644 --- a/syllabus/6-Polkadot-SDK/Substrate/2-Wasm-Meta-Protocol-Slides.md +++ b/syllabus/6-Polkadot-SDK/Substrate/2-Wasm-Meta-Protocol-Slides.md @@ -14,24 +14,11 @@ duration: 60 minutes --- -## It All Began With a Runtime.. - -- Personal opinion: - -> Substrate technology stack will put "Wasm stored onchain" on the map,
-> the same way Ethereum put "smart contracts stored onchain" on the map. - -Notes: - -> It is only a matter of time until every blockchain is doing the same thing. - ----v - -### It All Began With a Runtime.. +## Meta Protocol - The Node / Runtime division is one of the most important design decisions in Substrate. - - 👿 Bad: Fixed opinion. - - 😇 Good: Enables countless other things to not be fixed. + - ­ 👿 Bad: _One_ fixed opinion. + - ­ 😇 Good: _Countless_ other things to flexible. Notes: @@ -188,6 +175,8 @@ Notes: Short answer is: anything that is part of the STF definition must be opaque to the node, and is upgradeable, but we will learn this later. +Another way to think of it: Anything that we want to keep upgrade-able needs to be opaque. + ---v ### Example #2: Block Import @@ -668,6 +657,7 @@ This is why forkless upgrades are possible in substrate. - What about new host functions? - What about a new header field*? - What about a new Hashing primitive? +- A new consensus/networking engine? 🥺 No longer forkless. @@ -679,8 +669,7 @@ But, recall that substrate's extensibility and generic-ness clause applies here. For some, like header, some hacks exist, like the `digest` field. -Changing these is hard in a forkless manner. If you want to just change them at genesis and launch a -new chain, they are all VERY easy to change. +Changing these is hard in a forkless manner. If you want to just change them at genesis and launch a new chain, they are all VERY easy to change. --- @@ -894,79 +883,20 @@ wasm can be bigger than the actual hashing cost. --- -## Consideration: Native Runtime +### Runtime Version vs. Node Version - - ----v - -### Consideration: Native Runtime - -- Remember the `fn version()` in `Core` API! - -```rust [1-100|4-5] -/// Runtime version. +```rust #[sp_version::runtime_version] pub const VERSION: RuntimeVersion = RuntimeVersion { spec_name: create_runtime_str!("node"), spec_version: 268, - impl_name: create_runtime_str!("substrate-node"), impl_version: 0, - authoring_version: 10, - apis: RUNTIME_API_VERSIONS, - transaction_version: 2, - state_version: 1, + .. }; ``` ---v -### Consideration: Native Runtime - -- Native is only an option if spec versions match! - -```rust -fn execute_native_else_wasm() { - let native_version = runtime::native::api::version(); - let wasm_version = runtime::wasm::api::version(); - - // if spec name and version match. - if native_version == wasm_version { - runtime::native::execute(); - } else { - runtime::wasm::execute(); - } -} -``` - ----v - -### Consideration: Native Runtime - -- The days of native runtime are numbered 💀. - ----v - -### Consideration: Native Runtime - -- Question: what happens if you upgrade your runtime, but forget to bump the spec version? -- ­Question: What if a runtime upgrade is only tweaking implementation details, but not the specification? - -Notes: - -If everyone is executing wasm, technically nothing, but that's super confusing, don't do it. -But, if some are executing native, then you will have a consensus error. - ----v - -### Speaking of Versions.. - -- Make sure you understand the difference! 👍 - - Node Version - - Runtime Version - ----v - ### Speaking of Versions.. @@ -987,7 +917,7 @@ But, if some are executing native, then you will have a consensus error. ### Speaking of Versions.. -- What happens when Parity release a new `parity-polkadot` node binary? +- What happens when Parity release a new `polkadot` node binary? - What happens when the Polkadot fellowship wants to update the runtime? --- @@ -1130,7 +1060,7 @@ fn root(&mut self, version: StateVersion) -> Vec { .. } --- -## Workshop: Inspecting Wasm Code +## Activity: Inspecting Wasm Code ---v @@ -1269,7 +1199,7 @@ fn root(&mut self, version: StateVersion) -> Vec { .. } ---v -### Workshop: Inspecting Wasm Code +### Activity: Inspecting Wasm Code - Once you reach the Polkadot module, and you build your first parachain, repeat the same, I promise you will learn a thing or two :) @@ -1474,5 +1404,3 @@ https://paritytech.github.io/substrate/master/sc_basic_authorship/index.html ## Misc - ----v diff --git a/syllabus/6-Polkadot-SDK/Substrate/3-Merklized-Storage-slides.md b/syllabus/6-Polkadot-SDK/Substrate/3-Merklized-Storage-slides.md index 87c93e4fd..969a17af8 100644 --- a/syllabus/6-Polkadot-SDK/Substrate/3-Merklized-Storage-slides.md +++ b/syllabus/6-Polkadot-SDK/Substrate/3-Merklized-Storage-slides.md @@ -68,7 +68,7 @@ Notes: ## Key Value -> All this seems to indicate our storage externality is a simple key value database. +> This seems to indicate our storage could be key-value all the way.. 🤔 ---v @@ -174,7 +174,7 @@ flowchart TD -- Trie +- Trie aka. Prefix tree - Assuming only leafs have data, this is encoding: @@ -384,6 +384,7 @@ compact proof").
- Storage key (`balances_alice`) is the path on the trie. +- `sp_io::storage::get("balances_alice");`
@@ -427,51 +428,6 @@ it will be `O(LOG_n)`. --- -## Large data nodes 🤔 - -- Two common problems that merkle proofs have: - - If the one of the parent nodes has some large data. - - If you want to prove the deletion/non-existence of a leaf node. - ----v - - - ----v - -## Large data nodes 🤔 - -New "trie format" 🌈: - -- All data containing more than 32 bytes are replaced with their hash (pointer to the actual value). -- The (larger than 32 bytes) value itself stored in the database under this hash. - -```rust -struct RuntimeVersion { - ... - state_version: 0, -} -``` - - - ----v - - - - -What is the ramification of this for full nodes, and light clients? - -Notes: - -Both read and write have an extra step now, but proof are easier. - -Note from emeric: the green node is not really a "real" node, it is just `{ value: BIG_STUFF }` stored in the database. -I will skip this detail for the sake of simplicity. -One can assume that the green node is like any other node in the trie. - ---- - ### Unbalanced Tree @@ -511,7 +467,7 @@ All of that can be delayed. ## Overlay -- Is a cache layer outside of the Runtime. +- Is a cache layer **outside of the Runtime**. - It works **based on key-values**, **not trie-format**. ---v @@ -627,8 +583,7 @@ the trie layer and pull a whole lot of data back from the disk and build all the ### Overlay: More Caches -- There are more caches in the trie layer as well. - But outside of the scope of this lecture. +- There are more caches in the trie layer as well. But outside of the scope of this lecture. ```bash ./substrate --help | grep cache @@ -648,19 +603,6 @@ https://www.youtube.com/embed/OoMPlJKUULY ### Substrate Storage -There are multiple implementations of `Externalities`: - -- [`TestExternalities`](https://paritytech.github.io/substrate/master/sp_state_machine/struct.TestExternalities.html): - - `Overlay` - - `TrieDb` with `InMemoryBackend` -- [`Ext`](https://paritytech.github.io/substrate/master/sp_state_machine/struct.Ext.html) (the real thing 🫡) - - `Overlay` - - `TrieDb` with a real database being the backend - ----v - -### Substrate Storage - - Recall: Any code accessing host functions needs to wrapped in something that implements `Externalities` @@ -725,26 +667,6 @@ Notes: --- -## Child Trees - - - -Notes: -Shawn's answer on usecases: https://substrate.stackexchange.com/questions/139/what-are-good-use-cases-for-child-tries/144#144 - -- Used in smart contracts for being able to delete contract data and reinstate later. -- Generating proof of contribution to crowdloan. -- Alternative trie format. - ----v - -### Child Trees - -- Stored on a different DB Column (async-ish bulk deletion). -- Most importantly, alternative trie formats. - ---- - ## Base 2, Base 16, Base-26? - Instead of alphabet, we use the base-16 representation of everything. @@ -812,11 +734,15 @@ Notes: - KV-Based storage - Merklized storage, and proofs -- Large nodes - Unbalanced tree - State pruning - Radix order consequences +In the Appendix, you will find: + +- Large nodes affecting the proof +- Child tries + @@ -874,7 +800,13 @@ proof recorder, and see which parts of the trie is exactly part of the proof. --- -### Overlay +## Appendix + +Content that is not covered, but is relevant. + +--- + +### Overlay Limits - There is a limit to how many nested layers you can spawn - It is not free, thus it is attack-able. @@ -910,3 +842,82 @@ with_storage_layer(|| { Notes: Meaning, if another client wants to sync polkadot, it should know the details of the trie format. + +--- + +## Child Trees + + + +Notes: + +Shawn's answer on usecases: https://substrate.stackexchange.com/questions/139/what-are-good-use-cases-for-child-tries/144#144 + +- Used in smart contracts for being able to delete contract data and reinstate later. +- Generating proof of contribution to crowdloan. +- Alternative trie format. + +---v + +### Child Trees + +- Stored on a different DB Column (async-ish bulk deletion). +- Easily prove-able +- Most importantly, alternative trie formats. + +--- + +## Large data nodes 🤔 + +- Two common problems that merkle proofs have: + - If the one of the parent nodes has some large data. + - If you want to prove the deletion/non-existence of a leaf node. + +---v + + + +---v + +## Large data nodes 🤔 + +New "trie format" 🌈: + +- All data containing more than 32 bytes are replaced with their hash (pointer to the actual value). +- The (larger than 32 bytes) value itself stored in the database under this hash. + +```rust +struct RuntimeVersion { + ... + state_version: 0, +} +``` + + + +---v + + + +What is the ramification of this for full nodes, and light clients? + +Notes: + +Both read and write have an extra step now, but proof are easier. + +Note from emeric: the green node is not really a "real" node, it is just `{ value: BIG_STUFF }` stored in the database. +I will skip this detail for the sake of simplicity. +One can assume that the green node is like any other node in the trie. + +--- + +### Storage Implementation + +There are multiple implementations of `Externalities`: + +- [`TestExternalities`](https://paritytech.github.io/substrate/master/sp_state_machine/struct.TestExternalities.html): + - `Overlay` + - `TrieDb` with `InMemoryBackend` +- [`Ext`](https://paritytech.github.io/substrate/master/sp_state_machine/struct.Ext.html) (the real thing 🫡) + - `Overlay` + - `TrieDb` with a real database being the backend diff --git a/syllabus/6-Polkadot-SDK/Substrate/4-Transaction-Pool-slides.md b/syllabus/6-Polkadot-SDK/Substrate/4-Transaction-Pool-slides.md index 99bc986ae..f1482a5ad 100644 --- a/syllabus/6-Polkadot-SDK/Substrate/4-Transaction-Pool-slides.md +++ b/syllabus/6-Polkadot-SDK/Substrate/4-Transaction-Pool-slides.md @@ -98,6 +98,7 @@ Important, must pause and ask! - Why is it from the runtime? because the transaction format is opaque and the node doesn't even know what to do with it. - Why does it have to be cheap? wild west, unpaid, DoS! - Pesky question: but be aware that from the runtime's perspective, the node could be malicious. The runtime cannot trust the node to obey. + ** THE RUNTIME MUST RE-VALIDATE TRANSACTIONS LATER in block building and import as well ** ---v diff --git a/syllabus/6-Polkadot-SDK/img/multi-threaded.svg b/syllabus/6-Polkadot-SDK/img/multi-threaded.svg index 8743ef1d5..d93b29877 100644 --- a/syllabus/6-Polkadot-SDK/img/multi-threaded.svg +++ b/syllabus/6-Polkadot-SDK/img/multi-threaded.svg @@ -1 +1 @@ - \ No newline at end of file + \ No newline at end of file