Onion messages v1 #1503

valentinewallace · 2022-05-29T01:23:40Z

Implement v1 of onion messages per lightning/bolts#759. This will enable us to implement Offers messaging.

TODOs

~~tests from BOLT 7: Onion message support (features 38/39) lightning/bolts#759:~~
- ~~bolt04/enctlvs.json~~
- ~~bolt04/onion-route-blinding-test.json~~
- ~~bolt04/route-blinding-override-test.json~~
- ~~bolt04/route-blinding-test.json~~

our own integration tests
refuse to create packets that would cause their hop_data to exceed BIG_PACKET_HOP_DATA_LEN (because eclair will refuse to forward these)

Follow-ups documented in #1607

Based on #1518

codecov-commenter · 2022-05-29T01:32:59Z

Codecov Report

Merging #1503 (b8301da) into main (e403999) will decrease coverage by 0.07%.
The diff coverage is 88.84%.

❗ Current head b8301da differs from pull request most recent head 81e3c41. Consider uploading reports for the commit 81e3c41 to get more accurate results

@@            Coverage Diff             @@
##             main    #1503      +/-   ##
==========================================
- Coverage   91.03%   90.95%   -0.08%     
==========================================
  Files          80       85       +5     
  Lines       44194    44679     +485     
  Branches    44194    44679     +485     
==========================================
+ Hits        40233    40639     +406     
- Misses       3961     4040      +79

Impacted Files	Coverage Δ
lightning/src/lib.rs	`100.00% <ø> (ø)`
lightning/src/ln/channel.rs	`88.69% <0.00%> (-0.03%)`	⬇️
lightning/src/ln/mod.rs	`95.00% <ø> (ø)`
lightning/src/util/ser.rs	`89.24% <0.00%> (-1.54%)`	⬇️
lightning/src/chain/keysinterface.rs	`92.54% <50.00%> (-1.06%)`	⬇️
lightning/src/ln/msgs.rs	`86.18% <63.15%> (-0.32%)`	⬇️
lightning/src/util/test_utils.rs	`78.13% <75.00%> (-0.03%)`	⬇️
lightning/src/onion_message/packet.rs	`83.33% <83.33%> (ø)`
lightning/src/onion_message/messenger.rs	`89.15% <89.15%> (ø)`
lightning/src/onion_message/blinded_route.rs	`95.74% <95.74%> (ø)`
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e403999...81e3c41. Read the comment docs.

valentinewallace · 2022-06-01T00:03:28Z

According to @t-bast (and my experience), the spec test vectors are all broken and won't be fixed until lightning/bolts#765 has ACKs on the data model. Still gonna write our own tests, but removed the spec test vectors from the PR TODOs so they aren't blocking.

ariard

Browsed the new onion_message.rs API. Catching up with the spec itself.

ariard · 2022-06-03T18:27:55Z

lightning/src/ln/onion_message.rs

+	pub fn new<Signer: Sign, K: Deref>(node_pks: Vec<PublicKey>, keys_manager: K) -> Result<Self, ()>
+		where K::Target: KeysInterface<Signer = Signer>,
+	{
+		if node_pks.len() <= 1 { return Err(()) }


Would we like to enforce a max length ? From my understanding the specs isn't recommending one. I think we may be interested to check one at blinded route reception for memory bounds reasons, as it's a third-party input.

I added a TODO to ensure that outbound onion packets can't exceed BIG_PACKET_HOP_DATA_LEN. Reasoning is that Eclair will drop onion messages that exceed this length

If it's a value shared among implementation, nice to add it to the spec imo.

Ah, forgot to mention but as you mentioned below it's in the spec: https://github.com/lightning/bolts/pull/759/files#diff-ad65f0beaac5cef88f5fd7a8b9ca36cbc5a790f4815b4b50b9ea794a55aaf012R1421

lightning/src/ln/onion_message.rs

ariard · 2022-06-03T18:53:15Z

lightning/src/ln/onion_message.rs

+		/// If `path_id` is `Some`, it is used to identify the blinded route that this onion message is
+		/// sending to. This is useful for receivers to check that said blinded route is being used in
+		/// the right context.
+		path_id: Option<[u8; 32]>


Suggestion : if path_id usage is to identify that an encrypted_payload is arriving from the expected, last-hop node id, we should call it something like blinded_route_id to avoid confusion with the notion of a payment path.

A part of that, it's unclear to me what kind of further checks could be realized to verify the "right context usage". I would be concerned it could introduce "naive" deanonymization attacks, where the encrypted path_id is tampered with to observe success/errors from timing processing.

Sure thing. Thankfully, the path_id cannot be tampered with -- it's always provided by the receiver in an encrypted and HMAC-d blob

Right after reading the spec, if it's tampered it should break at the HMAC-level.

lightning/src/ln/onion_message.rs

ariard · 2022-06-03T19:14:46Z

lightning/src/ln/onion_message.rs

+		let node_secret = match self.keys_manager.get_node_secret(Recipient::Node) {
+			Ok(secret) => secret,
+			Err(e) => {
+				log_trace!(self.logger, "Failed to retrieve node secret: {:?}", e);


Maybe "Onion Reception : Failed to retrieve node secret" ? get_node_secret is used few times across the codebase, nice to provide more info to identify the callsite.

This log should be prefixed by the onion_messages module name, do you think that helps? 🤔

I would say yes, we start to have a lot of logs, being able to observe specific subsystem help to debug test from exp.

lightning/src/ln/onion_message.rs

To get the next hop's packet's pubkey. This will be used to DRY onion message forwarding in the upcoming Onion Messages PR lightningdevkit#1503

TheBlueMatt

Really quick initial first pass, but generally I think this is looking pretty good. The API seems sensible and the implementation is pretty straightforward.

lightning/src/ln/onion_message.rs

TheBlueMatt · 2022-06-22T15:04:01Z

lightning/src/ln/onion_message.rs

+	Receive {
+		/// If `path_id` is `Some`, it is used to identify the blinded route that this onion message is
+		/// sending to. This is useful for receivers to check that said blinded route is being used in
+		/// the right context.


Does this need a TODO to make it non-optional? Or are there reasons to set it to None once we're done with our full implementation?

It may be None in the future if someone were sending to our unblinded public key and not to a blinded route we create

lightning/src/ln/onion_message.rs

TheBlueMatt · 2022-06-22T15:56:19Z

lightning/src/ln/onion_message.rs

+}
+
+impl Writeable for Packet {
+	fn write<W: Writer>(&self, w: &mut W) -> Result<(), io::Error> {


Why is Packet never written/read in tests? At least according to codecov.

That's bizarre, is there a link to that? Can't find it

codecov may be wrong, but https://app.codecov.io/gh/lightningdevkit/rust-lightning/compare/1503/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr%20comments&utm_term=lightningdevkit#D8L53 you may have to click to open up the onion_message.rs diff.

Hmm, when I add logs to the read/write methods in the tests they def get hit 🤔

Ugh codecov sucks.

valentinewallace · 2022-06-23T19:56:38Z

Will add two missing tests next but pushed up the latest changes, main difference being somewhat extensive module-level documentation being added and the new sending errors.

valentinewallace · 2022-06-24T04:25:35Z

This is good for review!

ariard

Reviewed so far until 15983c2

ariard · 2022-06-25T00:53:45Z

lightning/src/onion_message.rs

+	/// Control TLVs for the final recipient of an onion message.
+	Receive {
+		/// If `path_id` is `Some`, it is used to identify the blinded route that this onion message is
+		/// sending to. This is useful for receivers to check that said blinded route is being used in


nit: As I understand the path_id is useful for the receiver, so it could say "this onion message is coming from".

Hm, in my mental model you send an onion message to a blinded route, which aligns with it being a Destination variant.

lightning/src/onion_message.rs

lightning/src/ln/onion_utils.rs

ariard · 2022-06-25T02:01:33Z

lightning/src/ln/onion_message.rs

+	pub fn new<Signer: Sign, K: Deref>(node_pks: Vec<PublicKey>, keys_manager: K) -> Result<Self, ()>
+		where K::Target: KeysInterface<Signer = Signer>,
+	{
+		if node_pks.len() <= 1 { return Err(()) }


If it's a value shared among implementation, nice to add it to the spec imo.

ariard · 2022-06-25T02:03:15Z

lightning/src/ln/onion_message.rs

+		/// If `path_id` is `Some`, it is used to identify the blinded route that this onion message is
+		/// sending to. This is useful for receivers to check that said blinded route is being used in
+		/// the right context.
+		path_id: Option<[u8; 32]>


Right after reading the spec, if it's tampered it should break at the HMAC-level.

ariard · 2022-06-25T02:04:20Z

lightning/src/ln/onion_message.rs

+		let node_secret = match self.keys_manager.get_node_secret(Recipient::Node) {
+			Ok(secret) => secret,
+			Err(e) => {
+				log_trace!(self.logger, "Failed to retrieve node secret: {:?}", e);


I would say yes, we start to have a lot of logs, being able to observe specific subsystem help to debug test from exp.

ariard

Code Review ACK e4a50a3

Changes since last review:

minor comments change around construct_onion_packet_with_init_noise
buffered read for Packet
clarification of ecdh() comment
new construct_onion_message_packet to initialize ChaCha20
restructuring of packet_payloads_and_keys
spec rational comment in construct_onion_message_packet
new SendError::TooFewBlindedHops and corresponding test

fuzz/src/chanmon_consistency.rs

lightning/src/ln/onion_utils.rs

lightning/src/util/ser.rs

Blinded routes can be provided as destinations for onion messages, when the recipient prefers to remain anonymous. We also add supporting utilities for constructing blinded path keys, and control TLVs structs representing blinded payloads prior to being encoded/encrypted. These utilities and struct will be re-used in upcoming commits for sending and receiving/forwarding onion messages. Finally, add utilities for reading the padding from an onion message's encrypted TLVs without an intermediate Vec.

lightning/src/onion_message/messenger.rs

TheBlueMatt · 2022-08-02T18:45:29Z

lightning/src/onion_message/messenger.rs

+				};
+
+				let mut pending_per_peer_msgs = self.pending_messages.lock().unwrap();
+				let pending_msgs = pending_per_peer_msgs.entry(next_node_id).or_insert(Vec::new());


We need to keep track of if we're actually connected to the given peer here, and simply drop the message if we're not. We can do this in the followup that wires this up to PeerManager, though.

Thanks, added a TODO on the follow-up PR to address soon

lightning/src/onion_message/packet.rs

TheBlueMatt · 2022-08-02T19:13:04Z

lightning/src/ln/onion_utils.rs

+	}
+}
+
+pub(crate) fn decode_next_hop<D: DecodeInput, R: ReadableArgs<D::Arg>, N: NextPacketBytes>(shared_secret: [u8; 32], hop_data: &[u8], hmac_bytes: [u8; 32], decode_input: D) -> Result<(R, Option<([u8; 32], N)>), OnionDecodeErr> {


I'm really not a fan of this being public with all the trait bounds that get auto-implemented based on what's passed in. Can we make this private and add a decode_next_message_hop like we have for decode_next_payment_hop, then we can make the DecodeInput and NextPacketBytes traits priv? Happy to see this in a followup, too, if you're feeling lazy.

Sounds good. Noted in the follow-up issue, will aim to be a quick follow-up

TheBlueMatt · 2022-08-02T20:37:17Z

LGTM, feel free to squash when other reviewers are happy.

jkczyz

LGTM modulo one comment. Feel free to squash.

lightning/src/util/ser.rs

…noise for it We need to add a new Packet struct because onion message packet hop_data fields can be of variable length, whereas regular payment packets are always 1366 bytes. Co-authored-by: Valentine Wallace <vwallace@protonmail.com> Co-authored-by: Jeffrey Czyz <jkczyz@gmail.com>

This method will help us avoid retrieving our node secret, something we want to get rid of entirely. It will be used in upcoming commits when decoding the onion message packet, and in future PRs to help us get rid of KeysInterface::get_node_secret usages across the codebase

OnionMessenger will be hooked up to the PeerManager to send and receive OMs in a follow-up PR.

This adds several utilities in service of then adding OnionMessenger::send_onion_message, which can send to either an unblinded pubkey or a blinded route. Sending custom TLVs and sending an onion message containing a reply path are not yet supported. We also need to split the construct_keys_callback macro into two macros to avoid an unused assignment warning.

This required adapting `onion_utils::decode_next_hop` to work for both payments and onion messages. Currently we just print out the path_id of any onion messages we receive. In the future, these received onion messages will be redirected to their respective handlers: i.e. an invoice_request will go to an InvoiceHandler, custom onion messages will go to a custom handler, etc.

Pre-existing to this PR, we were reading next packet bytes with io::Read::read, which is not guaranteed to read all the bytes we need, only guaranteed to read *some* bytes. We fix this to be read_exact, which is guaranteed to read all the next hop packet bytes.

valentinewallace changed the title ~~Onion messages~~ Onion messages v1 May 29, 2022

valentinewallace force-pushed the 2022-05-onion-msgs branch 2 times, most recently from eaecccd to 050268a Compare June 1, 2022 01:23

ariard reviewed Jun 3, 2022

View reviewed changes

valentinewallace mentioned this pull request Jun 4, 2022

Onion messages API #1432

Closed

valentinewallace force-pushed the 2022-05-onion-msgs branch from 050268a to 3d7d7ad Compare June 4, 2022 21:10

valentinewallace mentioned this pull request Jun 4, 2022

Onion messages v1 pre-refactor #1518

Merged

valentinewallace force-pushed the 2022-05-onion-msgs branch from 3d7d7ad to d2ef7c5 Compare June 22, 2022 03:00

TheBlueMatt reviewed Jun 22, 2022

View reviewed changes

valentinewallace force-pushed the 2022-05-onion-msgs branch from d2ef7c5 to d40751f Compare June 22, 2022 17:43

valentinewallace marked this pull request as ready for review June 22, 2022 17:44

valentinewallace force-pushed the 2022-05-onion-msgs branch from d40751f to 9223986 Compare June 22, 2022 23:28

jkczyz self-requested a review June 23, 2022 15:41

valentinewallace force-pushed the 2022-05-onion-msgs branch from 9223986 to fb10b10 Compare June 23, 2022 19:53

valentinewallace force-pushed the 2022-05-onion-msgs branch 3 times, most recently from 0dfad2a to 15983c2 Compare June 24, 2022 04:20

valentinewallace requested a review from TheBlueMatt June 24, 2022 19:33

ariard reviewed Jun 25, 2022

View reviewed changes

TheBlueMatt added the blocked on next release Should Wait Until Next Release To Land label Jun 27, 2022

valentinewallace force-pushed the 2022-05-onion-msgs branch 4 times, most recently from 282ed9e to a063de6 Compare June 27, 2022 21:00

valentinewallace requested a review from jkczyz August 1, 2022 16:37

ariard previously approved these changes Aug 1, 2022

View reviewed changes

TheBlueMatt reviewed Aug 2, 2022

View reviewed changes

fuzz/src/chanmon_consistency.rs Outdated Show resolved Hide resolved

lightning/src/ln/onion_utils.rs Show resolved Hide resolved

lightning/src/util/ser.rs Outdated Show resolved Hide resolved

valentinewallace dismissed ariard’s stale review via 3d6e0a8 August 2, 2022 16:53

valentinewallace force-pushed the 2022-05-onion-msgs branch from e4a50a3 to 3d6e0a8 Compare August 2, 2022 16:53

valentinewallace requested a review from TheBlueMatt August 2, 2022 17:19

valentinewallace force-pushed the 2022-05-onion-msgs branch from 3d6e0a8 to cb7bb84 Compare August 2, 2022 18:07

TheBlueMatt reviewed Aug 2, 2022

View reviewed changes

valentinewallace force-pushed the 2022-05-onion-msgs branch from cb7bb84 to 4919172 Compare August 2, 2022 19:25

jkczyz reviewed Aug 2, 2022

View reviewed changes

lightning/src/util/ser.rs Outdated Show resolved Hide resolved

valentinewallace and others added 10 commits August 2, 2022 19:17

Add baseline OnionMessenger and msgs::OnionMessage and its serialization

4c8dc2c

OnionMessenger will be hooked up to the PeerManager to send and receive OMs in a follow-up PR.

Significantly expand onion message documentation

b26fb85

Add test utilities and integration tests for onion messages

eaff561

Add SendError enum for onion messages and error on too-big packets

6500c99

Error when attempting to send an OM to a blinded route with 0 hops

39397d4

valentinewallace force-pushed the 2022-05-onion-msgs branch from 4919172 to 17ec697 Compare August 2, 2022 23:19

valentinewallace requested review from jkczyz and TheBlueMatt August 2, 2022 23:25

TheBlueMatt approved these changes Aug 3, 2022

View reviewed changes

jkczyz approved these changes Aug 3, 2022

View reviewed changes

TheBlueMatt merged commit 28c9b56 into lightningdevkit:main Aug 3, 2022

jkczyz mentioned this pull request May 10, 2023

BOLT12 Offers Tracking Issue #1970

Open

60 tasks

Onion messages v1 #1503

Onion messages v1 #1503

Conversation

valentinewallace commented May 29, 2022 • edited Loading

codecov-commenter commented May 29, 2022 • edited Loading

Codecov Report

valentinewallace commented Jun 1, 2022

ariard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valentinewallace commented Jun 23, 2022

valentinewallace commented Jun 24, 2022

ariard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ariard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Aug 2, 2022

jkczyz left a comment

Choose a reason for hiding this comment

valentinewallace commented May 29, 2022 •

edited

Loading

codecov-commenter commented May 29, 2022 •

edited

Loading