add support for OpenBSD/pf #1567

ge9 · 2024-06-24T17:19:24Z

Added support for OpenBSD/pf.
In OpenBSD's pf, divert-to is used for transparent proxying.
As described in https://man.openbsd.org/pf.conf, TCP destination address can be obtained by getsockname() (like Linux+TPROXY or FreeBSD+ipfw), and UDP destination address can be obtained by special socket options like IP_RECVDSTADDR.
send_to using "fake" source address fails, but using "0.0.0.0:0" instead works. Note that this is not related to IPv4-mapped IPv6 address issue (#1543), because OpenBSD doesn't support IPv4-mapped IPv6 address (https://man.openbsd.org/inet6) and shadowsocks can detect it.
Currently IPv6 is not supported. Also, --features "local-tun" doesn't work.

zonyitoo · 2024-06-24T17:22:54Z

crates/shadowsocks-service/src/local/net/udp/association.rs

@@ -625,6 +625,8 @@ where
                if bypassed { "bypassed" } else { "proxied" },
                err
            );
+            #[cfg(target_os = "openbsd")]
+            let _ = self.respond_writer.send_to(self.peer_addr, &Address::SocketAddress(SocketAddr::new(std::net::IpAddr::V4(std::net::Ipv4Addr::new(0,0,0,0)), 0)), data).await;


Why OpenBSD needs this "faked" destination address? addr is the actual address of client.

I think peer_addr is client (local) address and addr is remote address, which is not assigned to sslocal itself, resulting an error (Can't assign requested address, error 49).

That's not true. send_received_respond_packet is for sending back packets received from remote server to local client. So:

peer_addr is the receiver (client)'s address

addr is the sender (remote server)'s address

The code in respond_write.send_to, which is redir/udprelay/mod.rs:UdpRedirInboundWriter::send_to were doing:

Create an UDP socket, bind() to addr

Call sendto to peer_addr

This is the way to make an UDP packet that source address is addr and destination address is peer_addr.

If socket's bind() failed right here (EADDRNOTAVAIL), SO_BINDANY was not set or working as expected. Setting addr to 0.0.0.0:0 is actually let system to allocate one for this socket, which is not the expected behavior.

Sorry, I mistakenly disable set_bindany() function. If I enabled it, the original send_to worked correctly.
https://man.openbsd.org/getsockopt.2 says that we need divert-reply along with SO_BINDANY, but in my environment, it seems working correctly without divert-reply...🤔

SO_BINDANY allows socket to bind() an address that is not belongs to local environment. pf should be one of the use cases.

zonyitoo · 2024-06-24T17:23:54Z

crates/shadowsocks-service/src/local/redir/sys/unix/mod.rs

@@ -15,7 +15,7 @@ cfg_if! {
        #[allow(dead_code, non_upper_case_globals, non_snake_case, non_camel_case_types)]
        #[allow(clippy::useless_transmute, clippy::too_many_arguments, clippy::unnecessary_cast)]
        mod pfvar;
-    } else if #[cfg(target_os = "freebsd")] {
+    } else if #[cfg(any(target_os = "freebsd", target_os = "openbsd"))] {
        #[path = "pfvar_bindgen_freebsd.rs"]


pfvar_bindgen_freebsd.rs is generated on FreeBSD. I suggest you to generate one on OpenBSD with bindgen-cli.

Ah actually, for bindany, I just reversed the comment-out of the old existing code. Maybe that's not working correctly? I'll look into it.

WRONG COMMENT. :)

Sorry I'm not familiar with this tool, but how can I do this? Is there a commandline for generating this?

Is pfvar.h in OpenBSD included in release? If no, you should first download it from its source.

Create a header file, that only has #include <pfvar.h>.

$ cargo install bindgen-cli $ bindgen <your_created_header_file> -o pfvar_bindgen_openbsd.rs

More reference: https://rust-lang.github.io/rust-bindgen/command-line-usage.html

zonyitoo · 2024-06-24T17:24:49Z

What is that ktrace.out file?

ge9 · 2024-06-24T17:27:43Z

Ah sorry that's a stack trace...
I'll review and resend this PR sorry

ge9 · 2024-06-24T17:29:21Z

And I may be able to add IPv6 support soon.

zonyitoo · 2024-06-25T06:06:25Z

You should install an rustfmt, with:

rustup component add rustfmt
rustup component add rustfmt --toolchain nightly

Run rustfmt with:

cargo +nightly fmt

This project have set some unstable options, which requires nightly Rust.

ge9 · 2024-06-25T06:42:57Z

error[E0425]: cannot find value `IPV6_RECVDSTPORT` in crate `libc`
   --> crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/bsd.rs:272:37
    |
272 |         libc::IPPROTO_IPV6 => libc::IPV6_RECVDSTPORT,
    |                                     ^^^^^^^^^^^^^^^^ not found in `libc`
    |
note: constant `crate::local::redir::sys::unix::pfvar::IPV6_RECVDSTPORT` exists but is inaccessible
   --> crates/shadowsocks-service/src/local/redir/sys/unix/pfvar_bindgen_openbsd.rs:589:1
    |
589 | pub const IPV6_RECVDSTPORT: u32 = 64;
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not accessible

Sorry, I can't figure out how to solve this. Do you have any idea?

I think I successfully created pfvar_bindgen_openbsd and specified it in mod.rs

cfg_if! {
    if #[cfg(any(target_os = "macos",
                 target_os = "ios"))] {
        #[path = "pfvar_bindgen_macos.rs"]
        #[allow(dead_code, non_upper_case_globals, non_snake_case, non_camel_case_types)]
        #[allow(clippy::useless_transmute, clippy::too_many_arguments, clippy::unnecessary_cast)]
        mod pfvar;
    } else if #[cfg(target_os = "freebsd")] {
        #[path = "pfvar_bindgen_freebsd.rs"]
        #[allow(dead_code, non_upper_case_globals, non_snake_case, non_camel_case_types)]
        #[allow(clippy::useless_transmute, clippy::too_many_arguments, clippy::unnecessary_cast)]
        mod pfvar;
    } else if #[cfg(target_os = "openbsd")] {
        #[path = "pfvar_bindgen_openbsd.rs"]
        #[allow(dead_code, non_upper_case_globals, non_snake_case, non_camel_case_types)]
        #[allow(clippy::useless_transmute, clippy::too_many_arguments, clippy::unnecessary_cast)]
        mod pfvar;
    }
}

but it seems not working...

ge9 · 2024-06-25T06:52:18Z

There are already some hard-coded constant values for FreeBSD, like

shadowsocks-rust/crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/macos.rs

Line 167 in 6d7bc35

const IP_DONTFRAG: libc::c_int = 67; // don't fragment packet

. So, maybe pfvar_bindgen_freebsd.rs is not actually used either.
Should we write pub mod pfvar; instead of mod pfvar and use the constants in udprelay/sys/unix/bsd.rs?

zonyitoo · 2024-06-25T08:16:05Z

The first error indicates that OpenBSD doesn't have libc::IPV6_RECVDSTPORT in libc. Is that true? OpenBSD doesn't have IPV6_RECVDSTPORT?

If so, IPv6 support on OpenBSD have to be disabled.

Those hard-coded constant values exists because they are not defined in libc, so I copied them directly from source.

I would suggest to add IP_DONTFRAG for FreeBSD, OpenBSD independently instead of using the generated pfvar.rs, because they have no relation.

ge9 · 2024-06-25T08:41:13Z

Sorry, I pasted only the part of errors, but similar error happens for libc::IP_RECVDSTPORT.
These constants do exist in OpenBSD's C library, in netinet/in.h, and so included in generated pfvar_xxx.rs, but they seem not defined in Rust's libc.
The situation is exactly the same for IP_DONTFRAG. It's defined in FreeBSD's netinet/in.h and included in pfvar_xxx.rs, but not defined in Rust's libc.

zonyitoo · 2024-06-25T09:50:27Z

It would be great to open an PR to Rust's libc crate and add all those contants.

Before it got merge, it is Ok to just hard-coded.

All these constants are added by users, so if nobody is actually using it / depend on it, it won't exist in libc.

ge9 · 2024-06-25T10:19:16Z

Also, I added support for IPv6, but strangely enough, transparent proxying only works every other time. (I don't know detailed conditions)
I think this is OpenBSD's issue, because shadowsocks with -vvvv doesn't detect any packet.

crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/bsd.rs

crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/mod.rs

crates/shadowsocks-service/src/local/tun/mod.rs

zonyitoo · 2024-06-26T02:21:46Z

Consider update your branch with the latest master's commits.

ge9 · 2024-09-15T08:39:23Z

Now I'm working on completing this pull request.
Should I separate bsd.rs into openbsd.rs and freebsd.rs as you suggested?

zonyitoo · 2024-09-15T15:44:31Z

Now I'm working on completing this pull request. Should I separate bsd.rs into openbsd.rs and freebsd.rs as you suggested?

I would prefer the separation. openbsd and freebsd has too many differences.

ge9 · 2024-09-15T16:08:20Z

Is it okay if some part of code is shared (copy-pasted) between two xxxxbsd.rs files? (Maybe something is already shared between macos.rs and bsd.rs)

zonyitoo · 2024-09-16T17:17:03Z

Is it okay if some part of code is shared (copy-pasted) between two xxxxbsd.rs files? (Maybe something is already shared between macos.rs and bsd.rs)

Your choice. If those features are actually shared between all BSD systems, it could be shared.

ge9 · 2024-09-18T10:57:58Z

Now, I

separated bsd.rs into freebsd.rs and openbsd.rs
defined IP_RECVDSTPORT and IPV6_RECVDSTPORT in the global scope of openbsd.rs (as a temporary fix until these two are supported by libc)
added a boolean flag to confirm receiving both the address and port. But this still assumes we receive both the address and port only once respectively (in any order). If this is unacceptable, I'll fix it.
removed tun2 support

I think I've done all the remaining works (except for the flag issue)

zonyitoo · 2024-09-18T15:38:18Z

Current changes look good to me. But the code are not properly formatted. Run cargo fmt in the root directory and push it again.

ge9 · 2024-09-18T16:17:06Z

I applied rustfmt, except for the pfvar file. Is this okay?

zonyitoo · 2024-09-19T07:53:19Z

I applied rustfmt, except for the pfvar file. Is this okay?

It's Ok. But some of them are still not formatted.

ge9 · 2024-09-19T08:20:53Z

Oh, sorry, I forgot "git add"...
I pushed it again. Maybe is it fixed now?

crates/shadowsocks-service/src/local/redir/udprelay/sys/unix/openbsd.rs

zonyitoo reviewed Jun 24, 2024

View reviewed changes

ge9 force-pushed the master branch from 784ba6f to 938d296 Compare June 25, 2024 05:56

ge9 force-pushed the master branch from 938d296 to c71ce90 Compare June 25, 2024 10:16