Updating restriction file sometimes doesn't work, restrictions variable not updated #368

christopherhibbert · 2024-10-11T16:30:36Z

Describe the bug

Updating restriction file sometimes doesn't work, restrictions variable not updated

To Reproduce

We are running wstunnel in kubernetes. We're using a configmap which contains the restrictions file, and updating it using:
kubectl edit cm .....

We've added extra logging in mod.rs:
extra_debug.patch

impl RestrictionsRules {
    pub fn from_config_file(config_path: &Path) -> anyhow::Result<Self> {
        let restrictions: Self = serde_yaml::from_reader(BufReader::new(File::open(config_path)?))?;

        // Deserialize the YAML into a serde_yaml::Value to print it
        let yaml_content: serde_yaml::Value = serde_yaml::from_reader(BufReader::new(File::open(config_path)?))?;
        // Print out the contents of the file
        println!("YAML file content: {:?}", yaml_content);

        Ok(restrictions)
    }

We see that this logging is printing out the updated restrictions file.

But later in validate_tunnel (to which we've also added some extra logging), the OLD restrictions file contents being printed out and used, resulting in the tunnel connection being rejected.

Expected behavior

When the restrictions file is updated wstunnel should start checking new connections against the updated restrictions file.

Your wstunnel setup

Paste your logs of wstunnel, started with --log-lvl=DEBUG, and with the command line used

In the logs below with our extra debug, you see:

YAML file content: Mapping {"restrictions": Sequence [Mapping {"name": String("lv-encode"), "description": String("This is the config for lv-encode"), "match": Sequence [TaggedValue { tag: !PathPrefix, value: String("^990988e604e64195$") }], "allow": Sequence [TaggedValue { tag: !ReverseTunnel, value: Mapping {"protocol": Sequence [String("Tcp")], "port": Sequence [Number(49152)]} }]}]}
SNIP
2024-10-11T16:13:14.638326Z WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fe3f-7a30-89e8-2e9d351cc696" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }

server:

2024-10-11T16:13:11.580525Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-f24d-7713-aca9-ace2afdb99ca" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }
2024-10-11T16:13:11.580529Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-f24d-7713-aca9-ace2afdb99ca" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Rejecting connection with not allowed destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:12.599013Z  INFO cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel: wstunnel::tunnel::server::server: Request X-Forwarded-For: 10.244.0.1
2024-10-11T16:13:12.599050Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-f649-7110-99b9-d63ca5eeea95" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Got connection - destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:12.599056Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-f649-7110-99b9-d63ca5eeea95" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }
2024-10-11T16:13:12.599060Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-f649-7110-99b9-d63ca5eeea95" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Rejecting connection with not allowed destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:13.618212Z  INFO cnx{peer="[::ffff:10.244.0.43]:44532"}:tunnel: wstunnel::tunnel::server::server: Request X-Forwarded-For: 10.244.0.1
2024-10-11T16:13:13.618248Z  WARN cnx{peer="[::ffff:10.244.0.43]:44532"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fa44-7ef1-88d7-f3e83fd544ad" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Got connection - destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:13.618254Z  WARN cnx{peer="[::ffff:10.244.0.43]:44532"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fa44-7ef1-88d7-f3e83fd544ad" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }
2024-10-11T16:13:13.618258Z  WARN cnx{peer="[::ffff:10.244.0.43]:44532"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fa44-7ef1-88d7-f3e83fd544ad" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Rejecting connection with not allowed destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:14.561030Z TRACE notify::inotify: inotify event: Event { wd: WatchDescriptor { id: 9, fd: (Weak) }, mask: ATTRIB, cookie: 0, name: None }    
2024-10-11T16:13:14.561062Z TRACE wstunnel::restrictions::config_reloader: Received event: Event {
    kind: Modify(
        Metadata(
            Any,
        ),
    ),
    paths: [
        "/wstunnelserver-authfile/wstunnelserver-authfile.yaml",
    ],
    attr:tracker: None,
    attr:flag: None,
    attr:info: None,
    attr:source: None,
}    
2024-10-11T16:13:14.561085Z TRACE notify::inotify: inotify event: Event { wd: WatchDescriptor { id: 9, fd: (Weak) }, mask: DELETE_SELF, cookie: 0, name: None }    
2024-10-11T16:13:14.561121Z TRACE wstunnel::restrictions::config_reloader: Received event: Event {
    kind: Remove(
        File,
    ),
    paths: [
        "/wstunnelserver-authfile/wstunnelserver-authfile.yaml",
    ],
    attr:tracker: None,
    attr:flag: None,
    attr:info: None,
    attr:source: None,
}    
2024-10-11T16:13:14.561136Z  WARN wstunnel::restrictions::config_reloader: Restriction config file has been removed, trying to re-set a watch for it
2024-10-11T16:13:14.561179Z TRACE notify::inotify: inotify event: Event { wd: WatchDescriptor { id: 9, fd: (Weak) }, mask: IGNORED, cookie: 0, name: None }    
2024-10-11T16:13:14.561205Z TRACE notify::inotify: removing inotify watch: /wstunnelserver-authfile/wstunnelserver-authfile.yaml    
2024-10-11T16:13:14.561294Z TRACE notify::inotify: adding inotify watch: /wstunnelserver-authfile/wstunnelserver-authfile.yaml    
2024-10-11T16:13:14.561364Z TRACE wstunnel::restrictions::config_reloader: Received event: Event {
    kind: Create(
        Any,
    ),
    paths: [
        "/wstunnelserver-authfile/wstunnelserver-authfile.yaml",
    ],
    attr:tracker: None,
    attr:flag: None,
    attr:info: None,
    attr:source: None,
}    
YAML file content: Mapping {"restrictions": Sequence [Mapping {"name": String("lv-encode"), "description": String("This is the config for lv-encode"), "match": Sequence [TaggedValue { tag: !PathPrefix, value: String("^990988e604e64195$") }], "allow": Sequence [TaggedValue { tag: !ReverseTunnel, value: Mapping {"protocol": Sequence [String("Tcp")], "port": Sequence [Number(49152)]} }]}]}
2024-10-11T16:13:14.561574Z  INFO wstunnel::restrictions::config_reloader: Restrictions config file has been reloaded
YAML file content: Mapping {"restrictions": Sequence [Mapping {"name": String("lv-encode"), "description": String("This is the config for lv-encode"), "match": Sequence [TaggedValue { tag: !PathPrefix, value: String("^990988e604e64195$") }], "allow": Sequence [TaggedValue { tag: !ReverseTunnel, value: Mapping {"protocol": Sequence [String("Tcp")], "port": Sequence [Number(49152)]} }]}]}
2024-10-11T16:13:14.561704Z  INFO wstunnel::restrictions::config_reloader: Restrictions config file has been reloaded
2024-10-11T16:13:14.638273Z  INFO cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel: wstunnel::tunnel::server::server: Request X-Forwarded-For: 10.244.0.1
2024-10-11T16:13:14.638315Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fe3f-7a30-89e8-2e9d351cc696" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Got connection - destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:14.638326Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fe3f-7a30-89e8-2e9d351cc696" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }
2024-10-11T16:13:14.638333Z  WARN cnx{peer="[::ffff:10.244.0.43]:44506"}:tunnel{forwarded_for="10.244.0.1" id="01927c5a-fe3f-7a30-89e8-2e9d351cc696" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Rejecting connection with not allowed destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:15.657408Z  INFO cnx{peer="[::ffff:10.244.0.43]:45022"}:tunnel: wstunnel::tunnel::server::server: Request X-Forwarded-For: 10.244.0.1
2024-10-11T16:13:15.657446Z  WARN cnx{peer="[::ffff:10.244.0.43]:45022"}:tunnel{forwarded_for="10.244.0.1" id="01927c5b-023b-7a33-b83c-3b57562a0a4c" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Got connection - destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:15.657455Z  WARN cnx{peer="[::ffff:10.244.0.43]:45022"}:tunnel{forwarded_for="10.244.0.1" id="01927c5b-023b-7a33-b83c-3b57562a0a4c" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }
2024-10-11T16:13:15.657461Z  WARN cnx{peer="[::ffff:10.244.0.43]:45022"}:tunnel{forwarded_for="10.244.0.1" id="01927c5b-023b-7a33-b83c-3b57562a0a4c" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Rejecting connection with not allowed destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:16.676299Z  INFO cnx{peer="[::ffff:10.244.0.43]:45020"}:tunnel: wstunnel::tunnel::server::server: Request X-Forwarded-For: 10.244.0.1
2024-10-11T16:13:16.676347Z  WARN cnx{peer="[::ffff:10.244.0.43]:45020"}:tunnel{forwarded_for="10.244.0.1" id="01927c5b-0636-7b60-bb0d-d98bc25884ec" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Got connection - destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"
2024-10-11T16:13:16.676358Z  WARN cnx{peer="[::ffff:10.244.0.43]:45020"}:tunnel{forwarded_for="10.244.0.1" id="01927c5b-0636-7b60-bb0d-d98bc25884ec" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Restrictions: RestrictionsRules { restrictions: [] }
2024-10-11T16:13:16.676365Z  WARN cnx{peer="[::ffff:10.244.0.43]:45020"}:tunnel{forwarded_for="10.244.0.1" id="01927c5b-0636-7b60-bb0d-d98bc25884ec" remote="[::]:49152"}: wstunnel::tunnel::server::utils: Rejecting connection with not allowed destination: RemoteAddr { protocol: ReverseTcp, host: Ipv6(::), port: 49152 } - remote path prefix: "990988e604e64195"

Desktop (please complete the following information):

OS: Ubuntu
Version 20.04LTS

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

JamesTGrant · 2024-10-12T08:13:54Z

Here’s my guess… @erebe, I am a newbie at Rust. I think that the problem is that the variable ‘restrictions’ is not created as mutable (I think the fix is to make it mutable, for example: let mut restrictions).
When subsequent assignment(s) is(are) made, the original variable is still in use so what happens is there is an another variable with the same name, but the things we care about that are using the restrictions var are using the pointers pertaining to the original variable.
Whilst the variable remains in-scope, redefining it won’t work unless it is created as mutable.
What do you think?

erebe · 2024-10-12T08:52:14Z

Hello,

Thanks for reporting the issue ;-)

I think I understand where the problem is coming from, it is the same mistake I made when you reported the issue with the logs ever increasing on tunnel rejection.

For now, the update/use of the new reloaded restrictions is done only once at the beginning of a new TCP connections. And in your case you seem to have a reverse proxy in front of wstunnel that multiplex/reuse the TCP connections. So you end-up in the situation where the restrictions file is reloaded, but never updated for new requests that re-use the existing connections.

I will provide a fix next week regarding this, to update the restrictions before each request and not only at the beginning of the new tcp connections.

christopherhibbert · 2024-10-12T09:42:39Z

Ah, that makes sense, we're using an nginx ingress in kubernetes between wstunnel client and server, so yes makes sense that it is hiding some information. Thanks again for your very speedy response and analysis.

instead of only once per conections, to avoid using a stale restriction config when multiple request arrive on the same tcp stream.

erebe · 2024-10-14T06:45:51Z

It should be fixed in the latest release https://github.com/erebe/wstunnel/releases/tag/v10.1.5
Let me know if it is not the case

christopherhibbert · 2024-10-14T07:38:23Z

Will test today :-) Thanks

christopherhibbert · 2024-10-14T22:44:22Z

Fix looks good 💯 👍👍👍👍👍👍. Thank you ☺️

erebe · 2024-10-15T19:27:34Z

Awesome, thank you for reporting the issue ;)

christopherhibbert added the bug label Oct 11, 2024

erebe added a commit that referenced this issue Oct 14, 2024

fix(#368): Update the restrictions before each request

4b43dfc

instead of only once per conections, to avoid using a stale restriction config when multiple request arrive on the same tcp stream.

erebe closed this as completed Oct 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating restriction file sometimes doesn't work, restrictions variable not updated #368

Updating restriction file sometimes doesn't work, restrictions variable not updated #368

christopherhibbert commented Oct 11, 2024 •

edited

Loading

JamesTGrant commented Oct 12, 2024 •

edited

Loading

erebe commented Oct 12, 2024 •

edited

Loading

christopherhibbert commented Oct 12, 2024

erebe commented Oct 14, 2024

christopherhibbert commented Oct 14, 2024

christopherhibbert commented Oct 14, 2024

erebe commented Oct 15, 2024

Updating restriction file sometimes doesn't work, restrictions variable not updated #368

Updating restriction file sometimes doesn't work, restrictions variable not updated #368

Comments

christopherhibbert commented Oct 11, 2024 • edited Loading

Describe the bug

To Reproduce

Expected behavior

Your wstunnel setup

Desktop (please complete the following information):

JamesTGrant commented Oct 12, 2024 • edited Loading

erebe commented Oct 12, 2024 • edited Loading

christopherhibbert commented Oct 12, 2024

erebe commented Oct 14, 2024

christopherhibbert commented Oct 14, 2024

christopherhibbert commented Oct 14, 2024

erebe commented Oct 15, 2024

christopherhibbert commented Oct 11, 2024 •

edited

Loading

JamesTGrant commented Oct 12, 2024 •

edited

Loading

erebe commented Oct 12, 2024 •

edited

Loading