Address already in use #352

gilcu3 · 2022-09-07T09:37:41Z

Version: current from master branch
I was trying to setup sslh to redirect connections from port 443 for several services, mainly an openssh, apache2, shadowsocks and syncthing.

config: sslh.cfg.txt
iptables rules: rules.sh.txt
log: sslh.log.txt
command: sudo ./sslh-select --config=sslh.cfg --foreground --user=sslh
syncthing connections from one machine are correctly redirected, while connections from the same software on another machine get errors:

probing for ssh
probed for ssh: PROBE_NEXT
probing for tls
matching [bep/1.0] with [bep/1.0]
probed for tls: PROBE_MATCH
common.c:281:bind:98:Address already in use
common.c:351:bind_peer:98:Address already in use

All other services work correctly from all machines.

Weirdly, changing the connection route (using a vpn service for example) in the client machine with the error makes the error go away. Also, on few occasions that I cannot reproduce, the error doesn't happen.

The text was updated successfully, but these errors were encountered:

licaon-kter · 2022-09-07T09:58:41Z

sudo netstat -tupan | grep LISTEN says what ?

gilcu3 · 2022-09-07T10:04:34Z

sudo netstat -tupan | grep LISTEN says what ?

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      625/sslh-select     
tcp        0      0 127.0.0.1:8384          0.0.0.0:*               LISTEN      498/syncthing       
tcp        0      0 [server_ip]:1080        0.0.0.0:*               LISTEN      421/ss-server       
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      352/mariadbd        
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      259/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      422/sshd: /usr/bin/ 
tcp        0      0 127.0.0.1:6800          0.0.0.0:*               LISTEN      495/aria2c          
tcp        0      0 127.0.0.54:53           0.0.0.0:*               LISTEN      259/systemd-resolve 
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      490/sshd: [my_user]    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      259/systemd-resolve 
tcp6       0      0 :::22000                :::*                    LISTEN      498/syncthing       
tcp6       0      0 :::5355                 :::*                    LISTEN      259/systemd-resolve 
tcp6       0      0 :::8443                 :::*                    LISTEN      319/httpd           
tcp6       0      0 :::80                   :::*                    LISTEN      319/httpd           
tcp6       0      0 ::1:3306                :::*                    LISTEN      352/mariadbd        
tcp6       0      0 :::22                   :::*                    LISTEN      422/sshd: /usr/bin/ 
tcp6       0      0 ::1:8000                :::*                    LISTEN      490/sshd: [my_user]       
tcp6       0      0 ::1:6800                :::*                    LISTEN      495/aria2c

licaon-kter · 2022-09-07T10:15:24Z

This machine is directly connected to the internet? One of its interfaces has external_ip_address assigned?

Why does your .cfg mix between external_ip_address and localhost ? Why not make them all localhost ?

gilcu3 · 2022-09-07T10:31:03Z

This machine is directly connected to the internet? One of its interfaces has external_ip_address assigned?

yes, the machine has a fixed ip address on the internet.

Why does your .cfg mix between external_ip_address and localhost ? Why not make them all localhost ?

In all enabled services I use the external ip in the address. (I think this is required in the transparent setup)

licaon-kter · 2022-09-07T10:59:57Z

It's not required afaik. Can you put external IP as listen host and localhost for all redirects and see what error do you get?

gilcu3 · 2022-09-07T12:10:04Z

I got exactly the same error by putting localhost in all redirects.

hexdump of incoming packet:
0x000000: 16 03 01 00 f6 01 00 00 f2 03 03 4b 6b 86 32 13 ...........Kk.2.
0x000010: a7 a9 a9 70 4c 62 9b 97 85 13 49 41 86 43 50 ec ...pLb....IA.CP.
0x000020: e4 27 91 db e8 4a 5c 0c 6a 1d 8b 20 bc d1 c2 17 .'...J\.j.. ....
0x000030: ec d8 1b 17 6a f3 5c 2b 24 a1 49 1e 49 f2 67 3f ....j.\+$.I.I.g?
0x000040: 1d d0 dd 57 0e 74 48 91 b7 97 1f cf 00 26 c0 2b ...W.tH......&.+
0x000050: c0 2f c0 2c c0 30 cc a9 cc a8 c0 09 c0 13 c0 0a ./.,.0..........
0x000060: c0 14 00 9c 00 9d 00 2f 00 35 c0 12 00 0a 13 01 ......./.5......
0x000070: 13 02 13 03 01 00 00 83 00 05 00 05 01 00 00 00 ................
0x000080: 00 00 0a 00 0a 00 08 00 1d 00 17 00 18 00 19 00 ................
0x000090: 0b 00 02 01 00 00 0d 00 1a 00 18 08 04 04 03 08 ................
0x0000a0: 07 08 05 08 06 04 01 05 01 06 01 05 03 06 03 02 ................
0x0000b0: 01 02 03 ff 01 00 01 00 00 10 00 0a 00 08 07 62 ...............b
0x0000c0: 65 70 2f 31 2e 30 00 12 00 00 00 2b 00 03 02 03 ep/1.0.....+....
0x0000d0: 04 00 33 00 26 00 24 00 1d 00 20 b0 ca b5 06 1f ..3.&.$... .....
0x0000e0: 27 20 f5 3f 3a 7c 6d e8 b3 e5 ba e5 99 61 31 06 ' .?:|m......a1.
0x0000f0: 54 e2 dd e6 41 50 22 8a 35 ea 01                T...AP".5..     
probing for ssh
probed for ssh: PROBE_NEXT
probing for tls
matching [bep/1.0] with [bep/1.0]
probed for tls: PROBE_MATCH
common.c:281:bind:98:Address already in use
common.c:351:bind_peer:98:Address already in use

Edit: Also very strange, under this configuration using localhost for redirects, sslh crashes randomly:

fish: Job 1, 'sudo -u sslh bash -c './sslh-se…' terminated by signal SIGSEGV (Address boundary error)

utoni · 2022-09-07T12:29:49Z

@gilcu3
Can try to compile sslh with make ENABLE_SANITIZER=y and give us a stack trace of the crash? Thanks!

gilcu3 · 2022-09-07T12:31:29Z

@gilcu3 Can try to compile sslh with make ENABLE_SANITIZER=y and give us a stack trace of the crash? Thanks!

EDITED: (the previous output was due to missing capabilities in the new binary)
Correct error:

AddressSanitizer:DEADLYSIGNAL
=================================================================
==2409==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000002e (pc 0x7fbdef382737 bp 0x000000000043 sp 0x7ffe82616b40 T0)
==2409==The signal is caused by a WRITE memory access.
==2409==Hint: address points to the zero page.
    #0 0x7fbdef382737  (/usr/lib/libc.so.6+0x5b737)
    #1 0x7fbdef382c74  (/usr/lib/libc.so.6+0x5bc74)
    #2 0x7fbdefe5ce9c in __interceptor_vfprintf /usr/src/debug/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1661
    #3 0x55be977bf1e9 in print_message /home/user/opt/sslh/sslh/log.c:113
    #4 0x55be977bff18 in hexdump /home/user/opt/sslh/sslh/probe.c:119
    #5 0x55be977c000e in probe_buffer /home/user/opt/sslh/sslh/probe.c:425
    #6 0x55be977ca27a in probe_client_protocol /home/user/opt/sslh/sslh/tcp-probe.c:48
    #7 0x55be977cb837 in probing_read_process /home/user/opt/sslh/sslh/tcp-listener.c:271
    #8 0x55be977cb1cd in main_loop /home/user/opt/sslh/sslh/sslh-select.c:194
    #9 0x55be977b9b6d in main /home/user/opt/sslh/sslh/sslh-main.c:285
    #10 0x7fbdef34a2cf  (/usr/lib/libc.so.6+0x232cf)
    #11 0x7fbdef34a389 in __libc_start_main (/usr/lib/libc.so.6+0x23389)
    #12 0x55be977b9d64 in _start ../sysdeps/x86_64/start.S:115

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/usr/lib/libc.so.6+0x5b737) 
==2409==ABORTING

utoni · 2022-09-07T12:37:18Z

I am more interested in stack traces of SIGSEGV's.

utoni · 2022-09-10T10:52:55Z

@gilcu3
Does PR #353 fix your problem?

gilcu3 · 2022-09-10T17:16:44Z

@utoni Yes, #353 fixes the problem of the log.
Still, the original problem of the issue remains.

utoni · 2022-09-10T17:30:11Z

According to your previous answers. I presume that you've already tried to force sslh to listen only on localhost?

gilcu3 · 2022-09-10T18:05:38Z

According to your previous answers. I presume that you've already tried to force sslh to listen only on localhost?

Yes, my current config file is

# Default Arch configuration
# You can find more examples in /usr/share/doc/sslh

timeout: 2;
transparent: true;

verbose-fd: 0; # file descriptor activity, open/close/whatnot
verbose-packets: 1; # hexdump packets on which probing is done
verbose-probe-info: 1; # what's happening during the probe process

listen:
(
    { host: "[external_ip_address]"; port: "443"; }
);

protocols:
(
     { name: "ssh"; service: "ssh"; host: "localhost"; port: "22"; },
     { name: "tls"; host: "localhost"; port: "22000"; alpn_protocols: ["bep/1.0"];},
     # { name: "openvpn"; host: "localhost"; port: "1194"; probe: "builtin"; },
     # { name: "xmpp"; host: "localhost"; port: "5222"; probe: "builtin"; },
     # { name: "http"; host: "localhost"; port: "80"; probe: "builtin"; },
     { name: "tls"; host: "localhost"; port: "8443";},
     { name: "socks5"; host: "localhost"; port: "1080";},
     # { name: "anyprot"; host: "localhost"; port: "8443"; probe: "builtin"; }
);

# vim:set ts=4 sw=4 et:

utoni · 2022-09-10T18:57:51Z

Does sslh work if you change

listen:
(
    { host: "[external_ip_address]"; port: "443"; }
);
´´´

to

listen:
(
    { host: "127.0.0.1"; port: "443"; }
);
´´´

or

listen:
(
    { host: "0.0.0.0"; port: "443"; }
);
´´´

gilcu3 · 2022-09-10T21:15:21Z

@utoni
Using 127.0.0.1 doesnt work at all.
Using 0.0.0.0 is working up to now, I will report back if I see the same connection error again or the other way around.
EDIT: After some time the error began appearing again and the client fails to connect.

yrutschle · 2022-09-21T19:14:15Z

Does it work with transparent disabled? (Sorry if it was mentionned, I couldn't see it..)

gilcu3 · 2022-09-22T09:21:08Z

Does it work with transparent disabled? (Sorry if it was mentionned, I couldn't see it..)

Yes, with transparent mode disabled it seems to work just fine. Right now I am testing with transparent mode active just for the other services I need, so not globally, I will report back in case of error, as sometimes it takes hours for errors to appear.

gilcu3 · 2022-09-26T15:02:52Z

I can confirm that this only happens if the service
{ name: "tls"; host: "localhost"; port: "22000"; alpn_protocols: ["bep/1.0"];},
is in transparent mode. Whether the other services are in transparent mode or not doesn't seem to affect it.

licaon-kter · 2022-09-26T15:45:44Z

@gilcu3 can you link to the internet standard that documents bep/1.0 ?

gilcu3 · 2022-09-26T15:50:21Z

@licaon-kter This link is the best I could find

yrutschle · 2022-09-26T16:42:02Z

This is a bit weird. It would mean bind() fails because the remote address/port has been bound already, which would mean we're getting two connections from the same IP/port?
(or that there is a terrible bug in sslh)

gilcu3 · 2022-09-27T08:03:52Z

This is a bit weird. It would mean bind() fails because the remote address/port has been bound already, which would mean we're getting two connections from the same IP/port? (or that there is a terrible bug in sslh)

Is there any way to check something like that? Because then it would/could be a syncthing problem. Currently, I see that when this error happens, the number of open files by the sslh daemon increases with time.

yrutschle · 2022-10-29T20:43:43Z

(sorry for the delay) -- yes, tcpdump/wireshark, check the syncthing traffic and see if there are two opening TCP handshakes with no closing first (you know, the SYN-SYN/ACK-ACK exchange).

I sort-of would expect some operating systems to object and close the connection that sslh sees. I am not even sure what we should be doing if the OS don't do it for us: try to break the connection in case of a re-connection?

gilcu3 · 2022-11-04T12:10:02Z

Is this log enough?
capture.pcapng

yrutschle · 2022-11-20T17:28:51Z

Re-opening as I closed by accident (commited something for another issue and got the issue number wrong)

yrutschle · 2022-11-20T17:34:29Z

So the pcap shows:

client opens a socket to server (SYN, SYN/ACK, ACK)
client sends ClientHello
server acks
server closes socket (RST) 2s later

I am not sure what to make of this: if the server closes the socket, I'd expect the port to be available and bind() should not fail... Unless sslh needs to first close the socket as well, in which case maybe the client re-attempts connection before sslh did that?
I am still unsure!

ftasnetamot · 2024-08-11T18:45:58Z

This should be solved with my proposed patch: #461 , documentation, why this fails comes with #462
When vacation time is over, this may come true.

yrutschle · 2024-08-15T21:06:06Z

FYI @ftasnetamot ' s patch just got merged (during vacation! :-) )

gilcu3 mentioned this issue Sep 23, 2022

segfault when bind() fails on transparent connection #355

Closed

yrutschle closed this as completed in f77d08e Nov 20, 2022

yrutschle reopened this Nov 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Address already in use #352

Address already in use #352

gilcu3 commented Sep 7, 2022

licaon-kter commented Sep 7, 2022

gilcu3 commented Sep 7, 2022

licaon-kter commented Sep 7, 2022

gilcu3 commented Sep 7, 2022 •

edited

Loading

licaon-kter commented Sep 7, 2022

gilcu3 commented Sep 7, 2022 •

edited

Loading

utoni commented Sep 7, 2022

gilcu3 commented Sep 7, 2022 •

edited

Loading

utoni commented Sep 7, 2022

utoni commented Sep 10, 2022

gilcu3 commented Sep 10, 2022

utoni commented Sep 10, 2022

gilcu3 commented Sep 10, 2022

utoni commented Sep 10, 2022

gilcu3 commented Sep 10, 2022 •

edited

Loading

yrutschle commented Sep 21, 2022

gilcu3 commented Sep 22, 2022

gilcu3 commented Sep 26, 2022

licaon-kter commented Sep 26, 2022

gilcu3 commented Sep 26, 2022

yrutschle commented Sep 26, 2022

gilcu3 commented Sep 27, 2022 •

edited

Loading

yrutschle commented Oct 29, 2022

gilcu3 commented Nov 4, 2022

yrutschle commented Nov 20, 2022

yrutschle commented Nov 20, 2022

ftasnetamot commented Aug 11, 2024

yrutschle commented Aug 15, 2024

Address already in use #352

Address already in use #352

Comments

gilcu3 commented Sep 7, 2022

licaon-kter commented Sep 7, 2022

gilcu3 commented Sep 7, 2022

licaon-kter commented Sep 7, 2022

gilcu3 commented Sep 7, 2022 • edited Loading

licaon-kter commented Sep 7, 2022

gilcu3 commented Sep 7, 2022 • edited Loading

utoni commented Sep 7, 2022

gilcu3 commented Sep 7, 2022 • edited Loading

utoni commented Sep 7, 2022

utoni commented Sep 10, 2022

gilcu3 commented Sep 10, 2022

utoni commented Sep 10, 2022

gilcu3 commented Sep 10, 2022

utoni commented Sep 10, 2022

gilcu3 commented Sep 10, 2022 • edited Loading

yrutschle commented Sep 21, 2022

gilcu3 commented Sep 22, 2022

gilcu3 commented Sep 26, 2022

licaon-kter commented Sep 26, 2022

gilcu3 commented Sep 26, 2022

yrutschle commented Sep 26, 2022

gilcu3 commented Sep 27, 2022 • edited Loading

yrutschle commented Oct 29, 2022

gilcu3 commented Nov 4, 2022

yrutschle commented Nov 20, 2022

yrutschle commented Nov 20, 2022

ftasnetamot commented Aug 11, 2024

yrutschle commented Aug 15, 2024

gilcu3 commented Sep 7, 2022 •

edited

Loading

gilcu3 commented Sep 7, 2022 •

edited

Loading

gilcu3 commented Sep 7, 2022 •

edited

Loading

gilcu3 commented Sep 10, 2022 •

edited

Loading

gilcu3 commented Sep 27, 2022 •

edited

Loading