Could this be a new functionality for sslh? Or has anybody ideas, what to use?

[ftasnetamot]

Hello,

I am still in process upgrading old server configurations, make them stable and reach a better level of functionality.
Currently I am dealing with fail2ban, and I am unhappy, that the reaction a fail2ban block just looks like a broken server. That will be still no issue for 90-99% of the connections, beeing blocked by this way, but it will affect real users/customers/... as well. Especially, when the block occurs after 2 or 3 fails. So my current idea is, having a fail2ban instance, which has no banaction, but filters the log an fills its database with the result. Now I am able for example in exim, to deny incoming connections immediately, but with an error message, telling that this connection is blocked because of... and for so long ... and e.g. an additional information, how to fix

With this configuration, I protect the application against most of the usual threats, as I don't accept any data, which could lead to an exploit/account hack/...
But legit users will know, why they are blocked. Now I can run a second fail2ban jail, which just reads the related logs to this block (a very leightweight RegEx), and do the usual firewall block, but now for a real longer time. This allows differentation between legit clients, which will follow rules, posted in the deny message, and therefore not longer appearing with higher numbers in the log and those, who really earn beeing blocked for long.

Ok, this works fine with exim but when I looked at dovcot, it figured out, that dovecot does not support dnsbl/rbl or any other dynamic feedback mechanism, checking if a IP address should be denied.
Ok, I thought: No problem, let me use nginx as proxy, this is capable of transparent proxying like sslh and supports many protocols. So I guessed, this could be my swiss army knife, to fix all those applications, not able to talk to sql-databses or dns servers. Next big dissapointment: nginx supports this possibilities only in paid plus edition. So much for FOSS ...

So I am coming back to think: May it be a possible addition for sslh, that it can do decisions, based on the incoming ip and dnsbl queries, which target to be adressed? Doing this, I could define dummy targets, which will just give a short information, that the service is currently unavailable, and sslh will decide, whether to connect to the real or to the dummy target.

This could be adapted in a way, that sslh may listen of several incoming addresses/ports, and the routing decision can be just made for the incoming connection.
E.g.
If sslh would listen on ports 110 (pop3), 143 (imap2), 993 (imaps) and 995 (pops)
the target would either be the real dovecot or the fake dovecot, telling that no connections possible.

This service would be complete protocol agnostic, as decisions are only made on the source ip and destination port. Therefore it would help for almost any application.

The disadvantage is, that you need to configure dummy-applications, which are either configuration clones of the real application, or some stunnels to simple scripts, just answering to the first protocol command.

If anybody has better ideas, I really would glad to hear!

[yrutschle]

I am not sure I follow everything you're trying to do, but:

• Everytime someone mentions filtering based on source IP, I think of libwrap (because I don't want to re-write IP parsing and comparing :) ); then I remember it's already supported in sslh. I am not sure it works with the support that's already there, but I would expect something like:

     { name: "ssh"; service: "ssh1"; host: "localhost"; port: "22";  },
     { name: "ssh"; service: "ssh2"; host: "localhost"; port: "2222";  },

to work: with appropriate service definitions in /etc/hosts.allow, if ssh1 matches it would forward to 22, if ssh2 matches if would forward to 2222.
If that does not work, I except it would minor changes to the matching logic (and I'll have a look :-) ).

• I have never used dnsbl, but you could definitely change DNS values for some protocols using resolve_on_forward:

     { name: "ssh"; service: "ssh1"; host: "ssh.example.net"; port: "22"; resolve_on_forward: true  },

and then every time sslh forwards to ssh.example.net, it will resolve the name, which means calling the DNS (that's why it's not on by default: it'll generate additional load). Then you can change the IP that points to, which would allow you to alternate between the real and the fake server.

Does this help?

[ftasnetamot]

Unfortunately this does not work, as I need to do a gethostbyname() call, and do my decision based on this result. The good old Wietse Venema tcp-wrapper, was one of the first tools, I checked. But it depends on static files and can't do dynamic queries.

I have now a very first working configuration with openresty (a nginx clone), where I was able to encapsulate the core gethostbyname() C-call in lua and get results on this. 'resolve_on_forward' can't do that. A dnsblocklist/realtimeblocklist works like this:
the sender IP, e.g. 12.34.56.78 gets queried against the blacklist server and reversed for that. So I do a gethostbyname() query on '78.56.34.12.blocklist.mydomain.top`. If I get no result, its good news, but if I get results like 127.0.0.1, 127.0.0.2, 127.0.0.3, ... indicates this, that the ip is blocked and the value in the last position tells the reason.
So my solution only acts on result or not, and does (currently) no further decision, why this is blocked.

A short snippet of the openresty/nginx configuration shows how it works:

###this all happens in a stream context, not http!!
  map $server_port $good_target {
      80           192.168.255.254:80; #http
      443          192.168.255.254:442; #sslh->https or ssh or ...
      110          192.168.255.254:110; #pop
      143          192.168.255.254:143; #imap
      993          192.168.255.254:993; #imaps
      995          192.168.255.254:995; #pops
  }

  map $server_port $bad_target {
      80           192.168.255.254:5080;
      443          192.168.255.254:5443;
      110          192.168.255.254:5110;
      143          192.168.255.254:5143;
      993          192.168.255.254:5993;
      995          192.168.255.254:5995;
  }

##protection frontend
  server {
    access_log /var/log/nginx/protection.log reduced;
    listen xx.xx.xx.xx:80;
    listen xx.xx.xx.xx:443;
    listen xx.x.xx.xx:110;
    listen xx.xx.xx.xx:143;
    listen xx.xx.xx.xx:993;
    listen xx.xx.xx.xx:995;
    set $final_destination "";
    preread_by_lua_block {
      ngx.var.final_destination = require "block".decide_on_dest( "blacklist.mydomain.top",
                                           ngx.var.remote_addr,
                                           ngx.var.good_target, ngx.var.bad_target);
    }
    proxy_connect_timeout 5s;
    proxy_timeout 3m;
    proxy_bind $remote_addr transparent;
    proxy_pass $final_destination;
 }

When openresty initiates the connection to $final_destination it calls my lua function, as this is tied by required to this variable.
When using the variables $good_target and $bad_target Openresty calls the map function, and using the value, fitting to the incoming port of the current connection.
The decide_on_rest-function finally does the gethostbyname()-call, and if there are results (which means the ip is blacklisted) it sets the final-destination to the bad-target, where you get the error message in a protocol depended way (pop,imap,http,...), otherwise it redirects to the real destination (good_target).
The line proxy_bind $remote_addr transparent; works exactly, like sslh does, and my reflection routing works for this just in the same way. I have it now daisy-chained with sslh, which is now also in the second line listening on dummy0, but unfortunately every 4th to 10th connections gets interrupted, with error-messages, that there is a conflict with the used address. I assume, that it has something to do with the client port used in the connection.

Theorectical it is completly ok, that two applications (openresty and sslh) are using the same IP for outgoing connections (even if this is the IP from the client), This is the way, how every local connections of different command will look like. A more straight-forward test configuration could look like:

  server { ####  test conflicts with chaining sslh
    access_log /var/log/nginx/sslh-test.log
    listen xx.xx.xx.xx:443;
    proxy_connect_timeout 5s;
    proxy_timeout 3m;
    proxy_bind $remote_addr transparent;
    proxy_pass 192.168.255.254:443; ##sslh on dummy0
 }

I have some more ideas, what I can test, I will come back to you in the next days, maybe with questions on binding options for the outgoing connection.

Just to understand, what I do in the lua code, here my current 'block.lua' for completeness:

local ffi = require 'ffi'
local cdef = ffi.cdef
local C = ffi.C
-- redefine C-header of hostent, gethostbyname, and addr_t
cdef([[
  struct hostent {
    char  *h_name;            /* hostname */
    char **h_aliases;         /* alias list */
    int    h_addrtype;        /* address type */
    int    h_length;          /* length of address */
    char **h_addr_list;       /* list of addresses */
  };
  struct hostent *gethostbyname(const char *name);
  struct addr_t { uint8_t b1, b2, b3, b4; }; 
]])
local get_host_by_name = C.gethostbyname

local _M = {}

function _M.decide_on_dest( blocklist, ip2check, real_target, block_target)
-- blocklist:    name of blocklistserver inluding some key, if needed: e.g. blocklist.myDomain.top, keystring.zen.dq.spamhaus.net
-- ip2check:     IP address, which should be checked. In most cases $remote_addr
-- real_target:  service, which should be secured through blocklist check
-- block_target: service, which will be called for bad addresses, to respond with error code
  -- reverse ip2check byte tuples and concatenate with blocklist servername  
  local a,b,c,d = ip2check:match("([%d]+).([%d]+).([%d]+).([%d]+)")
  local dnsblhost = d.."."..c.."."..b.."."..a.."."..blocklist
  local final_destination
  local dnsbl_result = get_host_by_name(tostring(dnsblhost))
  if dnsbl_result == nil then
    -- no result is good news
    final_destination = real_target
  else
    final_destination = block_target
  end
  return final_destination
end

return _M

[ftasnetamot]

Hi again,

I am a step further, after looking into your source. In common.c you describe the issue, as it looks, you found that happening with stunnel also. This is something, I guessed, that this could be the problem.
Read this very interesting blogpost, as Cloudflare describes, how that can be solved.

It should be no issue, if sslh is connecting from 192.168.255.254:7777 to 192.168.255.254:443 while nginx is connecting from 192.168.255.254:7777 to 192.168.255.254:22, because the 4-tuple connections are different. When you however bind, like I have seen in your code (and perhaps also in stunnel) with a port, the two-tupel creates a EADDRINUSE error.
As the Operating System does not know in this situation, how you will use this socket. For listening connections it will fail, as the ip:port tuple is already used. For outgoing connections however, this combination is ok, but as this is not known in this state, you receive the error message.

With the 'IP_BIND_ADDRESS_NO_PORT' option, this situation could be avoided, and the connection can be established. With this option, the test against potential conflicts is done first, when establishing the connection, and the validity of the four-way-tuple can be checked. And as long, as not both applications will connect to the same target, there will be no longer a problem.
I am not so deep in C-programming, that I will try to create an pull-request at the current time. Because I must fully understand your code before, but I guess, if you read the linked article, you get the idea. So feel free, to implement. A solution from my side would last, as I have currently some other priorities.

90 minutes later:
I have now reread the linked blog article and had a closer look to your source code. I think, that the best solution would be, to add SO_REUSEADDR as socket option on first try, and IP_BIND_ADDRESS_NO_PORT to the second try, with port set to zero.

This would be similar to the connectx function, Cloudflare mentions. So in in first try, SO_REUSEADDR would make it possible, to reuse also the clients port number, and if that fails, IP_BIND_ADDRESS_NO_PORT would give the second option better cards.

Will sleep over it and look again tomorrow. But from my current understanding the change would be not more, than just two adapted setsockopt() lines.

[ftasnetamot]

It will not go out of my head, so I just did those changes and will give them a try the next days.

diff --git a/common.c b/common.c
index b04066a..7df8bff 100644
--- a/common.c
+++ b/common.c
@@ -244,11 +244,12 @@ int is_same_machine(struct addrinfo* from)
 /* Transparent proxying: bind the peer address of fd to the peer address of
  * fd_from */
 #define IP_TRANSPARENT 19
+#define IP_BIND_ADDRESS_NO_PORT 24
 int bind_peer(int fd, int fd_from)
 {
     struct addrinfo from;
     struct sockaddr_storage ss;
-    int res, trans = 1;
+    int res, enable = 1, disable = 0;
 
     memset(&from, 0, sizeof(from));
     from.ai_addr = (struct sockaddr*)&ss;
@@ -264,15 +265,17 @@ int bind_peer(int fd, int fd_from)
         return 0;
 
 #ifndef IP_BINDANY /* use IP_TRANSPARENT */
-    res = setsockopt(fd, IPPROTO_IP, IP_TRANSPARENT, &trans, sizeof(trans));
+    res = setsockopt(fd, IPPROTO_IP, IP_TRANSPARENT, &enable, sizeof(enable));
     CHECK_RES_DIE(res, "setsockopt IP_TRANSPARENT");
+    res = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &enable, sizeof(enable));
+    CHECK_RES_DIE(res, "setsockopt SO_REUSEADDR");
 #else
     if (from.ai_addr->sa_family==AF_INET) { /* IPv4 */
-        res = setsockopt(fd, IPPROTO_IP, IP_BINDANY, &trans, sizeof(trans));
+        res = setsockopt(fd, IPPROTO_IP, IP_BINDANY, &enable, sizeof(enable));
         CHECK_RES_RETURN(res, "setsockopt IP_BINDANY", res);
 #ifdef IPV6_BINDANY
     } else { /* IPv6 */
-        res = setsockopt(fd, IPPROTO_IPV6, IPV6_BINDANY, &trans, sizeof(trans));
+        res = setsockopt(fd, IPPROTO_IPV6, IPV6_BINDANY, &enable, sizeof(enable));
         CHECK_RES_RETURN(res, "setsockopt IPV6_BINDANY", res);
 #endif /* IPV6_BINDANY */
     }
@@ -294,6 +297,10 @@ int bind_peer(int fd, int fd_from)
          * have changed, but most people won't care.
          * Also note that stunnel uses the same logic for the same situation.
          */
+        res = setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &disable, sizeof(disable));
+        CHECK_RES_DIE(res, "setsockopt SO_REUSEADDR");
+        res = setsockopt(fd, IPPROTO_IP, IP_BIND_ADDRESS_NO_PORT, &enable, sizeof(enable));
+        CHECK_RES_DIE(res, "setsockopt IP_BIND_ADDRESS_NO_PORT");
         ((struct sockaddr_in *)from.ai_addr)->sin_port = 0;
         res = bind(fd, from.ai_addr, from.ai_addrlen);
         CHECK_RES_RETURN(res, "bind", res);

[ftasnetamot]

Damn, fighting at another front:
When building sslh from source, I can't go transparent:
sslh[204785]: common.c:269:setsockopt IP_TRANSPARENT: Operation not permitted
It dies with the first setsockopt()-call, when it tries to go transparent, just one line before I would like to do set SO_REUSEADDR.
I compiled finally just the current HEAD version, without any change, same result.
I have set all capabilities to sslh_fork and even set the sui-bit:

sslh-fork = cap_net_bind_service,cap_net_admin,cap_net_raw,cap_sys_admin+ep
-rwsr-xr-x 1 root root 589504 Jul 26 12:42 sslh-fork

but: the problem consist. Going back to my distribution binary, transparency works, even without any capabilities on the binary.
Do I miss some configuration options?
Unfortunately there is no documentation about best ways, to configure and compile the code.

[ftasnetamot]

While I had first my setup in this way, as this is the setup, I need for my idea to become real

                       +-->[dummy0:5443]->nginx-destination-for-bad-hosts
                       |       
[publicIP:443]->nginx--+                        +-->[dummy0:22]->sshd
                       |                        |
                       +-->[dummy0:1443]->sslh--+
                                                |
                                                +-->[dummy0:443]->apache

I tried now with changed roles, and had sslh (the distribution binary) doing the job as frontman, and assigned nginx to the second row:

                      +-->[dummy0:22]->sshd
                      |                         
[publicIP:443]->sslh--+                         +-->[dummy0:443]->apache
                      |                         |
                      +-->[dummy0:1443]->nginx--+
                                                |
                                                +-->[dummy0:5443]->nginx-destination-for-bad-hosts

This is working fine and Openresty/nginx doing its homework perfect and second in chain:

tcp        0      0 PUBLIC_IP:443   CLIENT_IP:49010 ESTABLISHED 210811/sslh         
tcp        0      0 CLIENT_IP:49010 DUMMY0:1443     ESTABLISHED 210811/sslh         
tcp        0      0 DUMMY0:1443     CLIENT_IP:49010 ESTABLISHED 207973/nginx: worker
tcp        0      0 CLIENT_IP:49010 DUMMY0:443      ESTABLISHED 207973/nginx: worker
tcp        0      0 DUMMY0:443      CLIENT_IP:49010 ESTABLISHED 202895/apache2      

tcp        0    588 PUBLIC_IP:443   CLIENT_IP:63495 ESTABLISHED 208131/sslh         
tcp        0      0 CLIENT_IP:63495 DUMMY0:22       ESTABLISHED 208131/sslh         
tcp        0      0 DUMMY0:22       CLIENT_IP:63495 ESTABLISHED 208132/sshd: user

As you can see, the client IP:PORT combinations stays the same through the chain, as the 4-value-tuple is always different.

That looks like, that it may be worth, to dig into the code of nginx, as they seem to do the things in a good way.

Unfortunately, for my solution, I need nginx in the first row, and sslh behind.

From the theory, coming from the Cloudflare article, my idea what to change, should work.

But unfortunately, all my tries to compile a own sslh will fail, because the binary gets transparency denied :-(
I need first an self-compiled binary, which will work the same as my distribution versions sslh 1.20-1 and sslh 1.20-1+b2

What I am doing wrong? Why can't the self-compiled binaries can't go transparent? Is there a configuration file missing?
I tried now compiling the current HEAD, and sslh-1.23.1 and sslh-2.1.2, none of them will go transparent.

I may miss a very obvious thing, but i have currently no clue.

[ftasnetamot]

OK, looks like I have found the reason:

I am starting sslh with the parameter --user sslh, to make sure, that sslh drops down to user sslh after establishing the connection.
Opening an issue!

[ftasnetamot]

Ok, besides the problem, why the "--user" parameter is not working, I am able to run the modified sslh without any problems behind nginx.
Now both combinations of nginx and sslh are working. I have done a stress test, with sslh behind nginx, which is that position, where the precursor has already bound the client:ip to the dummy0 and the modification needs to jump in.
I have run 1000 http-requests and about 200 ssh connects in 11 minutes, and have not seen any conflicting connection. So the solution from the mentioned blogpost solves the problem.

I did another test with stunnel, and yes: stunnel is also buggy. When stunnel runs in front of the modified sslh, everything is fine. But the other way round, the connections are failing as stunnel can't solve the conflict, with the ip-address sslh has already bound to dummy0.

The good news: nginx can do everything, stunnel does as well and works in both positions. For just having an proxytunnel-endpoint, both variants (the native nginx and openresty) will do the job.

[ftasnetamot]

OK, closing this discussion, as I have a solution with my proposed fix to sslh. As soon, as sslh can be used at any position in a daisy-chain of ip-transparent applications, my openresty sultion works.
Openresty is in this case a leightweight application firewall, which accepts connections at first place, and does rbl/dnsbl checks, to decide, where to route this connection to. Sslh is one of the possible targets, routing itself ip-transparent connections to other applications.
This solution complies with the core UNIX concept of small applications, doing their job and beeing able to combined to more compley solutions.