Skip to content

Mesh of Meshes: tinc & batman

Jump to bottom Edit New page
Franz Pletz edited this page Apr 27, 2015 · 1 revision

Motivation

We currently have a very big batman-adv network with over 450 nodes and an average of 400 clients (April 2015). Being a batman-adv network, all devices share in the same layer 2 network and broadcast domain. All internal network interfaces are bridged. We run 5 VPN gateways that also run batman-adv over VPN links with nodes connected to the internet.

The batman-adv broadcast traffic over the VPN links alone is currently about 6-10kb/s in both directions. Additionally, ARP requests & Neighbor Solicitations and other broadcast traffic for the whole network also need to be forward to the whole network. This overhead is a huge burden for node owners with a slow internet connection or on a volume-based data plan. But these messages must also be transmitted over wireless mesh links and thus reducing precious air time for real data.

Traffic between nodes that are only meshing over VPN is forwarded through the gateways because fastd, our current VPN software, is not able to mesh with all connected VPN peers directly.

Proposed Solution: Mesh of Meshes

Overview

Instead of running one big layer 2 network with batman-adv, we will only do batman-adv on the wireless mesh links. Thus every node will mesh with other nodes just like before but won't see any other batman-adv neighbors that are not in range or are reachable over a VPN link. This will drop all unnecessary batman-adv broadcast traffic.

To preserve support for meshing over VPN, fastd is replaced by tinc, which supports meshing with other VPN peers. This meshed network is used for efficiently routing traffic between nodes, gateways, services and the internet. This network can be managed with DHCP (nodes do NAT for IPv4) and DHCPv6 (nodes do prefix delegation). IPv6 subnet routes to nodes can be distributed with tinc.

Node Configuration

Nodes will have two different modes of operation, with and without connectivity to the Internet and thus the tinc mesh network. Compared to the old setup, the gateway functionality is now implemented on the nodes with tinc connectivity instead of machines in the internet. These machines are of course still used for routing traffic from the internal network to the internet but with regular layer 3 routing techniques instead of batman-adv.

Common

  • bat0: batman-adv only on wireless mesh interfaces
  • br-client: bridge with all client interfaces (wireless, wired) and bat0
  • br-wan: bridge with all wan interfaces (on most devices eth1)
    • DHCPv4 & RA/DHCPv6 for internet uplink, only used for tinc if available

With Internet Uplink

  • tinc mesh interface
    • tinc with tap device in switch mode
    • DHCPv4 for a single IPv4 and default gateway
    • DHCPv6 for a single IPv6, default gateway and prefix delegation for a /64 subnet
    • subnet-up/down scripts to add routes to other subnets to kernel routing table
  • announce as a batman-adv gateway (batctl gw server)
  • DNS resolver (unbound)
  • IPv4 for br-client
    • DHCPv4 server (common client network pool, 10.80.0.0/17)
    • SNAT to forward traffic to other networks
  • IPv6 for br-client
    • RA & DHCPv6 with delegated /64 prefix
    • TODO global addresses & routing to the internet?

Without Internet Uplink

  • batman-adv in client mode (select gateway)
  • IPv4 for br-client
    • no configuration, managed by selected gateway
  • IPv6 for br-client
    • RA with fixed /64 prefix, like fdef:ffc0:4fff:0::/64
    • no default router

Gateways

The gateways act as tinc connection brokers (meta connections) and additionally route traffic to the internet over VPN providers.

They need to provide the following services:

  • DNS recursor
  • NTP server
  • tinc mesh interface
    • IPv4 SNAT router
    • DHCPv4 server for backbone network
    • IPv6 router (TODO how to do global routing?)
    • RA/DHCPv6 with prefix delegation for client networks

Network Overview

We have allocated the following private networks:

  • 10.80.0.0/16
  • fdef:ffc0:4fff::/48

These will be divided into:

  • Client Networks
    • on every node & batman-adv segment
      • 10.80.0.0/17
      • fdef:ffc0:4fff:0::/64
    • for prefix delegation to nodes
      • fdef:ffc0:4fff:1:: - fdef:ffc0:4fff:efff::
  • Backbone Network (Nodes, Gateways, Services, Wireless Links)
    • 10.80.128.0/17
    • fdef:ffc0:4fff:feed::/64
Add a custom footer
Add a custom sidebar
Clone this wiki locally