Centre for Advanced Internet Architectures, Swinburne University of Technology, Melbourne, Australia 11th July, 2014 Multipath TCP For FreeBSD Kernel Patch v0.4 ---------------------------------------------- EXAMPLE USAGE ---------------------------------------------- This document provides several example topologies for testing the FreeBSD MPTCP implementation. It is supplimentary to the readme file: http://caia.swin.edu.au/urp/newtcp/mptcp/tools/mptcp-readme-v0.4.txt Single-subflow case: -------------------- The simplest MPTCP connection contains only a single subflow. As MPTCP is enabled by default, all TCP connections between MPTCP-enabled hosts will operate with multipath signalling. A simple scenario is illustrated below. In the figure, two single-interface hosts are connected via a switch. A - ACTIVE B - PASSIVE CLIENT SWITCH (SERVER) +----+ +----+ | | +---+ | | | A1| <========> | | <=========> |B1 | | | +---+ | | +----+ +----+ The client (active) host initiates a connection from interface A1 to the server (passive) host interface B1. To test that the the kernel patch is configured correctly, establish a connection between the hosts using Iperf*: (1) Set address on Host A: A1 = 192.168.0.10 ifconfig A1 inet 192.168.0.10/24 (2) Set address Host B: B1 = 192.168.0.20 ifconfig B1 inet 192.168.0.20/24 (3) Run Iperf on Host B (iperf server): iperf -s (4) Run Iperf on the Host A (the client), connecting to the server for 5 seconds: iperf -c 192.168.0.20 -t 5 A tool like tcpdump** can be used to capture and view the connection. You can confirm that MPTCP signalling is operating by capturing the exchange of packets and viewing the resulting trace file. Multiple subflow case --------------------- This patch supports FreeBSD-to-FreeBSD multipath sessions. ADD_ADDR options are sent immediately after the MP session becomes established, for any addresses added via the "mp_addresses" sysctl. When a host receives an ADD_ADDR, it will initiate an MP_JOIN to the advertised address. Multiple subflows, single network: In the figure, the Host A has two addresses and is connected to Host B via a switch. In this example both hosts are on the same subnet. A - ACTIVE B - PASSIVE CLIENT SWITCH (SERVER) +----+ +----+ | A1| <=======> +---+ | | | | | | <=========> |B1 | | A2| <-------> +---+ | | +----+ +----+ Host A opens a connection to host B via address A1. A multipath connection is then established between addresses A1 and B1. Once established, Host A sends an ADD_ADDR option to Host B, with the details of address A2. Host B2 associates this address with the established connection. Host B then sends a SYN from B1 to A2, with the MP_JOIN option. Host A recognises that this address is associated with the existing MPTCP session, and a handshake occurs that adds this new path to the connection. This scenario can be tested with any TCP application, for this example example we again use Iperf: (1) Set addresses on Host A: A1 = 192.168.0.10, A2 = 192.168.0.11 ifconfig A1 inet 192.168.0.10/24 ifconfig A2 inet 192.168.0.11/24 (2) Set address Host B: B1 = 192.168.0.20 ifconfig B1 inet 192.168.0.20/24 (3) On Host A use the sysctl variable to set 192.168.0.11 as a slave subflow: sysctl net.inet.tcp.mptcp.mp_addresses="192.168.0.11" (4) Run Iperf on Host B (iperf server). The '-l 512k' option causes the Iperf application to read in 512kB chunks. The '-k 1M' option causes the server to advertise a receive window of 1MB***: iperf -s -l 512k -k 1M (5) Run Iperf on the Host A (the client), connecting to the server for 5 seconds: iperf -c 192.168.0.20 -t 5 By default, Iperf will send data from the client to the server. Thus in this example Host A is the sender. Multiple subflows, multiple networks: In the figure, the Host A has two addresses and is connected to a router. Interfaces A1 and A2 have addresses in different networks. Host B has a single address in another network. A - ACTIVE B - PASSIVE (CLIENT) Router (SERVER) +----+ +---+ +----+ | A1| <-------> | | <=======> |B1 | | | | | | | | | | | | | | A2| <-------> | | | | +----+ +---+ +----+ A1: 192.168.5.2 (em0) on network 192.168.5.0/24 A2: 192.168.0.2 (em1) on network 192.168.0.0/24 B1: 10.0.1.2 (em0) on network 10.0.1.0/24 Host A gateways: +----+ | A1| <-------> gateway 192.168.5.1 | | | | | A2| <-------> gateway 192.168.0.1 +----+ Now that each of the connections (A1-B1, A2-B1) are between different networks, routing decisions can have an effect on the path that a packet takes. The table below shows how the routing table might look on Host A: Destination Gateway Netif default 192.168.5.1 em0 192.168.0.0/24 link#2 em0 192.168.5.0/24 link#3 em1 If Host A has only a single routing table configured, and the default route interface is A1 (em0), packets sent towards B1 will always exit via em0. This is regardless of the source address (routing lookups take into account destination addresses only). Setting a static route to B1 via A2 would not be suitable (as this would mean all traffic to B1 would be sent via A2). Ideally we would have a route table for each interface and select a route table based on the source address of the packet. However this functionality is not yet available in the FreeBSD implementation. The current solution is to use pf (packet filter) firewall rules to re-route packets to the correct interface based on the source address. To enable pf, add the following to /etc/rc.conf pf_enable="YES" Two pf filters allow the redirection of packets to a particular interface: (1) "route-to" For a packet originating internally, route it to (interface, gateway) based on the source IP address of the packet. In the topology above, we want to send packets from A1 (192.168.0.2) to B1 (10.0.1.2) via the interface of A1 (em1). The routing lookup will initially cause this packet to be sent via em0 - so we use a "route-to" filter to re-direct: pass out on em0 route-to {(em1 192.168.0.1)} from 192.168.5.0 to any This rule takes any packet that is outbound on the interface em0 with source network "192.168.0.0" and sends it towards the gateway "192.168.0.1" via em1. (2) "reply-to" When receiving packets on an interface, ensure that any responses are returned via the same path, specified by (interface, gateway). Example: When Host A receives a packet on {em1, 192.168.0.5} from "10.0.1.2", all packets in response to this packet (e.g. ACKs) should be sent back via {em1, 192.168.0.1}. A rule can also be specified to do the same for {em0, 192.168.5.1}. This gives two rules: pass in on em1 reply-to {(em1 192.168.0.1)} from any to any keep state pass in on em0 reply-to {(em0 192.168.5.1)} from any to any keep state The pf config file below will ensure that packets with source network of A2 are sent towards the gateway 192.168.0.1, and that any response to packets arriving on either interface occurs via that same interface. Adding the pf rules to /etc/pf.conf # The default interface, and the interface to be added to "mp_addresses" # Set aliases for A1, A2 interfaces def_if = "em0" mp_if = "em1" # default gateway of the A1 (default) interface def_gw = "192.168.5.1" # Network of the "A2" interface, and the gateway mp_net = "192.168.0.0" mp_gw = "192.168.0.1" # Any packets originating from this host and leaving "def_if" with source # network "mp_net" will be forwarded out of the "mp_if" interface instead. # Similarly, packets that should egress the "default" interface will be # redirected if being sent via the "mp_addresses" interface. pass out on $def_if route-to {($mp_if $mp_gw)} from $mp_net to any pass out on $mp_if route-to {($def_if $def_gw)} from $mp_net to any # Responses to any packets received on an interface are sent via that same # interface, irrespective of the source/destination networks. pass in on $mp_if reply-to {($mp_if $mp_gw)} from any to any keep state pass in on $def_if reply-to {($def_if $def_gw)} from any to any keep state Testing: This topology can be tested using the same method as for the "Multiple subflows, single network" topology. Multi-router topology --------------------- A more flexible testing topology, featuring two multi-homed hosts, two dummynet- enabled routers and several subnets is shown in: http://caia.swin.edu.au/urp/newtcp/mptcp/tools/mptcp-vm-topology-v0.4.pdf This topology has been tested using Oracle VirtualBox [1]. VirtualBox allows the hosting of multiple guest Virtual Machines (VMs) and the creation of virtual, VM-only networks. Configuration for the hosts are given in: http://caia.swin.edu.au/urp/newtcp/mptcp/tools/mptcp-vm-settings-v0.4.txt The routers are configured with dummynet [2] so that bandwidth and delay shaping can be applied to the various links. The routers do not need to be MPTCP- enabled. ---------------------------------------------- NOTES ---------------------------------------------- * Iperf is a network throughput testing utility. It can be installed from ports: ports/benchmarks/iperf ** tcpdump is a packet capture tool that is included as part of a standard FreeBSDinstallation. *** These options are available in a modified version of Iperf: PATCH: http://caia.swin.edu.au/newtcp/tools/caia_iperf204_1.1.patch README: http://caia.swin.edu.au/newtcp/tools/caia-iperf-readme-1.1.txt See 'Connection stalls triggered by Zero Window updates' in KNOWN LIMITATIONS for an explanation as to why these options are used. ---------------------------------------------- REFERENCES ---------------------------------------------- [1] "Oracle VM VirtualBox", Homepage, http://www.virtualbox.org [2] "The dummynet project", Homepage, http://info.iet.unipi.it/~luigi/dummynet/ ---------------------------------------------- DEVELOPMENT TEAM ---------------------------------------------- This FreeBSD MPTCP implementation was first released in 2013 by the Multipath TCP research project at Swinburne University of Technology's Centre for Advanced Internet Architectures (CAIA), Melbourne, Australia. The members of this project team are: Lead developer: Nigel Willams (njwilliams@swin.edu.au) Technical advisor/developer: Lawrence Stewart (lastewart@swin.edu.au) Project leader: Grenville Armitage (garmitage@swin.edu.au)