------------------------------------------------------------------------------- Centre for Advanced Internet Architectures, Swinburne University of Technology, Melbourne, Australia 26 February, 2016 Dummynet AQM v0.1 - CoDel and FQ-CoDel for FreeBSD's ipfw/dummynet framework Author: Rasool Al-Saadi ------------------------------------------------------------------------------- This document provides an overview of the v0.1 Dummynet AQM patch that includes CoDel and FQ-CoDel for FreeBSD's ipfw/dummynet framework For more details, refer to CAIA technical report CAIA-TR-160226A at http://caia.swin.edu.au/reports/160226A/CAIA-TR-160226A.pdf Dummynet AQM patch v0.1 is applied and has been tested against FreeBSD11.0-CURRENT Branch: freebsd-head Architecture: amd64 Revision: r295345* Date: 2016-02-06 Addtionally, we provided other patch for FreeBSD 10.x-RELEASE. However, it was tested in FreeBSD 10.1-RELEASE only. * Should also be applicable to revisions from r266941 and up to r295345 (newer revisions not tested). Table of Contents 1. OVERVIEW 1.1 Licence 2. CODEL OVERVIEW 2.1 CoDel Parameters 2.2 Codel Synopsis 2.3 Examples of using Codel 3. FQ-CODEL OVERVIEW 2.1 FQ-CoDel Parameters 2.2 fq_codel Synopsis 2.3 Examples of using fq_codel 4. APPLYING THE PATCH/INSTALLATION 4.1 Applying the patch 4.2 Testing the patched ipfw/dummynet 4.3 Install the patched ipfw/dummynet 5. DEVELOPMENT TEAM 6. ACKNOWLEDGEMENTS 1. OVERVIEW ------------ Dummynet AQM patch v0.1 provides CoDel AQM (Active Queue Management) and FQ-CoDel scheduler/AQM support for ipfw/dummynet to FreeBSD. It also implements loadable AQM modules framework for dummynet to make the process of implementing new AQM mechanisms much easier. 1.1 Licence Dummynet AQM patch v0.1 is released under a BSD licence and some parts released under LGPL dual-licensing. Refer to licence headers in each source file for further details. 2. CoDel ---------- CoDel (Controlled-Delay Active Queue Management) [1] is an AQM algorithm designed to address bufferbloat problem by drop packets depending on packet sojourn time in the queue. One of CoDel goals is to be parameterless and can works reasonably on the Internet without changing its parameters. 2.1 CoDel Parameters As mentioned, CoDel designed to work properly on the Internet without changing its default configurations. However, these default configurations are not suitable for all scenarios, for example high RTT, and can cause reduction in TCP connections throughput. Thus, our codel implementation provides options to change codel parameters for each pipe/queue individually as well as changing the default values. The following are codel configuration parameters and its default values: target: is the minimum acceptable persistent queue delay. Default: 5 ms, default can be changed by the sysctl variable net.inet.ip.dummynet.codel.target interval: CoDel does not drop a packet directly after packet sojourn becomes higher than target but wait for interval of time before dropping. interval should be set to maximum RTT for all expected connection. Default is 100 ms, default can be changed by the sysctl variable net.inet.ip.dummynet.codel.interval ecn: enable Explicit Congestion Notification message. Codel will mark packets with ECN instead of dropping them. Default: no ECN. 2.2 Codel Synopsis Codel AQM is used with dummynet 'pipe' or 'queue' and can be configured through ipfw interface. For more details about ipfw/dummynet, refer to ipfw(8) FreeBSD Man Pages [3]. It should be noted that any token after 'codel' token is considered as a parameter for codel. So, make sure that all pipe/queue configurations written before 'codel' token. Codel synopsis is as follow: ipfw pipe/queue x config [...] codel [target t] [interval t] [ecn] where t is time in millisecond. 2.3 Examples of using Codel a) One pipe controlled by CoDel AQM (default configuration) and rate limit to 1 Mbits/s. ipfw pipe 1 config bw 1mbits/s codel ipfw add 100 pipe 1 form any to any b) Two queues controlled by CoDel AQM using different CoDel configurations parameters. The pipe that queue 1 and 2 use has rate limit to 10 Mbits/s and 20ms emulated delay. In more details, queue 1 and 2 connected to an implicit WF2Q+ scheduler that use pipe 1 for traffic shaping and adding emulated delay. ipfw pipe 1 config bw 10mbits/s delay 20ms ipfw queue 1 config pipe 1 codel target 7 ecn ipfw queue 2 config pipe 1 codel target 8 interval 160 ecn ipfw add 100 queue 1 form 192.168.0.0/16 to 192.168.0.0/16 ipfw add 200 queue 2 from 172.16.0.0/16 to 172.16.0.0/16 c) Two queues - queue 1 controlled by CoDel AQM and queue 2 uses droptail. Both queues are connected to QFQ scheduler that uses pipe 1 for rate limit to 5 Mbits/s. ipfw pipe 1 config bw 5mbits/s ipfw sched 1 config pipe 1 type qfq ipfw queue 1 config sched 1 codel ipfw queue 2 config sched 1 ipfw add 100 queue 1 form 192.168.0.0/16 to 192.168.0.0/16 ipfw add 200 queue 2 from 172.16.0.0/16 to 172.16.0.0/16 3. FQ-CODEL OVERVIEW --------------------- FlowQueue-CoDel [2] is a scheduler/AQM hybrid scheme that ensures fairness among flows share same bottleneck and control queue delay at the same time. FQ-CoDel scheduler is based on DRR (Deficit Round Robin) scheduler and it manages two lists of queues (old queues and new queues) to ensure fairness between heavy and lightweight flows. Each queue of FQ-CoDel queues is controlled by CoDel AQM. 3.1 FQ-CoDel Parameters As FQ-CoDel uses CoDel AQM which works properly in most Internet situations using the default configurations, in most cases FQ-CoDel does not require any change to the default configurations. However, in certain scenarios the default parameters are not suitable and can lead to reduction in performance. Thus, our fq_codel implementation provides options to change FQ-CoDel parameters for each scheduler individually as well as changing the default values. The following are fq_codel configuration parameters and its default values: target: is the minimum acceptable persistent queue delay. Default: 5 ms, default can be changed by the sysctl variable net.inet.ip.dummynet.fqcodel.target interval: Codel does not drop a packet directly after packet sojourn becomes higher than target but wait for interval of time before dropping. Interval should be set to maximum RTT for all expected connection. Default is 100 ms, default can be changed by the sysctl variable net.inet.ip.dummynet.fqcodel.interval ecn: enable Explicit Congestion Notification message. fq_codel will mark packets with ECN instead of dropping them. quantum: is number of bytes a queue can be served before being moved it to the tail of old queues list. Default: 1514 bytes, default can be changed by the sysctl variable net.inet.ip.dummynet.fqcodel.quantum limit: is the hard limit of all queues size managed by fq_codel schedular instance. Default: 10240 packets, default can be changed by the sysctl variable net.inet.ip.dummynet.fqcodel.limit flows: is the number of flows that fq_codel creates and manages. Default: 1024 queues, default can be changed by the sysctl variable net.inet.ip.dummynet.fqcodel.flows 3.2 fq_codel Synopsis fq_codel is used with dummynet scheduler object ('schd') and can be configured through ipfw interface. It should be noted that any token after 'fq_codel' token is considered as a parameter for FQ-CoDel. Thus, make sure that all scheduler configurations written before 'codel' token. fq_codel synopsis is as follow: ipfw sched x config [...] type fq_codel [target t] [interval t] [ecn] [quantum n] [limit n] [flows n] where: t is time in millisecond. n is integer number 3.3 Examples of using fq_codel a) One scheduler with one queue, 2048 fq_codel sub-queues, target 7ms and quantum 2000 bytes ipfw pipe 1 config bw 10mbits/s ipfw sched 1 config pipe 1 type fq_codel target 7 quantum 2000 ecn ipfw queue 1 config sched 1 ipfw add 100 queue 1 form 192.168.0.0/16 to 192.168.0.0/16 b) One scheduler with two queues (1024 fq_codel sub-queues by default), interval 150ms, ECN enabled ipfw pipe 1 config delay 10ms ipfw sched 1 config pipe 1 type fq_codel interval 150 ecn ipfw queue 1 config sched 1 ipfw queue 2 config sched 1 ipfw add 100 queue 1 form 192.168.0.0/16 to 192.168.0.0/16 ipfw add 200 queue 2 from 172.16.0.0/16 to 172.16.0.0/16 4. APPLYING THE PATCH/TESTING/INSTALLATION ----------------------------------- Before applying Dummynet AQM patch v0.1, make sure you have installed FreeBSD11-CURRENT r295345 with its kernel source tree. You can checkout FreeBSD11-CURRENT r295345 using: svn checkout -r 295345 svn://svn.freebsd.org/base/head/ /usr/src/ We provided another patch for FreeBSD 10.x-RELEASE (10.0, 10.1, 10.2) that includes ECN marking code for FreeBSD 10.x-RELEASE (extracted from FreeBSD11.0-CURRENT r266941) in addition to codel/fq_codel code. Once the patch is applied, you only need to (re)build the dummynet.ko kernel module and ipfw userland command (rather than rebuild a complete kernel and world from source). As root user do the following steps: 4.1 Applying The Patch Apply the patch as follows: a) Extract the patch file tar -xvf dummynet-aqm-patch-0.1.tgz -C /usr/src/ cd /usr/src b) Apply the patch. For FreeBSD11-CURRENT use: patch -p1 < dummynet-aqm-patch-0.1/freebsd11-r295345.patch For FreeBSD 10.x-RELEASE use: patch -p1 < dummynet-aqm-patch-0.1/freebsd10.x.patch c) Copy ip_dummynet.h to include directory. cp /usr/src/sys/netinet/ip_dummynet.h /usr/include/netinet/ d) Build ipfw userland cd /usr/src/sbin/ipfw make e) Build dummynet kernel module cd /usr/src/sys/modules/dummynet/ make 4.2 Testing the patched ipfw/dummynet Use the following steps to check whether the patch applied cleanly and built both userland ipfw and dummynet kernel module: a) Check if old dummynet is already loaded kldstat | grep dummynet b) If dummynet is already loaded, unload it. kldunload dummynet c) Load the patched dummynet into FreeBSD kernel kldload /usr/src/sys/modules/dummynet/dummynet.ko d) Check debug messages using 'dmesg' command: dmesg | grep CODEL The Output should be something like: load_dn_aqm dn_aqm CODEL loaded load_dn_sched dn_sched FQ_CODEL loaded d) Use the patched ipfw interface (specify a full pathname when use ipfw): /usr/src/sbin/ipfw/ipfw pipe 1 config codel /usr/src/sbin/ipfw/ipfw pipe 1 show The Output should be something like: 00001: unlimited 0 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5 interval 100 sched 65537 type FIFO flags 0x0 0 buckets 0 active 4.3 Install the Patched ipfw/dummynet To use the patched ipfw/dummynet by default, install them as follow: a) Install ipfw interface, ignore any error if appears. cd /usr/src/sbin/ipfw make install b) Install dummynet kernel module cp /usr/src/sys/modules/dummynet/dummynet.ko /boot/kernel/ c) (Optional) To avoid the following warning "KLD '/boot/kernel/dummynet.ko' is newer than the linker.hints file" try to generate hints for the kernel loader using the following command: kldxref /boot/kernel 5. REFERENCES -------------- [1] Nichols, K. et al, "Controlled Delay Active Queue Management", Internet-draft, IETF, December 2015, https://tools.ietf.org/html/draft-ietf-aqm-codel-02 [2] Hoeiland-Joergensen, T. et al, "Flowqueue-codel", Internet-draft, IETF, February 2016, https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-04 [3] ipfw(8), FreeBSD Man Pages, https://www.freebsd.org/cgi/man.cgi?ipfw(8) 6. DEVELOPMENT TEAM -------------------- The members of this project team are: Lead developer: Rasool Al-Saadi (ralsaadi@swin.edu.au) Project leader: Grenville Armitage (garmitage@swin.edu.au) 7. ACKNOWLEDGEMENTS -------------------- This project has been made possible in part by a gift from The Comcast Innovation Fund.