CAIA NS-3/NSC FreeBSD 9 Based Simulation Environment Virtual Machine Image v0.1 Centre for Advanced Internet Architectures, Swinburne University of Technology, Melbourne, Australia http://www.caia.swin.edu.au/ 26th September, 2013 ---------------------------------------- 1. OVERVIEW ---------------------------------------- This virtual machine (VM) appliance provides a FreeBSD 9 based turn-key environment to experiment with the CAIA NS-3/NSC modifications. Patched versions of the NS-3 and NSC code are provided along with ready to compile and run example simulation code. Simplistic scripts to produce plots of simulation output are also provided as a starting point. This work is released as part of the "Exploring possible mitigation for incast TCP congestion in data centres" research project: http://www.caia.swin.edu.au/urp/incast/ Where possible we have licensed our code under a 2-clause BSD license. Refer to the license header in each individual source file for more detail. The patched NS-3 and NSC code is provided as two separate Mercurial repository + working copy directories in the "ns3" user's home directory to provide you with simple change tracking and source code management for any changes you wish to make. If you expect to undertake any NS-3/NSC code hacking, it is advised that you familiarise yourself with Mercurial [1,2] and Subversion [3,4]. Section 2 of this document provides a set of shell commands which get you going quickly and will give you some data to look at in a few minutes or less. Section 3 documents some important FreeBSD specific context which is treated as assumed knowledge elsewhere in this document. Section 4 and 5 provide more detailed information about the CAIA modifications to NS-3 and NSC respectively. Section 6 details administration information about the VM image itself. Section 7 lists known caveats and limitations of the code and simulation environment. Section 8 provides useful references. Section 9 and 10 document the authors and acknowledgements related with this project. ---------------------------------------- 2. QUICK START ---------------------------------------- To run the CAIA NS-3 "dumbbell" simulation with the FreeBSD 9-STABLE stack port and produce some simple X11-forwarded plots: ssh -XY -p2222 ns3@localhost cd /usr/home/ns3/ns-3-allinone/nsc ./scons.py freebsd-9stable cd /usr/home/ns3/ns-3-allinone/ns-3-dev ./waf --with-nsc=../nsc/freebsd-9stable configure ./waf build -j2 ./waf --run="dumbbell --data-store=/home/ns3/experiment_data" ~ns3/scripts/q_vs_time.sh /home/ns3/experiment_data//bottleneck.qtrace ~ns3/scripts/cwnd_vs_time.sh /home/ns3/experiment_data//allnodes.siftr ---------------------------------------- 3. IMPORTANT FREEBSD BACKGROUND ---------------------------------------- In FreeBSD parlance, X-STABLE (where X is a major version number) refers to the conservative development branch from which production-ready X.Y point releases are periodically made. At the time of writing, 9-STABLE is the branch from which production releases are being made, and 8-STABLE has essentially quiesced into caretaker mode as a legacy production branch which only receives bugfixes and minor feature updates. Development occurs on the FreeBSD "head" Subversion branch [5] (known as (X+1)-CURRENT i.e. 10-CURRENT as of writing), and then changes deemed appropriate for inclusion in production releases get merged back into the stable branch(es) after sufficient time has elapsed to uncover any problems with them in the head branch. Changes deemed too disruptive to merge back to stable branches remain in head and eventually become part of a .0 production release when a new stable branch is branched off head, which happens approximately every 18-24 months. 10-CURRENT is due to become 10-STABLE and produce a 10.0 production release by the end of 2013, at which point the head branch will become 11-CURRENT. ---------------------------------------- 4. CAIA NS-3 MODIFICATIONS ---------------------------------------- We have made a number of changes to the Network Simulator 3 and included them in the VM disk image as a patch applied to a local clone of the NS-3 development Mercurial repository at "/usr/home/ns3/ns-3-allinone/ns-3-dev". The patch was applied to revision d5b9559ce074 of the "default" branch as a Mercurial bookmark named "caia-freebsd", which effectively creates a separate line of development (known as a "head" in Mercurial parlance) in the local repository to keep our changes separate from the main NSC development branch (refer to output of "hg heads" run from the NS-3 development working copy directory). Here is a summary of the key changes: - Added a new QueryResponseApplication model to simulate one-to-many query-response network traffic patterns commonly found in data centres. - Added a new MultiQueue model to simulate queues which have a fixed resource limit that can be shared between logically partitioned sub-queues. This model forms a building block for the BridgeNetDevice overhaul. - Overhauled the BridgeNetDevice model to turn it into a round-robin, input queued device utilising virtual output queuing (VOQ) at each ingress port. - Extended the interface between NS-3 and NSC to support socket upcalls, generic route manipulation and an enhanced mechanism for NSC to obtain the virtual time from the simulator. - Added support for the RFC 1323 TCP window scale option to NS-3's TCP model. - Modified Queue tracing to make it possible to use a single callback function for selective per-queue and per-action tracing. - Added an "incast" simulation which utilises the VOQ BridgeNetDevice and QueryResponseApplication to simulate simple TCP incast scenarios. - Added a "dumbbell" simulation which utilises the PointToPointDumbbellHelper and BulkSendApplication to simulate simple TCP scenarios. The full set of changes can be viewed by looking at the diff: cd /usr/home/ns3/ns-3-allinone/ns-3-dev hg diff -r d5b9559ce074:caia-freebsd | vim - You will want to familiarise yourself with the wealth of NS-3 documentation available via the NS-3 website [6]. The tutorial [7] and API documentation [8] are particularly useful. To configure NS-3 with NSC FreeBSD stack port support after you have built the FreeBSD 9-STABLE stack port: cd /usr/home/ns3/ns-3-allinone/ns-3-dev ./waf --with-nsc=../nsc/freebsd-9stable configure To build NS-3 after configuring: cd /usr/home/ns3/ns-3-allinone/ns-3-dev ./waf build -j2 ## CAIA EXAMPLE SIMULATIONS ## We have included two simulations as a base for further experimentation. The code can be found in the "examples/tcp" directory of the NS-3 development working copy. Both the "dumbbell" and "incast" simulations share some common code in the "caia-util.cc" and "caia-util.h" files in the same directory. The simulations write output data to various files in a per-simulation-run directory under a top-level data store directory. "/home/ns3/experiment_data" has been created to act as the top level data store and should be specified using the "--data-store" command line switch when running simulations. The dumbbell simulation constructs a network with a "left" and "right" router connected via a bottleneck link, each of which have an equal number of leaf nodes connected to them which are paired as TCP senders and sinks. Congestion control algorithms are specified by their FreeBSD "net.inet.tcp.cc.algorithm" sysctl name. A number of simulation parameters can be controlled via command line options, and some must be changed by directly editing the code and recompiling. To see which command line parameters are available for run-time tweaking: cd /usr/home/ns3/ns-3-allinone/ns-3-dev ./waf --run="dumbbell --PrintHelp" To run a 2 leaf dumbbell simulation, with the first leaf pair using newreno congestion control and the second leaf pair using cubic congestion control: ./waf --run="dumbbell --data-store=/home/ns3/experiment_data --leaves=2 \ --cc-algos=newreno,cubic" The incast simulation constructs a set of nodes connected via a virtual output queued Ethernet switch model, where one of the nodes acts as a querier and the remainder act as responders. The traffic pattern generated is designed to simulate TCP incast when the responses converge towards the querier's switch port. To see which command line parameters are available for run-time tweaking: cd /usr/home/ns3/ns-3-allinone/ns-3-dev ./waf --run="incast --PrintHelp" To run a single querier, two responder incast simulation: cd /usr/home/ns3/ns-3-allinone/ns-3-dev ./waf --run="incast --data-store=/home/ns3/experiment_data --responders=2 \ --rxqueue-cells=256 --txqueue-cells=16 --response-size=100000" ## CREATING YOUR OWN TCP SIMULATIONS ## To add a new TCP simulation program to the build that uses one of the NSC FreeBSD stack ports, place your code in the "/usr/home/ns3/ns-3-allinone/ns-3-dev/examples/tcp" directory and modify the wscript file in that directory. You can duplicate the three key entries from either the dumbbell or incast simulations, updating file names and NS-3 modules your code depends on (second argument to bld.create_ns3_program) as appropriate. You may then need to re-run the waf configure and build steps in order for the build system to acknowledge your new simulation target program. ---------------------------------------- 5. CAIA NSC MODIFICATIONS ---------------------------------------- We have made a number of changes to the Network Simulation Cradle v0.5.3 and included them in the VM disk image as a patch applied to a local clone of the NSC Mercurial repository at "/usr/home/ns3/ns-3-allinone/nsc". The patch was applied to revision 08c391d0660a of the "default" branch as a Mercurial bookmark named "caia-freebsd", which effectively creates a separate line of development (known as a "head" in Mercurial parlance) in the local repository to keep our changes separate from the main NSC development branch (refer to output of "hg heads" run from the NSC working copy directory). Here is a summary of the key changes: - Added a "from scratch" port of the FreeBSD 9-STABLE network stack and associated kernel infrastructure, which is built as "libfreebsd-9stable.so". The port was most recently synced to revision 254312 of the FreeBSD 9-STABLE Subversion branch from 14th August 2013 - Added a port of the FreeBSD "head" network stack and associated kernel infrastructure, which is built as "libfreebsd-head.so". This stack port is a cut-and-paste of the 9-STABLE stack port which was then synced against revision 252531 of the FreeBSD head Subversion branch from 3rd July 2013. NB: The freebsd-head NSC stack port is not yet fully functional. Please refer to section 7 for details. The full set of changes can be viewed by looking at the diff: cd /usr/home/ns3/ns-3-allinone/nsc hg diff -r 08c391d0660a:caia-freebsd | vim - No effort has been made to make the pre-existing NSC stack ports compile on FreeBSD and they have been commented out of the top level SConstruct file. You must selectively build the FreeBSD 9-STABLE and/or head stack ports in order to get a usable NSC library. To build the FreeBSD 9-STABLE or head stack ports, where is "freebsd-9stable" or "freebsd-head": cd /usr/home/ns3/ns-3-allinone/nsc ./scons.py Note that the scons build system litters object files throughout the stack port directories during a build, which is not ideal and should be fixed. As it is though, you should be aware of this behaviour and remember to clean out the object files before doing tasks that care about the state of the working copy. To clean all build output from the FreeBSD 9-STABLE or head stack ports, where is "freebsd-9stable" or "freebsd-head": cd /usr/home/ns3/ns-3-allinone/nsc ./scons.py --clean Both FreeBSD stack ports are organised identically with the following directory layout: glue Contains various custom code to implement the NS-3/NSC interface and infrastructure required by the FreeBSD kernel code. kernconf Contains files generated by the FreeBSD build system as part of a kernel build which control which options are enabled in the kernel code. override Contains header files which transparently modify some aspect of the identically named header files in the pristine directory. pristine Unmodified kernel code managed as a Subversion working copy of the relevant FreeBSD branch. testharness A small test program which links with the NSC library to ensure all symbols are resolved. Eventually may become a proper test harness. ## MANAGING PRISTINE CODE ## The pristine directory of each stack is managed as a Subversion sparse working copy of the "sys" directory from the official FreeBSD Subversion repository branch. The entire Subversion working copy is then managed as part of the Mercurial working copy. Only the minimal set of files required for the NSC stack to function have been pulled in. The SConscript file of each FreeBSD stack port will automatically build any .c file it finds in the pristine directory. The most likely change you will want to make to the pristine code is to synchronise it against a more recent revision of the upstream FreeBSD source. The "update_freebsd_pristine.sh" shell script is used to partially automate the process, though does not handle every possibility and may require manual intervention. Running the script without any arguments will produce a usage message. You should not attempt an update if you have uncommitted changes to files in your pristine directory. To update the FreeBSD 9-STABLE or head NSC stack's pristine code to the latest revision of the respective FreeBSD Subversion branch, where is "freebsd-9stable" or "freebsd-head": cd /usr/home/ns3/ns-3-allinone/nsc ./scons.py --clean ./update_freebsd_pristine.sh update /pristine After an update, the pristine directory will be left in an uncommitted state from Mercurial's perspective, so you should commit the update once you have tested the stack and are satisfied it works well: cd /usr/home/ns3/ns-3-allinone/nsc hg commit /pristine If the update did not go to plan, do not commit the result and instead use the "revert" action of the script to restore the pristine directory to its pre-update state: cd /usr/home/ns3/ns-3-allinone/nsc ./update_freebsd_pristine.sh revert /pristine Files can be added, removed or updated by first using Subversion to manipulate the working copy and once you are happy with your changes, you can check them all in as a Mercurial commit. In the examples below, refers to the revision number that the pristine code was last updated against, which can be obtained from the "Revision: " line from "svn info". To explicitly add a new file netinet/ip_carp.c from the upstream FreeBSD repository that is not in the local sparse working copy: cd /usr/home/ns3/ns-3-allinone/nsc//pristine svn up -r netinet/ip_carp.c If a file needs to be pulled in to a directory that is not already in the sparse working copy, pull in the directory without all of its contents first: cd /usr/home/ns3/ns-3-allinone/nsc//pristine svn up -r --set-depth=empty netpfil svn up -r --set-depth=empty netpfil/ipfw svn up -r netpfil/ipfw/ip_dummynet.c To remove a file from the sparse working copy which you no longer want included when the NSC stack is built: cd /usr/home/ns3/ns-3-allinone/nsc//pristine svn up --set-depth=exclude netinet/igmp.c ---------------------------------------- 6. VM IMAGE CONFIGURATION & MAINTENANCE ---------------------------------------- ## CONFIGURATION ## The VM defaults to NATed networking with a port forward set up on localhost:2222 to facilitate inbound ssh to the VM i.e. ssh -p2222 ns3@localhost The root password is: ns3dev The "ns3" user account has been created and should be used for regular NS-3/NSC work. The account is a member of the "wheel" and "operator" groups and has no password. The wheel group has been granted full sudo privileges, which therefore extends to the ns3 user. The SSH daemon has been configured to allow passwordless logins, so ssh'ing into the VM as the "ns3" user will work without prompting for a password. The following useful third-party software is installed on the image: Mercurial - NS-3/NSC source code management Python - General purpose programming/scripting language R - Statistical computing package used to plot data rsync - Network file distribution/synchronization utility Subversion - FreeBSD source code management sudo - Run progs with another user's security privileges tmux - Terminal multiplexor similar to GNU "Screen" Vim - Featureful command line and X11 editor X.Org client libraries - Enables X11-forwarding from the VM Xpdf - Simple PDF viewer To test X11-forwarding to a *nix X11 server by running "xeyes" on the VM: ssh -XY -p2222 ns3@localhost "xeyes" ctrl-c to terminate ## MAINTENANCE ## The following maintenance information is provided for anyone wishing to update or further customise the VM environment. None of the procedures detailed in this subsection are necessary in order to begin running NS-3/NSC simulations. The FreeBSD handbook [9] is a good general reference for FreeBSD administration. The third-party software installed on the VM image came from the FreeBSD ports tree, and the portmaster(8) utility is used for managing installation and updating of third-party software. The newer pkg(8) package management utilities (also commonly referred to as "pkgng") are in use as per the setting in /etc/make.conf. The portsnap(8) utility is used to manage the ports tree. To list all currently installed third-party software: pkg info The ports tree in /usr/ports was deleted from the image to save space, but the steps below explain how to perform an update. NB: only perform the below portsnap fetch + extract step if /usr/ports is empty. Once you have /usr/ports populated from a previous portsnap fetch + extract, you can skip straight to the update step. To obtain a current snapshot of the ports tree (required when /usr/ports is empty): portsnap fetch portsnap extract To update from an existing snapshot of the ports tree: portsnap fetch portsnap update To update all currently installed third-party software to the latest respective versions in the local ports tree after a portsnap update: portmaster -ad To install a new piece of third-party software which lives in the x11/xcalc port directory within the ports tree: portmaster -d x11/xcalc The image is running revision 255014 of the FreeBSD 9-STABLE Subversion branch from 29th August 2013 as the base operating system. The source code was deleted from the image to save space, but the steps below explain how to perform an update. NB: only perform the below svn checkout step if /usr/src is empty. Once you have /usr/src populated from a previous svn checkout, you can skip straight to the update/compile step. To obtain a current snapshot of the FreeBSD base operating system source code (required when /usr/src is empty): svn checkout http://svn.freebsd.org/base/stable/9 /usr/src To update and compile the FreeBSD base operating system source code: cd /usr/src svn update make -s -j2 -DNO_CLEAN buildworld buildkernel If the build step fails, you can try it again but remove "-j2 -DNO_CLEAN". If it still fails, wait a day or two and attempt the above update + compilation procedure again in the hope that the breakage has been fixed by a developer. If the build step completes successfully, install the updated base operating system (refer to mergemaster(8) and etcupdate(8) for usage details): make -s installkernel mergemaster -p make -s installworld etcupdate yes | make delete-old shutdown -r now ---------------------------------------- 7. CAVEATS & KNOWN LIMITATIONS ---------------------------------------- - The freebsd-head NSC stack port does not yet work correctly. It compiles and initialises but is not yet able to complete a basic connection. This will be fixed in a future release. - To run a simulation with the freebsd-head NSC stack port, you must manually tweak "/usr/home/ns3/ns-3-allinone/ns-3-dev/examples/tcp/wscript" and change any reference of "freebsd-9stable" to "freebsd-head", and then re-run the waf configure step with "--with-nsc=../nsc/freebsd-head". This will be fixed in a future release. - A number of simulation parameters should ideally be made command line switches for runtime manipulation. - The freebsd-9stable stack port will sometimes "panic" (mapped to an abort() call) due to one or more latent bugs in the scheduler, which can manifest in a range of ways but typically as a "spin lock held too long" panic. Just re-run the simulation until it succeeds. ---------------------------------------- 8. LINKS ---------------------------------------- [1] "Mercurial SCM", http://mercurial.selenic.com/ [2] Bryan O'Sullivan, "Mercurial: The Definitive Guide", http://hgbook.red-bean.com/ [3] "Apache Subversion", http://subversion.apache.org/ [4] Ben Collins-Sussman, Brian W. Fitzpatrick, C. Michael Pilato, "Version Control with Subversion", http://svnbook.red-bean.com/ [5] "FreeBSD head Subversion branch", http://svnweb.freebsd.org/base/head/ [6] "NS-3", http://www.nsnam.org/ [7] "NS-3 Tutorial", http://www.nsnam.org/docs/release/3.16/tutorial/singlehtml/index.html [8] "NS-3 API Documentation", http://www.nsnam.org/docs/release/3.16/doxygen/index.html [9] "FreeBSD Handbook", http://www.freebsd.org/doc/handbook/ ---------------------------------------- 9. AUTHORS ---------------------------------------- This documentation, VM disk image and the NS-3/NSC modifications distributed with this disk image were created by Lawrence Stewart with input from Grenville Armitage . The work was undertaken as part of Lawrence's PhD studies. Lawrence was supported by a period of employment on the Cisco-funded "Exploring possible mitigation for incast TCP congestion in data centres" research project at Swinburne University of Technology's Centre for Advanced Internet Architectures, Melbourne, Australia. Grenville is Principle Investigator on this project, and further details are available at http://caia.swin.edu.au/urp/incast Questions, bug reports and feedback can be provided to Lawrence Stewart via email: or . ---------------------------------------- 10. ACKNOWLEDGEMENTS ---------------------------------------- This work has been made possible in part by a gift from The Cisco University Research Program Fund, a corporate advised fund of Silicon Valley Community Foundation.