#+LaTeX_HEADER:\usepackage[margin=2cm]{geometry}
#+LaTeX_HEADER:\usepackage{enumitem}
#+LaTeX_HEADER:\usepackage{tikz}
#+LATEX:\setitemize{noitemsep,topsep=0pt,parsep=0pt,partopsep=0pt}
#+LATEX:\lstdefinelanguage{javascript}{basicstyle=\scriptsize\ttfamily,numbers=left,numberstyle=\scriptsize,stepnumber=1,showstringspaces=false,breaklines=true,frame=lines}
#+OPTIONS: toc:nil ^:nil num:nil

#+TITLE: TIPSY: Telco pIPeline benchmarking SYstem

/Disclaimer: TIPSY is currently heavily work in progress!/

TIPSY is a benchmark suite to evaluate and compare the performance of
programmable data plane technologies and network-function virtualization
frameworks over a set of standard scenarios rooted in telecommunications
practice. Apart from simple L2 and L3 pipelines, currently there is a
rather complex BNG (Broadband Network Gateway) and 5G MGW (Mobile Gateway)
pipeline defined and implemented in TIPSY, with further pipelines and
implementations to follow soon.

The aim of TIPSY is to provide the networking community a set of
*standardized telco-oriented scenarios* on top of which different
*programmable data plane technologies can be fairly and comprehensibly
evaluated*.  The target audience is network operators who want to test new
data-plane equipment, network engineers evaluating the scalability of a
programmable switch in terms of increasingly complex configurations, or
researchers who want to compare a new data plane algorithm with existing
and established technology.

TIPSY comprises 6 elemental parts, currently existing at varying levels of
maturity:

- a set of telco pipelines with working implementations (currently OpenFlow
  and BESS are supported, contributions are welcome),
- a test suite to validate the implementations (ongoing),
- a configuration system that allows to fine-tune general parameters of the
  pipelines (supported),
- a tunable trace generator to produce deterministic traffic traces for
  repeatable experiments and reproducible results (supported),
- a distributed measurement infrastructure to feed the traffic traces to
  the system-under-test, controller code to drive dynamic benchmarks, and
  an evaluation framework that visualizes the results (rudimentary),
- an evaluation and visualization framework to generate production-quality
  reports from the benchmark results (planned).

** What TIPSY is

- A collection of pipelines that model real telco use cases: the intention
  for TIPSY is to serve as the "de facto" data-plane benchmarking suite for
  practitioners and researchers.
- A measurement support framework: TIPSY will automatically generate
  detailed configs, working pipelines, and complete traffic traces to start
  testing right away.
- A tool to perform repeatable experiments: TIPSY devotes huge emphasis to
  support reproducibility; it automatically generates deterministic traffic
  traces, supports multiple traffic-dropping policies to never lose
  packets, etc.
- A tool to present benchmark results in a comprehensible form, like
  diagrams, charts, and tables, and a flexible way to configure and
  fine-tune these visualizations (TODO).

** What TIPSY is not

- A standalone automatic measurement system: TIPSY will never be able to
  setup the measurement completely unattended and there are certain steps,
  like connecting the SUT and the tester, that will need to be done
  manually anyway; the intention is to minimize manual intervention as much
  as possible.
- An exhaustive collection of telco pipelines (contributions are welcome)
  or canonical implementations of the ones supported (of course, there is
  more that one way to do it).
- An Internet scale performance measurement framework: the TIPSY reference
  configuration is a single device-under-test connected back-to-back to a
  traffic generator/measurement device; benchmarking multi-hop setups or
  complex network topologies are out of scope for TIPSY.

** If you find TIPSY to not work for you

You can find a comprehensive list of alternative data plane benchmark
suites and lots of background on data plane testing in the below paper.

Matthias Holdorf,
[[https://www.net.in.tum.de/fileadmin/TUM/NET/NET-2016-07-1/NET-2016-07-1_05.pdf][How-To Compare Performance of Data Plane Devices]],
Proceedings of the Seminars Future Internet (FI) and Innovative Internet
Technologies and Mobile Communications (IITM), 2016.

* Architecture

The general TIPSY setup will contain a System-Under-Test (SUT) that runs
the data plane to be evaluated, a Tester that feeds the SUT with a traffic
trace and measures performance, and a Controller running on the SUT that
sets up the static pipeline and, optionally, exercises the performance of
the SUT under updates.  Drawing from telco practice, TIPSY distinguishes
between the *uplink* direction (user-to-network direction) and the
*downlink* direction (network-to-user direction); the downlink port is the
attachment port for the access network (users) and the uplink port connects
to the public Internet. In pratice, however, the uplink and downlink ports
of the SUT are connected back-to-back to the same Tester device. A
management access is also required between the two to let the Tester
configure the SUT via Secure Shell.

:   +----------------+             +---------------------+
:   |                |             |                     |
:   |                |             |                     |
:   |     uplink_port|<----------->|downlink_port        |
:   |                |             |                     |
:   |                |             |                     |
:   |    *Tester*    |             | *System Under Test* |
:   |                |             |        (SUT)        |
:   |                |             |          +          |
:   |                |             |     *Controller*    |
:   |                |             |                     |
:   |                |             |                     |
:   |   downlink_port|<----------->|uplink_port          |
:   |                |             |                     |
:   |                |             |                     |
:   |                |<---mgmt---->|                     |
:   +----------------+             +---------------------+

Apart from static pipeline configurations, aimed for evaluating the SUT in
steady state, TIPSY also contains additional scenarios to benchmark the SUT
in dynamic workloads, i.e., when the control plane updates the data plane
program at various (and configurable) intensities.  The implementations
that come with TIPSY pipelines contain code to feed the updates, via the
Controller module, into the SUT.

* Pipelines

The below table provides an overview of TIPSY pipelines along with the
features each one uses and the current availability of an implementation in
the TIPSY code.

#+ATTR_LaTeX: :align |c|c|c|c|c|c|c|c|c|c|
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
|                                                      | Encap/Decap | ParseField | SetField | RateLimit | Firewall | NAT | Backends |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| [[./doc/README.portfwd.org][Port forward (PORTfwd)]] | -           | -          | -        | -         | -        | -   | ovs, bess, erfs, lagopus, ofdpa, t4p4s, vpp |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| [[./doc/README.L2fwd.org][L2 forward (L2fwd)]]       | -           | L2         | -        | -         | -        | -   | ovs, bess, erfs, lagopus, ofdpa, t4p4s |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| [[./doc/README.L3fwd.org][L3 forward (L3fwd)]]       | -           | L2/L3      | L2/L3    | -         | -        | -   | ovs, bess, erfs, lagopus, ofdpa, t4p4s, vpp |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| Encap/Decap                                          | VXLAN       | -          | L2/L3    | -         | -        | -   |        |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| RateLimit                                            | -           | -          | -        | x         | -        | -   |        |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| [[./doc/README.fw.org][Firewall]]                                          | -           | L2/L3/L4   | -        | -         | x        | -   | ovs, bess, erfs, lagopus |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| NAT                                                  | -           | L2/L3/L4   | L2/L3/L4 | -         | -        | x   |ovs     |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| Data Center GW (DCGW)                                | VXLAN       | L2/L3      | L2/L3    | -         | -        | x   |        |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| [[./doc/README.mgw.org][Mobile GW (MGW)]]            | GTP         | L2/L3/L4   | L2/L3    | x         | -        | -   | ovs, bess, erfs, lagopus |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|
| [[./doc/README.bng.org][Broadband Network GW (BNG)]] | GRE         | L2/L3/L4   | L2/L3    | x         | x        | x   | ovs, bess, erfs, lagopus |
|------------------------------------------------------+-------------+------------+----------+-----------+----------+-----+--------|

* Installation and usage

** Installation
*** Install external software
TIPSY depends on external software components. To run TIPSY, it is
necessary to install the following software:

On SUT:
- sudo,
- ssh,
- screen.

On Tester:
- make,
- ssh,
- sudo,
- python-jsonschema,
- matplotlib,
- pdflatex,
- scapy.

*** Set PATH
TIPSY does not require explicit installation but the =tipsy= executable
must always be available and executable; the easiest setup is to add the
TIPSY main directory to the PATH.

#+BEGIN_SRC sh
git clone https://github.com/hsnlab/tipsy
cd tipsy
export PATH=$PWD:$PATH
#+END_SRC

*** Patch MoonGen
Apply the following patch to timestamping.lua of MoonGen, otherwise
this program cannot set the correct packet sizes, and might not be
able to read back timestamps from received packets.

#+BEGIN_SRC patch
diff --git a/lua/timestamping.lua b/lua/timestamping.lua
index bd6c81f..ee68513 100644
--- a/lua/timestamping.lua
+++ b/lua/timestamping.lua
@@ -87,7 +87,7 @@ function timestamper:measureLatency(pktSize, packetModifier, maxWait)
 			-- change timestamped UDP port as each packet may be on a different port
 			self.rxQueue:enableTimestamps(buf:getUdpPacket().udp:getDstPort())
 		end
-		buf:getUdpPtpPacket():setLength(pktSize)
+		-- buf:getUdpPtpPacket():setLength(pktSize)
 		self.txBufs:offloadUdpChecksums()
 		if self.rxQueue.dev.reconfigureUdpTimestampFilter and not skipReconfigure then
 			-- i40e driver fdir filters are broken
#+END_SRC

** Main TIPSY benchmark configuration

Input to TIPSY is a high-level description of the intended measurements and
the measuring environment. The environment configuration involves the
paramaters of the Tester (traffic generator, runtime of the measurement,
etc.) and the SUT (pipeline implementation, custom setup/teardown scripts,
etc.).

Descriptions of the measurements are the name and other general parameters
(number of users, number of Internet routes, etc.) of the pipeline to be
coded into the SUT. The high-level configurations are later transformed
into a set of configurations/data-plane programs that can be loaded into
the SUT and a set of traffic traces for each config.

A detailed TIPSY configuration guide can be found [[./doc/README.config.org][here]].
For the pipeline specific parameters, see the pipeline's
documentation. Below is a sample configuration snippet that defines a
benchmark on the [[./doc/README.mgw.org][Mobile Gateway (MGW)]] (=mgw=) pipeline,
with pipeline-specific settings =user= (number of users) and =bst= (number
of base stations) as set in the =pipeline= section.

#+BEGIN_SRC javascript
{
    "benchmark":
    [
        {
	    "id": "my_benchmark",
            "scale": "joint",
            "pipeline": {
                "name": "mgw",
                "user": [1,2],
                "bst": [5,10]
            },
            "traffic": {
                "pkt-num": 10000,
                "pkt-size": [64, 128],
                "dir": ["uplink"]
            },
            "sut": {
                "type": "bess"
            },
            "tester": {
                "type": "moongen",
                "test-time": 30
            }
        }
    ],

#+END_SRC

The =id= parameter sets a name for the benchmark and =scale= describes the
way the individual benchmark instances in the scalability benchmark are to
be executed. TIPSY allows to easily request and perform scalability tests
by repeating the benchmark multiple times, each time setting one or all
parameters as controlled by the =scale= setting:
- =none=: do not perform scalability tests (default),
- =outer=: take the outer product of all settings specified for the
  benchmark and generate a separate test case for all,
- =joint=: scale the parameters jointly.

In the above example =scale= is set to =joint=, which tells TIPSY to scale
the parameters specified as /lists/ in the config (=user= and =bst=)
jointly, that is, take the first setting in the list for each parameter,
then the second, etc., and generate a test for each such tuple.  In the
above example, this will result in two tests to be run, one when =user= is
set to 1 and =bst= is set to 5 (the first elements of the lists), and one
when =user= is set to 2 and =bst= is 10 (the second elements of the lists).

If =scale= is set to =outer=, then a separate test will be run for each
combination of the multi-valued settings (=user= and =bst=), i.e., we get 4
tests, first setting the (=user=, =bst=) tuple to (1,5), then to (1,10),
then to (2,5), and finally to (2,10).

Setting =scale= to =none= ignores parameter lists and generates a single
benchmark for the first scalar for each argument.

The =traffic= section sets the test traffic parameters. TIPSY calculates
the outer product of the scaled pipeline configurations and the traffic
configurations in order to measure each scenario with all types of test
traffic.

The =sut= and the =tester= section contains parameters of the SUT and
the Tester respectively.

In continuation of the example, the following snippet presents the
=default= section of the TIPSY configuration. It can contain the same
parameters as the benchmarks. The purpose of the =default= section is
to simplify the definition of recurring parameters of benchmarks.

Further details are in the [[./doc/README.config.org][detailed TIPSY configuration guide]].

#+BEGIN_SRC javascript
    "default":
        {
	"id": "my_benchmark",
        "scale": "joint",
        "pipeline": {
             "name": "mgw",
             "user": [1,2],
             "bst": [5,10]
        },
        "traffic": {
            "pkt-num": 10000,
            "pkt-size": [64, 128],
            "dir": ["uplink"]
        },
        "sut": {
            "type": "bess"
        },
        "tester": {
            "type": "moongen",
            "test-time": 30
        }
    }
}
#+END_SRC

For generating an empty configuration with the default setting for each
configurable parameter, use:

#+BEGIN_SRC sh
tipsy init <pipeline>
#+END_SRC

You may then start to edit the resultant JSON configuration accordingly.

** Run TIPSY

The first step to run a benchmark is to create a root directory that will
contain all files (configurations, traffic traces, data plane configs,
results, etc.) associated with the benchmark and to write the main JSON
configuration. Then, to actually run the benchmark, simply issue the
necessary TIPSY commands executing the benchmark driver =tipsy= in the
benchmark root directory with different command line arguments.

#+BEGIN_SRC sh
mkdir my_benchmark
cd my_benchmark
tipsy init <pipeline_name>
<edit TIPSY benchmark configuration JSON file>
tipsy config
make

tipsy clean
#+END_SRC

The basic workflow is described in more details [[./doc/README.workflow.org][here]].

* Miscellaneous

You can make the output a bit more readable by installing
=module/openflow/color_log.py=, read the header of
=module/openflow/color_log.py= for how to do that.

* License

TIPSY is a free software and licensed under [[./LICENSE][GPLv3+]].