Configuration and performance of policing with TCP traffic (Cat 6500)

Configuration, tuning and performance of policing on Catalyst 6506 switches
TCP traffic

Goal: to study the configuration, the effectiveness and the impact on end-to-end TCP performance.

Equipment:
- Catalyst 6506, with GigaEthernet interfaces
  - input queuing on GEthernet interfaces (receive queues): 1p4q (1 strict priority queue, 4 single-threshold WRR queues)
  - output queuing on GEthernet interfaces (transmit queues): 1p2q2t (1 strict priority queue, 2 double-threshold WRR queues)
  and IOS image: Version 12.1(8b)EX2
- Pool of Sun Ultra workstations running Solaris 7
- Pool of Linux PCs,
Test description:
Policing is enabled on one input interface according to the following configuration sample. Policing is presumably implemented according to a dual leaky bucket and depends on the following parameters:
- CIR: Committed Information Rate, the minimum sustainable rate
- norml_burst_size: tocket bucket size related to the CIR rate, minimum value: CIR bits/sec / 4000 (tokens are removed from the bucket every 0.25 msec)
- PIR: peak rate
- maximum_burst_size: tocket bucket size to let traffic rate peak to PIR. Minimum value as to be: PIR bits/sec / 4000 like in the previous case
  If not specified, the parameter is equal to the normal burst size
- Conform action: if rate is less than CIR, transmit/set-dscp-transmit/set-prec-transmit/drop
- Exceed action: if CIR is less than rate, and rate is less than PIR, drop (default value)/policed-dscp-transmit/transmit
- Violate action: if rate is larger than PIR
  The verified that both the normal and maximum burst can be set to a minimum value which depends on the following rule:
  - for policing rates up to 4 Mbps, the minimum value is equal to 2000 bytes;
  - for policing rates larger than 4 Mbps, the minimum value can be derived according to the following rule: Min_size = policing_rate / 2000
    If we attempt to set the burst size to a smaller value, then the burst size is set to 31250 Kby, i.e. the maximum configurable value.
Syntax:
```
police  CIR  normal-burst-size PIR maximum-burst-size conform-action  exceed-action violate-action 


mls qos

class-map match-all B1high
  match access-group 151
class-map match-all B2high
  match access-group 150


policy-map Bbandwidth
  class B2high
     police 300000000 2500000 2500000 conform-action set-prec-transmit 2 exceed-action drop
  class B1high
     police 50000000 2500000 2500000 conform-action set-prec-transmit 1 exceed-action drop


interface GigabitEthernet3/4
 description to sunlab3g
 no ip address
 wrr-queue queue-limit 10 45 
 wrr-queue cos-map 1 1 0 
 wrr-queue cos-map 1 2 3 
 wrr-queue cos-map 2 1 1 4 6 
 priority-queue cos-map 1 2 5 
 rcv-queue cos-map 1 2 3 
 service-policy input prova1
 switchport

access-list 150 permit tcp host 131.154.100.3 host 131.154.100.2 eq 50150
access-list 151 permit tcp host 131.154.100.3 host 131.154.100.2 eq 50151
```
Policing can be of two types:
- aggregate policer: policing is applied to a boundle of streams matching a given profile.
- per flow policing: policing applies to each flow in matched traffic separately
The PFC2 supports classification, marking and policing using policy maps.
Interfaces can be in one of the following states (interface state is fundamental to define the rule to set the internal DSCP value, i.e. the value used to set the TOS byte written in transmitted packets):
- untrusted (default): untrusted traffic can be marked with policy maps
- Trust IP precedence: internal DSCP value is derived from IP precedence value of a given packet
- Trust DSCP: DSCP code set by the end-system is trusted, i.e. the internal DSCP value is derived from the packet DSCP code
- Trust CoS (layer 2): internal DSCP code derived from layer 2 frame CoS
  - ISL frames: CoS indicated by the 3 less significant bits in the User field
  - IEEE 802.1Q frames: CoS indicated by User priority bits
Tolopology:
the pools of end-systems are directly connected to the GigaEthernet interfaces of switch Cat65006 (testing does not involved additional devices and test traffic is just local). The structure of the topology is illustrated in the following figure:
Parameters:
- CIR
- Normal burst size
- PIR
- Maximum burst size
Traffic profile
- a single point-to-point TCP is generated according to the following command line:
```
Tx: iperf -c 131.154.100.2 -t 3600 -p 50006 -l 16000 -w 64000 -N -i 10
Rx: iperf -s -p 50006 -i 10
	
```
  The application generates memory-to-memory data exchanges.
  The TCP socket size was set to 64000 by (sufficient to achieve maximum throughput, given the small local RTT). Tests with larger socket buffer sizes were run without significant different in performance.

Summary:

TEST B (CIR): the relationship between CIR and minimum normal burst size is peculiar, since in order to get relatively small CIR throughput values, the bucket size has to be increased to several Mbytes (for example, 5 Mbytes for a CIR of 100 Mbps).
This burst tolerance suggests that the minimum burst size corresponds to the maximum burstiness produced by the TCP stream. In other words, for smaller normal burst values than the minimum, a few TCP segments could get dropped by the policer, with the result that TCP reduces its rate.
The high burstiness which seems to be injected by the TCP stream could be due to the fact that source and receiver are connected through high-speed interfaces and to the fact that the RTT is very small. This may allow the sender to transmit several TCP windows back-to-back (at line speed), with a resulting instantaneous rate higher than the CIR.
The linearity between CIR and minimum normal burst cannot be reproduced on lower-speed devices like the C7200 and C7500. This could be due to the fact that the C6500 sends packets at wire speed, while the policer on the C7500 and C7200 is implemented in software, with a consequent internal higher delay which impacts the overall RTT.
TEST C (PIR): The addition of a PIR parameter greatly increases the tolerance to the burstiness sourced by the TCP transmitter, as expected.
With UDP traffic performance perfectly equals the confgiured PIR rate.
TEST D (masimum burst): maximum burst size has to be appropriately tuned to let a given application to reach the peak rate. As expected, the maximum burst greatly increases the tolerance of a policer to the injected TCP burstiness.

Comments:

Test A: variable Normal burst size
```
Command line:
police 50000000 normal-burst-size maximum-burst-size conform-action transmit exceed-action drop
```
Given a reference CIR value equal to 50 Mbps, we see how throughput varies as a function of the normal burst size. In general, for buffer sizes smaller than the congestion window of the TCP application throughput is penalized, since TCP incurs in packet loss as soon as the cwind is extended beyond the normal burst size. Similarly, if the normal burst size is larger than the maximum cwind, then no improvement in performance can be appreciated. This behavior was observed on Cisco routers C7200 and C7500 and IBM routers.
On the other hand, in this test we see that throughput increases linearly when normal burst size increases.
We assume that this behavior may be due to the implementation algorithm adopted on the device. More analysis and understanding of this are needed.
Figure 1: relationship between normal burst size and end-to-end TCP performance
Test B: variable CIR
```
Command line:
police  CIR  normal-burst-size  maximum-burst-size conform-action transmit exceed-action drop
```
For different CIR rates, once fixed a given CIR value, the normal burst size is varied so that the minimum value which allows the TCP applications to get an end-to-end throughput close the CIR is determined.
CIR varies in the range [10, 100] Mbps, the step chosen is fixed and equal to 10 Mbps. The following figure shows that the normal burst has to quite large, in comparison to the packet MTU in use (1500 by) and the socket size: in fact for a CIR target value of 10 Mbps, the minimum buffer size has to be equal to 600 Kby.
For each additional 10 Mbps of target rate approximately 500 Kby of burst size have to be added to the current burst size. This behaviour contradicts the behavior seen on different platforms, according to which for bucket sizes larger than a given threshold no increase in throughput is observed.
Figure 2: minimum normal burst size needed to achieve a target CIR throughput
Test C: variable PIR
```
Command line:
police 50000000 128000 128000 PIR (PIR bit/sec) conform-action transmit exceed-action set-prec-transmit 1 violate action drop
```
In this test we keep the CIR constant and we set the normal burst size to a a constant value (128 Kby), which is not sufficient to let the source get the committed access rate: the rate achieved without PIR configuration is only 5.4 Mbps.
Similarly, the maximum burst is constant and equal to the normal burst size.
The purpose of this test is to evaluate the advantage and the benefit for TCP-like application deriving from the possibility to peak up to a PIR value.
Test are run for a total length of 200 sec, throughput samples are collected every 10 sec, so that an overall number of 20 samples is gathered. The minimum, avg and maximimum throughput achieved during the 200 Mbps is plotted.
The following picture shows that the gain derived from the introduction of a PIR rate is considerable, so that the application is able to get a rate equivalente to the configured PIR.
Figure 3: throughput gain introduced with PIR
The following graph shows shows the different dipendency of the UDP and TCP protocol on the PIR configured for two identical streams, with the only difference that the rate of the UDP stream is constant and equal to 90 Mbps (including the IP and UDP overhead):
```
Tx: iperf -c 192.168.8.3 -t 200 -p 50006 -u -b 88.3 -i 10
Rx: iperf -s -u -p 50006 -i 10
```
As expected, UDP is not dependent of the burst tolerance configured, and the achieved rate exactly equals the PIR configured. This confirms the hypothesis that the performance issues of TCP streams are intrinsically related to the TCP rate control adaptivity. Figure 3b: comparison of TCP and UDP performance as a function of the PIR
Test D: variable Maximum burst size
```
Command line:
police 50000000 128000 maximum-burst-size 71000000 conform-action transmit exceed-action set-prec-transmit 1 violate action drop
```
In this test we keep all the parameters constant with the exception of the maximum burst size, the goal is to evaluate the benefit instroduced by having a maximum burst size and to define a configuration rule.
(CIR = 50Mbpss, normal-burst = 128 KBy, PIR = 71Mbps/s)
Similary to what seen with variable normal burst sizes, the maximum burst size tuning is critical for good TCP performance. The relationship between performance and bucket size is again linear.
The improvement we get when adding a burst size larger than the normal burst size is due to the capability of the second leaky bucket to absorb TCP packets that would be otherwise dropped by the first leaky bucket.
if normal burst size=maximum burst size= 128 Kby, the average throghput is only 33 Mbps, while if the maximum burst size is larger, then the achived throughput can freely increase up to CIR, this implies that PIR is not always sufficient to increase the expected TCP rate, especially when the maximum burst size is not sufficient to store the excess burstiness which was rejected by the first leaky bucket.
Figure 4: relationship between maximum burst size and throughput

Tiziana Ferrari, March 2002