Flexible Netflow on the 4500X

I’m a big fan of Solarwinds and their suite of network management products. If you’ve never seen or tried their products, head on over to their demo site and check it out. I recently added their Netflow product Network Traffic Analyzer and wanted to add netflow collection to my new 4500X switches.

The 4500X only support Flexible Netflow, aka Version 9, and doesn’t include any prebuilt flow record templates, so there are basically 4 steps to the configuration:

  • Create a flow record
  • Create a flow exporter
  • Create a flow monitor
  • Apply the monitor to an interface

Let’s look at each and go through a basic configuration.

Flow Record

The flow record defines the fields that will be used to group traffic into unique flows. Key fields are those which are used to distinguish traffic flows from each other. If a key field is different and doesn’t match an existing flow, a new flow will be added. Key fields are matched.

Non key fields aren’t used to distinguish flows from each other, but are collected as part of the data set you want to glean from each flow. Non key fields are collected.

For my flow record I used the following configuration:

flow record IPV4-FLOW-RECORD
    match ipv4 tos
    match ipv4 protocol
    match ipv4 source address
    match ipv4 destination address
    match transport source-port
    match transport destination-port
    collect interface input
    collect interface output
    collect counter bytes long
    collect counter packets long

So in my flow record, each flow will be distinguished by ToS, Protocol, Src/Dst address, src/dst port. I’m also interested in collecting the input and output interfaces, as well as the number of bytes and packets in each flow.

Flow Exporter

A flow exporter is basically a place to send the flow data you collect. By default, Cisco will send data to UDP/9995 but Orion expects it to arrive on UDP/2055. I also specified the source interface for the data so it will match the address that Orion uses to manage this node.

flow exporter Orion
    source Loopback0
    transport udp 2055

Flow Monitor

The flow monitor is where you link records and exporters together.

flow monitor IPV4-FLOW
    description Used for Monitoring IPv4 Traffic
    record IPV4-FLOW-RECORD
    exporter Orion

Once you’ve defined all the elements, it’s time to apply to an interface.

Applying the configuration

This particular 4500X install doesn’t have any routed interfaces, so my intention was to apply the flow monitor to an SVI. This resulted in the following error:

4500X-1(config-if)#ip flow monitor IPV4-FLOW input
% Flow Monitor: Flow Monitor 'IPV4-FLOW' : Configuring Flow Monitor on SVI interfaces is not allowed.
Instead configure Flow Monitor in vlan configuration mode via the command `vlan config <vlan number>'

Ok, we’ll try again:

4500X-1(config)#vlan config 2
4500X-1(config-vlan-config)#ip flow monitor IPV4-FLOW input

No problems there!

When I attempted to configure the flow monitor in the output direction, I received this error:

4500X-1(config-if)#ip flow monitor IPV4-FLOW output
% Flow Monitor: 'IPV4-FLOW' could not be added to interface due to invalid sub-traffic type: 0

I reread the Flexible netflow section in the configuration guide, and sure enough the very first limitation for 4500’s in a VSS configuration:

  1. The Catalyst 4500 series switch supports ingress flow statistics collection for switched and routed packets; it does not support Flexible Netflow on egress traffic.

Looks I won’t be able to configure collection for output statistics,at least not at this junction in the network.


Overall I thought the configuration was fairly straight forward. I ended up using the same configuration on the other routers in my network and this was the only instance where I was unable to collect output traffic statistics.

Cisco OTV – Overlay Transport Virtualization

First, let’s talk about what supports OTV — not much:

  • Nexus 7K
  • ASR 1K
  • CSR 1000V (For those of you not familiar with the Cloud Services router, I’d recommend reading this

What is OTV?

OTV is an encapsulation protocol that wraps L2 frames in IP packets in order to transport them between L2 domains. Typically this would be between remote datacenters, but it could also be within a datacenter if you needed an easy (expensive) way to extend a VLAN.

You will also see OTV referred to as ‘MAC Routing’, since the OTV devices are essentially performing routing decisions based on the destination MAC address in the L2 frame.

You might be thinking “Hey, I’ve already got this with EoMPLS and/or VPLS.” And you’d be right — you have the essence of what OTV accomplishes. What OTV adds, however, is simplicity and fault isolation.

When you configure OTV, you are defining 3 elements:

  • Join interface
    This is the interface that faces the IP core that will transport OTV encapsulated packets between sites.
  • Overlay InterfaceThis is the virtual interface that will handle the encapsulation and decapsulation of OTV packets sent between OTV edge devices.
  • Inside interface This is the interface that receives the traffic that will be sent across OTV.

What do I need before I can configure OTV?

Before you can setup OTV in your environment there are a few important details to know:

  • OTV adds 42 bytes of overhead into the packet header. This has implications if your MTU size is 1500 bytes (the default in most cases). You’ll need to either enable Jumbo frames across your core, or reduce the MTU size on your servers inside the OTV domain. UPDATE: You can enable OTV fragmentation by using the global command otv fragmentation join-interface.  I don’t know if this has any performance implications, but at least it’s an option for you if changing the MTU throughout your network is difficult.
  • With the latest code releases, I believe all platforms support either Unicast or Multicast for the OTV control-plane. If you have a multicast enabled core, use multicast — it’s really not too bad.

Topology and Configuration

For my topology I’m going to use two ASR 1K’s, a 4900M with two VRFs, and two 3550 switches. I know I could’ve left out the VRFs, but I wanted to make my topology as close as possible to real-life. So we end up with this:

Sample OTV topology

So let’s move on to the OTV configuration.

OTV Site information

Part of any OTV config will be defining the site identifiers and the Site Bridge-Domain. The site identifier is how an OTV device determines whether or not it is at the same location as another OTV device.


otv site-identifier 0001.0001.0001


otv site-identifier 0002.0002.0002

The site bridge-domain is the Vlan that the OTV edge devices at the same site will use for AED election. Since this VLAN will not be part of the overlay, we can use the same command on both routers.

otv site bridge-domain 100

The Join interface

The join interface will be the source for all OTV packets sent to remote OTV routers, and it will be the destination for OTV packets that need to come to the site. For multicast control-plane implementations you’ll need to enable Passive PIM and IGMPv3.


interface Gig0/0/1
mtu 8192
ip address
ip pim passive
ip igmp version 3

Also note that the MTU has been adjusted to accommodate the increased size of the OTV packet. This will be the same on the second OTV-RTR except for the IP address.

Overlay Interface

In the overlay interface configuration we have to specify the multicast group used for control messaging, as well as the range of multicast groups that will be used for passing multicast data within the VLAN. We will also specify which interface will be used as the join interface. This will be the same on both routers:

interface Overlay1
otv control-group
otv data-group
otv join-interface GigabitEthernet0/0/1
no shutdown

Once you turn up the Overlay interface on both sides, you should see your OTV adjacency form:

OTV-RTR1#show otv adjacency
Overlay 1 Adjacency Database
Hostname                       System-ID      Dest Addr       Up Time   State
OTV-RTR2                       c08c.6008.0f00       00:00:36  UP

At this point since there isn’t a Vlan bridged to the Overlay, there will be now OTV routing information:

OTV-RTR1#show otv route

Codes: BD - Bridge-Domain, AD - Admin-Distance,
       SI - Service Instance, * - Backup Route

OTV Unicast MAC Routing Table for Overlay1

 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)

0 unicast routes displayed in Overlay1

0 Total Unicast Routes Displayed

Adding Vlans to the Overlay

The last step will be to add the appropriate VLAN’s to the overlay. This config assumes that the router will receive the traffic from the switch with an 802.1Q tag:

interface GigabitEthernet0/0/0
service instance 250 ethernet
encapsulation dot1q 250
bridge-domain 250
interface Overlay1
service instance 250 ethernet
encapsulation dot1q 250
bridge-domain 250


I created a Vlan interface on each switch to use as my ‘hosts’ for the ping tests.

Sw-1 VL250 = 0009.b709.4b80

Sw-2 VL250 = 0009.b717.7880

Pinging between devices is successful. Let’s look at the switches to see how it looks:


Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 250    0009.b709.4b80    DYNAMIC     Gi0/1


OTV-RTR1#sh otv route

OTV Unicast MAC Routing Table for Overlay1

Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
 0    250  250    0009.b709.4b80 50    ISIS   OTV-RTR2
 0    250  250    0009.b716.7880 40    BD Eng Gi0/0/0:SI250

So we can see that SW-1 knows to reach Sw-2 out interface Gi0/1, which connects to OTV-RTR1. OTV-RTR1 shows that it’s learned the MAC for SW-2 via OTV(ISIS) from OTV-RTR2. So anytime it receives frames for this MAC, it knows to forward them across the overlay.


OTV-RTR2#sh otv route

OTV Unicast MAC Routing Table for Overlay1

Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
0    250  250    0009.b709.4b80 40    BD Eng Gi0/0/0:SI250
0    250  250    0009.b716.7880 50    ISIS   OTV-RTR1

OTV-RTR2 shows that SW-2 is out the local service-instance. Any packets that come across the overlay will be decapsulated and forwarded out the local interface.

Wrap Up

Getting a basic OTV config up and running is not that difficult. Next time I’ll talk about using unicast instead of multicast, and also about AED.

Multicast Performance on ESX

Multicast data has always been somewhat of a mystery to network engineers unless they have a very specific reason for using it. Since the financial industry is a heavy user of multicast, I have been fortunate to get my hands very dirty in it throughout my career.

One item that has always vexed our group is how we can consolidate our multicast workloads, and extend the efficiency gains of virtualization to this segment of our environment. These boxes represent a significant cost, and they often go under utilized in terms of CPU/Memory. But because of the nature of the data, it’s difficult to try anything that can degrade performance.

In ESX 5.0, Vmware introduced a new technology that is supposed to help alleviate the performance bottlenecks

  • splitRxMode

I’ll summarize the feature here as described in the Technical Whitepaper:


In previous versions of ESX, all network receive processing for a queue was performed inside a single context within the VMKernel. splitRxMode allows you to direct each vNIC (configured individually) to use a separate context for receive packet processing.

They make a special note to indicate that even though it improves receive processing for multicast, it does incur a CPU penalty due to the extra overhead per packet, so don’t enable it on every machine.


In their testing, VMWare labs reported that they observed 10-25% packet loss on a 16Kpps multicast stream once the number of subscriber VM’s went past 24. After they enabled splitRxMode, the packetloss was < 0.01% all the way up to 32 VM’s on the host.

My Take

Even though VMWare seems confident that the recent IO improvements with splitRxMode will increase multicast performance, there are some key considerations here:

  1. 0.01% is still a lot of packet loss — at 16Kpps, that’s still over 1pps
  2. The scenario they tested is for a one-to-many situation (one stream to multiple receivers). What if the packet rate is higher or the number of streams is higher, but the receiver count is low?

Obviously this requires a lot more testing on our part before we’d ever even consider rolling anything to production. If you have any experience in this regard, please feel free to comment and offer any insights/suggestions you might have.

NOTE: This entry is my first foray into technical blogging. I’ve learned a lot from the blogs I’ve read over the years, and I’ve also found that these types of blogs are the absolute best resource for solving real problems. I hope I can contribute something meaningful and perhaps repay some of what I’ve been given.