Multicast data check with Perl IO::Socket::Multicast

I spend a lot of time dealing with multicast, and one of the more tedious tasks is verifying that I’m receiving data across all of the groups I need. Whether it’s setting a baseline prior to a network change, or verifying everything works after the change, it’s an important task that gets more and more tedious as the number of groups increases.

I’m not a programmer, but I play one on TV.

I decided to try to write something in Perl, which is no small task considering I’ve never excelled when it comes to coding/scripting. Fortunately I know enough to be dangerous, and by combining some elements I’ve seen across various forums and reading the perl documentation, I was able to cobble together something that accomplishes the task of automating my tests.

In my first version I was able to successfully read in group information from a file, and wait for data on each line. The problem was that the program executed serially, so if I had a list of 50 groups, and I needed to wait a full 60 seconds for line, I could potentially wait 50 minutes for the test to complete.

The second version forks those tests out to individual child processes, dramatically decreasing the time it takes to verify a large list of groups.

I’m posting the code here in case someone out there is struggling with a similar issue, and is also not a programmer by nature. I’m sure there are lots of better ways to accomplish this in code. As I said, I’m not a programmer. But this seems to work. If you have any comments on ways to make it more efficient or handle something better, I’d love to hear about it in the comments.

Code


#!/usr/bin/perl

use strict;
use warnings;
use IO::Socket::Multicast;


my $grouplist = $ARGV[0];

my $GROUP = '';
my $PORT = '';
my @childproc;
my $data;
print "Parsing File...\n";

##Attempt to open the file provided on the CLI
open(FILE, $grouplist) || die "can't open file: $!";

my @list = ;

print "Contents: \n", @list;

print "Processed Data:\n";
foreach (@list) {
        chomp($_);
        ($GROUP, $PORT) = split (':',$_);
        my $pid = fork();
        if ($pid) {
                push(@childproc, $pid);
        } elsif ($pid == 0) {
                datacheck ($GROUP, $PORT);
                exit 0;
        } else {
                die "Couldn't Fork: $!\n";
        }
}

close(FILE);

foreach (@childproc) {
        my $tmp = waitpid($_, 0);
}

print "End of Testing\n";

sub datacheck {
        my $GROUPSUB = $_[0];
        my $PORTSUB = $_[1];
        my $msock = IO::Socket::Multicast->new(Proto=>'udp',LocalAddr=>$GROUPSUB,LocalPort=>$PORTSUB,ReuseAddr=>1);
        $msock->mcast_add($GROUPSUB) || die "Couldn't set group: $!\n";

    eval {
        $SIG{ALRM} = sub { die "timeout" };
        ## The Alarm value decides how long to wait before giving up on each test
        alarm(10);
        $msock->recv($data,64);
        alarm(0);
        };

        if ($@) {
                print "$GROUPSUB:$PORTSUB -- No data received\n";
        } else {
                print "$GROUPSUB:$PORTSUB  -- Data received\n";
        }

}

I’ve tested this on CentOS with Perl 5.8.8. This script requires the IO::Socket::Multicast perl module, so be sure to install that before you attempt to run it.

The script expects you to provide the name of a text file that contains a group:port mapping per line.

Update: I discovered that I could run into problems if my list ever contained more than one of the same group, or more than one of the same port. After consulting the module documentation, I realized that I needed to add the LocalAddr and the ReuseAddr options to the constructor code.

Redistributing Anyconnect VPN addresses into OSPF on Cisco ASA

I’m a big fan of the Cisco Anyconnect VPN client due to its easy configuration, and the relative ease of deployment to end users. When you deploy an Anyconnect VPN on your ASA, one of the important tasks is to decide how to advertise the VPN assigned addresses into the rest of your network. Fortunately, this is easy to accomplish using route redistribution.

Basic Setup

In this example, my VPN pool will be assigned from the 192.168.254.128/25 range, and I will redistribute these routes into OSPF. Notice that the ASA automatically creates a static host route for a connected client:

ASA# sh route | i 192.168.254
S    192.168.254.154 255.255.255.255 [1/0] via 1.1.1.1, Outside

So we have the building blocks for what we need, now let’s look at the configuration.

There are several different ways to accomplish this task, but I’ll demonstrate what I typically use.

Redistributing into OSPF

First, we’ll create a prefix list to match the address pool for our Anyconnect clients:

prefix-list VPN_PREFIX seq 1 permit 192.168.254.128/25 le 32

This prefix list entry matches the 192.168.254.128/25 subnet, as well as any routes with a mask less-than or equal to 32 bits. This works great, because our routes will all be /32.

Next we’ll create a route-map that we can reference inside OSPF:

route-map VPN_POOL permit 1
    match ip address prefix-list VPN_PREFIX

And finally, we’ll add enable redistribution in OSPF:

router ospf 1
    redistribute static subnets route-map VPN_POOL

If we look the routing table on another router in our network, we should see the route:

RTR#sh ip route | i 192.168.254
O E2     192.168.254.128/32 [110/20] via 10.5.2.6, 00:5:03, Vlan85

Advertising the subnet instead of individual host routes

If you like to keep your routing tables uncluttered, you might be inclined to only redistribute the entire VPN prefix, instead of the /32 routes. The important thing to remember here is that OSPF will not redistribute a route that is not already in the routing table.

We’ll simply add a static route for the VPN prefix:

route outside 192.168.254.128 255.255.255.128 1.1.1.1

Without any other modifications, we will now see routes like this in our network:

RTR#sh ip route | i 192.168.254
O E2     192.168.254.128/25 [110/20] via 10.5.2.6, 00:07:28, Vlan85
O E2     192.168.254.154/32 [110/20] via 10.5.2.6, 00:07:28, Vlan85

But we want to get rid of the /32 routes. So we have two options now:

  1. Modify the prefix-list to match only the /25 route
  2. Modify the OSPF redistribution command to ignore subnets.

Option 1: Modify the prefix-list

We’ll change the prefix list so we don’t even consider subnets with different masks:

no prefix-list VPN_PREFIX seq 1 permit 192.168.254.128/25 le 32
prefix-list VPN_PREFIX seq 1 permit 192.168.254.128/25

Our redistribution command still has the subnets keyword, but since the prefix list won’t even allow smaller prefix lengths, we end up with just the one route.

Option 2: Modify the OSPF redistribution command

You can also remove the subnets keyword from the redistribution command:

router ospf 1
    redistribute static route-map VPN_POOL

This way it doesn’t matter if the prefix-list matches longer routes, OSPF just won’t redistribute them.

Final Configuration

In the end we have a configuration that looks something like this:

route outside 192.168.254.128 255.255.255.128 1.1.1.1
!
prefix-list VPN_PREFIX seq 1 permit 192.168.254.128/25
!
route-map VPN_POOL permit 1
 match ip address prefix-list VPN_PREFIX
!
router ospf 100
 redistribute static route-map VPN_POOL

The ASA will still show all of the /32 routes, plus the /25 route:

ASA# sh route | i 192.168.254
S    192.168.254.154 255.255.255.255 [1/0] via 1.1.1.1, Outside
S    192.168.254.128 255.255.255.128 [1/0] via 1.1.1.1, Outside

But routers inside the network will only see the /25 route:

RTR#sh ip route | i 192.168.254
O E2     192.168.254.128/25 [110/20] via 10.5.2.6, 01:45:03, Vlan85

I didn’t talk about modifying any of the OSPF metrics as the routes are being injected, but that would be something to consider if you do this in your environment.

IOS CLI historical interface graphs

I was researching something for a project recently and came across a feature I hadn’t seen before:  historical interface graphs.

With this feature, you can enable up to 72 hours of traffic statistics on your interfaces, and you can view this data via the CLI, similar to how ‘show proc cpu history’ works.

Check it out:

Interface-history-cli

I’d never seen this before, so I was quite excited. If you’re like me, and haven’t tried this out yet, here is how you configure it:

interface Gig0/0
    history {bps | pps} [filter]

The filter can include a lot of different items, including:

  • input-drops
  • input-error
  • output-drops
  • output-errors
  • overruns
  • pause-input
  • pause-output
  • crcs

and the list goes on. You can see a full table of supported filters in the IOS Command reference.  I found this worked on my 7200s, ASR1Ks, ISRs.

A glimpse into LISP Control-Plane traffic

In our lab we were able to configure LISP and verify connectivity between our two hosts. One thing I noticed was the loss of the first two ICMP packets. Let’s walk through how LISP functions and examine what was happening behind the scene.

Upon receipt of the first packet, the SITE1 router (acting as an ITR) checked the LISP map cache to see if it already had an RLOC mapping for the destination EID (172.16.30.101):

SITE1#sh ip lisp map-cache
LISP IPv4 Mapping Cache for EID-table default (IID 0), 2 entries

0.0.0.0/0, uptime: 00:10:16, expires: never, via static send map-request
  Negative cache entry, action: send-map-request

This negative cache entry tells the router that it needs to send a map request to see if there’s an RLOC mapping available. The router will send a map request for EID 172.16.30.101/32 to the Map resolver at 192.168.255.3, and drop the initial data packet (ping #1) from our host:

LISP-Map-Reply

The Map Resolver/Map Server checks the namespaces that have registered, and looks for the RLOC address of the ETR that is authoritative for the EID prefix:

MR_MS#sh lisp site name SITE2
Site name: SITE2
Allowed configured locators: any
Allowed EID-prefixes:
  EID-prefix: 172.16.30.0/24
    First registered:     00:31:21
    Routing table tag:    0
    Origin:               Configuration
    Merge active:         No
    Proxy reply:          No
    TTL:                  1d00h
    State:                complete
    Registration errors:
    Authentication failures:   0
    Allowed locators mismatch: 0
    ETR 192.168.100.2, last registered 00:00:50, no proxy-reply, map-notify
                        TTL 1d00h, no merge, hash-function sha1, nonce 0x0524E21F-0x489614BB
                        state complete, no security-capability
                        xTR-ID 0xDC4A8044-0x87093251-0x602669CB-0x5B720F12
                        site-ID unspecified
    Locator        Local  State      Pri/Wgt
    192.168.100.2  yes    up          10/50 

Once the Map Server determines the RLOC for the authoritative ETR, it forwards the Map-Request message.  The ETR receives the forwarded Map-Request, and responds with a Map-Reply:

LISP-Map-Reply

Once the SITE1 router receives the reply, it updates the local cache:

SITE1#sh ip lisp map-cache
LISP IPv4 Mapping Cache for EID-table default (IID 0), 3 entries

0.0.0.0/0, uptime: 00:10:35, expires: never, via static send map-request
  Negative cache entry, action: send-map-request
0.0.0.0/1, uptime: 00:09:25, expires: 00:05:34, via map-reply, forward-native
  Negative cache entry, action: forward-native
172.16.30.0/24, uptime: 00:09:22, expires: 23:50:38, via map-reply, complete
  Locator        Uptime    State      Pri/Wgt
  192.168.100.2  00:09:22  up          10/50

Now that the router has a complete LISP cache, it can encapsulate packets in LISP headers and send them on their way.

LISP-Data-Packet-Header

LISP Data Packet Payload

In this setup, it’s interesting to note that we lost the first two ICMP packets to the control-plane process. The first packet was dropped by the SITE1 router as it went through the Map Request/Map Reply process to build the local cache. The second packet actually made it through to the other host, but the response was dropped by the SITE2 router as it also had to build the local cache. You can see some of that below:

Ping response sequence

Once the caches have been built, subsequent attempts are 100% successful:

[root@SITE1 ~]# ping -c 5 172.16.30.101
PING 172.16.30.101 (172.16.30.101) 56(84) bytes of data.
64 bytes from 172.16.30.101: icmp_seq=1 ttl=255 time=1.62 ms
64 bytes from 172.16.30.101: icmp_seq=2 ttl=255 time=1.41 ms
64 bytes from 172.16.30.101: icmp_seq=3 ttl=255 time=1.36 ms
64 bytes from 172.16.30.101: icmp_seq=4 ttl=255 time=1.48 ms
64 bytes from 172.16.30.101: icmp_seq=5 ttl=255 time=1.33 ms

--- 172.16.30.101 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4000ms
rtt min/avg/max/mdev = 1.338/1.442/1.626/0.114 ms

Conclusion

I think there are a few important items to remember about the LISP forwarding process. These probably seem obvious, but I still want to point them out:

  1. The mapping system is really more of a ‘director’, in that it doesn’t actually know the answers to queries, but knows who to talk to find out.
  2. LISP control-plane always uses the same source and destination ports: UDP/4342
  3. LISP data-plane packets are always destined for UDP/4341, but the source port will change.

Cisco has a website dedicated to LISP and you can find a great deal of information there, including the devices and software versions that support LISP.  I’d highly recommend checking it out:  Cisco LISP

Simple LISP Lab

We’re going to start with a very simple topology:

LISP-topology

Instead of using one of our standard routing protocols to advertise the host networks between the SITE1 and SITE2 routers, we’ll rely on LISP. OSPF will be used to advertise the loopback on the MS/MR router within the Core network. We’ll start configuring LISP assuming the basic network is ready to go.

First, lets setup LISP the SITE1 router:

router lisp
 locator-set SITE1
 192.168.100.1 priority 10 weight 50
 exit
!
database-mapping 192.168.5.0/24 locator-set SITE1
!
ipv4 itr map-resolver 192.168.255.3
ipv4 itr
ipv4 etr map-server 192.168.255.3 key secretkey
ipv4 etr
!

Now we’ll configure LISP on the SITE2 router:

router lisp
 locator-set SITE2
 192.168.100.2 priority 10 weight 50
 exit
!
database-mapping 172.16.30.0/24 locator-set SITE2
!
ipv4 itr map-resolver 192.168.255.3
ipv4 itr
ipv4 etr map-server 192.168.255.3 key secretkey
ipv4 etr
!

Finally, we’ll configure the Mapping Server and Mapping resolver functionality on the MS/MR router:

router lisp
site SITE1
 authentication-key secretkey
 eid-prefix 192.168.5.0/24
 exit
!
site SITE2
 authentication-key secretkey
 eid-prefix 172.16.30.0/24
 exit
!
ipv4 map-server
ipv4 map-resolver
exit

Once this is configured, our LISP infrastructure should be complete. Let’s check:

SITE1#sh ip lisp
  Instance ID:                      0
  Router-lisp ID:                   0
  Locator table:                    default
  EID table:                        default
  Ingress Tunnel Router (ITR):      enabled
  Egress Tunnel Router (ETR):       enabled
  Proxy-ITR Router (PITR):          disabled
  Proxy-ETR Router (PETR):          disabled
  Map Server (MS):                  disabled
  Map Resolver (MR):                disabled
  Delegated Database Tree (DDT):    disabled
  Map-Request source:               192.168.5.1
  ITR Map-Resolver(s):              192.168.255.3
  ETR Map-Server(s):                192.168.255.3 (00:00:55)
  xTR-ID:                           0x3D4C0900-0x95932BE0-0x2F5AF1F6-0x4C919E94
  ...

And over on SITE2:

SITE2#sh ip lisp
  Instance ID:                      0
  Router-lisp ID:                   0
  Locator table:                    default
  EID table:                        default
  Ingress Tunnel Router (ITR):      enabled
  Egress Tunnel Router (ETR):       enabled
  Proxy-ITR Router (PITR):          disabled
  Proxy-ETR Router (PETR):          disabled
  Map Server (MS):                  disabled
  Map Resolver (MR):                disabled
  Delegated Database Tree (DDT):    disabled
  Map-Request source:               172.16.30.1
  ITR Map-Resolver(s):              192.168.255.3
  ETR Map-Server(s):                192.168.255.3 (00:00:16)
  xTR-ID:                           0xDC4A8044-0x87093251-0x602669CB-0x5B720F12
  ...  

On the MS/MR Router, we can see that the ETR’s have registered with the LISP mapping system. Without this, LISP wouldn’t know the EID-to-RLOC mapping for each EID.

OTV-RTR3#sh lisp site
LISP Site Registration Information

Site Name      Last      Up   Who Last             Inst     EID Prefix
               Register       Registered           ID
SITE1          00:00:14  yes  192.168.100.1                 192.168.5.0/24
SITE2          00:00:17  yes  192.168.100.2                 172.16.30.0/24

Now, to show that LISP is working correctly, lets first show that we don’t have a route to the opposite site:

SITE1#sh ip route 172.16.30.101
% Network not in table
SITE1#ping 172.16.30.101
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.30.101, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

SITE2# sh ip route 192.168.5.101
% Network not in table
SITE2#ping 192.168.5.101
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.5.101, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

And since we didn’t configure a default route, there isn’t anything to fall back on.

Let’s test by pinging the SITE2 host:

[root@SITE1 ~]# ping 172.16.30.101
PING 172.16.30.101 (172.16.30.101) 56(84) bytes of data.
64 bytes from 172.16.30.101: icmp_seq=3 ttl=255 time=1.26 ms
64 bytes from 172.16.30.101: icmp_seq=4 ttl=255 time=4.57 ms
64 bytes from 172.16.30.101: icmp_seq=5 ttl=255 time=1.31 ms
^C
--- 172.16.30.101 ping statistics ---
5 packets transmitted, 3 received, 40% packet loss, time 4421ms
rtt min/avg/max/mdev = 1.268/1.374/1.466/0.086 ms

Looks good except for the two we lost at the beginning. Let’s try the reverse ping now:

[root@SITE2 ~]# ping 192.168.5.101
PING 192.168.5.101 (192.168.5.101) 56(84) bytes of data.
64 bytes from 192.168.5.101: icmp_seq=1 ttl=255 time=1.49 ms
64 bytes from 192.168.5.101: icmp_seq=2 ttl=255 time=1.31 ms
64 bytes from 192.168.5.101: icmp_seq=3 ttl=255 time=1.26 ms
64 bytes from 192.168.5.101: icmp_seq=4 ttl=255 time=4.57 ms
64 bytes from 192.168.5.101: icmp_seq=5 ttl=255 time=1.31 ms
^C
--- 192.168.5.101 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 1.178/1.278/1.398/0.090 ms

So we have two way communication across our network, using LISP to locate and reach the opposite site. Next time we’ll dig a little deeper on what’s happening behind the scenes.

Cisco LISP

With technologies like OTV, we need a method to optimize traffic destined for our mobile virtualized hosts, by tracking the location and updating our underlying routing system. One possible solution is known as LISP.

LISP stands for Locator ID Separation Protocol, and its function is to allow you to separate the location component of an IP address, from the identity portion of the address. Or as the Cisco LISP configuration guide states, LISP “implements the use of two namespaces instead of a single IP address.” What does this mean?

With LISP you are introduced to two new concepts:

  • Endpoint Identifiers (EID)
  • Routing Locators (RLOC)

The endpoint identifier (EID) is the address used to identify a specific host — this is the same as the IP addresses you use today, and it is said to be in the LISP namespace. The Routing Locator (RLOC) is the address of a router that is part of the normal routing domain but is connected to the LISP namespace and the non-LISP namespace. The RLOC is said to be part of the non-LISP namespace.

One significant difference with LISP is that you no longer have to advertise the EID address space into the normal routing domain. Instead, you rely on LISP to provide mappings between EIDs and RLOCs, and you route based on the RLOC address.

I like to think of LISP as a DNS-like system for routing, because an important part of LISP is a mapping system that maintains EID-to-RLOC mappings. Just like you use DNS to query for name-to-IP mappings, a LISP router performs a query against a LISP mapping server to find out the RLOC that should be used to reach the desired EID.

Components of a LISP system

An complete LISP infrastructure will consist of many parts:

  • Ingress Tunnel Routers (ITR)
  • Egress Tunnel Routers (ETR)
  • XTR
  • Map Server (MS)
  • Map Resolver (MR)
  • Proxy Ingress Tunnel Routers (PITR)
  • Proxy Egress Tunnel Routers (PETR)
  • PXTR

ITR

The ingress tunnel router receives unencapsulated IP packets from the EID namespace, and is responsible for performing lookups to identify EID-to-RLOC mappings for destination addresses. If the packet is destined for an EID in another LISP namespace, the ITR will encapsulate each packet with a LISP header and route the packet towards the identified RLOC. If the packet is destined for a non-LISP address, the packet is routed without any LISP modifications.

ETR

The egress tunnel router receives LISP encapsulated packets from the non-LISP portion of the network, removes the LISP header, and delivers the unencapsulated packets to the EID. The ETR is also responsible for keeping the Mapping system up to date with EID mappings and responding to Mapping system requests.

LISP XTR

An XTR is a router that performs both ETR and ITR functions.

Map Server

The map server receives EID registrations from ETRs, and responds to map request messages that are forwarded from map resolvers.

Map Resolver

The map resolver receives encapsulated map request messages from ITRs and forwards them to Map Servers that are authoritative for the EID namespace being queried.

PITR/PETR/PXTR

The Proxy ITR/Proxy ETR allows non-LISP sites to communicate with LISP sites, and vice-a-versa, by performing ITR and ETR functionality. They can be deployed separately, or together. If deployed together the device is referred to as a PXTR.

Conclusion

It took me a little while to wrap my head around the LISP concept, and one big help was working with it in a lab environment.  In the next post, I’ll walk through a lab scenario to demonstrate LISP in action.

OTV Traffic Flow Considerations

The beauty of OTV is that you are no longer limited to segregating your L2 VLAN’s based on site, or location within your network.  When used in conjunction with Virtual Machines, this means you can migrate machines between locations without having to modify IP addressing, giving you the ability to move entire server farms with only a few clicks.

Beauty has an ugly side, however. One of the not insignificant challenges with OTV is knowing how to best reach endpoints within the overlay network. Improper planning in this area can result in inefficient traffic flows through your network, and could possibly block end-end traffic altogether. Consider the following network:

lisp-network

Let’s say your host is in DC1, but your gateway is in DC2. How will traffic move through the network?  Often called ‘traffic tromboning,’ this is where traffic enters through one side of your network, and uses the overlay to trombone across to the opposite side before returning back through the original datacenter.

lisp-network-trombone

It’s ugly, but you can fix that by using an FHRP to have a gateway in each site. As we know, the ASR’s have FHRP filtering configured and enabled by default, and there is documentation on how to configure filtering for the N7K. After adding gateways to both sites, you end up with this:

lisp-network-trombone2

Well, that might be ok — if you turn a blind eye to the traffic flowing across  your core multiple times, but it’s certainly not the most efficient.  But to add insult to injury, what if your two sites have their own path out to the internet?  How will your edge firewall respond when it receives traffic for a connection it doesn’t know about?

Conclusion

These are just some of the issues that need to be considered when evaluating an OTV solution.  Multiple entry/exit points, firewall placement, flow lifetime, load-balancers, etc. combine to make the overall design complicated very quickly.   Add  in endpoint mobility (the whole point, right?), and you have to ensure that new flows will know how to reach the correct endpoint, and old flows either persist or can be reestablished quickly.  In my next post, I’ll discuss one of the solutions I’m exploring to solve these issues.