ICMP Fragmentation and Firewall

Following an installation of new firewalls, I was facing an issue of communication between a CMC (Central Management Console) and a Riverbed Steelhead.
In a normal way, the CMC open a SSH session to push rules to the Riverbed. But after the installation, it was not working anymore. I have investigated and found the root cause of this issue.

Packets between the CMC and the Riverbed are fragmented. Without firewalls, CMC and Riverbed use PMTU in order to discover the MTU small enough to traverse the entire path without fragmentation.
In a first step, Path MTU uses the option Don't Fragment (DF). When a packet is sent with this option, it cannot been fragmented. If a router with a smaller MTU receives this packet, it will drop it and send to the sender an ICMP Fragmentation Needed. This step is repeated until the source has reached the destination without fragmentation.

As you can see below, our problem come from this ICMP Fragmentation Needed packet.
When the steelhead sends a packet (with option Don't fragment) to the CMC, a router on the path sends a ICMP fragmentation Needed. However, the new firewall filters this packet and now the handshake failed.

Workaround:

• Allow ICMP option on the firewall.
• Reduce MTU on primary interface of the Riverbed.

QoS Traffic Policing (drop excess traffic)

Today, I was facing an issue with several users. These users were uploading big files on server (http). Unfortunately, they were using all available bandwidth. It's why, I have decided to police this specific traffic (any users to this server). With the following configuration, the bandwidth for users is limited at 3Mbps (configuration applied on a Layer 3 Switch):

• If the bandwidth exceeds 3Mbps, following packets are dropped:

access-list 100 permit tcp any 10.10.10.200 0.0.0.0 eq www

!

class-map match-all UserTraffic

match access-group 100

!

policy-map policeTraffic

class UserTraffic

    police 3000000 conform-action transmit  exceed-action drop

!

interface Vlan999

service-policy output policeTraffic

• Check statistics:

MYSWITCH#show policy-map  interface vlan 999

Vlan999

  Service-policy output: policeTraffic

    Class-map:UserTraffic (match-all)

      558663 packets, 827048161 bytes

      5 minute offered rate 3643000 bps, drop rate 645000 bps

      Match: access-group 100

      police:

          cir 3000000 bps, bc 93750 bytes

        conformed 460702 packets, 679305595 bytes; actions:

          transmit

        exceeded 97962 packets, 147744080 bytes; actions:

          drop

        conformed 2994000 bps, exceed 669000 bps

    Class-map: class-default (match-any)

      1626596 packets, 568490144 bytes

      5 minute offered rate 3555000 bps, drop rate 0 bps

      Match: any

BGP Decision Process

Nothing new in this post! It's just a reminder regarding the BGP process decision:

1. Weight (Bigger win, Cisco proprietary)
2. LOCAL_PREF (Bigger Win)
3. Locally injected routes (Locally injected win overiBGP/eBGP learned)
4. AS_PATH length (Smaller Win)
5. ORIGIN (code I win over E, E win over ? )
6. MED (Smaller Win)
7. Neighbor Type (eBGP win over iBGP)
8. IGP metric to NEXT_HOP (Smaller win)

Key Based SSH Authentication on a Linux Host using Putty

In order to be more secure, I have decided to use a key instead of a password authentication. I'm using putty to connect to my linux server. I have used the following steps in order to configure it:

• Generate a public and private key with PuTTYgen (click on Generate):

The key is generated by mouving randomly the mouse.

• Save the public and private key.
• The next step is to add the public key on the server:

mkdir ~/.ssh
chmod 700 ~/.ssh

• Paste the public key to the following file:

vim ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

• Configure the putty session:
• Create a new session and save it:

• Select data and configure 'Auto-login' (usersame used by the created key):

• Configure the path to the private key:

After this, if you open the session, you will be directly prompted to the linux host.

OSPF Database Reminder in the same area

This is a reminder in order to interpret an ospf database on a Cisco router. I have created the following lab to test the 'show ip ospf datase':

In red, the router ID:

R3#show ip ospf database

            OSPF Router with ID (3.3.3.3) (Process ID 1)

The list of all routers in the area:

                Router Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum Link count
1.1.1.1         1.1.1.1         129         0x80000002 0x0057AC 1
2.2.2.2         2.2.2.2         25          0x80000003 0x00536B 2
3.3.3.3         3.3.3.3         24          0x80000002 0x0016B6 2

All the network segments in the area and the DR for each segment. Here, 10.1.2.2 is the IP address of the DR on the segment and 2.2.2.2 is the router id:

                Net Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum
10.1.2.2        2.2.2.2         103         0x80000001 0x0021F5
10.1.4.2        2.2.2.2         25          0x80000001 0x006F9D

Each subnet, even if there is no OSPF neighbor:

R1#show ip ospf database internal

            OSPF Router with ID (1.1.1.1) (Process ID 1)

                Stub Link States (Area 0)

Link ID         ADV Router      Age         Seq#       Checksum Mask
10.1.2.255      1.1.1.1         355         0x0        0x006D69 /24
10.1.3.255      2.2.2.2         98          0x0        0x00448D /24
10.1.4.255      2.2.2.2         315         0x0        0x003997 /24
10.1.5.255      3.3.3.3         230         0x0        0x0010BB /24

CCIE Routing & Switching v5 !!

https://www.ciscolivemilan.com/connect/sessionDetail.ww?SESSION_ID=3480

WAIT and SEE!

Reflexive ACLs

Introduction:

By default, an ACL is not stateful. When an 'inside' client open a session to an 'outside' host you have to create 2 ACLs:
- Client to Server
- Server to Client
In our case, we would like open everything coming from the client side and going to the server side. We also deny each session coming from server side. With traditional ACLs it will be really difficult to realize this configuration. It's why, we will use reflexive ACLs. This kind of ACL record session open from the client side and open the necessary port and IP source for the traffic coming back from the specific server.

In our case, we are using a simple test infrastructure as can be seen in the following diagram:

The traffic coming from the client side and going to server side is authorized. Only the response to a session open by a client is authorized (handle dynamically by a recursive ACL). Any other traffic coming from the server side is denied.

Configuration:

In a first step, we create an ACL to reflect outgoing packets:

ip access-list extended OUTBOUND

  permit ip any any reflect MIRROR

!

interface FastEthernet0/1

  ip access-group OUTBOUND out

When the client initiates a session, a reflected ACL is created in MIRROR.

Router#show ip access-lists MIRROR

Reflexive IP access list MIRROR

  permit tcp host 10.1.3.10 eq telnet host 10.1.1.10 eq 27797 (27 matches) (time left 159)

In a second step, we create an ACL which uses the MIRROR entry to authorize the traffic.
ip access-list extended INBOUND

  evaluate MIRROR

!

interface FastEthernet0/1

  ip access-group INBOUND in

 

Results:

Connexion coming from the server side are denied:

SERVER#telnet 10.1.1.10

Trying 10.1.1.10 ...

% Destination unreachable; gateway or host down

Connexion coming from the client side are authorized:

client#telnet 10.1.3.10

 Trying 10.1.3.10 ... Open

 User Access Verification

 Password:

 SERVER>en

As you can see below, a 'reflexive' session is authorized from the server to the client:

Router#show ip access-lists

Extended IP access list INBOUND

  10 evaluate Mirror

Reflexive IP access list Mirror

  permit tcp host 10.1.3.10 eq telnet host 10.1.1.10 eq 27797 (27 matches) (time left 159)

Extended IP access list OUTBOUND

10 permit ip any any reflect Mirror (27 matches)
 

Remarks:

　

By default the timeout for a reflexive ACL is 300sec.

Change phpmyadmin well-know url

In order to access to the web interface of phpmyadmin, by default you have to use the well-know url: mywebsite.com/phpmyadmin.
If you want to change this default url, you can use the following procedure:

• Open the apache.conf file, this file is located in /etc/phpmyadmin (for many Linux servers).

By default, this file looks like:

# phpMyAdmin default Apache configuration

Alias /phpmyadmin /usr/share/phpmyadmin

• Edit the file apache.conf by changing the alias

# phpMyAdmin default Apache configuration

Alias /urlsecret /usr/share/phpmyadmin

• Restart the apache service:

sudo /etc/init.d/apache2 restart

TACACs Attributes

You will find below a compilation of attributes used for authenticate several components with an ACS server. The ACS returns this attribute to the server in order to be authenticated with the good shell privilege.

Nexus:

• Attribute: cisco-avp-pair
• Requirement: Mandatory
• Value: shell:roles*"network-admin vdc-admin"

Riverbed (Steelhead):

• Attribute: local-user-name
• Requirement: Mandatory
• Value: admin

WLC (Cisco Wireless Controller):

• Attribute: role1
• Requirement: Mandatory
• Value: ALL

These equipment have been tested with an ACS 5.4. Don't hesitate to add your remarks.

EIGRP Summary and Leak Map

Route leaking can be used when you have summarized a subnet but at the same time, you also want to announce a more specific subnet. For my lab, I have used the following design:

• In a first step, I have configured my network without summarization:

R1:

interface Loopback10
 ip address 10.1.0.1 255.255.255.0
!
interface Loopback20
 ip address 10.1.1.1 255.255.255.0
!
interface Loopback30
 ip address 10.1.2.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 192.168.1.1 255.255.255.0
!
router eigrp 1
 network 10.0.0.0
 network 192.168.1.0

R2:

interface Loopback10

 ip address 192.168.2.2 255.255.255.0

!

interface FastEthernet0/1

 ip address 192.168.1.2 255.255.255.0

!

router eigrp 1

 network 192.168.0.0 0.0.255.255
 

Results (on R2):

R2# show ip route eigrp

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/24 is subnetted, 3 subnets

D        10.1.0.0 [90/409600] via 192.168.1.1, 00:01:11, FastEthernet0/1

D        10.1.1.0 [90/409600] via 192.168.1.1, 00:01:11, FastEthernet0/1

D        10.1.2.0 [90/409600] via 192.168.1.1, 00:01:11, FastEthernet0/1

As you can see, 3 subnets are seen in the routing table of R2.

• Now, I have configured summary network on R1:

R1:

interface FastEthernet0/0
 ip address 192.168.1.1 255.255.255.0
 ip summary-address eigrp 1 10.1.0.0 255.255.252.0

Results (on R2):

R2#show ip route eigrp

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/22 is subnetted, 1 subnets

D        10.1.0.0 [90/409600] via 192.168.1.1, 00:00:39, FastEthernet0/1

Only one summarized subnet is seen on R2.

• The final step is to configure the route leaking on R1:

R1:

ip access-list standard LEAK-MAP
 permit 10.1.1.0 0.0.0.255
!
route-map LEAK-MAP permit 10
 match ip address LEAK-MAP
!
interface FastEthernet0/0
 ip address 192.168.1.1 255.255.255.0
 ip summary-address eigrp 1 10.1.0.0 255.255.252.0 100 leak-map LEAK-MAP

Results (on R2):

R2#show ip route eigrp

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       + - replicated route, % - next hop override

Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks

D        10.1.0.0/22 [90/409600] via 192.168.1.1, 00:06:10, FastEthernet0/1

D        10.1.1.0/24 [90/409600] via 192.168.1.1, 00:00:34, FastEthernet0/1

 

Both subnet are seen on R2:

• Summarized (10.1.0.0/22)
• Specific (10.1.1.0/24)

 

For me, it's impossible to realize an equivalent configuration with OSPF.

Don't hesitate to leave a comment if you have already seen this configuration for OSPF.

 

MAC Flapping detection

By default, the command mac address-table notification mac-move is not configured on a 4k5. This command permits to detect a MAC flapping with a log message (see below generated logs):

#mac address-table notification mac-move

Jul  9 08:12:26.707: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/16 to port Gi2/18
Jul  9 08:12:28.111: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/18 to port Gi2/16
Jul  9 08:12:28.499: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/16 to port Gi2/18
Jul  9 08:12:28.519: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/18 to port Gi2/16

If this command is not configured you will certainly detect the MAC flapping with a high CPU (if it's monitored!). When you detect a CPU peak, it's not trivial to detect the root cause. To determine whether it's a MAC flapping you can use the following command

C4510#show platform health

                                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total

                                     Target    Actual  Target Actual    Fg   Bg 5Sec Min Hour  CPU

RkiosObflMan                     0.50   0.00      4      0  100  500    0   0    0  51:24

GalChassisVp-review           3.00   0.11     10     40  100  500    0   0    0  1260:27

S2w-JobEventSchedule      10.00   0.90     10      8  100  500    1   1    1  15025:10

...

K5L3Unicast Adj Chan         2.00   0.00     15      1  100  500    0   0    0  12:14

K5L3Unicast Adj Tabl         2.00  11.07     15     11  100  500    6   7    5  55618:52

K5L3AdjStatsMan Revi        2.00   0.15     10     11  100  500    0   0    0  3972:07

...

K5 L2 Unicast Addres          2.00   0.00     20     10  100  500    0   0    0  0:18

K5 L2 Multicast Addr           2.00   0.00     20      0  100  500    0   0    0  0:08

K5 L2 Hardware Addre       2.00  15.06     20     14  100  500   11  12    7  88117:09

As seen above, 2 variables are abnormally high:

• K5L3Unicast Adj Tabl: due to the ARP table refresh. a new MAC address is permanently associated to an IP address.
• K5 L2 Hardware Addre : the relation MAC/physical interface is also permanently refreshed.

To be sure, enter the command mac address-table notification mac-move and check the logs messages.

Find the plain text authentication of a peer RIP router

Followings commands can be useful if you need to use authenticated 'RIP updates' between 2 routers and whether you don't have access to the peer (you don't know the authentication message used by the peer). If it's a clear text, you can use the following debug:

access-list 100 permit udp any any eq rip
debug ip packet 100 dump

Normally, the password will appear in a clear text as below:

*Sep  4 17:27:54.815: IP: s=10.10.10.2 (FastEthernet/1), d=224.0.0.9, len 72, input feature
AC43C9C0:        0100 5E000009 AABBCC00 07030800    ..^...*;L.....
AC43C9D0:  45C00048 00000000 0111C4D0 0A0A0A02  E@.H......DP....
AC43C9E0:  E0000009 02080208 003453BE 02020000  `........4S>....
AC43C9F0:  FFFF0002 54455354 00000000 00000000  ....TEST........
AC43CA00:  00000000 00020000 0A010100 FFFFFF00  ................
AC43CA10:  00000000 00000001                    ........
Router#, MCI Check(68), rtype 0, forus FALSE, sendself FALSE, mtu 0

Track and modify a route with EEM!

This script EEM aims to add or delete a static ip route. Each 10 seconds a ping is sent to a host. If this ping fails a static route is deleted. If this host responds, a route is added. If this host responds and if the route is already in the routing table, no change is done.

event manager applet Route_redisribute
 event tag 1.0 timer watchdog time 10
 action 001 cli command "enable"
 action 002 cli command "ping 10.1.37.3"
 action 003 regexp "!!" "$_cli_result"
 action 004 if $_regexp_result eq 1
 action 005  cli command "show ip route static"
 action 006  regexp "10.12.37.0/24" "$_cli_result"
 action 007  if $_regexp_result ne 1
 action 008   cli command "conf t"
 action 009   cli command "ip route 10.12.37.0 255.255.255.0 10.1.37.8 tag 100"
 action 010  end
 action 011 else
 action 012  continue
 action 013 else
 action 014  cli command "conf t"
 action 015  cli command "no ip route 10.12.37.0 255.255.255.0 10.1.37.8 tag 100"
 action 016 end
!
end


In order to debug this script, the following command is helpful:
debug event manager action cli

Basic NAT reminders

Below, there are some simple examples of NAT configurations on a Cisco router. 

STATIC NAT:

interface fa0/0

 ip address 10.1.1.2 255.255.255.0

 ip nat outside

!

interface fa0/1

 ip address 10.1.3.2 255.255.255.0

 ip nat inside

!

ip nat inside source static 10.1.3.1 10.1.1.10

R3#show ip nat translations

Pro Inside global      Inside local       Outside local      Outside global

--- 10.1.1.10          10.1.3.1           ---                ---

host1#ping 10.1.1.10

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 10.1.1.10, timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms

R3#show ip nat translations

Pro Inside global      Inside local       Outside local      Outside global

icmp 10.1.1.10:5       10.1.3.1:5         10.1.1.1:5         10.1.1.1:5

--- 10.1.1.10          10.1.3.1           ---                ---

PAT:

interface fa0/0
 ip address 10.1.1.2 255.255.255.0
 ip nat inside
!
interface fa0/1
 ip address 10.1.3.2 255.255.255.0
 ip nat outside
!
ip access-list extended NET_INSIDE
 permit ip 10.1.1.0 0.0.0.255 any
!
ip nat inside source list NET_INSIDE interface fa0/1 overload
!


R3#show ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 10.1.3.2:6        10.1.1.1:6         10.1.3.1:6         10.1.3.1:6

NAT SRC & DST:

interface fa0/0

 ip address 10.1.1.2 255.255.255.0

 ip nat inside

!

interface fa0/1

 ip address 10.1.3.2 255.255.255.0

 ip nat outside

!

ip nat inside source static 10.1.1.1 10.1.3.10

ip nat outside source static 10.1.3.1 10.1.1.10

!

ip route 10.1.1.10 255.255.255.255 10.1.3.1

!

R3#show ip nat translations

Pro Inside global      Inside local       Outside local      Outside global

--- ---                ---                10.1.1.10          10.1.3.1

--- 10.1.3.10          10.1.1.1           ---                ---

R3#show ip nat translations

Pro Inside global      Inside local       Outside local      Outside global

--- ---                ---                10.1.1.10          10.1.3.1

icmp 10.1.3.10:10      10.1.1.1:10        10.1.1.10:10       10.1.3.1:10

icmp 10.1.3.10:18      10.1.1.1:18        10.1.1.10:18       10.1.3.1:18

--- 10.1.3.10          10.1.1.1           ---                ---

Dynamic NAT:

interface fa0/0
 ip address 10.1.1.2 255.255.255.0
 ip nat inside
!
interface fa0/1
 ip address 10.1.3.2 255.255.255.0
 ip nat outside
!
ip access-list extended NET_INSIDE
 permit ip 10.1.1.0 0.0.0.255 any
!
ip nat pool NAT_OUTSIDE 10.1.3.1 10.1.3.254 prefix-length 24 type match-host
!!!!!!!!!!!!!!!!!!!!!!
! Alternative:
!ip nat pool NAT_OUTSIDE 10.1.3.9 10.1.3.254 prefix-length 24 type rotary
!!!!!!!!!!!!!!!!!!!!!!!!
!
ip nat inside source list NET_INSIDE pool NAT_OUTSIDE

R3#show ip nat translations
Pro Inside global      Inside local       Outside local      Outside global
icmp 10.1.3.10:27      10.1.1.10:27       10.1.4.1:27        10.1.4.1:27
--- 10.1.3.10          10.1.1.10          ---                ---

Troubleshoot HIGH CPU during the night !

 

You will find below another EEM configuration. This script is useful in order to determine the root cause of an high CPU.

Some high CPU alerts can be generated during off hours. Obviously, nobody is working during the night in order to diagnose this alert! It's why I have created the script below.

If an high CPU syslog message ("%SYS-1-CPURISINGTHRESHOLD") is detected, the command "show proc cpu sorted 5min" is executed. The result of this command is then send to a mailbox.

 

process cpu threshold type total rising 80 interval 30

!

event manager applet ALERT-CPU

event syslog pattern "%SYS-1-CPURISINGTHRESHOLD"

 action 1.0 cli command "enable"

 action 2.0 cli command "show proc cpu sorted 5min"

 action 3.0 mail server "172.16.10.10" to "NetAdmin@mybox.com" from "myswitch@mylab.lab" subject "CPU Alert 5 min" body "$_cli_result"

Cisco Prompt Modification

 

I have recently discovered a practice tips. As on UNIX you can easily modify you prompt on Cisco equipment. You have to use the prompt command with the following variables:
%h - hostname
%n - tty command counter number
%p - prompt character (> or #)
%s - Space
In the following example, I have modified the prompt in order to see on which line I'm connected.
With this variable, I can diretly see if another person is connected on the router:

Before:
RouterLAB #

Command:
#prompt %h%s-%s%n%p

After:
RouterLAB - 2#

Detect High CPU on a Cisco Switch

Recently, I have had an issue on a Core Switch. This switch was running at 80% of CPU for 3 days. The CPU was not monitored and we have not received a syslog message on our server.
Also, I have decided to use the following command (on 4k5) in order to trigger a syslog alert in case of High CPU:

process cpu threshold type total rising 60 interval 20

This command trigger a syslog alert if the CPU exceeds 60 percent for a period of 20 seconds.

EEM - Generate a customized syslog message

In order to generate a customized syslog message, you can use EEM (Embedded Event Manager). This is a Cisco tool. It helps to monitor events and generates an action when an particular event occur. In my case, I would like generate a customized Cisco log message if my primary WAN link is down. This message will have an 'alerts' level with my own description. By default, if an interface goes down it will only generate a 'informational' message. The policy of my syslog server is to generate an email only if I receive an 'error' level message.
I monitor interface giga1/1. If this this interface goes down (log message), I generate my customized message. You can see below the configuration:


event manager applet WanMonitor
 event syslog pattern "Interface GigabitEthernet1/1, changed state to administratively down"
 action 1.0 syslog priority alerts msg "PRIMARY WAN LINK is DOWN on Core 1"