Troubleshoot HIGH CPU during the night !

 

You will find below another EEM configuration. This script is useful in order to determine the root cause of an high CPU.

Some high CPU alerts can be generated during off hours. Obviously, nobody is working during the night in order to diagnose this alert! It's why I have created the script below.

If an high CPU syslog message ("%SYS-1-CPURISINGTHRESHOLD") is detected, the command "show proc cpu sorted 5min" is executed. The result of this command is then send to a mailbox.

 

process cpu threshold type total rising 80 interval 30

!

event manager applet ALERT-CPU

event syslog pattern "%SYS-1-CPURISINGTHRESHOLD"

 action 1.0 cli command "enable"

 action 2.0 cli command "show proc cpu sorted 5min"

 action 3.0 mail server "172.16.10.10" to "NetAdmin@mybox.com" from "myswitch@mylab.lab" subject "CPU Alert 5 min" body "$_cli_result"

Cisco Prompt Modification

 

I have recently discovered a practice tips. As on UNIX you can easily modify you prompt on Cisco equipment. You have to use the prompt command with the following variables:
%h - hostname
%n - tty command counter number
%p - prompt character (> or #)
%s - Space
In the following example, I have modified the prompt in order to see on which line I'm connected.
With this variable, I can diretly see if another person is connected on the router:

Before:
RouterLAB #

Command:
#prompt %h%s-%s%n%p

After:
RouterLAB - 2#

Detect High CPU on a Cisco Switch

Recently, I have had an issue on a Core Switch. This switch was running at 80% of CPU for 3 days. The CPU was not monitored and we have not received a syslog message on our server.
Also, I have decided to use the following command (on 4k5) in order to trigger a syslog alert in case of High CPU:

process cpu threshold type total rising 60 interval 20

This command trigger a syslog alert if the CPU exceeds 60 percent for a period of 20 seconds.

EEM - Generate a customized syslog message

In order to generate a customized syslog message, you can use EEM (Embedded Event Manager). This is a Cisco tool. It helps to monitor events and generates an action when an particular event occur. In my case, I would like generate a customized Cisco log message if my primary WAN link is down. This message will have an 'alerts' level with my own description. By default, if an interface goes down it will only generate a 'informational' message. The policy of my syslog server is to generate an email only if I receive an 'error' level message.
I monitor interface giga1/1. If this this interface goes down (log message), I generate my customized message. You can see below the configuration:


event manager applet WanMonitor
 event syslog pattern "Interface GigabitEthernet1/1, changed state to administratively down"
 action 1.0 syslog priority alerts msg "PRIMARY WAN LINK is DOWN on Core 1"

Giant frames on VSL link

I was checking the VSL link on a network and I have found a lot of giant:

show int Te1/5/5TenGigabitEthernet1/5/4 is up, line protocol is up (connected)
       8 runts, 1739095 giants, 0 throttles

I have searched on cisco.com and I have found the following:
The VSL carries data traffic and in-band control traffic between the two chassis. All frames forwarded over the VSL link are encapsulated with a special 32-byte header, which provides information for the VSS to forward the packet on the peer chassis. 

I guess if a packet close to1500 MTU is received and a 32-byte header is added, we have giant packet. If I'm wrong, don't hesitate to correct me.

VPN connexion and Internet Access

When you are connected to a VPN session, normally you can access to the corporate LAN.
However, it's also possible to have access (in the same time) to Internet.

To do this on ASA, you have to configure some NAT rule.
All traffic that will go to Internet have to use the Internet facing interface like source IP address.

The configuration below is an example that can be applied:

ip local pool Pool_VPN 192.168.40.10-192.168.40.250 mask 255.255.255.0

!

interface GigabitEthernet0/1

 description Outside Facing Interface

 nameif INTERNET

 security-level 0

 ip address 199.199.199.199 255.255.255.240

!

object network NAT-VPN-POOL

 subnet 192.168.40.0 255.255.255.0

 description Pool VPN

!

object network NAT-VPN-POOL

 nat (any,INTERNET) dynamic interface

All traffic with VPN IP Pool source address and with Internet destination will match this NAT rule.
Source address will be modified with the outside interface.

Are you impacted by DNS Changer ?

You can test if you are impacted by the virus DNS changer with the web site below:

http://www.dns-ok.ca/