Command to see all floatings routes

With the classic 'show ip route static', the backup floating route is not seen in the display:

Router#show  ip route static
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       + - replicated route, % - next hop override

Gateway of last resort is 10.10.2.1 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 10.10.2.1

In order to see both routes, active and non-active route, we have to use the command 'show ip static route'. With this command, both routes and metrics are seen:

Router#show ip static route
Codes: M - Manual static, A - AAA download, N - IP NAT, D - DHCP,
       G - GPRS, V - Crypto VPN, C - CASA, P - Channel interface proces
       B - BootP, S - Service selection gateway
       DN - Default Network, T - Tracking object
       L - TL1, E - OER, I - iEdge
       D1 - Dot1x Vlan Network, K - MWAM Route
       PP - PPP default route, MR - MRIPv6, SS - SSLVPN
       H - IPe Host, ID - IPe Domain Broadcast
       U - User GPRS, TE - MPLS Traffic-eng, LI - LIIN
       IR - ICMP Redirect
Codes in []: A - active, N - non-active, B - BFD-tracked, D - Not Track

Static local RIB for default

M  0.0.0.0/0 [1/0] via 10.10.2.1 [A]
M            [5/0] via 10.10.25.1 [N]

TCN received from stackport1 on 3750

I was troubleshooting a network issue and during this time, I have done the following command on a 3750 stack:

show spanning-tree vlan 200 detail

Bridge Identifier has priority 20480, sysid 200, address a8b1.d36f.c670

  Configured hello time 2, max age 20, forward delay 15

  Current root has priority 8392, address 0021.e809.0670

  Root port is 568 (Port-channel11), cost of root path is 3

  Topology change flag not set, detected flag not set

  Number of topology changes 27 last change occurred 1w2d ago

          from StackPort1

As you can see, the source of the TCN is the StackPort1 on the switch!!!

But what is the meaning of this StackPort1 and how to determinate the real source of the topology change ?

In fact as my SSH session is open on the master and the TCN come from another switch of the stack, the source for my session is the stack port of the master switch. In order to determine which interface is really the source I have to open a session on each switch in order to find the TCN source.

For example, I have opened a session on the 2nd switch of the stack:

C3750#session 2

C3750-2# show spanning-tree detail | i from|exec|occur

 VLAN0001 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 26 last change occurred 1w2d ago

          from GigabitEthernet2/0/52

 VLAN0040 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 26 last change occurred 1w2d ago

          from GigabitEthernet2/0/52

As you can see above, we have found the real source of the TCN.

Troubleshoot OSPF neighbors (Hellos check)

In order to become neighbors, routers perform several checks. If this check fails, we have to troubleshoot and find the cause of this issue. You will find below several examples of neighbor failed. I have added logs messages and debug messages in order to easily find the cause.

• Area mismatch:

• Authentication key mismatch:

• Duplicate Router-id:

• Subnet/mask Mismatch:

• Area Type Mismatch:

MAC Flapping detection

By default, the command mac address-table notification mac-move is not configured on a 4k5. This command permits to detect a MAC flapping with a log message (see below generated logs):

#mac address-table notification mac-move

Jul  9 08:12:26.707: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/16 to port Gi2/18
Jul  9 08:12:28.111: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/18 to port Gi2/16
Jul  9 08:12:28.499: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/16 to port Gi2/18
Jul  9 08:12:28.519: %C4K_EBM-4-HOSTFLAPPING: Host A4:CA:DB:48:05:B0 in vlan 50 is moving from port Gi2/18 to port Gi2/16

If this command is not configured you will certainly detect the MAC flapping with a high CPU (if it's monitored!). When you detect a CPU peak, it's not trivial to detect the root cause. To determine whether it's a MAC flapping you can use the following command

C4510#show platform health

                                     %CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total

                                     Target    Actual  Target Actual    Fg   Bg 5Sec Min Hour  CPU

RkiosObflMan                     0.50   0.00      4      0  100  500    0   0    0  51:24

GalChassisVp-review           3.00   0.11     10     40  100  500    0   0    0  1260:27

S2w-JobEventSchedule      10.00   0.90     10      8  100  500    1   1    1  15025:10

...

K5L3Unicast Adj Chan         2.00   0.00     15      1  100  500    0   0    0  12:14

K5L3Unicast Adj Tabl         2.00  11.07     15     11  100  500    6   7    5  55618:52

K5L3AdjStatsMan Revi        2.00   0.15     10     11  100  500    0   0    0  3972:07

...

K5 L2 Unicast Addres          2.00   0.00     20     10  100  500    0   0    0  0:18

K5 L2 Multicast Addr           2.00   0.00     20      0  100  500    0   0    0  0:08

K5 L2 Hardware Addre       2.00  15.06     20     14  100  500   11  12    7  88117:09

As seen above, 2 variables are abnormally high:

• K5L3Unicast Adj Tabl: due to the ARP table refresh. a new MAC address is permanently associated to an IP address.
• K5 L2 Hardware Addre : the relation MAC/physical interface is also permanently refreshed.

To be sure, enter the command mac address-table notification mac-move and check the logs messages.