TCN received from stackport1 on 3750

I was troubleshooting a network issue and during this time, I have done the following command on a 3750 stack:

show spanning-tree vlan 200 detail

Bridge Identifier has priority 20480, sysid 200, address a8b1.d36f.c670

  Configured hello time 2, max age 20, forward delay 15

  Current root has priority 8392, address 0021.e809.0670

  Root port is 568 (Port-channel11), cost of root path is 3

  Topology change flag not set, detected flag not set

  Number of topology changes 27 last change occurred 1w2d ago

          from StackPort1

As you can see, the source of the TCN is the StackPort1 on the switch!!!

But what is the meaning of this StackPort1 and how to determinate the real source of the topology change ?

In fact as my SSH session is open on the master and the TCN come from another switch of the stack, the source for my session is the stack port of the master switch. In order to determine which interface is really the source I have to open a session on each switch in order to find the TCN source.

For example, I have opened a session on the 2nd switch of the stack:

C3750#session 2

C3750-2# show spanning-tree detail | i from|exec|occur

 VLAN0001 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 26 last change occurred 1w2d ago

          from GigabitEthernet2/0/52

 VLAN0040 is executing the ieee compatible Spanning Tree protocol

  Number of topology changes 26 last change occurred 1w2d ago

          from GigabitEthernet2/0/52

As you can see above, we have found the real source of the TCN.

Install Netflow collector on Cacti

This tuto explains how to install the flowview plugin on Cacti. I have worked with Cacti 0.8.8b installed on an Ubuntu server.

Install and configure flow-capture

In order to capture netflow traffic, I have used flow-capture. In order to install it on Ubuntu, you can use apt:
apt-get install flow-capture

Once flow-capture is installed, you can configure the flow-capture.conf file:
vim /etc/flow-tools/flow-capture.conf

# Example 1:
# Capture flows from router at 10.1.1.10, listening at port 3000.
# Store flows in /var/netflow/flows/myrouter.
-w /var/netflow/flows/myrouter 0/10.1.1.10/3000

Add the file in dedicated folder:
mkdir /var/netflow/flows/myrouter

Configure a Router in order to export netflow

This configuration is different for each constructor. For example, I have configured an netflow export on a Cisco 4500:

flow record R1
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 collect counter bytes
!
flow exporter CACTI
 destination 10.10.10.10
 export-protocol netflow-v5 => flow-capture is only v5 capable
!
flow monitor M1
 exporter CACTI
 cache entries 1000
 record R1
!
interface Port-channel1
 ip flow monitor M1 input

Install and configure flowview on Cacti

Download the flowview plugin (http://docs.cacti.net/plugin:flowview) and untar it in:
/usr/share/cacti/site/plugins

Go to the Cacti console
Configuration>Plugin Management
And enable Flowview

Configure the path in order to read the netflow file created by flow-capture:
Go to the Cacti console
Configuration>Settings>Misc
Under Flows directory, specified your folder (for example /var/netflow/flows/)

Not able to execute 'copy running-config startup-config' command

Today, I have encountered the following problem:

MY-SWITCH#copy running-config startup-config
startup-config file open failed (Device or resource busy)

In a first step, I was thinking that the nvram: was corrupted because I was not able to see files in the nvram (dir nvram:). In fact, we were 2 users connected on the switch.


MY-SWITCH#show users
    Line       User       Host(s)              Idle       Location
*  1 vty 0     admin      idle                 00:00:00 10.10.10.10 => My session
   2 vty 1     admin      idle                 00:07:09 10.10.20.10

 Also, I have just ejected my colleague with the following command:

MY-SWITCH#clear line 2

After that, I was able to backup my configuration.

Troubleshoot OSPF neighbors (Hellos check)

In order to become neighbors, routers perform several checks. If this check fails, we have to troubleshoot and find the cause of this issue. You will find below several examples of neighbor failed. I have added logs messages and debug messages in order to easily find the cause.

• Area mismatch:

• Authentication key mismatch:

• Duplicate Router-id:

• Subnet/mask Mismatch:

• Area Type Mismatch:

How a phone choose the voice vlan ?

Just by listening the CDP announce coming from the switch:

ICMP Fragmentation and Firewall

Following an installation of new firewalls, I was facing an issue of communication between a CMC (Central Management Console) and a Riverbed Steelhead.
In a normal way, the CMC open a SSH session to push rules to the Riverbed. But after the installation, it was not working anymore. I have investigated and found the root cause of this issue.

Packets between the CMC and the Riverbed are fragmented. Without firewalls, CMC and Riverbed use PMTU in order to discover the MTU small enough to traverse the entire path without fragmentation.
In a first step, Path MTU uses the option Don't Fragment (DF). When a packet is sent with this option, it cannot been fragmented. If a router with a smaller MTU receives this packet, it will drop it and send to the sender an ICMP Fragmentation Needed. This step is repeated until the source has reached the destination without fragmentation.

As you can see below, our problem come from this ICMP Fragmentation Needed packet.
When the steelhead sends a packet (with option Don't fragment) to the CMC, a router on the path sends a ICMP fragmentation Needed. However, the new firewall filters this packet and now the handshake failed.

Workaround:

• Allow ICMP option on the firewall.
• Reduce MTU on primary interface of the Riverbed.

QoS Traffic Policing (drop excess traffic)

Today, I was facing an issue with several users. These users were uploading big files on server (http). Unfortunately, they were using all available bandwidth. It's why, I have decided to police this specific traffic (any users to this server). With the following configuration, the bandwidth for users is limited at 3Mbps (configuration applied on a Layer 3 Switch):

• If the bandwidth exceeds 3Mbps, following packets are dropped:

access-list 100 permit tcp any 10.10.10.200 0.0.0.0 eq www

!

class-map match-all UserTraffic

match access-group 100

!

policy-map policeTraffic

class UserTraffic

    police 3000000 conform-action transmit  exceed-action drop

!

interface Vlan999

service-policy output policeTraffic

• Check statistics:

MYSWITCH#show policy-map  interface vlan 999

Vlan999

  Service-policy output: policeTraffic

    Class-map:UserTraffic (match-all)

      558663 packets, 827048161 bytes

      5 minute offered rate 3643000 bps, drop rate 645000 bps

      Match: access-group 100

      police:

          cir 3000000 bps, bc 93750 bytes

        conformed 460702 packets, 679305595 bytes; actions:

          transmit

        exceeded 97962 packets, 147744080 bytes; actions:

          drop

        conformed 2994000 bps, exceed 669000 bps

    Class-map: class-default (match-any)

      1626596 packets, 568490144 bytes

      5 minute offered rate 3555000 bps, drop rate 0 bps

      Match: any