Cisco 2911 with VPN Module: High CPU Useage Problem

I have been experiencing very high CPU uesage on a Cisco 2911. It is currently our edge device for a 50Mbps data center WAN link. This connection is connecting in to our DMVPN. Here is the first page of output for the sh proc cpu s 5s:

CPU utilization for five seconds: 83%/83%; one minute: 86%; five minutes: 86%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

  96     1440856   154699951          9  0.16%  0.21%  0.21%   0 Ethernet Msec Ti

   2      887752      260607       3406  0.08%  0.06%  0.06%   0 Load Meter      

  67      537724     1298481        414  0.08%  0.03%  0.02%   0 Per-Second Jobs 

 119      444236    38568359         11  0.08%  0.04%  0.05%   0 IPAM Manager    

 192        1796       83981         21  0.08%  0.00%  0.00%   0 FW DP Inspect pr

   3           0           1          0  0.00%  0.00%  0.00%   0 LICENSE AGENT   

   4           0           1          0  0.00%  0.00%  0.00%   0 EDDRI_MAIN      

   7         756         471       1605  0.00%  0.00%  0.00%   0 Pool Manager    

   5           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers

   6     4118308      293279      14042  0.00%  0.04%  0.05%   0 Check heaps     

  11           0           1          0  0.00%  0.00%  0.00%   0 License Client N

   8           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro

  13    13264548       21497     617070  0.00%  0.17%  0.18%   0 Licensing Auto U

  14     4343012     1289560       3367  0.00%  0.05%  0.05%   0 Environmental mo

   9           0           2          0  0.00%  0.00%  0.00%   0 Timers          

  16         924       21497         42  0.00%  0.00%  0.00%   0 IPC Dynamic Cach

  10          28        1332         21  0.00%  0.00%  0.00%   0 WATCH_AFS       

  18           0           1          0  0.00%  0.00%  0.00%   0 IPC Zone Manager

  19       16240     1265108         12  0.00%  0.00%  0.00%   0 IPC Periodic Tim

  12           0           1          0  0.00%  0.00%  0.00%   0 Image License br

  21           0           1          0  0.00%  0.00%  0.00%   0 IPC Process leve

  22           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat Manager

  23           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat RX Cont

  15        3292      257950         12  0.00%  0.00%  0.00%   0 IPC Event Notifi

  25        1564      130306         12  0.00%  0.00%  0.00%   0 IPC Keep Alive M

  26       10204      258446         39  0.00%  0.00%  0.00%   0 IPC Loadometer   

  27           0           1          0  0.00%  0.00%  0.00%   0 Crash writer    

  28           0           1          0  0.00%  0.00%  0.00%   0 Exception contro

  29       19032     1265100         15  0.00%  0.00%  0.00%   0 BGP Scheduler   

  17           0           1          0  0.00%  0.00%  0.00%   0 IPC Session Serv

  31       26916     1340717         20  0.00%  0.00%  0.00%   0 ARP Background  

  32           0           2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer  

  33           0           1          0  0.00%  0.00%  0.00%   0 ATM ASYNC PROC  

  20       14780     1265108         11  0.00%  0.00%  0.00%   0 IPC Deferred Por

  35           0          45          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT

  36           0           1          0  0.00%  0.00%  0.00%   0 Policy Manager  

  37           0           2          0  0.00%  0.00%  0.00%   0 DDR Timers      

  38           0           3          0  0.00%  0.00%  0.00%   0 Entity MIB API  

  24           0           1          0  0.00%  0.00%  0.00%   0 IPC Seat TX Cont

  40        2192          33      66424  0.00%  0.00%  0.00%   0 PrstVbl         

  41           0           2          0  0.00%  0.00%  0.00%   0 Serial Backgroun

  42           0           1          0  0.00%  0.00%  0.00%   0 RMI RM Notify Wa

  43           4           2       2000  0.00%  0.00%  0.00%   0 SMART           

  44       22804     1297957         17  0.00%  0.00%  0.00%   0 GraphIt         

  45           0           2          0  0.00%  0.00%  0.00%   0 Dialer event    

  46           0           1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect 

  47           0           2          0  0.00%  0.00%  0.00%   0 XML Proxy Client

  30      269096     1078488        249  0.00%  0.00%  0.00%   0 ARP Input       

  49       33436      135132        247  0.00%  0.00%  0.00%   0 Net Background  

The sh proc cpu h command shows the router at or above 90% for the last 72 hours with spikes up to 100%.

The device on the other end of this conenction is a Cisco 2821 with a VPN module WS-X4548-RJ45V+. This device is running at 30-34% useage.

Is anyone aware of any problems with the 2911?

The answer:

2911 throughput is 35 Mb. 2821 throuput is 86 Mb. VPN accelerator card can just help for encryption of IPSEC. all IP flow conttrols by CPU. Because of it you depend on throuput of your router model.

Regarding the 2911 you can read in this one: http://www.3anetwork.com/blog/

WS-X4448 uplink trunk

As far I read on official Cisco website, the 48 ports 10.100.1000 line card WS-X4606-X2-E is made of 6 groups of 8 ports each connected to 1Gbits trunk to Supervisor.

What if I have 4 servers, so 4 ports, in the same vlan, same subnet, connected to the same group of 8 ports. Those 4 servers need to communicate at 1Gbits rate between eachother. Does the traffic goes up to the supervisor, so I would have 250mbits per server ? or the local traffic stays in the group on the local card and everybody got 1gbits ?

The solution:

When you have devices in the same subnet, the switch only needs to use the CAM table to forward the traffic.  The CAM table contains the MAC address and associated port in which the MAC address is attached. So basically only L2 is used. 

6500 and 4500 series switches use centralized switching, "With centralized switching, routing, ACL, QoS, and forwarding decisions are made on the Supervisor Engine in a modular chassis"

Forwarding architecture: These modules use the central Cisco Express Forwarding engine located on the supervisor engine.

• Forwarding performance: These modules forward packets up to 30 Mpps per system and up to 15 Mpps per slot if upgraded to support distributed forwarding.

You are correct in your assumption - there is no "local switching" on that linecard, so every packet must go through the oversubscrbied 1GigE backplane connection.

If you have a 4500 and any Sup before the Sup6, and have servers that must communicate at 1GigE, then you need to put the servers into different port groups.  For example, server #1 in port 1, then shut down ports 2 through 8 with description "OVERSUBSCRIBED."  Server #2 would go into port 9, then shut down ports 10 through 16, etc.

This is a limitation of the 4500...  For the newer 4500-E and the Sup6-E, the backplane has increased from 6GigE to 24GigE.  You have a couple options:

  WS-X4624-SFP-E: 24-port SFP (pluggable) with full line-rate.  For typical server connectivity you would have to get a TX SFP (GLC-T)

  WS-X4648-RJ45-E: 24-port 10/100/1000 with 2:1 oversubscription.  For a high-traffic server, you would just connect on every other port.

Cisco WS-C6504-E Swap Out

I have a WS-C6504-E (similar to WS-X4606-X2-E) switch that is failing to power up one of the two PWR-2700-AC PSU.  I have another WS-C6504-E switch on the way and i am going to swap them out.

My question is, my configuration files that are on my current switch, are they stored on the chassis themselves or one of my cards in the chassis?

Currently i have the following installed in my 6504

1- VS-S720-10G-SC 720 Supervisor blade

1- WS-X6724-SFP  24 port gig SFP blade

I was wondering if it would be as simple as taking these cards out and popping them in the new chassis.

The best solution:

All your configuration is saved on the Supervisor.

Cisco 4500 6500 and 7600 are modular design, apart from the main chassis, supervisor engine, linecard, power supplies and modules will be also required for these products. 3Anetwork.com wholesales Cisco 4500 6500 7600 Modules, ship to worldwide.

3Anetwork.com is a world leading Cisco networking products wholesaler, we wholesale original new Cisco networking equipments, including Cisco Catalyst switches, Cisco routers, Cisco firewalls, Cisco wireless products, Cisco modules and interface cards products at competitive price and ship to worldwide.

Our website: http://www.3anetwork.com

Telephone: +852-3069-7733

Email:  info@3Anetwork.com

Address: 23/F Lucky Plaza, 315-321 Lockhart Road, Wanchai, Hongkong

How to Reload the Cisco Modules

How to Reload the Cisco switching Modules? Below are the details and steps.

Switching modules automatically download their images from the supervisor module, and do not need a force download. This procedure is provided for reference should a need arise.

To replace the image on a switching module, follow these steps:

Step 1   Identify the switching module that requires the new image (see the "Upgrading Software Images on Modules" section).

Step 2   Issue the reload module number force-dnld command to update the image on the switching module.

switch# reload module number force-dnld

Where number indicates the slot in which the identified module resides. For example, if the identified module resides in slot 9:

switch# reload module 9 force-dnld...

Jan  1 00:00:46 switch %LC-2-MSG:SLOT9 LOG_LC-2-IMG_DNLD_COMPLETE: COMPLETED

downloading of linecard image. Download successful...

Customers interested in purchasing Cisco modules or the price, two great modules for you:

WS-X4448-GB-SFP

WS-X4548-RJ45V+

Request a quote for Cisco modules by emailing us at info@3Anetwork.com or pick up the phone and speak to one of our friendly and helpful sales reps at +852-3069-7733.

3Anetwork.com is a world leading Cisco networking products wholesaler, we wholesale original new Cisco networking equipments, including Cisco Catalyst switches, Cisco routers, Cisco firewalls, Cisco wireless products, Cisco modules and interface cards products at competitive price and ship to worldwide.

Our website: http://www.3anetwork.com

Telephone: +852-3069-7733

Email:  info@3Anetwork.com

Address: 23/F Lucky Plaza, 315-321 Lockhart Road, Wanchai, Hongkong.

NM-CUE Module in Cisco 2911 Router, How to upgrade

I have a Cisco 2911 Router running CME and CUE.  I am running CUE on a NM-CUE Module.  I found out yesterday that this module had reached end of life and end of support from Cisco.  I have a support contract on this Router, and the NM-CUE Module was listed as a minor serial number and was covered under the contract.  I just renewed the cisco smartnet contract and the NM-CUE module did not show up as a minor serial number anymore.  I contacted Cisco and they said it was because it at end of support.  What module do I need to upgrade to?  How do I upgrade?

The solution:

You can use nme-cue module but I believe that is also going for an end of sales soon. (not sure about the date).

NME-CUE / AIM2-CUE-K9 / NME-UMG-EC / NME-UMG are the modules available as of now for 2811. But the router itself is end of sales and end of life.

We would suggest to run with the existing module you have and when cisco no longer will support 2911 (in a year or 2), go for a new setup with new router and module.

To be precise you will need NME-CUE, just in case you want to upgrade. That's not going to be a very wise investment though considering end of sales/life support for Cisco router 2911. It should be an emergency plan ONLY in case the nm-cue stops working.

What is my SFP model number, without unplugging it?

I'm needing to know the model number of an SFP connector plugged in to a Cisco Catalyst 3750 [WS-C3750X-48PF-S , IOS 12.2(50)SE2, C3750-IPBASEK9-M image].  I know it's using multi-mode fiber and I have a list of supported SFP connectors from Cisco's web site:

GLC-T (10/100/1000)

GLC-SX-MM

GLC-LH-SM

GLC-SX-MMD

GLC-LH-SMD

GLC-ZX-SM

CWDM SFP

SFP-GE-S

SFP-GE-L

SFP-GE-Z

GLC-BX-D

GLC-BX-U

DWDM SFP

DWDM SFP (add.)

But I can't figure out which one it is, and what the wavelength is.  I can connect to the switch fine, and all "show int" tells me is that the media type is 1000BaseSX SFP.  Is there a way to figure this out?  That network segment shouldn't go down at all, which is why I don't want to unplug the SFP transceiver.  Thanks.

The advice:

Try show inventory raw and see if that shows the info you are looking for. Also try:

show interface gix/x capabilities  (x/x being the interface numbering).

If the show interface gi1/0/1 capabilities didn't  list the model number, post the output.

It certainly shows the model number of the switch itself for some reason:

NBU-SC-1#show interfaces gigabitEthernet 1/0/1 capabilities

GigabitEthernet1/0/1

  Model:                 WS-C3750X-48P-S

  Type:                  1000BaseSX SFP

  Speed:                 1000

  Duplex:                full

  Trunk encap. type:     802.1Q,ISL

  Trunk mode:            on,off,desirable,nonegotiate

  Channel:               yes

  Broadcast suppression: percentage(0-100)

  Flowcontrol:           rx-(off,on,desired),tx-(none)

  Fast Start:            yes

  QoS scheduling:        rx-(not configurable on per port basis),

                         tx-(4q3t) (3t: Two configurable values and one fixed.)

  CoS rewrite:           yes

  ToS rewrite:           yes

  UDLD:                  yes

  Inline power:          no

  SPAN:                  source/destination

  Por

tSecure:            yes

  Dot1x:                 yes

NBU-SC-1#

Then, Base on the above you have GLC-SX-MM. That is a common SFP used.

Cisco 10 GIG SFP Problems

I am having an issue with 10 gig interfacing between a 6507 Core Switch with a 4 port 10 Gig  optical blade using SC GBIC's connecting to a Nexus 5548 using 10 Gig optical SFP's.

The link is configured as a Trunk link and is passing traffic.  The issue is with MTU...no packet larger than 255 bytes will pass.

6507 -  SC -------------------------------------LC  NEXUS

GBIC    -------------------------------------------  SFP

10/GIG OPTICAL  _  -----------------   10 GIG OPTICAL

TRUNK ----------------------------------------- TRUNK

The solution:

You can set upto 9216 bytes of MTU size in Nexus.(jumbo frames)

http://www.cisco.com/en/US/products/ps9670/products_configuration_example09186a0080b44116.shtml

but the factor depends on ASIC..

You may be interested in configuring jumbo frame on 6500 series

Configure in CatOS

    Cat6509≶ (enable) set port jumbo

    Usage: set port jumbo <mod/port> <enable|disable>

    Cat6509> (enable) set port jumbo 1/1 enable

    Jumbo frames enabled on port  1/1.

    Cat6509> (enable) 2002 May 29 12:34:35 %PAGP-5-PORTFROMSTP:

    Port 1/1 left bridge port 1/1

    2002 May 29 12:34:38 %PAGP-5-PORTTOSTP:Port 1/1 joined bridge port 1/1

Verify in CatOS

    Cat6509> (enable) show port jumbo

    Jumbo frames MTU size is 9216 bytes.

    Jumbo frames enabled on port(s) 1/1,9/1.

keep MTU size same at two ends.

Or if not interested in jumbo..simply configure the mtu size to 1500 b

You may check for the commands on ur switches

Something like

Switch(config)# mtu.

The Cisco module you may want to know, please click the below links:

WS-X4748-RJ45-E

WS-X4748-RJ45V+E