rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_green
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_green
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#191 — FS#4279 — p19-53/p19-54

Attached to Project— Network
Incident
Whole Network
CLOSED
100%
Since 7 am this morning, we had a strange problem between Nexus 5000 and Catalyst 6500.



Date:  Sunday, 13 June 2010, 12:41PM
Reason for closing:  Done
Comment by OVH - Saturday, 12 June 2010, 09:32AM

We have an input packet problem on the 6000. None on the Nexus 5000.

We have restarted the p19-53-n5 then p19-54-n5.
They are in cluster so each took the other's relay. The problem persists.

The problem is on all 6K ports that are connected towards N5.


p19-57-6k#sh inter counters errors module 6

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Te6/1 0 0 0 0 0 0
Te6/2 0 0 0 0 0 0
Te6/3 0 0 0 0 0 0
Te6/4 0 0 0 74 0 0

On the N5 we use a virtual port channel on 2 equipments.


187 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

It is like if the Nexus are balancing such traffic that 6k did not take ...

hmm ... we will increase the input's queue size on the 6k

wrr-queue bandwidth 255 255 255 255 255 255 255
wrr-queue queue-limit 100 100 100 100 100 100 100
wrr-queue threshold 1 100 100 100 100 100 100 100 100
wrr-queue threshold 2 100 100 100 100 100 100 100 100
wrr-queue random-detect min-threshold 3 100 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100
wrr-queue random-detect max-threshold 2 100 100 100 100 100 100 100 100
wrr-queue cos-map 1 4 0 1
wrr-queue cos-map 3 1 6
wrr-queue cos-map 7 8 7
rcv-queue bandwidth 255 255 255 255 255 255 255 255
rcv-queue queue-limit 100 100 100 100 100 100 100 100
rcv-queue threshold 1 100 100 100 100 100 100 100 100
rcv-queue random-detect min-threshold 1 100 100 100 100 100 100 100 100
rcv-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100
rcv-queue cos-map 1 1 2 3
rcv-queue cos-map 1 8 0 1
rcv-queue cos-map 2 8 4 6
rcv-queue cos-map 7 8 7
rcv-queue cos-map 8 8 5


Comment by OVH - Saturday, 12 June 2010, 09:34AM

We will try to check if it would be better with the flowcontrol.


Comment by OVH - Saturday, 12 June 2010, 09:46AM

On n5 we can put flowcontrol on the port channel, but not on the physical ports, and on 6K we can put flowcontrol on the physical port but not on port channel.
I love it.


Comment by OVH - Saturday, 12 June 2010, 09:48AM

The errors still increase, yet there is no problem of MAC detection
between the cards of load repartition and the servers.

p19-57-6k#sh inter counters errors module 6

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Te6/1 0 0 0 0 0 0
Te6/2 0 0 0 0 0 0
Te6/3 0 0 0 0 0 0
Te6/4 0 0 0 146 0 0

Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants
Te6/1 0 0 0 0 0 0 0
Te6/2 0 0 0 0 0 0 0
Te6/3 0 0 0 0 0 0 15
Te6/4 0 0 0 0 0 0 0

Port SQETest-Err Deferred-Tx IntMacTx-Err IntMacRx-Err Symbol-Err
Te6/1 0 0 0 0 0
Te6/2 0 0 0 0 0
Te6/3 0 0 0 0 0
Te6/4 0 0 0 0 0

We will add a 2nd 10G port in the channels port
between the 6k and the N5. If we distribute the traffic
on 2 10G ports instead of 1, it must functions well.


Comment by OVH - Sunday, 13 June 2010, 12:41PM

With 4x10G /6500, it goes well as soon as a 10G port
takes on more than 300'000 packets/seconds from a N5,
nevertheless it has errors on the input.

We will ascend the errors at Cisco.


Comment by OVH - Sunday, 13 June 2010, 12:41PM

p19-57-6k#sh inter t6/4 | i 30 sec
30 second input rate 1276919000 bits/sec, 316372 packets/sec
30 second output rate 1149697000 bits/sec, 204839 packets/sec
p19-57-6k#sh inter t7/1 | i 30 sec
30 second input rate 364833000 bits/sec, 68728 packets/sec
30 second output rate 1142250000 bits/sec, 205848 packets/sec
p19-57-6k#sh inter t7/3 | i 30 sec
30 second input rate 1404992000 bits/sec, 344054 packets/sec
30 second output rate 1042913000 bits/sec, 199855 packets/sec
p19-57-6k#sh inter t7/4 | i 30 sec
30 second input rate 342808000 bits/sec, 65034 packets/sec
30 second output rate 1081846000 bits/sec, 194390 packets/sec

p19-57-6k#sh inter cou err mod 6
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Te6/4 0 0 0 271 0 0
Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants
Te6/4 0 0 0 0 0 0 0
Port SQETest-Err Deferred-Tx IntMacTx-Err IntMacRx-Err Symbol-Err
Te6/4 0 0 0 0 0
p19-57-6k#sh inter cou err mod 7
Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards
Te7/1 0 0 0 0 0 0
Te7/3 0 0 0 142 0 0
Te7/4 0 0 0 0 0 0
Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants
Te7/1 0 0 0 0 0 0 0
Te7/3 0 0 0 0 0 0 0
Te7/4 0 0 0 0 0 0 0
Port SQETest-Err Deferred-Tx IntMacTx-Err IntMacRx-Err Symbol-Err
Te7/1 0 0 0 0 0
Te7/3 0 0 0 0 0
Te7/4 0 0 0 0 0