Subscribe to Brad Reese speaks out

Has the Cisco CRS-3 caused network outages at both Comcast and AT&T?
Tue, 1/18/11 - 11:59pm    View comments

It's nice to have loyal readers.

Why?

Because they pass along great tips about Cisco!

And their latest Cisco tip concerns the CRS-3 (Carrier Routing System) which on March 8, 2010 Cisco proudly announced and ballyhooed to the entire world:

"Cisco will host a live media, blogger and industry analyst webcast Tuesday, March 9, at 8:00 a.m. (PST) to make a significant announcement that will forever change the Internet and its impact on consumers, businesses and governments."

So here's the tip:

The Cisco CRS-3 has caused network outages at both Comcast and AT&T while Cisco continues to deny any problems with the CRS-3, blaming carrier configs for the issue.

Meanwhile, Comcast and AT&T have formally petitioned both Cisco and Broadcom on the issue...

And so what's exactly the issue?

Well, its called bit flipping.

The bit flip is an interesting phenomenon because the Cisco CRS-3 is rumored to have used Broadcom 3rd party silicon (view Cisco Systems' Fear of a Broadcom Planet), which did not use ECC protected memory subsystems and did not use low-alpha particle lead. Normal lead occasionally kicks out an alpha particle and with transistor densities being what they are today it can cause a bit to "stick" in a memory subsystem. This then causes the memory to corrupt and then pretty much snowballs into at least a minimum of having the ASIC being reset, and then possibly having an entire system reset.

This happened most famously with the Cisco 4500 and 6500 and the Toshiba SRAMs back in the 2002-2004 time frame, causing tens of millions of dollars of hardware recall.

Bottom line: There's no real "field fix" in most cases and a line card swap out is called for.

I find this interesting because Cisco CEO John Chambers stated during Cisco's Q1'FY11 earnings conference call:

"Just as an update, the customer acceptance from the pilot perspective on the CRS-3 is off to a great start. In Q1, we shipped our first CRS-3 system. We received $51 million in orders from 30 customers. However, we expect there will probably be several more quarter before we see rapid increase in these accounts from dual high-end routers as they test out all the new systems of this magnitude before they begin volume commitments."

Finally in my opinion, the above Cisco CRS-3 bit flipping issue should be brought to the attention of Cisco's customers.

Related stories:

Cisco's CRS-3 flexible packet transport vs. Juniper's T series, JCS1200 and PTX series

Wall Street's take on the new Cisco CRS-3 router

What's your take, have you heard about or experienced this issue with the Cisco CRS-3?

Subscribe to Brad Reese speaks out

1. Cisco's the worst performing stock on the world famous Dow Jones Industrial Average (DJIA)
2. Download for FREE - Enterprise Application Firewalls for Dummies
3. Cisco 4G LTE Wireless WAN WICs: EHWIC-4G-LTE-A, EHWIC-4G-LTE-G, EHWIC-4G-LTE-V
4. A whopping 75% of Cisco's new job creation was in locations outside the United States
5. HP networking Cisco trade-in program - A Catalyst for Change
6. Cisco's SP edge router market share down -12% since 2005
7. 14% of Cisco CCIEs appear to be inactive or suspended
8. HP E5400 zl and HP 8200 zl switches vs. Cisco Catalyst 3750-X and 4500 switches
9. Cisco CEO John Chambers appears to have knowingly lyed to Wall Street
10. Slide presentation detailing Cisco's strategy for acquiring LineSider
11. Cisco's 2010 annual meeting of shareholders visual slide presentation
12. Cisco's management and employees appear to have pocketed $32 billion of Cisco shareholder cash
13. Cisco's tax scandal growing $1.510 million per day
14. Cisco earned lowly niche player status in new Gartner magic quadrant report while simultaneously losing market share
15. Why it's going to get ugly at Cisco
16. Aruba beats Cisco in WLAN user survey as revenue soars +44%
17. Aruba Networks Q1'FY11 sales expected to increase +47% Y/Y
18. According to the stock market, Cisco's issues appear to be Cisco specific
19. Padmasree Warrior is leading Cisco's sequentially declining security and switch sales
20. View archive of Brad Reese speaks out