Cisco Nexus 5010 and 5020 vs. Arista 7124S and 7148SX competitive lab test
Sat, 4/23/11 - 6:00am View comments
Cisco commissioned the networking product testing consultancy, Miercom, to conduct an independent third-party performance test of the Cisco Nexus switch models 5010 and 5020 versus the Arista switch models 7124S and 7148SX.
The testing focused on evaluating the switches with bursty traffic profiles and included measurements of latency and packet loss with traffic for 128-byte frames. The systems under test included:
Cisco Nexus 5010 switch - running Cisco NX-OS v 4.1(3)N1(1). The switch is a 1 RU, 10 Gigabit Ethernet switch supporting 20 fixed 10 Gigabit Ethernet ports and one expansion module slot fitted with a 6 port 10 GE module with only four used for testing to ensure the products compared had the same port count for fair comparison.
Cisco Nexus 5020 switch - running Cisco NX-OS v 4.1(3) N2 (1a). The switch is a 2 RU, 10 Gigabit Ethernet switch supporting 40 fixed 10 Gigabit Ethernet ports and two expansion module slot fitted with an 8 port 10 GE module with only four used on each for testing to ensure the products compared had the same port count for fair comparison.
Arista 7124S switch - running EOS v 4.3.0. The switch is a 1 RU supporting (24) 10-Gigabit Ethernet ports.
Arista 7148SX switch - running EOS v 4.3.2. The switch is a 1 RU supporting 48 fixed 10-Gigabit Ethernet ports.
The tests measured the latency and loss characteristics of datacenter traffic patterns traversing through a network platform. Seven traffic profiles were simulated: 2-to-many multicast, 4-to-many multicast, 8-to-many multicast, 23-to-1 unicast, full-mesh multicast, 47-to-many unicast and full-mesh unicast. All tests were conducted for 128-byte frame sizes:
The Arista 7124S starts dropping packets at bursts of 1250 frames with a 0.05% traffic loss observed. 1250 frames equates to 148.0 µsec at 128 bytes. The Nexus 5010 continues to forward frames with increasing average latency until the burst size reaches 3250 frames. At this burst rate, the Arista switch drops 31.0% of traffic and the Nexus 5010 drops 2.5% of traffic. For this traffic pattern the Nexus can support 2.6 (3250/1250) times the level of burst traffic versus Arista without loss.
23-to-One Unicast
The Arista switch starts dropping packets at bursts of 60 frames with a 5.0% traffic loss observed. 60 frames equates to 7.1 µsec at 128 bytes. The Nexus 5010 continues to forward frames with increasing average latency until the burst size reaches 1600 frames. At this burst rate, the Arista switch drops 92.2% of traffic and the Nexus 5010 drops 3.9% of traffic. For this traffic pattern the Nexus can support 26.7 (1600/60) times the level of burst traffic versus Arista without loss.
Full-Mesh Multicast
12 Ports
The Arista switch starts dropping packets at bursts of 150 frames with a 15.7% traffic loss observed. 150 frames equates to 17.8 µsec at 128 bytes. Cisco Nexus continues to forward frames with increasing average latency until the burst size reaches 1600 frames. At this burst rate, the Arista switch drops 83.9% of traffic and the Nexus 5010 drops 0.6% of traffic. For this traffic pattern the Nexus 5010 can support 10.7 (1600/150) times the level of burst traffic versus Arista without loss.
24 Ports
Arista 7124S starts dropping packets at bursts of 60 frames with a 5.0% traffic loss observed. 60 frames equates to 7.1 µsec at 128 bytes. The Nexus 5010 continues to forward frames with increasing average latency until the burst size reaches 1600 frames. At this burst rate, the Arista switch drops 92.3% of traffic and the Nexus 5010 drops 4.2% of traffic. For this traffic pattern the Nexus can support 26.7 (1600/60) times the level of burst traffic versus Arista without loss.
Full-Mesh Unicast
12 Ports
The Arista 7124S starts dropping packets at bursts of 60 frames with a 6.8% traffic loss observed. 60 frames equates to 7.1 µsec at 128 bytes. Cisco Nexus continues to forward frames with increasing average latency until the burst size reaches 275 frames. At this burst rate, the Arista switch drops 50.7% of traffic and the Nexus 5010 drops 0.6% of traffic. For this traffic pattern the Nexus 5010 can support 4.6 (275/60) times the level of burst traffic versus Arista without loss.
24 Ports
The Arista switch starts dropping packets at bursts of 15 frames with a 9.6% traffic loss observed. 15 frames equates to 1.8 µsec at 128 bytes. The Nexus 5010 continues to forward frames with increasing average latency until the burst size reaches 120 frames. At this burst rate, the Arista switch drops 64.6% of traffic and the Nexus 5010 drops 1.9% of traffic. For this traffic pattern the Nexus can support 8.0 (120/15) times the level of burst traffic versus Arista without loss.
Arista 7148SX starts dropping packets at bursts of 1250 frames with a 0.1% traffic loss. 1250 frames equates to 148.0 µsec at 128 bytes. The Nexus 5020 continues to forward frames with increasing average latency until the burst size reaches 3250 frames. At this rate, the Arista switch drops 20.9% of traffic and the Nexus 5020 drops 1% of traffic. With this traffic pattern the Nexus can support 2.6 (3250/1250) times the level of burst traffic versus Arista without loss.
4-to-Many Multicast
Arista 7148SX starts dropping packets at bursts of 425 frames with a 0.4% traffic loss. 425 frames equates to 50.3 µsec at 128 bytes. The Nexus 5020 continues to forward frames with increasing latency until the burst size reaches 2100 frames. At this rate, the Arista switch drops 48.3% of traffic and the Nexus 5020 drops 1.1% of traffic. In this traffic pattern the Nexus can support 4.9 (2100/425) times the level of burst traffic versus Arista without loss.
8-to-Many Multicast
Arista 7148SX starts dropping packets at bursts of 180 frames with a 0.2% traffic loss. 180 frames equates to 21.3 µsec at 128 bytes. The Nexus 5020 continues to forward frames with increasing latency until the burst size reaches 1750 frames. At this rate, the Arista switch drops 77.1% of traffic and the Nexus 5020 drops 0.2% of traffic. With this traffic pattern the Nexus can support 9.7 (1750/180) times the level of burst traffic versus Arista without loss.
Full Mesh Multicast
Arista 7148SX starts dropping packets at bursts of 35 frames with a 5.1% traffic loss. 35 frames equates to 4.14 µsec at 128 bytes. The Nexus 5020 continues to forward frames with increasing latency until the burst size reaches 1600 frames. At this rate, the Arista switch drops 94.3% of traffic and the Nexus 5020 drops 3.3% of traffic. In this traffic pattern the Nexus can support 45.7 (1600/35) times the level of burst traffic versus Arista without loss.
47-to-1 Unicast
Arista 7148SX starts dropping packets at bursts of 30 frames with a 1.3% traffic loss. 30 frames equates to 3.5 µsec at 128 bytes. The Nexus 5020 continues to forward frames with increasing latency until the burst size reaches 1600 frames. At this rate, the Arista switch drops 91.1% of traffic and the Nexus 5020 drops 3.1% of traffic. With this traffic pattern the Nexus can support 53.3 (1600/30) times the level of burst traffic versus Arista without loss.
Full-Mesh Unicast
Arista 7148SX starts dropping packets at bursts of 15 frames with a 9.5% traffic loss. 15 frames equates to 1.78 µsec at 128 bytes. The Nexus 5020 continues to forward frames with increasing latency until the burst size reaches 55 frames. At this rate, the Arista switch drops 54.4% of traffic and the Nexus 5020 drops 1.7% of traffic. In this traffic pattern the Nexus can support 3.6 (55/15) times the level of burst traffic versus Arista without loss.
Arista Networks' official response to the above Miercom lab test results:
Cisco commissioned Miercom to compare Arista 7100S series switches against the Cisco Nexus 5000 series switches. The tests used so called "typical real world application traffic scenarios" and were focused on finding thresholds where packets were dropped by the respective switches. To Miercom's credit, these tests are clearly described, correctly run, and accurately reported.
Arista objects to Cisco's characterization of these tests as being similar to "real world" traffic. This claim is deeply misleading. These tests were clearly designed to make the Nexus 5010 appear better than the Arista 7124S at all costs. Most of the testing was focused on fan-in with multicast cases - as much as 23-to-1 or even 47-to-1 fan-in. Each test case is custom - no standard RFCs were used, which have been a benchmark in the industry for several years now to compare products.
Product Architectures:
First, it is important to understand the product architectures - we will leave it to the reader to decide the co-relation between the architecture and the tests noted in the report.
Nexus 5000:
The Nexus 5000 buffer memory is statically associated with a given input port, 480KB per port for each of its ports. Nexus buffer is allocated in chunks of 140 bytes.
Arista 7100:
The 7124S has 2MB of centralized packet buffer. After subtracting off various overheads, there are 1718KB of usable memory with 20KB reserved per port and an additional 1238KB usable by any port experiencing congestion. The 7148SX has 4 times the memory of 7124S.
The two architectures are substantially different when it comes to buffering. The Nexus 5000 will perform better when most ports are congested at the same time. Arista 7100 switches will perform better when some ports get congested at any given time, which is the common case in most networks.
What was Tested:
96 and 128 byte packets were used. Tests were n-to-many unicast or multicast - essentially the same test repeated over and over again where as many as 23 or 47 sources were used, all simultaneously sending packets to each of the remaining ports, one-by-one. This ganging up on the destination port results in massive congestion, resulting in packet drops. Multicasts, as we all know it is a one-to-many protocol. Not many-to-one! In these tests, that was turned around, simply to cause congestion and show drops. Which network in the world has 23 hosts sending to one destination, at exactly the same time?
What happens in the real world:
In real world financial trading networks, multicast feeds come in from the exchanges (such as NASDAQ, NYSE, BATS, etc.). These multicast feeds are processed for synchronization, accuracy and optimizations before being relayed to algorithmic trading engines. In today's trading networks, all available market data feeds added together sum to 3.6Gbps (obtained by adding all feeds available through Savvis). Thus, the massive many-to-many fan-in cited in Cisco's report is not seen in real world networks.
Why 96 and 128 byte packets?
The Nexus buffer is allocated in chunks of 140 bytes. Thus by picking packet sizes that fit nicely in the Nexus architecture, Cisco was able to achieve better performance.
What happens in the real world:
In real world financial networks, the packets are of mixed sizes. The NYSE market data feed for example contains packets from 64 bytes to 1273 bytes. The average packet size is 192 bytes. Thus the tests skipped a key component: using real world traffic. In addition, since various packet sizes appear in data feeds, a latency test with random packet sizes, or an IMIX pattern needs to be used. With such tests, Nexus 5000 switches have much higher jitter (3 to 4 µsec), while the Arista 7100 series switches can handle such patterns with less than 30 nanoseconds of jitter.
Congestion & Latency:
In all of these custom tests, the objective was to find the breaking point of the Arista 7100S switches. However, the most common cases of congestion, 2:1 fan in, was not tested. In this most relevant, real-world case, the Arista 7100 switch outperforms the Nexus 5000.
The test summary argues that low latency is not a good way to measure performance if you drop packets during congestion. Well, that depends on the application and how much congestion needs to be handled before dropping a packet. In some applications, low latency is critical - buffering traffic for 50 µsec or more does not help. The solution in these low-latency cases is to identify the sources of congestion and eliminate them by re-architecting the network.
Arista does understand the need for buffering in various applications. In cases where more buffering is required, sub-micro second latency is not a requirement - such as storage. Arista 7048 and Arista 7500 series switches offer very deep-buffer solutions for both 1GbE and 10GbE, buffering over 40ms of traffic on all ports simultaneously.
Summary:
All tests commissioned by Cisco have the same basic structure. Each sender sweeps across all recipients in perfect lock-step, simultaneously blasting each recipient with line-rate bursts of small packets, so as to create uniform congestion across all ports simultaneously, hiding the Nexus' inflexible buffer allocation policy while exaggerating the impact of a shared memory buffer.
These tests use exclusively packets that are just under 140 bytes in size. At this size, the Nexus 5010 uses 140 bytes of buffer per packet (wasting a tiny handful), whereas the 7100 uses 512 bytes of buffer per packet (wasting 384 bytes of buffer per packet).
The tests spread congestion uniformly across all ports of the switch. Real world scenarios have anything but uniformly spread congestion.
While any-to-any multicast exists in financial trading applications, with all market data feeds combined at 3.6Gbps today, these "micro-bursts" are no longer an issue in today's low-latency 10GbE networks.
Conclusion:
Custom test reports can make one product look better than the other. All customers are encouraged to do their own analysis before making a purchasing decision.
While the results of these tests are valid, the tests themselves do not resemble anything in the real world. Instead, these tests commissioned by Cisco are designed to find extreme corner cases to show the Nexus 5010 performing better than the Arista 7124S while hiding the deficiencies of the Nexus 5000 product line. An industry wide comparison was done by Network World where all vendors were invited to participate under a common test criterion. Arista 7100 switches outperformed all other switches in that testing, including Cisco's Nexus 5000.
