To flow control or to not flow control, that is the question. The answer is, it depends, but what does it depend on?
- First, what is flow control at a high level? Technical report TR-4182 does a great job explaining the mechanism.
Page 30, section 6.1 Ethernet Flow Control
“Ethernet flow control is a layer 2 network mechanism that is used to manage the rate of data transmission between two endpoints. It provides a mechanism for one network node to control the transmission speed of another so that the receiving node is not overwhelmed with data.”
2. Next, what are the ONTAP Flow Control defaults?
The default in ONTAP is disabled for Cluster Interconnect ports and enabled on data ports. There have been many different flow control recommendations over the years. The knowledge base article 1002403 provides the best practice to match the setting end-to-end for data ports.
3. What are the flow control best practices for 10g Ethernet?
- Disable flow control on cluster network ports in the Data ONTAP cluster. This is the default, so do not change this.
- Flowcontrol on the remaining network ports (the ports that provide data, management, and intercluster connectivity) should be configured to match the settings within the rest of your environment.
4. ONTAP output explained
When flow control is disabled on a physical port, any interface group (ifgrp) or VLAN will show enabled regardless of the underlying port setting. The flow control setting is handled by the underlying physical ports. Note that a0a, a0a-11 and a0a-12 below show “full” however they are not enabled based on the underlying e0a and e0b ports set to “none”.
5. Follow up and next steps
You will need to talk to the network and server teams to inquire what the settings are on the switch ports and the hosts, then match on the storage ports. Below are two real customer examples, where the opposite setting increased performance. The key metric is matching the environment.
Example 1: Enabling Flow Control
I had one customer with Cisco CNAs that did not disable flow control on the host. Throughput was throttled to 1Gb on 10GbE ports. By enabling flow control on both the switches and NetApp, we were able to saturate the 10GbE ports.
Example 2: Disabling Flow Control
At another customer with large sequential IO, we increased throughput 20% by disabling flow control end-to-end.