As shown in the previous part, Data-center Bridging (DCB) has these CoS values (0-7) and based on that the network stack will prioritize traffic and when it becomes too much even pause a specific flow. I also told you that every node in the network in an active DCB node and needs to have DCB configured to make it work. This also means that next to configuring the switches, we will need to configure the windows servers as well.
Configuring the windows servers includes installing the DCB feature and specifying which traffics belongs to which CoS value.
Part 1: RDMA, DCB, PFC, ETS, etc
Part 2: Configuring DCB in Windows
Part 3: Optimizing Network settings
Part 4: Test DCB and RDMA (coming)
Enabling traffic tagging with a CoS value
Traffic tagging can be configured by using the New-NetQosPolicy Powershell cmdlet.
The cmdlets uses different types of filters to identify traffic, for example:
This new policy filters outgoing Live Migration traffic and adds a CoS value of 1 to it.
Enabling Lossless networks
To let Windows know which networks you want to transmit lossless you will have to enable it on a specified CoS value.
In this case the Powershell Enable-NetQosFlowControl cmdlet accepts a “Priority” parameter which is equal to a CoS value.
By default, no CoS value is set lossless.
Enabling ETS policies
The New-NetQosTrafficClass cmdlet creates a new traffic class, next to the default one where all traffic lands in.
This way we can make sure that a specific CoS value always has a minimum of x percentage of outgoing bandwidth.
In this example we specified that traffic with CoS value 1 always should have 20% outgoing bandwidth when it needs it.
If the Live Migration traffic is 0%, other traffic classes can use the bandwidth when needed.
Configuration Example
In the following configuration example I’m using RDMA capable 10GbE NICs so I can benefit from SMB direct, which I highly recommend for Storage Spaces Direct. This is why I want to match traffic on port “445” (SMB) and tag it with a CoS value so I can enable lossless transmission and ETS on that value.
Cluster traffic is also the kind of traffic I would like to give a dedicated traffic flow and tag with a CoS value, although lossless transmission is not necessary, giving it minimum outgoing bandwidth with ETS is something I prefer. This is specific per scenario and not a hard requirement.
Get-NetQosPolicy | Remove-NetQosPolicy -Confirm:$False
Disable-NetQosFlowControl -Priority 0,1,2,3,4,5,6,7
Get-NetQosTrafficClass | Remove-NetQosTrafficClass -Confirm:$False
#Disable DCBx
Set-NetQosDcbxSetting –Willing $false -Confirm:$False
# Set tagging policies
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "Cluster"-IPDstPort 3343 –PriorityValue8021Action 1
# Turn on Flow Control (Lossless) for SMB
Enable-NetQosFlowControl -Priority 3
# Make sure flow control is off for other traffic
Disable-NetQosFlowControl -Priority 0,1,2,4,5,6,7
# Enable QoS on Adapters
Get-NetAdapterQos | Enable-NetAdapterQos
# Configure ETS with Bandwidth minimums, default class will take 35%
New-NetQosTrafficClass "SMB" -Priority 3 -BandwidthPercentage 60 -Algorithm ETS
New-NetQosTrafficClass "Cluster" -Priority 1 -BandwidthPercentage 5 -Algorithm ETS
Note that the “BandwidthPercentage” is relative to the amount of bandwidth you have. In this example I have 2x10GbE which makes 5% bandwidth a total of 1GbE for Cluster traffic which is more than enough.
Windows QoS
ETS is something handled in the hardware layer and also referred to as “Hardware QoS”, but we also have the ability to do QoS on the software layer, in Windows Server itself. You can use either one to configure QoS, your choice to go for Hardware or software QoS.
In Windows Server 2016 Software QoS exists in two forms:
- SDN-QoS
- VM-QoS
SDN-QoS
SDN-QoS settings can only be set in the Network Controller (or VMM talking to the Network Controller). If you don’t have the Network Controller installed, you can’t and aren’t using SDN-QoS. SDN-QoS is ETS aware (but only applies to the default traffic class). It can do egress bandwidth limits and egress bandwidth reservations. Default is that it’s off and the switch just tries to fairly send egress traffic on behalf of all VMs that are using the default traffic class.
VM-QoS
VM-QoS settings are set through PowerShell. This is the same VM-QoS settings that existed in WS2012 and WS2012 R2 using the Set-VMNetworkAdapter cmdlet. They work (sometimes) but have known issues. VM-QoS is not ETS aware. It will try to reserve bandwidth above what the switch can use.
It’s possible to set vmQoS in configurations where SDN QoS should be used. It’s even possible to set them both at the same time. Either one (using them in the wrong place or using both together) is a recipe for disaster.
Continue to Part 3: Optimizing Network settings >>
—
Thank you for reading my blog.
If you have any questions or feedback, leave a comment or drop me an email.
Darryl van der Peijl
http://www.twitter.com/DarrylvdPeijl
Cant wait for Part 4.