Windows Server 2016 Networking – Multiple DCB priorities

When configuring DCB on Windows Server 2016, you can create policies to tag specific traffic with a specific priority to let DCB handle the right QoS. For more info how this works, please see this blog series: https://www.darrylvanderpeijl.com/windows-server-2016-networking-rdma-dcb-pfc-ets-etc/

When you have multiple types of traffic that you want to separate into DCB priorities, e.g. Storage and Cluster traffic it could be that it not works as you would expect. It will look like DCB is not working right and I incorrectly blamed the NIC drivers..

The Problem

I have created two NetQosPolicy entries with Powershell using the following commands:

This would mean that every outgoing packet on port 445 would be tagged with CoS value ‘3’.
Windows also has ‘templates’ build in as shown above, so you can specify “-Cluster” and then all cluster traffic will be tagged, so every packet using port 3343 will be tagged with CoS value ‘5’.

Fortunately Mellanox is offering a great set of tools which also include counters for Performance Monitor.
One of those counters is the “QoS counter” which exactly shows how much traffic is flowing through a specific CoS Value.

As you can see, my server is receiving and sending packets with CoS Value ‘3’, but nothing happens on ‘5’.

The Solution

It seemed like this problem introduced itself with the release of the SET switch, but that is not the case. With 2012 R2 we did not create SMB/Storage NICs on the vSwitch as they wouldn’t support RDMA, but now with SET switches we don’t have that limitation anymore. The problem lays within traffic flowing through the vSwitch. By default all traffic that is originated from vNICs (Host & VM adapters) is stripped from the CoS value that is assigned to it. This makes sense, especially for service providers,  as you don’t want people configuring DCB in their VMs to mess up your datacenter networks and QoS. Imagine a customers web traffic getting the same prioritization as your Storage Spaces Direct traffic 🙂

I can hear you thinking “But then why is CoS value 3 working?”
This has to do with RDMA / SMB Direct traffic. This traffic bypasses the vSwitch so the CoS value is not stripped from the traffic.

The default behavior to strip the CoS value can be changed by changing the VM Network adapter setting called “IeeePriorityTag” (Technet) :

Set-VMNetworkAdapter -Name “SMB1” -ManagementOS -IeeePriorityTag On

On = Trusted, Packets with CoS values are allowed
Off (default) = Untrusted, Packets with CoS values are reset to CoS value ‘0’

After we changed the setting for the storage NICs we can now see that traffic is flowing with both CoS values, ‘3’ and ‘5’.

Whoohoo, success!

—

Thank you for reading my blog.
If you have any questions or feedback, leave a comment or drop me an email.

Darryl van der Peijl
http://www.twitter.com/DarrylvdPeijl