Windows Server 2016 Networking – Multiple DCB priorities

When configuring DCB on Windows Server 2016, you can create policies to tag specific traffic with a specific priority to let DCB handle the right QoS. For more info how this works, please see this blog series: http://www.darrylvanderpeijl.com/windows-server-2016-networking-rdma-dcb-pfc-ets-etc/

When you have multiple types of traffic that you want to separate into DCB priorities, e.g. Storage and Cluster traffic it could be that it not works as you would expect. It will look like DCB is not working right and I incorrectly blamed the NIC drivers..

The Problem

I have created two NetQosPolicy entries with Powershell using the following commands:

# Set a policy for SMB-Direct
New-NetQosPolicy "SMB" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 3
New-NetQosPolicy "Cluster" -Cluster -PriorityValue8021Action 5

This would mean that every outgoing packet on port 445 would be tagged with CoS value ‘3’.
Windows also has ‘templates’ build in as shown above, so you can specify “-Cluster” and then all cluster traffic will be tagged, so every packet using port 3343 will be tagged with CoS value ‘5’.

Fortunately Mellanox is offering a great set of tools which also include counters for Performance Monitor.
One of those counters is the “QoS counter” which exactly shows how much traffic is flowing through a specific CoS Value.

 

As you can see, my server is receiving and sending packets with CoS Value ‘3’, but nothing happens on ‘5’.

The Solution

It seemed like this problem introduced itself with the release of the SET switch, but that is not the case. With 2012 R2 we did not create SMB/Storage NICs on the vSwitch as they wouldn’t support RDMA, but now with SET switches we don’t have that limitation anymore. The problem lays within traffic flowing through the vSwitch. By default all traffic that is originated from vNICs (Host & VM adapters) is stripped from the CoS value that is assigned to it. This makes sense, especially for service providers,  as you don’t want people configuring DCB in their VMs to mess up your datacenter networks and QoS. Imagine a customers web traffic getting the same prioritization as your Storage Spaces Direct traffic 🙂

I can hear you thinking “But then why is CoS value 3 working?”
This has to do with RDMA / SMB Direct traffic. This traffic bypasses the vSwitch so the CoS value is not stripped from the traffic.

The default behavior to strip the CoS value can be changed by changing the VM Network adapter setting called “IeeePriorityTag” (Technet) :

Set-VMNetworkAdapter -Name “SMB1” -ManagementOS -IeeePriorityTag On

On = Trusted, Packets with CoS values are allowed
Off (default) = Untrusted, Packets with CoS values are reset to CoS value ‘0’

After we changed the setting for the storage NICs we can now see that traffic is flowing with both CoS values, ‘3’ and ‘5’.

Whoohoo, success!

Thank you for reading my blog.
If you have any questions or feedback, leave a comment or drop me an email.

Darryl van der Peijl
http://www.twitter.com/DarrylvdPeijl

 

 

 

2 thoughts to “Windows Server 2016 Networking – Multiple DCB priorities”

  1. Hi,
    Thank you for the article. If the priority flow is enabled for SMB traffic and apply ETS as shown below
    Server Side config:
    New-NetQosTrafficClass “SMB” -Priority 3 -BandwidthPercentage 50 -Algorithm ETS
    Switch Config:
    On mellanox switch we enable priory flow traffic class ‘3’ and we do not mention bandwidth allocation and I believe we instead configure ECN something like this on mellanox switch

    ### ECN config traffic class-3
    #interface ethernet 1/1-1/32 traffic-class 3 congestion-control ecn minimum-absolute 150 maximum-absolute 1500

    If we have other traffic such as ‘Cluster’, Backup traffic and we create traffic class with some priority and ETS, Should we also configure the these on network side as well i.e on mellanox switch?
    If we only configure ETS for traffic based on for example IP-Subnet with no priority set, would that be acceptable? the idea is just assign priority to SMB and treat all other traffic as default, but allocate bandwidth for certain type of traffic such as backup and cluster on server side and not configuring anything on switch side as it is a outbound traffic..
    what is best way of allocating bandwidth for the following traffic types given 2X40Gig NICs
    Cluster
    Management
    Live Migration – (priority 3)
    SMB – (Priority-3)
    Backup
    VM-Network
    I appreciate your help.
    regards
    john

    1. Hi John,

      If I understand correctly, you could configure policies on the server and use this as outgoing QoS.
      On the switch/network side, they are equal.

Leave a Reply

Your email address will not be published. Required fields are marked *