NVIDIA Mellanox Bluefield-2 SmartNIC Hands-On Tutorial: “Rig for Dive” — Part VII/A: To Offload or Not To Offload?
Table of Contents
Is it beneficial to offload OvS datapath to the hardware? Does it matter if the kernel or DPDK datapath is offloaded? In this episode, I dig a bit deeper into the OvS offloading matters, and I also explain how the packet processing is done with OvS running on the SmartNIC.
In the previous episodes, I have already been dealing with OvS and DPDK on the Bluefield-2 DPU SmartNIC; however, the OFFLOADing part was not working completely. Then, I had to manually reinstall different parts of the whole ecosystem, starting from the BlueField OS through DPDK and OvS.
In this episode, finally, I show how exactly you can achieve to have all these ‘supposed-to-be-there-by-default’ applications be indeed there by default.
Install Everything Properly
After I have been granted to early access the DOCA SDK, I could download the so-called SDK manager. It is an all-in-one installation suite with a graphical user interface, and it supposes to install all packages you would ever need for the Bluefield.
SDK manager
Once you have access to the SDK manager, you will also have access to the documentation. Hence, I don’t want to repeat that here. Just do the Next-Next-Finish chain, and you are mostly done.
Attention Cloudlab Users
Cloudlab machines may not have enough free space on their root partition. For development, the SDKmanager also downloads a docker container, which is around 10 GB. If we also count the drivers and Bluefield OS images being downloaded (among other more minor things), we would definitely need approximately 16 GB of free space.
However, on a Cloudlab machine, your root partition is just about 16 GB — it is insufficient. Yet, we can overcome the issue with two simple steps.
Add More Storage to Cloudlab
Once your system is running, you can attach an extra 1+ TB temporary space to your machine by the following command:
# /usr/local/etc/emulab/mkextrafs.pl /mydataHere, I created a /mydata directory in the root directory, and this will be used for all NVIDIA / Mellanox Bluefield-2 DPU related sources. Also, for brevity, set all permission to this directory to let the SDKmanager download everything there.
# chmod -R 777 /mydataReconfigure Docker
We have to reconfigure docker in order to store its files under /mydata instead of */var/lib/docker/. *Therefore, on a freshly booted system, where docker is not even installed (like on a Cloudlab machine), it is better to install docker first before running the SDKmanager. Note, SDKmanager would anyway install docker for you if you don’t have it, but it is already too late. So, install docker first by the official guide.
Then, stop docker daemon
# /etc/init.d/docker stopCreate a file daemon.json under */etc/docker, *which has the following lines
{
"data-root": "/mydata/docker"
}Save it, then copy every docker related files you might already have to this new location.
# rsync -aP /var/lib/docker/ /mydata/dockerRestart docker daemon.
# /etc/init.d/docker restartAnd, you are done! Docker will download and manage all its sources under /mydata/docker, which has plenty of storage.
Offload Packet Processing to the BlueField-2 SmartNIC
NOTE: every network testing or performance measurement application runs on the host machines. On the other hand, every OvS-related command, e.g., enable/disable offload, adding flow rules, monitoring flow tables and datapath cache, is issued on the Bluefield. You can also refer to the BASH PROMPTS on screenshots.
Once the Bluefield SmartNIC and all its drivers are correctly installed, it is already running in SmartNIC mode. If you are unfamiliar with the mode of operation and don’t know how to change it, please refer to Part II.
In Part III, I have already investigated the performance with iperf3 tests. However, I have just realized how the packet processing has been done in that setup. Moreover, it turned out, I was already using an offloaded open vswitch (OvS) for packet processing. How you can prove this is when you run an iperf3 session between the two Hosts; if you try to do tcpdump on any of the Bluefields to see which interfaces the packets are going through, you will not see any of them. However, the Bluefields are in SmartNIC mode, and they “somehow” cross the OvS switch running on the Bluefields by default. Let’s have a closer look.
Assume you have configured one of the SmartNIC Ethernet ports on the Hosts that are directly connected. Let the Host1 (H1) IP address be 10.0.0.1, while the same for Host2 (H2) be 10.0.0.2. I also assume here that you can ping from H1 to H2 (and vice versa).
Ping is working from Host 1 to Host 2 (and vice versa)
If you go to the Bluefield, you can check whether an OvS instance is running at all. Let’s do so. First, we focus on one Bluefield only on H1; let it be BF2_1.
On the Bluefield, issue the following command.
# ovs-vsctl show
OvS is running on the Bluefield
We see, there are two bridges (ovsbr1 and ovsbr2). While the former connects the first physical port p0 to the logical representor of the host *pf0hpf, *the latter does the same for the second port p1 and pf1hpf.
Let’s dump the flow table of ovsbr1.
# ovs-ofctl dump-flows ovsbr1
cookie=0x0, duration=5717.678s, table=0, n_packets=7671, n_bytes=617129, priority=0 actions=NORMALWe can see it is working in NORMAL mode, which is the default L2 switch behavior. Even though this single flow rule matched already on 7671 packets, we don’t know what those packets were or whether this switch takes part in forwarding our ping packets from the host.
First, let’s stop ping if it is not stopped already and give the system around 10 seconds of idle time to flush all its caches for sure. Then, remove the flow rule from ovsbr1.
# ovs-ofctl del-flows ovsbr1Now, start ping again from H1. We can observe that it is not working anymore. This proves that our packets were processed by OvS, but why did I not see any packets with tcpdump?
To investigate this, I will reinstall flow rules to ovsbr1, but now I make them more explicit to easily keep track of them later in other parts of OvS.
Let’s enable ARP FLOOD on the switch and also add the two (back-and-forth) IP forwarding rule to the flow table explicitly.
# ovs-ofctl -O OpenFlow12 add-flow ovsbr1 arp,actions=FLOOD
# ovs-ofctl -O OpenFlow12 add-flow ovsbr1 ip,in_port=pf0hpf,ip_dst=10.0.0.2,ip_src=10.0.0.1,actions=output:p0
# ovs-ofctl -O OpenFlow12 add-flow ovsbr1 ip,in_port=p0,ip_dst=10.0.0.1,ip_src=10.0.0.2,actions=output:pf0hpfBear in mind the IP addresses and the ports. This OvS runs “beneath” H1, so the host-facing logical interface is pf0hpf, and the packets coming from H1 have an IP of 10.0.0.1. Accordingly, all such packets should be sent out on port p0 towards H2. Conversely, I also set a similar rule for the reverse direction.
Try pinging again.
Ping works again between the host after manually installing explicit flow rules into ovsbr1
You can see, it is working again. Then, OvS is indeed responsible for the packet processing.
Offload or Not Offload? — Offload with TC
There is a user-space application for OvS to monitor the flow cache, in particular, the MegaFlow Cache of OvS. Recently, there was a study about how this flow cache can be populated in a covert way with only a few number of packets that causes a Denial-of-Service attack. Here, I do not want to dig deeper in the caching architecture of OvS; let it be enough that flow rules matched in the flow table are cached for future reference, i.e., to make packet processing faster for those flows.
Dump the Flow Cache
Let’s check the flow cache.
# ovs-dpctl dump-flows
Flow cache of OvS on the Bluefield
We see that there is nothing related to the ping flows.
It led me to one conclusion, the flows should be already offloaded to the hardware. However, since it is not a DPDK-enabled OvS yet, it should be using the good-old TC flower-based offloading.
Dump the Hardware Flow Cache
There is another user-space tool for OvS that allows us to dump any flow cache entry that exists in the system.
Let’s use that command and also use grep to quickly find the exciting entries if they exist.
# ovs-appctl dpctl/dump-flows -m |grep 10.0.0.
The flow cache entries are offloaded to the hardware
We can indeed see the corresponding flow rules along with all the conntrack (connection tracking) information (ct_X). Yet, the most essential part of the entries can be found at the end of the lines. We can see the offloaded flag is set to yes, while the dp (corresponding to the datapath) flag is set to TC flowers.
Performance of the TC Flowers
When we do an iperf3 session between the hosts.
We can observe that the performance is around 22.3 Gbps.
iperf3 performance snippet when using TC flower-based hardware offloading on the Bluefield
This concludes that the Bluefield in SmartNIC mode uses TC flowers to offload the OvS instance running by default.
Offload or Not Offload? — Not Offload
Next, we try to disable hardware offloading altogether to see how the OvS will perform. First, stop the iperf3 session and let the system flush the caches.
On the Bluefield, set the OvS-database configuration for hardware offloading to False and restart OvS.
# ovs-vsctl --no-wait set Open_vSwitch . other_config:hw-offload=false
# /etc/init.d/openvswitch-switch restartSimilarly, add the flow rules again to ovsbr1.
# ovs-ofctl -O OpenFlow12 add-flow ovsbr1 arp,actions=FLOOD
# ovs-ofctl -O OpenFlow12 add-flow ovsbr1 ip,in_port=pf0hpf,ip_dst=10.0.0.2,ip_src=10.0.0.1,actions=output:p0
# ovs-ofctl -O OpenFlow12 add-flow ovsbr1 ip,in_port=p0,ip_dst=10.0.0.1,ip_src=10.0.0.2,actions=output:pf0hpfNow, if we start pinging, it still works; however, the corresponding flow cache entries are not offloaded to the hardware. We can use both flow cache monitoring applications to make sure of this.
# ovs-appctl dpovs-appctl dpctl/dump-flows -m
Flow cache is now NOT offloaded
We can see in the highlighted row that the flow cache entry corresponding to the pings have no offloaded flag now, and the dp flag is set to ovs.
Let’s see with the other flow cache monitor that sees the software-based flow cache only.
# ovs-dpctl dump-flows
output of ovs-dpctl dump-flows when flow caching is not offloaded to the hardware
We also see here the corresponding cache entries in the non-offloaded kernel datapath.
Performance of the Pure Software OvS
The performance of OvS with the kernel datapath is much worse than when its datapath is offloaded to the hardware (via TC flowers).
iperf3 performance of OvS when the kernel datapath is not offloaded to the hardware
Overall, we can observe that from the average of ~22.3 Gbps, the performance of the pure kernel datapath without hardware offloading is around 88% worse.
Offload or Not Offload? — Offload with DPDK
And now, we reached the section that is of utmost interest for most of the readers who ended up here. To use OvS with DPDK, there are further OvS-database configurations we have to change before starting OvS itself.
Issue all commands below on the Bluefield.
Initialize OvS with DPDK and Offloading
First, let’s set the hw-offload flag back to True
# ovs-vsctl --no-wait set Open_vSwitch . other_config:hw-offload=trueInitialize DPDK (for more details, refer to Part II). This first library export is needed for any DPDK application running on the system to find the corresponding DPDK libraries.
# export LD_LIBRARY_PATH=/opt/mellanox/dpdk/lib/aarch64-linux-gnu/Enable hugepages and mount them if you did not so already
# echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepagesSet hugepage configuration accordingly for OvS
# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"Set DPDK initialization flag to True
# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=trueFinally, restart OvS
# /etc/init.d/openvswitch-switch restartCreate OvS-DPDK bridge
Now, to make sure only the DPDK OvS bridge will process packets, remove ovsbr1 and ovsbr2 created by default.
# ovs-vsctl del-br ovsbr1
# ovs-vsctl del-br ovsbr2Add a new bridge to the OvS system that is DPDK-enabled
# ovs-vsctl --no-wait add-br ovs_dpdk_br0 -- set bridge ovs_dpdk_br0 datapath_type=netdevWe can further restrict OvS to only be able to use the ports that we let it operate. Accordingly, let’s assign only port0 (our former p0 port from above), particularly port0’s PCI ID to OvS.
# ovs-vsctl set Open_vSwitch . other_config:dpdk-extra="-a 0000:03:00.0,representor=[0,65535]"Add the two ports to the bridge — one for the physical port (dpdk0) and one for the logical port using the representor.
The latter is actually the VF-PF (virtual function — physical function mapper). There is a pretty good paragraph here (20.1.1) about the VFs and PFs; basically, you can define many VFs that can be assigned to any application/VM/container/whatnot, and all are connected to the one and only physical function (PF) that represents the physical port itself.
Add the PF first
# ovs-vsctl --no-wait add-port ovs_dpdk_br0 dpdk0 -- set Interface dpdk0 type=dpdk -- set Interface dpdk0 options:dpdk-devargs=0000:03:00.0Then, add the VFs (we add all VFs here without any restritction as all packets we want to receive on the host)
# ovs-vsctl --no-wait add-port ovs_dpdk_br0 dpdk1 -- set Interface dpdk1 type=dpdk -- set Interface dpdk1 options:dpdk-devargs=0000:03:00.0,representor=[0,65535]Finally, restart OvS again to enforce all our settings
# /etc/init.d/openvswitch-switch restartNow, let’s see how OvS looks like
# ovs-vsctl show
DPDK-based OvS bridge having dpdk0 for the PF while dpdk1 for the VFs
It looks okay, doesn’t it? There is no error, and no other OvS bridge is running; only the DPDK-based one is present with the two ports we have just assigned.
Let’s delete the default flow table and add the flow rules again we had for the use case used above.
# ovs-ofctl del-flows ovs_dpdk_br0
# ovs-ofctl -O OpenFlow12 add-flow ovs_dpdk_br0 arp,actions=FLOOD
# ovs-ofctl -O OpenFlow12 add-flow ovs_dpdk_br0 ip,in_port=dpdk0,ip_dst=10.0.0.1,ip_src=10.0.0.2,actions=output:dpdk1
# ovs-ofctl -O OpenFlow12 add-flow ovs_dpdk_br0 ip,in_port=dpdk1,ip_dst=10.0.0.2,ip_src=10.0.0.1,actions=output:dpdk0Do the same on the other host?
If we really want to benefit from the DPDK-based hardware offloading, and on the receiving side you also run a Bluefield-2 DPU “beneath” the host and it is running in SmartNIC mode, you have to repeat the same step on the other Host (H2), more precisely, on the other Bluefield (BF2_2).
Why? Because, we saw above that by default, the the OvS datapath is offloaded via the TC flowers by default. Hence, it would introduce a bottleneck in our measurements, hindering to achieve the highest performance possible.
As I said, the steps are the same on the other host, except the flow rules as the IPs should be reversed. Or, since it is in NORMAL mode by default, and we no longer need explicit flow rules to identify the presence of offloading (we confirmed that in this blog post), you might leave the other OvS bridge’s flow table on BF2_2 as it is.
DISCLAIMER: In my current setup, on the receiving side, the Bluefield is in SEPRATED_HOST mode and, therefore, the host (H2) receiveds the packets without crossing the ARM cores on the Bluefield.
Let’s see the offloaded cache entries on BF2_1
# ovs-appctl dpctl/dump-flows -m
The relevant part of the flow cache corresponding to the flow rules and ping communication.
Now, we can see that the offloaded flag for the IP-based forwarding rule is set to yes, and dp is set to dpdk in this case. On the other hand, the ARP-based rule is only partially offloaded. We do not cover this because it is related to the architecture and the flow rule representation themselves. The ARP rules are probably not offloaded by default as they are anyway short-lived entries as they are required at the beginning of a flow establishment. And, most importantly, inserting, updating, and/or removing entries from the cache is a costly action — as pointed out in this study and in this one.
Performance of the DPDK offload
Somewhat unexpectedly, I did not encounter any non-negligible performance improvement when running the same iperf3 session.
iperf3 performance when OvS DPDK datapath is offloaded to the hardware — not much improvement compared to the TC flower-based one
While the average became 27.2 Gbps, we can see that most of the time the throughput is around **~26.3 Gbps **(from the 22.3 Gbps of the TC flower-based offloading). However, at the end of the iperf3 measurement, the achieved throughput is almost identical to the TC flower-based offloading results)
I did not define explicitly the number of cores to use, though, but using multiple iperf3 parallel sessions does not help either…actually, the cumulative throughput is less.
iperf3 performance does not benefit from the offloaded OvS DPDK even with multiple threads
I have played around with OvS-DPDK settings, like pinning the handler and revalidator thread to core 7, while assigning core 0–3 to the datapath.
# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x80
# ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=0x0fHowever, the whole datapath core assignment would only make more sense if we were optimizing for receiving traffic. Put differently, I could assign 4 cores to the datapath, then I could define 4 receive queues; each handled by one core exclusively. For transmitting the packets, i.e., for TX queue, there is no such option.
There are some interesting observations, we should make.
First, the methods used to offload the datapaths to the hardware are different. In case of the kernel datapath, the “OVS data plane rules are converted to TC filters”. On the other hand, in case of the DPDK datapath, “OVS data plane rules are downloaded to the NIC via rte_flow API” — according to an OvS Conference 2019 presentation by Mellanox.
But, eventually, when the datapath is offloaded to the hardware, the same “hardware block” handles the flows, or more precisely, the rest of the packets of the flows. Hence, I would assume that the performance after offloading is pretty much the same irrespective to the offloading mechanism.
Second, we can see from the iperf3 measurements that, in case of the TC flower-based offloading, the first reported throughput is less than the consecutive ones.
Yes, I am aware of the fact that iperf3 is not the most robust tool for performance measurement and it, by default, reports in every second only (though, this can be set — but I stick with the defaults).
According to the expected performance improvement of the hardware offloading, once the flows are offloaded to the hardware, the performance jumps up from ~19.8 Gbps to ~22.3 Gbps. For brevity, let’s assume that for a flow to be completely offloaded to the hardware, OvS requires 1 second.
However, in case of OvS-DPDK, we see the opposite in terms of performance changes. It starts with a high throughput of ~31.1 Gbps, then, when the assumed 1 second of offloading time is elapsed, the throughput **drops down to 27 Gbps **(which, at the end, dropped down to 22.3 Gbps).
Question? Does it mean that the user-space DPDK datapath is actually performing better (i.e., achieves higher throughput) when it is running on the ARM cores compared to the case, when the datapath itself is offloaded to the hardware?
Conclusion
OvS running on the Bluefield by default in SmartNIC mode already offloads its kernel datapath to the hardware. Although, it is using the TC flower-based approach.
In this episode, we investigated whether the offloading can be disabled completely, or whether we can enhance the performance by offloading the DPDK datapath instead.
We used simple iperf3-based measurements to investigate the performance.
We observed that without hardware offloading, the throughput of OvS is around 2.73 Gbps.
When the default TC flower-based hardware offloading is enabled, the performance boost is close to 10x, i.e., the throughput is 22.3 Gbps.
When the DPDK datapath is offloaded, the overall performance does not boost that much; even after fine-tuning some DPDK-related settings, the throughput is only ~26.3 Gbps.
Stay tuned! In order to get rid of the biases imposed by iperf3, in the next post, I reexamine the packet processing performance with the DPDK-based pktgen.