NVIDIA Mellanox Bluefield-2 SmartNIC Hands-On Tutorial: “Rig for Dive” — Part IV: Baseline Throughput Tests with iperf3

Table of Contents

In this episode, we scrutinize the performance of the Bluefield-2 SmartNIC in our two-node cluster deployed at Cloudlab. We had quite a bumpy road in reaching this point, but we finally did it. If you missed the former parts, read Part I to know how to install the necessary drivers; read Part II to see how DPDK can be installed, and read Part III to see how the environment we further work in can be achieved in Cloudlab.

As a quick recap, the topology we use is shown below. Two servers are connected via three links. Two of them practically wires the two ports of the Bluefield SmartNICs, while the third uses the other smartNIC (with a non-capitalized ‘s’) port. The latter is only to force Cloudlab to indeed connect the Bluefields.

The topology we are going to use onward.

Why 40G only if Bluefield has dual ports of QSFP56?

First, after pinging one SmartNIC from the other, I also wanted to check via traceroute “how far” the servers are from each other, but it returned 1 hop. This is, of course, not surprising. Due to the Software Defined Networking architecture used by Cloudlab, the networking devices do not necessarily behave as expected when seeing ICMP messages.

Speaking of SDN switches, this can also explain why the Bluefield shows 40Gbps throughput at most. Since all QSFP ports (QSFP+, QSFP28, QSFP56, QSFP56-DD) have the same QSFP form factor, they are backward compatible.

While QSFP+ has 4 lanes of 10G, QSFP56 (installed on the Bluefield) supports 4 lanes of 50Gbps; physically capable of 200 Gbps for each port. However, NVIDIA claims that the Bluefield (the one installed at Cloudlab) can only support 100G. This is probably the limitations imposed by the SoC architecture, not the physical ports soldered on the board. Put differently, this claim should mean that whatever you want to offload to the SmartNIC, as long as you write your code efficiently and your application relies only on the hardware accelerators, you can achieve 100G. But, the hardware is incapable of pushing more packets per second through the NIC.

Even though the QSFP56 form factor, Bluefield-2 only supports 100G (https://store.mellanox.com/products/nvidia-mbf2h516a-ceeot-bluefield-2-smartnic-p-series-dpu-100gbe-dual-port-qsfp56-pcie-4-x16-crypto-enabled-16gb-on-board-ddr-1gbe-oob-management-tall-bracket-fhhl.html)

Anyway, QSFP56 can then support QSFP+’s 4x10G lanes instead of its official 4x50G…this can be the reason for having only 40G interfaces at the machines.

Throughput of the Bluefield SmartNICs measured from the Host

Before being able to measure the throughput of/from the Host, you have to be sure that the Bluefield is in Separated Host Mode. We discussed this in Part III, so I only copy-paste the final command below. If it does not work for you, refer to Part III, to obtain the correct device descriptors.

# mlxconfig -d /dev/mst/mt41686_pciconf0 q | grep -i internal_cpu_model
         INTERNAL_CPU_MODEL                  SEPERATED_HOST(0)

If you see SEPARATED_HOST(0), you are fine.

Assign IP Addresses to the Ports

On the first host machine

# ifconfig ens5f0 10.10.10.1/24 up

On the second host machine

# ifconfig ens5f0 10.10.10.2/24 up

Ensure connection via ping from Host 1 to Host 2:

# ping 10.10.10.2
PING 10.10.10.2 (10.10.10.2) 56(84) bytes of data.
64 bytes from 10.10.10.2: icmp_seq=1 ttl=64 time=0.326 ms

Okay, we got the connection up and running. Let’s check throughput via iperf3 on the hosts. In case you don’t have iperf3 installed, install it with

# apt-get update
# apt-get install iperf3

And, now run iperf3 measurement from one host towards the other; the latter running iperf3 -s, i.e., iperf3 in server mode.

iperf3 performance on the hosts

Not too bad, but not amazing either. This average of 26.5 Gbps will serve as a baseline later.

Evaluate Bluefield

Let’s start to repeat this simple iperf3 measurement on the Bluefield.

Login to the Bluefield

In order to do so, just use what we learnt from Part I, and access Bluefield via rshim.

# ssh ubuntu@192.168.100.2
Password: ubuntu

If you had any problem accessing the Bluefield, or just want to have the most latest BlueOS installed before digging deeper, check Part V.

Throughput of the Bluefield in Separated Host Mode

Again, ensure that the SmartNIC is still in **Separated Host Mode. **Install iperf3 just as we did before on the hosts.

Stop OVS on the Bluefield

Now, let us ensure that no OVS instance is running on the host, accidentally taking over the network control on the SmartNIC ports.

# ps aux | grep ovs

In my case, OVS is running on the Host

If you don’t have any OVS instance running, jump to the next section.

Remove ovsbrX Instances

Even though you stop the OVS process (see below), networking interfaces can remain, in particular, the OVS bridges. In our case, we have two OVS bridges running by default, ovsbr1, and ovsbr2. While OVS processes can be stopped, I encountered that these remainder “interfaces” can still cause problems later. So, let us ensure to remove them first. To do so, we cannot stop the whole OVS subsystem; at least we need ovsdb-server to run.

Let’s remove the bridges:

# ovs-vsctl del-br ovsbr1
# ovs-vsctl del-br ovsbr2

Now, we are ready to stop OVS processes.

The easiest way to stop it is to use ovs-ctl, instead of killing each process one-by-one. However, ovs-ctl is sometimes not part of your PATH environment variable and/or simply installed under a different path than /usr/bin, /usr/local/bin, etc. Mine is located under /snap/ directory.

# /snap/lxd/18152/share/openvswitch/scripts/ovs-ctl stop

Assign IP Addresses to the Ports

I am going to configure the interface p0 on each Bluefield SmartNIC. Let’s use an IP address range for these ports different from those we used for the hosts.

On the first Bluefield SmartNIC:

# ifconfig p0 10.1.1.1/24 up

On the second Bluefield SmartNIC:

# ifconfig p0 10.1.1.2/24 up

Similarly to the host-based measurement, ensure the connection is working from the first Bluefield.

# ping 10.1.1.2
PING 10.1.1.2 (10.1.1.2) 56(84) bytes of data.
64 bytes from 10.1.1.2: icmp_seq=1 ttl=64 time=0.461 ms
64 bytes from 10.1.1.2: icmp_seq=2 ttl=64 time=0.125 ms
64 bytes from 10.1.1.2: icmp_seq=3 ttl=64 time=0.174 ms

Good, we are connected. Let’s repeat the iperf3 measurement accordingly, but now we will see the performance of the Bluefield itself.

iperf3 performance on the Bluefield-2 DPUs

We can observe that the performance on the less performant ARM system is only around 7.5 Gbps (compared to the 26.5 Gbps we achieved from the Host). Actually, the performance results are literally the single-threaded, single-core performance of the systems as iperf3 by default is single-threaded. We see the difference between the power-efficient ARM core and a beefy server core — the Cloudlab facility at Clemson that has the Bluefield-2 DPUs installed has AMD EPYC 7542 32-core processors.

Multi-threaded iperf3

Simply adding -P flag to iperf3 only initiates more parallel streams, however, they are still using the same CPU core. Therefore, we have to rely on another approach suggested by the iperf3 developers, i.e., ESnet itself.

In particular, we have to start multiple iperf3 instances both on the server-side and the client-side. Start multiple servers, say three, on the second Bluefield with different ports:

# iperf3 -s -p 5101 &
# iperf3 -s -p 5102 &
# iperf3 -s -p 5103 &
# iperf3 -s -p 5104

Run multiple clients accordingly:

# iperf3 -c 10.1.1.2 -T s1 -p 5101 &
# iperf3 -c 10.1.1.2 -T s2 -p 5102 &
# iperf3 -c 10.1.1.2 -T s3 -p 5103 &
# iperf3 -c 10.1.1.2 -T s4 -p 5104 &

The -T parameter just adds a prefix to the processes outputs.

Let’s make a for loop to reach the same end:

# num_threads=4
# for i in `seq 1 $num_threads` ; do iperf3 -c 10.1.1.2 -T s${i} -p 510${i} & done

Performance results of four iperf3 threads on the Bluefield

We can see that each thread has around** 4.5Gbps** throughput, summing up to about 18 Gbps in total.

Let’s try with six threads. After initializing two more iperf3 servers and running the for loop above with $num_threads=6, we got the following results.

Performance results of six iperf3 threads on the Bluefield

We can observe that the per-thread performance is much more fluctuating than before in the interval of around [1.5–5] Gbps. These average numbers sum up in 17.85 Gbps, close to the one we measured in the previous four-threaded scenario.

Let’s utilize all 8 cores the Bluefield-2 DPU has:

Performance results of eight iperf3 threads on the Bluefield

The measured fluctuation is still visible, but why would it be different, right? Anyway, the sum of the throughput we could reach is around 23 Gbps.

From these numbers, we can conclude that, by default, the Bluefield-2 DPU’s TCP performance utilizing all CPU cores can go up to 23 Gbps.

Performance tuning

I tried to play around with the ARM cores’ CPU frequency and install cpufrequtils to set governor mode to performance. However, the cores already seem to be scaled up…at least, cat /proc/cpuinfo shows constant 400.0 BogoMIPS for all cores. Having MIPS values instead of MHz might also explain why setting the governor to performance by *cpufreq-set *resulted in a **Segmentation fault **:) Thus, I did not go further on this topic.

You might go further with ESnet’s Test/Measurement Host Tuning blog post; however, after a quick look, there seems to be no tuning option for extra gain on the ARM-based Bluefield-2 DPUs.

Multi-threaded Measurements on the Host

Just briefly…after figuring out how to carry out the proper multi-threaded scenario on the Bluefield, I gave it a try on the Host as well. I was playing around with six to eight threads but have not been able to go much above 32–33 Gbps.

Accordingly, I would conclude that with iperf3, we can reach around 33 Gbps on the Host by default by using the Bluefield-2 DPUs ports.

Note, in this Cloudlab environment, 40 Gbps is the theoretical maximum anyway.

Summary

Using the Bluefield-2 DPU SmartNIC as a regular NIC, we can achieve around 33 Gbps throughput by multi-threaded iperf3 sessions running on the host. On the other hand, with a similar experiment is purely running on the SmartNIC, i.e., on top of the ARM cores, we could achieve around 23 Gbps throughput by utilizing all eight cores.

In the next episode, namely Part V, I show how to (re-)install the operating system on the Bluefield. Furthermore, I also discuss in what circumstances it can become necessary.