NVIDIA Mellanox Bluefield-2 SmartNIC Hands-On Tutorial: “Rig for Dive” — Part VI: DPDK Performance.

Table of Contents

How I get DPDK up and running on a Bluefield-2 SmartNIC, and what is its performance when measured with pktgen and testpmd.

In the previous part, we have installed the latest Ubuntu operating system on the Bluefield, supplied by the vendor itself. This release includes DOCA, DPDK, OvS, and all necessary tools by default. Hence, if you follow Part V in installing this system, you can probably skip the DPDK-related part of my blog post in Part II :)

Let’s see whether “probably” can be transformed to “surely”.

Enable SmartNIC Mode

First of all, we have to switch from the default Separated Host Mode to the SmartNIC mode. Follow the instructions in Part II.

After this, I thought I would not be able to ping a host from the other host as properly configured OvS would be required. However, the ping is working correctly.

Pinging from Host 1 to Host 2 using the SmartNIC in SmartNIC mode

After checking the OvS instances running on the Bluefields, I saw they are fired up in NORMAL operation, which means they work as an Ethernet Switch. This is good then…however, I did not see any ICMP packets on any interfaces in the Bluefields. According to the documentation we covered and referred to many times in this blog post series, in SmartNIC mode, packets should be going through the ARM cores.

Enable OvS-DPDK on the BlueField

Anyway, let’s enable OvS-DPDK on the SmartNIC as suggested on a Mellanox blog post.

Access the Bluefield and verify you have OvS installed and have DPDK support.

# ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.14.1
MLNX_DPDK 20.11.1.1.10

Okay, we have DPDK installed, and OvS configured and compiled with it.

Let’s see whether the running OvS has DPDK initialized.

# ovs-vsctl get Open_vSwitch . dpdk_initialized
false

In my case, we can see that it is not running with DPDK enabled.

Let’s overcome this.

Enable hugepages

First, follow the instructions again in Part II to enable hugepages. I make your task easier by repeating the steps below but without explanation. Please, read the post if you also want to know why we are doing what.

# mountpoint -q /dev/hugepages || mount -t hugetlbfs nodev /dev/hugepages
# echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

Ensure both commands took effect.

# mount |grep huge

2M hugetlbfs is mounted and ready to use.

# cat /proc/meminfo |grep -i hugepages_

Hugepage information from /proc/meminfo.

We are ready roll. The required filesystem is mount under */dev/hugepages *and we have all allocated 1024 hugepages free to use.

Enable RDMA

We discussed in Part II that enabling RDMA both on the Host and the Bluefield is required for DPDK. Accordingly, let’s do this on the host first.

# /opt/mellanox/iproute2/sbin/rdma link

Enabling RDMA on the host

We can see that, in contrast to Part II, since our Bluefield is connected, the status of the physical ports are ACTIVE/LINK_UP.

Let’s do the same on the Bluefield.

Enabling RDMA on the Bluefield

Again, we see a lot of entries, but the important parts are the ones with ACTIVE/LINK_UP states. If you have a similar output to this, you are can proceed further.

For the rest of the instructions, I am following the “official” guide. However, as I found most of the time, that guide was not enough on its own either. For instance, it does not deal with the above RDMA-related settings, but you cannot get OvS-DPDK running properly without them. You are lucky that I cover you with this blog post ;)

Identify SmartNIC PCI Device Identifiers

# mst start
# mst status -v

Okay, the IDs did not change since the first time I played around with Bluefield :)

Initialize DPDK for OvS

# ovs-vsctl --no-wait set Open_vSwitch . dpdk_initialized=true

Enable HW Offload

# ovs-vsctl --no-wait set Open_vSwitch . other_config:hw-offload=true

Create a DPDK bridge

# ovs-vsctl --no-wait add-br ovs_dpdk_br0 -- set bridge ovs_dpdk_br0 datapath_type=netdev

Add Port to bridge

First, we only add one interface to the newly created OvS-DPDK bridge

# ovs-vsctl --no-wait add-port ovs_dpdk_br0 dpdk0 -- set Interface dpdk0 type=dpdk -- set Interface dpdk0 options:dpdk-devargs=0000:03:00.0

Okay, let’s check the status

# ovs-vsctl show

OvS-DPDK is not running properly — Port could not be added

As you can see, the good-old vendor-supplied products do not work all the time. During the blog post series, I anyway did not encounter things to work out of the box, so I will not expect this anymore :(

[Side-track] What about testpmd

Whenever it comes to DPDK, I like to see whether any example application works at all. Since now we have an NVIDIA / Mellanox NIC, which uses the mlx5 driver being famous for not requiring it to be bound to the UIO driver, the simple dpdk-devbind.py application cannot help confirm this.

Thus, the next step is testpmd, the example DPDK application that does nothing just sends packets back to the ports configured.

First, let’s find where testpmd is located. It is located under /opt/mellanox/dpdk/bin.

Okay, then let’s try to run it.

# opt/mellanox/dpdk/bin/dpdk-testpmd -c7 -w 0000:03:000 -w 0000:03:00.1 -- -i -a
/opt/mellanox/dpdk/bin/dpdk-testpmd: error while loading shared libraries: librte_ethdev.so.21: cannot open shared object file: No such file or directory

And, of course, it does not work either…

Install DPDK from Scratch

I am so happy that I already did this before. For the complete guide, please read Part II, and install DPDK via the recommended way, i.e., via meson and ninja.

Then, I also installed pktgen as discussed in Part II

pktgen on the Bluefield

After compiling pktgen, let’s fire it up.

First, let’s see what ports it finds and whether there are any issues during startup.

Go to pktgen’s root directory and start the application.

# ./usr/local/bin/pktgen -c7 -n 4 --socket-mem “1024” -- -T -P

The command has nothing interesting but some EAL feature DPDK requires us to specify.

The first argument defines the core mask in HEX, which, when converted to 8-ary binary, means 0b00000111. This actually assigns the first 3 cores to the DPDK application, in this case, to pktgen.

The second argument defines the number of memory channels to use. You can omit this and use the default; I just set it to four.

The third argument defines how many of the available hugepages we want to use. Since I only allocated 1024 pieces of 2048kB hugepages, I can only assign 1024 to the app. Of course, I can live with less, but since no other DPDK app will run in this experiment, I just give all of them to pktgen.

After the EAL option, -T just enables a pretty colorized control panel, while -P puts the interfaces in promiscuous mode.

pktgen first run on the Bluefield

We don’t have to bother ourselves with the warnings as they are already handled. For instance, since Bluefield is not a NUMA architecture, we cannot even define NUMA, or the system cannot even get the NUMA id since we do not have such a descriptor.

Astute pkgten users should have already observed that I have not pinned the used cores to the used ports. Hence, the output of pktgen only shows one column, and it did even say before the colored output that besides the first core used for the application output itself, the other two cores, namely *lcore 1 *and lcore 2, are not used at all. Anyway, the purpose of this fresh run is to see how many ports pktgen will discover — and it found four.

Let’s see whether I can fire up all four ports to see which one is which.

# ./usr/local/bin/pktgen -c1f -n 4 --socket-mem “1024” -- -T -P -p f -m "[1].0,[2].1,[3].2,[4].3"

Mind the double-dashes in the command above.

pktgen on the Bluefield with all the supported ports brought up

If you have checked before what your interfaces MAC addresses are, they can be a good indicator to figure out which port is what.

The ports with *source MAC address 0c:42:a1:a4:8a:44/45 *are the physical port identifiers facing the outside world. In other words, they are port p0 and p1.

Since the Bluefield is connected to another Bluefield deployed in another server in my environment, I can check whether packets indeed leave the system in the way we would expect. So, the p0 ports on the two Bluefields are connected to each other, while the same applies to the p1 ports. Let’s verify this.

Packet sent from pktgen on the Bluefield is received by the other Bluefield it is connected to

Since my p0 interface on the Bluefield running the pktgen is the first port, i.e., port 0, I set it to send 3 packets only. On the other Bluefield (right-hand side of the figure above), I am running a tcpdump on the other p0 interface filtering on the IP range pktgen uses (across all its ports) to avoid seeing any unrelated status messages, e.g., STP protocol updates.

After starting port 0 in pktgen, it sends 3 packets that are observed on the other system. I confirmed this via *port 2 *in pktgten, i.e., via the p1 interfaces on both systems, and they worked the same. As expected, on the other hand, port 1 and port 3 with MAC addresses belonging to pf0hpf and pf1hpf interfaces (facing towards up to the host), respectively, are NOT connected to the other Bluefield. No packets are received on any of the interfaces on the other side when packets are sent on these ports.

However, when running tcpdump on the same host, we can see the packets sent from port 1 in pktgen on host interface ens5f0.

Packets sent on ports of the Bluefield facing up towards the Host are indeed received.

By this quick examination, I can confirm that Bluefield interfaces work as expected. To sum it up:

• PHY interface p0/p1 is the physical port on the Bluefield
• LOGICAL interface pf0/1hpf is facing up towards the Host
• MAC addresses of the physical ports differ in one digit only at the back, while logical ports’ MAC addresses are pretty random — this can also be used to infer from the MAC addresses which port is which.

Bluefield-2 SmartNIC DPDK Throughput

After getting DPDK, testpmd, and pktgen working, it is time to puzzle things together and see what they can achieve in terms of performance. Below, I show the environment I am working with. I also highlighted the port-related findings from above.

The environment I use for this experiment.

As one can see, we are working on the Bluefields exclusively in this experiment. The** Host BF2_1**, or more precisely, its Bluefield-2 SmartNIC (BF2_1), runs the dpdk-testpmd application, which forwards all packets coming through one of its ports to the other.

On the other hand, BF2_2 (at the other host, Host BF2_2) runs pktgen and generates packets from p0 while receiving them on p1.

Let’s see how we can achieve this setup.

Start testpmd on BF2_1

To start testpmd on BF2_1, we use the following command:

# ./build/app/dpdk-testpmd -c7 -n 4 --socket-mem 1024 -w 0000:03:00.0 -w 0000:03:00.1 -- -i -a

What you can observe here is that I explicitly whitelisted the ports, namely p0 and p1. Other settings are almost the default ones you have to set for interactive mode.

It is always a good idea to double-check the ports, though.

testpmd during start

As you can see at the bottom of the image, the ports used have the MAC addresses that only have the last digit to be different. From the above, we can conclude what it means — they are the physical ports, so we are OK with the configuration.

testpmd by default uses 1 core to do the forwarding

When we scroll a bit lower, we can see from the output of testpmd that it uses 1 core exclusively to forward packets from one port to another. This is good for now; let’s check the performance of one ARM core then.

Start pktgen on BF2_2

For pktgen, we follow a similar approach. Since it can realize more ports than we would need for this experiment, we also explicitly whitelist the ports, set the portmask to 3 (i.e., 0b0011 to use two ports only), and assign one CPU core to each port.

# ./usr/local/bin/pktgen -cf -n 4 --socket-mem 1024 -w 0000:03:00.0 -w 0000:03:00.1 -- -T -p 3 -P -m "[1].0,[2].1"

Then, start sending packets on port 0 without any limit.

Pktgen:/> start 0

Throughput of the Bluefield-2 with 1 core can be observed on port 1

We can see that with sending, one ARM core can be fully saturated with the default smallest possible packet size of 64B. I say saturated because usually, even on x86 machines, one CPU core can send 15Gbps traffic when packet size is 64B only.

Let’s increase the number of cores assigned for TX. Restart pktgen with the following command.

# ./usr/local/bin/pktgen -c 1f -n 4 --socket-mem 1024 -w 0000:03:00.0 -w 0000:03:00.1 -- -T -p 3 -P -m "[1:2-3].0,[4].1"

Observe, I have changed the coremask and define core 1 to handle RX, while core 2–3 to handle TX on port 0. Port 1 is configured as it was before.

At least doubled transfer throughput when using 2 cores

We can see the sending rate boosted up to 34 Gbps, more than double the rate than it was before. Increasing the number of assigned cores with even one already hits the 40Gbps limit we have in our environment. Refer back to Part IV., where we discuss why we are probably having only 40GbE “ports” instead of 100GbE.

Assigning even 4 cores in total for TX; rate is bounded by 40G

On the receiving side, however, we only see around 8.5 Gbps throughput. This can be caused by the testpmd application itself on the other server. Remember it runs everything with one core only.

Let’s increase first the number of cores for the testpmd application on BF2_1. For some reason, however, testpmd does not allow to assign more than 3 cores to the whole application (at least in my configuration). Furthermore, assigning more RX queues to the ports also does not make too much sense for now, as they would only be utilized if RSS (receive side scaling was working). But the latter only works if incoming packets have different headers, otherwise, they will be hashed to the same queue.

Generate PCAP

Hence, I will create a pcap file for this purpose. I will use my own pcap_generator available on Github.

# git clone https://github.com/cslev/pcap_generator

It uses a simple text file describing the header fields, which is fed to the Python script.

The text file should look like this:

src_mac=00:00:00:00:00:01,dst_mac=00:00:00:00:00:02, src_ip=10.0.0.1, dst_ip=10.0.0.2, dst_port=22

First, let’s generate a similar file having 100 random entries. It will have 100 entries with a random destination IP, source port, and destination port. Other fields are left static. The lines below are in different lines for easier consumption but they can be in one BASH line only.

# for i in {1..100}; 
do 
  ip=$(printf "%d.%d.%d.%d\n" "$((RANDOM % 256))" "$((RANDOM % 256))" "$((RANDOM % 256))" "$((RANDOM % 256))"); 
  src_port=$(printf "%d\n" "$((RANDOM % 65535))");
  dst_port=$(printf "%d\n" "$((RANDOM % 65535))"); 
  echo "src_mac=00:00:00:00:00:01,dst_mac=00:00:00:00:00:02, src_ip=10.0.0.1, dst_ip=${ip}, src_port=${src_port}, dst_port=${dst_port}" >> rss_capable.txt;
done

Now, let’s make the pcap file:

# ./pcap_generator_from_csv.py -i rss_capable.txt -o rss_capable

The resulted file will have 64-byte packets stored in rss_capable.64bytes.pcap.

You can check whether your pcap file has been indeed created as intended.

# tcpdump -nr rss_capable.64bytes.pcap
reading from file rss_capable.64bytes.pcap, link-type EN10MB (Ethernet)
01:32:21.6884408 IP 10.0.0.1.7876 > 146.38.156.123.22979: UDP, length 18
01:32:21.6955194 IP 10.0.0.1.22576 > 59.181.196.80.19067: UDP, length 18
01:32:21.6977010 IP 10.0.0.1.8005 > 51.74.42.117.17745: UDP, length 18
01:32:21.6997299 IP 10.0.0.1.8491 > 31.197.10.94.578: UDP, length 18
01:32:21.7017112 IP 10.0.0.1.13091 > 223.21.36.26.21426: UDP, length 18
01:32:21.7036495 IP 10.0.0.1.31004 > 74.186.216.88.18412: UDP, length 18
01:32:21.7055593 IP 10.0.0.1.27656 > 51.81.30.38.13146: UDP, length 18
...

Let’s use this pcap file for pktgen, and see what the throughput of testpmd will become.

# ./usr/local/bin/pktgen -c ff -n 4 --socket-mem 1024 -w 0000:03:00.0 -w 0000:03:00.1 -- -T -p 3 -P -m "[1:2-4].0,[5-6].1" -s 0:../pcap_generator/rss_capable.64bytes.pcap

Throughput with RSS-capable pcap file. Testpmd has more cores to process input, pktgen on the other side similarly assigns 2 cores for receiving.

We can see that the throughput/packet per second rate is similar on both the sending and the receiving sides. Although, the sending rate is a bit higher.

The other observation is that even if three cores are still assigned for pktgen to send the traffic, now as a pcap is sent instead of the same standard packet every time, the sending rate is highly affected.

The most I could do is add one more core for sending; otherwise, I will not have enough cores, i.e., two, for receiving.

# /usr/local/bin/pktgen -c ff -n 4 --socket-mem 1024 -w 0000:03:00.0 -w 0000:03:00.1 -- -T -p 3 -P -m "[1:2-5].0,[6-7].1" -s 0:../pcap_generator/rss_capable.64bytes.pcap

Sending rate increased when one more core is assigned. Receiving rate remained the same

We can see that the sending rate is indeed increased up to 21 Gbps, while the receiving rate remained the same.

Let me update the post later when I can run pktgen on BF2_1, and then I can assign more cores for the receiving side on BF2_2.

UPDATE@22/04/2021:

I have managed to run pktgen on BF2_1 and use that Bluefield to send packets, while BF2_2 only has to deal with receiving the packets.

**Sender statistics: **No matter how many cores I set for transfer, I cannot reach 30Gbps sending rate with the pcap file created above, i.e., 64B packets with different Layer 3/4 header fields.

Receiver statistics: Increasing the number of cores assigned for RX, increases the attainable receive rate as well. Setting the number of cores to 1, 2, 3, 4, and 5 results in around 8.5, 16, 22, 26, and max of 28 Gbps throughput.

Below, you can find a snippet of the last setting, i.e., when the traffic is received by utilizing 5 cores.

By using 5 cores at the receiving side, we can achieve more or less the same receivingrate as the sending rate itself.

Conclusion

I showed how we can install DPDK from scratch when the vendor-supplied package is not working as expected.

After putting things together, I have run a throughput measurement with pktgen and a testpmd connected back to back on the two Bluefields. We observed that pktgen on the Bluefield-2 SmartNIC can generate 40Gbps traffic (64B packets) with three cores; just as the same cores that we would need on an x86 system. When it comes to pcap file-based transfer, the *three cores *are only enough to send around 21 Gbps traffic (64B packets).

Finally, when three cores (the maximum allowed in my system) are assigned to testpmd, I measured around 15.3 Gbps receive rate at the packet generator side, for which I had the remaining two cores to assign.

P.S.: to answer the question I asked in the beginning whether we can turn “probably” to “surely”, I can only say it is rather a “might”, i.e., you might need to install DPDK from scratch :)

UPDATE@22/04/2021:

The maximum receive rate by utilizing 5 cores reaches around 28 Gbps, which is the maximum sending rate by using a pcap file.