NVIDIA Mellanox Bluefield-2 SmartNIC Hands-On Tutorial: "Rig for Dive" — Part I: Install Drivers and Access the SmartNIC

Table of Contents

I have gotten my hands dirty with NVIDIA Bluefield-2 SmartNIC deployed at Cloudlab’s facility @ Clemson. If you ever wondered to buy a Bluefield SmartNIC, now I can show you how to test them and get your first impressions for free.

NVIDIA Bluefield-2 DPU

What is Mellanox Bluefield-2

For some time ago, the networking industry is going through a big revolution. There have been many headlines lately about the end of Moore’s law and how the continuous close-to-exponential improvement of general-purpose processors is degrading. However, the data and the corresponding network traffic are doing the opposite. To efficiently keep up with this increasing data processing need, common network interface cards, a.k.a. NICs, have become programmable. Programmable means that we can offload some parts of the data processing to the NIC (which are thus termed as SmartNIC), thereby alleviating the host CPU to do the actual processing, i.e., data interpretation, visualization, content generation, etc.

There are different types of SmartNICs, and various branches of research or IT stakeholders look at them differently. Even the definition of “programmable” is (unfortunately) not carved into stone. Like in programmable network devices, “programmable” can mean remotely configurable, extended features, or explicitly that the device itself can be programmed like a general-purpose architecture.

This approach even applies to NICs, more precisely, SmartNICs, too. For instance, many traditional NICs (found in your laptop, server, etc.) are already “smart” at some point. Some part of the packet processing is offloaded to the NIC, i.e., the NIC does more than just packet buffer allocations and sending packets back and forth from the physical ports to the CPU. Your NIC already does checksum calculations and TCP segmentation offloading in hardware, just to mention the most common ones. Below, you can see all the features my laptop’s Wi-Fi interface supports, i.e., can do in hardware. According to the ON/OFF settings, you can observe what are the actual tasks my laptop (by default) offloads to the NIC.

This is a snippet of the offloading features set for my Wi-Fi networking interface

Then came the smartNICs with non-capitalized ‘s’ providing even more complex tasks to be offloaded, but still not “really programmable”.

Without the need for completeness, consider Mellanox ConnectX-4/5 that, by utilizing Mellanox ASAP2 and Linux’s TC, can offload a part of the Open vSwitch packet processing logic to hardware.

ASAP2 OVS Offload Solution (source:https://blog.mellanox.com/2018/05/mellanox-red-hat-nfv-infrastructure/)

Lastly, we reached SmartNICs (with capitalized ‘S’) that are indeed programmable, meaning that we do not only utilize some features for packet processing, but we can actually define the packet processing logic itself!

SmartNICs are usually categorized into three main branches: ASIC-based, FPGA-based, and SoC-based. And, in this order, the performance decreases but programmability and its easiness increase. There are pretty good and short descriptions about these branches on Mellanox’s blog.

The three branches of SmartNICs (source:https://blog.mellanox.com/2018/08/defining-smartnic/)

Okay, let’s cut the brief history lesson short here; otherwise we never get to the point.

So, Mellanox Bluefield-2 is the latest flagship SmartNIC of Mellanox. It is actually termed as DPU (Data Processing Unit) (instead of simply saying the heavily overused word SmartNIC) to make Mellanox, more precisely, NVIDIA (since it had acquired Mellanox quite some time ago) stand out from the crowd.

Bluefield-2 actually belongs to the SoC branch, meaning it is less performant but much more programmable. It is “so programmable” that it can actually be thought of as a Server in your Server. It has an 8-core ARM64 CPU, 8–16GB DRAM, up to 2x100G Ethernet interface + 1Gb OOB (out-of-band) management port and/or UART (universal asynchronous receiver-transmitter) to do the same. It can also be connected to an IPMI network to monitor it just like a server.

Accordingly, Bluefield-2 DPU runs its own Linux distribution on top and has a Mellanox ConnectX-6 Ethernet module. It also supports DPDK and even P4 via the DOCA SDK specifically tailored to program the Bluefield via “native C” libraries and APIs.

In terms of (publicly available/claimed) performance (metrics) Bluefield-2DPU enabled offloading software-defined storage, zero-trust security, networking and management.

“By offloading these functions from the CPU, NVIDIA says a single BlueField-2 can perform the same services that can require as many as 125 CPU cores. Just think about what those freed-up CPU cores could accomplish.” — Forbes

What is Cloudlab? Can I use it?

For those, who don’t know what Cloudlab is, it is a

“Flexible, scientific infrastructure for research on the future of cloud computing.”

Cloudlab’s site

In a nutshell: It is free to use and consists of many sites mostly distributed in the US only. But, Cloudlab also somewhat federates Onelab, another similar facility situated in the EU. This means you can manage and interconnect sites from Cloudlab to sites in Onelab.

Cloudlab and Bluefield-2 DPU

Recently Cloudlab, more specifically, its cluster maintained at the University of Clemson, has upgraded its system and installed Dual-port Mellanox BlueField2 100Gb SmartNIC into their nodes termed as r7525.

Therefore, we can now play around with this technology for free. This blog post is all about my first experience with it.

Create experiment profile

Here, I assume that the reader is familiar with the Cloudlab dashboard and can define topology describing profiles.

First, we have to create an experiment with its topology defined to be deployed at the Clemson cluster and consists of r7525 nodes only. Let’s now only have a single Server topology running the latest Ubuntu 20.04.

Create proper profile for it. Pay attention to hardware type and disk image

Once this ready, and you are lucky (i.e., there is free resource), you can instantiate your profile and your server will be up and running within a minute (usually).

Use your Cloudlab credentials to access the host machine via the ssh connection details cloudlab dashboard shows you.

Configure Bluefield and Access It

TL;DR

In case you already read this blog post and just want to quickly reproduce everything, jump to the end of the post.

Install drivers

Even though the servers have the SmartNICs installed, it does not necessarily mean that all the essential drivers are installed. Therefore, start by installing the necessary Mellanox drivers.

For a more comprehensive quick start guide with Bluefield, you can refer to this. Below, I show the already digested long story and summarize the main steps (without missing any important one).

Download drivers

# cd /opt
# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-5.3-1.0.0.1/MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64.tgz

Uncompress and Install

# tar -xzvf MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64.tgz
# cd MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64
# ./mlnxofedinstall --auto-add-kernel-support --without-fw-update

You will probably be asked for confirmation as the install script will remove all libraries you might have from previous installation. Additionally, it will install new packages. Just hit ‘y’ when the question below appears.

Accept that the script will remove all previous installations and will install new packages, too.

This is a good time to grab a coffee or just stand up, walk around, stretch your fingers and squeeze a stress ball — this process can take a few minutes.

Everything should be OK without any error, and the script will print out all Mellanox ethernet interfaces that are affected after the installation:

The script tells us what ethernet interfaces it found and will apply the newly installed drivers. We can see that besides the smartNIC (non-capitalized ‘s’), i.e., ConnectX-5, our SmartNIC (a.k.a. Bluefield) is also found.

Restart MLNX_OFED’s openibd:

Let’s do what the script tells us to do.

# /etc/init.d/openibd restart
Unloading HCA drvier:                [ OK ]
Loading HCA driver and Access Layer: [ OK ]

Install rshim separately ?

rshim is actually a shim layer we can use from the host system to communicate with the SmartNIC through Ethernet. In other words, the rshim layer creates dummy interfaces with local IP addresses on both the host and the Bluefield, then we can easily SSH into the NIC from the host (see later).

By default, the above install script will also install the rshim drivers. You can confirm this by checking whether you have an rshim0 device in /dev/. If there is one, you are fine.

In case you do not have, install it separately from the DEBS directory of the drivers we downloaded above.

# cd MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64/DEBS
# dpkg --force-all -i rshim-<version>.deb

Enable rshim and start it

Let’s enable rshim after the IP assignments.

# systemctl enable rshim
# systemctl start rshim

Check status

# systemctl status rshim
● rshim.service - rshim driver for BlueField SoC
 Loaded: loaded (/lib/systemd/system/rshim.service; enabled; vendor preset: enabled)
 Active: active (running) since Wed 2021-04-07 20:56:36 EDT; 19min ago
 Docs: man:rshim(8)
 Main PID: 438156 (rshim)
 Tasks: 6 (limit: 618619)
 Memory: 1.8M
 CGroup: /system.slice/rshim.service
 └─438156 /usr/sbin/rshim
Apr 07 20:56:36 node-0.cslev-qv95721.k8s-dataplane-pg0.clemson.cloudlab.us systemd[1]: Starting rshim driver for BlueField SoC...
Apr 07 20:56:36 node-0.cslev-qv95721.k8s-dataplane-pg0.clemson.cloudlab.us systemd[1]: Started rshim driver for BlueField SoC.
Apr 07 20:56:36 node-0.cslev-qv95721.k8s-dataplane-pg0.clemson.cloudlab.us rshim[438156]: Probing pcie-0000:81:00.2
Apr 07 20:56:36 node-0.cslev-qv95721.k8s-dataplane-pg0.clemson.cloudlab.us rshim[438156]: create rshim pcie-0000:81:00.2
Apr 07 20:56:37 node-0.cslev-qv95721.k8s-dataplane-pg0.clemson.cloudlab.us rshim[438156]: rshim0 attached

Get Hardware string

Although it might not be the most important thing, getting to know what exactly you are working with can be beneficial in the future when you plan to buy your own DPU. And, of course, you want the same you have been working with before to reduce the number of things that can go wrong. Unfortunately, lspci and lshw do tell you the exact product ID.

This is what lspci can tell…not too much about the actual product, just the ethernet interface and the chipset.

While lshw is more talky, as expected, it rather talks about where the Bluefield is installed and what its system-wide properties are.

After installing rshim, however, we can obtain the burnt hardware string from the Bluefield. First, set the rshim’s miscellaneous display level to 1:

# echo "DISPLAY_LEVEL 1" > /dev/rshim0/misc

Then, we can get the necessary information by reading this file via:

# cat /dev/rshim0/misc

Getting further hardware information through rshim’s misc

Look for OPN_STR, in our case it is:

OPN_STR MBF2H516A-CENO

Mind the missing ‘T’ at the end? According to the guide, this Bluefield should be MBF2H516A-CENOT , which means that Crypto is NOT enabled. We verify (?) this below after accessing the Bluefield.

Access Bluefield via rshim

First, we have to set up an IP address to the rshim host interface *tmfifo_net0 *that is already visible after install rshim itself.

The rshim interface is visible to the host system: tmfifo_net0

On Ubuntu, we already use netplan (instead of the /etc/network/interfaces.conf approach available in Debian)

Assign IP

On my Cloudlab host, the default netplan configuration yaml file is missing, so we can easily create one from scratch. We only have to pay attention to not screw up the base networking interface we are using to access the system itself.

# nano /etc/netplan/01-netcfg.yaml

Add the following lines:

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: yes
    tmfifo_net0:
      addresses: [192.168.100.1/24]
      dhcp4: no

Pay attention to the eno1 part. This is the main interface, so double-check what is yours by looking at the output of ifconfig.

Also, this is a indentation-sensitive yaml file, so pay attention to make it indented as shown — maybe copy-paste will carry this forward inappropriately. The indentation size does not matter (1/2 whitespace or Tabs), just make it consistent.

Restart networking

Option 1 — restarting the system’s networking daemon:

# systemctl restart systemd-networkd

You may have a different network manager (if you did not instantiate the host with Ubuntu 20.04 as I did), like NetworkManager, etc. Get to know your related networking services by this command:

# systemctl list-unit-files | grep -i network
emulab-networkd@.service static enabled 
networkd-dispatcher.service enabled enabled 
systemd-network-generator.service disabled enabled 
systemd-networkd-wait-online.service enabled enabled 
systemd-networkd.service disabled enabled 
systemd-networkd.socket disabled enabled 
network-online.target static enabled 
network-pre.target static disabled 
network.target static disabled

In our case, it was pretty obvious which one to pick.

Option 2 — applying the new netplan configuration directly

In some cases, the above restart does not work as the new netplan configuration has to be explicitly applied first. If you see that after the above restart commands, your tmfifo_net0 interface has still no IP, issue the following command, too:

# netplan apply

If there is an error, due to typos or whatever, use the command in debug mode to find out more details:

# netplan --debug apply

Test

If everything is fine, then we can try pinging Bluefield-2

# ping -I tmfifo_net0 192.168.100.2 -c2

Ping works; We can now access the Bluefield

Connect

# ssh ubuntu@192.168.100.2
Password: ubuntu

Done, you are in the SmartNIC.

Crypto enabled or disabled?

This is the question that I *cannot *answer at the moment. After logging into the Bluefield, we can see what “PCI-attached” hardwares does it see via the lspci command.

lspci output on the Bluefield

We only see the Ethernet interfaces and the PCI bridge. The latter, in contrast to our expectations after reading the hardware string above, the crypto is enabled. Maybe, this is an SoC feature that is enabled/disabled by default, however, by purchasing licence or whatever extra, NVIDIA can get this enabled for you if you had a Crypto disabled variant.

Install drivers on the Bluefield ?

The short answer is NO. Since the operating system installed on the Bluefield comes from the vendor, the necessary Mellanox drivers and tools are installed under /opt/mellanox directory. You could observe that on the Host machine, after installing the MLNX_OFED drivers, the same directory appeared. In contrast, on the Bluefield the directory is already there. Hence, there is nothing to do (for now) regarding the drivers.

Access the Internet from the Bluefield

According to my experience, every time I am logged into a new non-x86 system, and I got to know the basic things, my first action is to ping google.com.

Luckily, through *rshim *— as we were connecting to the Bluefield via SSH — we can provide Internet access to the Bluefield via NAT. It’s pretty straightforward to do so, and we only need three things to carry out.

Enable IP forwarding on the host

# echo 1 | tee /proc/sys/net/ipv4/ip_forward

Enable NAT on the Internet-facing interface on the host

# iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE

As you can see, in my Cloudlab experiment, the Internet-facing interface is termed eno1. Double-check yours.

In case your Internet access (via pinging 8.8.8.8) does not work, ensure IP forwarding is set properly.

# sysctl -a | grep forwarding

And, look for your device on which you enabled forwarding.

net.ipv4.conf.eno1.forwarding = 1

If this is indeed enabled, but still no Internet access, then it might have happened that your default firewall rules are configured to an implicit deny policy. Let’s change that:

# iptables -A FORWARD -o eno1 -j ACCEPT
# iptables -A FORWARD -m state --state ESTABLISHED,RELATED -i eno1 -j ACCEPT

Now, you should definitely have access to the Internet. However, the DNS settings on the Bluefield itself is still messed up.

Set Google’s DNS on the Bluefield

# echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf

Now, full internet access is enabled:

# ping google.com

Bluefield is now connected to the Internet

In Part II., we are going to install the latest DPDK on the Bluefield.

“Rig for Dive and Take Her Down”, i.e., we are diving deeper :)

Summary

Below, you find all the commands and settings in a concise way:

On the Host

# cd /opt
# wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-5.3-1.0.0.1/MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64.tgz
# tar -xzvf MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64.tgz
# cd MLNX_OFED_LINUX-5.3-1.0.0.1-ubuntu20.04-x86_64
# ./mlnxofedinstall --auto-add-kernel-support --without-fw-update
# /etc/init.d/openibd restart
# systemctl enable rshim
# systemctl start rshim
# systemctl status rshim
# echo "DISPLAY_LEVEL 1" > /dev/rshim0/misc
# cat /dev/rshim0/misc
# nano /etc/netplan/01-netcfg.yaml
--- YAML FILE ---
network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: yes
    tmfifo_net0:
      addresses: [192.168.100.1/24]
      dhcp4: no
=== YAML FILE END ===
# systemctl restart systemd-networkd
# netplan apply
# ifconfig tmfifo_net0
# echo 1 | tee /proc/sys/net/ipv4/ip_forward
# iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE
^^^^^^^^^^^^^^^ IN CASE IP FORWARDING IS NOT WORKING ^^^^^^^^^^^^^
# iptables -A FORWARD -o eno1 -j ACCEPT
# iptables -A FORWARD -m state --state ESTABLISHED,RELATED -i eno1 -j ACCEPT
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

On the Bluefield

# ssh ubuntu@192.168.100.2
Password: ubuntu
# echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
# ping google.com