Part VIII — NVIDIA Mellanox Bluefield-2 SmartNIC Hands-On Tutorial: Install that frickin DPDK from source
Table of Contents
In previous tutorials, I’ve covered installing DPDK from its source. However, the true challenge comes when you attempt to use it practically, such as compiling a DPDK app. You will only realize the need for additional system-wide optimizations, especially for Bluefield-2 compatibility.
“Installing DPDK on the Bluefield-2” revisited
In this brief episode, we explore the nuances of compiling DPDK effectively on the Bluefield-2 DPU and the essential system-wide configurations required to harness its full potential. As we’ll discuss, many of the previously documented DPDK installation methods may no longer suffice.
It is especially evidenced in Part II, where even the compilation fails due to some missing rxp-compiler.h header file.
Note that all commands below are supposed to be executed on the Bluefield.
What causes issues and interference?
When it comes to compiling DPDK on the Bluefield, we already have a sense that the platform should be somewhat different. But is it just simply aarch64, arm64, or what?
It took some time for me to find the correct way as no explicit tutorial was given. I already shared some thoughts in this regard on the Suricata forum, which eventually caused me to install DPDK from source.
There is a trick with setting the platform from native to the actual one. I found the correct one from config/arm/meson.build where there is something called soc_bluefield. This is the key, so let’s use this during compilation. Furthermore, to also play around with the example applications let’s also compile all of them at once. It will take some time on the Bluefield but at least we do not have to do it anymore.
Below, I use the latest LTS version, 22.11.4 for DPDK.
# wget https://fast.dpdk.org/rel/dpdk-22.11.4.tar.xz
# tar -xJf dpdk-22.11.3.tar.xz
# ln -s dpdk-stable-22.11.3/ dpdk
# cd dpdk/
# apt install meson cmake
# meson setup -Dplatform=bluefield -Dexamples=all build
# ninja -C build
# ninja -C build install
# ldconfigIf meson succeeds, but you find that some libraries are missing that you might need, just install them and run meson setup --wipe build , then it will redo meson with your previously set configuration, i.e., -Dplatform, -Dexamples.
In case you want to uninstall and try another version, just issue this command:
# ninja -C build uninstallPay attention to the output as it might not delete all directories. In that case, check those and delete them manually (if there wouldn’t be any interference with other things.
Following this step, assuming all other prerequisites and dependencies are satisfied (which I won’t delve into here), you should observe successful compilation without encountering any errors. The final command, which installs the libraries into their system-wide recognizable locations, confirms the actual path (without requiring additional specifications) as:
/usr/local/include
However, even if you issue the ldconfig command as a last step, when you invoke pkg-config to see if everything is alright, you see this (or something similar w.r.t. your BFB image installed on the Bluefield — mine is 2.6.0):
# pkg-config --modversion libdpdk
22.11.2401.1.0Clearly, this version is not the one we installed, but let’s examine the compilation flags and include paths we would use if we were to compile a DPDK application in the future.
# pkg-config --cflags libdpdk
-include rte_config.h -mcpu=cortex-a72
-I/opt/mellanox/dpdk/include/dpdk
-I/opt/mellanox/dpdk/include/dpdk/../aarch64-linux-gnu/dpdk
-I/opt/mellanox/dpdk/include/dpdk
-I/usr/include/libnl3
# pkg-config --libs libdpdk
-L/opt/mellanox/dpdk/lib/aarch64-linux-gnu -Wl,--as-needed -lrte_node
-lrte_graph -lrte_pipeline -lrte_table -lrte_pdump -lrte_port -lrte_fib
-lrte_ipsec -lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder
-lrte_rib -lrte_dmadev -lrte_regexdev -lrte_rawdev -lrte_power -lrte_pcapng
-lrte_member -lrte_lpm -lrte_latencystats -lrte_jobstats -lrte_gso -lrte_gro
-lrte_gpudev -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev
-lrte_compressdev -lrte_cfgfile -lrte_bpf -lrte_bitratestats -lrte_bbdev
-lrte_acl -lrte_timer -lrte_metrics -lrte_cmdline -lrte_pci -lrte_ethdev
-lrte_ip_frag -lrte_hash -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool
-lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargsWe can confirm that neither the cflags nor the libs that would be included are not pointing to our DPDK installation, but rather are set as default, i.e., the system would still use the default DPDK installation coming from Nvidia.
One might assume that removing the mlnx-dpdk(-dev) package would resolve the issue. However, due to dependencies, removing it could also result in the removal of many doca libs. This could potentially cause further issues down the line, so it’s preferable to avoid that approach.
If you navigate to /etc/ld.so.conf.d, you will find a respective file to the Mellanox/NVidia DPDK installation, which is there to help ldconfig command where to look for the .pc pkg-config files. You will see a file there mlnx-dpdk-aarch64-linux-gnu.conf with the following content:
# cat /etc/ld.so.conf.d/mlnx-dpdk-aarch64-linux-gnu.conf
/opt/mellanox/dpdk/lib/aarch64-linux-gnuAccordingly, let’s create a file for our new DPDK installation:
# echo "/usr/local/lib/aarch64-linux-gnu" > /etc/ld.so.conf.d/dpdk-aarch64-linux-gnu.conf After this, let’s remove the old one to make sure it won’t get picked up. In fact, just rename it and adding .bak suffix should do the job — then, we will still have it later if needed.
# mv /etc/ld.so.conf.d/mlnx-dpdk-aarch64-linux-gnu.conf /etc/ld.so.conf.d/mlnx-dpdk-aarch64-linux-gnu.conf.bakAs an additional measure, modify the $PKG_CONFIG_PATH environment variable. I only modify it for the root user, because I anyway do everything as root on the Bluefield.
The easiest and most permanent solution is to edit /root/.bashrc and append the following line to the end of it:
# export PKG_CONFIG_PATH=${PKG_CONFIG_PATH}:/usr/local/lib/aarch64-linux-gnu/pkgconfigNote this only changes how the root user “sees” the system. The default ubuntu user will still (probably) use the mlnx-dpdk package. Hence, if you plan to configure/compile apps with DPDK links (e.g., Suricata) as a user, you need to do the above export for the ubuntu user as well. What I observed is that the PKG_CONFIG_PATH for the user to have the /opt/mellanox/dpdk/... already, so it is better to reset PKG_CONFIG_PATH without just appending the correct DPDK path. To do so, add this line to /home/ubuntu/.bashrc
# export PKG_CONFIG_PATH=/opt/mellanox/collectx/lib/aarch64-linux-gnu/pkgconfig:/opt/mellanox/doca/lib/aarch64-linux-gnu/pkgconfig:/opt/mellanox/flexio/lib/pkgconfig:/opt/mellanox/grpc/lib/pkgconfig:/opt/mellanox/doca/lib/aarch64-linux-gnu/pkgconfig:/usr/local/lib/aarch64-linux-gnu/pkgconfigAfter all these things are done, it’s better to reboot the Bluefield. Issue a reboot command on it, or use rshim on the host to do the same via
Host# echo "SW_RESET 1" > /dev/rshim0/miscTesting
When the Bluefield came back up, let’s check all pkg-config commands above:
# pkg-config --modversion libdpdk
22.11.4
# pkg-config --cflags libdpdk
-include rte_config.h -mcpu=cortex-a72 -I/usr/local/include
-I/usr/include/libnl3
# pkg-config --libs libdpdk
-L/usr/local/lib/aarch64-linux-gnu -Wl,--as-needed -lrte_node -lrte_graph
-lrte_pipeline -lrte_table -lrte_pdump -lrte_port -lrte_fib -lrte_ipsec
-lrte_vhost -lrte_stack -lrte_security -lrte_sched -lrte_reorder -lrte_rib
-lrte_dmadev -lrte_regexdev -lrte_rawdev -lrte_power -lrte_pcapng -lrte_member
-lrte_lpm -lrte_latencystats -lrte_jobstats -lrte_ip_frag -lrte_gso -lrte_gro
-lrte_gpudev -lrte_eventdev -lrte_efd -lrte_distributor -lrte_cryptodev
-lrte_compressdev -lrte_cfgfile -lrte_bpf -lrte_bitratestats -lrte_bbdev
-lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline -lrte_pci
-lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool -lrte_rcu
-lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lbsd
Photo by Andre Hunter on Unsplash
Hooray! At last, we’ve successfully compiled, installed, and linked the DPDK version we wanted. Now, let’s attempt to re-compile an example app.
Try the helloworld app. Let’s go to the source:
# cd /home/ubuntu/dpdk-stable-22.11.4/examples/helloworld
# ls
Makefile build main.c meson.build
# make
cc -O3 -include rte_config.h -mcpu=cortex-a72 -I/usr/local/include
-I/usr/include/libnl3 -DALLOW_EXPERIMENTAL_API main.c
-o build/helloworld-shared -L/usr/local/lib/aarch64-linux-gnu
-Wl,--as-needed -lrte_node -lrte_graph -lrte_pipeline -lrte_table
-lrte_pdump -lrte_port -lrte_fib -lrte_ipsec -lrte_vhost -lrte_stack
-lrte_security -lrte_sched -lrte_reorder -lrte_rib -lrte_dmadev
-lrte_regexdev -lrte_rawdev -lrte_power -lrte_pcapng -lrte_member
-lrte_lpm -lrte_latencystats -lrte_jobstats -lrte_ip_frag -lrte_gso
-lrte_gro -lrte_gpudev -lrte_eventdev -lrte_efd -lrte_distributor
-lrte_cryptodev -lrte_compressdev -lrte_cfgfile -lrte_bpf -lrte_bitratestats
-lrte_bbdev -lrte_acl -lrte_timer -lrte_hash -lrte_metrics -lrte_cmdline
-lrte_pci -lrte_ethdev -lrte_meter -lrte_net -lrte_mbuf -lrte_mempool
-lrte_rcu -lrte_ring -lrte_eal -lrte_telemetry -lrte_kvargs -lbsd
ln -sf helloworld-shared build/helloworldAs you can see, the compilation finished without errors, and the correct libraries were included, thanks to our pkg-config “hack”. The system is now confirmed to be set up with the DPDK version of our choice.
Lastly, run the app to double-check:
# ./helloworld-shared
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:03:00.0 (socket -1)
EAL: Probe PCI driver: mlx5_pci (15b3:a2d6) device: 0000:03:00.1 (socket -1)
TELEMETRY: No legacy callbacks, legacy socket not created
hello from core 1
hello from core 2
hello from core 3
hello from core 4
hello from core 5
hello from core 6
hello from core 7
hello from core 0Enjoy…