Native Infiniband Support

BeeGFS requires OFED API version 1.2 or higher for kernel and userspace Infiniband support.


Native Infiniband support in BeeGFS is based on the Open Fabrics Enterprise Distribution ibverbs API ( To enable native Infiniband support, the BeeGFS Client kernel modules have to be compiled with Infiniband support. Client Infinband support is enabled by setting corresponding the buildArgs option in the client autobuild file (/etc/beegfs/beegfs-client-autobuild.conf). This file also contains more details on the values that you need to set to enable Infiniband support.


The BeeGFS OpenTk communication library ( for userspace daemons comes pre-built with and without native Infiniband support. You can use the following command to enable the shared library version with native Infiniband support:
$ beegfs-setup-rdma

(Note that the command above is also automatically executed after the beegfs-opentk-lib package installation.)

Verifying Infiniband Connectivity

At runtime, you can check whether your IB devices have been discovered by using the BeeGFS online configuration tool. The tool mode LISTNODES will show a list of all running nodes and their configured network interfaces in order of preference. The word "RDMA" will be appended to interfaces that are enabled for the native Infiniband protocol. Use the following command to list the servers and their available network interfaces:
$ beegfs-ctl --listnodes --nodetype=storage --details
$ beegfs-ctl --listnodes --nodetype=meta --details

To check whether the clients are connecting to the servers via RDMA or whether they are falling back to TCP because of configuration problems, use the following command to list active connections on a client:
$ beegfs-net

(Note that the command above reads information from /proc/fs/beegfs/<clientID>/X_nodes.)

A typical source of trouble is to have the ibacm service (/etc/init.d/ibacm) running on the machines. This service causes RDMA connections attempts to stall and should be disabled in all nodes.

In addition to the commands above, the log files also provide information on established connections and connection failures (if you are using at least logLevel=3). See /var/log/beegfs-X.log on clients and servers.

General Infiniband Tuning Settings

Intel/QLogic TrueScale Infiniband Tuning

Intel OmniPath Architecture (OPA) Tuning

Mellanox Infiniband Tuning

Back to User Guide - Tuning and advanced Configuration
Valid XHTML 1.0 Transitional :: Valid CSS :: Powered by WikkaWiki