Native Infiniband Support


BeeGFS requires OFED API version 1.2 or higher for kernel and userspace Infiniband support.

Clients
Native Infiniband support in BeeGFS is based on the Open Fabrics Enterprise Distribution ibverbs API (http://www.openfabrics.org). To enable native Infiniband support, the BeeGFS Client kernel modules have to be compiled with Infiniband support. Client Infinband support is enabled by setting corresponding the buildArgs option in the client autobuild file (/etc/beegfs/beegfs-client-autobuild.conf). This file also contains more details on the values that you need to set to enable Infiniband support.

Servers
The BeeGFS OpenTk communication library (libbeegfs-opentk.so) for userspace daemons comes pre-built with and without native Infiniband support. You can use the following command to enable the shared library version with native Infiniband support:
$ beegfs-setup-rdma

(Note that the command above is also automatically executed after the BeeGFS opentk-lib package installation.)

Verifying Infiniband Connectivity
At runtime, you can check whether your IB devices have been discovered by using the BeeGFS online configuration tool. The tool mode “LISTNODES” will show a list of all running nodes and their configured network interfaces in order of preference. The word “RDMA” will be appended to interfaces that are enabled for the native Infiniband protocol. Use the following command to list the servers and their available network interfaces:
$ beegfs-ctl --listnodes --nodetype=storage --details
$ beegfs-ctl --listnodes --nodetype=meta --details


To check whether the clients are connecting to the servers via RDMA or whether they are falling back to TCP because of configuration problems, use the following command to list active client connections:
$ beegfs-net

(Note that the command above reads information from /proc/fs/beegfs/<…>/X_nodes.)

In addition to the commands above, the log files also provide information on established connections and connection failures (if you are using at least logLevel=3). See /var/log/beegfs-X.log on clients and servers.

General Infiniband Tuning Settings

QLogic/Intel Infiniband Tuning

Mellanox Infiniband Tuning


Back to User Guide - Tuning and advanced Configuration
Valid XHTML 1.0 Transitional :: Valid CSS :: Powered by WikkaWiki