Native Infiniband Support


FhGFS requires OFED API version 1.2 or higher for kernel and userspace Infiniband support.

Clients
Native Infiniband support in FhGFS is based on the Open Fabrics Enterprise Distribution ibverbs API (http://www.openfabrics.org). To enable native Infiniband support, the FhGFS Client kernel modules have to be compiled with Infiniband support. Client Infinband support is enabled by setting corresponding the buildArgs option in the client autobuild file (/etc/fhgfs/fhgfs-client-autobuild.conf). This file also contains more details on the values that you need to set to enable Infiniband support.

Servers
The FhGFS OpenTk communication library (libfhgfs-opentk.so) for userspace daemons comes pre-built with and without native Infiniband support. You can use the following command to enable the shared library version with native Infiniband support:
$ fhgfs-opentk-lib-update-ib

(Note that the command above is also automatically executed after the FhGFS opentk-lib package installation.)

Verifying Infiniband Connectivity
At runtime, you can check whether your IB devices have been discovered by using the FhGFS online configuration tool. The tool mode “LISTNODES” will show a list of all running nodes and their configured network interfaces in order of preference. The word “RDMA” will be appended to interfaces that are enabled for the native Infiniband protocol. Use the following command to list the servers and their available network interfaces:
$ fhgfs-ctl --listnodes --nodetype=storage --details
$ fhgfs-ctl --listnodes --nodetype=meta --details


To check whether the clients are connecting to the servers via RDMA or whether they are falling back to TCP because of configuration problems, use the following command to list active client connections:
$ fhgfs-net

(Note that the command above reads information from /proc/fs/fhgfs/<…>/X_nodes.)

In addition to the commands above, the log files also provide information on established connections and connection failures (if you are using at least logLevel=3). See /var/log/fhgfs-X.log on clients and servers.

General Infiniband Tuning Settings

QLogic/Intel Infiniband Tuning


Back to User Guide - Tuning and advanced Configuration
Valid XHTML 1.0 Transitional :: Valid CSS :: Powered by WikkaWiki