Frequently Asked Questions (FAQ)


General Questions

  1. Who should use FhGFS?
  2. Is FhGFS suitable for production use?
  3. What are the minimum system requirements to run FhGFS?
  4. Which network types are supported by FhGFS?
  5. Can I test-drive FhGFS for free?
  6. Do I need a SAN or any kind of shared storage to run FhGFS?
  7. Does FhGFS provide high-availability?
  8. Can I do everything that can be done with the Admon GUI from the command-line?
  9. Is FhGFS open-source?
  10. What does the abbreviation FhGFS mean?
  11. How is FhGFS related to BeeGFS?
  12. How does FhGFS distribute metadata?
  13. How to backup metadata?


Installation & Startup

  1. Where can I find information about why an Admon-based installation of FhGFS fails?
  2. We are not using one of the distributions you are providing packages for, which means we cannot install RPM or DEB packages. What can we do to try FhGFS?
  3. Do I need to be ''root'' to run FhGFS?
  4. Is it possible to run storage servers and clients on the same machine?
  5. Is it possible to run multiple instances of the same daemon on the same machine?
  6. Do I need a special Linux kernel for FhGFS?
  7. Does FhGFS need time synchronization between different machines?
  8. How to upgrade from one FhGFS major release series to another?
  9. Why do I get an 'Access denied' error on the client even with correct permissions?


Configuration & Tuning

  1. Where can I find system tuning recommendations?
  2. How can I remove a node from an existing file system?
  3. Is it possible to free up a storage target before removal?
  4. I did some testing and want to start over again. Is there an easy way to delete all residues from the old file system?
  5. My hosts have more than one network interface. Is there a way of configuring FhGFS to use only certain interfaces or certain networks?
  6. Is it possible to force a certain node ID or target ID for FhGFS servers?
  7. What needs to be done when a server hostname has changed?
  8. My client refuses to mount because of an 'unknown storage target'
  9. How can I solve problems with the .Xauthority file for home dirs?




[General Questions]



Who should use FhGFS?


Everyone with an interest in fast, flexible, and easy to manage storage. This is typically the case for HPC clusters, but FhGFS was designed to also work well on smaller deployments like a group of workstations or even heterogeneous systems with different hardware architectures.


Is FhGFS suitable for production use?


Yes, absolutely! FhGFS is not a proof-of-concept or research implementation. FhGFS was designed for production use since the beginning of its development and is fully supported by Fraunhofer and our international partners.


What are the minimum system requirements to run FhGFS?


Currently, native FhGFS client and server components are available for Linux on x86, x86_64 and PowerPC/Cell architectures. In general, all FhGFS components can run on a single machine with only a single CPU core, a single hard disk and less than 1GB of RAM. But this is probably not what you want to do in practice, so here are some recommendations for your hardware configuration:







Which network types are supported by FhGFS?


FhGFS supports all TCP/IP based networks and the native Infiniband protocol (based on OFED ibverbs). Servers and clients can handle requests from different networks at the same time (e.g. your servers can be equipped with Infiniband and Ethernet interconnects and some clients connect via native Infiniband while the rest connects via TCP/Ethernet). Clients with multiple connection paths (like Infiniband and Ethernet or multiple Ethernet ports) can also do network failover if the primary connection path fails.


Can I test-drive FhGFS for free?


Yes, FhGFS can be downloaded and used free of charge without any limitations.


Do I need a SAN or any kind of shared storage to run FhGFS?


No, FhGFS is not a storage area network (SAN) file system and it does not rely on shared access to storage. Typical hardware configurations for FhGFS consist of multiple servers with internal (or external non-shared) RAID storage.


Does FhGFS provide high-availability?


Starting with the 2012.10 release series, FhGFS provides optional metadata and file contents redundancy (replication). See here for more information. Further extensions of this feature are still work in progress and will later provide high availability.

Otherwise, clients will ignore the fact that a server is down until they need data that is located on the missing host. In this case, they can either wait for the server to come back online or let the IO operation fail after a configurable period of time.

You are also free to implement cold failover with active/active server pairs based on shared storage and external tools like e.g. Heartbeat or Pacemaker. Though FhGFS is using write caching on the servers, this will allow access to your existing data in case of a machine failure.
To setup active/active pairs with failover based on external tools, you will need to run a new instance of the failed daemon on another machine, so that two instances of the same daemon are running on one machine (using different network protocol ports) when one machine is down. To make the failover FhGFS daemon instance appear like the original daemon, you will need to also move IP addresses to the failover machine and make sure the failover daemon instance uses the same NodeID on the failover host. (See section "Configuration & Tuning" on this page for information on how to manually define a NodeID instead of using the hostname as NodeID.)
Note that the customer wiki contains more details on this topic.


Can I do everything that can be done with the Admon GUI from the command-line?


Yes, installation, configuration and status queries can also be done via the command line:


Is FhGFS open-source?


Currently, only the client source code is available under the GPL.

It was publicly announced at the International Supercomputing Conference 2013 in Leipzig and it is guaranteed as part of the DEEP-ER project that the server components of FhGFS will also be made available under an open-source license.
Until this happens, we make agreements with some supported customers to share parts of the source code on a per-project basis and can in special cases also provide the full source code to customers for whom it might be business critical.

Anyways, you can download and use all components of FhGFS free of charge.


What does the abbreviation FhGFS mean?


FhG: The German name of the Fraunhofer Society is Fraunhofer Gesellschaft, its official abbreviation is FhG.
FS: FS is short for File System.

Note: Since the meaning of the letter G in FhGFS is not always obvious, some people started calling FhGFS the Fraunhofer Global File System or Fraunhofer GFS or sometimes even just Fraunhofer.


How is FhGFS related to BeeGFS?


BeeGFS® is the new name for FhGFS. With the community of users and partners getting more and more active and involved, we felt like it would be a good idea to pick a new name.
However, to make the transition as smooth as possible, the change to BeeGFS is rather a process, not an event, so the old name will still be around for quite a while.


How does FhGFS distribute metadata?


FhGFS distributes metadata on a per-directory basis. This means each directory in your filesystem can be managed by a different metadata server. When a new directory is created, the system automatically picks one of the available metadata servers to be responsible for the files inside this directory. New subdirectories of this directory can be assigned to other metadata servers, so that load is balanced across all available metadata servers.


How to backup metadata?






Additional Notes:





[Installation and Startup]



Where can I find information about why an Admon-based installation of FhGFS fails?


If you tried to install FhGFS by using the Administration and Monitoring GUI, there are two log facilities that can provide useful information:


We are not using one of the distributions you are providing packages for, which means we cannot install RPM or DEB packages. What can we do to try FhGFS?


You can try to download the RPM packages for Suse or Red Hat and unpack them with rpm2cpio and cpio (rpm2cpio packname | cpio -i). Afterwards, you might need to change the init scripts to work with your distribution. We cannot guarantee that FhGFS will work with your distribution, but it is worth a try.


Do I need to be ''root'' to run FhGFS?


Yes and no. You do not need root access to start the servers, as these are all userspace processes (of course, you need to change the configuration files to store the file system data and log files at places, where you have write access).
The client is a kernel module. To load the module and mount the file system, you normally need to have root privileges. As an alternative, it is also possible to grant permissions to execute the corresponding commands for non-root users via /etc/sudoers.


Is it possible to run storage servers and clients on the same machine?


Yes, it is. You do not need dedicated hosts for any service in FhGFS. For example, it is possible to have one host running a management daemon, a metadata server, a storage server and a client at the same time.


Is it possible to run multiple instances of the same daemon on the same machine?


Yes, it is. To do this, you will need to create a separate configuration file for the other daemon instance, using different network ports, a different storage directory and so on. You might also need to set a different NodeID manually for the other instance, in case the second daemon instance should register at the same management daemon where the first daemon instance on this machine is already registered. (See "Configuration & Tuning" section on this page for information on how to set the NodeID.)

Starting with the 2012.10 release series, the standard FhGFS service init scripts (/etc/init.d/fhgfs-XY) can manage multiple instances of the same daemon on a machine. To enable support for this, see the comment on MULTI_MODE in /etc/default/fhgfs-XY.

The client mount config file (/etc/fhgfs/fhgfs-mounts.conf) allows specification of multiple mount points by adding one or more additional lines of mount path and corresponding config file. (Note: In this case, make sure to specify different client ports in the different client config files.)

The FhGFS-on-demand script (/opt/fhgfs/sbin/fhgfs-ondemand) from the fhgfs-utils package is another possible way to run a separate FhGFS instance, especially on a per-job basis for clusters or in cloud environments.


Do I need a special Linux kernel for FhGFS?


FhGFS client modules require at least kernel version 2.6.16, but apart from that, FhGFS does not need a special kernel: The client kernel modules were designed patchless (so you don't need to apply any kernel patches to run FhGFS) and the server components of FhGFS run as userspace daemons.


Does FhGFS need time synchronization between different machines?


Yes, the time of all FhGFS client and server machines needs to be synchronized for various reasons, e.g. to provide consistent file modification timestamps and for consistent ID generation. Make sure all server clocks are set to the correct time and date (e.g. with date or ntpdate) before starting up FhGFS services.
A service like ntp can then be used to keep the clocks of all machines in sync.


How to upgrade from one FhGFS major release series to another?


For an upgrade from version 2009.08 to version 2011.04, see here for instructions.
For an upgrade from version 2011.04 to version 2012.10, see here for instructions.
For an upgrade from version 2012.10 to version 2014.01, see here for instructions.


Why do I get an 'Access denied' error on the client even with correct permissions?


Please check if you have SELinux enabled on the client machine. If it is enabled, disabling it should solve your problem. SELinux can be disabled by setting SELINUX=disabled in the config file /etc/selinux/config. Afterwards, you might need to reboot your client for the new setting to become effective.





[Configuration and Tuning]



Where can I find system tuning recommendations?


There are a lot of tuning possibilities to increase performance - not only in FhGFS, but also in the Linux kernel, the underlying local file systems, the hardware drivers etc. Please have a look at the Tuning Guide for tips and recommendations on system tuning.


How can I remove a node from an existing file system?


Use the FhGFS Control Tool fhgfs-ctl (contained in the fhgfs-utils package) if you need to unregister a server from the file system:

Note: If you want to move files from a storage server to the other storage servers before removing, see here: "Is it possible to free up a storage target before removal?"


Is it possible to free up a storage target before removal?


The fhgfs-ctl tool has a mode called "migrate", which allows you to move all files from a certain storage target to other storage targets.

Note: Migration is directory-based and currently single-threaded, so a single migration instance may perform well below the capabilities of the hardware. It is possible to start multiple non-interfering instances of "fhgfs-ctl --migrate" on the same client (or different clients) for different directory trees, e.g. one instance for /mnt/fhgfs/subdir1 and another instance for /mnt/fhgfs/subdir2.


I did some testing and want to start over again. Is there an easy way to delete all residues from the old file system?


To revert your installation to a completely clean file system, you can follow these steps:
  1. Stop all clients and servers (via the Admon GUI or via /etc/init.d/fhgfs-X stop)
  2. Delete the data directories of the metadata servers, storage servers and the management daemon (these are the directories named "store...Directory" in the corresponding /etc/fhgfs/fhgfs-X.conf config files)
  3. Start all servers and clients again


My hosts have more than one network interface. Is there a way of configuring FhGFS to use only certain interfaces or certain networks?


Yes, there are two different settings that can be used to achieve this:


Is it possible to force a certain node ID or target ID for FhGFS servers?


First of all you have to know that FhGFS uses 2 different kinds of IDs for a server node.

  • A string-based node ID. By default, the hostname is used for this.
  • A numeric ID. By default, this ID is randomly generated by the management daemon.
    • Numeric IDs are 16-bit values in range 1..65535.

  • The string-based node ID is most important to the user/administrator, because it is the ID you will see in log messages to conveniently identify the servers. But internally, FhGFS uses the numeric IDs, mostly because they can be used more efficiently.

    Each FhGFS server daemon checks for special files inside their storage directory during startup. To force certain IDs instead of having them generated automatically, you would create these special files before first startup of a FhGFS server daemon.
    The names of these special files are:

    Example:
    Assuming you want to set the string-based node ID of your first storage server to "storage01" and the numeric ID to "1". The first storage server also provides the first storage target, so you would want to set the string-based target ID to "target01" and the numeric target ID to "1". (The storeStorageDirectory in /etc/fhgfs/fhgfs-storage.conf for this example is set to /mnt/myraid/fhgfs_storage.)
    To force these IDs, you would use the commands below before starting up the fhgfs-storage daemon for the first time:
    $ echo storage01 > /mnt/myraid/fhgfs_storage/nodeID
    $ echo 1 > /mnt/myraid/fhgfs_storage/nodeNumID
    $ echo target01 > /mnt/myraid/fhgfs_storage/targetID
    $ echo 1 > /mnt/myraid/fhgfs_storage/targetNumID

    The ID settings can be confirmed by checking the server log file (/var/log/fhgfs-storage.log) after starting up the daemon. Or by querying the management server:
    $ fhgfs-ctl --listnodes --nodetype=storage
    $ fhgfs-ctl --listtargets --longnodes

    Important notes:


    What needs to be done when a server hostname has changed?


    Scenario: 'hostname' or '$HOSTNAME' report a different name than during the FhGFS installation and FhGFS servers refuse to start up. Logs tell the nodeID has changed and therefore a startup was refused.

    Note that by default, node IDs are generated based on the hostname of a server. As IDs are not allowed to change, see here for information on how to manually set your ID back to the previous value: "Is it possible to force a certain node ID or target ID for FhGFS servers?"


    My client refuses to mount because of an 'unknown storage target'


    Scenario: While testing FhGFS, you removed the storage directory of a storage server, but kept the storage directory of the management server. Now the FhGFS client refuses to mount and prints an error about an unknown storage target to the log file.

    What happened to your file system: When you start a new fhgfs-storage daemon with a given storage directory, the daemon initializes this directory by assigning an ID to this storage target path and registering this targetID at the management server.
    When you delete this directory, the storage server creates a new directory on next startup with a new ID and also registers this ID at the management server. (Because the storage server cannot know what happend to the old directory and whether you might have just moved the data to another machine, so it needs a new ID here.)

    When the client starts, it performs a sanity check by querying all registered targetIDs from the mangement server and checks whether all of them are accessible. If you removed a storage directory, this check fails and thus the client refuses to mount. (Note: This sanity check can be disabled, but it is definitely a good thing in this case and saves you from more trouble.)

    Now you have two alternative options...

    Solution A: Simply remove the storage directories of all FhGFS services to start with a clean new file system:

    1) Stop all the FhGFS server daemons, i.e. fhgfs-mgmtd, fhgfs-meta, fhgfs-storage:
    $ /etc/init.d/fhgfs-... stop (or use the Admon GUI).
    2) Delete ("rm -rf") all their storage directories. The paths to the server storage directores can be looked up in the server config files:
    3) Restart the daemons ("/etc/init.d/fhgfs-... start" or use the GUI).

    Now you have a fresh new file system without any of the previously registered targetIDs.

    Solution B: Unregister the invalid targetID from the management server:
    For this, you would first use the fhgfs-ctl tool (the tool is part of the fhgfs-utils package on a client) to list the registered target IDs:
    $ fhgfs-ctl --listtargets
    Then check the contents of the file "targetID" in your storage directory on the storage server to find out which targetID is the current one that you want to keep.
    For all other targetIDs from the list, which are assigned to this storage server but are no longer valid, use this command to unregister them from the management daemon:
    $ fhgfs-ctl --unmaptarget <targetID>
    Afterwards, your client will no longer complain about the missing storage targets.

    Note: There are options in the server config files to disallow initialization of new storage directories and registration of new servers or targets, which are not set by default, but should be set for production environments. See storeAllowFirstRunInit and sysAllowNewServers.


    How can I solve problems with the .Xauthority file for home dirs?


    If users have their home directory on FhGFS, the .Xauthority file for X-forwarding cannot be created correctly, due to missing hardlink support in the FhGFS 2011.04 release series.
    The solution is to simply create the .Xauthority file on a different file system, e.g. in /var/tmp:

    /usr/local/bin/xauth.sh:
    ========================
    #!/bin/sh
    export XAUTHORITY=/var/tmp/Xauthority-$USER
    /usr/bin/xauth

    /etc/ssh/sshd_config:
    =====================
    XAuthLocation=/usr/local/bin/xauth.sh

    /etc/profile.d/xauth.sh:
    ========================
    export XAUTHORITY=/var/tmp/Xauthority-$USER

    /etc/profile.d/xauth.csh:
    =========================
    setenv XAUTHORITY "/var/tmp/Xauthority-${USER}"



    List of all categories
    Valid XHTML 1.0 Transitional :: Valid CSS :: Powered by WikkaWiki