Frequently Asked Questions (FAQ)


General Questions

  1. Who should use BeeGFS?
  2. Is BeeGFS suitable for production use?
  3. What are the minimum system requirements to run BeeGFS?
  4. Which network types are supported by BeeGFS?
  5. Can I test-drive BeeGFS for free?
  6. Do I need a SAN or any kind of shared storage to run BeeGFS?
  7. Does BeeGFS provide high-availability?
  8. Can I do everything that can be done with the Admon GUI from the command-line?
  9. Is BeeGFS open-source?
  10. What does the abbreviation FhGFS mean?
  11. How does BeeGFS distribute metadata?
  12. How to backup metadata?
  13. How can I move a storage target to a different server?
  14. How can I move a metadata target to a different server?
  15. How can I move the management service to a different server?
  16. Can I export BeeGFS via Samba/CIFS to connect Windows hosts?
  17. Can I export BeeGFS via NFS?


Installation & Startup

  1. Where can I find information about why an Admon-based installation of BeeGFS fails?
  2. We are not using one of the distributions you are providing packages for, which means we cannot install RPM or DEB packages. What can we do to try BeeGFS?
  3. Do I need to be ''root'' to run BeeGFS?
  4. Is it possible to run storage servers and clients on the same machine?
  5. Is it possible to run multiple instances of the same daemon on the same machine?
  6. Do I need a special Linux kernel for BeeGFS?
  7. Does BeeGFS need time synchronization between different machines?
  8. How to upgrade from one BeeGFS major release series to another?
  9. Why do I get an 'Access denied' error on the client even with correct permissions?
  10. How can I configure BeeGFS to use ACLs?
  11. How much disk-space is required for metadata?


Configuration & Tuning

  1. Where can I find system tuning recommendations?
  2. How can I remove a node from an existing file system?
  3. Is it possible to free up a storage target before removal?
  4. I did some testing and want to start over again. Is there an easy way to delete all residues from the old file system?
  5. My hosts have more than one network interface. Is there a way of configuring BeeGFS to use only certain interfaces or certain networks?
  6. Is it possible to force a certain node ID or target ID for BeeGFS servers?
  7. What needs to be done when a server hostname has changed?
  8. My client refuses to mount because of an 'unknown storage target'
  9. What happens when I delete a file that is still being used by a process?




[General Questions]



Who should use BeeGFS?


Everyone with an interest in fast, flexible, and easy to manage storage. This is typically the case for HPC clusters, but BeeGFS was designed to also work well on smaller deployments like a group of workstations or even heterogeneous systems with different hardware architectures.


Is BeeGFS suitable for production use?


Yes, absolutely! BeeGFS is not a proof-of-concept or research implementation. BeeGFS was designed for production use since the beginning of its development and is fully supported by Fraunhofer and our international partners.


What are the minimum system requirements to run BeeGFS?


Currently, native BeeGFS client and server components are available for Linux on x86, x86_64 and PowerPC/Cell architectures. In general, all BeeGFS components can run on a single machine with only a single CPU core, a single hard disk and less than 1GB of RAM. But this is probably not what you want to do in practice, so here are some recommendations for your hardware configuration:







Which network types are supported by BeeGFS?


BeeGFS supports all TCP/IP based networks and the native Infiniband protocol (based on OFED ibverbs). Servers and clients can handle requests from different networks at the same time (e.g. your servers can be equipped with Infiniband and Ethernet interconnects and some clients connect via native Infiniband while the rest connects via TCP/Ethernet). Clients with multiple connection paths (like Infiniband and Ethernet or multiple Ethernet ports) can also do network failover if the primary connection path fails.


Can I test-drive BeeGFS for free?


Yes, BeeGFS can be downloaded and used free of charge without any limitations.


Do I need a SAN or any kind of shared storage to run BeeGFS?


No, BeeGFS is not a storage area network (SAN) file system and it does not rely on shared access to storage. Typical hardware configurations for BeeGFS consist of multiple servers with internal (or external non-shared) RAID storage.


Does BeeGFS provide high-availability?


Starting with the 2012.10 release series, BeeGFS provides optional metadata and file contents redundancy (replication). In the 2015.03 release series this concept was extended with additional high availability features for transparent failover. Please see here for more information (or here if you still use a release prior to 2015.03).

Besides that, you are also free to implement cold failover with active/active server pairs based on shared storage and external tools like e.g. Heartbeat or Pacemaker. This will allow access to your existing data in case of a machine failure, although BeeGFS is using write caching on the servers, so cold failovers might not be fully transparent for running applications.
To setup active/active pairs with failover based on external tools, you will need to run a new instance of the failed daemon on another machine, so that two instances of the same daemon are running on one machine (using different network protocol ports) when one machine is down. To make the failover BeeGFS daemon instance appear like the original daemon, you will need to also move IP addresses to the failover machine and make sure the failover daemon instance uses the same NodeID on the failover host. (See section "Configuration & Tuning" on this page for information on how to manually define a NodeID instead of using the hostname as NodeID.)
Note that the customer wiki contains more details on this topic.


Can I do everything that can be done with the Admon GUI from the command-line?


Yes, installation, configuration and status queries can also be done via the command line:


Is BeeGFS open-source?


Yes, see here: Source Download


What does the abbreviation FhGFS mean?


FhGFS is the old name of BeeGFSĀ®, which is now named after the nice and busy animals that work together as a swarm and do their important job mostly unnoticed in the background.

FhG: The German name of the Fraunhofer Society is Fraunhofer Gesellschaft, its official abbreviation is FhG.
FS: FS is short for File System.

Note: Since the meaning of the letter G in FhGFS is not always obvious, some people started calling FhGFS the Fraunhofer Global File System or Fraunhofer GFS or sometimes even just Fraunhofer.


How does BeeGFS distribute metadata?


BeeGFS distributes metadata on a per-directory basis. This means each directory in your filesystem can be managed by a different metadata server. When a new directory is created, the system automatically picks one of the available metadata servers to be responsible for the files inside this directory. New subdirectories of this directory can be assigned to other metadata servers, so that load is balanced across all available metadata servers.


How to backup metadata?


As extended attributes are typically not copied by default by many backup tools, here are three different ways to backup metadata, which is stored in extended attributes. However, these are just examples. Other tools like rsync also have options to preserve extended attributes and hardlinks and thus could be used to backup BeeGFS metadata.

To reduce the overall runtime, you might want to combine your backup with tools like "xargs" (see parameter "--max-procs") or "GNU parallel" to run multiple processes in parallel, each on a subset of the directory structure.




Additional Notes:


How can I move a storage target to a different server?


Take the following steps to move a storage target to a different machine. Please, read the whole procedure before taking any action.

  1. Firstly, evaluate how much time you need to take all the following steps. If you need more than the amount of time specified by option sysTargetOfflineTimeoutSecs in file /etc/beegfs/beegfs-client.conf, by default 15 minutes, you will need to shut the BeeGFS clients down or at least remount the file system as readonly, in order to prevent error messages from annoying the users. On the other hand, if you are able to take all the steps under the specified timeout, the clients won't show any error message. They will stall if they try to connect the moving storage target, but they will resume right after the target is back online. Of course you can also raise the timeout and restart the client services.

  2. Then, edit the file /etc/beegfs/beegfs-storage.conf on the current machine and remove the path of the moving storage target from the comma-separated list of storage targets' directories, defined by option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

  3. Stop the storage service on the current machine and unmount the storage target device.
    $ service beegfs-storage stop
    $ umount /data/mydisk

  4. Start the storage service afterwards if it has some remaining storage targets. If you don't need the storage service running on the machine, please uninstall it.
    $ service beegfs-storage start

  5. Check if the storage service is already installed on the new machine. If not, please install it. Do not start the service at this point.
    • If the storage service is already installed on the new server machine and before the move you had multiple instances of the service running on different machines, and now you want to have 2 instances of the service running on the same machine, each using a different node ID, please configure the service to run in multi-mode .

    • In case you don't mind having the storage target associated to the node ID used by the new server, you don't need to configure the storage service to be multi-mode. Later on, in a future step of this procedure, you will be able to simply add the storage targets to the existing instance of the storage service. Only configure the service to be multi-mode if you really want to keep the moved storage target associated to the previous node ID.

  6. Check if the storage target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.

  7. Otherwise, if the storage target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of new machine.

  8. Then, edit the file /etc/beegfs/beegfs-storage.conf on the new machine and add the path of the moved storage target to the comma-separated list of storage targets' directories, defined by option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

  9. Make sure that the service directory contains the right nodeID file.
    • If you are moving a storage target to a machine that already has a storage service running and this service is not in multi-mode, remove all files whose names match the patterns *node*ID and *Node*ID located at the storage target directory being moved. This will cause the storage service to recreate those files with the node ID used by the existing storage service daemon.

    • In any other case, make sure that the file nodeID exists on the service directory and that it contains the ID that the storage service daemon should use to identify itself with the management service. If the file does not exist, just create it with the same content of the originalNodeID file.

  10. Start or restart the storage service.
    $ service beegfs-storage restart

  11. Test if the target is working properly.
    • Check if log files contain any error message.
      $ less /var/log/beegfs-mgmtd.log
      $ less /var/log/beegfs-meta.log
      $ less /var/log/beegfs-storage.log
      $ less/var/log/beegfs-client.log

    • List all storage targets of your system and check if the moved one appears online.
      $ beegfs-ctl --listtargets --longnodes --state

    • If you moved a storage target to an existing storage service running on another server, you will see the previous target listed as unreachable. That happens because the management service still remembers it. If you want to make the management service to forget it, please use the command below.
      $ beegfs-ctl --removenode <NodeID>


How can I move a metadata target to a different server?


Take the following steps to move a metadata target to a different machine. Please, read the whole procedure before taking any action.

  1. Firstly, evaluate how much time you need to take all the following steps. If you need more than the amount of time specified by option sysTargetOfflineTimeoutSecs in file /etc/beegfs/beegfs-client.conf, by default 15 minutes, you will need to shut the BeeGFS clients down or at least remount the file system as readonly, in order to prevent error messages from annoying the users. On the other hand, if you are able to take all the steps under the specified timeout, the clients won't show any error message. They will stall if they try to connect the moving metadata target, but they will resume right after the target is back online. Of course you can also raise the timeout and restart the client services.

  2. Stop the metadata service on the current machine and unmount the metadata target device. If the service is configured to run in multi-mode, be careful to stop the right instance.
    $ service beegfs-meta stop
    $ umount /data/ssd

  3. If you don't need the metadata service running on the current machine, please uninstall it.

  4. Check if the metadata target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.

  5. Otherwise, if the metadata target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of new machine.

  6. Check if the metadata service is already installed on the new machine. If not, please install it. If the metadata service is already installed on the new server machine, please configure the service to run in multi-mode . Do not start the service at this point.

  7. Then, edit the file /etc/beegfs/beegfs-meta.conf on the new machine and add the path of the moved metadata target to option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

  8. Make sure that the file nodeID exists on the service directory and that it contains the ID that the metadata service daemon should use to identify itself with the management service. If the file does not exist, just create it with the same content of the originalNodeID file.

  9. Start or restart the metadata service.
    $ service beegfs-meta restart

  10. Test if the target is working properly.
    • Check if log files contain any error message.
      $ less /var/log/beegfs-mgmtd.log
      $ less /var/log/beegfs-meta.log
      $ less /var/log/beegfs-storage.log
      $ less/var/log/beegfs-client.log

    • List all metadata nodes of your system and check if the moved one appears online.
      $ beegfs-ctl --listnodes --nodetype=meta --reachable


How can I move the management service to a different server?


Take the following steps to move the management service to a different machine. Please, read the whole procedure before taking any action.

  1. Firstly, evaluate how much time you need to take all the following steps. If you need more than the amount of time specified by option sysTargetOfflineTimeoutSecs in file /etc/beegfs/beegfs-client.conf, by default 15 minutes, you will need to shut the BeeGFS clients down or at least remount the file system as readonly, in order to prevent error messages from annoying the users. On the other hand, if you are able to take all the steps under the specified timeout, the clients won't show any error message. They will stall if they try to connect the moving management service, but they will resume right after the service is back online. Of course you can also raise the timeout and restart the client services.

  2. Stop the management service on the current machine. If the service is configured to run in multi-mode, be careful to stop the right instance.
    $ service beegfs-mgmtd stop
    $ umount /data/ssd

  3. If you don't need the management service running on the current machine, please uninstall it.

  4. Check if the storage device where the management directory resides can be moved or connected to the new machine. If so, unmount it, move it, mount it on a directory of the new machine, and make sure it is configured to be mounted at boot time.

  5. Otherwise, if the device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of new machine.

  6. Check if the management service is already installed on the new machine. If not, please install it. Do not start the service at this point. If the management service is already installed on the new server machine, please configure the service to run in multi-mode .

  7. Then, edit the file /etc/beegfs/beegfs-mgmtd.conf on the new machine and add the path to the management service directory to option storeMgmtdDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

  8. Make sure that the file nodeID exists on the service directory and that it contains the ID that the management service daemon should use to identify itself. If the file does not exist, just create it with the same content of the originalNodeID file.

  9. If the new machine has a different hostname, edit the configuration file of each other BeeGFS service running in your system, which are associated to that management service, and write the new hostname as the value of option sysMgmtdHost.

  10. Make sure that the new machine is reachable from the other BeeGFS servers, specially if you are using the client config options connInterfacesFile or connNetFilterFile to restrict which network interfaces can be used by BeeGFS services. Its IP address may be changed freely, because BeeGFS keeps no record of such addresses.

  11. Start or restart the management service.
    $ service beegfs-mgmtd restart

  12. Test if the target is working properly.
    • Check if log files contain any error message.
      $ less /var/log/beegfs-mgmtd.log
      $ less /var/log/beegfs-meta.log
      $ less /var/log/beegfs-storage.log
      $ less/var/log/beegfs-client.log

    • List all management nodes of your system and check if the moved one appears online.
      $ beegfs-ctl --listnodes --nodetype=mgmt --reachable


Can I export BeeGFS via Samba/CIFS to connect Windows hosts?


Yes, you can. Please, take the following steps.
  1. Configure a BeeGFS client host to be the Samba server. Make sure you are using at least:
    • BeeGFS 2015.03-r12, which contains optimizations for Samba, and
    • Samba 4.2, which contains optimizations for BeeGFS.

  2. Add a share definition to file /etc/samba/smb.conf for the BeeGFS mountpoint, as shown in the example below. For more information on these options, please have a look at the Samba documentation.
    [beegfs]
    comment = BeeGFS file system
    public = yes
    path = /mnt/beegfs
    browsable = yes
    writable = yes
    read only = no

  3. Restart the samba server service. Now, you should be able access the BeeGFS file system from a Samba client.

Hints
  1. Consider exporting BeeGFS via NFS, which usually has a better performance than Samba and is supported by recent Windows versions.

  2. If you are using Windows Samba clients, please consider adding the option case sensitive = true to the share definition above, in order to make the BeeGFS export case sensitive. This setting has a positive performance impact on the client, as case-insensitive lookups perform fewer system calls on the exported file system. Nevertheless, take into consideration that such feature may be inconvenient for some windows users and applications.

  3. Another way of improving performance of Windows clients, but keeping the BeeGFS export case-insensitive for ASCII file names, is to store the BeeGFS metadata in a case-insensitive XFS partition, formatted using the command mkfs.xfs with the option -n version=ci, as seen in the example below. This allows now to set the Samba option case sensitive = true, as suggested above, so that Samba is led to perform less syscalls on the exported file system.
        $ mkfs.xfs -d su=128k,sw=8 -l version=2,su=128k -isize=512 -n version=ci /dev/sdx -f


Can I export BeeGFS via NFS?


Yes, you can. Starting with the 2014.01 release series, BeeGFS supports Linux kernel NFS server exports. For previous release series, it is recommended to use unfs3 to export BeeGFS via NFS 3.

In order to serve files via NFS, configure a BeeGFS client host to be an NFS server, and re-export the BeeGFS client mountpoint via NFS. On that system, the following beegfs-client.conf option should be set for kernel NFS export. It makes sure that a file cache entry is refreshed every time its attributes are queried.

Example NFSv4 export (knfsd, /etc/exports):

Example NFSv4 mount:
$ mount -t nfs -overs=4 myserver:/ /mnt/beegfs_via_nfs/


Important Notes

  • Kernel NFS 3 has some limitations that prevent BeeGFS from supporting it. For example, NFS clients exchange cookies with the servers for identifying files when they are no longer on the server cache. The maximum size of such cookies in NFS 3 is too small for encoding BeeGFS entry lookup information. In NFS 4, the maximum cookie size was raised, enabling BeeGFS to use NFS cookies. Although kernel NFS 3 exports might sometimes work in your system, eventually, they are very likely to fail as soon as cookies are used (i.e. when the NFS server starts dropping entries from its cache and the cookie information is needed for lookups). So, make sure that you use kernel NFS version 4.
  • It is a good idea to restrict the NFS protocol versions used in your system, as explained in the section A4 of the NFS documentation. The best way to do that is to add the definition RPCMOUNTDOPTS="--no-nfs-version 3" to the file /etc/sysconfig/nfs, which is read by the script /etc/rc.d/init.d/nfs.
  • Beware that disabling NFS 3 in the server configuration and even restarting the NFS server machine may not be enough for enforcing NFS 4 in your network, because NFS stores allowed connections and sessions on disk. So, when the NFS server is restarted, the allowed connections are loaded from disk and thus, old NFS clients could still be allowed to reconnect via NFS 3. Only after all NFS clients are restarted, NFS 4 will really be used by all parts.





  • [Installation and Startup]



    Where can I find information about why an Admon-based installation of BeeGFS fails?


    If you tried to install BeeGFS by using the Administration and Monitoring GUI, there are two log facilities that can provide useful information:


    We are not using one of the distributions you are providing packages for, which means we cannot install RPM or DEB packages. What can we do to try BeeGFS?


    You can try to download the RPM packages for Suse or Red Hat and unpack them with rpm2cpio and cpio (rpm2cpio packname | cpio -i). Afterwards, you might need to change the init scripts to work with your distribution. We cannot guarantee that BeeGFS will work with your distribution, but it is worth a try.


    Do I need to be ''root'' to run BeeGFS?


    Yes and no. You do not need root access to start the servers, as these are all userspace processes (of course, you need to change the configuration files to store the file system data and log files at places, where you have write access).
    The client is a kernel module. To load the module and mount the file system, you normally need to have root privileges. As an alternative, it is also possible to grant permissions to execute the corresponding commands for non-root users via /etc/sudoers.


    Is it possible to run storage servers and clients on the same machine?


    Yes, it is. You do not need dedicated hosts for any service in BeeGFS. For example, it is possible to have one host running a management daemon, a metadata server, a storage server and a client at the same time.


    Is it possible to run multiple instances of the same daemon on the same machine?


    Yes, it is. Starting with the 2012.10 release series, the standard BeeGFS service init scripts (/etc/init.d/beegfs-XY) can manage multiple instances of the same daemon on a machine. To enable support for this, see the comment on MULTI_MODE in /etc/default/beegfs-XY.

    For multi-mode, you will need to create a separate configuration file for the other daemon instance, using different network ports, a different storage directory, a different log file and so on. If the second daemon instance on a machine should become part of the same file system instance (i.e. it registers at the same management daemon as the first daemon instance on this machine), then you would also need to set a different NodeID manually for the second daemon instance. (See "Configuration & Tuning" section on this page for information on how to manually set the NodeID.)

    For the client, multi-mode is also available, but the client mount config file (/etc/beegfs/beegfs-mounts.conf) also allows specification of multiple mount points without enabled multi-mode, by adding one or more additional lines of mountpoint and corresponding client config file. (Note: Make sure to specify different client ports in the different client config files.)

    The BeeGFS-on-demand script (/opt/beegfs/sbin/beegfs-ondemand-v2) from the beegfs-utils package is another possible way to run a separate BeeGFS instance, especially on a per-job basis for clusters or in cloud environments.


    Do I need a special Linux kernel for BeeGFS?


    BeeGFS client modules require at least kernel version 2.6.18, but apart from that, BeeGFS does not need a special kernel: The client kernel modules were designed patchless (so you don't need to apply any kernel patches and don't even need to reboot to install and run the BeeGFS client) and the server components of BeeGFS run as userspace daemons, so they are independent of the kernel version.


    Does BeeGFS need time synchronization between different machines?


    Yes, the time of all BeeGFS client and server machines needs to be synchronized for various reasons, e.g. to provide consistent file modification timestamps and for consistent ID generation. Make sure all server clocks are set to the correct time and date (e.g. with date or ntpdate) before starting up BeeGFS services.
    A service like ntp can then be used to keep the clocks of all machines in sync.


    How to upgrade from one BeeGFS major release series to another?


    For an upgrade from version 2009.08 to version 2011.04, see here for instructions.
    For an upgrade from version 2011.04 to version 2012.10, see here for instructions.
    For an upgrade from version 2012.10 to version 2014.01, see here for instructions.


    Why do I get an 'Access denied' error on the client even with correct permissions?


    Please check if you have SELinux enabled on the client machine. If it is enabled, disabling it should solve your problem. SELinux can be disabled by setting SELINUX=disabled in the config file /etc/selinux/config. Afterwards, you might need to reboot your client for the new setting to become effective.


    How can I configure BeeGFS to use ACLs?


    In BeeGFS, Access Control Lists (ACLs) are stored as extended file attributes of metadata files. Therefore, you need to make sure that extended attributes are enabled in the underlying file systems of the metadata services. For example, in an ext4 file system, the options used to mount the storage devices must include user_xattr. Nevertheless, in recent Linux distributions, this option is set by default when file systems that support extended attributes are mounted.

    After that, make sure that you are using at least the BeeGFS release 2015.03-r13, which fully supports ACLs. Then, edit the metadata configuration file (/etc/beegfs/beegfs-meta.conf) and check if the following options are set to true.
    storeUseExtendedAttribs = true
    storeClientXAttrs       = true
    storeClientACLs         = true

    By default, the option storeUseExtendedAttribs is set to true during installation. This particular option can only be changed before the first service startup, while the options storeClientXAttrs and storeClientACLs may also be enabled later.

    In addition, edit the client configuration file (/etc/beegfs/beegfs-client.conf) and check if the following options are set to true.
    sysXAttrsEnabled = true
    sysACLsEnabled   = true        

    If you change the configuration files, remember to restart their respective services afterwards.



    How much disk-space is required for metadata?


    This depends on things like the average file size that you have or how many files you want to be able to create in total.

    In general, we recommend to have about 0.5% of the total storage capacity for metadata. However, this number is based on gathered statistics of file systems at different cluster sites and thus might or might not fit for your case.

    As a rule of thumb, 500GB of metadata capacity are sufficient for about 200 million files, if the underlying metadata storage is formatted with ext4 according to the recommendations in the metadata server tuning guide: Metadata Server Tuning

    More specifically, for every file that a user creates, one metadata file is created on one of the metadata servers. For every directory that a user creates, two directories and two metadata files are created on one of the metadata servers.

    For file metadata, if the underlying local metadata server file system (e.g. ext4) is formatted according to our recommendations with large inodes (e.g. "mkfs.ext4 -I 512", as described here: Metadata Server Tuning) then the BeeGFS metadata as extended attribute fits completely into the inode of the underlying local file system and does not use any additional disk space. If the underlying metadata server file system is not formatted with large inodes, then the underlying local file system will need to allocate a full block (usually 4KB) to store the BeeGFS metadata information in addition to using up one inode.

    Access Control Lists (ACLs) and user extended attributes are also stored as extended attributes in the corresponding metadata files and thus add to the required disk space, depending on whether they still fit into the inode or whether a new block needs to be allocated.

    For each directory, one inode and one directory contents block (usually 4KB) are used on the underlying local file system until there are so many subentries in the directory that another directory contents block needs to be allocated by the underlying file system. How many entries fit into one block depends on things like user file name length, but usually 10+ entries fit into one directory block.
    So if a user creates e.g. many directories with only one file inside them, this will significantly increase the used number of inodes and disk space on the underlying local file system compare to the case where the same number of files is stored in fewer directories.

    Note that while ext4 is generally recommended for metadata storage because of its performance advantages for BeeGFS metadata workloads compared to other local Linux file systems, xfs has the advantage of using a dynamic number of inodes, meaning new inodes can be created as long as there is free disk space. ext4 on the other is based on a static number maximum number of inodes that is defined when the file system is formatted via mkfs (e.g. "mkfs.ext4 -i <number>"). So it can happen with ext4 that you have disk space left but run out of available inodes or vice versa. Use "df -ih" to see information on available inodes for mounted file systems.





    [Configuration and Tuning]



    Where can I find system tuning recommendations?


    There are a lot of tuning possibilities to increase performance - not only in BeeGFS, but also in the Linux kernel, the underlying local file systems, the hardware drivers etc. Please have a look at the Tuning Guide for tips and recommendations on system tuning.


    How can I remove a node from an existing file system?


    Use the BeeGFS Control Tool beegfs-ctl (contained in the beegfs-utils package) if you need to unregister a server from the file system:

    Note: If you want to move files from a storage server to the other storage servers before removing, see here: "Is it possible to free up a storage target before removal?"


    Is it possible to free up a storage target before removal?


    The beegfs-ctl tool has a mode called "migrate", which allows you to move all files from a certain storage target to other storage targets.

    Note: Migration is directory-based and currently single-threaded, so a single migration instance may perform well below the capabilities of the hardware. It is possible to start multiple non-interfering instances of "beegfs-ctl --migrate" on the same client (or different clients) for different directory trees, e.g. one instance for /mnt/beegfs/subdir1 and another instance for /mnt/beegfs/subdir2.


    I did some testing and want to start over again. Is there an easy way to delete all residues from the old file system?


    To revert your installation to a completely clean file system, you can follow these steps:
    1. Stop all clients and servers (via the Admon GUI or via /etc/init.d/beegfs-X stop)
    2. Delete the data directories of the metadata servers, storage servers and the management daemon (these are the directories named "store...Directory" in the corresponding /etc/beegfs/beegfs-X.conf config files)
    3. Start all servers and clients again


    My hosts have more than one network interface. Is there a way of configuring BeeGFS to use only certain interfaces or certain networks?


    Yes, there are two different settings that can be used to achieve this:


    Is it possible to force a certain node ID or target ID for BeeGFS servers?


    Short answer:
    Yes. Use the corresponding setup script of each service to manually define IDs:
    $ /opt/beegfs/sbin/beegfs-setup-XY
    (Add "-h" argument for help and usage examples.)

    Long answer:
    First of all you should know that BeeGFS uses 2 different kinds of IDs for a server node.

  • A string-based node ID. By default, the hostname is used for this.
  • A numeric ID. By default, this ID is sequentially generated by the management daemon in 2015.x and higher releases (and randomly generated in 2014.x releases).
    • Numeric IDs are 16-bit values in range 1..65535.

  • The string-based node ID is most important to the user/administrator, because it is the ID you will see in log messages to conveniently identify the servers. But internally, BeeGFS uses the numeric IDs, mostly because they can be used more efficiently.

    Each BeeGFS server daemon checks for special files inside their storage directory during startup. To force certain IDs instead of having them generated automatically, you would create these special files before first startup of a BeeGFS server daemon.
    The names of these special files are:

    Example:
    Assuming you want to set the string-based node ID of your first storage server to "storage01" and the numeric ID to "1". The first storage server also provides the first storage target, so you would want to set the string-based target ID to "target01" and the numeric target ID to "1". (The storeStorageDirectory in /etc/beegfs/beegfs-storage.conf for this example is set to /mnt/myraid/beegfs_storage.)
    To force these IDs, you would use the commands below before starting up the beegfs-storage daemon for the first time:
    $ echo storage01 > /mnt/myraid/beegfs_storage/nodeID
    $ echo 1 > /mnt/myraid/beegfs_storage/nodeNumID
    $ echo target01 > /mnt/myraid/beegfs_storage/targetID
    $ echo 1 > /mnt/myraid/beegfs_storage/targetNumID

    The ID settings can be confirmed by checking the server log file (/var/log/beegfs-storage.log) after starting up the daemon. Or by querying the management server:
    $ beegfs-ctl --listnodes --nodetype=storage
    $ beegfs-ctl --listtargets --longnodes

    Important notes:


    What needs to be done when a server hostname has changed?


    Scenario: 'hostname' or '$HOSTNAME' report a different name than during the BeeGFS installation and BeeGFS servers refuse to start up. Logs tell the nodeID has changed and therefore a startup was refused.

    Note that by default, node IDs are generated based on the hostname of a server. As IDs are not allowed to change, see here for information on how to manually set your ID back to the previous value: "Is it possible to force a certain node ID or target ID for BeeGFS servers?"


    My client refuses to mount because of an 'unknown storage target'


    Scenario: While testing BeeGFS, you removed the storage directory of a storage server, but kept the storage directory of the management server. Now the BeeGFS client refuses to mount and prints an error about an unknown storage target to the log file.

    What happened to your file system: When you start a new beegfs-storage daemon with a given storage directory, the daemon initializes this directory by assigning an ID to this storage target path and registering this targetID at the management server.
    When you delete this directory, the storage server creates a new directory on next startup with a new ID and also registers this ID at the management server. (Because the storage server cannot know what happend to the old directory and whether you might have just moved the data to another machine, so it needs a new ID here.)

    When the client starts, it performs a sanity check by querying all registered targetIDs from the mangement server and checks whether all of them are accessible. If you removed a storage directory, this check fails and thus the client refuses to mount. (Note: This sanity check can be disabled, but it is definitely a good thing in this case and saves you from more trouble.)

    Now you have two alternative options...

    Solution A: Simply remove the storage directories of all BeeGFS services to start with a clean new file system:

    1) Stop all the BeeGFS server daemons, i.e. beegfs-mgmtd, beegfs-meta, beegfs-storage:
    $ /etc/init.d/beegfs-... stop (or use the Admon GUI).
    2) Delete ("rm -rf") all their storage directories. The paths to the server storage directores can be looked up in the server config files:
    3) Restart the daemons ("/etc/init.d/beegfs-... start" or use the GUI).

    Now you have a fresh new file system without any of the previously registered targetIDs.

    Solution B: Unregister the invalid targetID from the management server:
    For this, you would first use the beegfs-ctl tool (the tool is part of the beegfs-utils package on a client) to list the registered target IDs:
    $ beegfs-ctl --listtargets --longnodes
    Then check the contents of the file "targetNumID" in your storage directory on the storage server to find out which targetID is the current one that you want to keep.
    For all other targetIDs from the list, which are assigned to this storage server but are no longer valid, use this command to unregister them from the management daemon:
    $ beegfs-ctl --unmaptarget <targetID>
    Afterwards, your client will no longer complain about the missing storage targets.

    Note: There are options in the server config files to disallow initialization of new storage directories and registration of new servers or targets, which are not set by default, but should be set for production environments. See storeAllowFirstRunInit and sysAllowNewServers.


    What happens when I delete a file that is still being used by a process?


    When a file is deleted while it is still open by a process, its inode is temporarily moved to the directory disposal on the metadata node. Similarly, its data chunks are moved to directories with the same name on the storage targets. Later, when the file is closed, its inode and chunks are finally erased. So, it is normal to see files in the disposal directories, as long as processes still hold them open.

    However, if a failure occurs on the metadata server machine and those processes don't have the chance of closing the files, the disposal entries won't be erased. In order to remove such disposal entries, you can execute the command below.
        $ beegfs-ctl  --disposeunused --dispose

    If a disposal file cannot be deleted because it is still being used by some process, you will see a message like the one below.
        [1-573EC7CC-C] File still in use

    Nothing to worry about. Just wait the process end and that disposal file will be deleted. If you want to identify such process, please run the command below on your client nodes:
        $ lsof | grep "(deleted)"

    In case the process is aborted, the Linux Kernel will automatically close all files the process was using and the BeeGFS client module will keep on trying to send the close operation to the metadata server. The client will do that also if the network was disconnected at the time of the close operation.

    In case the client node crashes or is rebooted, the disposal files will be removed by the metadata server after about 30 minutes, when the client node is marked as dead.


    List of all categories
    Valid XHTML 1.0 Transitional :: Valid CSS :: Powered by WikkaWiki