Since a cluster uses a large amount of resources, a large organization is likely to have only one and to share it among multiple groups. DataFlow provides basic resource scheduling services for the cluster, but this does not guarantee that jobs fairly share resources on the nodes. Jobs still compete for resources, with one possibly acquiring CPU or memory at the expense of the other. These are controlled at the operating system level to enable fairness with low-level support.

Linux allows controlling resource usage by user processes, known as control groups or cgroups. Cgroups are arranged in hierarchies with subsystems like CPU, memory, and I/O associated with a hierarchy. For more information about cgroups and their management, Red Hat provides some useful documentation. DataFlow can be configured to leverage cgroups to ensure jobs do not consume more resources than they have been allotted by use of adhoc cgroups created on a per-job basis.

The use of cgroups is optional. If cgroups are used along with DataFlow, first you must create their hierarchy. Since DataFlow restricts only CPU and memory usage, at least those subsystems should be attached.

After the hierarchy is created, a cgroup that can be managed by the dataflow user should be created on every machine where a node manager is running. As processes are launched by the node manager, subgroups within this cgroup are created for each process to execute within. For example, the following command creates a cgroup named dataflow administered by the dataflow user that can control CPU and memory subsystem usage.

The name of this group must be the same on every machine and is required later when configuring the cluster to use cgroups for managing resource usage. For more information, see Configuring Resource Fairness.

We recommend you create cgroups using the cgconfig service. This will automate the above mentioned steps and can be automatically done on system startup. The configuration details are specified in the file /etc/cgconfig.conf. If the cluster is configured to use cgroups but cannot access them, no job can run on the cluster.