VectorH interacts with YARN through a custom implementation of the YARN Client API, named DbAgent.

The DbAgent allocates the amount of resources required by VectorH (memory and cores per container) in a YARN environment. It then starts the set of VectorH backend processes with specific slave nodes given as a list of parameters.

During startup, the required set of cluster resources is registered and resource amounts are computed based on the vectorwise.conf file. If YARN is not present, the Vector Server starts without the DbAgent.

The following YARN settings are required.

For Hadoop version 2.4 (and older) the CapacityScheduler out of the box uses the DefaultResourceCalculator, which takes into account only the memory dimension for computing the number of available containers; the amount of virtual cores is ignored. We recommend using DominantResourceCalculator instead.

The installer creates actian as a service ID (system account) to reduce the chances of conflict on one or more of the other nodes in the cluster. Service accounts have uids below 1000, 500 on some systems. This causes YARN to complain.

To stop YARN complaining about the service ID, add the following line to container-executor.cfg under HADOOP_CONF_DIR:

To work around YARN-2892 an "actian" (short-name) proxy user must be created. The following YARN configuration snippet allows the user "actian" (which can be the Kerberos principal actian@domain) to impersonate the local user "actian" by its short-name.

Also, make sure that YARN is configured to support long running applications. By default, any delegation token has a maximum lifetime of seven days. The following properties allow YARN ResourceManager to request new tokens when the existing ones are past their maximum lifetime. YARN is then able to continue performing localization and log-aggregation on behalf of the hdfs user.

By default, integration with Apache Hadoop YARN is disabled. During installation, you are asked if you want to enable it.

You can configure integration with YARN by using the iisetres command to change the following parameters in config.dat:

Cluster resources are allocated statically at startup time for the lifetime of the Vector Server. After the database is shut down, DbAgent terminates and deallocates the resources (killing all YARN containers registered for the current database).

To influence the amount of resources allocated, you can modify the following parameters in vectorwise.conf:

The Dbagent determines the set of slave nodes by selecting the best num_nodes nodes (N) from the given list of DataNodes (M) with respect to their resource (such as CPU and memory) capabilities and local data (VectorH columnar files stored in HDFS), if any.

DbAgent handles the process transparently, as follows:

Initially, when no data is stored in the system, this strategy determines a set of (more or less) homogeneous nodes, which is preferable for load-balancing query workloads.

After loading data into the database, the VectorH slave set is kept the same over any system restart because the data resides on those nodes. If a restart is due to a system failure (node crash) and your VectorH version comes with HDFS Custom Block Placement support, the missing replicas are determined by looking at the data-locality information and are re-replicated to other (possibly new) slave nodes. Moreover, in the latter case, the slave set will automatically adjust to a new set of slaves based on what nodes from the remaining DataNodes are still running. You do not have to manually modify the configuration files to remove the failed nodes.

Preemption support allows YARN to preempt a VectorH job with a higher priority job.

This feature is important in a scenario where a queue has a defined level of cluster resources, but must wait to run applications because other queues are using all of its available resources.

If YARN preemption is enabled, a higher-priority application does not have to wait for a lower priority application (such as a VectorH query) that has taken up the available capacity. An under-served queue can begin to claim its allocated cluster resources almost immediately, without having to wait for an application in another queue to finish running.

After the slave nodes are started, VectorH preemption support in YARN works as follows:

To configure VectorH to use preemption, set the following configuration parameters in the [dbagent] section of vectorwise.conf:

These parameters define the minimum resources VectorH is to allocate. The difference between these values and the [system] num_cores and [memory] max_memory_size values, respectively, defines the target level of performance, which is then requested in a separate lower priority container.

The second application master (WorkloadAppMaster) tries to reach the target amount of resources, given a queue with sufficient resources. The corresponding containers for the target resources may be preempted by YARN at any time (because they are allocated with lower priority) and eventually re-allocated to VectorH as soon as the job that needed extra resources has finished. On the other hand, the required amount of resources is requested with a higher priority (10 by default) and if the job happens to be preempted by YARN, the DbAgent will enter a fallback procedure, which ends by shutting down all Vector servers.

For example, the following settings in vectorwise.conf tell the WsetApplicationMaster to request 6 GB and 4 cores, and the WorkloadAppMaster a further 2 GB and 2 more cores.

Running two or more VectorH clusters at the same time is not supported if their entire amount of resources cannot be allocated, even though VectorH has YARN preemption enabled to displace some containers from other queues. YARN, during preemption, ignores the node locations from where VectorH want resources to be allocated and offers other containers than expected. The major consequence is that a subsequent VectorH cluster may not get the required one-container per node to continue starting the Vector servers. This issue cannot be reproduced if the amount of resources needed is idle at the time VectorH attempts to allocate the containers.

The following YARN configuration settings are relevant for VectorH.

Memory related settings in YARN are as follows:

CPU related settings in YARN are as follows:

We recommend keeping the yarn.scheduler.minimum-allocation-mb constant (1024MB, for example) and setting the yarn.scheduler.maximum-allocation-mb to the same value as yarn.nodemanager.resource.memory-mb. These settings allow applications to acquire memory within the minimum-maximum range. If a YARN application wants to acquire one large container per node, as Vector does, then it can do so. In case of multiple applications (or YARN enabled data frameworks) running in the same cluster, the yarn.scheduler.maximum-allocation-mb property can be adjusted (for example, 40% of yarn.nodemanager.resource.memory-mb). In this case, however, the difference between the (max_memory_size + bufferpool_size) minus yarn.scheduler.minimum-allocation-mb will be out-of-band from YARN and this can increase the resource contention in the cluster. The same applies for the YARN vcores and Vector num_cores configuration options.

You can add nodes to or remove nodes from a VectorH cluster by using a manual process.

Adding or removing nodes requires a full restart and reconfiguration of the cluster, but this typically takes only a few minutes. Individual data partitions, however, are allocated uniquely and evenly to each node based on the original cluster configuration. Queries run against that data are intended to be executed solely on the node where the data resides. Without repartitioning of data, any additional nodes in the cluster are required to access the data remotely, which may introduce a bottleneck on the system, resulting in sub-optimal performance. This will be particularly acute during "cold" runs immediately after the restart. If the [cbm] bufferpool_size is sufficiently large, performance impact will be negligible once the workload is entirely resident in memory.

Note: The VectorH HDFS block placement policy does not help in this situation, given that it is installed and configured beforehand. It will help, however, when nodes fail and are then restarted or replaced, and only if, after the restart, the number of nodes from the slave set (that is, num_nodes) remains the same. Only in such a situation will the HDFS Block Placement program, upon calling hdfs fsck /path/to/hdfs/data/location ‑blocks ‑locations ‑files, handle the re-replication and co-location of the data that was lost.

Similarly, after removing nodes from the cluster, data that was resident on the removed nodes must be retrieved remotely by the remaining nodes, again leading to a performance degradation. This can be minimized after the initial run by increasing the [cbm] bufferpool_size enough so that the entire workload can still be loaded into memory.