Preemption Support
Preemption support allows YARN to preempt a VectorH job with a higher priority job.
This feature is important in a scenario where a queue has a defined level of cluster resources, but must wait to run applications because other queues are using all of its available resources.
If YARN preemption is enabled, a higher-priority application does not have to wait for a lower priority application (such as a VectorH query) that has taken up the available capacity. An under-served queue can begin to claim its allocated cluster resources almost immediately, without having to wait for an application in another queue to finish running.
How VectorH Preemption Support Works
After the slave nodes are started, VectorH preemption support in YARN works as follows:
1. Starts a subsequent application master, WorkloadAppMaster, which handles YARN containers preemption.
2. The WorkloadAppMaster allocates a fraction of VectorH cluster resources (cores and query execution memory) and requests the needed containers with a lower priority.
3. YARN preemption acts over this set of resources first and not over the WsetApplicationMaster's core set of resources, which is required for the entire database lifespan.
Configuring VectorH for Preemption Support
To configure VectorH to use preemption, set the following configuration parameters in the [dbagent] section of vectorwise.conf:
• required_num_cores
• required_max_memory_size
These parameters define the minimum resources VectorH is to allocate. The difference between these values and the [system] num_cores and [memory] max_memory_size values, respectively, defines the target level of performance, which is then requested in a separate lower priority container.
The second application master (WorkloadAppMaster) tries to reach the target amount of resources, given a queue with sufficient resources. The corresponding containers for the target resources may be preempted by YARN at any time (because they are allocated with lower priority) and eventually re-allocated to VectorH as soon as the job that needed extra resources has finished. On the other hand, the required amount of resources is requested with a higher priority (10 by default) and if the job happens to be preempted by YARN, the DbAgent will enter a fallback procedure, which ends by shutting down all Vector servers.
Note: In YARN, because containers cannot have only core resources or only memory resources, both parameters must be set for the DbAgent to correctly compute the target amount of resources and start the subsequent application master (WorkloadAppMaster).
For example, the following settings in vectorwise.conf tell the WsetApplicationMaster to request 6 GB and 4 cores, and the WorkloadAppMaster a further 2 GB and 2 more cores.
[dbagent]
required_num_cores=4
required_max_memory_size=4G
[system]
num_cores=6
[memory]
max_memory_size=6G
[cbm]
bufferpool_size=2G
Preemption Support Limitations
Running two or more VectorH clusters at the same time is not supported if their entire amount of resources cannot be allocated, even though VectorH has YARN preemption enabled to displace some containers from other queues. YARN, during preemption, ignores the node locations from where VectorH want resources to be allocated and offers other containers than expected. The major consequence is that a subsequent VectorH cluster may not get the required one-container per node to continue starting the Vector servers. This issue cannot be reproduced if the amount of resources needed is idle at the time VectorH attempts to allocate the containers.