Quota Preemption

Queues can be configured with a quota. The quota for a queue can be changed while the system is running. In the case that a quota is changed the new quota is applied immediately in the next scheduling cycle. Depending on the type of change there are different impacts.

For a queue that had its quota increased: no impact. The queue could not have used more than its old quota and the new quota is higher providing more resources to be allocated by the workloads running in the queue.

For a queue that had its quota decreased there are two cases.

1. the new, lowered, quota is larger than the current usage in the queue: no impact. Workloads will be allocated until the new quota is reached. All running workloads are unaffected.
2. the new, lowered, quota is smaller than the current usage in the queue: the queue is impacted. Any workloads that were pending in the queue will need to wait until resources become available. Workloads will keep on running until they are done.

This second case is what is targeted by quota preemption. Quota preemption provides the administrator the option to intervene in the running workload when lowering a quota. This document guide users to set up preemption delay for more details on the design, please refer design doc.

Global configuration

Quota preemption is available in YuniKorn 1.8 or later and turned off by default.

To turn on quota preemption it must be turned on globally at the partition level first in the YuniKorn config:

partitions:
  - name: <name of the partition>
    preemption:
      quotapreemptionenabled: <boolean value>

The default value for quotapreemptionenabled is false. Allowed values: true or false, any other value will cause a parse error.

When quota preemption is turned on at the partition level quota changes could trigger a preemption when a queue quota is changed.

Queue configuration

With the global configuration turned on, each queue must be configured to opt in to quota preemption. A queue can opt in by setting the quota.preemption.delay property on the queue.

queues:
  - name: default
    properties:
      quota.preemption.delay: <delay value>

The delay value must be specified in Go time.Duration format. Valid units are s (seconds), m (minutes), and h (hours). Units can be combined.

Examples of valid delay values:

• 15s — 15 seconds
• 5m — 5 minutes
• 1h30m — 1 hour and 30 minutes
• 2h — 2 hours

The delay when not specified defaults to 0. A delay value explicitly set to 0 will prevent the quota change of the queue from triggering preemption. Any non-zero value for the delay will be added to the time the change of the quota was applied to the queue. That timestamp defines the trigger point for quota preemption. Quota preemption will only be triggered if the queue at the point in time of the change is above the new quota. The standard scheduling quota enforcement will immediately enforce the new quota in all other cases and no further preemption actions are needed.

The scheduler will not trigger quota preemption until the delay has passed. If at that point in time the queue usage has dropped below the quota set, no actions will be taken. The quota preemption tracking information will be cleaned up in that case.

To prevent multiple quota changes from impacting each other quota preemption works top down in the queue hierarchy. If a change of a quota has triggered preemption on a queue none of the children of that queue will be able to trigger quota preemption. This prevents complex victim selection interactions if multiple changes are made.

Victims for quota preemption can come from any queue below the queue that triggered the quota preemption. Quota preemption follows the same rules for victim selection as normal preemption. It cannot cause a queue to go below its guaranteed, allocations are sorted based on priority and can opt out. See the description in the preemption documentation for details.

An example configuration turning on quota preemption and setting a delay of 15 minutes on the "prod" queue:

partitions:
  - name: default
    preemption:
      quotapreemptionenabled: true
    queues:
      - name: root
        queues:
          - name: prod
            parent: false
            resources:
              max:
                {memory: 10T, vcore: 1000}
            properties:
              quota.preemption.delay: 15m

Dynamic Queues

Dynamic queues do not support quota preemption.

Inheritance

The current configuration does not support inheritance of the quota.preemption.delay value. YUNIKORN-3208 has been logged to support that functionality.

Triggering preconditions

Quota preemption is triggered when all of the following conditions are met:

1. Quota preemption is enabled at the partition level (quotapreemptionenabled: true)
2. The queue has a non-zero quota.preemption.delay configured
3. The queue is a managed queue (not dynamically created)
4. The queue's allocated resources exceed its configured max resources
5. No quota preemption is currently running for that queue
6. The configured delay has elapsed since the trigger event

If the queue usage drops below the max resources before the delay expires, the preemption is cancelled and the tracking state is cleared.

note

Quota preemption does not check for pending pods. It triggers based solely on allocated resources exceeding the max quota. This means preemption can occur even if no workloads are waiting for resources.

Configuration change scenarios

The following table describes when and how quota preemption is triggered for various configuration changes:

Scenario	Behavior
Max resources decreased	Start time is set to now + delay. Preemption will trigger after the delay if usage still exceeds the new max.
Delay value changed	The start time is adjusted by the difference: startTime += (newDelay - oldDelay).
New allocation pushes queue over max	Start time is set to now + delay. This can occur when a running allocation is added to the queue.
Consecutive quota decreases	The earliest start time is preserved. Subsequent decreases do not reset the timer.
Quota increased back (usage < new max)	Start time is cleared. Preemption is cancelled.
Max resources removed entirely	Start time is cleared. No quota means no limit to enforce.

caution

If preemption has already been triggered and is in progress, a subsequent quota increase or removal cannot stop the running preemption process. Only future preemption cycles are affected.

Restart behavior

The quota preemption start time is held in memory and is not persisted to any external store. When the YuniKorn scheduler restarts:

1. The preemption start time is lost.
2. During recovery, allocations are restored via the resource manager (shim). Each allocation increments the queue's allocated resources.
3. If, after recovery, a queue's usage exceeds its max resources and a non-zero delay is configured, a new start time is set to now + delay.
4. Preemption will only trigger after the full delay has elapsed from the restart time.

This means that a restart effectively resets the preemption timer. Preemption will not fire immediately after a restart even if the delay had already elapsed before the restart.

Recommendations

Quota preemption should be used with care. Using short delays is not recommended. Although no minimum delay is enforced any delay below a minute (60 seconds) should not be used.

If a queue mainly runs service type workloads up and down scaling of deployments should be considered when changing quotas. Workloads will not exit automatically and will be restarted if preempted. Using quota preemption could cause the service to be left in a degraded state. The controller will also try to recreate the workload.

When running batch workloads, the delay should be based on the runtime of the workloads. Preempting workloads should be a last resort. A workload that finishes automatically lowers the queue usage and will not require to be re-run. Preempted workloads have already used resources and will be more expensive overall.

tip

For consistency: until inheritance is provided setting a delay on a parent queue should not be set unless all children below it are also updated with the same delay.