Use K8s Namespace in YuniKorn
In K8s, user can setup namespace with resource quotas to limit aggregated resource consumption in this namespace. The validation of namespace resource quotas is handled in api-server directly, therefore YuniKorn simply honors the quotas like the default scheduler.
It is not mandatory to setup YuniKorn queues with respect of namespaces. However, in practice, it makes more sense to do so. Namespace is often used to set a cap for resource consumptions per user-group/team, YuniKorn queue is also meant to divide cluster resource into multiple groups. Let's go through an example.
1. Setup namespace
Create the namespace
2. Setup YuniKorn queues
QueueMaxResource <= NamespaceResourceQuotaRequests
3. Mapping applications to queues & namespace
In a pod spec
Now submit another application,
pod will not be able to submitted to api-server, because the requested cpu
200m + used cpu
300m which exceeds the resource quota.
For compatibility, we should respect namespaces and resource quotas.
Resource quota is overlapped with queue configuration in many ways,
for example the
requests quota is just like queue's max resource. However,
there are still a few features resource quota can do but queue cannot, such as
limits. The aggregated resource from all pods in a namespace cannot exceed this limit.
- Storage Resource Quota, e.g storage size, PVC number, etc.
- Object Count Quotas, e.g count of PVCs, services, configmaps, etc.
- Resource Quota can map to priority class.
Probably we can build something similar to cover (3) in this list. But it would be hard to completely support all these cases.
But currently, setting applications mapping to a queue as well as a corresponding namespace is over complex. Some future improvements might be:
- Automatically detects namespaces in k8s-shim and map them to queues. Behind the scenes, we automatically generates queue configuration based on namespace definition. Generated queues are attached under root queue.
- When new namespace added/updated/removed, similarly to (1), we automatically update queues.
- User can add more configuration to queues, e.g add queue ACL, add child queues on the generated queues.
- Applications submitted to namespaces are transparently submitted to corresponding queues.