Namespace Resource Quota
Use K8s Namespace in YuniKorn
In K8s, user can setup namespace with resource quotas to limit aggregated resource consumption in this namespace. The validation of namespace resource quotas is handled in api-server directly, therefore YuniKorn simply honors the quotas like the default scheduler.
Best practice
It is not mandatory to setup YuniKorn queues with respect of namespaces. However, in practice, it makes more sense to do so. Namespace is often used to set a cap for resource consumptions per user-group/team, YuniKorn queue is also meant to divide cluster resource into multiple groups. Let's go through an example.
1. Setup namespace
Namespace: advertisement
:
Create the namespace
2. Setup YuniKorn queues
Queue: advertisement
:
ensure QueueMaxResource <= NamespaceResourceQuotaRequests
3. Mapping applications to queues & namespace
In a pod spec
Check Quota
Now submit another application,
pod will not be able to submitted to api-server, because the requested cpu 200m
+ used cpu 100m
= 300m
which exceeds the resource quota.
Future Work
For compatibility, we should respect namespaces and resource quotas.
Resource quota is overlapped with queue configuration in many ways,
for example the requests
quota is just like queue's max resource. However,
there are still a few features resource quota can do but queue cannot, such as
- Resource
limits
. The aggregated resource from all pods in a namespace cannot exceed this limit. - Storage Resource Quota, e.g storage size, PVC number, etc.
- Object Count Quotas, e.g count of PVCs, services, configmaps, etc.
- Resource Quota can map to priority class.
Probably we can build something similar to cover (3) in this list. But it would be hard to completely support all these cases.
But currently, setting applications mapping to a queue as well as a corresponding namespace is over complex. Some future improvements might be:
- Automatically detects namespaces in k8s-shim and map them to queues. Behind the scenes, we automatically generates queue configuration based on namespace definition. Generated queues are attached under root queue.
- When new namespace added/updated/removed, similarly to (1), we automatically update queues.
- User can add more configuration to queues, e.g add queue ACL, add child queues on the generated queues.
- Applications submitted to namespaces are transparently submitted to corresponding queues.