Vendor-Agnostic Reconfiguration of Kubernetes Clusters in Cloud Federations

Kubernetes (K8s) defines standardized APIs for container-based cluster orchestration so it becomes possible for application managers to deploy their applications in a unified manner across different cloud providers. A practical problem is however feature incompatibility between different K8s vendors, who offer commercial K8s products based on the open-source K8s distribution. A large number of documented features in this open-source distribution are optional features that are turned off by default, but can be activated by setting specific combinations of parameters and plug-in components in configuration manifests for the K8s control plane and worker node agents. However, none of these configuration manifests are standardized, giving K8s vendors the freedom to hide the manifests behind a single, more restricted, and proprietary customization interface. Therefore some optional K8s features cannot be activated consistently across K8s vendors and applications that require these features cannot be run on those vendors. In this paper we present a unified, vendor-agnostic feature management approach that bypasses the proprietary customization interface of K8s vendors in order to consistently activate optional K8s features across a federation of clusters hosted by different Kubernetes vendors. We describe vendor-agnostic reconfiguration tactics that are already applied in industry and cover a wide range of optional K8s features. Based on these tactics, we design and implement an autonomic controller for declarative feature compatibility management across a cluster federation. We found that the features configured through our vendor-agnostic approach have no impact on application performance when compared with a cluster where the features are configured using the configuration manifests of the open-source K8s distribution. Moreover, the maximum time to complete reconfiguration of a single feature is within 100 seconds, which is 6 times faster than using proprietary customization interfaces of mainstream K8s vendors such as Google Kubernetes Engine. However, there is a non-negligible disruption to running applications when performing the reconfiguration; this disruption impact does not appear using the proprietary customization methods of the K8s vendors. Therefore, our approach is best applied in the following three use cases: (i) when starting up new K8s clusters, (ii) when optional K8s features of existing clusters must be activated as quickly as possibly and temporary disruption to running applications can be tolerated or (iii) when proprietary customization interfaces do not allow to activate the desired optional feature.