servala/documentation
5
1
Fork 0
Code Issues Pull requests 1 Packages 1 Activity Actions

document decision for talos

Browse source
This commit is contained in:
Tobias Brunner 2025-12-08 17:05:39 +01:00
parent bd103e17fe
commit f5edfc5364
Signed by: tobru
SSH key fingerprint: SHA256:kOXg1R6c11XW3/Pt9dbLdQvOJGFAy+B2K6v6PtRWBGQ
1 changed files with 32 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

35
docs/ADRs/adr002.md
View file

@ -8,13 +8,16 @@ author: Marco De Luca and Tobias Brunner
## Context and Problem Statement
TODO
We're building a multi-CSP platform that runs VSHN AppCat workloads across different cloud providers. We need to decide how to provision and manage Kubernetes clusters on each CSP. The choice affects operational complexity, cost structure, consistency across environments, and our ability to support AppCat reliably.
Each CSP offers their own managed Kubernetes service (EKS, AKS, GKE, SKS, etc.), but these services differ significantly in behavior, versioning, available features, and upgrade paths. For a platform like Servala, where predictable behavior and consistency are essential for AppCat development and operations, these differences create friction.
## Considered Options
We evaluated two options for running Kubernetes on each CSP: using the CSP's managed Kubernetes offering or deploying our own Kubernetes distribution on top of their compute layer (e.g., Talos Linux). We explicitly decided not to use OpenShift or OKE, mainly due to their high cost, added complexity, and the amount of bloat they introduce for a platform that only needs to run hosted AppCat workloads. Lower infrastructure cost directly benefits Servala and allows us to offer more competitive pricing.
We evaluated two main options for running Kubernetes on each CSP:
For Servala, consistency across CSPs and predictable behavior for AppCat are essential. Managed Kubernetes offerings differ significantly between providers, resulting in fragmentation and making AppCat development, testing, and support more difficult. A BYO Kubernetes approach gives us full control over versions, components, and security defaults, enabling a standardized setup across all CSPs.
1. using the CSP's managed Kubernetes offering
1. deploying our own Kubernetes distribution on top of their compute layer (e.g., Talos Linux).
The table below summarizes the main differences we identified.
@ -33,6 +36,32 @@ The table below summarizes the main differences we identified.
| :lucide-x: Cope with infrastructure | |
| :lucide-x: Manage whole stack | |
Based on this, BYO Kubernetes is a clear winner.
### Kubernetes distributions
We evaluated the following Kubernetes distributions:
**OpenShift**: Too expensive due to subscription costs and high infrastructure requirements. Adds significant complexity and bloat for a platform that only needs to run hosted AppCat workloads. The overhead doesn't justify the benefits for our use case.
**Rancher**: Past experience with Rancher has been negative. The additional management layer introduces complexity and potential points of failure that we want to avoid.
**k3s**: Lightweight and easy to deploy, but lacks full integration with the underlying operating system. We would still need to manage a traditional Linux distribution separately, which adds operational burden.
**Talos Linux**: Purpose-built for Kubernetes with an immutable, API-driven design. No SSH, no shell, minimal attack surface. The OS and Kubernetes are managed as a single unit with declarative configuration. Produces consistent behavior across all environments.
## Decision
We're choosing a BYO Kubernetes approach using Talos Linux across all CSPs.
Talos Linux provides an immutable, API-driven operating system purpose-built for Kubernetes. It eliminates SSH access, uses a declarative configuration model, and produces identical cluster behavior regardless of where it runs. This gives us the consistency and control we need without the operational burden of managing traditional Linux distributions.
Each CSP will run Talos Linux on their compute layer (usually virtual machines). We control the Kubernetes version, component configuration, security defaults, and upgrade cadence. The same cluster configuration works everywhere, which simplifies AppCat development, testing, and support.
### Consequences
The operational model shifts from consuming managed services to managing infrastructure. We take on responsibility for Kubernetes upgrades, security patches, and cluster lifecycle management. This requires investment in automation and tooling, but we already have experience and concepts in place from previous work.
In return, we get full control over our platform. AppCat can be developed and tested against a single, predictable Kubernetes environment instead of accounting for differences between different flavors of Kubernetes distributions. Troubleshooting becomes easier because cluster behavior is consistent. Cost becomes predictable and tied to compute resources rather than per-cluster service fees.
The security posture improves. Talos Linux has no SSH, no shell, and a minimal attack surface. We define exactly what runs on the nodes. There are no surprises from CSP-specific components or automatic updates we don't control.