My view on this is a bit different. I see Kubernetes as the abstraction layer on top of the cloud providers. In the last few years I have set up multiple k8s clusters for clients who specifically do not want to be locked in to a certain cloud provider. Once the software is running on top of k8s it is easy to switch cloud providers without changing the software.
Switching to another cloud provider this way is trivial and usually only involves changing the Terraform configuration to setup a k8s cluster on another cloud. All k8s-specific config/deploy files can be reused on the new cluster.
This of course only works if (as you suggest) you stay away from cloud-specific services (SQS, aurora, ECS, S3) and run everything in-cluster, or use managed services that are available on multiple providers (Postgresql via RDS, or Digital Ocean managed Postgres, Cloud SQL on GCP)
> Switching to another cloud provider this way is trivial
Based on my limited experience, I highly doubt this. Have you actually deployed cross-cloud k8s setups, or is this merely a theoretical statement on your part? Deploying to another cloud provider brings a whole new universe of failure modes and auth quirks, let alone migration and switch-over woos.
I’ve done a couple cluster switches from k8s on AWS to other providers like Digital Ocean and GCP. As far as I recall we had no issues and one of those was done in about an hour were most of the time was spent waiting for pg_dump/restore.
Note that most of these were not production clusters so switch-over was just data restore and DNS changes.
I build clusters from the start to not use cloud-specifics where possible and all cloud-specific configuration is on the cluster edges in terraform which you have to rewrite anyway when switching clouds.
Auth things like IAM permissions are not an issue if everything is “in cluster” and auth/permissions are checked there.
Most of these deployments consist of several application servers, PG databases, redis, rabbitmq etc
Switching to another cloud provider this way is trivial and usually only involves changing the Terraform configuration to setup a k8s cluster on another cloud. All k8s-specific config/deploy files can be reused on the new cluster.
This of course only works if (as you suggest) you stay away from cloud-specific services (SQS, aurora, ECS, S3) and run everything in-cluster, or use managed services that are available on multiple providers (Postgresql via RDS, or Digital Ocean managed Postgres, Cloud SQL on GCP)