Two epic migrations, one decade of scars — hardware → AWS → EKS
Over four years at Flo Health I led two migrations that most companies only survive once. First: from racked, on-prem hardware to AWS. Second: from EC2 to Amazon EKS. One company. Two generational shifts. Here's the unvarnished version.
Migration one — bare metal to AWS
Flo was running on physical hardware in colos when I joined the infra org. The product was growing faster than the rack rollout. Capacity planning had become a 6-month lead-time problem. Moving to AWS wasn't an optimization — it was the difference between scaling with demand or capping it.
What we got right
We treated it as a one-way door. No bi-directional replication, no "we'll keep colo as DR for a year." Every service that moved, moved for good. Dual-running drains leadership attention and blames every incident on "the migration" for months.
We picked a landing zone early. Multi-account AWS Organization, centralised identity, Terraform baseline, tagging standard, network plan. The week we finalised the landing zone was the week the migration actually started. Everything before that was yak shaving.
We migrated stateful services last. Stateless workloads first (API, workers). Then caches. Then primary stores. Never ever migrate a primary database before the application layer is already living happily on the other side — the pressure to roll back a stateful cutover will kill you.
What I'd never do again
Underestimate data egress. Moving terabytes out of a colo over the internet looks cheap on the spreadsheet and punishes you on the bill. Use Direct Connect, Snowball, or an ISP deal. Budget 2x whatever the vendor says.
Let every team "re-platform" in their own way. Uniformity beats cleverness. Pick one deployment pattern, one logging shape, one secrets story. Let the exceptions earn their right to exist.
Migration two — EC2 to Amazon EKS
A couple of years later the EC2 estate that had absorbed the colo workloads was creaking. Per-service autoscaling groups. Custom AMIs. Bespoke deploy scripts. Every service a snowflake. The case for Kubernetes wasn't "Kubernetes is cool" — it was "we need one shared substrate, one control plane, one way to deploy, or the platform team will be the bottleneck forever."
What we got right
EKS, not self-managed. I have enormous respect for the teams that run kops or kubeadm at scale. You shouldn't be one of them unless you have a very specific reason. AWS running the control plane buys you back a full-time engineer's worth of attention.
Platform team owned the cluster, product teams owned the workload. Clear split. Platform owns the cluster, base images, network policy, observability, the golden deployment path. Product owns the manifest. Ambiguity here is where every bad Kubernetes rollout starts.
We invested in the paved road first. Before the first production workload moved, the CI template, the manifest template, the on-call runbook, the dashboard template — all existed. Migration velocity is gated by the golden path, not by the migration tooling.
What I'd never do again
Let Helm become a wild west. Give teams one base chart. Do not let twelve teams write twelve different charts for the same pattern. The future self who has to upgrade all of them in parallel will curse you.
Move critical stateful workloads into the cluster early. Run them on managed services (RDS, ElastiCache, MSK) or keep them on EC2 until the cluster has years on it. Postgres in Kubernetes is a solved problem if you have a whole team for it; if you don't, it's an incident waiting to happen.
Underestimate the IAM work. IRSA and pod identity are the only correct answer, and getting every team to migrate off node-level credentials takes twice as long as the rest of the migration combined.
The meta-lesson
The hardest part of any infrastructure migration is never the technology. It's the two years of political capital you need to finish what you started — when the novelty wears off, when half the company has moved and the other half is hostile, when the next shiny thing shows up and everyone wants to skip ahead.
Pick migrations the company has to do or die. Don't pick the ones where "it would be nice to be on Kubernetes." The ones you can justify in a single sentence to the CEO are the ones you'll be allowed to finish.
In public: the slides · the talk (video, RU).
Related: Platform engineering for the AI era · How I've saved $10M+ in AWS spend