<div dir="ltr"><div dir="ltr">I'm assuming on-prem k8s vs EC2[1] with automation managing deployment, configuration and scaling of nodes/containers. Your decision will be different from other folks', but here are what I would consider the primary axes of comparison:<br>- Cost<br> - Hardware<br> - k8s: High up-front, but amortized over time. Possibly includes support contracts.<br> - ec2: Per instance based on time used on an ongoing basis[2]. And there is not an upper bound if anything gets misconfigured, so best be very careful about putting in limits.<br> - Infra (network, storage, AC, etc)<br> - k8s: Depends on your pre-existing infrastructure.<br> - ec2: Ongoing cost per MB for both network and persistent storage.</div><div> - Time</div><div> - k8s: Much longer initial setup time. In addition, ops/eng folks will need to ramp up on the necessary skills to properly manage the cluster *and* deploy services to it.</div><div> - ec2: You can be up and running in a few hours. Ops/eng setup and testing can be done via GUI initially, with automation and "config as code" deployment coming later.<br></div><div dir="ltr">- Scalability<br> - Scaling up<br> - k8s: For nodes, it's based on lead time for hardware acquisition and installation. For containers (assuming adequate compute/storage is available), from dozens of seconds to minutes depending on application complexity.<br> - ec2: On the order of single-digit minutes (assuming proper auto-scaling) or low double-digit minutes (if done manually).<br> - Scaling down<br> - k8s: Your workloads can be managed dynamically, but your nodes are assumed to stay in place until they are decommissioned and removed once the hardware reaches end of life.<br> - ec2: On the order of minutes.<br>- Ongoing maintenance<br> - k8s: The versions of the OS and of Kubernetes will need to be periodically updated. Generally done by completely rebuilding a node, then re-introducing it to the running cluster (if the k8s upgrades allow it); or taking several nodes, doing upgrades, creating a new cluster with them, migrating workloads, then upgrading and migrating individual nodes when demand is low enough to allow it.<br> - ec2: The suggested method is to deploy new instances with software/OS/"hardware" upgrades, then add them to the service rotation while removing old instances. Once everything is net new and there are no problems, old ec2 instances are shut down/deleted.<br>- Management Complexity<br> - For both options, this will depend on your comfort with the platform and will require an initial learning period as well as refreshing knowledge on an ongoing basis.<br> - k8s: Will require more effort/knowledge to properly maintain and provide supporting services (IAM, Storage, etc)<br> - ec2: Lets you avoid handling complexity for things you don't want to tackle right away. But you will pay for it on an ongoing basis<br><br>High level:<br>- If I'm part of an existing org; the costs (time, money, space, etc) are small compared to the overall budgets; the org is willing to invest in the operations/engineering staff; and the ops/eng staff are interested/eager to learn.... I'd go with k8s. If I could make use of k8s to run other existing services in a more efficient fashion[3], this would tip the scales toward k8s even more.<br>- If I'm building a small startup and this is my primary app, I'd deploy to the cloud. Having a lower initial investment a) should everything go pear shaped or b) if I need to devote resources to another part of the business; would be my priority.<br>- In most/all other cases, the cost-benefit analysis would depend on what, if any, available resources are there; what is the requirements and criticality of the service(s) to be provided; what is the priority (and, therefore, organizational support) for doing what needs done to stand up the service(s).<br><br>In general on-prem kubernetes has a higher up-front cost in terms of time, money, complexity, and knowledge; and EC2 has much lower initial investment but you will always be paying for what you use as long as you use it. It's similar to whether you buy or rent your living abode. If you're in a position to commit to the long-term, buying has a significant upside. If you're not sure you'll be in the same city in five years, renting allows the required flexibility, even if you accrue no equity during the time you rent.<br><br>Just some idle thoughts on how I've come to think about these decisions.<br><br><br>[1]: Substitute GCP, Azure, or any other cloud provider of choice here.<br>[2]: You can get discounts for reserved instances, but the complexity of determining the optimum choice has significant overhead. <br>[3]: In case you didn't know, AWS grew from Amazon trying to make more money from all the compute resources they built for peak times during the other 98% of the year.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 28, 2024 at 9:41 PM Leam Hall via Ale <<a href="mailto:ale@ale.org">ale@ale.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p dir="ltr">Dozens to hundreds. EC2 can be created via automation, and autoscaling can expand as needed, right?</p>
<br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 28, 2024, 20:01 DJPfulio--- via Ale <<a href="mailto:ale@ale.org" target="_blank">ale@ale.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 8/28/24 17:53, Leam Hall via Ale wrote:<br>
> <br>
> What am I missing?<br>
<br>
Do you have 4 instances or 200+ instances?<br>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" rel="noreferrer" target="_blank">Ale@ale.org</a><br>
<a href="https://mail.ale.org/mailman/listinfo/ale" rel="noreferrer noreferrer" target="_blank">https://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" rel="noreferrer noreferrer" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
</blockquote></div>
_______________________________________________<br>
Ale mailing list<br>
<a href="mailto:Ale@ale.org" target="_blank">Ale@ale.org</a><br>
<a href="https://mail.ale.org/mailman/listinfo/ale" rel="noreferrer" target="_blank">https://mail.ale.org/mailman/listinfo/ale</a><br>
See JOBS, ANNOUNCE and SCHOOLS lists at<br>
<a href="http://mail.ale.org/mailman/listinfo" rel="noreferrer" target="_blank">http://mail.ale.org/mailman/listinfo</a><br>
</blockquote></div><br clear="all"><br><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Dylan Northrup<br>"Now that is a powerful cat"<br></div> - Aesop Rock<br></div></div></div>