Deploying AI at Scale
Containers. Microservices. Kubernetes. Where to begin?
Containers and Microservices have become the most popular way for new software deployments, and for application re-factoring, such as twelve factor apps.
At Modzy, we use containers and Kubernetes for our entire platform. This blog covers choices, lessons learned, and how Modzy uses Kubernetes — and what that means for our customers.
Why Kubernetes?
Choice isn’t a huge factor when it comes to container orchestrators. Kubernetes, Hashicorp Nomad, Docker Swarm, and Marathon/Mesos are the primary tools in use today. In addition to cloud provider managed offerings such as Elastic Kubernetes Service (EKS) from Amazon and Azure Kubernetes Service (AKS) from Microsoft Azure, companies such as Rancher and Red Hat/IBM also have their own Kubernetes offerings. These companies contribute heavily to the open source community, which is a bonus.
So, why would you use Kubernetes vs. another solution? There are a number of factors we considered.
First, Kubernetes is the defacto orchestrator, with the largest open source community out of all the orchestrators. It is also the most supported in terms of vendors, features, and contributors. Kubernetes also scales very well, with good documentation, including potential issues that a user might encounter based on scale. While Nomad and Mesos also scale well, they lack Kubernetes’ open source contribution and the documentation. Additionally, out of all the orchestration tools, Kubernetes has the widest variety of options for deployment methods, tools, cloud provider support, tutorials, docs, guides, etc.
While deploying Kubernetes has its own challenges, the community support, managed offerings, additional tools, and vendor choices provided for the deployment process are much more robust than those of other orchestration tools. Deployment spans a wide range industry verticals and customer use cases, and the managed Kubernetes offering offloads daily maintenance from our DevOps and SRE teams.
Kubernetes offers flexible deployment options, with many managed distributions for software deployment. Kubernetes offers support for everything from large datacenters, all the way down to Raspberry Pis. Kubernetes is open source, by default — meaning no licensing required. From a security perspective, Kubernetes has robust open source tooling, as well as the option to add additional security via vendor tools. Tools like Vault can be used to secure credentials, secrets, and more. Open source products such as Falco does runtime security, allowing you to detect potential threats in real time.
Application Features and Integrations
In addition to the orchestration features, Kubernetes offers a number of application features and integrations.
- Application Troubleshooting — How do we log, monitor, and trace? Kubernetes has many open source options for this as do vendor paid for offerings. Opensource offerings from Jaeger, Open Telemetry, Zipkin, Open Tracing, and more.
- Application Deployment — Kubernetes offers multiple deployment options such as using Helm, Kustomize, and more. Matching your deployment requirements to application requirements becomes easier and requires less customization.
- Application Integrations/Support — Does the orchestrator support your application’s need for integrations, libraries, and more? What needs to be added to your application to work in the orchestrator, if anything? Are there plugins available or integrations? Kubernetes offers all of these and has the ability to buy or build integrations, as needed.
Kubernetes in the real world
We’ve learned a lot about Kubernetes in AI and ModelOps (MLOps) in the last year at Modzy. We realized that AI and MLOps require good documentation for deployment purposes. And, scaling is different between a Web frontend and an AI platform. Model deployments require additional thinking around scale, resource use, and security.
Other best practices we follow for consideration:
- “Stateless-ness” — This is one of the core tenets of microservices — and Modzy is designed with this same mindset. This makes for easier scaling and independently updating services.
- “Dogfooding” — We use what we ship. Period. From dev to release branches, we have been able to catch errors from all microservices, our deployment model, and more using this approach. Modzy’s data science team uses our product to run their models.
- Choice — Modzy needs Kubernetes to run — that’s it. We do not want our customers to be locked into a cloud provider, distribution, or company to use our product. Modzy is here to enable your models, and align with your current cloud or infrastructure of choice.
- Scaling — We use automated scaling in a few different parts of Modzy. Our AWS components have their autoscaling enabled in their AWS configs. On the Kubernetes side, we utilize horizontal pod autoscalers (HPA). HPA allows Modzy to scale horizontally — which is something required when running dynamic workloads at scale. Our platform does not require manual scaling, allowing our customers to focus on models, not cluster management.
- Security — Kubernetes has a large vendor community supporting security. This ranges from open source to paid for products. Runtime security, storage encryption, auditing, and more are all available. We implement security in all aspects of our platform.
What are your thoughts around scaling AI? Is it something you’ve done? We’d love to chat further and share our lessons learned with you.
