Kobai’s Lean Platform Team and Cutting Edge Practices

In this series, we’ll go over how the platform engineering team at Kobai is able to handle cutting edge DevOps implementations like ephemeral environments, manage to wrangle our cloud spending to just a few thousand dollars a month for over 20 environments, and create automated regression testing suites — all while maintaining a lean team.

Preamble

I have a couple of key opinions about this field. I’ll reference DevOps engineers, platform engineers, and site reliability engineers (SREs), but in reality given how vague each role actually is and the size of this team, I’ll be using them almost interchangeably. It’s not about the title of the role, but what the individual does — and we do a bit of everything.

DevOps isn’t about doing everything between development and operations but by doing everything from development to operations. Inclusive of the ends (if you’re a fan of set theory [Development, Operations]). What we do isn’t just automation or just managing infrastructure, but creating a culture that fosters engineering by enabling everyone in our organization to do their best work.

This series is going to assume familiarity with the tools and concepts mentioned and will dive into how we’re using them, what problems we’re solving with them, and any “gotchas”s that we ran into.

Who are we?

I’m Varun, one of the platform engineers on this team. I’ve dabbled in a lot of different things, physics, astronomy, robotics, muay thai and cars — to name a few. I’m always a fan of taking on new ideas and seeing how they can be applied to our current challenges even if they’re not apparent at first.

My peer Akilan is an avid player of both badminton and pickleball, bringing the same energy and precision he shows on the court to his work. Brilliant at navigating Kubernetes clusters like the back of his hand, he leverages that expertise to drive our cost-saving initiatives with remarkable efficiency and insight.

We’re both extremely passionate about what we do and always strive for elegant and comprehensive solutions. A big focus on what we do though is solving tangible problems. There’s no reason to solve problems that might exist. Solve problems that do exist and you’ll be providing value. You can think of this as a version of YAGNI that’s focused on our way of working.

Why’d we even bother doing this stuff?

Cost Savings

As a startup, we are always aiming to reduce our cloud costs to allocate to other, more impactful parts of the business. I’m sure that’s true for all businesses, but as everyone knows, that AWS bill can really jump up at you if you’re not careful.

Developer enablement

Making the developers’ lives easier allows them to work more effectively and with less stress. It’s always worth making the lives of your peers easier, but from a business perspective, this also allows them to not worry about limitations when they’re working — they can just do the work.

Product Stability

Ensuring the product is stable directly affects our customers. As DevOps engineers, we have a stake in everything from development to operations, so detecting defects and how quickly we can test and release a patch are also in our purview.

Foundational DevOps Initiatives

Given all the above, we need a comprehensive platform to manage all this and a consistent pattern to deal with it all. Rome wasn’t built in a day!

What did we do?

To give a bit of an overview to supply context on our solutions, it’s important to know that the Kobai platform is a set of services hosted on Kubernetes. We’re able to set up the platform on any Kubernetes platform (AKS, EKS, GKE, etc.) and this is the core of what we do. Internally, most of our environments are based on AWS’s EKS.

Our DevOps stack consists of Terraform, Github Actions, Snyk, FluxCD with Helm and Kustomize. We use Grafana, Loki, and Prometheus for monitoring.

Following the goals from above we introduced certain technologies to help address some of these problems. In this series, we’ll dive deeper into each of the solutions and what we did for it

Foundational DevOps Initiatives

Obviously, you don’t get things like Ephemeral environments working off of a weak foundation, so we’ll go over some of the foundational work that we started with, including choosing the right CI provider, how we use that CI provider (spoiler alert: it’s Github Actions) to its fullest potential, setting up monitoring and how we manage our Kubernetes environments at scale.

Cost Savings

Cost savings happen in tandem with almost all the other work that we do. Everyone at Kobai has a very large degree of ownership, but especially as the arbiters of the infrastructure (sounds like a movie title) we have to keep in mind a lot of things that others would take for granted.

We’ll talk about how we use Karpenter to optimize our node usage and how we automatically scale our internal environments so we can run clusters at minimal costs.

Developer Enablement

Using our strong foundations, we’re able to create some really interesting, cutting-edge tools for our developers to use like Ephemeral environments. Helping our developers maintain their own code bases is also something that we’re able to help with tools like super-linter.

Product Stability

We’ll round out everything with our implementation on regression testing and how we manage this for our frontend and our backend to ensure stability in our product.

Conclusion

We’ll be releasing additional articles in the weeks to dive deeper into each of these topics and get into the nitty gritty. If you’re also doing DevOps for a startup, I hope that you can get a sense of what’s important when working with a small team. Contrary to popular belief, you can get a lot done with less.

Happy New Years and I hope to see you in the next article!

Feel free to follow us on LinkedIn or checkout our website.