Horizontally scaling consumers with Kubernetes -

(This is part of a larger series on scaling autonomous components. If you like, you can start at the beginning.)

Today’s the daaaaaaay! Watch me try to flex my k8s knowledge here.

Okay, from the last time, we learned that scaling Eventide components requires starting the consumers with an identifier and that it makes sense to include the group_size and group_member in that identifier. It’s up to you to figure out how to inject those values into your running component, but here’s how I’ve done it before with k8s.

You can’t do this with a regular Deployment because Deployments create random identifiers for your pods. If you have 2 pods running, you need one to know that it’s instance 0 and the other to know that it’s instance 1. If, say, instance 1 goes down, the pod that replaces it needs to also know that it’s instance 1.

I won’t post the entire yaml file, but we use the StatefulSet construct instead of a Deployment. Within the yaml, you’ll have a metadata object with a name key. Suppose we’re dealing with the account-component , and that will look something like:

# stuff before
metadata:
  # other metadata
  name: account-component

Sibling to metadata you’ll have spec. Spec has a replicas key. We’ll set it to 2 for our example.

spec will also have template, with child spec (yes, repeated), with child containers. containers is an array, and its members will have attribute env, another array. You’ll put whatever env vars you need in there, but for our purposes, we’ll include two, namely GROUP_SIZE and POD_ID. All this will look something like:

# stuff before
metadata:
  # other metadata
  name: account-component
spec:
  # stuff
  replicas: 2
  template:
	  spec:
	    containers:
	      - name: account-component
	        # other properties
	        env:
	          - name: GROUP_SIZE
	            value: '2'
	          - name: POD_ID
	            valueFrom:
	              fieldRef:
	                fieldPath: metadata.name

There’s more that goes into this file than this, of course, but I trust that if you’re going this far, you know enough about k8s to fill in the rest. These are just the essential bits to make it work for our consumer group.

With that in place then, k8s will supply you GROUP_SIZE and POD_ID as env vars, both strings, mind you. POD_ID will take your [metadata.name](http://metadata.name) and append -<n> where n goes from 0 to your number of replicas minus 1.

To use these in your component’s start.rb, you’d do some like:

module AccountComponent
  module Start
    group_size = ENV['GROUP_SIZE'].to_i
    pod_id = ENV['POD_ID']
    group_member = pod_id.split('-').last.to_i
    identifier = "#{group_size}-#{group_member}"

    Consumers::Commands.start("account:command", group_size:, group_member:, identifier:)
    # other consumer...
  end
end

Note that we have to split the POD_ID env var because it is the concatenation of the name and the index. As of this writing, there isn’t a value k8s can supply that is just the index. Maybe once Eventide has taken the world by storm, we can flex our collective might and get them to add that index so we can all scale our components. Also, the GROUP_SIZE is a string, so we have to convert it to an integer.

You may want to extract that fiddling with the env vars into a different file, especially if you need to repeat this pattern, but that’s the gist of using StatefulSets to have concurrent instances of a component running in k8s.

I’ve never been a fan of articles that post things like these yaml snippets without the full context. It’s a fairly large file though. Let me know if you’d like to see the entire example, and I’ll figure out how to post it somewhere.

Get the book!