AWS CDK for EKS — Kubernetes Manifest Handling
YAML — Build vs. Parse
…Jimmy Ray follows up from a previous post on using AWS CDK to deploy a sample Java application on Amazon EKS, with this post that dives deeper into Kubernetes YAML manifests. From what these are to how to use them with AWS CDK, this is a great post to understand what your options are when managing Kubernetes manifests and the resulting Kubernetes resources that get created. — Ricardo Sueiras, AWS open source news and updates #68
In a previous post, I briefly covered how to use the AWS Cloud Development Kit (CDK) with Java, to build Amazon EKS clusters and associated AWS resources (VPC, Subnets, etc.), and deploy applications to the created cluster. In this post I will look at how CDK can handle Kubernetes YAML manifests.
Manifests
Anyone that has worked with Kubernetes (a.k.a. k8s) has probably also worked with the YAML manifest files. The following example is a YAML manifest for creating a k8s namespace.
apiVersion: v1
kind: Namespace
metadata:
name: read-only
labels:
owner: jimmy
env: dev
app: read-onlyOften, these files are applied to k8s clusters via the kubectl apply command. It is considered a best practice to apply changes to the cluster in this declarative fashion, instead of using the more imperative kubectl create command.
When using the AWS CDK with Java, the KubernetesManifest.Builder can be used to create k8s manifests. The following example creates a k8s manifest, using multiple k8s manifest objects (namespace, deployment, service).
KubernetesManifest.Builder.create(this, "read-only")
.cluster(cluster)
.manifest((List<? extends Map<String, ? extends Object>>) List.of(ReadOnlyNamespace.manifest,
ReadOnlyDeployment.manifest, ReadOnlyService.manifest))
.overwrite(true)
.build();As seen in the code, the KubernetesManifest object is a list (sequence) of map objects. These maps can also contain maps and lists that represent the YAML structure, as seen in the following example.
public final class ReadOnlyNamespace {
public static Map<String, ? extends Object> manifest;
// Get properties object
private static final Properties properties = Config.properties;
static {
manifest = Map.of("apiVersion", "v1",
"kind", "Namespace",
"metadata", Map.of("name", "read-only", "labels",
Map.of("owner", Strings.getPropertyString("labels.owner", properties,
Constants.NOT_FOUND.getValue()),
"env",
Strings.getPropertyString("labels.env", properties,
Constants.NOT_FOUND.getValue()),
"app",
Strings.getPropertyString("labels.app", properties,
Constants.NOT_FOUND.getValue())))
);
}
}When the CDK creates the CloudFormation template for this stack, the k8s manifest is stored in the CloudFormation template as a JSON custom resource. In the following example, the three k8s configs (Namespace, Deployment, Service) are stored in the CloudFormation custom resource, as JSON.
readonlyB1EB06D3:
Type: Custom::AWSCDK-EKS-KubernetesResource
Properties:
ServiceToken:
Fn::GetAtt:
- awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B
- Outputs.EksStackawscdkawseksKubectlProviderframeworkonEvent47AB1AD9Arn
Manifest: '[{"apiVersion":"v1","kind":"Namespace","metadata":{"name":"read-only","labels":{"aws.cdk.eks/prune-c8157df28ab1a464bab539b75e7483fab124b22805":"","owner":"jimmy","env":"dev"}}},{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"name":"read-only","namespace":"read-only","labels":{"aws.cdk.eks/prune-c8157df28ab1a464bab539b75e7483fab124b22805":"","app":"read-only","owner":"jimmy","env":"dev"}},"spec":{"revisionHistoryLimit":3,"selector":{"matchLabels":{"app":"read-only"}},"replicas":3,"strategy":{"type":"RollingUpdate","rollingUpdate":{"maxSurge":10,"maxUnavailable":1}},"template":{"metadata":{"labels":{"app":"read-only","owner":"jimmy","env":"dev"}},"spec":{"securityContext":{"fsGroup":2000},"containers":[{"name":"read-only","image":"public.ecr.aws/r2l1x4g2/go-http-server:v0.1.0-23ffe0a715","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false,"runAsUser":1000,"readOnlyRootFilesystem":true},"resources":{"limits":{"cpu":"200m","memory":"20Mi"},"requests":{"cpu":"100m","memory":"10Mi"}},"readinessProbe":{"tcpSocket":{"port":8080},"initialDelaySeconds":5,"periodSeconds":10},"livenessProbe":{"tcpSocket":{"port":8080},"initialDelaySeconds":15,"periodSeconds":20},"ports":[{"containerPort":8080}],"volumeMounts":[{"mountPath":"/tmp","name":"tmp"}]}],"volumes":[{"name":"tmp","emptyDir":{}}]}}}},{"kind":"Service","apiVersion":"v1","metadata":{"name":"read-only","namespace":"read-only","labels":{"aws.cdk.eks/prune-c8157df28ab1a464bab539b75e7483fab124b22805":"","app":"read-only","owner":"jimmy","env":"dev"}},"spec":{"ports":[{"port":80,"targetPort":8080,"protocol":"TCP","name":"http"}],"type":"LoadBalancer","selector":{"app":"read-only"}}}]'
ClusterName:
Ref: cdkeksDB67CD5C
RoleArn:
Fn::GetAtt:
- cdkeksCreationRole8B89769F
- Arn
PruneLabel: aws.cdk.eks/prune-c8157df28ab1a464bab539b75e7483fab124b22805
Overwrite: true
DependsOn:
- cdkeksKubectlReadyBarrierA155E2C2
UpdateReplacePolicy: Delete
DeletionPolicy: Delete
Metadata:
aws:cdk:path: EksStack/read-only/Resource/DefaultParsing YAML
Depending on the complexity of the k8s object configuration, these KubernetesManifest objects can be quite complex to build in Java as well. If the k8s configs already exist as maintained YAML files, then SnakeYAML can be used to parse the raw YAML and create the list-of-map-of objects needed by the KubernetesManifest object.
In the following example, I have stored the raw YAMLs as static text-blocks in a final Java class. Java text-blocks are a relatively new addition to Java that resemble the triple-double quoted strings from the Groovy language. Text-blocks are literal, multiline, strings that preserve whitespace; they are ideal for storing YAML. These YAMLs could also be stored in external files and read via the Java FileReader or InputStream classes.
/**
* Contains static text blocks of K8s manifest YAMLs
*/
public final class Yamls {
public static final String namespace = """
apiVersion: v1
kind: Namespace
metadata:
name: read-only
labels:
owner: jimmy
env: dev""";
public static final String deployment = """
apiVersion: apps/v1
kind: Deployment
metadata:
name: read-only
namespace: read-only
labels:
app: read-only
owner: jimmy
env: dev
spec:
revisionHistoryLimit: 3
selector:
matchLabels:
app: read-only
replicas: 3
...To parse the YAMLs, I use SnakeYAML. In the following example, I wrote a utility class with a static method to use the non-thread-safe Yaml object to parse strings.
package io.jimmyray.utils;
import io.jimmyray.aws.cdk.manifests.Yamls;
import org.yaml.snakeyaml.Yaml;
import java.util.Map;
/**
* Provides helper methods for SnakeYaml parser
*/
public class YamlParser {
public static void main(final String[] args) {
Map<String, Object> out = YamlParser.parse(Yamls.namespace);
System.out.println(out);
out = YamlParser.parse(Yamls.deployment);
System.out.println(out);
out = YamlParser.parse(Yamls.service);
System.out.println(out);
}
/**
* Parses YAML String and returns Map
* @param in
* @return
*/
public static Map<String, Object> parse(final String in) {
Yaml yaml = new Yaml();
return yaml.load(in);
}
/**
* Parses YAML String of multiple objects and returns Iterable
* @param in
* @return
*/
public static Iterable<Object> parseMulti(final String in) {
Yaml yaml = new Yaml();
return yaml.loadAll(in);
}
}The returned object from the parse method can be used in the KubernetesManifest list object, using the List.of() method, as seen in the following snippet.
KubernetesManifest.Builder.create(this, "read-only")
.cluster(cluster)
.manifest(List.of(YamlParser.parse(Yamls.namespace),
YamlParser.parse(Yamls.deployment),
YamlParser.parse(Yamls.service.replace("<REMOTE_ACCESS_CIDRS>",
Strings.getPropertyString("remote.access.cidrs", properties, "")))))
.overwrite(true)
.build();Note that I am replacing the <REMOTE_ACCESS_CIDRS> string with a resolved property. This inserts comma-separated CIDR ranges into the service.beta.kubernetes.io/load-balancer-source-ranges annotation. This results in adding inbound IP rules for the security group created when the load balancer is created.
annotations:
service.beta.kubernetes.io/load-balancer-source-ranges: <REMOTE_ACCESS_CIDRS>
The stack created, using the preceding KubernetesManifest object, uses the CDK assets, installed in the CDKToolkit stack during the cdk bootstrap operation, to make the kubectl calls to the target cluster via AWS Lambda. The result is the objects seen in the following kubectl command.
kubectl -n read-only get all
Pulling Raw Files from Hosted Git Repositories
Another way to harvest YAML is to pull raw files directly from hosted Git repositories, such as GitHub. The following WebRetriever method does just that, using the URLConnection and BufferedReader objects. Files are downloaded and stored in Java text-blocks.
/**
* Get raw file from GitHub
* @param in
* @return
* @throws IOException
*/
public static String getRaw(final String in) throws IOException {
URL url;
String file = """
""";
url = new URL(in);
URLConnection uc;
uc = url.openConnection();
uc.setRequestProperty("X-Requested-With", "Curl");
BufferedReader reader = new BufferedReader(new InputStreamReader(uc.getInputStream()));
String line = null;
while ((line = reader.readLine()) != null)
file = file + line + "\n";
return file;
}Once downloaded, the YamlParser.parseMulti(…) method parses the multiple manifests in the downloaded file.
/**
* Parses YAML String of multiple objects and returns Iterable
* @param in
* @return
*/
public static Iterable<Object> parseMulti(final String in) {
Yaml yaml = new Yaml();
return yaml.loadAll(in);
}Then parsed YAML is returned as an Iterable containing each individual YAML manifest, that can then be iterated over and added to the KubernetesManifest.Builder.
/*
* Parse multiple docs in same string
*/
String yamlFile = null;
/*
* Try to get the YAML from GitHub
*/
try {
yamlFile = WebRetriever.getRaw(Strings.getPropertyString("ssm.agent.installer.url", properties, ""));
} catch (IOException e) {
e.printStackTrace();
}
if (yamlFile == null) yamlFile = Yamls.ssmAgent;
if (null != yamlFile && !yamlFile.isBlank()) {
Iterable<Object> manifestYamls = YamlParser.parseMulti(yamlFile);
List manifestList = new ArrayList();
for (Object doc : manifestYamls) {
manifestList.add((Map<String, ? extends Object>) doc);
}
KubernetesManifest.Builder.create(this, "ssm-agent")
.cluster(cluster)
.manifest(manifestList)
.overwrite(true)
.build();
}In the above example, the YAML is retrieved and used for an AWS Systems Manager (SSM) agent installer k8s Daemonset resource that will connect the nodes to the SSM for fleet management and other operational use cases. Using the SSM agent will eliminate the need for SSH keys for nodes.
Sequencing Configurations
According to kubernetes.io, managing multiple and related configurations in the same file (or even directory) is considered a best practice. In the AWS CDK, multiple and related configurations should be managed within the same KubernetesManifest.Builder object. Using multiple KubernetesManifest.Builder objects to manage related k8s resources, such as those with dependencies, can result in non-deterministic behavior (race-conditions) that cause the kubectl apply calls to fail. Configurations are stored in Java lists, that are ordered sequences, so let the code manage the dependencies and order of operations.
Summary
With the AWS CDK, there are multiple ways to handle YAML and resultant object graphs that are used to create Kubernetes resources.
The example code for this post can be had from this GitHub repo.
