AWS: Karpenter and SSH for Kubernetes WorkerNodes

What this system really lacks, is access to servers via SSH, without which you feel like… Well, like DevOps, not Infrastructure Engineer. In short, SSH access is sometimes necessary, but — surprise — Karpenter does not allow you to add a key to the WorkerNodes it manages out of the box.

Although, what’s the problem to add in the suggests using either , or “the old school way” — add the public part of the key via

But here, too, we have the connection error — “ SSM Agent is not online ”:

And the policies connected to it — there is nothing about SSM:

Attach the AmazonSSMManagedInstanceCore:

for SSM.

See .

AWS CLI: SessionManagerPlugin is not found

However, after the IAM fix, when connecting from a workstation using the AWS CLI, we can get the “ SessionManagerPlugin is not found ” error:

$ aws --profile work ssm start-session --target i-011b1c0b5857b0d92

SessionManagerPlugin is not found. Please refer to SessionManager Documentation here: http://docs.aws.amazon.com/console/systems-manager/session-manager-plugin-not-found

Install it locally — see the documentation package in AUR:

$ yay -S aws-session-manager-plugin

And now we can connect:

$ aws --profile work ssm start-session --target i-011b1c0b5857b0d92

Starting session with SessionId: arseny-33ahofrlx7bwlecul2mkvq46gy

sh-4.2$

All that’s left is to add it to the automation.

Terraform: EKS module, and adding an IAM Policy

For the Terraform EKS module from Anton Babenko, we can add a policy through the iam_role_additional_policies parameter - see the .

In the 20.0 module, the parameter name has changed — iam_role_additional_policies => node_iam_role_additional_policies, but we are still using version 19.21.0, and the role is added in this way:

...

module "eks" {

  source = "terraform-aws-modules/eks/aws"

  version = "~> 19.21.0"

  cluster_name = local.env_name

  cluster_version = var.eks_version

  ...

  vpc_id = local.vpc_out.vpc_id

  subnet_ids = data.aws_subnets.private.ids

  control_plane_subnet_ids = data.aws_subnets.intra.ids

  manage_aws_auth_configmap = true



  eks_managed_node_groups = {

      ...

      # allow SSM

      iam_role_additional_policies = {

        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"

      }

...

Remove what we did manually, deploy the Terraform code, and check that the Policy has been added:

.

It also requires an agent, which is also installed by default on the Amazon Linux.

For instances on private networks, EC2 Instance Connect VPC Endpoint is required for connection.

SecurityGroup and SSH

Instance Connect through the endpoint requires access to port 22, SSH (as opposed to SSM, which opens a connection through the agent itself).

Open the port for all addresses in the VPC:

EC2 Instance Connect VPC Endpoint

Go to the VPC Endpoints and create an endpoint:

Choose a Subnet — we have most of the resources in us-east-1a, so we’ll use it to avoid unnecessary cross-AvailabilityZone traffic (see

Wait a few minutes for the Active status:

And connect using AWS CLI by specifying --connection-type eice, because the instances are on a private network:

$ aws --profile work ec2-instance-connect ssh --instance-id i-011b1c0b5857b0d92 --connection-type eice

...

[ec2-user@ip-10-0-34-239 ~]$

Terraform: EC2 Instance Connect, EKS, and VPC

For the Terraform, here you will need to add the for SSH access, and create an EC2 Instance Connect Endpoint for the VPC, as in my case we create VPC and EKS separately.

...

module "eks" {

  source = "terraform-aws-modules/eks/aws"

  version = "~> 19.21.0"



  cluster_name = local.env_name

  cluster_version = var.eks_version



  ...

      # allow SSM

      iam_role_additional_policies = {

        AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"

      }

      ...

  }



  node_security_group_name = "${local.env_name}-node-sg"

  cluster_security_group_name = "${local.env_name}-cluster-sg"



  # to use with EC2 Instance Connect

  node_security_group_additional_rules = {

    ingress_ssh_vpc = {

      description = "SSH from VPC"

      protocol = "tcp"

      from_port = 22

      to_port = 22

      cidr_blocks = [local.vpc_out.vpc_cidr]

      type = "ingress"

    }

  }



  node_security_group_tags = {

    "karpenter.sh/discovery" = local.env_name

  }

  ...

}

...

If you created it manually, as described above, then remove the rule from SecurityGroup with SSH and deploy it from Terraform.

For the VPC EC2 Instance Connect Endpoint, I did not find how to do this through Anton Babenko’s module :

resource "aws_ec2_instance_connect_endpoint" "example" {

  subnet_id = module.vpc.private_subnets[0]

  security_group_ids = ["sg-0b70cfd6019c635af"]

}

However, here you need to pass the SecurityGroup ID from the cluster, and the cluster is created after the VPC, so there is a chicken-and-egg problem.

In general, the Instance Connect seems to be a little more complicated than SSM in the automation, because there are more changes in the code, and in different modules.

However, it is a working option, and if your automation allows it, you can use it.

Option 3: the old-fashioned way with SSH Public Key via EC2 User Data

And the oldest and perhaps the simplest option is to create an SSH key yourself and add its public part to EC2 when creating an instance.

The disadvantages here are that it will be difficult to add many keys in this way, and EC2 User Data can sometimes go sideways, but if you need to add only one key, a kind of “super-admin” in case of emergency, then this is a perfectly valid option.

Moreover, if you have a VPN to the VPC (see :

resource "kubectl_manifest" "karpenter_node_class" {

  yaml_body = <<-YAML

    apiVersion: karpenter.k8s.aws/v1beta1

    kind: EC2NodeClass

    metadata:

      name: default

    spec:

      amiFamily: AL2

      role: ${module.eks.eks_managed_node_groups["${local.env_name_short}-default"].iam_role_name}

      subnetSelectorTerms:

        - tags:

            karpenter.sh/discovery: "atlas-vpc-${var.environment}-private"

      securityGroupSelectorTerms:

        - tags:

            karpenter.sh/discovery: ${local.env_name}

      tags:

        Name: ${local.env_name_short}-karpenter

        environment: ${var.environment}

        created-by: "karpneter"

        karpenter.sh/discovery: ${local.env_name}

      userData: |

        #!/bin/bash

        mkdir -p ~ec2-user/.ssh/

        touch ~ec2-user/.ssh/authorized_keys

        echo "${var.karpenter_nodeclass_ssh}" >> ~ec2-user/.ssh/authorized_keys

        chmod -R go-w ~ec2-user/.ssh/authorized_keys

        chown -R ec2-user ~ec2-user/.ssh       

  YAML



  depends_on = [

    helm_release.karpenter

  ]

}

If you are using not Amazon Linux, then change the ec2-user to the desired one.

Important Note : Keep in mind that changes to EC2NodeClass will recreate all instances, and that your services are configured for stable operation, see )

Originally published at RTFM: Linux, DevOps, and system administration.