How to Create an AWS MSK Kafka Cluster via Terraform: A Step-by-Step Guide

Creating an AWS Managed Streaming for Apache Kafka (MSK) Cluster via Terraform is a great way to automate and manage infrastructure efficiently. In this blog post, I’ll walk you through the steps of setting up an MSK cluster using Terraform, ensuring that your Kafka infrastructure is easy to deploy and maintain.

Prerequisites:

• AWS Account: You must have an AWS account to create resources.
• Terraform Installed: Ensure Terraform is installed on your local machine. You can verify installation by running terraform -version.
• IAM Permissions: You will need permissions to create MSK clusters, VPCs, security groups, IAM roles, and other related resources.
• Basic Understanding of Kafka and Terraform: This guide assumes you have basic knowledge of both.

Steps Overview:

1. Configure the VPC and Subnets
2. Create Security Groups
3. Create the MSK Cluster via Terraform
4. Enable CloudWatch and S3 Logging
5. Deploy the Infrastructure

Step 1: Configure the VPC and Subnets

Kafka clusters must be deployed within a Virtual Private Cloud (VPC). We’ll first create the VPC, subnets, and necessary components.

Here’s a Terraform snippet to create a VPC and its subnets:

resource "aws_vpc" "msk_vpc" {
  cidr_block = "10.0.0.0/16"
  tags = {
    Name = "msk-vpc"
  }
}

resource "aws_subnet" "msk_subnet_a" {
  vpc_id            = aws_vpc.msk_vpc.id
  cidr_block        = "10.0.1.0/24"
  availability_zone = "us-east-1a"
}

resource "aws_subnet" "msk_subnet_b" {
  vpc_id            = aws_vpc.msk_vpc.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-east-1b"
}

Step 2: Create Security Groups

Kafka clusters need to have security rules in place to control inbound and outbound traffic. We’ll set up security groups that allow necessary traffic between your applications and MSK.

resource "aws_security_group" "msk_security_group" {
  vpc_id = aws_vpc.msk_vpc.id

  ingress {
    from_port   = 9092
    to_port     = 9092
    protocol    = "tcp"
    cidr_blocks = ["10.0.0.0/16"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "msk-security-group"
  }
}

Step 3: Create the MSK Cluster via Terraform

Next, we’ll create the MSK cluster using Terraform. Here, you’ll define the cluster properties, including the number of brokers, Kafka version, and more

resource "aws_msk_cluster" "msk_cluster" {
  cluster_name           = "my-msk-cluster"
  kafka_version          = "2.6.0"
  number_of_broker_nodes = 3
  broker_node_group_info {
    instance_type = "kafka.m5.large"
    client_subnets = [
      aws_subnet.msk_subnet_a.id,
      aws_subnet.msk_subnet_b.id
    ]
    security_groups = [aws_security_group.msk_security_group.id]
  }

  encryption_info {
    encryption_in_transit {
      client_broker = "TLS"
      in_cluster    = true
    }
  }

  logging_info {
    broker_logs {
      cloudwatch_logs {
        enabled         = true
        log_group       = aws_cloudwatch_log_group.msk_log_group.name
      }
      s3 {
        enabled         = true
        bucket          = aws_s3_bucket.msk_bucket.bucket
        prefix          = "logs/"
      }
    }
  }

  tags = {
    Name = "MyMSKCluster"
  }
}

Step 4: Enable CloudWatch and S3 Logging

It’s crucial to monitor Kafka operations, and enabling logging to CloudWatch and S3 helps track and troubleshoot issues.

resource "aws_cloudwatch_log_group" "msk_log_group" {
  name = "/aws/msk/my-msk-cluster"
}

resource "aws_s3_bucket" "msk_bucket" {
  bucket = "my-msk-logs"
}

Setting Up IAM Roles for MSK

You’ll need to create an IAM role for the MSK cluster to write logs to CloudWatch and S3.

resource "aws_iam_role" "msk_iam_role" {
  name = "MSKCloudWatchLoggingRole"
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": {
    "Action": "sts:AssumeRole",
    "Principal": {
      "Service": "kafka.amazonaws.com"
    },
    "Effect": "Allow",
    "Sid": ""
  }
}
POLICY
}

Step 5: Deploy the Infrastructure

Now that all resources are defined, you can deploy them by following the standard Terraform workflow:

1. Initialize the Terraform working directory to install the provider plugins:

terraform init

Validate your Terraform configuration to ensure everything is set up properly:

terraform validate

Apply the Terraform configuration to create the MSK cluster and related resources:

terraform apply

Conclusion

By following these steps, you’ve successfully created an AWS MSK Kafka cluster using Terraform. This setup provides a scalable, managed Kafka solution with encryption, security groups, logging, and monitoring in place. AWS MSK abstracts the complexity of managing Kafka clusters, and using Terraform makes your infrastructure as code, ensuring consistency, version control, and ease of updates.

What’s Next?

• Scaling: You can easily modify the Terraform code to scale the MSK cluster by adjusting the number_of_broker_nodes.
• Monitoring: Use Prometheus and Grafana to monitor Kafka metrics for performance tuning.
• Topic Management: Once the cluster is running, explore Kafka’s capabilities by creating topics and managing producers and consumers.

Feel free to tweak the configuration and explore additional features of AWS MSK and Terraform to best suit your use case. Happy coding!