Kubernetes Development and Operations Best Practices

---
description: This rule provides comprehensive best practices for developing and maintaining Kubernetes applications and infrastructure, covering coding standards, security, performance, testing, and deployment.
globs: **/*.{go,yaml,yml,sh,tf,tfvars,json}
---

# Kubernetes Development and Operations Best Practices

This document outlines a collection of guidelines, style suggestions, and tips for writing code and managing infrastructure within the Kubernetes ecosystem. It emphasizes clarity, maintainability, security, and performance.

## 1. Code Organization and Structure

### 1.1 Directory Structure

- **Root Level:**
  - `cmd/`:  Main application entry points. Each subdirectory represents a separate command-line tool or service.
  - `pkg/`: Reusable libraries and components that can be imported by other projects.
  - `internal/`: Private code that should not be imported by external projects.  Enforces encapsulation.
  - `api/`:  API definitions, including protobuf files and OpenAPI specifications.
  - `config/`: Configuration files, such as YAML manifests, Kustomize configurations, and Helm charts.
  - `scripts/`: Utility scripts for building, testing, and deploying the application.
  - `docs/`: Documentation for the project.
  - `examples/`: Example usage of the library or application.
  - `vendor/`: (If using `go modules` without external dependency management) Contains vendored dependencies.  Generally discouraged in modern Go with `go modules`.
- **Component-Specific Directories:** Inside `pkg/` or `internal/`, organize code by component or module. Each component should have its own directory with clear separation of concerns.

Example:


my-kubernetes-project/
├── cmd/
│   └── controller/
│       └── main.go
├── pkg/
│   └── api/
│       ├── types.go
│   └── controller/
│       ├── controller.go
│       ├── reconciler.go
│   └── util/
│       └── util.go
├── internal/
│   └── admission/
│       └── webhook.go
├── config/
│   ├── deploy/
│   │   └── deployment.yaml
│   └── kustomize/
│       ├── base/
│       │   ├── kustomization.yaml
│       │   └── ...
│       └── overlays/
│           ├── dev/
│           │   ├── kustomization.yaml
│           │   └── ...
│           └── prod/
│               ├── kustomization.yaml
│               └── ...
├── scripts/
│   └── build.sh
├── docs/
│   └── architecture.md
└── go.mod


### 1.2 File Naming Conventions

- **Go Files:** Use lowercase with underscores (e.g., `my_controller.go`).
- **YAML Files:** Use lowercase with dashes (e.g., `deployment.yaml`).
- **Configuration Files:** Be descriptive and consistent (e.g., `config.yaml`, `kustomization.yaml`).
- **Test Files:** Follow the standard Go convention: `*_test.go` (e.g., `my_controller_test.go`).

### 1.3 Module Organization (Go)

- **Packages:**  Organize code into meaningful packages that represent logical units of functionality.
- **Internal Packages:** Use `internal/` directories to create packages that are only visible within the project.
- **Interfaces:** Define interfaces to abstract dependencies and promote testability.

### 1.4 Component Architecture

- **Microservices:** Design applications as a collection of loosely coupled microservices.
- **Separation of Concerns:** Each component should have a single responsibility and well-defined interfaces.
- **API Gateway:** Use an API gateway to handle routing, authentication, and rate limiting for external requests.
- **Service Mesh:** Consider using a service mesh (e.g., Istio, Linkerd) to manage inter-service communication, observability, and security.

### 1.5 Code Splitting Strategies

- **Feature-Based Splitting:** Group code by feature or functionality.
- **Layer-Based Splitting:** Separate code into layers, such as data access, business logic, and presentation.
- **Component-Based Splitting:** Divide code into reusable components that can be shared across multiple projects.

## 2. Common Patterns and Anti-Patterns

### 2.1 Design Patterns

- **Controller Pattern:** Implement controllers to reconcile the desired state of Kubernetes resources with the actual state.
- **Operator Pattern:** Extend the Kubernetes API with custom resources and controllers to automate complex application management tasks.
- **Sidecar Pattern:** Deploy a sidecar container alongside the main application container to provide supporting functionality, such as logging, monitoring, or security.
- **Ambassador Pattern:**  Use an ambassador container to proxy network traffic to the main application container, providing features such as load balancing, routing, and authentication.
- **Adapter Pattern:** Translate requests from one interface to another, allowing different components to work together.
- **Singleton Pattern:** Implement a singleton pattern for managing global resources, such as database connections or configuration settings. Be extremely cautious, as this can hurt testability and introduce implicit dependencies.

### 2.2 Recommended Approaches for Common Tasks

- **Resource Management:** Use Kubernetes resource requests and limits to ensure that applications have sufficient resources and prevent resource contention.
- **Configuration Management:** Use ConfigMaps and Secrets to manage configuration data and sensitive information separately from the application code.
- **Service Discovery:** Use Kubernetes services to provide a stable endpoint for accessing applications, even when pods are scaled up or down.
- **Health Checks:** Implement liveness and readiness probes to monitor the health of applications and automatically restart unhealthy pods.
- **Logging and Monitoring:** Use a centralized logging and monitoring system to collect and analyze application logs and metrics.
- **Rolling Updates:** Use Kubernetes deployments to perform rolling updates of applications with zero downtime.

### 2.3 Anti-Patterns and Code Smells

- **Naked Pods:** Avoid creating pods directly without a deployment or replica set, as they will not be automatically rescheduled if a node fails.
- **Hardcoded Configuration:** Avoid hardcoding configuration data in the application code. Use ConfigMaps and Secrets instead.
- **Ignoring Resource Limits:** Failing to set resource requests and limits can lead to resource contention and performance issues.
- **Oversized Containers:**  Keep container images small and focused to improve startup time and reduce security risks.
- **Privileged Containers:** Avoid running containers in privileged mode, as it can create security vulnerabilities.
- **Long-Lived Branches:** Avoid creating long-lived branches, and prefer small, frequent merges to the main branch.
- **God Classes:** Avoid creating classes that are too large and complex.  Break them down into smaller, more manageable classes.
- **Shotgun Surgery:**  Avoid making changes to multiple classes when a single feature is modified. This suggests poor class design and coupling.
- **Feature Envy:**  Avoid methods that access the data of another object more than their own.  This suggests that the method might be in the wrong class.

### 2.4 State Management Best Practices

- **Stateless Applications:** Prefer stateless applications whenever possible, as they are easier to scale and manage.
- **Persistent Volumes:** Use Persistent Volumes to store persistent data for stateful applications.
- **External Databases:** Consider using external databases for managing application state, such as databases hosted on cloud providers.
- **Kubernetes Operators:** Implement Kubernetes operators to automate the management of stateful applications.
- **Etcd:** Understand the importance of etcd as Kubernetes' data store and protect it accordingly.

### 2.5 Error Handling Patterns

- **Centralized Error Handling:** Implement a centralized error handling mechanism to handle exceptions and log errors consistently.
- **Retry Mechanism:** Implement a retry mechanism to automatically retry failed operations.
- **Circuit Breaker Pattern:** Use a circuit breaker pattern to prevent cascading failures in distributed systems.
- **Logging Error Details:** Log detailed error messages, including stack traces and relevant context, to help with debugging.
- **Graceful Degradation:** Design applications to gracefully degrade functionality when errors occur.
- **Alerting on Critical Errors:** Set up alerts to notify administrators when critical errors occur.

## 3. Performance Considerations

### 3.1 Optimization Techniques

- **Caching:** Implement caching to reduce latency and improve performance.
- **Load Balancing:** Use load balancing to distribute traffic across multiple instances of an application.
- **Connection Pooling:** Use connection pooling to reuse database connections and reduce overhead.
- **Compression:** Use compression to reduce the size of data transmitted over the network.
- **Gzip:** Enable Gzip compression in web servers to reduce the size of HTTP responses.

### 3.2 Memory Management

- **Memory Profiling:** Use memory profiling tools to identify memory leaks and optimize memory usage.
- **Garbage Collection:** Understand how garbage collection works in the programming language used for the application.
- **Resource Limits:** Set memory resource limits for containers to prevent them from consuming excessive memory.
- **Monitor Memory Usage:** Monitor memory usage regularly to identify potential issues.

### 3.3 Rendering Optimization

- **Minimize DOM Manipulation:** Reduce the number of DOM manipulations to improve rendering performance in web applications.
- **Virtual DOM:** Use a virtual DOM to optimize rendering updates in web applications.
- **Lazy Loading:** Use lazy loading to load images and other resources only when they are needed.

### 3.4 Bundle Size Optimization

- **Code Minification:** Use code minification to reduce the size of JavaScript and CSS files.
- **Tree Shaking:** Use tree shaking to remove unused code from JavaScript bundles.
- **Image Optimization:** Optimize images to reduce their file size without sacrificing quality.
- **Code Splitting:** Split the application code into smaller bundles that can be loaded on demand.

### 3.5 Lazy Loading Strategies

- **On-Demand Loading:** Load resources only when they are needed by the application.
- **Intersection Observer:** Use the Intersection Observer API to detect when elements are visible in the viewport and load them accordingly.
- **Placeholder Images:** Use placeholder images while loading the actual images to improve the user experience.

## 4. Security Best Practices

### 4.1 Common Vulnerabilities and How to Prevent Them

- **Injection Attacks:** Prevent injection attacks by validating and sanitizing all user input.
- **Cross-Site Scripting (XSS):** Prevent XSS attacks by escaping all user-generated content before rendering it in the browser.
- **Cross-Site Request Forgery (CSRF):** Prevent CSRF attacks by using anti-CSRF tokens.
- **Authentication and Authorization Flaws:** Implement robust authentication and authorization mechanisms to protect sensitive data and resources.
- **Security Misconfiguration:** Avoid using default configurations and ensure that all components are properly configured with security in mind.
- **Using Components with Known Vulnerabilities:** Keep all dependencies up to date to patch known vulnerabilities.
- **Insufficient Logging and Monitoring:** Implement comprehensive logging and monitoring to detect and respond to security incidents.
- **Container Security:** Follow best practices for securing containers, such as using minimal images, running as non-root, and limiting capabilities.
- **Network Policies:** Use network policies to restrict network traffic between pods.
- **RBAC (Role-Based Access Control):** Implement RBAC to control access to Kubernetes resources.
- **Secrets Management:** Use Kubernetes Secrets to store sensitive information, and encrypt secrets at rest.
- **Pod Security Policies/Pod Security Standards:** Enforce Pod Security Standards to restrict the capabilities of pods.

### 4.2 Input Validation

- **Validate All Input:** Validate all input, including user input, API requests, and configuration data.
- **Use Strong Data Types:** Use strong data types to enforce data integrity.
- **Sanitize Input:** Sanitize input to remove potentially harmful characters and prevent injection attacks.
- **Whitelist Input:** Use a whitelist approach to only allow known good input.
- **Blacklist Input:** Avoid using a blacklist approach, as it can be easily bypassed.

### 4.3 Authentication and Authorization Patterns

- **Multi-Factor Authentication (MFA):** Use MFA to enhance authentication security.
- **OAuth 2.0:** Use OAuth 2.0 for authorization and delegation of access.
- **JSON Web Tokens (JWT):** Use JWTs for securely transmitting claims between parties.
- **Role-Based Access Control (RBAC):** Implement RBAC to control access to resources based on roles.
- **Least Privilege Principle:** Grant users and applications only the minimum necessary permissions.

### 4.4 Data Protection Strategies

- **Encryption at Rest:** Encrypt sensitive data at rest to protect it from unauthorized access.
- **Encryption in Transit:** Encrypt sensitive data in transit using HTTPS or other secure protocols.
- **Data Masking:** Mask sensitive data to prevent it from being exposed to unauthorized users.
- **Data Anonymization:** Anonymize data to remove personally identifiable information (PII).
- **Data Loss Prevention (DLP):** Implement DLP measures to prevent sensitive data from leaving the organization.

### 4.5 Secure API Communication

- **HTTPS:** Use HTTPS for all API communication to encrypt data in transit.
- **API Authentication:** Implement API authentication to verify the identity of clients.
- **API Authorization:** Implement API authorization to control access to API endpoints.
- **Rate Limiting:** Implement rate limiting to prevent abuse and denial-of-service attacks.
- **Input Validation:** Validate all API requests to prevent injection attacks and other vulnerabilities.

## 5. Testing Approaches

### 5.1 Unit Testing Strategies

- **Test-Driven Development (TDD):** Write unit tests before writing the application code.
- **Mock Dependencies:** Use mocks to isolate the unit being tested from its dependencies.
- **Test Boundary Conditions:** Test boundary conditions and edge cases to ensure that the code handles them correctly.
- **Test Error Conditions:** Test error conditions to ensure that the code handles errors gracefully.
- **Code Coverage:** Aim for high code coverage to ensure that all parts of the code are tested.
- **Table-Driven Tests:**  Use table-driven tests to easily test multiple inputs and outputs.

### 5.2 Integration Testing

- **Test Interactions Between Components:** Test the interactions between different components of the application.
- **Test with Real Dependencies:** Use real dependencies or integration mocks for integration tests.
- **Test Data Flows:** Test the data flows through the application to ensure that data is processed correctly.
- **Contract Tests:**  Use contract tests to ensure that services adhere to a defined contract.

### 5.3 End-to-End Testing

- **Test the Entire System:** Test the entire system from end to end to ensure that all components work together correctly.
- **Automate End-to-End Tests:** Automate end-to-end tests to ensure that they are run regularly.
- **Use Realistic Test Data:** Use realistic test data to simulate real-world scenarios.
- **CI/CD Integration:** Integrate end-to-end tests into the CI/CD pipeline.

### 5.4 Test Organization

- **Keep Tests Separate from Code:** Keep tests separate from the application code in a dedicated `test/` directory.
- **Organize Tests by Component:** Organize tests by component or module to make them easier to find and maintain.
- **Use Clear Naming Conventions:** Use clear naming conventions for test files and test functions.

### 5.5 Mocking and Stubbing

- **Use Mocking Frameworks:** Use mocking frameworks to simplify the creation of mocks and stubs.
- **Mock External Dependencies:** Mock external dependencies, such as databases and APIs, to isolate the unit being tested.
- **Stub Responses:** Use stubs to provide predefined responses for external dependencies.
- **Verify Interactions:** Verify that the code under test interacts with dependencies as expected.

## 6. Common Pitfalls and Gotchas

### 6.1 Frequent Mistakes Developers Make

- **Ignoring Error Handling:** Failing to handle errors properly can lead to unexpected behavior and crashes.
- **Not Using Version Control:** Not using version control can lead to lost code and conflicts.
- **Hardcoding Configuration:** Hardcoding configuration data can make it difficult to deploy the application in different environments.
- **Not Securing Sensitive Data:** Not securing sensitive data can lead to security breaches.
- **Not Testing Thoroughly:** Not testing thoroughly can lead to bugs and performance issues.
- **Over-Engineering:** Adding unnecessary complexity to the code can make it difficult to understand and maintain.
- **Premature Optimization:** Optimizing code before it is necessary can waste time and make the code harder to read.
- **Not Documenting Code:** Not documenting code can make it difficult for others to understand and maintain it.

### 6.2 Edge Cases to Be Aware Of

- **Network Connectivity Issues:** Handle network connectivity issues gracefully.
- **Resource Exhaustion:** Handle resource exhaustion gracefully.
- **Concurrency Issues:** Avoid concurrency issues by using proper synchronization mechanisms.
- **Data Corruption:** Protect against data corruption by using checksums and other data integrity techniques.
- **Time Zone Issues:** Be aware of time zone issues when working with dates and times.

### 6.3 Version-Specific Issues

- **API Version Compatibility:** Be aware of API version compatibility issues when upgrading Kubernetes or other dependencies.
- **Feature Deprecation:** Be aware of feature deprecation when upgrading Kubernetes or other dependencies.
- **Configuration Changes:** Be aware of configuration changes when upgrading Kubernetes or other dependencies.

### 6.4 Compatibility Concerns

- **Operating System Compatibility:** Ensure that the application is compatible with the target operating systems.
- **Architecture Compatibility:** Ensure that the application is compatible with the target architectures (e.g., x86, ARM).
- **Browser Compatibility:** Ensure that web applications are compatible with the target browsers.

### 6.5 Debugging Strategies

- **Logging:** Use detailed logging to help identify the root cause of issues.
- **Debugging Tools:** Use debugging tools, such as debuggers and profilers, to analyze the code and identify performance bottlenecks.
- **Remote Debugging:** Use remote debugging to debug applications running in Kubernetes.
- **Log Aggregation:**  Use log aggregation tools (e.g., Elasticsearch, Loki) to centralize and analyze logs.
- **Metrics Monitoring:** Use metrics monitoring tools (e.g., Prometheus, Grafana) to track application performance.
- **Tracing:** Implement distributed tracing (e.g., Jaeger, Zipkin) to track requests across multiple services.

## 7. Tooling and Environment

### 7.1 Recommended Development Tools

- **IDE:** Use a modern IDE with support for the programming language used for the application (e.g., VS Code, IntelliJ IDEA, GoLand).
- **Kubectl:** Use `kubectl` for interacting with Kubernetes clusters.
- **Minikube/Kind:** Use Minikube or Kind for local Kubernetes development.
- **Helm:** Use Helm for managing Kubernetes packages.
- **Kustomize:** Use Kustomize for customizing Kubernetes configurations.
- **Docker:** Use Docker for building and managing container images.
- **Tilt:** Use Tilt for fast, local Kubernetes development.
- **Skaffold:** Use Skaffold for automated build, push, and deploy workflows.
- **Telepresence:** Use Telepresence to debug applications running in Kubernetes from your local machine.

### 7.2 Build Configuration

- **Makefile:** Use a Makefile to automate common build tasks.
- **CI/CD Pipeline:** Integrate the build process into a CI/CD pipeline.
- **Dependency Management:** Use a dependency management tool, such as `go modules`, to manage dependencies.
- **Version Control:** Use version control to track changes to the build configuration.

### 7.3 Linting and Formatting

- **Linters:** Use linters to enforce code style and best practices (e.g., `golangci-lint`, `eslint`, `stylelint`).
- **Formatters:** Use formatters to automatically format code according to a predefined style (e.g., `go fmt`, `prettier`).
- **Pre-Commit Hooks:** Use pre-commit hooks to run linters and formatters before committing code.

### 7.4 Deployment Best Practices

- **Infrastructure as Code (IaC):** Use IaC tools, such as Terraform or CloudFormation, to manage infrastructure.
- **Immutable Infrastructure:** Deploy immutable infrastructure to ensure consistency and repeatability.
- **Blue-Green Deployments:** Use blue-green deployments to minimize downtime during deployments.
- **Canary Deployments:** Use canary deployments to test new versions of the application with a small subset of users.
- **Rolling Updates:** Use rolling updates to gradually update the application with zero downtime.
- **Automated Rollbacks:** Implement automated rollbacks to quickly revert to a previous version of the application if something goes wrong.

### 7.5 CI/CD Integration

- **Automated Testing:** Automate all tests in the CI/CD pipeline.
- **Automated Deployment:** Automate the deployment process in the CI/CD pipeline.
- **Continuous Integration:** Use continuous integration to automatically build and test the application whenever code is committed.
- **Continuous Delivery:** Use continuous delivery to automatically deploy the application to production whenever a new version is released.
- **Pipeline Security:** Secure the CI/CD pipeline to prevent unauthorized access and code injection.

## Bibliography

- Kubernetes documentation: [https://kubernetes.io/docs/](https://kubernetes.io/docs/)
- Kubernetes Best Practices: [https://kubernetes.io/docs/concepts/configuration/overview/](https://kubernetes.io/docs/concepts/configuration/overview/)
- Application Security Checklist: [https://kubernetes.io/docs/concepts/security/application-security-checklist/](https://kubernetes.io/docs/concepts/security/application-security-checklist/)
- Kubernetes coding conventions: [https://www.kubernetes.dev/docs/guide/coding-convention/](https://www.kubernetes.dev/docs/guide/coding-convention/)