DevOps Security Best Practices Every Engineering Team Should Follow

I've spent 25 years securing Linux boxes, cloud accounts, CI/CD pipelines, and production clusters. The single most consistent lesson across all of it is this: the teams that get breached aren't the ones who lacked a security department. They're the ones who treated security as something a separate department would handle later.

Security is not a phase. It's not a gate at the end of the pipeline, and it's not a quarterly audit. It's a property of how you write infrastructure code, manage secrets, ship containers, and run production every single day. When security lives inside the daily workflow — in the merge request, the pipeline stage, the Terraform plan — it costs almost nothing. When it lives in a separate review at the end, it's expensive, late, and routinely skipped.

This is the checklist I'd hand a new engineering team. Everything here is defensive: hardening, detection, and recovery. Work through it section by section.

Why DevOps security belongs in the daily workflow

The whole premise of DevOps was to stop throwing work over the wall between dev and ops. Security is the last wall standing in most orgs, and it has to come down the same way: by moving the controls into the tools engineers already use.

• Treat every pull/merge request as a security review surface, not just a code review.


• Run security checks as pipeline stages that fail the build, not as advisory reports nobody reads.


• Make the secure path the easy path — a hardened base image, a vetted Terraform module, a secrets helper — so engineers don't route around it.


• Assign a security owner per service, not a security team for the whole company. Ownership beats oversight.


• Measure mean-time-to-remediate for vulnerabilities the same way you measure deploy frequency.

If a control only exists in a wiki page, it doesn't exist. If it exists in the pipeline, it's real.

Secure access control and least privilege

Most incidents I've cleaned up came down to one over-privileged credential. Least privilege is boring and it's the highest-leverage thing on this list.

• Default every IAM role, Kubernetes ServiceAccount, and Linux user to zero permissions, then add only what's needed.


• Replace standing admin access with just-in-time elevation that expires automatically.


• Scope cloud roles to specific resources and actions — no *:* policies, ever.


• In Kubernetes, use RBAC Roles bound to namespaces rather than ClusterRole bindings wherever possible.


• Separate human identities from machine identities. Humans get SSO; services get workload identity.


• Audit who can sudo, who's in the docker group (that's root-equivalent), and who holds cloud admin — quarterly, in writing.

SSH key management and MFA

SSH is still how a huge amount of production gets touched, and it's still where credential hygiene quietly rots.

• Disable password authentication entirely: PasswordAuthentication no and PermitRootLogin no in sshd_config.


• Use per-user keys, never a shared key passed around in a chat thread.


• Prefer short-lived SSH certificates from a CA over long-lived static keys; rotate the rest on a schedule.


• Put a bastion/jump host in front of production and log every session through it.


• Require MFA on every identity provider, VPN, and cloud console — phishing-resistant (WebAuthn/hardware keys) for anyone with production access.


• Pull keys for departed team members the same day, and audit authorized_keys files for orphans.

Secrets management: API keys, passwords, and tokens

The fastest way to leak a secret is to commit it. The second fastest is to print it. Both are entirely preventable.

• Never store secrets in git — not in code, not in .env, not in a "temporary" YAML file. Add a pre-commit secret scanner (gitleaks or trufflehog) to block it.


• Centralize secrets in a real secrets manager: HashiCorp Vault, a cloud secrets manager, or equivalent.


• For Kubernetes, use Sealed Secrets or an external-secrets operator so the cluster pulls from Vault at runtime — plain Secret objects are only base64, not encrypted.


• Give every secret a rotation policy and an owner. Static credentials that never rotate are time bombs.


• Inject secrets as runtime environment values or mounted files, not baked into container images or Terraform state.


• Scan your git history, not just the current tree — a secret deleted in HEAD is still in the log until you rotate it.

CI/CD pipeline security

Your pipeline has credentials to everything. That makes it one of the highest-value targets you own, and it's frequently the least hardened.

• Protect your main branches: require reviews, status checks, and signed commits before merge.


• Mark CI/CD variables as protected and masked so they're only exposed on protected branches and never echoed to logs.


• In GitLab CI, scope variables to environments and never echo a secret — masking helps, but the discipline of not printing it is what saves you.


• Replace long-lived cloud keys in CI with short-lived credentials via OIDC. Let the pipeline exchange its identity for a temporary, scoped token instead of holding a static AWS_SECRET_ACCESS_KEY.


• Pin and review your pipeline dependencies — third-party CI templates and actions run with your pipeline's privileges.


• Require manual approval for production deploys, and make the deploy job itself least-privileged.

A leaked CI variable is a leaked production credential. Treat the pipeline config with the same care you'd treat root.

Container image scanning

A container is only as trustworthy as the layers underneath it. Most images ship with known CVEs the team never looked at.

• Scan every image with Trivy (or Grype) as a GitLab pipeline stage before push, and fail the build on high/critical findings:
  

  container_scan:

    stage: test

    image: aquasec/trivy:latest

    script:

      - trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMAGE_TAG"

• Start from minimal base images (distroless or slim) to shrink the attack surface.


• Run containers as a non-root user (USER in the Dockerfile) with a read-only root filesystem where possible.


• Drop all Linux capabilities and add back only what's required.


• Pin base images by digest, not by floating :latest tags, and rebuild regularly to pick up patches.


• Sign images and verify signatures at admission so only your builds run in your cluster.

Infrastructure as Code security

IaC is where a one-line mistake becomes a fleet-wide misconfiguration. The good news: it's also where automated policy catches it before it ships.

• Review Terraform and Ansible changes like application code — every change goes through a merge request with a human reviewer.


• Run static analysis on IaC in the pipeline: tfsec/Checkov for Terraform, ansible-lint and kube-linter for the rest.


• Adopt policy-as-code (OPA/Conftest or Sentinel) so rules like "no public S3 buckets" and "no 0.0.0.0/0 on port 22" are enforced automatically, not remembered by reviewers.


• Protect and encrypt Terraform state — it contains secrets in plaintext. Use a remote backend with locking and access controls.


• For Ansible, encrypt sensitive variables with Vault and avoid become where it isn't needed.


• Diff the plan before every apply and require approval for changes to security groups, IAM, and networking.

If you want a structured second opinion on a risky module, an automated cover image scanning, Linux hardening, and IaC review, and the broader — practical AI workflows for cloud engineers.