Setting Up a CI/CD Pipeline for Docker Images with GitLab

Introduction

Hello, tech enthusiasts! Today, we’re diving into the world of Continuous Integration and Continuous Deployment (CI/CD) for Docker images. We’ll explore how to set up a robust pipeline using GitLab, covering everything from building images with Docker Buildx or Buildah to tagging, pushing, and automating cleanup. Whether you’re a seasoned developer or just starting, this guide will help streamline your workflow.

Building Docker Images: Docker Buildx vs. Buildah

When it comes to building Docker images, you have two powerful tools at your disposal: Docker Buildx and Buildah. Here’s a quick comparison to help you decide:

  • Docker Buildx:

    • Integrates seamlessly with the Docker ecosystem.
    • Supports building images for multiple architectures.
    • Ideal for those already familiar with Docker.

  • Buildah:

    • Offers a rootless mode for enhanced security.
    • Suitable for environments where security is a priority.

Example: Building with Docker Buildx

.build_template:
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  before_script:
    - docker info
    - docker buildx create --use
  script:

    - docker buildx build --platform linux/amd64 -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
  • docker:dind: Enables Docker-in-Docker to build images directly in CI.
  • DOCKER_DRIVER: overlay2: Specifies the storage driver for layers.
  • docker buildx create —use: Sets up the builder for handling requests.
  • —platform linux/amd64: Specifies the architecture explicitly.

Tagging: Assigning the Right Tags

Tagging is crucial for identifying and managing your Docker images. Use GitLab environment variables to automate this process:

  • CI_COMMIT_SHORT_SHA: Short hash of the commit.
  • CI_COMMIT_REF_NAME: Name of the branch.

Tagging Example

docker buildx build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME .
  • The first tag uniquely identifies the image by commit.
  • The second tag helps track images built from specific branches.

To automate adding the latest tag for the main branch:

script:
  - |
    TAGS="-t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME"
    if [ "$CI_COMMIT_REF_NAME" = "main" ]; then
      TAGS="$TAGS -t $CI_REGISTRY_IMAGE:latest"
    fi
    docker buildx build --platform linux/amd64 $TAGS .

Pushing to Registries: GitLab, DockerHub, GHCR

Once built, your images need a home. GitLab Registry is a great choice due to its seamless integration. However, DockerHub and GitHub Container Registry (GHCR) are also viable options.

GitLab Registry Example

docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME
  • docker login: Authenticates using GitLab-provided variables.
  • Push images with both commit and branch tags for flexibility.

DockerHub Example

variables:
  DOCKERHUB_IMAGE: yourdockerhubuser/yourproject
  DOCKERHUB_USERNAME: yourdockerhubuser
  DOCKERHUB_PASSWORD: $DOCKERHUB_PASSWORD

script:
  - echo "$DOCKERHUB_PASSWORD" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
  - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $DOCKERHUB_IMAGE:$CI_COMMIT_SHORT_SHA
  - docker push $DOCKERHUB_IMAGE:$CI_COMMIT_SHORT_SHA
  • DockerHub is straightforward for small projects but watch out for request limits in large CI/CD systems.

Complete .gitlab-ci.yml

Here’s a comprehensive example of a .gitlab-ci.yml file that includes caching, artifacts, testing, and deployment:

stages:
  - build
  - test
  - deploy

variables:
  IMAGE_TAG: $CI_COMMIT_SHORT_SHA
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - docker info
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker buildx create --use
  script:
    - |
      TAGS="-t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA -t $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_NAME"
      if [ "$CI_COMMIT_REF_NAME" = "main" ]; then

        TAGS="$TAGS -t $CI_REGISTRY_IMAGE:latest"
      fi
      docker buildx build --platform linux/amd64 $TAGS --push .
  artifacts:
    expire_in: 1 hour
    paths:
      - docker-compose.yml
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - .cache

test:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  script:
    - echo "Running tests..."

    - docker pull $CI_REGISTRY_IMAGE:$IMAGE_TAG
    - docker run --rm $CI_REGISTRY_IMAGE:$IMAGE_TAG pytest tests/
  artifacts:
    paths:
      - test-reports/
    when: always

deploy:
  stage: deploy
  image: alpine:latest
  script:
    - echo "Starting deployment..."
    - apk add --no-cache curl
    - curl -X POST "https://my-deploy-endpoint/internal/redeploy?image=$CI_REGISTRY_IMAGE:$IMAGE_TAG"
  environment:
    name: production
    url: https://my-production-app.example.com
  • Stages: Defined as build, test, and deploy for a streamlined process.
  • Artifacts: Save useful files like docker-compose.yml and test reports for analysis.
  • Cache: Retain intermediate data between builds.
  • Environment: Track deployment environments with GitLab.

Cleanup: Automating Image Cleanup

Over time, your registry may accumulate outdated images. Automate cleanup using the GitLab API.

Python Script for Cleanup

import requests

# Set your values
GITLAB_TOKEN = "your_token"
PROJECT_ID = 12345
REGISTRY_URL = f"https://gitlab.com/api/v4/projects/{PROJECT_ID}/registry/repositories"

headers = {"PRIVATE-TOKEN": GITLAB_TOKEN}

def get_repositories():
    response = requests.get(REGISTRY_URL, headers=headers)
    response.raise_for_status()
    return response.json()

def get_tags(repo_id):
    tags_url = f"{REGISTRY_URL}/{repo_id}/tags"

    response = requests.get(tags_url, headers=headers)
    response.raise_for_status()
    return response.json()

def delete_tag(repo_id, tag_name):
    delete_url = f"{REGISTRY_URL}/{repo_id}/tags/{tag_name}"
    response = requests.delete(delete_url, headers=headers)
    if response.status_code == 204:
        print(f"Tag {tag_name} successfully deleted from repository {repo_id}")
    else:
        print(f"Error deleting tag {tag_name} from repository {repo_id}: {response.text}")

def cleanup_old_tags():
    repositories = get_repositories()
    for repo in repositories:

        repo_id = repo['id']
        tags = get_tags(repo_id)
        # Delete all tags except 'latest' and 'main'
        tags_to_delete = [tag['name'] for tag in tags if tag['name'] not in ('latest', 'main')]
        for tag in tags_to_delete:
            delete_tag(repo_id, tag)

if __name__ == "__main__":
    cleanup_old_tags()
  • Fetch all container registry repositories using the API.
  • Retrieve tags for each repository and filter out non-essential ones.
  • Sequentially delete unnecessary tags.

Conclusion

Setting up a CI/CD pipeline for Docker images with GitLab can significantly enhance your development workflow. By automating builds, tagging, and cleanup, you ensure a smooth and efficient process. If you have any questions or want to share your experiences, feel free to comment below. For those eager to delve deeper into automation, consider joining Otus’s open lessons to explore real-world cases and tools that complement GitLab and Docker.

  • April 3: Terraform: Working with it through GitLab. Learn More
  • April 16: Setting up GitLab Runner without regrets. Learn More

Happy coding!