Skip to content

Latest commit

 

History

History
130 lines (93 loc) · 4.99 KB

docker.md

File metadata and controls

130 lines (93 loc) · 4.99 KB

Docker Implementation in IndexAgent

This document provides comprehensive details about the Docker implementation in the IndexAgent project and its companion airflow-hub repository.

Overview

IndexAgent follows a Docker-first approach, with all components running in containers to ensure consistent deployment, isolation, and scalability. The architecture consists of multiple containers that work together to provide code search, indexing, and automated maintenance capabilities.

Container Architecture

IndexAgent Repository Containers

The IndexAgent stack consists of three primary containers:

  1. zoekt-indexserver

    • Image: sourcegraph/zoekt-indexserver:latest
    • Purpose: Provides high-performance code indexing and search capabilities
    • Ports: 6070 (Zoekt API and web UI)
    • Volumes: Mounts $HOME/repos to access source code repositories
  2. sourcebot

    • Image: ghcr.io/sourcebot-dev/sourcebot:latest
    • Purpose: Handles source code discovery, repository syncing, and metadata extraction
    • Ports: 3000 (Sourcebot UI)
    • Volumes: Mounts $HOME/repos to access source code repositories
  3. indexagent

    • Image: indexagent:latest (built from local Dockerfile)
    • Purpose: Runs maintenance scripts and Claude Code CLI operations
    • Ports: 8080 (IndexAgent API)
    • Volumes:
      • $HOME/repos: Access to source code repositories
      • ../reports: For storing analysis reports
      • ./docs/auto: For generated documentation
    • Environment Variables:
      • CONFIG_PATH: Path to configuration file
      • CLAUDE_MODEL: Claude model version (defaults to claude-3-opus-2025-05-02)
    • Secrets:
      • claude_api_key: For authenticating with the Claude API

Docker Compose Configuration

The Docker Compose configuration is defined in config/docker-compose.yml and orchestrates the deployment of all containers. Key features include:

  • Volume mounting for code repositories
  • Port mapping for service access
  • Secret management for API keys
  • Container restart policies
  • Environment variable configuration

Dockerfile Details

The IndexAgent container is built from a custom Dockerfile that:

  1. Uses python:3.11-slim as the base image
  2. Sets up the Python environment and working directory
  3. Copies source code, tests, and scripts into the container
  4. Installs Python dependencies (pytest, pytest-cov)
  5. Installs Node.js and npm for Claude CLI and markdownlint
  6. Installs the Claude Code CLI and markdownlint
  7. Configures the container to remain running for service availability

Integration with airflow-hub

The airflow-hub repository uses Docker to run Apache Airflow 3.0, which orchestrates maintenance tasks by calling the IndexAgent services. The integration works as follows:

  1. Airflow DAGs in airflow-hub use SSHOperator to execute scripts in the IndexAgent container
  2. The scripts access mounted repositories, perform maintenance tasks, and push changes back to source repositories
  3. Metrics and reports generated by IndexAgent are available to Airflow for monitoring and alerting

Docker Secrets Management

IndexAgent uses Docker secrets for managing sensitive information:

  1. In production environments, secrets are stored using Docker's secrets management
  2. For local development, a .env file can be used
  3. The Claude API key is stored as a secret and mounted into the indexagent container

Docker Networking

The containers communicate over a Docker network:

  1. Zoekt and Sourcebot expose web UIs and APIs on mapped ports
  2. The IndexAgent container can access these services via internal Docker networking
  3. External access is controlled through port mapping in the docker-compose.yml

Best Practices Implemented

  1. Isolation: Each component runs in its own container
  2. Reproducibility: Container definitions ensure consistent environments
  3. Volume Mounting: Efficient sharing of code repositories
  4. Secret Management: Secure handling of API keys
  5. Port Mapping: Controlled access to services

Running the Docker Stack

To start the entire stack:

# From the IndexAgent repository root
docker compose -f config/docker-compose.yml up -d
# Or using the Makefile
make up

To stop the stack:

# From the IndexAgent repository root
docker compose -f config/docker-compose.yml down
# Or using the Makefile
make down

Customization

The Docker setup can be customized by:

  1. Modifying the docker-compose.yml file to change port mappings, volume mounts, or add new services
  2. Updating the Dockerfile to install additional dependencies or change the base image
  3. Adjusting environment variables in the docker-compose.yml or through a .env file

Troubleshooting

Common issues and solutions:

  1. Container fails to start: Check logs with docker logs <container_name>
  2. Volume mounting issues: Ensure the host directories exist and have appropriate permissions
  3. API key problems: Verify that secrets are correctly configured
  4. Network connectivity: Use docker network inspect to troubleshoot network issues