> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pullbase.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations & Troubleshooting

> Keep Pullbase healthy and resolve common issues quickly.

Operating Pullbase involves monitoring service health, responding to drift, and collecting diagnostics during incidents.

## Health checks

* **API health:** `curl http://localhost:8080/api/v1/healthz`
* **Database (SQLite):** Check the database file exists and is accessible
* **Database (PostgreSQL):** `docker compose exec db pg_isready -U pullbaseuser -d pullbasedb`
* **Web UI:** Sign in and confirm the dashboard loads recent activity.
* **Agents:** List servers and check `last_timestamp` in status history via CLI or UI.

## Routine operations checklist

<CardGroup cols={2}>
  <Card title="Daily" icon="calendar">
    * Review drift alerts from status history
    * Check audit logs for unexpected actions
    * Ensure webhooks (if enabled) are succeeding
  </Card>

  <Card title="Weekly" icon="rotate">
    * Rotate admin tokens or ensure expiration dates are set
    * Back up the database (SQLite file or PostgreSQL dump)
    * Verify TLS certificates (at reverse proxy) aren't approaching expiry
  </Card>

  <Card title="Monthly" icon="shield-halved">
    * Test agent token rotation on a sample server
    * Exercise rollback on a staging environment
    * Review GitHub App access scopes
  </Card>

  <Card title="Quarterly" icon="file-signature">
    * Run disaster recovery drill (bootstrap new server + restore DB)
    * Patch base OS/images used for agents and central server hosts
  </Card>
</CardGroup>

## Troubleshooting guide

<AccordionGroup>
  <Accordion title="Bootstrap secret already used">
    * Delete `bootstrap-admin-secret.txt` inside the config directory and restart to regenerate.
    * Or provide a new secret via `PULLBASE_BOOTSTRAP_SECRET` environment variable.
    * Create a new admin via CLI if another admin exists: `pullbasectl users create`
  </Accordion>

  <Accordion title="Agents reporting 401 Unauthorized">
    * Token revoked? Generate a new token and update the deployment.
    * Clock skew? Ensure NTP is running—JWT validation is time-sensitive.
    * Incorrect `SERVER_ID`? Confirm environment variable matches the server record.
  </Accordion>

  <Accordion title="Git clone failures">
    * Check outbound network connectivity to the Git provider.
    * For GitHub Apps, verify installation scopes and ensure the private key matches the app.
    * Confirm agents trust the TLS certificate (`CACERT_PATH` points to the CA bundle).
  </Accordion>

  <Accordion title="Service manager not detected">
    * Override auto-detection by setting `system.serviceManager` in `config.yaml`.
    * For Docker containers, use `supervisor` as the service manager.
    * Ensure service manager commands are in the agent's PATH.
  </Accordion>

  <Accordion title="Database migration errors">
    * Review container logs around startup for SQL errors.
    * Migrations run automatically for both SQLite and PostgreSQL.
    * Check for schema drift caused by manual database edits.
    * Restore from backup if migrations fail due to corruption.
  </Accordion>

  <Accordion title="Webhook retries failing">
    * Inspect container logs: `docker compose logs pullbase`
    * Ensure the webhook secret in GitHub matches `PULLBASE_WEBHOOK_SECRET_KEY`.
    * Confirm Pullbase is reachable from your Git provider (network ACLs, firewalls).
  </Accordion>

  <Accordion title="TLS certificate permission denied">
    When TLS is enabled, the server attempts to generate self-signed certificates on startup. If you see:

    ```
    failed to generate self-signed certificates: mkdir certs: permission denied
    ```

    **Option 1: Disable TLS** (recommended if behind a reverse proxy or for testing):

    ```bash theme={null}
    echo "PULLBASE_TLS_ENABLED=false" | sudo tee -a /etc/pullbase/pullbase.env
    sudo systemctl restart pullbase
    ```

    **Option 2: Create the certs directory with proper permissions**:

    ```bash theme={null}
    sudo mkdir -p /var/lib/pullbase/certs
    sudo chown pullbase:pullbase /var/lib/pullbase/certs

    # Update env to use this path
    sudo tee -a /etc/pullbase/pullbase.env > /dev/null <<'EOF'
    PULLBASE_TLS_CERT_PATH=/var/lib/pullbase/certs/server.crt
    PULLBASE_TLS_KEY_PATH=/var/lib/pullbase/certs/server.key
    EOF

    sudo systemctl restart pullbase
    ```

    For production, use CA-signed certificates or terminate TLS at a reverse proxy.
  </Accordion>

  <Accordion title="Exec format error">
    If you see `Failed to execute /usr/local/bin/pullbase-server: Exec format error`:

    You downloaded the wrong binary for your system architecture.

    ```bash theme={null}
    # Check your architecture
    uname -m
    ```

    | Output               | Binary to download            |
    | -------------------- | ----------------------------- |
    | `x86_64`             | `pullbase-server-linux-amd64` |
    | `aarch64` or `arm64` | `pullbase-server-linux-arm64` |

    Download the correct binary and reinstall:

    ```bash theme={null}
    sudo systemctl stop pullbase
    curl -fsSL -o pullbase-server "https://github.com/pullbase/pullbase/releases/latest/download/pullbase-server-linux-arm64"
    sudo install -m 0755 pullbase-server /usr/local/bin/pullbase-server
    sudo systemctl start pullbase
    ```
  </Accordion>
</AccordionGroup>

## Collect diagnostics

```bash theme={null}
# Pullbase server logs
docker compose logs pullbase --since 1h

# Database logs
docker compose logs db --since 1h

# Agent logs (container)
docker logs pullbase-agent --since 1h

# Agent logs (systemd)
journalctl -u pullbase-agent --since "1 hour ago"
```

<Steps>
  <Step title="Export environment metadata">
    ```bash theme={null}
    curl -H "Authorization: Bearer $ADMIN_TOKEN" \
      http://localhost:8080/api/v1/environments
    ```
  </Step>

  <Step title="Export server status history">
    ```bash theme={null}
    curl -H "Authorization: Bearer $ADMIN_TOKEN" \
      http://localhost:8080/api/v1/servers/web-01/status/history
    ```
  </Step>

  <Step title="Capture database snapshot">
    **SQLite:**

    ```bash theme={null}
    cp /data/pullbase.db pullbase-$(date +%F).db
    ```

    **PostgreSQL:**

    ```bash theme={null}
    docker compose exec db \
      pg_dump -U pullbaseuser pullbasedb \
      > pullbasedb-$(date +%F).sql
    ```
  </Step>
</Steps>

## Resetting the installation

If you need a clean slate (lab environments):

1. Stop services: `docker compose down`
2. Remove data **only if you accept data loss**:
   * **SQLite:** `docker volume rm pullbase_pullbase_data` or delete the database file
   * **PostgreSQL:** `docker volume rm pullbase_postgres_data`
3. Start again: `docker compose up -d`

<Warning>
  Removing the database wipes environments, servers, tokens, and audit history. Always back up before destructive operations.
</Warning>

## Getting help

* Review [GitHub issues](https://github.com/pullbase/pullbase/issues) for known bugs.
* Collect the diagnostic bundle above before filing a bug report.
* Provide Pullbase version and deployment details.

Keep this playbook handy so you can respond quickly when drift occurs or when an agent experiences connectivity issues.