> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pullbase.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Operations & Troubleshooting

> Keep Pullbase healthy and resolve common issues quickly.

Operating Pullbase involves monitoring service health, responding to drift, and collecting diagnostics during incidents.

## Health checks

* **API health:** `curl http://localhost:8080/api/v1/healthz`
* **Database (SQLite):** Check the database file exists and is accessible
* **Database (PostgreSQL):** `docker compose exec db pg_isready -U pullbaseuser -d pullbasedb`
* **Web UI:** Sign in and confirm the dashboard loads recent activity.
* **Agents:** List servers and check `last_timestamp` in status history via CLI or UI.

## Routine operations checklist

<CardGroup cols={2}>
  <Card title="Daily" icon="calendar">
    * Review drift alerts from status history
    * Check audit logs for unexpected actions
    * Ensure webhooks (if enabled) are succeeding
  </Card>

  <Card title="Weekly" icon="rotate">
    * Rotate admin tokens or ensure expiration dates are set
    * Back up the database (SQLite file or PostgreSQL dump)
    * Verify TLS certificates (at reverse proxy) aren't approaching expiry
  </Card>

  <Card title="Monthly" icon="shield-halved">
    * Test agent token rotation on a sample server
    * Exercise rollback on a staging environment
    * Review GitHub App access scopes
  </Card>

  <Card title="Quarterly" icon="file-signature">
    * Run disaster recovery drill (bootstrap new server + restore DB)
    * Patch base OS/images used for agents and central server hosts
  </Card>
</CardGroup>

## Troubleshooting guide

<AccordionGroup>
  <Accordion title="Bootstrap secret already used">
    * Delete `bootstrap-admin-secret.txt` inside the config directory and restart to regenerate.
    * Or provide a new secret via `PULLBASE_BOOTSTRAP_SECRET` environment variable.
    * Create a new admin via CLI if another admin exists: `pullbasectl users create`
  </Accordion>

  <Accordion title="Agents reporting 401 Unauthorized">
    * Token revoked? Generate a new token and update the deployment.
    * Clock skew? Ensure NTP is running—JWT validation is time-sensitive.
    * Incorrect `SERVER_ID`? Confirm environment variable matches the server record.
  </Accordion>

  <Accordion title="Git clone failures">
    * Check outbound network connectivity to the Git provider.
    * For GitHub Apps, verify installation scopes and ensure the private key matches the app.
    * Confirm agents trust the TLS certificate (`CACERT_PATH` points to the CA bundle).
  </Accordion>

  <Accordion title="Service manager not detected">
    * Override auto-detection by setting `system.serviceManager` in `config.yaml`.
    * For Docker containers, use `supervisor` as the service manager.
    * Ensure service manager commands are in the agent's PATH.
  </Accordion>

  <Accordion title="Database migration errors">
    * Review container logs around startup for SQL errors.
    * Migrations run automatically for both SQLite and PostgreSQL.
    * Check for schema drift caused by manual database edits.
    * Restore from backup if migrations fail due to corruption.
  </Accordion>

  <Accordion title="Webhook retries failing">
    * Inspect container logs: `docker compose logs pullbase`
    * Ensure the webhook secret in GitHub matches `PULLBASE_WEBHOOK_SECRET_KEY`.
    * Confirm Pullbase is reachable from your Git provider (network ACLs, firewalls).
  </Accordion>

  <Accordion title="TLS certificate permission denied">
    When TLS is enabled, the server attempts to generate self-signed certificates on startup. If you see:

    ```
    failed to generate self-signed certificates: mkdir certs: permission denied
    ```

    **Option 1: Disable TLS** (recommended if behind a reverse proxy or for testing):

    ```bash theme={null}
    echo "PULLBASE_TLS_ENABLED=false" | sudo tee -a /etc/pullbase/pullbase.env
    sudo systemctl restart pullbase
    ```

    **Option 2: Create the certs directory with proper permissions**:

    ```bash theme={null}
    sudo mkdir -p /var/lib/pullbase/certs
    sudo chown pullbase:pullbase /var/lib/pullbase/certs

    # Update env to use this path
    sudo tee -a /etc/pullbase/pullbase.env > /dev/null <<'EOF'
    PULLBASE_TLS_CERT_PATH=/var/lib/pullbase/certs/server.crt
    PULLBASE_TLS_KEY_PATH=/var/lib/pullbase/certs/server.key
    EOF

    sudo systemctl restart pullbase
    ```

    For production, use CA-signed certificates or terminate TLS at a reverse proxy.
  </Accordion>

  <Accordion title="Exec format error">
    If you see `Failed to execute /usr/local/bin/pullbase-server: Exec format error`:

    You downloaded the wrong binary for your system architecture.

    ```bash theme={null}
    # Check your architecture
    uname -m
    ```

    | Output | Binary to download |
    | - | - |
    | `x86_64` | `pullbase-server-linux-amd64` |
    | `aarch64` or `arm64` | `pullbase-server-linux-arm64` |

    Download the correct binary and reinstall:

    ```bash theme={null}
    sudo systemctl stop pullbase
    curl -fsSL -o pullbase-server "https://github.com/pullbase/pullbase/releases/latest/download/pullbase-server-linux-arm64"
    sudo install -m 0755 pullbase-server /usr/local/bin/pullbase-server
    sudo systemctl start pullbase
    ```
  </Accordion>
</AccordionGroup>

## Collect diagnostics

```bash theme={null}
# Pullbase server logs
docker compose logs pullbase --since 1h

# Database logs
docker compose logs db --since 1h

# Agent logs (container)
docker logs pullbase-agent --since 1h

# Agent logs (systemd)
journalctl -u pullbase-agent --since "1 hour ago"
```

<Steps>
  <Step title="Export environment metadata">
    ```bash theme={null}
    curl -H "Authorization: Bearer $ADMIN_TOKEN" \
      http://localhost:8080/api/v1/environments
    ```
  </Step>

  <Step title="Export server status history">
    ```bash theme={null}
    curl -H "Authorization: Bearer $ADMIN_TOKEN" \
      http://localhost:8080/api/v1/servers/web-01/status/history
    ```
  </Step>

  <Step title="Capture database snapshot">
    **SQLite:**

    ```bash theme={null}
    cp /data/pullbase.db pullbase-$(date +%F).db
    ```

    **PostgreSQL:**

    ```bash theme={null}
    docker compose exec db \
      pg_dump -U pullbaseuser pullbasedb \
      > pullbasedb-$(date +%F).sql
    ```
  </Step>
</Steps>

## Resetting the installation

If you need a clean slate (lab environments):

1. Stop services: `docker compose down`
2. Remove data **only if you accept data loss**:
   * **SQLite:** `docker volume rm pullbase_pullbase_data` or delete the database file
   * **PostgreSQL:** `docker volume rm pullbase_postgres_data`
3. Start again: `docker compose up -d`

<Warning>
  Removing the database wipes environments, servers, tokens, and audit history. Always back up before destructive operations.
</Warning>

## Getting help

* Review [GitHub issues](https://github.com/pullbase/pullbase/issues) for known bugs.
* Collect the diagnostic bundle above before filing a bug report.
* Provide Pullbase version and deployment details.

Keep this playbook handy so you can respond quickly when drift occurs or when an agent experiences connectivity issues.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.