> ## Documentation Index
> Fetch the complete documentation index at: https://backstage.spotify.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Managing Portal Connect

> Configure, monitor, and troubleshoot Portal Connect in production

This guide covers everything you need to configure, operate, monitor, and troubleshoot Portal Connect in production.

## Configuring Plugins

Once you have Portal Connect Agent deployed and running, you can configure your Backstage plugins to access internal services through the Agent. The key principle is simple: **configure plugins as if they could access the services directly within the same network**.

### How It Works

When you configure a Backstage plugin to access an internal service:

1. Use the internal hostname/URL directly in your plugin configuration (e.g., `https://github.internal.company.com`, `https://artifactory.corp.net`)
2. The Agent registers routes for these internal services with Portal
3. When Portal makes a request to that URL, Portal Connect automatically tunnels the request through the Agent
4. The Agent executes the request from within your network and returns the response

**No special configuration needed.** Portal Connect handles the routing transparently once the Agent has registered the route.

### Configuration Approach

**Internal Service URLs**

Configure your plugins with internal URLs exactly as you would if Portal were running inside your network:

**Example - GitHub Enterprise:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
github:
  - host: github.internal.company.com
    apiBaseUrl: https://github.internal.company.com/api/v3
    token: ${GITHUB_TOKEN} # or dummy value
```

**Example - Internal Artifactory:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
catalog:
  locations:
    - type: url
      target: https://artifactory.corp.net/api/storage/backstage/catalog-info.yaml
```

### Authentication and Credentials

For plugin configuration that requires authentication headers or API tokens:

**Option 1: Use dummy values in plugin configuration**

Since the Portal Connect Agent injects the real authentication headers (configured in the Agent's routes), you can use placeholder values in your plugin configuration:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
integrations:
  github:
    - host: github.internal.company.com
      token: dummy-token-not-used # Real token configured in Agent
```

**Option 2: Reference environment variables**

You can reference environment variables in your configuration, but the actual authentication will happen at the Agent level:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
integrations:
  github:
    - host: github.internal.company.com
      token: ${GITHUB_TOKEN} # Can be dummy value
```

The Agent configuration would include the real authentication:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
# Agent app-config.yaml
agent:
  routes:
    'https://github.internal.company.com/*':
      headers:
        authorization: 'Bearer ${REAL_GITHUB_TOKEN}'
```

### Multi-Step Configuration Example

**Step 1: Configure the Agent** with the route and real credentials:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
# Portal Connect Agent configuration
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    'https://github.internal.company.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_ENTERPRISE_TOKEN}'

    'https://artifactory.corp.net/*':
      headers:
        x-api-key: '${ARTIFACTORY_API_KEY}'

  mtls:
    clientCertPath: /certs/portal-connect-client-<timestamp>.crt
    clientKeyPath: /certs/portal-connect-client-<timestamp>.key
```

**Step 2: Configure your plugin** to use the internal URL:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
integrations:
  github:
    - host: github.internal.company.com
      apiBaseUrl: https://github.internal.company.com/api/v3
      token: placeholder # Real token injected by Agent
```

**Step 3:** Deploy or restart the Agent to register the routes.

**Step 4:** Your plugins now work seamlessly—Portal Connect handles the routing and authentication.

### Benefits of This Approach

* **Simple plugin configuration** - No special proxy settings or complex routing rules in Backstage
* **Centralized credential management** - All internal service credentials stay within your network at the Agent level
* **Security** - Credentials never leave your network perimeter
* **Transparency** - Plugins work as if Portal were running inside your network
* **Maintainability** - Plugin configurations remain clean and straightforward

## Adding and Managing Routes

This section covers how to add, modify, and remove routes in Portal Connect Agent.

### Understanding Routes

Routes define which external URLs the Agent can access on behalf of your applications.

**Route Components**

A route consists of:

* **Protocol:** `http://` or `https://`
* **Hostname:** Domain name or IP address
* **Path:** Optional path component (exact or wildcard)
* **Headers:** Optional per-route default headers (advanced)

**Route Matching**

**Exact match:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/users
```

* Matches only: `https://api.github.com/users`
* Does not match: `https://api.github.com/users/octocat`

**Wildcard match:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/*
```

* Matches: `https://api.github.com/users`
* Matches: `https://api.github.com/users/octocat`
* Matches: `https://api.github.com/repos/owner/repo`
* Does not match: `http://api.github.com/users` (different protocol)
* Does not match: `https://github.com/users` (different hostname)

**Route Precedence**

When multiple routes could match a URL:

1. Exact matches take precedence over wildcards
2. More specific wildcards take precedence over less specific
3. Routes are evaluated in the order configured

Example:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/users/octocat # Most specific (exact)
  - https://api.github.com/users/* # More specific wildcard
  - https://api.github.com/* # Less specific wildcard
```

Request to `https://api.github.com/users/octocat` matches the first route.

### Adding Routes

**Simple Routes**

To add routes without custom headers, use the array format:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com
  routes:
    - https://api.github.com/*
    - https://api.gitlab.com/*
    - https://internal-api.company.com/v1/*
```

**Routes with Headers**

To add per-route authentication or custom headers:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'
        accept: 'application/vnd.github.v3+json'

    'https://internal-api.company.com/v1/*':
      headers:
        x-api-key: '${INTERNAL_API_KEY}'
        x-forwarded-by: 'portal-connect-agent'
```

Set the referenced environment variables when starting the Agent:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
docker run -d \
  -e GITHUB_TOKEN="ghp_xxxxxxxxxxxx" \
  -e INTERNAL_API_KEY="api-key-xxxxxxxxxxxx" \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest
```

### Route Planning

Before adding routes, consider:

**Scope:** Use wildcards appropriately

* Too broad: `https://*/*` (avoid - too permissive)
* Too narrow: Individual endpoints (maintenance overhead)
* Recommended: API version level `https://api.example.com/v1/*`

**Authentication:** Does the service require headers?

* API keys
* Bearer tokens
* Custom authentication headers

**Security:** Should this route be accessible?

* Apply principle of least privilege
* Only add routes that are necessary

### Adding a New Route

**Step 1: Update configuration**

Edit `app-config.yaml`:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  routes:
    - https://api.github.com/*
    - https://api.example.com/* # New route
```

Or with headers:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'

    'https://api.example.com/*': # New route
      headers:
        x-api-key: '${EXAMPLE_API_KEY}'
```

**Step 2: Set environment variables** (if using headers)

Update your deployment to include new environment variables:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
# Docker
docker stop portal-connect-agent
docker rm portal-connect-agent
docker run -d \
  --name portal-connect-agent \
  -e GITHUB_TOKEN="..." \
  -e EXAMPLE_API_KEY="new-api-key-here" \
  -v $(pwd)/app-config.yaml:/app/config/app-config.yaml:ro \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest

# Docker Compose - update .env file, then:
docker-compose restart

# Kubernetes - update Secret, then:
kubectl create secret generic portal-connect-secrets \
  --from-literal=example-api-key='new-api-key' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/portal-connect-agent
```

**Step 3: Restart Agent**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
# Docker
docker restart portal-connect-agent

# Docker Compose
docker-compose restart

# Kubernetes
kubectl rollout restart deployment/portal-connect-agent
```

**Step 4: Verify route registration**

Check logs:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
docker logs portal-connect-agent | grep "Registered route"
```

Expected output:

```
Registered route: https://api.github.com/*
Registered route: https://api.example.com/*
```

Check metrics:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
curl -s http://localhost:9466/metrics | grep agent_routes_registered
```

Expected output:

```
agent_routes_registered{} 2
```

### Modifying Routes

**Changing Route Patterns**

To change a route pattern (e.g., from exact to wildcard):

Before:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/users
```

After:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/users/*
```

Restart the Agent to apply changes.

**Adding Headers to Existing Route**

To add headers to a route that previously had none:

Before:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/*
```

After:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'
```

Don't forget to:

1. Set the `GITHUB_TOKEN` environment variable
2. Restart the Agent

**Updating Headers**

To change header values, update the configuration and environment variables:

Configuration change:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'
      accept: 'application/vnd.github.v3+json' # New header
```

If changing token values, update the environment variable and restart.

**Converting Between Formats**

From array to object format:

Before:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/*
  - https://api.gitlab.com/*
```

After (adding headers to one route):

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'

  'https://api.gitlab.com/*':
    headers:
      private-token: '${GITLAB_TOKEN}'
```

<Warning>
  **Format Consistency**

  You cannot mix array and object formats. Choose one format for all routes.
</Warning>

### Route Security Best Practices

**Principle of Least Privilege**

Only add routes that are absolutely necessary:

❌ **Bad - too permissive:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://*/*
  - http://*/*
```

✅ **Good - specific routes:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.github.com/*
  - https://internal-api.company.com/v1/*
```

**Credential Management**

Never hardcode credentials:

❌ **Bad - credentials in config file:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ghp_AbCdEfGhIjKlMnOpQrStUvWxYz'
```

✅ **Good - use environment variables:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'
```

**Credential Rotation**

When rotating API credentials:

1. Generate new credential
2. Update environment variable or secret
3. Restart Agent
4. Revoke old credential

Kubernetes example:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
# Update secret with new token
kubectl create secret generic portal-connect-secrets \
  --from-literal=github-token='new-token-here' \
  --dry-run=client -o yaml | kubectl apply -f -

# Rolling restart to pick up new secret
kubectl rollout restart deployment/portal-connect-agent

# Verify Agent is healthy
kubectl get pods -l app=portal-connect-agent

# Once verified, revoke old token in GitHub
```

### Common Route Patterns

**GitHub API**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'
      accept: 'application/vnd.github.v3+json'
```

**GitLab API**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://gitlab.com/api/*':
    headers:
      private-token: '${GITLAB_TOKEN}'
```

**Internal REST API**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  'https://internal-api.company.com/v1/*':
    headers:
      x-api-key: '${INTERNAL_API_KEY}'
      x-forwarded-by: 'portal-connect-agent'
```

**Prometheus Metrics Endpoint**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://prometheus.company.com/api/v1/*
```

**Health Check Endpoints**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://service1.company.com/health
  - https://service2.company.com/health
  - https://service3.company.com/healthz
```

**Multiple API Versions**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.company.com/v1/*
  - https://api.company.com/v2/*
```

### Troubleshooting Routes

**Route Not Registered**

**Symptom:** Route doesn't appear in logs or metrics

**Diagnostic:**

1. Check configuration syntax:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   cat app-config.yaml
   ```

2. Validate YAML:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   python -c "import yaml; yaml.safe_load(open('app-config.yaml'))"
   ```

3. Check for configuration errors in logs:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker logs portal-connect-agent | grep -i error
   ```

**Solutions:**

* Fix YAML syntax errors
* Ensure routes are under `agent.routes` key
* Restart Agent after configuration changes

**Requests Not Matching Route**

**Symptom:** Requests returning 404 or not being processed

**Diagnostic:**

* Verify exact URL being requested
* Check route pattern matches the URL
* Review logs for rejected requests

**Common mismatches:**

* Protocol mismatch: Route is `https://` but request is `http://`
* Hostname mismatch: Different subdomain or domain
* Path mismatch: Wildcard doesn't cover the requested path

**Solutions:**

* Adjust route pattern to match intended URLs
* Use wildcards appropriately
* Add specific routes for edge cases

**Headers Not Being Sent**

**Symptom:** External service returns authentication errors

**Diagnostic:**

1. Check environment variables are set:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker exec portal-connect-agent env | grep TOKEN
   ```

2. Verify header syntax in configuration

3. Check logs for request details (debug level)

**Solutions:**

* Set missing environment variables
* Fix header syntax in configuration
* Ensure variable names match between config and environment

**Route with Wrong Credentials**

**Symptom:** Authentication failures for specific route

**Diagnostic:**

Test the credential directly:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
curl -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/user
```

**Solutions:**

* Update credential in environment variable
* Verify credential is valid and not expired
* Check credential has necessary permissions for the API

## Monitoring and Operations

### Metrics Endpoint

The Agent exposes Prometheus metrics on port 9466 (configurable via `AGENT_METRICS_PORT`).

<Info>
  **Metrics Port**

  The metrics endpoint is optional and only needs to be accessible from your monitoring infrastructure within your network. It does not require inbound access from the internet - the Agent still only makes outbound connections for its primary function.
</Info>

**Accessing Metrics:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
curl http://localhost:9466/metrics
```

### Key Metrics

**Connection Metrics**

**`agent_websocket_connected`**

* **Type:** Gauge
* **Description:** WebSocket connection state (0 = disconnected, 1 = connected)
* **Use:** Monitor gateway connectivity

```
agent_websocket_connected{} 1
```

**`agent_websocket_reconnections_total`**

* **Type:** Counter
* **Description:** Total number of reconnection attempts
* **Use:** Track connection stability

**Request Metrics**

**`agent_http_requests_total`**

* **Type:** Counter
* **Labels:** method, status\_code, route
* **Description:** Total HTTP requests processed
* **Use:** Track request volume and status codes

```
agent_http_requests_total{method="GET",status_code="200",route="https://api.github.com/*"} 152
agent_http_requests_total{method="POST",status_code="201",route="https://api.github.com/*"} 23
```

**`agent_http_request_duration_seconds`**

* **Type:** Histogram
* **Labels:** method, route
* **Description:** HTTP request duration distribution
* **Use:** Monitor request latency

```
agent_http_request_duration_seconds_bucket{method="GET",route="https://api.github.com/*",le="0.1"} 120
agent_http_request_duration_seconds_bucket{method="GET",route="https://api.github.com/*",le="0.5"} 145
agent_http_request_duration_seconds_sum{method="GET",route="https://api.github.com/*"} 18.5
agent_http_request_duration_seconds_count{method="GET",route="https://api.github.com/*"} 152
```

**`agent_http_request_errors_total`**

* **Type:** Counter
* **Labels:** method, route, error\_type
* **Description:** Total HTTP request errors
* **Use:** Monitor error rates

```
agent_http_request_errors_total{method="GET",route="https://api.github.com/*",error_type="timeout"} 3
agent_http_request_errors_total{method="GET",route="https://api.github.com/*",error_type="network"} 1
```

**Route Metrics**

**`agent_routes_registered`**

* **Type:** Gauge
* **Description:** Number of routes currently registered
* **Use:** Verify route configuration

```
agent_routes_registered{} 5
```

### Alerting

**Recommended Alerts**

**Critical Alerts**

**WebSocket Disconnected:**

```
agent_websocket_connected{} == 0
```

* **Threshold:** Alert if disconnected for > 5 minutes
* **Action:** Check network connectivity, Portal status, certificate validity

**Certificate Expiring Soon:**

```
agent_certificate_expiry_seconds{} < 604800  # 7 days
```

* **Threshold:** Alert 7 days before expiration
* **Action:** Generate and register new certificate

**High Error Rate:**

```
(
  sum(rate(agent_http_request_errors_total[5m]))
  /
  sum(rate(agent_http_requests_total[5m]))
) > 0.05  # 5% error rate
```

* **Threshold:** > 5% error rate over 5 minutes
* **Action:** Investigate logs, check external service status

**Agent Down:**

```
up{job="portal-connect-agent"} == 0
```

* **Threshold:** Alert immediately
* **Action:** Restart Agent, check logs for crash reason

**Warning Alerts**

**Elevated Error Rate:**

```
(
  sum(rate(agent_http_request_errors_total[15m]))
  /
  sum(rate(agent_http_requests_total[15m]))
) > 0.01  # 1% error rate
```

* **Threshold:** > 1% error rate over 15 minutes
* **Action:** Monitor, investigate if sustained

**Slow Requests:**

```
histogram_quantile(0.95,
  rate(agent_http_request_duration_seconds_bucket[5m])
) > 5  # 5 seconds
```

* **Threshold:** P95 latency > 5 seconds
* **Action:** Check external service performance

**Frequent Reconnections:**

```
rate(agent_websocket_reconnections_total[15m]) > 0.1
```

* **Threshold:** > 0.1 reconnections/second
* **Action:** Investigate network stability

### Monitoring WebSocket Connection

The Agent automatically reconnects if the WebSocket connection drops:

* Initial retry: 1 second
* Maximum retry interval: 60 seconds
* Uses exponential backoff with jitter

Monitor connection stability:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
# Count disconnection events in last hour
docker logs --since 1h portal-connect-agent | grep -c "WebSocket disconnected"
```

### Monitoring Request Success

View request metrics:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
curl -s http://localhost:9466/metrics | grep agent_http_requests_total
```

Calculate error rate:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
# View error count
curl -s http://localhost:9466/metrics | grep agent_http_request_errors_total
```

### Health Checks

**Docker Health Check**

```dockerfile theme={"theme":{"light":"github-light","dark":"dracula"}}
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:9466/metrics || exit 1
```

**Kubernetes Probes**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
livenessProbe:
  httpGet:
    path: /metrics
    port: 9466
  initialDelaySeconds: 10
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /metrics
    port: 9466
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3
```

## Troubleshooting

### Connection Issues

**WebSocket Connection Fails**

**Symptoms:**

* `agent_websocket_connected{} == 0`
* Logs show `WebSocket connection failed` or `Connection refused`

**Diagnostic Steps:**

1. Verify WebSocket URL:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   grep brokerUrl app-config.yaml
   ```

2. Test network connectivity:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   curl -v wss://broker.${subdomain}.spotifyportal.com
   ```

3. Check firewall rules:

   * Ensure outbound WSS (port 443 or 8080) allowed
   * Test from Agent host

4. Verify certificates:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   openssl x509 -in /certs/client.crt -noout -dates
   ```

**Solutions:**

* Fix WebSocket URL in configuration
* Update firewall rules to allow outbound WSS
* Renew expired certificates
* Contact Spotify to verify Portal is operational

**Frequent Disconnections**

**Symptoms:**

* `agent_websocket_reconnections_total` increasing
* Logs show repeated disconnect/reconnect cycles

**Diagnostic Steps:**

1. Check connection stability:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   # Monitor reconnections over time
   curl -s http://localhost:9466/metrics | grep reconnections
   ```

2. Review network logs:

   * Check for intermittent network issues
   * Look for firewall or proxy interruptions

3. Verify Portal health:
   * Contact Spotify support
   * Check for Portal maintenance windows

**Solutions:**

* Resolve network stability issues
* Adjust firewall/proxy timeouts for WebSocket connections
* Coordinate with Spotify on Portal issues

**SSL Inspection / Corporate Proxy Intercepting TLS**

**Symptoms:**

* Repeated reconnection failures in logs
* Error message similar to: `Error: self-signed certificate in certificate chain`
* `agent_websocket_connected{} == 0` even though broker URL and network access are correct
* Other outbound HTTPS requests from the Agent may also fail

**Diagnostic Steps:**

1. **Check whether your organization performs SSL/TLS inspection**

   * Firewalls such as Palo Alto, Zscaler, Cisco, or Blue Coat may intercept and re-sign outbound TLS connections
   * If so, Portal Connect's WebSocket connection will be re-signed using a corporate CA that is not trusted by Node.js by default

2. **Inspect Agent logs for certificate errors:**

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker logs portal-connect-agent | grep -i "self-signed"
   ```

   or:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   kubectl logs deployment/portal-connect-agent | grep -i certificate
   ```

3. **Verify whether Node.js trusts your corporate CA**

   Inside the Agent container:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker exec portal-connect-agent node -p "require('tls').rootCertificates"
   ```

   If your corporate CA is not listed, Node.js will reject the connection.

4. **Ensure you have the correct TLS inspection certificates**
   * Export the root + intermediate CA certificates from your firewall
   * Certificates must be PEM-encoded:
     ```
     -----BEGIN CERTIFICATE-----
     ...
     -----END CERTIFICATE-----
     ```

**Solutions:**

1. **Create a CA bundle** containing your corporate inspection certificates:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   cat root-ca.pem intermediate-ca.pem > custom-ca-bundle.pem
   ```

2. **Mount the CA bundle** into the Agent container

   Example:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   -v ./certs:/certs:ro
   ```

3. **Tell Node.js to trust your corporate CA** using `NODE_EXTRA_CA_CERTS`

   **Docker run:**

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   -e NODE_EXTRA_CA_CERTS=/certs/custom-ca-bundle.pem
   ```

   **Docker Compose:**

   ```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
   environment:
     - NODE_EXTRA_CA_CERTS=/certs/custom-ca-bundle.pem
   ```

   **Kubernetes:**

   ```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
   env:
     - name: NODE_EXTRA_CA_CERTS
       value: /etc/portal-connect/custom-ca-bundle.pem
   ```

4. **Restart or redeploy the Agent:**

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker restart portal-connect-agent
   ```

   or:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   kubectl rollout restart deployment/portal-connect-agent
   ```

5. **Verify WebSocket connectivity:**

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   curl http://localhost:9466/metrics | grep agent_websocket_connected
   ```

   Expected:

   ```
   agent_websocket_connected{} 1
   ```

### Request Failures

**All Requests Failing**

**Symptoms:**

* High error rate across all routes
* Logs show consistent request failures

**Diagnostic Steps:**

1. Check Agent connectivity:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker logs portal-connect-agent | grep "WebSocket connected"
   ```

2. Verify routes are registered:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   curl http://gateway:8080/routes
   ```

3. Test external service directly from Agent host:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker exec portal-connect-agent curl -v https://api.github.com
   ```

**Solutions:**

* Restore WebSocket connection
* Fix route configuration and restart Agent
* Resolve network access to external services

**Specific Route Failing**

**Symptoms:**

* Errors for specific route/URL pattern
* Other routes working normally

**Diagnostic Steps:**

1. Check route configuration:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   grep -A5 "routes:" app-config.yaml
   ```

2. Test service directly:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   curl -v https://failing-service.com/endpoint
   ```

3. Check service-specific authentication:
   * Verify headers configured correctly
   * Test with actual credentials

**Solutions:**

* Fix route pattern (wildcard, exact match)
* Update authentication headers
* Verify external service is operational

**Timeout Errors**

**Symptoms:**

* Logs show `Network timeout` or `Request timeout`
* Slow response times

**Diagnostic Steps:**

1. Check request latency:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   curl -s http://localhost:9466/metrics | grep duration_seconds
   ```

2. Test external service response time:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   time curl https://external-service.com/endpoint
   ```

3. Review Agent resource usage:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker stats portal-connect-agent
   ```

**Solutions:**

* Increase timeout configuration (if configurable)
* Optimize external service performance
* Scale Agent resources if CPU/memory constrained

### Certificate Issues

**Certificate Validation Fails**

**Symptoms:**

* `mTLS certificate validation failed` in logs
* Agent fails to start

**Diagnostic Steps:**

1. Verify certificate files exist:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   ls -la /certs/
   ```

2. Check certificate validity:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   openssl x509 -in /certs/portal-connect-client-<timestamp>.crt \
     -noout -dates -subject -issuer
   ```

3. Verify certificate and key match:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   openssl x509 -in /certs/portal-connect-client-<timestamp>.crt \
     -noout -modulus | openssl md5
   openssl rsa -in /certs/portal-connect-client-<timestamp>.key \
     -noout -modulus | openssl md5
   ```

4. Check file permissions:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   stat /certs/portal-connect-client-<timestamp>.crt \
     /certs/portal-connect-client-<timestamp>.key
   ```

**Solutions:**

* Generate new certificates
* Fix file permissions (readable by container user)
* Ensure certificate is registered with Spotify
* Verify certificate and key are matching pair

**Certificate Expired**

**Symptoms:**

* `Certificate expired` in logs
* `agent_certificate_expiry_seconds` negative or very low

**Diagnostic:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
openssl x509 -in /certs/portal-connect-client-<timestamp>.crt -noout -enddate
```

**Solutions:**

1. Generate new certificate
2. Register with Spotify
3. Update certificate files
4. Restart Agent

### Configuration Issues

**Configuration File Not Found**

**Symptoms:**

* `Configuration file not found` or `ENOENT` error
* Agent exits immediately

**Diagnostic Steps:**

1. Check `CONFIG_PATH`:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker inspect portal-connect-agent | grep CONFIG_PATH
   ```

2. Verify volume mount:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker inspect portal-connect-agent | grep -A10 Mounts
   ```

3. Check file exists in container:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker exec portal-connect-agent ls -la /app/config/
   ```

**Solutions:**

* Fix `CONFIG_PATH` environment variable
* Fix volume mount path
* Verify file exists on host at mount source

**Invalid Configuration Syntax**

**Symptoms:**

* `Invalid configuration` or `YAML parse errors`
* Agent fails to start

**Diagnostic Steps:**

1. Validate YAML syntax:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   python -c "import yaml; yaml.safe_load(open('app-config.yaml'))"
   ```

2. Check for common errors:
   * Incorrect indentation
   * Missing required fields
   * Invalid route patterns

**Solutions:**

* Fix YAML syntax errors
* Validate against configuration reference
* Use YAML linter

**Environment Variable Not Substituted**

**Symptoms:**

* Headers contain literal `${VAR_NAME}`
* Authentication fails for external services

**Diagnostic Steps:**

1. Check environment variables are set:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker exec portal-connect-agent env | grep TOKEN
   ```

2. Review configuration:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker exec portal-connect-agent cat /app/config/app-config.yaml
   ```

**Solutions:**

* Set missing environment variables when starting container
* Fix environment variable name mismatch
* Use correct syntax: `${VAR_NAME}` for env vars, `${VAR_NAME_FILE}` for file-based

### Resource Issues

**High CPU Usage**

**Symptoms:**

* Container using > 80% CPU consistently
* Slow request processing

**Diagnostic:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
docker stats portal-connect-agent
```

**Solutions:**

* Increase CPU limits
* Scale horizontally (add more Agent replicas)
* Investigate unusually high request volume

**High Memory Usage**

**Symptoms:**

* Container approaching memory limit
* `OOMKilled` events (Kubernetes)

**Diagnostic:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
docker stats portal-connect-agent
kubectl describe pod portal-connect-agent-xxx
```

**Solutions:**

* Increase memory limits
* Check for memory leaks (contact Spotify support)
* Restart Agent to clear memory

**Container Keeps Restarting**

**Symptoms:**

* `docker ps` shows frequent restarts
* Pod in `CrashLoopBackOff` (Kubernetes)

**Diagnostic Steps:**

1. Check logs:

   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker logs portal-connect-agent
   ```

2. Check exit code:
   ```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
   docker inspect portal-connect-agent | grep ExitCode
   ```

**Solutions:**

* Fix configuration errors causing startup failure
* Resolve certificate validation issues
* Check resource limits not too restrictive
* Review logs for specific error messages

## Configuration Reference

This reference documents all configuration options for the Portal Connect Agent.

### Configuration File Format

The Portal Connect Agent uses YAML format for configuration. The default configuration file location is `app-config.yaml`, but you can specify a custom path using the `CONFIG_PATH` environment variable.

### Configuration Structure

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: string (required)
  routes: array or object (required)
  mtls: object (optional, enabled by default)
```

### Configuration Options

#### agent.brokerUrl

* **Type:** String (required)
* **Description:** WebSocket URL to connect the Agent to your Portal instance
* **Format:** `wss://broker.${subdomain}.spotifyportal.com`

**Example:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com
```

**Notes:**

* Must use `wss://` protocol (secure WebSocket)
* Connection URL provided by Spotify
* Do not modify the path component

#### agent.routes

* **Type:** Array or Object (required)
* **Description:** Defines which external URLs the Agent can access

**Simple Array Format**

Use this format for routes without custom headers:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  routes:
    - https://api.github.com/*
    - https://internal-api.company.com/v1/*
    - https://status.company.com/health
```

**Advanced Object Format**

Use this format to specify per-route headers:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'

    'https://internal-api.company.com/v1/*':
      headers:
        x-api-key: '${INTERNAL_API_KEY}'
        x-forwarded-by: 'portal-connect-agent'
```

**Route Patterns**

**Exact match:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.example.com/v1/users
```

* Matches only: `https://api.example.com/v1/users`

**Wildcard match:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.example.com/v1/*
```

* Matches:
  * `https://api.example.com/v1/users`
  * `https://api.example.com/v1/users/123`
  * `https://api.example.com/v1/anything`
* Does not match:
  * `https://api.example.com/v2/users` (different path)
  * `http://api.example.com/v1/users` (different protocol)

**Route Requirements**

* Must include protocol (`http://` or `https://`)
* Must include hostname
* Path is optional but recommended
* Wildcards (`*`) only supported at end of path
* No query parameters or fragments in route patterns

**Valid routes:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - https://api.example.com/*
  - https://api.example.com/v1/*
  - https://api.example.com/v1/users
  - http://legacy-api.company.com/health
```

**Invalid routes:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
routes:
  - api.example.com/* # Missing protocol
  - https://*.example.com/* # Wildcard in hostname not supported
  - https://api.example.com/*/users # Wildcard in middle of path
```

#### agent.mtls

* **Type:** Object (optional, enabled by default)
* **Description:** Mutual TLS (mTLS) configuration for secure authentication

**agent.mtls.enabled**

* **Type:** Boolean (optional)
* **Default:** `true`
* **Description:** Enable or disable mTLS authentication

<Warning>
  **Production Use**

  mTLS is required for production use. Only disable for local development.
</Warning>

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  mtls:
    enabled: true
```

**agent.mtls.clientCertPath**

* **Type:** String (required when mTLS enabled)
* **Description:** Path to client certificate file in PEM format

**Example:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  mtls:
    clientCertPath: /etc/ssl/certs/portal-connect-client.crt
```

**Notes:**

* Must be PEM-encoded X.509 certificate
* Maximum certificate lifespan: 180 days
* Certificate must be registered with Spotify

**agent.mtls.clientKeyPath**

* **Type:** String (required when mTLS enabled)
* **Description:** Path to private key file in PEM format

**Example:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  mtls:
    clientKeyPath: /etc/ssl/private/portal-connect-client.key
```

**Notes:**

* Must be PEM-encoded RSA or ECDSA private key
* For ECDSA: P-256 or P-384 elliptical curve (P-384 recommended)
* For RSA: Minimum 2048-bit key size (4096-bit recommended)
* Keep file secure with restricted permissions (400 or 600)

**agent.mtls.validateOnStartup**

* **Type:** Boolean (optional)
* **Default:** `true`
* **Description:** Validate certificates when Agent starts

**Example:**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  mtls:
    validateOnStartup: true
```

**Notes:**

* Recommended to keep enabled
* Startup will fail if certificates are invalid or expired
* Helps catch certificate issues early

### Environment Variables

**CONFIG\_PATH**

* **Type:** String
* **Default:** `./app-config.yaml`
* **Description:** Path to the Agent configuration file

**Example:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
-e CONFIG_PATH=/app/config/app-config.yaml
```

**AGENT\_METRICS\_PORT**

* **Type:** Integer
* **Default:** `9466`
* **Description:** Port for Prometheus metrics endpoint

**Example:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
-e AGENT_METRICS_PORT=9466
```

**LOG\_LEVEL**

* **Type:** String
* **Default:** `info`
* **Valid values:** `debug`, `info`, `warn`, `error`
* **Description:** Logging verbosity level

**Example:**

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
-e LOG_LEVEL=debug
```

### Environment Variable Substitution

Configuration files support environment variable substitution for sensitive values.

**Using Environment Variables**

Reference environment variables in configuration with `${VAR_NAME}`:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'
```

Set the environment variable when running the container:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
docker run -d \
  -e GITHUB_TOKEN="ghp_xxxxxxxxxxxx" \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest
```

**File-Based Secrets**

For secrets stored in files (e.g., Kubernetes secrets), use `${VAR_NAME_FILE}`:

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN_FILE}'
```

The Agent will read the secret from the file path:

```bash theme={"theme":{"light":"github-light","dark":"dracula"}}
docker run -d \
  -e GITHUB_TOKEN_FILE="/run/secrets/github_token" \
  -v /path/to/secrets:/run/secrets \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest
```

### Complete Configuration Examples

**Minimal Configuration**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    - https://api.github.com/*

  mtls:
    clientCertPath: /certs/client.crt
    clientKeyPath: /certs/client.key
```

**Configuration with Multiple Routes**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    - https://api.github.com/*
    - https://api.gitlab.com/*
    - https://internal-api.company.com/v1/*
    - https://metrics.company.com/prometheus/*
    - https://legacy-service.company.com/health

  mtls:
    enabled: true
    clientCertPath: /etc/ssl/certs/portal-connect-client.crt
    clientKeyPath: /etc/ssl/private/portal-connect-client.key
    validateOnStartup: true
```

**Configuration with Authentication Headers**

```yaml theme={"theme":{"light":"github-light","dark":"dracula"}}
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'
        accept: 'application/vnd.github.v3+json'

    'https://api.gitlab.com/*':
      headers:
        private-token: '${GITLAB_TOKEN}'

    'https://internal-api.company.com/v1/*':
      headers:
        x-api-key: '${INTERNAL_API_KEY}'
        x-forwarded-by: 'portal-connect-agent'
        x-environment: 'production'

  mtls:
    clientCertPath: /certs/client.crt
    clientKeyPath: /certs/client.key
```