Managing Portal Connect - Spotify for Backstage

This guide covers everything you need to configure, operate, monitor, and troubleshoot Portal Connect in production.

Configuring Plugins

Once you have Portal Connect Agent deployed and running, you can configure your Backstage plugins to access internal services through the Agent. The key principle is simple: configure plugins as if they could access the services directly within the same network.

How It Works

When you configure a Backstage plugin to access an internal service:

Use the internal hostname/URL directly in your plugin configuration (e.g., https://github.internal.company.com, https://artifactory.corp.net)
The Agent registers routes for these internal services with Portal
When Portal makes a request to that URL, Portal Connect automatically tunnels the request through the Agent
The Agent executes the request from within your network and returns the response

No special configuration needed. Portal Connect handles the routing transparently once the Agent has registered the route.

Configuration Approach

Internal Service URLs Configure your plugins with internal URLs exactly as you would if Portal were running inside your network: Example - GitHub Enterprise:

github:
  - host: github.internal.company.com
    apiBaseUrl: https://github.internal.company.com/api/v3
    token: ${GITHUB_TOKEN} # or dummy value

Example - Internal Artifactory:

catalog:
  locations:
    - type: url
      target: https://artifactory.corp.net/api/storage/backstage/catalog-info.yaml

Authentication and Credentials

For plugin configuration that requires authentication headers or API tokens: Option 1: Use dummy values in plugin configuration Since the Portal Connect Agent injects the real authentication headers (configured in the Agent’s routes), you can use placeholder values in your plugin configuration:

integrations:
  github:
    - host: github.internal.company.com
      token: dummy-token-not-used # Real token configured in Agent

Option 2: Reference environment variables You can reference environment variables in your configuration, but the actual authentication will happen at the Agent level:

integrations:
  github:
    - host: github.internal.company.com
      token: ${GITHUB_TOKEN} # Can be dummy value

The Agent configuration would include the real authentication:

# Agent app-config.yaml
agent:
  routes:
    'https://github.internal.company.com/*':
      headers:
        authorization: 'Bearer ${REAL_GITHUB_TOKEN}'

Multi-Step Configuration Example

Step 1: Configure the Agent with the route and real credentials:

# Portal Connect Agent configuration
agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    'https://github.internal.company.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_ENTERPRISE_TOKEN}'

    'https://artifactory.corp.net/*':
      headers:
        x-api-key: '${ARTIFACTORY_API_KEY}'

  mtls:
    clientCertPath: /certs/portal-connect-client-<timestamp>.crt
    clientKeyPath: /certs/portal-connect-client-<timestamp>.key

Step 2: Configure your plugin to use the internal URL:

integrations:
  github:
    - host: github.internal.company.com
      apiBaseUrl: https://github.internal.company.com/api/v3
      token: placeholder # Real token injected by Agent

Step 3: Deploy or restart the Agent to register the routes. Step 4: Your plugins now work seamlessly—Portal Connect handles the routing and authentication.

Benefits of This Approach

Simple plugin configuration - No special proxy settings or complex routing rules in Backstage
Centralized credential management - All internal service credentials stay within your network at the Agent level
Security - Credentials never leave your network perimeter
Transparency - Plugins work as if Portal were running inside your network
Maintainability - Plugin configurations remain clean and straightforward

Adding and Managing Routes

This section covers how to add, modify, and remove routes in Portal Connect Agent.

Understanding Routes

Routes define which external URLs the Agent can access on behalf of your applications. Route Components A route consists of:

Protocol: http:// or https://
Hostname: Domain name or IP address
Path: Optional path component (exact or wildcard)
Headers: Optional per-route default headers (advanced)

Route Matching Exact match:

routes:
  - https://api.github.com/users

Matches only: https://api.github.com/users
Does not match: https://api.github.com/users/octocat

Wildcard match:

routes:
  - https://api.github.com/*

Matches: https://api.github.com/users
Matches: https://api.github.com/users/octocat
Matches: https://api.github.com/repos/owner/repo
Does not match: http://api.github.com/users (different protocol)
Does not match: https://github.com/users (different hostname)

Route Precedence When multiple routes could match a URL:

Exact matches take precedence over wildcards
More specific wildcards take precedence over less specific
Routes are evaluated in the order configured

Example:

routes:
  - https://api.github.com/users/octocat # Most specific (exact)
  - https://api.github.com/users/* # More specific wildcard
  - https://api.github.com/* # Less specific wildcard

Request to https://api.github.com/users/octocat matches the first route.

Adding Routes

Simple Routes To add routes without custom headers, use the array format:

agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com
  routes:
    - https://api.github.com/*
    - https://api.gitlab.com/*
    - https://internal-api.company.com/v1/*

Routes with Headers To add per-route authentication or custom headers:

agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'
        accept: 'application/vnd.github.v3+json'

    'https://internal-api.company.com/v1/*':
      headers:
        x-api-key: '${INTERNAL_API_KEY}'
        x-forwarded-by: 'portal-connect-agent'

Set the referenced environment variables when starting the Agent:

docker run -d \
  -e GITHUB_TOKEN="ghp_xxxxxxxxxxxx" \
  -e INTERNAL_API_KEY="api-key-xxxxxxxxxxxx" \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest

Route Planning

Before adding routes, consider: Scope: Use wildcards appropriately

Too broad: https://*/* (avoid - too permissive)
Too narrow: Individual endpoints (maintenance overhead)
Recommended: API version level https://api.example.com/v1/*

Authentication: Does the service require headers?

API keys
Bearer tokens
Custom authentication headers

Security: Should this route be accessible?

Apply principle of least privilege
Only add routes that are necessary

Adding a New Route

Step 1: Update configuration Edit app-config.yaml:

agent:
  routes:
    - https://api.github.com/*
    - https://api.example.com/* # New route

Or with headers:

agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'

    'https://api.example.com/*': # New route
      headers:
        x-api-key: '${EXAMPLE_API_KEY}'

Step 2: Set environment variables (if using headers) Update your deployment to include new environment variables:

# Docker
docker stop portal-connect-agent
docker rm portal-connect-agent
docker run -d \
  --name portal-connect-agent \
  -e GITHUB_TOKEN="..." \
  -e EXAMPLE_API_KEY="new-api-key-here" \
  -v $(pwd)/app-config.yaml:/app/config/app-config.yaml:ro \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest

# Docker Compose - update .env file, then:
docker-compose restart

# Kubernetes - update Secret, then:
kubectl create secret generic portal-connect-secrets \
  --from-literal=example-api-key='new-api-key' \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/portal-connect-agent

Step 3: Restart Agent

# Docker
docker restart portal-connect-agent

# Docker Compose
docker-compose restart

# Kubernetes
kubectl rollout restart deployment/portal-connect-agent

Step 4: Verify route registration Check logs:

docker logs portal-connect-agent | grep "Registered route"

Expected output:

Registered route: https://api.github.com/*
Registered route: https://api.example.com/*

Check metrics:

curl -s http://localhost:9466/metrics | grep agent_routes_registered

Expected output:

agent_routes_registered{} 2

Modifying Routes

Changing Route Patterns To change a route pattern (e.g., from exact to wildcard): Before:

routes:
  - https://api.github.com/users

After:

routes:
  - https://api.github.com/users/*

Restart the Agent to apply changes. Adding Headers to Existing Route To add headers to a route that previously had none: Before:

routes:
  - https://api.github.com/*

After:

routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'

Don’t forget to:

Set the GITHUB_TOKEN environment variable
Restart the Agent

Updating Headers To change header values, update the configuration and environment variables: Configuration change:

routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'
      accept: 'application/vnd.github.v3+json' # New header

If changing token values, update the environment variable and restart. Converting Between Formats From array to object format: Before:

routes:
  - https://api.github.com/*
  - https://api.gitlab.com/*

After (adding headers to one route):

routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'

  'https://api.gitlab.com/*':
    headers:
      private-token: '${GITLAB_TOKEN}'

Format ConsistencyYou cannot mix array and object formats. Choose one format for all routes.

Route Security Best Practices

Principle of Least Privilege Only add routes that are absolutely necessary: ❌ Bad - too permissive:

routes:
  - https://*/*
  - http://*/*

✅ Good - specific routes:

routes:
  - https://api.github.com/*
  - https://internal-api.company.com/v1/*

Credential Management Never hardcode credentials: ❌ Bad - credentials in config file:

routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ghp_AbCdEfGhIjKlMnOpQrStUvWxYz'

✅ Good - use environment variables:

routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'

Credential Rotation When rotating API credentials:

Generate new credential
Update environment variable or secret
Restart Agent
Revoke old credential

Kubernetes example:

# Update secret with new token
kubectl create secret generic portal-connect-secrets \
  --from-literal=github-token='new-token-here' \
  --dry-run=client -o yaml | kubectl apply -f -

# Rolling restart to pick up new secret
kubectl rollout restart deployment/portal-connect-agent

# Verify Agent is healthy
kubectl get pods -l app=portal-connect-agent

# Once verified, revoke old token in GitHub

Common Route Patterns

GitHub API

routes:
  'https://api.github.com/*':
    headers:
      authorization: 'Bearer ${GITHUB_TOKEN}'
      accept: 'application/vnd.github.v3+json'

GitLab API

routes:
  'https://gitlab.com/api/*':
    headers:
      private-token: '${GITLAB_TOKEN}'

Internal REST API

routes:
  'https://internal-api.company.com/v1/*':
    headers:
      x-api-key: '${INTERNAL_API_KEY}'
      x-forwarded-by: 'portal-connect-agent'

Prometheus Metrics Endpoint

routes:
  - https://prometheus.company.com/api/v1/*

Health Check Endpoints

routes:
  - https://service1.company.com/health
  - https://service2.company.com/health
  - https://service3.company.com/healthz

Multiple API Versions

routes:
  - https://api.company.com/v1/*
  - https://api.company.com/v2/*

Troubleshooting Routes

Route Not Registered Symptom: Route doesn’t appear in logs or metrics Diagnostic:

Check configuration syntax:
```
cat app-config.yaml
```

Validate YAML:

python -c "import yaml; yaml.safe_load(open('app-config.yaml'))"

Check for configuration errors in logs:

docker logs portal-connect-agent | grep -i error

Solutions:

Fix YAML syntax errors
Ensure routes are under agent.routes key
Restart Agent after configuration changes

Requests Not Matching Route Symptom: Requests returning 404 or not being processed Diagnostic:

Verify exact URL being requested
Check route pattern matches the URL
Review logs for rejected requests

Common mismatches:

Protocol mismatch: Route is https:// but request is http://
Hostname mismatch: Different subdomain or domain
Path mismatch: Wildcard doesn’t cover the requested path

Solutions:

Adjust route pattern to match intended URLs
Use wildcards appropriately
Add specific routes for edge cases

Headers Not Being Sent Symptom: External service returns authentication errors Diagnostic:

Check environment variables are set:

docker exec portal-connect-agent env | grep TOKEN

Verify header syntax in configuration
Check logs for request details (debug level)

Solutions:

Set missing environment variables
Fix header syntax in configuration
Ensure variable names match between config and environment

Route with Wrong Credentials Symptom: Authentication failures for specific route Diagnostic: Test the credential directly:

curl -H "Authorization: Bearer ${GITHUB_TOKEN}" https://api.github.com/user

Solutions:

Update credential in environment variable
Verify credential is valid and not expired
Check credential has necessary permissions for the API

Monitoring and Operations

Metrics Endpoint

The Agent exposes Prometheus metrics on port 9466 (configurable via AGENT_METRICS_PORT).

Metrics PortThe metrics endpoint is optional and only needs to be accessible from your monitoring infrastructure within your network. It does not require inbound access from the internet - the Agent still only makes outbound connections for its primary function.

Accessing Metrics:

curl http://localhost:9466/metrics

Key Metrics

Connection Metrics agent_websocket_connected

Type: Gauge
Description: WebSocket connection state (0 = disconnected, 1 = connected)
Use: Monitor gateway connectivity

agent_websocket_connected{} 1

agent_websocket_reconnections_total

Type: Counter
Description: Total number of reconnection attempts
Use: Track connection stability

Request Metrics agent_http_requests_total

Type: Counter
Labels: method, status_code, route
Description: Total HTTP requests processed
Use: Track request volume and status codes

agent_http_requests_total{method="GET",status_code="200",route="https://api.github.com/*"} 152
agent_http_requests_total{method="POST",status_code="201",route="https://api.github.com/*"} 23

agent_http_request_duration_seconds

Type: Histogram
Labels: method, route
Description: HTTP request duration distribution
Use: Monitor request latency

agent_http_request_duration_seconds_bucket{method="GET",route="https://api.github.com/*",le="0.1"} 120
agent_http_request_duration_seconds_bucket{method="GET",route="https://api.github.com/*",le="0.5"} 145
agent_http_request_duration_seconds_sum{method="GET",route="https://api.github.com/*"} 18.5
agent_http_request_duration_seconds_count{method="GET",route="https://api.github.com/*"} 152

agent_http_request_errors_total

Type: Counter
Labels: method, route, error_type
Description: Total HTTP request errors
Use: Monitor error rates

agent_http_request_errors_total{method="GET",route="https://api.github.com/*",error_type="timeout"} 3
agent_http_request_errors_total{method="GET",route="https://api.github.com/*",error_type="network"} 1

Route Metrics agent_routes_registered

Type: Gauge
Description: Number of routes currently registered
Use: Verify route configuration

agent_routes_registered{} 5

Alerting

Recommended Alerts Critical Alerts WebSocket Disconnected:

agent_websocket_connected{} == 0

Threshold: Alert if disconnected for > 5 minutes
Action: Check network connectivity, Portal status, certificate validity

Certificate Expiring Soon:

agent_certificate_expiry_seconds{} < 604800  # 7 days

Threshold: Alert 7 days before expiration
Action: Generate and register new certificate

High Error Rate:

(
  sum(rate(agent_http_request_errors_total[5m]))
  /
  sum(rate(agent_http_requests_total[5m]))
) > 0.05  # 5% error rate

Threshold: > 5% error rate over 5 minutes
Action: Investigate logs, check external service status

Agent Down:

up{job="portal-connect-agent"} == 0

Threshold: Alert immediately
Action: Restart Agent, check logs for crash reason

Warning Alerts Elevated Error Rate:

(
  sum(rate(agent_http_request_errors_total[15m]))
  /
  sum(rate(agent_http_requests_total[15m]))
) > 0.01  # 1% error rate

Threshold: > 1% error rate over 15 minutes
Action: Monitor, investigate if sustained

Slow Requests:

histogram_quantile(0.95,
  rate(agent_http_request_duration_seconds_bucket[5m])
) > 5  # 5 seconds

Threshold: P95 latency > 5 seconds
Action: Check external service performance

Frequent Reconnections:

rate(agent_websocket_reconnections_total[15m]) > 0.1

Threshold: > 0.1 reconnections/second
Action: Investigate network stability

Monitoring WebSocket Connection

The Agent automatically reconnects if the WebSocket connection drops:

Initial retry: 1 second
Maximum retry interval: 60 seconds
Uses exponential backoff with jitter

Monitor connection stability:

# Count disconnection events in last hour
docker logs --since 1h portal-connect-agent | grep -c "WebSocket disconnected"

Monitoring Request Success

View request metrics:

curl -s http://localhost:9466/metrics | grep agent_http_requests_total

Calculate error rate:

# View error count
curl -s http://localhost:9466/metrics | grep agent_http_request_errors_total

Health Checks

Docker Health Check

HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:9466/metrics || exit 1

Kubernetes Probes

livenessProbe:
  httpGet:
    path: /metrics
    port: 9466
  initialDelaySeconds: 10
  periodSeconds: 30
  timeoutSeconds: 5
  failureThreshold: 3

readinessProbe:
  httpGet:
    path: /metrics
    port: 9466
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

Troubleshooting

Connection Issues

WebSocket Connection Fails Symptoms:

agent_websocket_connected{} == 0
Logs show WebSocket connection failed or Connection refused

Diagnostic Steps:

Verify WebSocket URL:
```
grep brokerUrl app-config.yaml
```

Test network connectivity:

curl -v wss://broker.${subdomain}.spotifyportal.com

Check firewall rules:
- Ensure outbound WSS (port 443 or 8080) allowed
- Test from Agent host

Verify certificates:

openssl x509 -in /certs/client.crt -noout -dates

Solutions:

Fix WebSocket URL in configuration
Update firewall rules to allow outbound WSS
Renew expired certificates
Contact Spotify to verify Portal is operational

Frequent Disconnections Symptoms:

agent_websocket_reconnections_total increasing
Logs show repeated disconnect/reconnect cycles

Diagnostic Steps:

Check connection stability:

# Monitor reconnections over time
curl -s http://localhost:9466/metrics | grep reconnections

Review network logs:
- Check for intermittent network issues
- Look for firewall or proxy interruptions
Verify Portal health:
- Contact Spotify support
- Check for Portal maintenance windows

Solutions:

Resolve network stability issues
Adjust firewall/proxy timeouts for WebSocket connections
Coordinate with Spotify on Portal issues

SSL Inspection / Corporate Proxy Intercepting TLS Symptoms:

Repeated reconnection failures in logs
Error message similar to: Error: self-signed certificate in certificate chain
agent_websocket_connected{} == 0 even though broker URL and network access are correct
Other outbound HTTPS requests from the Agent may also fail

Diagnostic Steps:

Check whether your organization performs SSL/TLS inspection
- Firewalls such as Palo Alto, Zscaler, Cisco, or Blue Coat may intercept and re-sign outbound TLS connections
- If so, Portal Connect’s WebSocket connection will be re-signed using a corporate CA that is not trusted by Node.js by default

Inspect Agent logs for certificate errors:

docker logs portal-connect-agent | grep -i "self-signed"

or:

kubectl logs deployment/portal-connect-agent | grep -i certificate

Verify whether Node.js trusts your corporate CA Inside the Agent container:
```
docker exec portal-connect-agent node -p "require('tls').rootCertificates"
```
If your corporate CA is not listed, Node.js will reject the connection.
Ensure you have the correct TLS inspection certificates
- Export the root + intermediate CA certificates from your firewall
- Certificates must be PEM-encoded:
  -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

Solutions:

Create a CA bundle containing your corporate inspection certificates:
```
cat root-ca.pem intermediate-ca.pem > custom-ca-bundle.pem
```
Mount the CA bundle into the Agent container Example:
```
-v ./certs:/certs:ro
```

Tell Node.js to trust your corporate CA using NODE_EXTRA_CA_CERTS Docker run:

-e NODE_EXTRA_CA_CERTS=/certs/custom-ca-bundle.pem

Docker Compose:

environment:
  - NODE_EXTRA_CA_CERTS=/certs/custom-ca-bundle.pem

Kubernetes:

env:
  - name: NODE_EXTRA_CA_CERTS
    value: /etc/portal-connect/custom-ca-bundle.pem

Restart or redeploy the Agent:

docker restart portal-connect-agent

or:

kubectl rollout restart deployment/portal-connect-agent

Verify WebSocket connectivity:

curl http://localhost:9466/metrics | grep agent_websocket_connected

Expected:

agent_websocket_connected{} 1

Request Failures

All Requests Failing Symptoms:

High error rate across all routes
Logs show consistent request failures

Diagnostic Steps:

Check Agent connectivity:

docker logs portal-connect-agent | grep "WebSocket connected"

Verify routes are registered:
```
curl http://gateway:8080/routes
```

Test external service directly from Agent host:

docker exec portal-connect-agent curl -v https://api.github.com

Solutions:

Restore WebSocket connection
Fix route configuration and restart Agent
Resolve network access to external services

Specific Route Failing Symptoms:

Errors for specific route/URL pattern
Other routes working normally

Diagnostic Steps:

Check route configuration:
```
grep -A5 "routes:" app-config.yaml
```

Test service directly:

curl -v https://failing-service.com/endpoint

Check service-specific authentication:
- Verify headers configured correctly
- Test with actual credentials

Solutions:

Fix route pattern (wildcard, exact match)
Update authentication headers
Verify external service is operational

Timeout Errors Symptoms:

Logs show Network timeout or Request timeout
Slow response times

Diagnostic Steps:

Check request latency:

curl -s http://localhost:9466/metrics | grep duration_seconds

Test external service response time:

time curl https://external-service.com/endpoint

Review Agent resource usage:
```
docker stats portal-connect-agent
```

Solutions:

Increase timeout configuration (if configurable)
Optimize external service performance
Scale Agent resources if CPU/memory constrained

Certificate Issues

Certificate Validation Fails Symptoms:

mTLS certificate validation failed in logs
Agent fails to start

Diagnostic Steps:

Verify certificate files exist:
```
ls -la /certs/
```

Check certificate validity:

openssl x509 -in /certs/portal-connect-client-<timestamp>.crt \
  -noout -dates -subject -issuer

Verify certificate and key match:

openssl x509 -in /certs/portal-connect-client-<timestamp>.crt \
  -noout -modulus | openssl md5
openssl rsa -in /certs/portal-connect-client-<timestamp>.key \
  -noout -modulus | openssl md5

Check file permissions:

stat /certs/portal-connect-client-<timestamp>.crt \
  /certs/portal-connect-client-<timestamp>.key

Solutions:

Generate new certificates
Fix file permissions (readable by container user)
Ensure certificate is registered with Spotify
Verify certificate and key are matching pair

Certificate Expired Symptoms:

Certificate expired in logs
agent_certificate_expiry_seconds negative or very low

Diagnostic:

openssl x509 -in /certs/portal-connect-client-<timestamp>.crt -noout -enddate

Solutions:

Generate new certificate
Register with Spotify
Update certificate files
Restart Agent

Configuration Issues

Configuration File Not Found Symptoms:

Configuration file not found or ENOENT error
Agent exits immediately

Diagnostic Steps:

Check CONFIG_PATH:

docker inspect portal-connect-agent | grep CONFIG_PATH

Verify volume mount:

docker inspect portal-connect-agent | grep -A10 Mounts

Check file exists in container:

docker exec portal-connect-agent ls -la /app/config/

Solutions:

Fix CONFIG_PATH environment variable
Fix volume mount path
Verify file exists on host at mount source

Invalid Configuration Syntax Symptoms:

Invalid configuration or YAML parse errors
Agent fails to start

Diagnostic Steps:

Validate YAML syntax:

python -c "import yaml; yaml.safe_load(open('app-config.yaml'))"

Check for common errors:
- Incorrect indentation
- Missing required fields
- Invalid route patterns

Solutions:

Fix YAML syntax errors
Validate against configuration reference
Use YAML linter

Environment Variable Not Substituted Symptoms:

Headers contain literal ${VAR_NAME}
Authentication fails for external services

Diagnostic Steps:

Check environment variables are set:

docker exec portal-connect-agent env | grep TOKEN

Review configuration:

docker exec portal-connect-agent cat /app/config/app-config.yaml

Solutions:

Set missing environment variables when starting container
Fix environment variable name mismatch
Use correct syntax: ${VAR_NAME} for env vars, ${VAR_NAME_FILE} for file-based

Resource Issues

High CPU Usage Symptoms:

Container using > 80% CPU consistently
Slow request processing

Diagnostic:

docker stats portal-connect-agent

Solutions:

Increase CPU limits
Scale horizontally (add more Agent replicas)
Investigate unusually high request volume

High Memory Usage Symptoms:

Container approaching memory limit
OOMKilled events (Kubernetes)

Diagnostic:

docker stats portal-connect-agent
kubectl describe pod portal-connect-agent-xxx

Solutions:

Increase memory limits
Check for memory leaks (contact Spotify support)
Restart Agent to clear memory

Container Keeps Restarting Symptoms:

docker ps shows frequent restarts
Pod in CrashLoopBackOff (Kubernetes)

Diagnostic Steps:

Check logs:
```
docker logs portal-connect-agent
```

Check exit code:

docker inspect portal-connect-agent | grep ExitCode

Solutions:

Fix configuration errors causing startup failure
Resolve certificate validation issues
Check resource limits not too restrictive
Review logs for specific error messages

Configuration Reference

This reference documents all configuration options for the Portal Connect Agent.

Configuration File Format

The Portal Connect Agent uses YAML format for configuration. The default configuration file location is app-config.yaml, but you can specify a custom path using the CONFIG_PATH environment variable.

Configuration Structure

agent:
  brokerUrl: string (required)
  routes: array or object (required)
  mtls: object (optional, enabled by default)

Configuration Options

agent.brokerUrl

Type: String (required)
Description: WebSocket URL to connect the Agent to your Portal instance
Format: wss://broker.${subdomain}.spotifyportal.com

Example:

agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

Notes:

Must use wss:// protocol (secure WebSocket)
Connection URL provided by Spotify
Do not modify the path component

agent.routes

Type: Array or Object (required)
Description: Defines which external URLs the Agent can access

Simple Array Format Use this format for routes without custom headers:

agent:
  routes:
    - https://api.github.com/*
    - https://internal-api.company.com/v1/*
    - https://status.company.com/health

Advanced Object Format Use this format to specify per-route headers:

agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'

    'https://internal-api.company.com/v1/*':
      headers:
        x-api-key: '${INTERNAL_API_KEY}'
        x-forwarded-by: 'portal-connect-agent'

Route Patterns Exact match:

routes:
  - https://api.example.com/v1/users

Matches only: https://api.example.com/v1/users

Wildcard match:

routes:
  - https://api.example.com/v1/*

Matches:
- https://api.example.com/v1/users
- https://api.example.com/v1/users/123
- https://api.example.com/v1/anything
Does not match:
- https://api.example.com/v2/users (different path)
- http://api.example.com/v1/users (different protocol)

Route Requirements

Must include protocol (http:// or https://)
Must include hostname
Path is optional but recommended
Wildcards (*) only supported at end of path
No query parameters or fragments in route patterns

Valid routes:

routes:
  - https://api.example.com/*
  - https://api.example.com/v1/*
  - https://api.example.com/v1/users
  - http://legacy-api.company.com/health

Invalid routes:

routes:
  - api.example.com/* # Missing protocol
  - https://*.example.com/* # Wildcard in hostname not supported
  - https://api.example.com/*/users # Wildcard in middle of path

agent.mtls

Type: Object (optional, enabled by default)
Description: Mutual TLS (mTLS) configuration for secure authentication

agent.mtls.enabled

Type: Boolean (optional)
Default: true
Description: Enable or disable mTLS authentication

Production UsemTLS is required for production use. Only disable for local development.

agent:
  mtls:
    enabled: true

agent.mtls.clientCertPath

Type: String (required when mTLS enabled)
Description: Path to client certificate file in PEM format

Example:

agent:
  mtls:
    clientCertPath: /etc/ssl/certs/portal-connect-client.crt

Notes:

Must be PEM-encoded X.509 certificate
Maximum certificate lifespan: 180 days
Certificate must be registered with Spotify

agent.mtls.clientKeyPath

Type: String (required when mTLS enabled)
Description: Path to private key file in PEM format

Example:

agent:
  mtls:
    clientKeyPath: /etc/ssl/private/portal-connect-client.key

Notes:

Must be PEM-encoded RSA or ECDSA private key
For ECDSA: P-256 or P-384 elliptical curve (P-384 recommended)
For RSA: Minimum 2048-bit key size (4096-bit recommended)
Keep file secure with restricted permissions (400 or 600)

agent.mtls.validateOnStartup

Type: Boolean (optional)
Default: true
Description: Validate certificates when Agent starts

Example:

agent:
  mtls:
    validateOnStartup: true

Notes:

Recommended to keep enabled
Startup will fail if certificates are invalid or expired
Helps catch certificate issues early

Environment Variables

CONFIG_PATH

Type: String
Default: ./app-config.yaml
Description: Path to the Agent configuration file

Example:

-e CONFIG_PATH=/app/config/app-config.yaml

AGENT_METRICS_PORT

Type: Integer
Default: 9466
Description: Port for Prometheus metrics endpoint

Example:

-e AGENT_METRICS_PORT=9466

LOG_LEVEL

Type: String
Default: info
Valid values: debug, info, warn, error
Description: Logging verbosity level

Example:

-e LOG_LEVEL=debug

Environment Variable Substitution

Configuration files support environment variable substitution for sensitive values. Using Environment Variables Reference environment variables in configuration with ${VAR_NAME}:

agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'

Set the environment variable when running the container:

docker run -d \
  -e GITHUB_TOKEN="ghp_xxxxxxxxxxxx" \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest

File-Based Secrets For secrets stored in files (e.g., Kubernetes secrets), use ${VAR_NAME_FILE}:

agent:
  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN_FILE}'

The Agent will read the secret from the file path:

docker run -d \
  -e GITHUB_TOKEN_FILE="/run/secrets/github_token" \
  -v /path/to/secrets:/run/secrets \
  europe-west1-docker.pkg.dev/spc-global-admin/spotify-portal-connect/portal-connect-agent:latest

Complete Configuration Examples

Minimal Configuration

agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    - https://api.github.com/*

  mtls:
    clientCertPath: /certs/client.crt
    clientKeyPath: /certs/client.key

Configuration with Multiple Routes

agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    - https://api.github.com/*
    - https://api.gitlab.com/*
    - https://internal-api.company.com/v1/*
    - https://metrics.company.com/prometheus/*
    - https://legacy-service.company.com/health

  mtls:
    enabled: true
    clientCertPath: /etc/ssl/certs/portal-connect-client.crt
    clientKeyPath: /etc/ssl/private/portal-connect-client.key
    validateOnStartup: true

Configuration with Authentication Headers

agent:
  brokerUrl: wss://broker.${subdomain}.spotifyportal.com

  routes:
    'https://api.github.com/*':
      headers:
        authorization: 'Bearer ${GITHUB_TOKEN}'
        accept: 'application/vnd.github.v3+json'

    'https://api.gitlab.com/*':
      headers:
        private-token: '${GITLAB_TOKEN}'

    'https://internal-api.company.com/v1/*':
      headers:
        x-api-key: '${INTERNAL_API_KEY}'
        x-forwarded-by: 'portal-connect-agent'
        x-environment: 'production'

  mtls:
    clientCertPath: /certs/client.crt
    clientKeyPath: /certs/client.key

Documentation Index

​Configuring Plugins

​How It Works

​Configuration Approach

​Authentication and Credentials

​Multi-Step Configuration Example

​Benefits of This Approach

​Adding and Managing Routes

​Understanding Routes

​Adding Routes

​Route Planning

​Adding a New Route

​Modifying Routes

​Route Security Best Practices

​Common Route Patterns

​Troubleshooting Routes

​Monitoring and Operations

​Metrics Endpoint

​Key Metrics

​Alerting

​Monitoring WebSocket Connection

​Monitoring Request Success

​Health Checks

​Troubleshooting

​Connection Issues

​Request Failures

​Certificate Issues

​Configuration Issues

​Resource Issues

​Configuration Reference

​Configuration File Format

​Configuration Structure

​Configuration Options

​agent.brokerUrl

​agent.routes

​agent.mtls

​Environment Variables

​Environment Variable Substitution

​Complete Configuration Examples

Configuring Plugins

How It Works

Configuration Approach

Authentication and Credentials

Multi-Step Configuration Example

Benefits of This Approach

Adding and Managing Routes

Understanding Routes

Adding Routes

Route Planning

Adding a New Route

Modifying Routes

Route Security Best Practices

Common Route Patterns

Troubleshooting Routes

Monitoring and Operations

Metrics Endpoint

Key Metrics

Alerting

Monitoring WebSocket Connection

Monitoring Request Success

Health Checks

Troubleshooting

Connection Issues

Request Failures

Certificate Issues

Configuration Issues

Resource Issues

Configuration Reference

Configuration File Format

Configuration Structure

Configuration Options

agent.brokerUrl

agent.routes

agent.mtls

Environment Variables

Environment Variable Substitution

Complete Configuration Examples