Skip to main content

AI Gateway

Overview​

What is AI Gateway?​

The AI Gateway is a centralized plugin for configuring and managing AI capabilities in Backstage. It allows you to set up and manage various AI models, providers, and re-ranking capabilities that can be used across different plugins, such as AiKA.

Supported Providers​

AI Gateway currently supports the following providers

  • AWS Bedrock
  • Azure OpenAI
  • OpenAI
  • Cohere
  • Ollama

Installation​

Enabling AI Gateway in Portal​

You can enable or disable the AI Gateway at any time through the Config Manager.

Configuring your integrations​

We provide support for a variety of AI providers and models. You can configure these in the Config Manager under the AI Gateway section.

Screenshot detailing AiKA Config

tip

You can have one or multiple providers configured at the same time. Plugins will allow you to select which provider and model to use, or you can set a default provider to use across all plugins.

note

The baseURL and headers configuration options are useful when routing requests through a proxy or gateway. For direct connections to AI providers, these options can be omitted and the default endpoints will be used.

ai:
  providers:
    amazonBedrock:
      region: us-east-1
      accessKeyId: your-access-key-id
      secretAccessKey: your-secret-access-key
      sessionToken: your-session-token # Optional, if using temporary credentials
      baseURL: https://ai-gateway.example.com/bedrock # Optional, to point at a gateway (defaults to AWS Bedrock's regional endpoint)
      headers: # Optional, e.g. for authentication with gateway
        - key: X-Custom-Header
          value: custom-value
    azure:
      apiKey: your-azure-api-key
      resourceName: your-resource-name # Used in URL: https://{resourceName}.openai.azure.com/openai/v1
      apiVersion: 2024-02-15-preview # Optional, defaults to preview
      baseURL: https://ai-gateway.example.com/azure # Optional, to point at a gateway (overrides resourceName when provided)
      headers: # Optional, e.g. for authentication with gateway
        - key: X-Custom-Header
          value: custom-value
    cohere:
      apiKey: your-cohere-api-key
      baseURL: https://ai-gateway.example.com/cohere # Optional, to point at a gateway (defaults to https://api.cohere.com/v2)
      headers: # Optional, e.g. for authentication with gateway
        - key: X-Custom-Header
          value: custom-value
    openai:
      apiKey: your-openai-api-key
      baseURL: https://ai-gateway.example.com/openai # Optional, to point at a gateway (defaults to https://api.openai.com/v1)
      headers: # Optional, e.g. for authentication with gateway
        - key: X-Custom-Header
          value: custom-value
    ollama:
      baseURL: http://your-ollama-host:11434 # Required: URL to your Ollama instance
      headers: # Optional, e.g. for authentication with gateway
        - key: X-Custom-Header
          value: custom-value

Configuring a Default Provider​

You can set a default provider that will be used across all plugins that support AI Gateway. The model fields are formatted as provider|model.

ai:
  models:
    languageModel: openai|gpt-4
    rerankingModel: cohere|rerank-001
    embedding: openai|text-embedding-ada-002
    imageModel: ollama|llama3