Skip to main content

GitLab

Similar to the Source Control Management (SCM) integration plugin, the GitLab integration plugin for Soundcheck provides out-of-box integration with GitLab by leveraging Backstage's GitLab integration to implement collection of facts from GitLab repositories.

The purpose of the GitLab integration plugin is to provide GitLab-specific fact collection (like branch protections), while the SCM integration plugin provides the collection of facts based on project content.

The GitLab integration plugin supports the collection of the following facts:

Prerequisites​

Configure GitLab integration in Backstage​

Integrations are configured at the root level of app-config.yaml. Here's an example configuration for GitLab:

integrations:
  gitlab:
    - host: gitlab.com
      token: ${GITLAB_TOKEN}

Consult the Backstage GitLab integration instructions for full configuration details.

Add the GitlabFactCollector to Soundcheck​

GitLab integration for Soundcheck is not installed by default. It must be manually installed and configured for the GitLab Fact Collector to work.

First, add the @spotify/backstage-plugin-soundcheck-backend-module-gitlab package:

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-gitlab

Legacy Backend​

If you are still using the legacy backend, in packages/backend/src/plugins/soundcheck.ts, add the GitlabFactCollector:

packages/backend/src/plugins/soundcheck.ts
import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';
import { GitlabFactCollector } from '@spotify/backstage-plugin-soundcheck-backend-module-gitlab';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return SoundcheckBuilder.create({ ...env })
    .addFactCollectors(
      GitlabFactCollector.create(env.config, env.logger, env.cache),
    )
    .build();
}

New Backend System​

If you are using the New Backend System, you can just add the following:

packages/backend/src/index.ts
const backend = createBackend();

backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(
  import('@spotify/backstage-plugin-soundcheck-backend-module-gitlab'),
);
// ...

backend.start();

Consult the Soundcheck Backend documentation for additional details on setting up the Soundcheck backend.

Plugin Configuration​

The collection of facts is driven by configuration. To learn more about the configuration, jump to the Defining GitLab Fact Collectors section.

  1. Create a gitlab-facts-collectors.yaml file in the root of your Backstage repository and fill in all your GitLab Fact Collectors. A simple example GitLab Fact Collector is listed below.

    gitlab-facts-collectors.yaml
    ---
    frequency:
      cron: '0 * * * *'
    collects:
      - type: branch_protections
      - type: project_details
      - type: project_languages

    Note: this file will be loaded at runtime along with the rest of your Backstage configuration files. Therefore, make sure that it's available in deployed environments in the same way as your app-config.yaml files are.

  2. Add a soundcheck collectors field to app-config.yaml and reference the newly created gitlab-facts-collectors.yaml

    app-config.yaml
    soundcheck:
      collectors:
        gitlab:
          $include: ./gitlab-facts-collectors.yaml

Rate Limiting (Optional)​

This fact collector can be rate limited in Soundcheck using the following configuration:

app-config.yaml
soundcheck:
  job:
    workers:
      gitlab:
        limiter:
          max: 1900
          duration: 60000

GitLab API has a limit of 2000 requests per minute (Authenticated API traffic for a user). We recommend setting your rate limit to something below this, i.e. in the example above, we set the rate limit to 1900 executions every minute.

This fact collector handles rate limit errors per the recommendation from GitLab. Soundcheck will automatically wait and retry requests that are rate limited.

Defining GitLab Fact Collectors​

This section describes the data shape and semantics of GitLab Fact Collectors.

Overall Shape Of A GitLab Fact Collector​

The following is an example of a descriptor file for a GitLab Fact Collector:

---
frequency:
  cron: '0 * * * *'
initialDelay:
  seconds: 30
filter:
  kind: 'Component'
cache:
  duration:
    hours: 2
collects:
  - type: branch_protections
  - type: project_details
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
  - type: project_languages

Below are the details for each field.

frequency [optional]​

The frequency at which the collector should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. This is the default frequency for each fact type. Example:

frequency:
  minutes: 10

initialDelay [optional]​

The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration. This is the default initial delay for each fact type. Example:

initialDelay:
  seconds: 30

filter [optional]​

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. This is the default filter for each fact type. Example:

filter:
  - spec.lifecycle: 'production'

cache [optional]​

If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. This is the default cache config for each fact type. Example:

cache:
  duration:
    hours: 24

collects [required]​

An array describing which facts to collect and how to collect them. See below for details about the configuration of fact collection for each fact type.

  • factName [optional]​

    The name of the fact to be collected (must be unique within GitLab collector).

    • Minimum length of 1
    • Maximum length of 100
    • Alphanumeric with single separator instances of periods, dashes, underscores, or forward slashes

    If not provided it defaults to the type value (see below).

  • type [required]​

    The type of the collector (e.g. branch_protections, project_details, project_languages).

  • frequency [optional]​

    The frequency at which the fact collection should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided, it overrides the default frequency provided at the top level. If not provided, it defaults to the frequency provided at the top level. If neither collector's frequency, nor default frequency is provided, the fact will only be collected on demand.

  • initialDelay [optional]​

    The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided, it overrides the default initial delay provided at the top level. If not provided, it defaults to the initial delay provided at the top level. If neither collector's initial delay, nor default initial delay is provided, the fact will be collected with no initial delay.

  • filter [optional]​

    A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. If provided, it overrides the default filter provided at the top level. If not provided, it defaults to the filter provided at the top level. If neither collector's filter, nor default filter is provided, the fact will be collected for all entities.

  • cache [optional]​

    If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. If provided, it overrides the default cache config provided at the top level. If not provided, it defaults to the cache config provided at the top level. If neither collector's cache nor default cache config is provided, the fact will not be cached.

Collecting Branch Protections Fact​

The branch_protections fact contains information about configured branch protections for a given branch in a GitLab project.

Shape of A Branch Protections Fact Collector​

The shape of a branch_protections Fact Collector extends the Overall Shape Of A GitLab Fact Collector (restriction: type: branch_protections).

Additional fields:

  • branch [optional]​

    The branch to collect the fact from. If not provided, defaults to the project's default branch. If provided, collected fact scope will be set to the configured branch name.

The following is an example of the branch_protections Fact Collector configuration:

collects:
  - type: branch_protections
    frequency:
      cron: '0 * * * *'
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    cache: false

Shape of A Branch Protections Fact​

The shape of a branch_protections Fact is based on the Fact Schema.

For a description of the data collected regarding branch protection, refer to the GitLab API documentation.

The following is an example of the collected branch_protections fact:

factRef: gitlab:default/branch_protections
entityRef: component:default/queue-proxy
timestamp: 2023-02-24T15:50+00Z
data:
  id: 1
  name: 'main'
  push_access_levels:
    - id: 1
      access_level: 40
      user_id: null
      group_id: null
      access_level_description: 'Maintainers'
  merge_access_levels:
    - id: 1
      access_level: null
      user_id: null
      group_id: 1234
      access_level_description: 'Example Merge Group'
  unprotect_access_levels:
    - id: 1
      access_level: 40
      user_id: null
      group_id: null
      access_level_description: 'Maintainers'
  allow_force_push: false
  code_owner_approval_required: false
  inherited: false

Shape of A Branch Protections Fact Check​

The shape of a branch_protections Fact Check matches the Shape of a Fact Check.

The following is an example of the branch_protections fact checks:

soundcheck:
  checks:
    - id: denies_force_push
      rule:
        factRef: gitlab:default/branch_protections
        path: $.allow_force_push
        operator: equal
        value: false
    - id: code_owner_approval_required
      rule:
        factRef: gitlab:default/branch_protections
        path: $.code_owner_approval_required
        operator: equal
        value: true

Collecting Project Details Fact​

The project_details fact contains information about a GitLab project.

Shape of A Project Details Fact Collector​

The shape of a project_details Fact Collector matches the Overall Shape Of A GitLab Fact Collector (restriction: type: project_details).

The following is an example of the project_details Fact Collector configuration:

collects:
  - type: project_details
    frequency:
      cron: '0 * * * *'
    filter:
      - spec.lifecycle: 'production'
    cache: true

Shape of A Project Details Fact​

The shape of a project_details Fact is based on the Fact Schema.

For a description of the data collected about project, refer to the GitLab API documentation.

The following is an example of the collected project_details fact:

factRef: gitlab:default/project_details
entityRef: component:default/queue-proxy
timestamp: 2023-02-24T15:50+00Z
data:
  id: 3
  description: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit.'
  description_html: '<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>'
  default_branch: 'main'
  visibility: 'private'
  ssh_url_to_repo: 'git@example.com:diaspora/diaspora-project-site.git'
  http_url_to_repo: 'http://example.com/diaspora/diaspora-project-site.git'
  web_url: 'http://example.com/diaspora/diaspora-project-site'
  readme_url: 'http://example.com/diaspora/diaspora-project-site/blob/main/README.md'
  tag_list:
    - 'example'
    - 'disapora project'
  topics:
    - 'example'
    - 'disapora project'
  owner:
    id: 3
    name: 'Diaspora'
    created_at: '2013-09-30T13:46:02Z'
  name: 'Diaspora Project Site'
  name_with_namespace: 'Diaspora / Diaspora Project Site'
  path: 'diaspora-project-site'
  path_with_namespace: 'diaspora/diaspora-project-site'
  issues_enabled: true
  open_issues_count: 1
  merge_requests_enabled: true
  jobs_enabled: true
  wiki_enabled: true
  snippets_enabled: false
  can_create_merge_request_in: true
  resolve_outdated_diff_discussions: false
  container_registry_enabled: false
  container_registry_access_level: 'disabled'
  security_and_compliance_access_level: 'disabled'
  container_expiration_policy:
    cadence: '7d'
    enabled: false
    keep_n: null
    older_than: null
    name_regex: null
    name_regex_delete: null
    name_regex_keep: null
    next_run_at: '2020-01-07T21:42:58.658Z'
  created_at: '2013-09-30T13:46:02Z'
  updated_at: '2013-09-30T13:46:02Z'
  last_activity_at: '2013-09-30T13:46:02Z'
  creator_id: 3
  namespace:
    id: 3
    name: 'Diaspora'
    path: 'diaspora'
    kind: 'group'
    full_path: 'diaspora'
    avatar_url: 'http://localhost:3000/uploads/group/avatar/3/foo.jpg'
    web_url: 'http://localhost:3000/groups/diaspora'
  import_url: null
  import_type: null
  import_status: 'none'
  import_error: null
  permissions:
    project_access:
      access_level: 10
      notification_level: 3
    group_access:
      access_level: 50
      notification_level: 3
  archived: false
  avatar_url: 'http://example.com/uploads/project/avatar/3/uploads/avatar.png'
  license_url: 'http://example.com/diaspora/diaspora-client/blob/main/LICENSE'
  license:
    key: 'lgpl-3.0'
    name: 'GNU Lesser General Public License v3.0'
    nickname: 'GNU LGPLv3'
    html_url: 'http://choosealicense.com/licenses/lgpl-3.0/'
    source_url: 'http://www.gnu.org/licenses/lgpl-3.0.txt'
  shared_runners_enabled: true
  group_runners_enabled: true
  forks_count: 0
  star_count: 0
  runners_token: 'b8bc4a7a29eb76ea83cf79e4908c2b'
  ci_default_git_depth: 50
  ci_forward_deployment_enabled: true
  ci_forward_deployment_rollback_allowed: true
  ci_allow_fork_pipelines_to_run_in_parent_project: true
  ci_separated_caches: true
  ci_restrict_pipeline_cancellation_role: 'developer'
  public_jobs: true
  shared_with_groups:
    - group_id: 4
      group_name: 'Twitter'
      group_full_path: 'twitter'
      group_access_level: 30
    - group_id: 3
      group_name: 'Gitlab Org'
      group_full_path: 'gitlab-org'
      group_access_level: 10
  repository_storage: 'default'
  only_allow_merge_if_pipeline_succeeds: false
  allow_merge_on_skipped_pipeline: false
  restrict_user_defined_variables: false
  only_allow_merge_if_all_discussions_are_resolved: false
  remove_source_branch_after_merge: false
  printing_merge_requests_link_enabled: true
  request_access_enabled: false
  merge_method: 'merge'
  squash_option: 'default_on'
  auto_devops_enabled: true
  auto_devops_deploy_strategy: 'continuous'
  approvals_before_merge: 0
  mirror: false
  mirror_user_id: 45
  mirror_trigger_builds: false
  only_mirror_protected_branches: false
  mirror_overwrites_diverged_branches: false
  external_authorization_classification_label: null
  packages_enabled: true
  service_desk_enabled: false
  service_desk_address: null
  autoclose_referenced_issues: true
  suggestion_commit_message: null
  enforce_auth_checks_on_uploads: true
  merge_commit_template: null
  squash_commit_template: null
  issue_branch_template: 'gitlab/%{id}-%{title}'
  marked_for_deletion_at: '2020-04-03'
  marked_for_deletion_on: '2020-04-03'
  compliance_frameworks:
    - 'sox'
  warn_about_potentially_unwanted_characters: true
  statistics:
    commit_count: 37
    storage_size: 1038090
    repository_size: 1038090
    wiki_size: 0
    lfs_objects_size: 0
    job_artifacts_size: 0
    pipeline_artifacts_size: 0
    packages_size: 0
    snippets_size: 0
    uploads_size: 0
  container_registry_image_prefix: 'registry.example.com/diaspora/diaspora-client'
  _links:
    self: 'http://example.com/api/v4/projects'
    issues: 'http://example.com/api/v4/projects/1/issues'
    merge_requests: 'http://example.com/api/v4/projects/1/merge_requests'
    repo_branches: 'http://example.com/api/v4/projects/1/repository_branches'
    labels: 'http://example.com/api/v4/projects/1/labels'
    events: 'http://example.com/api/v4/projects/1/events'
    members: 'http://example.com/api/v4/projects/1/members'
    cluster_agents: 'http://example.com/api/v4/projects/1/cluster_agents'

Shape of A Project Details Fact Check​

The shape of a project_details Fact Check matches the Shape of a Fact Check.

The following is an example of the project_details fact checks:

soundcheck:
  checks:
    - id: only_allows_merge_if_all_discussions_are_resolved
      rule:
        factRef: gitlab:default/project_details
        path: $.only_allow_merge_if_all_discussions_are_resolved
        operator: equal
        value: true
    - id: not_archived
      rule:
        factRef: gitlab:default/project_details
        path: $.archived
        operator: equal
        value: false

Collecting Project Languages Fact​

The project_languages fact contains information about languages used within a GitLab project (percentage value).

Shape of A Project Languages Fact Collector​

The shape of a project_languages Fact Collector matches the Overall Shape Of A GitLab Fact Collector (restriction: type: project_languages).

The following is an example of the project_languages Fact Collector configuration:

collects:
  - type: project_languages
    frequency:
      cron: '0 * * * *'
    filter:
      - spec.lifecycle: 'production'
    cache: true

Shape of A Project Languages Fact​

The shape of a project_languages Fact is based on the Fact Schema.

For a description of the data collected about project languages, refer to the GitLab API documentation.

The following is an example of the collected project_languages fact:

factRef: gitlab:default/project_languages
entityRef: component:default/queue-proxy
timestamp: 2023-02-24T15:50+00Z
data:
  Ruby: 66.69
  JavaScript: 22.98
  HTML: 7.91
  CoffeeScript: 2.42

Shape of A Project Languages Fact Check​

The shape of a project_languages Fact Check matches the Shape of a Fact Check.

The following is an example of the project_languages fact checks:

soundcheck:
  checks:
    - id: uses_java_script
      rule:
        factRef: gitlab:default/project_languages
        path: $.JavaScript
        operator: greaterThan
        value: 0