GitLab

Similar to the Source Control Management (SCM) integration plugin, the GitLab integration plugin for Soundcheck provides out-of-box integration with GitLab by leveraging Backstage's GitLab integration to implement collection of facts from GitLab repositories.

The purpose of the GitLab integration plugin is to provide GitLab-specific fact collection (like branch protections), while the SCM integration plugin provides the collection of facts based on project content.

The GitLab integration plugin supports the collection of the following facts:

• branch_protections
• project_details
• project_languages

Prerequisites

Configure GitLab integration in Backstage

Integrations are configured at the root level of app-config.yaml. Here's an example configuration for GitLab:

integrations:
  gitlab:
    - host: gitlab.com
      token: ${GITLAB_TOKEN}

Consult the Backstage GitLab integration instructions for full configuration details.

Add the GitlabFactCollector to Soundcheck

GitLab integration for Soundcheck is not installed by default. It must be manually installed and configured for the GitLab Fact Collector to work.

First, add the @spotify/backstage-plugin-soundcheck-backend-module-gitlab package:

yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-gitlab

Then add the following to your packages/backend/src/index.ts file:

packages/backend/src/index.ts

const backend = createBackend();

backend.add(import('@spotify/backstage-plugin-soundcheck-backend'));
backend.add(
  import('@spotify/backstage-plugin-soundcheck-backend-module-gitlab'),
);
// ...

backend.start();

Consult the Soundcheck Backend documentation for additional details on setting up the Soundcheck backend.

Legacy Backend

warning

If you are still using the Legacy Backend you can follow these instructions but we highly recommend migrating to the New Backend System.

First add the package: yarn workspace backend add @spotify/backstage-plugin-soundcheck-backend-module-gitlab

Then in packages/backend/src/plugins/soundcheck.ts, add the GitlabFactCollector:

packages/backend/src/plugins/soundcheck.ts

import { SoundcheckBuilder } from '@spotify/backstage-plugin-soundcheck-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';
import { GitlabFactCollector } from '@spotify/backstage-plugin-soundcheck-backend-module-gitlab';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  return SoundcheckBuilder.create({ ...env })
    .addFactCollectors(
      GitlabFactCollector.create(env.config, env.logger, env.cache),
    )
    .build();
}

Plugin Configuration

The collection of facts is driven by configuration. To learn more about the configuration, jump to the Defining GitLab Fact Collectors section.

GitLab Fact Collector can be configured via YAML or No-Code UI. If you configure it via both YAML and No-Code UI, the configurations will be merged. It's preferable to choose a single source for the Fact Collectors configuration (either No-Code UI or YAML) to avoid confusing merge results.

No-Code UI Configuration Option

1. Make sure the prerequisite Configure GitLab integration in Backstage is completed and GitLab instance details are configured.

2. To enable the GitLab Integration, go to Soundcheck > Integrations > GitLab and click the Configure button. To learn more about the No-Code UI config, see the Configuring a fact collector (integration) via the no-code UI.

YAML Configuration Option

1. Create a gitlab-facts-collectors.yaml file in the root of your Backstage repository and fill in all your GitLab Fact Collectors. A simple example GitLab Fact Collector is listed below.

   gitlab-facts-collectors.yaml

   ---
   frequency:
     cron: '0 * * * *'
   collects:
     - type: branch_protections
     - type: project_details
     - type: project_languages

   Note: this file will be loaded at runtime along with the rest of your Backstage configuration files. Therefore, make sure that it's available in deployed environments in the same way as your app-config.yaml files are.

2. Add a soundcheck collectors field to app-config.yaml and reference the newly created gitlab-facts-collectors.yaml

   app-config.yaml

   soundcheck:
     collectors:
       gitlab:
         $include: ./gitlab-facts-collectors.yaml

Rate Limiting (Optional)

This fact collector can be rate limited in Soundcheck using the following configuration:

app-config.yaml

soundcheck:
  job:
    workers:
      gitlab:
        limiter:
          max: 1900
          duration: 60000

GitLab API has a limit of 2000 requests per minute (Authenticated API traffic for a user). We recommend setting your rate limit to something below this, i.e. in the example above, we set the rate limit to 1900 executions every minute.

This fact collector handles rate limit errors per the recommendation from GitLab. Soundcheck will automatically wait and retry requests that are rate limited.

Defining GitLab Fact Collectors

This section describes the data shape and semantics of GitLab Fact Collectors.

Overall Shape Of A GitLab Fact Collector

The following is an example of a descriptor file for a GitLab Fact Collector:

---
frequency:
  cron: '0 * * * *'
initialDelay:
  seconds: 30
filter:
  kind: 'Component'
cache:
  duration:
    hours: 2
collects:
  - type: branch_protections
  - type: project_details
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
  - type: project_languages

Below are the details for each field.

frequency [optional]

The frequency at which the collector should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. This is the default frequency for each fact type. Example:

frequency:
  minutes: 10

initialDelay [optional]

The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration. This is the default initial delay for each fact type. Example:

initialDelay:
  seconds: 30

batchSize [optional]

The number of entities to collect facts for at once. Optional, the default value is 1.

Note: Fact collection for a batch of entities is still considered as one hit towards the rate limits by the Soundcheck Rate Limiting engine, while the actual number of hits will be equal to the batchSize.

Example:

batchSize: 100

filter [optional]

A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. This is the default filter for each fact type. Example:

filter:
  - spec.lifecycle: 'production'

exclude [optional]

Entities matching this filter will be skipped during the fact collection process. Can be used in combination with filter. Matches the filter format used by the Catalog API.

filter:
  - kind: component
exclude:
  - spec.type: documentation

cache [optional]

If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. This is the default cache config for each fact type. Example:

cache:
  duration:
    hours: 24

collects [required]

An array describing which facts to collect and how to collect them. See below for details about the configuration of fact collection for each fact type.

• factName [optional]

  The name of the fact to be collected (must be unique within GitLab collector).

  • Minimum length of 1
  • Maximum length of 100
  • Alphanumeric with single separator instances of periods, dashes, underscores, or forward slashes

  If not provided it defaults to the type value (see below).

• type [required]

  The type of the collector (e.g. branch_protections, project_details, project_languages).

• frequency [optional]

  The frequency at which the fact collection should be executed. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided, it overrides the default frequency provided at the top level. If not provided, it defaults to the frequency provided at the top level. If neither collector's frequency, nor default frequency is provided, the fact will only be collected on demand.

• initialDelay [optional]

  The amount of time that should pass before the first invocation happens. Possible values are either a cron expression { cron: ... } or HumanDuration. If provided, it overrides the default initial delay provided at the top level. If not provided, it defaults to the initial delay provided at the top level. If neither collector's initial delay, nor default initial delay is provided, the fact will be collected with no initial delay.

• batchSize [optional]

  The number of entities to collect facts for at once. Optional, the default value is 1. If provided it overrides the default batchSize provided at the top level. If not provided it defaults to the batchSize provided at the top level. If neither collector's batchSize nor default batchSize is provided the fact will be collected for one entity at a time.

  Note: Fact collection for a batch of entities is still considered as one hit towards the rate limits by the Soundcheck Rate Limiting engine, while the actual number of hits will be equal to the batchSize.

  Example:

  batchSize: 100

• filter [optional]

  A filter specifying which entities to collect the specified facts for. Matches the filter format used by the Catalog API. If provided, it overrides the default filter provided at the top level. If not provided, it defaults to the filter provided at the top level. If neither collector's filter, nor default filter is provided, the fact will be collected for all entities.

• exclude [optional]

  Entities matching this filter will be skipped during the fact collection process. Can be used in combination with filter. Matches the filter format used by the Catalog API.

  filter:
    - kind: component
  exclude:
    - spec.type: documentation

• cache [optional]

  If the collected facts should be cached, and if so for how long. Possible values are either true or false or a nested { duration: HumanDuration } field. If provided, it overrides the default cache config provided at the top level. If not provided, it defaults to the cache config provided at the top level. If neither collector's cache nor default cache config is provided, the fact will not be cached.

Collecting Branch Protections Fact

The branch_protections fact contains information about configured branch protections for a given branch in a GitLab project.

Shape of A Branch Protections Fact Collector

The shape of a branch_protections Fact Collector extends the Overall Shape Of A GitLab Fact Collector (restriction: type: branch_protections).

Additional fields:

• branch [optional]

  The branch to collect the fact from. If not provided, defaults to the project's default branch. If provided, collected fact scope will be set to the configured branch name.

The following is an example of the branch_protections Fact Collector configuration:

collects:
  - type: branch_protections
    frequency:
      cron: '0 * * * *'
    filter:
      - spec.lifecycle: 'production'
        spec.type: 'website'
    cache: false

Shape of A Branch Protections Fact

The shape of a branch_protections Fact is based on the Fact Schema.

For a description of the data collected regarding branch protection, refer to the GitLab API documentation.

The following is an example of the collected branch_protections fact:

factRef: gitlab:default/branch_protections
entityRef: component:default/queue-proxy
timestamp: 2023-02-24T15:50+00Z
data:
  id: 1
  name: 'main'
  push_access_levels:
    - id: 1
      access_level: 40
      user_id: null
      group_id: null
      access_level_description: 'Maintainers'
  merge_access_levels:
    - id: 1
      access_level: null
      user_id: null
      group_id: 1234
      access_level_description: 'Example Merge Group'
  unprotect_access_levels:
    - id: 1
      access_level: 40
      user_id: null
      group_id: null
      access_level_description: 'Maintainers'
  allow_force_push: false
  code_owner_approval_required: false
  inherited: false

Shape of A Branch Protections Fact Check

The shape of a branch_protections Fact Check matches the Shape of a Fact Check.

The following is an example of the branch_protections fact checks:

soundcheck:
  checks:
    - id: denies_force_push
      rule:
        factRef: gitlab:default/branch_protections
        path: $.allow_force_push
        operator: equal
        value: false
    - id: code_owner_approval_required
      rule:
        factRef: gitlab:default/branch_protections
        path: $.code_owner_approval_required
        operator: equal
        value: true

Collecting Project Details Fact

The project_details fact contains information about a GitLab project.

Shape of A Project Details Fact Collector

The shape of a project_details Fact Collector matches the Overall Shape Of A GitLab Fact Collector (restriction: type: project_details).

The following is an example of the project_details Fact Collector configuration:

collects:
  - type: project_details
    frequency:
      cron: '0 * * * *'
    filter:
      - spec.lifecycle: 'production'
    cache: true

Shape of A Project Details Fact

The shape of a project_details Fact is based on the Fact Schema.

For a description of the data collected about project, refer to the GitLab API documentation.

The following is an example of the collected project_details fact:

factRef: gitlab:default/project_details
entityRef: component:default/queue-proxy
timestamp: 2023-02-24T15:50+00Z
data:
  id: 3
  description: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit.'
  description_html: '<p data-sourcepos=\"1:1-1:56\" dir=\"auto\">Lorem ipsum dolor sit amet, consectetur adipiscing elit.</p>'
  default_branch: 'main'
  visibility: 'private'
  ssh_url_to_repo: 'git@example.com:diaspora/diaspora-project-site.git'
  http_url_to_repo: 'http://example.com/diaspora/diaspora-project-site.git'
  web_url: 'http://example.com/diaspora/diaspora-project-site'
  readme_url: 'http://example.com/diaspora/diaspora-project-site/blob/main/README.md'
  tag_list:
    - 'example'
    - 'disapora project'
  topics:
    - 'example'
    - 'disapora project'
  owner:
    id: 3
    name: 'Diaspora'
    created_at: '2013-09-30T13:46:02Z'
  name: 'Diaspora Project Site'
  name_with_namespace: 'Diaspora / Diaspora Project Site'
  path: 'diaspora-project-site'
  path_with_namespace: 'diaspora/diaspora-project-site'
  issues_enabled: true
  open_issues_count: 1
  merge_requests_enabled: true
  jobs_enabled: true
  wiki_enabled: true
  snippets_enabled: false
  can_create_merge_request_in: true
  resolve_outdated_diff_discussions: false
  container_registry_enabled: false
  container_registry_access_level: 'disabled'
  security_and_compliance_access_level: 'disabled'
  container_expiration_policy:
    cadence: '7d'
    enabled: false
    keep_n: null
    older_than: null
    name_regex: null
    name_regex_delete: null
    name_regex_keep: null
    next_run_at: '2020-01-07T21:42:58.658Z'
  created_at: '2013-09-30T13:46:02Z'
  updated_at: '2013-09-30T13:46:02Z'
  last_activity_at: '2013-09-30T13:46:02Z'
  creator_id: 3
  namespace:
    id: 3
    name: 'Diaspora'
    path: 'diaspora'
    kind: 'group'
    full_path: 'diaspora'
    avatar_url: 'http://localhost:3000/uploads/group/avatar/3/foo.jpg'
    web_url: 'http://localhost:3000/groups/diaspora'
  import_url: null
  import_type: null
  import_status: 'none'
  import_error: null
  permissions:
    project_access:
      access_level: 10
      notification_level: 3
    group_access:
      access_level: 50
      notification_level: 3
  archived: false
  avatar_url: 'http://example.com/uploads/project/avatar/3/uploads/avatar.png'
  license_url: 'http://example.com/diaspora/diaspora-client/blob/main/LICENSE'
  license:
    key: 'lgpl-3.0'
    name: 'GNU Lesser General Public License v3.0'
    nickname: 'GNU LGPLv3'
    html_url: 'http://choosealicense.com/licenses/lgpl-3.0/'
    source_url: 'http://www.gnu.org/licenses/lgpl-3.0.txt'
  shared_runners_enabled: true
  group_runners_enabled: true
  forks_count: 0
  star_count: 0
  runners_token: 'b8bc4a7a29eb76ea83cf79e4908c2b'
  ci_default_git_depth: 50
  ci_forward_deployment_enabled: true
  ci_forward_deployment_rollback_allowed: true
  ci_allow_fork_pipelines_to_run_in_parent_project: true
  ci_separated_caches: true
  ci_restrict_pipeline_cancellation_role: 'developer'
  public_jobs: true
  shared_with_groups:
    - group_id: 4
      group_name: 'Twitter'
      group_full_path: 'twitter'
      group_access_level: 30
    - group_id: 3
      group_name: 'Gitlab Org'
      group_full_path: 'gitlab-org'
      group_access_level: 10
  repository_storage: 'default'
  only_allow_merge_if_pipeline_succeeds: false
  allow_merge_on_skipped_pipeline: false
  restrict_user_defined_variables: false
  only_allow_merge_if_all_discussions_are_resolved: false
  remove_source_branch_after_merge: false
  printing_merge_requests_link_enabled: true
  request_access_enabled: false
  merge_method: 'merge'
  squash_option: 'default_on'
  auto_devops_enabled: true
  auto_devops_deploy_strategy: 'continuous'
  approvals_before_merge: 0
  mirror: false
  mirror_user_id: 45
  mirror_trigger_builds: false
  only_mirror_protected_branches: false
  mirror_overwrites_diverged_branches: false
  external_authorization_classification_label: null
  packages_enabled: true
  service_desk_enabled: false
  service_desk_address: null
  autoclose_referenced_issues: true
  suggestion_commit_message: null
  enforce_auth_checks_on_uploads: true
  merge_commit_template: null
  squash_commit_template: null
  issue_branch_template: 'gitlab/%{id}-%{title}'
  marked_for_deletion_at: '2020-04-03'
  marked_for_deletion_on: '2020-04-03'
  compliance_frameworks:
    - 'sox'
  warn_about_potentially_unwanted_characters: true
  statistics:
    commit_count: 37
    storage_size: 1038090
    repository_size: 1038090
    wiki_size: 0
    lfs_objects_size: 0
    job_artifacts_size: 0
    pipeline_artifacts_size: 0
    packages_size: 0
    snippets_size: 0
    uploads_size: 0
  container_registry_image_prefix: 'registry.example.com/diaspora/diaspora-client'
  _links:
    self: 'http://example.com/api/v4/projects'
    issues: 'http://example.com/api/v4/projects/1/issues'
    merge_requests: 'http://example.com/api/v4/projects/1/merge_requests'
    repo_branches: 'http://example.com/api/v4/projects/1/repository_branches'
    labels: 'http://example.com/api/v4/projects/1/labels'
    events: 'http://example.com/api/v4/projects/1/events'
    members: 'http://example.com/api/v4/projects/1/members'
    cluster_agents: 'http://example.com/api/v4/projects/1/cluster_agents'

Shape of A Project Details Fact Check

The shape of a project_details Fact Check matches the Shape of a Fact Check.

The following is an example of the project_details fact checks:

soundcheck:
  checks:
    - id: only_allows_merge_if_all_discussions_are_resolved
      rule:
        factRef: gitlab:default/project_details
        path: $.only_allow_merge_if_all_discussions_are_resolved
        operator: equal
        value: true
    - id: not_archived
      rule:
        factRef: gitlab:default/project_details
        path: $.archived
        operator: equal
        value: false

Collecting Project Languages Fact

The project_languages fact contains information about languages used within a GitLab project (percentage value).

Shape of A Project Languages Fact Collector

The shape of a project_languages Fact Collector matches the Overall Shape Of A GitLab Fact Collector (restriction: type: project_languages).

The following is an example of the project_languages Fact Collector configuration:

collects:
  - type: project_languages
    frequency:
      cron: '0 * * * *'
    filter:
      - spec.lifecycle: 'production'
    cache: true

Shape of A Project Languages Fact

The shape of a project_languages Fact is based on the Fact Schema.

For a description of the data collected about project languages, refer to the GitLab API documentation.

The following is an example of the collected project_languages fact:

factRef: gitlab:default/project_languages
entityRef: component:default/queue-proxy
timestamp: 2023-02-24T15:50+00Z
data:
  Ruby: 66.69
  JavaScript: 22.98
  HTML: 7.91
  CoffeeScript: 2.42

Shape of A Project Languages Fact Check

The shape of a project_languages Fact Check matches the Shape of a Fact Check.

The following is an example of the project_languages fact checks:

soundcheck:
  checks:
    - id: uses_java_script
      rule:
        factRef: gitlab:default/project_languages
        path: $.JavaScript
        operator: greaterThan
        value: 0