Container registry for a secondary site | GitLab Docs
/ Container registry for a secondary site Tier : Premium, Ultimate Offering : GitLab Self-Managed You can set up a container registry on your secondary Geo site that replicates container images from the one on the primary Geo site. This container image replication is used only for disaster recovery purposes. Do not push to the container registry on the secondary Geo site, because the data is not propagated to the primary site.
We do not recommend pulling container registry data from the secondary site because it may be stale. The feature request issue 365864 would solve this problem. You are encouraged to upvote the issue to register your interest. Important: The container registry metadata database is separate from container image replication. While container images replicate from primary to secondary sites, the metadata database does not.
When using GitLab Geo with the container registry metadata database enabled, you must configure separate, external PostgreSQL instances for the container registry at each Geo site (both primary and secondary). The container registry metadata database cannot use the default GitLab-managed PostgreSQL database. Each site’s metadata database operates independently without replication between them. For setup instructions, see Using an external database .
Supported container registries Geo supports the following types of container registries: Docker OCI Supported image formats The following container image formats are supported by Geo: Docker V2, schema 1 Docker V2, schema 2 OCI (Open Container Initiative) In addition, Geo also supports BuildKit cache images . Supported storage Docker For more information on supported registry storage drivers see Docker registry storage drivers Read the Load balancing considerations when deploying the Registry, and how to set up the storage driver for the GitLab integrated container registry .
Registries that support OCI artifacts The following registries support OCI artifacts: CNCF Distribution - local/offline verification Azure Container Registry (ACR) Amazon Elastic Container Registry (ECR) Google Artifact Registry (GAR) GitHub Packages container registry (GHCR) Bundle Bar For more information, see the OCI Distribution Specification . Configure container registry replication You can enable a storage-agnostic replication so it can be used for cloud or local storage.
Whenever a new image is pushed to the primary site, each secondary site pulls it to its own container repository. To configure container registry replication: Configure the primary site . Configure the secondary site . Verify container registry replication . Configure primary site Make sure that you have container registry set up and working on the primary site before following the next steps. To be able to replicate new container images, the container registry must send notification events to the primary site for every push.
The token shared between the container registry and the web nodes on the primary is used to make communication more secure. SSH into your GitLab primary server and sign in as root (for GitLab HA, you only need a Registry node): shell sudo -i Edit /etc/gitlab/gitlab.rb : ruby # Configure the registry to listen on the public/internal interface # Replace with the appropriate interface (for example, '0.0.0.0' for all interfaces) registry [ 'registry_http_addr' ] = '0.0.0.0:5000' registry [ 'notifications' ] = [ { 'name' => 'geo_event' , 'url' => 'https://<example.com>/api/v4/container_registry_event/events' , 'timeout' => '500ms' , 'maxretries' => 5 , 'backoff' => '1s' , 'headers' => { 'Authorization' => [ '<replace_with_a_secret_token>' ] } } ] Replace <example.com> with the external_url defined in your primary site’s /etc/gitlab/gitlab.rb file, and replace <replace_with_a_secret_token> with a case-sensitive alphanumeric string that starts with a letter.
You can generate one with /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c 32 | sed "s/^[0-9]*//"; echo Unless you set gitlab_rails['registry_notification_secret'] explicitly, name this endpoint geo_event . On a node whose /etc/gitlab/gitlab.rb contains a registry['notifications'] block, the Linux package derives that secret from this endpoint’s Authorization header only when the endpoint name matches. With any other name, and no explicit setting, GitLab has no notification secret and the registry’s notifications are rejected 401 Unauthorized .
If you use an external registry (not the one integrated with GitLab), there is no registry['notifications'] block to derive the secret from, so you must set it yourself in /etc/gitlab/gitlab.rb : ruby gitlab_rails [ 'registry_notification_secret' ] = '<replace_with_a_secret_token>' For GitLab HA only. Edit /etc/gitlab/gitlab.rb on every web node: ruby gitlab_rails [ 'registry_notification_secret' ] = '<replace_with_a_secret_token_generated_above>' Web nodes have no registry['notifications'] block, so there is nothing to derive the secret from and it must be set explicitly.
A web node that does carry the block, because the same /etc/gitlab/gitlab.rb is copied to every node, derives the secret the same way the registry node does. Reconfigure each node you just updated: shell gitlab-ctl reconfigure Configure secondary site Make sure you have container registry set up and working on the secondary site before following the next steps. The following steps should be done on each secondary site you’re expecting to see the container images replicated.
Because we need to allow the secondary site to communicate securely with the primary site container registry, we need to have a single key pair for all the sites. The secondary site uses this key to generate a short-lived JWT that is pull-only-capable to access the primary site container registry. For each application and Sidekiq node on the secondary site: SSH into the node and sign in as the root user: shell sudo -i Copy /var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key from the primary to the node.
Edit /etc/gitlab/gitlab.rb and add: ruby gitlab_rails [ 'geo_registry_replication_enabled' ] = true # Primary registry's hostname and port, it will be used by # the secondary node to directly communicate to primary registry gitlab_rails [ 'geo_registry_replication_primary_api_url' ] = 'https://primary.example.com:5050/' Reconfigure the node for the change to take effect: shell gitlab-ctl reconfigure Disable container registry replication When the container registry service is disabled, Geo automatically skips verification attempts for container repositories.
Container registry metrics are hidden from the Geo dashboard. Existing verification states remain in the database but are not displayed. To stop replicating container images to a secondary site and disable the local container registry: SSH into the primary server node and sign in as the root user: shell sudo -i Edit /etc/gitlab/gitlab.rb and add: ruby # Disable Geo replication for the container registry gitlab_rails [ 'geo_registry_replication_enabled' ] = false # Disable the container registry service itself registry [ 'enable' ] = false Reconfigure the node for the change to take effect: shell sudo gitlab-ctl reconfigure Verify replication To verify container registry replication is working, on the secondary site: In the upper-right corner, select Admin .
In the left sidebar, select Geo > Nodes . The initial replication, or “backfill”, is probably still in progress. You can monitor the synchronization process on each Geo site from the primary site’s Geo Nodes dashboard in your browser. Garbage collection on Geo sites Garbage collection on the primary site does not free storage on the secondary site. Each site must clean up its own container registry storage. When you delete a tag on the primary site, the deletion is replicated to the secondary site.
However, the image layers remain in storage on both sites until garbage collection removes them. How you run garbage collection depends on your registry configur
The rest of this page is on the live site.