Text of docs.gitlab.com › development › github_importer as Styx crawled it on Oct 12, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

GitHub importer developer documentation | GitLab Docs

/ GitHub importer developer documentation The GitHub importer is a parallel importer that uses Sidekiq. Prerequisites Sidekiq workers that process the github_importer and github_importer_advance_stage queues (enabled by default). Octokit (used for interacting with the GitHub API). Code structure The importer’s codebase is broken up into the following directories: lib/gitlab/github_import : this directory contains most of the code such as the classes used for importing resources. app/workers/gitlab/github_import : this directory contains the Sidekiq workers. app/workers/concerns/gitlab/github_import : this directory contains a few modules reused by the various Sidekiq workers.

Architecture overview When a GitHub project is imported, work is divided into separate stages, with each stage consisting of a set of Sidekiq jobs that are executed. Between every stage a job is scheduled that periodically checks if all work of the current stage is completed, advancing the import process to the next stage when this is the case. The worker handling this is called Gitlab::GithubImport::AdvanceStageWorker .

An import is initiated via an API request to POST /import/github The API endpoint calls Import::GitHubService . Which calls Gitlab::LegacyGithubImport::ProjectCreator Which calls Projects::CreateService Which calls @project.import_state.schedule Which calls project.add_import_job Which calls RepositoryImportWorker Stages 1. RepositoryImportWorker This worker calls Projects::ImportService.new.execute , which calls importer.execute .

In this context, importer is an instance of Gitlab::ImportSources.importer(project.import_type) , which for github import types maps to ParallelImporter . ParallelImporter schedules a job for the next worker. 2. Stage::ImportRepositoryWorker This worker imports the repository and wiki, scheduling the next stage when done. 3. Stage::ImportBaseDataWorker This worker imports base data such as labels, milestones, and releases.

This work is done in a single thread because it can be performed fast enough that we don’t need to perform this work in parallel. 4. Stage::ImportPullRequestsWorker This worker imports all pull requests. For every pull request a job for the Gitlab::GithubImport::ImportPullRequestWorker worker is scheduled. 5. Stage::ImportCollaboratorsWorker This worker imports only direct repository collaborators who are not outside collaborators.

For every collaborator, we schedule a job for the Gitlab::GithubImport::ImportCollaboratorWorker worker. This stage is optional (controlled by Gitlab::GithubImport::Settings ) and is selected by default. 6. Stage::ImportIssuesAndDiffNotesWorker This worker imports all issues and pull request comments. For every issue, we schedule a job for the Gitlab::GithubImport::ImportIssueWorker worker. For pull request comments, we instead schedule jobs for the Gitlab::GithubImport::DiffNoteImporter worker.

This worker processes both issues and diff notes in parallel so we don’t need to schedule a separate stage and wait for the previous one to complete. Issues are imported separately from pull requests because only the “issues” API includes labels for both issue and pull requests. Importing issues and setting label links in the same worker removes the need for performing a separate crawl through the API data, reducing the number of API calls necessary to import a project.

7. Stage::ImportIssueEventsWorker This worker imports all issues and pull request events. For every event, we schedule a job for the Gitlab::GithubImport::ImportIssueEventWorker worker. We can import both issues and pull request events by single stage because of a specific aspect of the GitHub API. It looks like that under the hood, issues and pull requests GitHub are stored in a single table. Therefore, they have globally-unique IDs and so: Every pull request is an issue.

Issues aren’t pull requests. Therefore, both issues and pull requests have a common API for most related things. To facilitate the import of pull request review requests using the timeline events endpoint, events must be processed sequentially. Given that import workers do not execute in a guaranteed order, the pull request review requests events are initially placed in a Redis ordered list. Subsequently, they are consumed in sequence by the Gitlab::GithubImport::ReplayEventsWorker .

8. Stage::ImportAttachmentsWorker This worker imports note attachments that are linked inside Markdown. For each entity with Markdown text in the project, we schedule a job of: Gitlab::GithubImport::Importer::Attachments::ReleasesImporter for every release. Gitlab::GithubImport::Importer::Attachments::NotesImporter for every note. Gitlab::GithubImport::Importer::Attachments::IssuesImporter for every issue. Gitlab::GithubImport::Importer::Attachments::MergeRequestsImporter for every merge request.

Each job: Iterates over all attachment links inside of a specific record. Downloads the attachment. Replaces the old link with a newly-generated link to GitLab. It’s an optional stage that could consume significant extra import time (controlled by Gitlab::GithubImport::Settings ). 9. Stage::ImportProtectedBranchesWorker This worker imports protected branch rules. For every rule that exists on GitHub, we schedule a job of Gitlab::GithubImport::ImportProtectedBranchWorker .

Each job compares the branch protection rules from GitHub and GitLab and applies the strictest of the rules to the branches in GitLab. 10. Stage::FinishImportWorker This worker completes the import process by performing some housekeeping (such as flushing any caches) and by marking the import as completed. Advancing stages Advancing stages is done in one of two ways: Scheduling the worker for the next stage directly. Scheduling a job for Gitlab::GithubImport::AdvanceStageWorker which will advance the stage when all work of the current stage has been completed.

The first approach should only be used by workers that perform all their work in a single thread, while AdvanceStageWorker should be used for everything else. An example of the first approach is how ImportBaseDataWorker invokes PullRequestWorker directly . An example of the second approach is how PullRequestsWorker invokes the AdvanceStageWorker when its own work has been completed . When you schedule a job, AdvanceStageWorker is given a project ID, a list of Redis keys, and the name of the next stage.

The Redis keys (produced by Gitlab::JobWaiter ) are used to check if the running stage has been completed or not. If the stage has not yet been completed AdvanceStageWorker reschedules itself. After a stage finishes, or if more jobs have been finished after the last invocation. AdvanceStageworker refreshes the import JID (more on this below) and schedule the worker of the next stage. To reduce the number of AdvanceStageWorker jobs scheduled this worker briefly waits for jobs to complete before deciding what the next action should be.

For small projects, this may slow down the import process a bit, but it also reduces pressure on the system as a whole. User contribution mapping The GitHub importer supports user contribution mapping , which allows imported records to be attributed to placeholder users until a real user can be assigned after the import completes. Key classes Class Purpose GithubImport::UserFinder Class that calls Import::SourceUserMapper to map user data from GitHub API responses to GitLab User records GithubImport::Importer::CollaboratorImporter calls PlaceholderMemberships::CreateService to create placeholder memberships for the source users Deleted users and ghost user handling GitHub has a special “ghost” user (username: ghost ) that represents deleted users.

When the importer encounters this user, it maps the user directly to the GitLab ghost user without creating an Import::SourceUser or placeholder user. Refreshing import job IDs GitLab includes a worker called Gitlab::Import::StuckProjectImportJobsWorker that periodically runs a

The rest of this page is on the live site.