PEP 512 – Migrating from hg.python.org to GitHub | peps.python.org
PEP 512 – Migrating from hg.python.org to GitHub PEP 512 – Migrating from hg.python.org to GitHub Author : Brett Cannon <brett at python.org> Discussions-To : Core-Workflow list Status : Final Type : Process Created : 17-Jan-2015 Post-History : 17-Jan-2016, 19-Jan-2016, 23-Jan-2016 Table of Contents Abstract Rationale Repositories to Migrate Migration Plan Requirements for Code-Only Repositories Create a ‘Python core’ team Define commands to move a Mercurial repository to Git CLA enforcement Adding GitHub username support to bugs.python.org A bot to enforce CLA signing Make old repository read-only Requirements for Web-Related Repositories Requirements for the cpython Repository Document steps to commit a pull request Linking pull requests to issues Linking a pull request to an issue Notify the issue if a commit is made Update the linking service for mapping commit IDs to URLs Deprecate sys._mercurial Update the devguide Update PEP 101 Optional, Planned Features Handling Misc/NEWS Handling Misc/ACKS Create https://git.python.org Backup of pull request data Bot to generate cherry-pick pull requests Pull request commit queue A CI service Test coverage report Notifying issues of pull request comments Allow bugs.python.org to use GitHub as a login provider Web hooks for re-generating web content Link web content back to files that it is generated from Splitting out parts of the documentation into their own repositories Backup of Git repositories Identify potential new core developers Status cpython repo Open Issues The fate of hg.python.org Git CLI commands for committing a pull request to cpython Naming the bots Rejected Ideas Separate Python 2 and Python 3 repositories Commit multi-release changes in bugfix branch first Deriving Misc/NEWS from the commit logs Deriving Misc/NEWS from bugs.python.org References Copyright Note CPython’s development process moved to https://github.com/python/cpython on 2017-02-10.
Abstract This PEP outlines the steps required to migrate Python’s development process from Mercurial [3] as hosted at hg.python.org [1] to Git [4] on GitHub [2] . Meeting the minimum goals of this PEP should allow for the development process of Python to be as productive as it currently is, and meeting its extended goals should improve the development process from its status quo. Rationale In 2014, it became obvious that Python’s custom development process was becoming a hindrance.
As an example, for an external contributor to submit a fix for a bug that eventually was committed, the basic steps were: Open an issue for the bug at bugs.python.org [5] . Checkout out the CPython source code from hg.python.org [1] . Make the fix. Upload a patch. Have a core developer review the patch using our fork of the Rietveld code review tool [6] . Download the patch to make sure it still applies cleanly. Run the test suite manually.
Update the NEWS , ACKS , and “What’s New” document as necessary Pull changes to avoid a merge race. Commit the change manually. If the change was for a bugfix release, merge into the in-development branch. Run the test suite manually again. Commit the merge. Push the changes. This is a very heavy, manual process for core developers. Even in the simple case, you could only possibly skip the code review step, as you would still need to build the documentation.
This led to patches languishing on the issue tracker due to core developers not being able to work through the backlog fast enough to keep up with submissions. In turn, that led to a side-effect issue of discouraging outside contribution due to frustration from lack of attention, which is a dangerous problem for an open source project with no corporate backing as it runs counter to having a viable future for the project.
While allowing patches to be uploaded to bugs.python.org [5] is potentially simple for an external contributor, it is as slow and burdensome as it gets for a core developer to work with. Hence the decision was made in late 2014 that a move to a new development process was needed. A request for PEPs proposing new workflows was made, in the end leading to two: PEP 481 and PEP 507 proposing GitHub [2] and GitLab [7] , respectively.
The year 2015 was spent off-and-on working on those proposals and trying to tease out details of what made them different from each other on the core-workflow mailing list [8] . PyCon US 2015 also showed that the community was a bit frustrated with our process due to both cognitive overhead for new contributors and how long it was taking for core developers to look at a patch (see the end of Guido van Rossum’s keynote at PyCon US 2015 [9] as an example of the frustration).
On January 1, 2016, the decision was made by Brett Cannon to move the development process to GitHub. The key reasons for choosing GitHub were [10] : Maintaining custom infrastructure has been a burden on volunteers (e.g., an unmaintained, custom fork of Rietveld [6] is currently being used). The custom workflow is very time-consuming for core developers (not enough automated tooling built to help support it). The custom workflow is a hindrance to external contributors (acts as a barrier of entry due to time required to ramp up on development process unique to CPython itself).
There is no feature differentiating GitLab from GitHub beyond GitLab being open source. Familiarity with GitHub is far higher among core developers and external contributors than with GitLab. Our BDFL prefers GitHub (who would be the first person to tell you that his opinion shouldn’t matter, but the person making the decision felt it was important that the BDFL feel comfortable with the workflow of his own programming language to encourage his continued participation).
There’s even already an unofficial logo to represent the migration to GitHub [22] . The overarching goal of this migration is to improve the development process to the extent that a core developer can go from external contribution submission through all the steps leading to committing said contribution from within a browser on a tablet with WiFi using some development process (this does not inherently mean GitHub’s default workflow).
The final solution will also allow an external contributor to contribute even if they chose not to use GitHub (although there is not guarantee in feature parity). Repositories to Migrate While hg.python.org [1] hosts many repositories, there are only five key repositories that need to move: devinabox [12] (done) benchmarks [11] (skipped) peps [13] (done) devguide [14] (done) cpython [15] The devinabox repository is code-only.
The peps and devguide repositories involve the generation of webpages. And the cpython repository has special requirements for integration with bugs.python.org [5] . Migration Plan The migration plan is separated into sections based on what is required to migrate the repositories listed in the Repositories to Migrate section. Completion of requirements outlined in each section should unblock the migration of the related repositories.
The sections are expected to be completed in order, but not necessarily the requirements within a section. Requirements for Code-Only Repositories Completion of the requirements in this section will allow the devinabox repository to move to GitHub. Create a ‘Python core’ team To manage permissions, a ‘Python core’ team will be created as part of the python organization [16] . Any repository that is moved will have the ‘Python core’ team added to it with write permissions [17] .
Anyone who previously had rights to manage SSH keys on hg.python.org will become a team maintainer for the ‘Python core’ team. Define commands to move a Mercurial repository to Git Since moving to GitHub also entails moving to Git [4] , we must decide what tools and commands we will run to translate a Mercurial repository to Git. The tools developed specifically for this migration are hosted at https://github.com/orsenthil/cpython-h
The rest of this page is on the live site.