Cloud PyPI / Proposal - Python Wiki
Python Wiki Python Wiki / Python Wiki / Packaging / Cloud PyPI / Cloud PyPI / Proposal View as Markdown Cloud PyPI / Proposal ¶ Legacy Wiki Page This page was migrated from the old MoinMoin-based wiki. Information may be outdated or no longer applicable. For current documentation, see python.org . Cloud PyPI Proposal ¶ This is the third revision of the proposal text which was accepted by the PSF board in their December 2010 meeting.
PSF-Proposal: 100 Title: Move PyPI static data to the cloud for better availability Version: 3 Last-Modified: 2010-08-16 Author: mal@lemburg.com (Marc-André Lemburg) Discussions-To: catalog-sig@python.org Status: Draft Type: Informational Created: 2010-06-14 Post-History: Proposal: Move PyPI static data to the cloud for better availability ======================================================================== Motivation ---------- PyPI has in recent months seen several outages with the index being unavailable to both users using the web GUI interface as well as package administration tools such as easy_install from setuptools.
As more and more Python applications rely on tools such as easy_install for direct installation, or zc.buildout to manage the complete software configuration cycle, the PyPI infrastructure receives more and more attention from the Python community. While we just started to have monitoring for PyPI installed and the first reports of outtages are now being sent to the catalog-sig mailing list, we don't have reliable numbers available yet.
However, the number of discussions about PyPI outtages in the mailing lists has increased to a point where we cannot simply ignore those complaints anymore. In order to maintain its credibility as software repository, to support the many different projects relying on the PyPI infrastructure and the many users who rely on the simplified installation process enabled by PyPI, the PSF needs to take action and move the essential parts of PyPI to a more robust infrastructure that provides: * scalability * 24/7 outsourced system administration management * redundant storage * geo-localized fast and reliable access Current Situation ----------------- PyPI is currently run from a single server hosted in The Netherlands (ximinez.python.org).
This server is run by a very small team of sys admin. PyPI itself has in recent months been mostly maintained by one developer: Martin von Löwis. Projects are underway to enhance PyPI in various ways, including a proposal to add external mirroring (PEP 381), but these are still not fully finalized and require implementation changes/new features in the existing client tools. According to Martin, the server side features of PEP 381, including a few undocumented extensions to provide package signatures, are already implemented.
However, without client tools to make use of them, this is not going to change the current situation for existing PyPI users. Furthermore those client tools enhancements would first have to get adopted by PyPI users by either replacing their client tools with updated versions or switching to new client tools, which is likely going to take months to years. Existing client tool users won't see an immediate improvement. Usage ----- PyPI provides four different mechanisms for accessing the stored information: * a web GUI that is meant for use by humans * an RPC interface which is mostly used for uploading new content * a semi-static /simple package listing, used by setuptools * a static area /packages for package download files and documentation, used by both the web GUI and setuptools The /simple package listing is dump of all packages in PyPI using a simple HTML page with links to sub-pages for each package.
These sub-pages provide links to download files and external references. External tools like easy_install only use the /simple package listing together with the hosted package download files. While the /simple package listing is currently dynamically created from the database in real-time, this is not really needed for normal operation. A static copy created every 10-20 minutes would provide the same level of service in much the same way.
Moving static data to a CDN --------------------------- Under the proposal the static information stored in PyPI (meta-information as well as package download files and documentation) is moved to a content delivery network (CDN). For this purpose, the /simple package listing is replaced with a static copy that is recreated every 10-20 minutes using a cronjob on the PyPI server. At the same intervals, another script will scan the package and documentation files under /packages for updates and upload any changes to the CDN for neartime availability.
By using a CDN the PSF will enable and provide: * high availability of the static PyPI content * offload management to the CDN * enable geo-localized downloads, i.e. the files are hosted on a nearby server * faster downloads * more reliability and scalability * move away from a single point of failure setup Note that the proposal does not cover distribution of the dynamic parts of PyPI. As a result uploads to PyPI may still fail if the PyPI server goes down.
However, these dynamic parts are currently not being used by the existing package installation tools. Choice of CDN: Amazon Cloudfront -------------------------------- To keep the costs low for the PSF, Amazon Cloudfront appears to be the bext choice for CDN. Cloudfront is supported by a set of Python libraries (e.g. Amazon S3 lib and boto), upload scripts are readily available and can easily be customized. http://www.saltycrane.com/blog/2008/12/card-store-project-4-notes-using-amazons-cloudfront/ Other CDNs, such as Akamai, are either more expensive or require custom integration.
Availability of Python-based tools is not always given, in fact, accessing such information is difficult for most of the proprietary CDNs. Cloudfront: quality of service ------------------------------ Amazon Cloudfront uses S3 as basis for the service, S3 has been around for years and has a very stable uptime: http://www.readwriteweb.com/archives/amazon_s3_exceeds_9999_percent_uptime.php Cloudfront itself has been around since Nov 2008.
Amazon still uses the web 2.0 "beta" marketing term on it. You can check their current online status using this panel: http://status.aws.amazon.com/ Apart from the gained availability and outsourced management, we'd also get faster downloads in most parts of the world, due to the local caching Cloudfront is applying. This caching can be used to further increase the availability, since we can control the expiry time of those local copies.
So in summary, we are replacing a single point of failure with an N server fail-over system (with N being the number of edge caching servers they use). How Cloudfront works -------------------- Cloudfront uses Amazon's S3 storage system which is based on "buckets". These can store any number of files in a directory-like structure. The only limit is a 5GB per file limit - more than enough for any PyPI package file. Cloudfront provides a domain for each registered S3 bucket via a "distribution" which is then made available through local cache servers in various locations around the world.
The management of which server to use for an incoming request is transparently handled by Amazon. Once uploaded to the S3 bucket, the files will be distributed to the cache servers on demand and as necessary. Each edge server server maintains a cache of requested files and refetches the files after an expiry time which can be defined when uploading the file to the bucket. To simplify things on our side, we'll setup a CNAME DNS alias for the Cloudfront domain issued by Amazon to our bucket: pypi-static.python.org.
IN CNAME d32z1yuk7jeryy.cloudfront.net. In the unlikely event of a longer downtime of the whole Amazon Cloudfront system, our system administrators could then easily change the DNS alias pypi-static.python.org to point back to the PyPI serv
The rest of this page is on the live site.