urllib.request — Extensible library for opening URLs — Python 3.16.0a1 documentation
Navigation index modules | next | previous | Python » 3.16.0a1 Documentation » The Python standard library » Internet Protocols and Support » urllib.request — Extensible library for opening URLs | Theme | urllib.request — Extensible library for opening URLs ¶ Source code: Lib/urllib/request.py The urllib.request module defines functions and classes which help in opening URLs (mostly HTTP) in a complex world — basic and digest authentication, redirections, cookies and more.
See also The Requests package is recommended for a higher-level HTTP client interface. Warning On macOS it is unsafe to use this module in programs using os.fork() because the getproxies() implementation for macOS uses a higher-level system API. Set the environment variable no_proxy to * to avoid this problem (e.g. os.environ["no_proxy"] = "*" ). Availability : not WASI. This module does not work or is not available on WebAssembly.
See WebAssembly platforms for more information. The urllib.request module defines the following functions: urllib.request. urlopen ( url , data=None , [ timeout , ] * , context=None ) ¶ Open url , which can be either a string containing a valid, properly encoded URL, or a Request object. data must be an object specifying additional data to be sent to the server, or None if no such data is needed. See Request for details. urllib.request module uses HTTP/1.1 and includes Connection:close header in its HTTP requests.
The optional timeout parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS and FTP connections. If context is specified, it must be a ssl.SSLContext instance describing the various SSL options. See HTTPSConnection for more details. This function always returns an object which can work as a context manager and has the properties url , headers , and status .
See urllib.response.addinfourl for more detail on these properties. For HTTP and HTTPS URLs, this function returns a http.client.HTTPResponse object slightly modified. In addition to the three new methods above, the msg attribute contains the same information as the reason attribute — the reason phrase returned by server — instead of the response headers as it is specified in the documentation for HTTPResponse . For FTP, file, and data URLs, this function returns a urllib.response.addinfourl object.
Raises URLError on protocol errors. Note that None may be returned if no handler handles the request (though the default installed global OpenerDirector uses UnknownHandler to ensure this never happens). In addition, if proxy settings are detected (for example, when a *_proxy environment variable like http_proxy is set), ProxyHandler is default installed and makes sure the requests are handled through the proxy. The legacy urllib.urlopen function from Python 2.6 and earlier has been discontinued; urllib.request.urlopen() corresponds to the old urllib2.urlopen .
Proxy handling, which was done by passing a dictionary parameter to urllib.urlopen , can be obtained by using ProxyHandler objects. The default opener raises an auditing event urllib.Request with arguments fullurl , data , headers , method taken from the request object. Changed in version 3.2: cafile and capath were added. HTTPS virtual hosts are now supported if possible (that is, if ssl.HAS_SNI is true). data can be an iterable object.
Changed in version 3.3: cadefault was added. Changed in version 3.4.3: context was added. Changed in version 3.10: HTTPS connection now send an ALPN extension with protocol indicator http/1.1 when no context is given. Custom context should set ALPN protocols with set_alpn_protocols() . Changed in version 3.13: Remove cafile , capath and cadefault parameters: use the context parameter instead. urllib.request. install_opener ( opener ) ¶ Install an OpenerDirector instance as the default global opener.
Installing an opener is only necessary if you want urlopen to use that opener; otherwise, simply call OpenerDirector.open() instead of urlopen() . The code does not check for a real OpenerDirector , and any class with the appropriate interface will work. urllib.request. build_opener ( [ handler , ... ] ) ¶ Return an OpenerDirector instance, which chains the handlers in the order given. handler s can be either instances of BaseHandler , or subclasses of BaseHandler (in which case it must be possible to call the constructor without any parameters).
Instances of the following classes will be in front of the handler s, unless the handler s contain them, instances of them or subclasses of them: ProxyHandler (if proxy settings are detected), UnknownHandler , HTTPHandler , HTTPDefaultErrorHandler , HTTPRedirectHandler , FTPHandler , FileHandler , HTTPErrorProcessor . If the Python installation has SSL support (i.e., if the ssl module can be imported), HTTPSHandler will also be added.
A BaseHandler subclass may also change its handler_order attribute to modify its position in the handlers list. urllib.request. pathname2url ( path , * , add_scheme = False ) ¶ Convert the given local path to a file: URL. This function uses quote() function to encode the path. If add_scheme is false (the default), the return value omits the file: scheme prefix. Set add_scheme to true to return a complete URL. This example shows the function being used on Windows: >>> from urllib.request import pathname2url >>> path = 'C: \\ Program Files' >>> pathname2url ( path , add_scheme = True ) 'file:///C:/Program%20Files' Changed in version 3.14: Windows drive letters are no longer converted to uppercase, and : characters not following a drive letter no longer cause an OSError exception to be raised on Windows.
Changed in version 3.14: Paths beginning with a slash are converted to URLs with authority sections. For example, the path /etc/hosts is converted to the URL ///etc/hosts . Changed in version 3.14: The add_scheme parameter was added. urllib.request. url2pathname ( url , * , require_scheme = False , resolve_host = False ) ¶ Convert the given file: URL to a local path. This function uses unquote() to decode the URL. If require_scheme is false (the default), the given value should omit a file: scheme prefix.
If require_scheme is set to true, the given value should include the prefix; a URLError is raised if it doesn’t. The URL authority is discarded if it is empty, localhost , or the local hostname. Otherwise, if resolve_host is set to true, the authority is resolved using socket.gethostbyname() and discarded if it matches a local IP address (as per RFC 8089 §3 ). If the authority is still unhandled, then on Windows a UNC path is returned, and on other platforms a URLError is raised.
This example shows the function being used on Windows: >>> from urllib.request import url2pathname >>> url = 'file:///C:/Program %20F iles' >>> url2pathname ( url , require_scheme = True ) 'C:\\Program Files' Changed in version 3.14: Windows drive letters are no longer converted to uppercase, and : characters not following a drive letter no longer cause an OSError exception to be raised on Windows. Changed in version 3.14: The URL authority is discarded if it matches the local hostname.
Otherwise, if the authority isn’t empty or localhost , then on Windows a UNC path is returned (as before), and on other platforms a URLError is raised. Changed in version 3.14: The URL query and fragment components are discarded if present. Changed in version 3.14: The require_scheme and resolve_host parameters were added. urllib.request. getproxies ( ) ¶ This helper function returns a dictionary of scheme to proxy server URL mappings.
It scans the environment for variables named <scheme>_proxy , in a case insensitive approach, for all operating systems first, and when it cannot find it, looks for proxy information from System Configuration for macOS and Windows Systems Registry for Windows. If both lowercase
The rest of this page is on the live site.