Python:urllib

lastmod 2024-08-05created 2015-10-2913 edits

Example

The following is an example of making a PUT request using Request:

import urllib.request
DATA = b'some data'
req = urllib.request.Request(url='http://localhost:8080', data=DATA, method='PUT')
with urllib.request.urlopen(req) as f:
    pass
print(f.status)
print(f.reason)
print(u.getcode())
print(u.headers)
print(u.read())

Example of requesting josn data with POST:

req = Request("http://localhost:48888/live", headers={"Content-Type": "application/json"}, method="POST")
data = b'{"live": [], "history": []}'
with urlopen(req, data) as f
    f.read()
    # ...

application/x-www-form-urlencoded

urllib.parse — Parse URLs into components

String arguments (application/x-www-form-urlencoded MIME) can be parsed using urllib.parse.parse_qsl or urllib.parse.parse_qs.

To convert to a query string (URL encoding), use urllib.parse.urlencode.

The ParseResult object returned as the result of urlencode can be turned back into the original URL text using urllib.parse.urlunparse.

Troubleshooting

CERTIFICATE_VERIFY_FAILED

In Python, the following error may occur when connecting over https.

/usr/local/c2core/lib/python3.7/site-packages/torch/hub.py:429: UserWarning: torch.hub._download_url_to_file has been renamed to            torch.hub.download_url_to_file to be a public API,            _download_url_to_file will be removed in after 1.3 release
  _download_url_to_file will be removed in after 1.3 release')
Traceback (most recent call last):
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 1317, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/local/c2core/lib/python3.7/http/client.py", line 1392, in connect
    server_hostname=server_hostname)
  File "/usr/local/c2core/lib/python3.7/ssl.py", line 412, in wrap_socket
    session=session
  File "/usr/local/c2core/lib/python3.7/ssl.py", line 853, in _create
    self.do_handshake()
  File "/usr/local/c2core/lib/python3.7/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "webcam.py", line 80, in <module>
    main()
  File "webcam.py", line 64, in main
    min_image_size=args.min_image_size,
  File "/home/aaa/Project/_c2core/script/python/maskrcnn_benchmark/demo/predictor.py", line 117, in __init__
    _ = checkpointer.load(cfg.MODEL.WEIGHT)
  File "/usr/local/c2core/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/utils/checkpoint.py", line 61, in load
    checkpoint = self._load_file(f)
  File "/usr/local/c2core/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/utils/checkpoint.py", line 129, in _load_file
    cached_f = cache_url(f)
  File "/usr/local/c2core/lib/python3.7/site-packages/maskrcnn_benchmark-0.1-py3.7-linux-x86_64.egg/maskrcnn_benchmark/utils/model_zoo.py", line 55, in cache_url
    _download_url_to_file(url, cached_file, hash_prefix, progress=progress)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/hub.py", line 430, in _download_url_to_file
    download_url_to_file(url, dst, hash_prefix, progress)
  File "/usr/local/c2core/lib/python3.7/site-packages/torch/hub.py", line 384, in download_url_to_file
    u = urlopen(url)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 1360, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/local/c2core/lib/python3.7/urllib/request.py", line 1319, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1056)>

This is because, under the revised PEP 467, all https communication checks the required certificate and hostname by default. The affected libraries are urllib, urllib2, http, and httplib.

Install the certificates.

$ pip install certifi

If you want to fix it directly in code, pass ssl._create_unverified_context() as the context parameter of urllib.request.urlopen:

context = ssl._create_unverified_context()
response = urllib.request.urlopen(requests, data=data.encode('utf-8'), context=context)

Or change the default context as shown below.

import ssl
try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context

Alternative when curl is not available

python -c "import urllib.request as r; exit(0 if r.urlopen('http://www.python.org/').status == 201 else 1)"

Favorite site

Web client programming with Python

author Zer0 sitemap