Python Web Scraping: Verify SSL certificates for HTTPS requests using requests module

Python Web Scraping: Exercise-27 with Solution

Write a Python program to verify SSL certificates for HTTPS requests using requests module.

Note: Requests verifies SSL certificates for HTTPS requests, just like a web browser. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate

Sample Solution:

Python Code:

import requests
print(requests.get('https://w3resource.com'))
print(requests.get('https://wayback.com'))
  
 

Sample Output:

<Response [200]>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 364, in connect
    _match_hostname(cert, self.assert_hostname or server_hostname)
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connection.py", line 374, in _match_hostname
    match_hostname(cert, asserted_hostname)
  File "/usr/lib/python3.6/ssl.py", line 327, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'wayback.com' doesn't match either of '*.prod.iad2.secureserver.net', 'prod.iad2.secureserver.net'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/lib/python3.6/dist-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='wayback.com', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'wayback.com' doesn't match either of '*.prod.iad2.secureserver.net', 'prod.iad2.secureserver.net'",),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/sessions/d2a193285f0c778a/main.py", line 3, in <module>
    print(requests.get('https://wayback.com'))
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='wayback.com', port=443): Max retries exceeded with url: / (Caused by SSLError(CertificateError("hostname 'wayback.com' doesn't match either of '*.prod.iad2.secureserver.net', 'prod.iad2.secureserver.net'",),))

Flowchart:

Python Code Editor:

Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Write a Python program to display the contains of different attributes like status_code, headers, url, history, encoding, reason, cookies, elapsed, request and content of a specified resource.