Logging HTTP Requests in Python urllib3: Understanding Request Hooks

Last update on December 21 2024 07:48:56 (UTC/GMT +8 hours)

Write a Python program that uses request hooks to log information about each request made with urllib3.

Sample Solution:

Python Code :

# Import the urllib3 library
import urllib3
# Custom request hook function to log information
def logging_request_hook(response, *args, **kwargs):
    # Log information about the response
    print(f"Received response with status code: {response.status}")
    print(f"Response Headers: {response.headers}")

# Function to make a sample request using the custom hook
def make_sample_request():
    # Create a PoolManager with the custom request hook
    http = urllib3.PoolManager()

    # Define the URL for the sample request
    sample_url = 'https://www.example.com'

    try:
        # Make a GET request using the PoolManager
        response = http.request('GET', sample_url, headers={'User-Agent': 'Custom User Agent'},
                                preload_content=False, retries=False)

        # Check if the request was successful (status code 200)
        if response.status == 200:
            print("Request Successful:")
            print(response.data.decode('utf-8'))
        else:
            print(f"Error: Unable to fetch data. Status Code: {response.status}")

    except urllib3.exceptions.RequestError as e:
        print(f"Error: {e}")

if __name__ == "__main__":
    # Make a sample request with the custom request hook
    make_sample_request()

Sample Output:

Request Successful:
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
        
    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }
    </style>    
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

Explanation:

Here's a brief explanation of the above Python urllib3 library code:

Custom Request Hook Function (logging_request_hook):

The "logging_request_hook()" function is a custom request hook provided to the urllib3 library.
It takes the "response" object, which contains information about the HTTP response, as well as additional "args" and "*kwargs" parameters.

Function to Make a Sample Request (make_sample_request):

The "make_sample_request()" function demonstrates how to use the custom request hook.
It creates a PoolManager from urllib3, specifying the custom request hook (hooks={'response': logging_request_hook}).
Then, it makes a sample GET request to 'https://www.example.com' with a custom User-Agent header.
The 'preload_content' and 'retries' parameters are set to 'False' to avoid loading the content into memory and disabling retries.

Execution in the Main Block (`if name == "main":):

Calls the "make_sample_request()" function when the script is executed.

Output:

The custom request hook logs information about the received response, including the status code and headers.

Flowchart:

Python Code Editor :

Have another way to solve this solution? Contribute your code (and comments) through Disqus.

Previous: Constructing URLs with Python urllib3: Query Parameter Encoding Explained.
Next: Making API Requests with digest authentication in Python urllib3.