Python Requests & HTTPX: The Complete Guide for 2026

Making HTTP requests is one of the most common tasks in Python development. Whether you are consuming a REST API, scraping a webpage, downloading files, or integrating microservices, you need a reliable HTTP client. Python offers two excellent options: the battle-tested requests library and the modern httpx library that adds async support and HTTP/2.

This guide covers both libraries in depth. You will learn every major feature from basic GET and POST requests through authentication, file uploads, session management, retries, and error handling. We then explore HTTPX's async capabilities and HTTP/2 support, compare the two libraries head-to-head, and finish with real-world patterns for pagination, rate limiting, and testing.

1. Requests Basics: GET, POST, PUT, DELETE

The requests library is the most downloaded Python package on PyPI. Install it with pip and start making HTTP calls immediately:

pip install requests

Every HTTP method has a corresponding function:

import requests

# GET - retrieve data
response = requests.get("https://api.example.com/users")
print(response.status_code)  # 200
print(response.text)         # raw response body

# POST - create a resource
response = requests.post("https://api.example.com/users", json={
    "name": "Alice",
    "email": "alice@example.com"
})
print(response.json())  # parsed JSON response

# PUT - update a resource completely
response = requests.put("https://api.example.com/users/42", json={
    "name": "Alice Smith",
    "email": "alice.smith@example.com"
})

# PATCH - partial update
response = requests.patch("https://api.example.com/users/42", json={
    "email": "new-email@example.com"
})

# DELETE - remove a resource
response = requests.delete("https://api.example.com/users/42")

The json parameter automatically serializes the dictionary to JSON and sets the Content-Type: application/json header. For form data, use the data parameter instead.

2. Query Parameters, Headers, and Authentication

Pass query parameters as a dictionary instead of manually building URL strings:

# Query parameters - requests encodes them for you
response = requests.get("https://api.example.com/search", params={
    "q": "python http client",
    "page": 1,
    "per_page": 20,
    "sort": "relevance"
})
# Actual URL: https://api.example.com/search?q=python+http+client&page=1&...

# Custom headers
response = requests.get("https://api.example.com/data", headers={
    "Accept": "application/json",
    "X-API-Version": "2",
    "User-Agent": "MyApp/1.0"
})

# Basic authentication
response = requests.get("https://api.example.com/me",
    auth=("username", "password"))

# Bearer token authentication
response = requests.get("https://api.example.com/me", headers={
    "Authorization": "Bearer eyJhbGciOiJIUzI1NiIs..."
})

For APIs requiring API key authentication, many accept the key as either a header or a query parameter. Check the API documentation for the expected location.

3. Handling JSON Responses

Most modern APIs return JSON. The .json() method parses the response body into a Python dictionary or list:

response = requests.get("https://api.github.com/repos/psf/requests")

# Parse JSON response
data = response.json()
print(data["full_name"])       # "psf/requests"
print(data["stargazers_count"])

# Always check the status code before parsing
if response.ok:  # True for any 2xx status
    data = response.json()
    process_data(data)
else:
    print(f"Error {response.status_code}: {response.text}")

# Access response headers
content_type = response.headers["Content-Type"]
rate_limit = response.headers.get("X-RateLimit-Remaining", "unknown")

The response.ok property returns True for any 2xx status code. For stricter checking, use response.raise_for_status() which raises an HTTPError for 4xx and 5xx responses.

4. File Uploads and Downloads

Upload files using the files parameter and download large files with streaming:

# Upload a file
with open("report.pdf", "rb") as f:
    response = requests.post("https://api.example.com/upload", files={
        "document": ("report.pdf", f, "application/pdf")
    })

# Upload multiple files
response = requests.post("https://api.example.com/upload", files=[
    ("images", ("photo1.jpg", open("photo1.jpg", "rb"), "image/jpeg")),
    ("images", ("photo2.jpg", open("photo2.jpg", "rb"), "image/jpeg")),
])

# Download a large file with streaming
with requests.get("https://example.com/large-file.zip", stream=True) as r:
    r.raise_for_status()
    with open("large-file.zip", "wb") as f:
        for chunk in r.iter_content(chunk_size=8192):
            f.write(chunk)

Streaming is essential for large downloads. Without stream=True, requests loads the entire response into memory at once, which can crash your program for multi-gigabyte files.

5. Session Objects and Connection Pooling

A Session object persists settings across requests and reuses the underlying TCP connection, which is significantly faster for multiple calls to the same host:

session = requests.Session()

# Set default headers for all requests in this session
session.headers.update({
    "Authorization": "Bearer my-token",
    "Accept": "application/json",
    "User-Agent": "MyApp/2.0"
})

# All these requests reuse the same TCP connection
users = session.get("https://api.example.com/users").json()
posts = session.get("https://api.example.com/posts").json()
comments = session.get("https://api.example.com/comments").json()

# Sessions also persist cookies automatically
session.post("https://example.com/login", data={
    "username": "alice",
    "password": "secret"
})
# Subsequent requests include the session cookie
profile = session.get("https://example.com/profile").json()

# Always close sessions when done, or use a context manager
session.close()

# Better: use as context manager
with requests.Session() as s:
    s.headers["Authorization"] = "Bearer my-token"
    data = s.get("https://api.example.com/data").json()

Connection pooling means the TCP handshake and TLS negotiation happen only once per host. For applications making many requests to the same API, sessions can reduce latency by 50% or more.

6. Timeout and Retry Strategies

Never make HTTP requests without a timeout. The default in requests is to wait forever:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# Always set a timeout (connect_timeout, read_timeout)
response = requests.get("https://api.example.com/data", timeout=(3.05, 30))

# Configure automatic retries with exponential backoff
retry_strategy = Retry(
    total=3,                    # 3 retries maximum
    backoff_factor=1,           # wait 1s, 2s, 4s between retries
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD", "OPTIONS"],
)

adapter = HTTPAdapter(max_retries=retry_strategy)
session = requests.Session()
session.mount("https://", adapter)
session.mount("http://", adapter)

# This request will automatically retry on failure
response = session.get("https://api.example.com/data", timeout=10)

The backoff_factor controls exponential delays between retries. A factor of 1 produces waits of 1, 2, and 4 seconds. The status_forcelist defines which HTTP status codes trigger a retry. Always include 429 (rate limited) and 5xx server errors.

7. HTTPX: Why It Exists and Getting Started

HTTPX was created to provide a requests-compatible API with modern features that requests cannot add without breaking backward compatibility. The key additions are async support, HTTP/2, stricter timeout defaults, and a unified sync/async interface.

pip install httpx

# HTTPX sync API is almost identical to requests
import httpx

response = httpx.get("https://api.example.com/users")
print(response.status_code)
print(response.json())

# POST with JSON
response = httpx.post("https://api.example.com/users", json={
    "name": "Alice",
    "email": "alice@example.com"
})

# HTTPX enforces timeouts by default (5 seconds)
# Override with custom timeout
response = httpx.get("https://api.example.com/slow", timeout=30.0)

# Use a Client for connection pooling (like requests.Session)
with httpx.Client(
    base_url="https://api.example.com",
    headers={"Authorization": "Bearer my-token"},
    timeout=10.0,
) as client:
    users = client.get("/users").json()
    posts = client.get("/posts").json()

Migrating from requests to HTTPX is straightforward. Replace import requests with import httpx, replace requests.Session() with httpx.Client(), and your existing code will work with minimal changes. The biggest difference is that HTTPX has a default 5-second timeout instead of no timeout.

8. HTTPX Async Support

The async client is where HTTPX truly shines. It lets you make concurrent HTTP requests using Python's async/await syntax. See our Python Asyncio Guide for deeper coverage of async programming.

import httpx
import asyncio

async def fetch_user(client: httpx.AsyncClient, user_id: int) -> dict:
    response = await client.get(f"/users/{user_id}")
    response.raise_for_status()
    return response.json()

async def main():
    async with httpx.AsyncClient(
        base_url="https://api.example.com",
        headers={"Authorization": "Bearer my-token"},
        timeout=10.0,
    ) as client:
        # Sequential requests (slow)
        user1 = await fetch_user(client, 1)
        user2 = await fetch_user(client, 2)

        # Concurrent requests (fast!) - all run at the same time
        tasks = [fetch_user(client, uid) for uid in range(1, 11)]
        users = await asyncio.gather(*tasks)
        print(f"Fetched {len(users)} users concurrently")

asyncio.run(main())

If you are building an async web application with FastAPI, HTTPX's AsyncClient integrates naturally into your async endpoints:

from fastapi import FastAPI, Depends
import httpx

app = FastAPI()

async def get_http_client():
    async with httpx.AsyncClient(timeout=10.0) as client:
        yield client

@app.get("/proxy/weather")
async def get_weather(client: httpx.AsyncClient = Depends(get_http_client)):
    resp = await client.get("https://api.weather.com/current?city=London")
    return resp.json()

9. HTTP/2 Support in HTTPX

HTTP/2 enables multiplexing (multiple requests over a single TCP connection), header compression, and server push. HTTPX is the only major Python HTTP client with built-in HTTP/2 support:

pip install httpx[http2]

import httpx

# Enable HTTP/2 on the client
with httpx.Client(http2=True) as client:
    response = client.get("https://http2.github.io/")
    print(response.http_version)  # "HTTP/2"

# Async with HTTP/2
async with httpx.AsyncClient(http2=True) as client:
    response = await client.get("https://api.example.com/data")
    print(response.http_version)  # "HTTP/2" if server supports it

HTTP/2 multiplexing is especially valuable when making many concurrent requests to the same host. Instead of opening multiple TCP connections, all requests share a single connection with interleaved streams. This reduces latency, avoids TCP connection limits, and is more efficient for both client and server.

10. Requests vs HTTPX Comparison

Both libraries handle SSL verification, proxy support, redirects, and cookie management out of the box. The main reasons to choose HTTPX over requests are async support, HTTP/2, and saner timeout defaults.

11. Error Handling and Status Codes

Robust HTTP code must handle network failures, timeouts, and unexpected status codes:

import requests
from requests.exceptions import (
    ConnectionError, Timeout, HTTPError, RequestException
)

def fetch_data(url: str) -> dict | None:
    try:
        response = requests.get(url, timeout=(3.05, 30))
        response.raise_for_status()  # raises HTTPError for 4xx/5xx
        return response.json()
    except Timeout:
        print(f"Request to {url} timed out")
    except ConnectionError:
        print(f"Could not connect to {url}")
    except HTTPError as e:
        print(f"HTTP error {e.response.status_code}: {e.response.text}")
    except RequestException as e:
        print(f"Unexpected request error: {e}")
    return None

# HTTPX equivalent with similar exception hierarchy
import httpx

async def fetch_data_async(url: str) -> dict | None:
    try:
        async with httpx.AsyncClient(timeout=30.0) as client:
            response = await client.get(url)
            response.raise_for_status()
            return response.json()
    except httpx.TimeoutException:
        print(f"Request to {url} timed out")
    except httpx.ConnectError:
        print(f"Could not connect to {url}")
    except httpx.HTTPStatusError as e:
        print(f"HTTP error {e.response.status_code}: {e.response.text}")
    except httpx.RequestError as e:
        print(f"Request error: {e}")
    return None

Always handle specific exceptions before general ones. The raise_for_status() pattern is cleaner than checking status_code manually for every request. For APIs that return error details in the JSON body, catch the HTTPError and parse e.response.json() for structured error messages.

12. Real-World Patterns

API Pagination

Most APIs paginate results. Here is a reusable pattern for cursor-based and page-number pagination:

def fetch_all_pages(session: requests.Session, url: str) -> list:
    """Fetch all pages from a paginated API."""
    results = []
    params = {"per_page": 100, "page": 1}

    while True:
        response = session.get(url, params=params, timeout=15)
        response.raise_for_status()
        data = response.json()

        if not data:
            break

        results.extend(data)
        params["page"] += 1

        # Respect rate limits
        if "X-RateLimit-Remaining" in response.headers:
            remaining = int(response.headers["X-RateLimit-Remaining"])
            if remaining < 5:
                import time
                time.sleep(2)

    return results

Rate Limiting with Backoff

import time

def request_with_rate_limit(session, url, max_retries=5):
    for attempt in range(max_retries):
        response = session.get(url, timeout=15)

        if response.status_code == 429:
            retry_after = int(response.headers.get("Retry-After", 2 ** attempt))
            print(f"Rate limited. Waiting {retry_after}s...")
            time.sleep(retry_after)
            continue

        response.raise_for_status()
        return response

    raise Exception(f"Failed after {max_retries} retries")

OAuth2 Client Credentials Flow

def get_oauth_token(client_id: str, client_secret: str) -> str:
    response = requests.post("https://auth.example.com/oauth/token", data={
        "grant_type": "client_credentials",
        "client_id": client_id,
        "client_secret": client_secret,
        "scope": "read write",
    }, timeout=10)
    response.raise_for_status()
    return response.json()["access_token"]

# Use the token in subsequent requests
token = get_oauth_token("my-client-id", "my-secret")
session = requests.Session()
session.headers["Authorization"] = f"Bearer {token}"
data = session.get("https://api.example.com/protected", timeout=15).json()

For a deeper dive into API design and security, see our API Testing Guide.

13. Testing HTTP Calls

Never let your test suite make real HTTP requests. Use mocking libraries to intercept calls and return predictable responses. Read our Python Testing with Pytest guide for broader testing strategies.

# Testing requests-based code with the "responses" library
import responses
import requests

@responses.activate
def test_get_users():
    responses.add(
        responses.GET,
        "https://api.example.com/users",
        json=[{"id": 1, "name": "Alice"}],
        status=200,
    )

    response = requests.get("https://api.example.com/users")
    assert response.status_code == 200
    assert response.json()[0]["name"] == "Alice"
    assert len(responses.calls) == 1

# Testing httpx-based code with the "respx" library
import httpx
import respx
import pytest

@respx.mock
@pytest.mark.anyio
async def test_get_users_async():
    respx.get("https://api.example.com/users").mock(
        return_value=httpx.Response(200, json=[{"id": 1, "name": "Alice"}])
    )

    async with httpx.AsyncClient() as client:
        response = await client.get("https://api.example.com/users")
        assert response.status_code == 200
        assert response.json()[0]["name"] == "Alice"

# VCR.py - record real HTTP interactions and replay them
import vcr

@vcr.use_cassette("tests/cassettes/github_repo.yaml")
def test_github_repo():
    response = requests.get("https://api.github.com/repos/psf/requests")
    assert response.json()["full_name"] == "psf/requests"
    # First run: records to YAML file. Subsequent runs: replays from file.

14. Performance Tips and Best Practices

These practices apply to both requests and HTTPX:

Always use a Session/Client. Creating a new connection for every request wastes time on DNS resolution, TCP handshakes, and TLS negotiation. A session reuses connections and can be 3-10x faster for multiple requests to the same host.

# Bad: new connection every time
for user_id in range(100):
    requests.get(f"https://api.example.com/users/{user_id}")

# Good: reuse connection via session
with requests.Session() as session:
    for user_id in range(100):
        session.get(f"https://api.example.com/users/{user_id}", timeout=10)

Always set timeouts. The default in requests is no timeout, meaning a hung server will block your program forever. Set both connect and read timeouts.

Use async for concurrent I/O. When you need to call multiple endpoints, async HTTPX with asyncio.gather() can be 10-50x faster than sequential requests:

# Sequential: ~10 seconds for 10 API calls at 1s each
# Concurrent: ~1 second for the same 10 calls
import asyncio, httpx

async def fetch_all():
    async with httpx.AsyncClient(timeout=15.0) as client:
        tasks = [client.get(f"https://api.example.com/items/{i}") for i in range(10)]
        responses = await asyncio.gather(*tasks)
        return [r.json() for r in responses]

Stream large responses. Never load a multi-megabyte response into memory when you only need to write it to disk or process it in chunks.

Respect rate limits. Check X-RateLimit-Remaining and Retry-After headers. Getting blocked by an API is worse than going slower.

Use connection pooling limits. Both libraries allow you to configure the maximum number of connections. For high-throughput applications, tune the pool size:

# HTTPX: configure pool limits
limits = httpx.Limits(max_keepalive_connections=20, max_connections=100)
async with httpx.AsyncClient(limits=limits, timeout=15.0) as client:
    # handle many concurrent requests efficiently
    pass

Conclusion

Python has two excellent HTTP client libraries. requests remains the reliable default for synchronous code and scripts. httpx is the modern choice that adds async support, HTTP/2, and better defaults while maintaining near-perfect API compatibility with requests. For new projects, especially those using async frameworks like FastAPI, HTTPX is the stronger choice.

Whichever library you choose, follow the fundamentals: always set timeouts, use session or client objects for connection pooling, handle errors explicitly, implement retries with backoff for production code, and never let your tests make real HTTP requests. These practices will make your HTTP code reliable, performant, and maintainable.

Related Resources

• Python Asyncio Guide — master async/await for concurrent HTTPX requests
• FastAPI Guide — build high-performance APIs that use HTTPX as an HTTP client
• Python Testing with Pytest — comprehensive testing strategies including HTTP mocking
• API Testing Guide — end-to-end API testing patterns and tools