How to test a Python app uses an HTTP proxy to connect to AWS

Some organizations require that all outbound application traffic run through an HTTP proxy so that traffic can be inspected and limited to certain allowlisted domains. But how do you test your application to verify your client libraries have all been configured to use the proxy correctly? In this post, you’ll learn how to configure a local HTTP proxy test environment using Docker, Squid, the AWS CLI, and your application.

Prerequisites

The only prerequisite is to have Docker installed and running on a host with access to the Internet.

Create the networks

Let’s create two Docker networks, a public-net and a private-net.

First, create the private-net which will not have outbound access to the host’s network and internet:

docker network create private-net --driver=bridge --internal

The --internal flag restricts traffic on this network to only this network. Containers attached to this network will only be able to talk to each other, not the Internet.

Second, create the public-net with:

docker network create public-net --driver=bridge

Now let’s attach containers to these networks.

Run an HTTP Proxy

Run the Squid HTTP proxy in a container and attach it to both networks with the following command:

docker container run --rm -d --name http-proxy \
  --network private-net \
  --network public-net \
  -e TZ=UTC -p 3128:3128 ubuntu/squid:5.2-22.04_beta

That command runs the Squid packaged by Ubuntu (ubuntu/squid) in a container named http-proxy. If you’re following along, the http-proxy container is now attached to both private-net and public-net. This will allow http-proxy to act as a gateway for applications on private-net.

You can verify the http-proxy is attached to both networks with the docker container inspect http-proxy command. Inspect just the network attachments with docker container inspect http-proxy | jq '.[0].NetworkSettings.Networks':

{
  "private-net": {
    "IPAMConfig": null,
    "Links": null,
    "Aliases": null,
    "MacAddress": "02:42:ac:13:00:03",
    "DriverOpts": null,
    "NetworkID": "9a8c62d162c0193d1d513a5530005cff0199ec6ded9b8762d365b8549a1dc9a1",
    "EndpointID": "06a4349060f8d90c1c1f833ae3196e9b09152770cfcf5aa2c219102a523dc7a8",
    "Gateway": "",
    "IPAddress": "172.19.0.3",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "DNSNames": [
      "http-proxy",
      "0bf03a850e00"
    ]
  },
  "public-net": {
    "IPAMConfig": {},
    "Links": null,
    "Aliases": [],
    "MacAddress": "02:42:ac:14:00:02",
    "DriverOpts": {},
    "NetworkID": "f824b5799f53018f5a2015b04b0e32c74c39788f97430d001faeeaff090691c0",
    "EndpointID": "26d4fd1bd11cecb3dcede97d9da1675067e17c0a94f6730cf749b2755db6f43e",
    "Gateway": "172.20.0.1",
    "IPAddress": "172.20.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "DNSNames": [
      "http-proxy",
      "0bf03a850e00"
    ]
  }
}

Notice that:

1. http-proxy is attached to private-net with IP address 172.19.0.3 and that Gateway is empty ("Gateway": "").
2. http-proxy is attached to public-net with IP address 172.20.0.2 and that public-net has a Gateway at 172.20.0.1.

(Note: Docker conveniently creates networks with non-overlapping network ranges by default.)

Run the AWS CLI

Now let’s attach a container to the private network and try to interact with AWS.

Populate your shell with the usual environment variables to access AWS. Don’t forget to specify AWS_REGION and AWS_DEFAULT_REGION; the SDK needs these to resolve API endpoints. If you use aws-vault, then aws-vault exec <profile-name> -- bash is perfect.

Now run the official amazon/aws-cli container image:

docker container run --rm -it --name aws-cli \
  --network private-net \
  -e PS1='private-net.aws-cli# ' \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
  -e AWS_REGION=$AWS_REGION \
  -e AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION \
  --entrypoint /bin/bash \
  amazon/aws-cli

This command runs the AWS CLI container interactively, attaches it to only the private-net network, and drops you into a bash shell.

You should have a prompt like:

private-net.aws-cli#

Now let’s use the CLI and see what access we have. Run:

aws sts get-caller-identity

You should see a failure like:

private-net.aws-cli# aws sts get-caller-identity

Could not connect to the endpoint URL: "https://sts.us-east-1.amazonaws.com/"

Excellent! The aws-cli container network is isolated and does not have access to AWS. Yet.

(You can also inspect the aws-cli‘s network config with docker container inspect and compare it to http-proxy‘s)

Now export the https_proxy variable to point to our squid instance running on port 3128:

export https_proxy=http://http-proxy:3128

Note: There is no official standard for casing or even spelling of the HTTP proxy variables; it’s implementation dependent. http_proxy (no ‘s’) is probably most widely supported, however it doesn’t work for the AWS CLI!

Rerun the get-caller-identity command. You should get output similar to:

private-net.aws-cli# aws sts get-caller-identity
{
    "UserId": "AROASBB3_EXAMPLE:console",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/networking-pro"
}

Woohoo! 🙌

Your isolated HTTP proxy setup is working!

Now you can switch to doing what you probably wanted to do all along. Test your application.

Test your application

Every Python application is a little different, but fortunately if you’re using the boto3 and botocore SDKs, the previous AWS CLI exercise showed that the underlying networking setup is correct. Now we need to run a container with your app in it. Here are the general steps to do that.

First, launch a new terminal and populate your shell with AWS credential and config variables. Change to your project’s source directory if you’re not already there.

Second, start a container with your application’s source code mounted at the working directory appropriate for that image. If you are using a container image derived from the official Python Lambda images, then this is a good start:

docker container run \
  --network private-net \
  -e PS1='private-net.dev-shell# ' \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
  -e AWS_REGION=$AWS_REGION \
  -e AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION \
  -v $(PWD):/var/task \
  --workdir /var/task \
  --entrypoint /bin/bash \
  --rm -it public.ecr.aws/docker/library/python:3.11-slim

That command runs a container using the AWS Lambda python:3.11-slim image:

• attaches it to private-net
• mounts the current directory into /var/task (the normal directory for Lambda function code)
• gives you a shell in the working directory: /var/task
  

Now you can set up your application’s environment then perform your testing.

First, configure https_proxy:

export https_proxy=http://http-proxy:3128

Then install dependencies, e.g.:

pip3 install --proxy $https_proxy -r requirements.txt

Run your tests, e.g.:

pytest tests/functional/test_e2e.py

Now you should see your app & tests making requests through the proxy or emit some sort of ‘failed to connect to host xyz’ error.

We hope this helps you sort out issues with HTTP proxy configuration and support. You can also use AWS Security’s guide for configuring an outbound VPC proxy using Squid to verify your app works as expected in AWS.

Happy testing!

Fun Facts

But wait, there’s more! Here are some fun facts about HTTP proxy and Python:

Fun Fact #1: boto3 and botocore do not currently support NO_PROXY, but they did before March 2022 when boto3 switched from requests to urllib3!

• boto3 Issue 3179 NO_PROXY not considered in boto lambda local invocation
• boto3 Issue 3193 Remove NO_PROXY from configuration docs
• botocore Config class does not support configuration of NO_PROXY

Fun Fact #2: NO_PROXY support is inconsistent

Many libraries and tools allow you to bypass sending requests to the proxy for a configured list of IPs, hosts, or domains. But this is very implementation-dependent. Here’s what we know about NO_PROXY support in Python