proxy.py

Table of Contents

• Features
• Install
  • Using PIP
    • Stable version
    • Development version
  • Using Docker
    • Stable version from Docker Hub
    • Development Version from GHCR
    • Build container locally
  • Using HomeBrew
    • Stable version
    • Development version
• Start proxy.py
  • From command line when installed using PIP
    • Run it
    • Understanding logs
    • Enable DEBUG logging
  • From command line using repo source
  • Docker Image
    • Customize Startup Flags
• Plugin Examples
  • HTTP Proxy Plugins
    • ShortLink Plugin
    • Modify Post Data Plugin
    • Mock Api Plugin
    • Redirect To Custom Server Plugin
    • Filter By Upstream Host Plugin
    • Cache Responses Plugin
    • Cache By Response Type
    • Man-In-The-Middle Plugin
    • Proxy Pool Plugin
    • Filter By Client IP Plugin
    • Modify Chunk Response Plugin
    • Modify Request Header Plugin
    • Cloudflare DNS Resolver Plugin
    • Custom DNS Resolver Plugin
    • Custom Network Interface
    • Program Name Plugin
  • HTTP Web Server Plugins
    • Web Server Route
  • Reverse Proxy Plugins
    • Reverse Proxy
  • Plugin Ordering
• End-to-End Encryption
• TLS Interception
  • Insecure TLS Interception
  • TLS Interception With Docker
• GROUT (NGROK Alternative)
  • Grout Usage
  • Grout Authentication
  • Grout Paths
  • Grout Wildcard Domains
  • "Host" Header Based Wildcard Routing
  • "Dynamic" Routing
  • Grout using Docker
  • How Grout works
  • Self-hosted Grout
• Proxy Over SSH Tunnel
  • Proxy Remote Requests Locally
  • Proxy Local Requests Remotely
• Embed proxy.py
  • Blocking Mode
  • Non-blocking Mode
  • Ephemeral Port
  • Loading Plugins
• Unit testing with proxy.py
  • proxy.TestCase
  • Override Startup Flags
  • With unittest.TestCase
• Utilities
  • TCP
    • new_socket_connection
    • socket_connection
  • Http
    • build_http_request
    • build_http_response
  • Public Key Infrastructure
    • API Usage
    • CLI Usage
• Run Dashboard
  • Inspect Traffic
• Chrome DevTools Protocol
• Prometheus Metrics
• Frequently Asked Questions
  • Deploying proxy.py in production
    • What not to do?
    • Via Requirements
    • Via Docker Container
    • Integrate your CI/CD with proxy.py
  • Stable vs Develop
    • Release Schedule
  • Threads vs Threadless
  • Threadless Remote vs Local Execution Mode
  • SyntaxError: invalid syntax
  • Unable to load plugins
  • Unable to connect with proxy.py from remote host
  • Basic auth not working with a browser
  • Docker image not working on MacOS
  • ValueError: filedescriptor out of range in select
  • None:None in access logs
  • OSError when wrapping client for TLS Interception
• Plugin Developer and Contributor Guide
  • High level architecture
  • Everything is a plugin
  • Managing states for your stateless plugins
  • Passing processing context between plugins
  • Internal Documentation
    • Read The Doc
    • pydoc
    • pyreverse
  • Development Guide
    • Setup Local Environment
    • Setup Git Hooks
    • Sending a Pull Request
• Projects Using Proxy.Py
• Benchmarks
• Flags
• Changelog
  • v2.x
  • v1.x
  • v0.x

Features

• A drop-in alternative to ngrok

• Fast & Scalable

  • Scale up by using all available cores on the system

  • Threadless executions using asyncio

  • Made to handle tens-of-thousands connections / sec

    # On Macbook Pro M2 2022
    ❯ python --version
    Python 3.11.8
    ❯ oha --version
    oha 1.4.3
    ❯ ./benchmark/compare.sh
      CONCURRENCY: 100 workers, DURATION: 1m, TIMEOUT: 1sec
      =============================
      Benchmarking Proxy.Py
      Server (pid:75969) running
      Summary:
        Success rate: 100.00%
        Total:        60.0006 secs
        Slowest:      0.2525 secs
        Fastest:      0.0002 secs
        Average:      0.0019 secs
        Requests/sec: 51667.3774
    
        Total data:   56.17 MiB
        Size/request: 19 B
        Size/sec:     958.64 KiB
    
      Response time histogram:
        0.000 [1]       |
        0.025 [3073746] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
        0.051 [10559]   |
        0.076 [4980]    |
        0.101 [2029]    |
        0.126 [5896]    |
        0.152 [2466]    |
        0.177 [116]     |
        0.202 [40]      |
        0.227 [52]      |
        0.253 [87]      |
    
      Response time distribution:
        10.00% in 0.0005 secs
        25.00% in 0.0007 secs
        50.00% in 0.0009 secs
        75.00% in 0.0014 secs
        90.00% in 0.0021 secs
        95.00% in 0.0035 secs
        99.00% in 0.0198 secs
        99.90% in 0.1262 secs
        99.99% in 0.1479 secs
    
      Details (average, fastest, slowest):
        DNS+dialup:   0.0018 secs, 0.0004 secs, 0.0031 secs
        DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0002 secs
    
      Status code distribution:
        [200] 3099972 responses
    
      Error distribution:
        [100] aborted due to deadline
      =============================
    

    Consult Threads vs Threadless and Threadless Remote vs Local Execution Mode to control number of CPU cores utilized.

    See Benchmark for more details and for how to run benchmarks locally.

• Lightweight

  • Uses only ~5-20 MB RAM
    • No memory leaks
    • Start once and forget, no restarts required
  • Compressed containers size is only ~25 MB
  • No external dependency other than standard Python library

• Programmable

  • Customize proxy behavior using Proxy Server Plugins. Example:
    • --plugins proxy.plugin.ProxyPoolPlugin
  • Enable builtin Web Server. Example:
    • --enable-web-server --plugins proxy.plugin.WebServerPlugin
  • Enable builtin Reverse Proxy Server. Example:
    • --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
  • Plugin API is currently in development phase. Expect breaking changes. See Deploying proxy.py in production on how to ensure reliability across code changes.

• Can listen on multiple addresses and ports

  • Use --hostnames flag to provide additional addresses
  • Use --ports flag to provide additional ports
  • Optionally, use --port flag to override default port 8899
  • Capable of serving multiple protocols over the same port

• Real-time Dashboard

  • Optionally, enable proxy.py dashboard.
    • Use --enable-dashboard
    • Then, visit http://localhost:8899/dashboard
  • Inspect, Monitor, Control and Configure proxy.py at runtime
  • Chrome DevTools Protocol support
  • Extend dashboard frontend using typescript based plugins
  • Dashboard is currently in development phase Expect breaking changes.

• Secure

  • Enable end-to-end encryption between clients and proxy.py
  • See End-to-End Encryption

• Private

  • Protection against DNS based traffic blockers
  • Browse with malware and adult content protection enabled
  • See DNS-over-HTTPS

• Man-In-The-Middle

  • Can decrypt TLS traffic between clients and upstream servers
  • See TLS Interception

• Supported http protocols for proxy requests

  • http(s)
    • http1
    • http1.1 with pipeline
  • http2
  • websockets

• Support for HAProxy Protocol

  • See --enable-proxy-protocol flag

• Static file server support

  • See --enable-static-server and --static-server-dir flags

• Optimized for large file uploads and downloads

  • See --client-recvbuf-size, --server-recvbuf-size, --max-sendbuf-size flags

• IPv4 and IPv6 support

  • See --hostname flag

• Unix domain socket support

  • See --unix-socket-path flag

• Basic authentication support

  • See --basic-auth flag

• PAC (Proxy Auto-configuration) support

  • See --pac-file and --pac-file-url-path flags

Install

Consult Deploying proxy.py in production when deploying production grade applications using proxy.py.

Using PIP

Stable Version with PIP

Install from PyPi

❯ pip install --upgrade proxy.py

or from GitHub master branch

❯ pip install git+https://github.com/abhinavsingh/proxy.py.git@master

Development Version with PIP

❯ pip install git+https://github.com/abhinavsingh/proxy.py.git@develop

Using Docker

Multi-platform containers are available via:

• Docker Hub
  • latest tag points to last stable release
  • docker pull abhinavsingh/proxy.py:latest
• GitHub container registry (GHCR)
  • latest tag points to last develop release
  • docker pull ghcr.io/abhinavsingh/proxy.py:latest

Stable version container releases are available for following platforms:

• linux/386
• linux/amd64
• linux/arm/v6
• linux/arm/v7
• linux/arm64/v8
• linux/ppc64le
• linux/s390x

Stable Version from Docker Hub

Run proxy.py latest container:

❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest

Docker daemon will automatically pull the matching platform image. To run specific target platform container on multi-platform supported servers:

❯ docker run -it -p 8899:8899 --rm --platform linux/arm64/v8 abhinavsingh/proxy.py:latest

Development Version from GHCR

Run proxy.py container from cutting edge code in the develop branch:

❯ docker run -it -p 8899:8899 --rm ghcr.io/abhinavsingh/proxy.py:latest

Build Development Version Locally

❯ git clone https://github.com/abhinavsingh/proxy.py.git
❯ cd proxy.py && make container
❯ docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest

docker image is currently broken on macOS due to incompatibility with vpnkit.

Using HomeBrew

Updated formulae for HomeBrew are maintained in develop branch under the helper/homebrew directory.

• stable formulae installs the package from master branch.
• develop formulae installs the package from develop branch.

Stable Version with HomeBrew

❯ brew install https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/helper/homebrew/stable/proxy.rb

Development Version with HomeBrew

❯ brew install https://raw.githubusercontent.com/abhinavsingh/proxy.py/develop/helper/homebrew/develop/proxy.rb

Start proxy.py

From command line when installed using PIP

When proxy.py is installed using pip, an executable named proxy is placed under your $PATH.

Run it

Simply type proxy on command line to start with default configuration.

❯ proxy
...[redacted]... - Loaded plugin proxy.http.proxy.HttpProxyPlugin
...[redacted]... - Started 8 threadless workers
...[redacted]... - Started 8 acceptors
...[redacted]... - Listening on 127.0.0.1:8899

Understanding logs

Things to notice from above logs:

• Loaded plugin

  • proxy.py will load proxy.http.proxy.HttpProxyPlugin by default
  • As name suggests, this core plugin adds http(s) proxy server capabilities to proxy.py instance

• Started N threadless workers

  • By default, proxy.py will start as many worker processes as there are CPU cores on the machine
  • Use --num-workers flag to customize number of worker processes
  • See Threads vs Threadless to understand how to control execution mode

• Started N acceptors

  • By default, proxy.py will start as many acceptor processes as there are CPU cores on the machine
  • Use --num-acceptors flag to customize number of acceptor processes
  • See High Level Architecture to understand relationship between acceptors and workers

• Started server on ::1:8899

  • By default, proxy.py listens on IPv6 ::1, which is equivalent of IPv4 127.0.0.1
  • If you want to access proxy.py from external host, use --hostname :: or --hostname 0.0.0.0 or bind to any other interface available on your machine.
  • See CustomNetworkInterface for how to customize proxy.py public IP seen by upstream servers.

• Port 8899

  • Use --port flag to customize default TCP port.

Enable DEBUG logging

All the logs above are INFO level logs, default --log-level for proxy.py

Lets start proxy.py with DEBUG level logging:

❯ proxy --log-level d
...[redacted]... - Open file descriptor soft limit set to 1024
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
...[redacted]... - Started 8 workers
...[redacted]... - Started server on ::1:8899

You can use single letter to customize log level. Example:

• d = DEBUG
• i = INFO
• w = WARNING
• e = ERROR
• c = CRITICAL

As we can see from the above logs, before starting up:

• proxy.py tried to set open file limit ulimit on the system
• Default value for --open-file-limit used is 1024
• --open-file-limit flag is a no-op on Windows operating systems

See flags for full list of available configuration options.

From command line using repo source

If you are trying to run proxy.py from source code, there is no binary file named proxy in the source code.

To start proxy.py from source code follow these instructions:

• Clone repo

  ❯ git clone https://github.com/abhinavsingh/proxy.py.git
  ❯ cd proxy.py
  

• Create a Python 3 virtual env

  ❯ python3 -m venv venv
  ❯ source venv/bin/activate
  

• Install deps

  ❯ make lib-dep
  

• Generate proxy/common/_scm_version.py

  NOTE: Following step is not necessary for editable installs.

  This file writes SCM detected version to proxy/common/_scm_version.py file.

  ❯ ./write-scm-version.sh
  

• Optionally, run tests

  ❯ make
  

• Run proxy.py

  ❯ python -m proxy
  

See Plugin Developer and Contributor Guide if you plan to work with proxy.py source code.

Docker image

Customize startup flags

By default docker binary is started with IPv4 networking flags:

--hostname 0.0.0.0 --port 8899

You can override flag from command line when starting the docker container. For example, to check proxy.py version within the docker container, run:

❯ docker run -it \
    -p 8899:8899 \
    --rm abhinavsingh/proxy.py:latest \
    -v

Plugin Examples

• See plugin module for full code.
• All the bundled plugin examples also works with https traffic
  • Require additional flags and certificate generation
  • See TLS Interception.
• Plugin examples are also bundled with Docker image.
  • See Customize startup flags to try plugins with Docker image.

HTTP Proxy Plugins

ShortLinkPlugin

Add support for short links in your favorite browsers / applications.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ShortLinkPlugin

Now you can speed up your daily browsing experience by visiting your favorite website using single character domain names :). This works across all browsers.

Following short links are enabled by default:

Short Link	Destination URL
a/	amazon.com
i/	instagram.com
l/	linkedin.com
f/	facebook.com
g/	google.com
t/	twitter.com
w/	web.whatsapp.com
y/	youtube.com
proxy/	localhost:8899

ModifyPostDataPlugin

Modifies POST request body before sending request to upstream server.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ModifyPostDataPlugin

By default plugin replaces POST body content with hard-coded b'{"key": "modified"}' and enforced Content-Type: application/json.

Verify the same using curl -x localhost:8899 -d '{"key": "value"}' http://httpbin.org/post

{
  "args": {},
  "data": "{\"key\": \"modified\"}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "19",
    "Content-Type": "application/json",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "json": {
    "key": "modified"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://httpbin.org/post"
}

Note following from the response above:

1. POST data was modified "data": "{\"key\": \"modified\"}". Original curl command data was {"key": "value"}.
2. Our curl command did not add any Content-Type header, but our plugin did add one "Content-Type": "application/json". Same can also be verified by looking at json field in the output above:

   "json": {
    "key": "modified"
   },
   

3. Our plugin also added a Content-Length header to match length of modified body.

MockRestApiPlugin

Mock responses for your server REST API. Use to test and develop client side applications without need of an actual upstream REST API server.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ProposedRestApiPlugin

Verify mock API response using curl -x localhost:8899 http://api.example.com/v1/users/

{"count": 2, "next": null, "previous": null, "results": [{"email": "[email protected]", "groups": [], "url": "api.example.com/v1/users/1/", "username": "admin"}, {"email": "[email protected]", "groups": [], "url": "api.example.com/v1/users/2/", "username": "admin"}]}

Verify the same by inspecting proxy.py logs:

... [redacted] ... - access_log:1210 - ::1:64792 - GET None:None/v1/users/ - None None - 0 byte

Access log shows None:None as server ip:port. None simply means that the server connection was never made, since response was returned by our plugin.

Now modify ProposedRestApiPlugin to returns REST API mock responses as expected by your clients.

RedirectToCustomServerPlugin

Redirects all incoming http requests to custom web server. By default, it redirects client requests to inbuilt web server, also running on 8899 port.

Start proxy.py and enable inbuilt web server:

❯ proxy \
    --enable-web-server \
    --plugins proxy.plugin.RedirectToCustomServerPlugin

Verify using curl -v -x localhost:8899 http://google.com

... [redacted] ...
< HTTP/1.1 404 NOT FOUND
< Server: proxy.py v1.0.0
< Connection: Close
<
* Closing connection 0

Above 404 response was returned from proxy.py web server.

Verify the same by inspecting the logs for proxy.py. Along with the proxy request log, you must also see a http web server request log.

... [redacted] ... - access_log:1241 - ::1:49525 - GET /
... [redacted] ... - access_log:1157 - ::1:49524 - GET localhost:8899/ - 404 NOT FOUND - 70 bytes

FilterByUpstreamHostPlugin

Drops traffic by inspecting upstream host. By default, plugin drops traffic for facebook.com and www.facebok.com.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.FilterByUpstreamHostPlugin

Verify using curl -v -x localhost:8899 http://facebook.com:

... [redacted] ...
< HTTP/1.1 418 I'm a tea pot
< Proxy-agent: proxy.py v1.0.0
* no chunk, no close, no size. Assume close to signal end
<
* Closing connection 0

Above 418 I'm a tea pot is sent by our plugin.

Verify the same by inspecting logs for proxy.py:

... [redacted] ... - handle_readables:1347 - HttpProtocolException type raised
Traceback (most recent call last):
... [redacted] ...
... [redacted] ... - access_log:1157 - ::1:49911 - GET None:None/ - None None - 0 bytes

CacheResponsesPlugin

Caches Upstream Server Responses.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.CacheResponsesPlugin

You may also use the --cache-requests flag to enable request packet caching for inspection.

Verify using curl -v -x localhost:8899 http://httpbin.org/get:

... [redacted] ...
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/json
< Date: Wed, 25 Sep 2019 02:24:25 GMT
< Referrer-Policy: no-referrer-when-downgrade
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Frame-Options: DENY
< X-XSS-Protection: 1; mode=block
< Content-Length: 202
< Connection: keep-alive
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://httpbin.org/get"
}
* Connection #0 to host localhost left intact

Get path to the cache file from proxy.py logs:

... [redacted] ... - GET httpbin.org:80/get - 200 OK - 556 bytes
... [redacted] ... - Cached response at /var/folders/k9/x93q0_xn1ls9zy76m2mf2k_00000gn/T/httpbin.org-1569378301.407512.txt

Verify contents of the cache file cat /path/to/your/cache/httpbin.org.txt

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 02:24:25 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://httpbin.org/get"
}

CacheByResponseType

CacheResponsesPlugin plugin can also automatically cache responses by content-type. To try this, you must be running under TLS Interception mode and then pass --cache-by-content-type flag. Example:

❯ proxy \
    --plugins proxy.plugin.CacheResponsesPlugin \
    --cache-by-content-type \
    --ca-key-file ca-key.pem \
    --ca-cert-file ca-cert.pem \
    --ca-signing-key ca-signing-key.pem

Make a few requests to the proxy server and you shall see data under ~/.proxy/cache directory.

You should see 2 folders:

• content: Contains parsed jpg, css, js, html, pdf etc by content type
• responses: Contains raw responses as received (of-course decrypted because of interception)

ManInTheMiddlePlugin

Modifies upstream server responses.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ManInTheMiddlePlugin

Verify using curl -v -x localhost:8899 http://google.com:

... [redacted] ...
< HTTP/1.1 200 OK
< Content-Length: 28
<
* Connection #0 to host localhost left intact
Hello from man in the middle

Response body Hello from man in the middle is sent by our plugin.

ProxyPoolPlugin

Forward incoming proxy requests to a set of upstream proxy servers.

Let's start 2 upstream proxies first. To simulate upstream proxies, start proxy.py on port 9000 and 9001

❯ proxy --port 9000

❯ proxy --port 9001

Now, start proxy.py with ProxyPoolPlugin (on default 8899 port), pointing to our upstream proxies at 9000 and 9001 port.

❯ proxy \
    --plugins proxy.plugin.ProxyPoolPlugin \
    --proxy-pool localhost:9000 \
    --proxy-pool localhost:9001

Make a curl request via 8899 proxy:

curl -v -x localhost:8899 http://httpbin.org/get

Verify that 8899 proxy forwards requests to upstream proxies by checking respective logs.

If an upstream proxy require credentials, pass them as arguments. Example:

--proxy-pool user:[email protected]:port

FilterByClientIpPlugin

Reject traffic from specific IP addresses. By default this plugin blocks traffic from 127.0.0.1 and ::1.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.FilterByClientIpPlugin

Send a request using curl -v -x localhost:8899 http://google.com:

... [redacted] ...
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 418 I'm a tea pot
< Connection: close
<
* Closing connection 0

Modify plugin to your taste e.g. Allow specific IP addresses only.

ModifyChunkResponsePlugin

This plugin demonstrate how to modify chunked encoded responses. In able to do so, this plugin uses proxy.py core to parse the chunked encoded response. Then we reconstruct the response using custom hard-coded chunks, ignoring original chunks received from upstream server.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ModifyChunkResponsePlugin

Verify using curl -v -x localhost:8899 http://httpbin.org/stream/5:

... [redacted] ...
modify
chunk
response
plugin
* Connection #0 to host localhost left intact
* Closing connection 0

Modify ModifyChunkResponsePlugin to your taste. Example, instead of sending hard-coded chunks, parse and modify the original JSON chunks received from the upstream server.

ModifyRequestHeaderPlugin

This plugin demonstrate how to modify outgoing HTTPS request headers under TLS interception mode.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ModifyRequestHeaderPlugin \
    ... [TLS interception flags] ...

Verify using curl -x localhost:8899 --cacert ca-cert.pem https://httpbin.org/get:

{
  "args": {},
  "headers": {
    ... [redacted] ...,
    "X-Proxy-Py-Version": "2.4.4rc6.dev15+gf533c711"
  },
  ... [redacted] ...
}

CloudflareDnsResolverPlugin

This plugin uses Cloudflare hosted DNS-over-HTTPS API (json).

DoH mandates a HTTP2 compliant client. Unfortunately proxy.py does not provide that yet, so we use a dependency. Install it:

❯ pip install "httpx[http2]"

Now start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.CloudflareDnsResolverPlugin

By default, CloudflareDnsResolverPlugin runs in security mode and provides malware protection. Use --cloudflare-dns-mode family to also enable adult content protection too.

CustomDnsResolverPlugin

This plugin demonstrate how to use a custom DNS resolution implementation with proxy.py. This example plugin currently uses Python's in-built resolution mechanism. Customize code to your taste. Example, query your custom DNS server, implement DoH or other mechanisms.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.CustomDnsResolverPlugin

CustomNetworkInterface

HttpProxyBasePlugin.resolve_dns callback can also be used to configure network interface which must be used as the source_address for connection to the upstream server.

See this thread for more details.

PS: There is no plugin named, but CustomDnsResolverPlugin can be easily customized according to your needs.

ProgramNamePlugin

Attempts to resolve program (application) name for proxy requests originating from the local machine. If identified, client IP in the access logs is replaced with program name.

Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.ProgramNamePlugin

Make a request using curl:

❯ curl -v -x localhost:8899 https://httpbin.org/get

You must see log lines like this:

... [redacted] ... - [I] server.access_log:419 - curl:58096 - CONNECT httpbin.org:443 - 6010 bytes - 1824.62ms

Notice curl in-place of ::1 or 127.0.0.1 as client IP.

If ProgramNamePlugin does not work reliably on your operating system, kindly contribute by sending a pull request and/or open an issue. Thank you!!!

HTTP Web Server Plugins

Web Server Route

Demonstrates inbuilt web server routing using plugin.

Start proxy.py as:

❯ proxy --enable-web-server \
    --plugins proxy.plugin.WebServerPlugin

Verify using curl -v localhost:8899/http-route-example, should return:

HTTP route response

Reverse Proxy Plugins

Extends in-built Web Server to add Reverse Proxy capabilities.

Reverse Proxy

Start proxy.py as:

❯ proxy --enable-reverse-proxy \
    --plugins proxy.plugin.ReverseProxyPlugin

With default configuration, ReverseProxyPlugin plugin is equivalent to following Nginx config:

location /get {
    proxy_pass http://httpbin.org/get;
}

Verify using curl -v localhost:8899/get:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "localhost",
    "User-Agent": "curl/7.64.1"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://localhost/get"
}

Rewrite Host Header

With above example, you may sometimes see:

>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

This is happenening because our default reverse proxy plugin ReverseProxyPlugin is configured with a http and a https upstream server. And, by default ReverseProxyPlugin preserves the original host header. While this works with https upstreams, this doesn't work reliably with http upstreams. To work around this problem use the --rewrite-host-header flags.

Example:

❯ proxy --enable-reverse-proxy \
    --plugins proxy.plugin.ReverseProxyPlugin \
    --rewrite-host-header

This will ensure that Host header field is set as httpbin.org and works with both http and https upstreams.

NOTE: Whether to use --rewrite-host-header or not depends upon your use-case.

Plugin Ordering

When using multiple plugins, depending upon plugin functionality, it might be worth considering the order in which plugins are passed on the command line.

Plugins are called in the same order as they are passed. Example, say we are using both FilterByUpstreamHostPlugin and RedirectToCustomServerPlugin. Idea is to drop all incoming http requests for facebook.com and www.facebook.com and redirect other http requests to our inbuilt web server.

Hence, in this scenario it is important to use FilterByUpstreamHostPlugin before RedirectToCustomServerPlugin. If we enable RedirectToCustomServerPlugin before FilterByUpstreamHostPlugin, facebook requests will also get redirected to inbuilt web server, instead of being dropped.

End-to-End Encryption

By default, proxy.py uses http protocol for communication with clients e.g. curl, browser. For enabling end-to-end encrypting using tls / https first generate certificates. Checkout the repository and run:

make https-certificates

Start proxy.py as:

❯ proxy \
    --cert-file https-cert.pem \
    --key-file https-key.pem

Verify using curl -x https://localhost:8899 --proxy-cacert https-cert.pem https://httpbin.org/get:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://httpbin.org/get"
}

If you want to avoid passing --proxy-cacert flag, also consider signing generated SSL certificates. Example:

First, generate CA certificates:

make ca-certificates

Then, sign SSL certificate:

make sign-https-certificates

Now restart the server with --cert-file https-signed-cert.pem flag. Note that you must also trust generated ca-cert.pem in your system keychain.

TLS Interception

By default, proxy.py will not decrypt https traffic between client and server. To enable TLS interception first generate root CA certificates:

❯ make ca-certificates

Lets also enable CacheResponsePlugin so that we can verify decrypted response from the server. Start proxy.py as:

❯ proxy \
    --plugins proxy.plugin.CacheResponsesPlugin \
    --ca-key-file ca-key.pem \
    --ca-cert-file ca-cert.pem \
    --ca-signing-key-file ca-signing-key.pem

Also provide explicit CA bundle path needed for validation of peer certificates. See --ca-file flag.

Verify TLS interception using curl

❯ curl -v -x localhost:8899 --cacert ca-cert.pem https://httpbin.org/get

*  issuer: C=US; ST=CA; L=SanFrancisco; O=proxy.py; OU=CA; CN=Proxy PY CA; [email protected]
*  SSL certificate verify ok.
> GET /get HTTP/1.1
... [redacted] ...
< Connection: keep-alive
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://httpbin.org/get"
}

The issuer line confirms that response was intercepted.

Also verify the contents of cached response file. Get path to the cache file from proxy.py logs.

❯ cat /path/to/your/tmp/directory/httpbin.org-1569452863.924174.txt

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Wed, 25 Sep 2019 23:07:05 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Length: 202
Connection: keep-alive

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "1.2.3.4, 5.6.7.8",
  "url": "https://httpbin.org/get"
}

Viola!!! If you remove CA flags, encrypted data will be found in the cached file instead of plain text.

Now use CA flags with other plugin examples to see them work with https traffic.

Insecure TLS Interception

To intercept TLS traffic from a server using a self-signed certificate add the --insecure-tls-interception flag to disable mandatory TLS certificate validation.

NOTE: This flag disables certificate check for all servers.

TLS Interception With Docker

Important notes about TLS Interception with Docker container:

• Since v2.2.0, proxy.py docker container also ships with openssl. This allows proxy.py to generate certificates on the fly for TLS Interception.

• For security reasons, proxy.py docker container does not ship with CA certificates.

Here is how to start a proxy.py docker container with TLS Interception:

1. Generate CA certificates on host computer

   ❯ make ca-certificates
   

2. Copy all generated certificates into a separate directory. We'll later mount this directory into our docker container

   ❯ mkdir /tmp/ca-certificates
   ❯ cp ca-cert.pem ca-key.pem ca-signing-key.pem /tmp/ca-certificates
   

3. Start docker container

   ❯ docker run -it --rm \
       -v /tmp/ca-certificates:/tmp/ca-certificates \
       -p 8899:8899 \
       abhinavsingh/proxy.py:latest \
       --hostname 0.0.0.0 \
       --plugins proxy.plugin.CacheResponsesPlugin \
       --ca-key-file /tmp/ca-certificates/ca-key.pem \
       --ca-cert-file /tmp/ca-certificates/ca-cert.pem \
       --ca-signing-key /tmp/ca-certificates/ca-signing-key.pem
   

   • -v /tmp/ca-certificates:/tmp/ca-certificates flag mounts our CA certificate directory in container environment
   • --plugins proxy.plugin.CacheResponsesPlugin enables CacheResponsesPlugin so that we can inspect intercepted traffic
   • --ca-* flags enable TLS Interception.

4. From another terminal, try TLS Interception using curl. You can omit --cacert flag if CA certificate is already trusted by the system.

   ❯ curl -v \
       --cacert ca-cert.pem \
       -x 127.0.0.1:8899 \
       https://httpbin.org/get
   

5. Verify issuer field from response headers.

   * Server certificate:
   *  subject: CN=httpbin.org; C=NA; ST=Unavailable; L=Unavailable; O=Unavailable; OU=Unavailable
   *  start date: Jun 17 09:26:57 2020 GMT
   *  expire date: Jun 17 09:26:57 2022 GMT
   *  subjectAltName: host "httpbin.org" matched cert's "httpbin.org"
   *  issuer: CN=example.com
   *  SSL certificate verify ok.
   

6. Back on docker terminal, copy response dump path logs.

   ...[redacted]... [I] access_log:338 - 172.17.0.1:56498 - CONNECT httpbin.org:443 - 1031 bytes - 1216.70 ms
   ...[redacted]... [I] close:49 - Cached response at /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txt
   

7. In another terminal, cat the response dump:

   ❯ docker exec -it $(docker ps | grep proxy.py | awk '{ print $1 }') cat /tmp/httpbin.org-ae1a927d064e4ab386ea319eb38fe251.txt
   HTTP/1.1 200 OK
   ...[redacted]...
   {
     ...[redacted]...,
     "url": "http://httpbin.org/get"
   }
   

GROUT (NGROK Alternative)

1. grout is a drop-in alternative for ngrok and frp
2. grout comes packaged within proxy.py

Grout Usage

❯ grout
NAME:
  grout - securely tunnel local files, folders and services to public URLs

USAGE:
  grout route [name]

DESCRIPTION:
  grout exposes local networked services behinds NATs and firewalls to the
  public internet over a secure tunnel. Share local folders, directories and websites,
  build/test webhook consumers and self-host personal services to public URLs.

EXAMPLES:
  Share Files and Folders:
    grout C:\path\to\folder                          # Share a folder on your system
    grout /path/to/folder                            # Share a folder on your system
    grout /path/to/folder --basic-auth user:pass     # Add authentication for shared folder
    grout /path/to/photo.jpg                         # Share a specific file on your system

  Expose HTTP, HTTPS and Websockets:
    grout http://localhost:9090                      # Expose HTTP service running on port 9090
    grout https://localhost:8080                     # Expose HTTPS service running on port 8080
    grout https://localhost:8080 --path /worker/     # Expose only certain paths of HTTPS service on port 8080
    grout https://localhost:8080 --basic-auth u:p    # Add authentication for exposed HTTPS service on port 8080

  Expose TCP Services:
    grout tcp://:6379                                # Expose Redis service running locally on port 6379
    grout tcp://:22                                  # Expose SSH service running locally on port 22

  Custom URLs:
    grout https://localhost:8080 abhinavsingh        # Custom URL for HTTPS service running on port 8080
    grout tcp://:22 abhinavsingh                     # Custom URL for SSH service running locally on port 22

  Custom Domains:
    grout tcp://:5432 abhinavsingh.domain.tld        # Custom URL for Postgres service running locally on port 5432

  Self-hosted solutions:
    grout tcp://:5432 abhinavsingh.my.server         # Custom URL for Postgres service running locally on port 5432

  (*) Wildcard Domains:
    grout https://host:443 do.main --wildcard        # Receive traffic on provided domain and all it's subdomains

  (*) Host based routing for Wildcard Domains:
    grout ... --tunnel-route-url host=https://h:p    # When using wildcards, optionally route traffic by incoming host header

SUPPORT:
  Write to us at [email protected]

  Privacy policy and Terms & conditions
  https://jaxl.com/privacy/

  Created by Jaxl™
  https://jaxl.io

Grout Authentication

Grout supports authentication to protect your files, folders and services from unauthorized access. Use --basic-auth flag to enforce authentication. Example:

grout /path/to/folder --basic-auth user:pass
grout https://localhost:8080 --basic-auth u:p

Grout Paths

By default, Grout allows access to all paths on the services. Use --path flag to restrict access to only certain paths on your web service. Example:

grout https://localhost:8080 --path /worker/
grout https://localhost:8080 --path /webhook/ --path /callback/

Grout Wildcard Domains

By default, Grout client serves incoming traffic on a dedicated subdomain. However, some services (e.g. Kubernetes) may want to serve traffic on adhoc subdomains. Starting a dedicated Grout client for every adhoc subdomain may not be a practical solution.

For such scenarios, Grout supports wildcard domains. Here is how to configure your own wildcard domain for use with Grout clients.

1. Choose a domain e.g. custom.example.com
2. Your service wants to serve traffic for custom.example.com and *.custom.example.com
3. If you plan on using https://, you need to setup a load balancer:
   • Setup a HTTPS load balancer (LB)
   • Configure LB with certificate generated for custom.example.com and *.custom.example.com
   • Point traffic to Grout service public IP addresses
4. Contact Grout team at [email protected] to whitelist custom.example.com. Grout team will make sure you really own the domain and you have configured a valid SSL certificate as described above

Start Grout with --wildcard flag. Example:

grout https://localhost:8080 custom.example.com --wildcard
2024-08-05 18:24:59,294 - grout - Logged in as [email protected]
2024-08-05 18:25:03,159 - setup - Grouting https://*.custom.domain.com

Grout Wildcard Domain Routing Based Upon "Host" Header

Only available with --wildcard

Along with the default route you can also provide additional routes which takes precedence when the host field matches. Example:

grout https://localhost:8080 custom.example.com \
  --wildcard \
  --tunnel-route-url stream.example.com=http://localhost:7001

You can provide multiple custom routes by repeating this flag.

Grout Client Plugin

GroutClientBasePlugin allows you to dynamically route traffic to different upstreams. Below is a simple implementation with some description about how to use it.

class GroutClientPlugin(GroutClientBasePlugin):

    def resolve_route(
        self,
        route: str,
        request: HttpParser,
        origin: HostPort,
        server: HostPort,
    ) -> Tuple[Optional[str], HttpParser]:
        print(request, origin, server, '->', route)
        print(request.header(b'host'), request.path)
        #
        # Here, we send traffic to localhost:7001 irrespective
        # of the original "route" value provided to the grout
        # client OR any custom host:upstream mapping provided
        # through the --tunnel-route-url flags (when using
        # --wildcard).
        #
        # Optionally, you can also strip path before
        # sending traffic to upstrem, like:
        # request.path = b"/"
        #
        # To drop the request, simply return None for route
        # return None, request
        #
        return 'http://localhost:7001', request

See grout_client.py for more information. To try this out, start by passing --plugin proxy.plugin.grout_client.GroutClientPlugin when starting the grout client.

Grout using Docker

❯ docker run --rm -it \
  --entrypoint grout \
  -v ~/.proxy:/root/.proxy \
  abhinavsingh/proxy.py:latest \
  http://host.docker.internal:29876

Above:

• We changed --entrypoint to grout
• We replaced localhost with host.docker.internal, so that grout can route traffic to port 29876 running on the host machine
• (Optional) Mount host machine ~/.proxy folder, so that grout credentials can persist across container restarts

How Grout works

• grout infrastructure has 2 components: client and server
• grout client has 2 components: a thin and a thick client
• grout thin client is part of open source proxy.py (BSD 3-Clause License)
• grout thick client and servers are hosted at jaxl.io and a copyright of Jaxl Innovations Private Limited
• grout server has 3 components: a registry server, a reverse proxy server and a tunnel server

Self-Hosted grout

• grout thick client and servers can also be hosted on your GCP, AWS, Cloud infrastructures
• With a self-hosted version, your traffic flows through the network you control and trust
• grout developers at jaxl.io provides GCP, AWS, Docker images for self-hosted solutions
• Please drop an email at [email protected] to get started.

Proxy Over SSH Tunnel

This is a WIP and may not work as documented

Requires paramiko to work. Install dependencies using pip install "proxy.py[tunnel]"

Proxy Remote Requests Locally

                        |
+------------+          |            +----------+
|   LOCAL    |          |            |  REMOTE  |
|   HOST     | <== SSH ==== :8900 == |  PROXY   |
+------------+          |            +----------+
:8899 proxy.py          |
                        |
                     FIREWALL
                  (allow tcp/22)

What

Proxy HTTP(s) requests made on a remote proxy server through proxy.py server running on localhost.

How

• Requested remote port is forwarded over the SSH connection.
• proxy.py running on the localhost handles and responds to remote proxy requests.

Requirements

1. localhost MUST have SSH access to the remote server
2. remote server MUST be configured to proxy HTTP(s) requests through the forwarded port number e.g. :8900.
   • remote and localhost ports CAN be same e.g. :8899.
   • :8900 is chosen in ascii art for differentiation purposes.

Try it

Start proxy.py as:

❯ # On localhost
❯ proxy --enable-ssh-tunnel \
    --tunnel-username username \
    --tunnel-hostname ip.address.or.domain.name \
    --tunnel-port 22 \
    --tunnel-remote-port 8899 \
    --tunnel-ssh-key /path/to/ssh/private.key \
    --tunnel-ssh-key-passphrase XXXXX
...[redacted]... [I] listener.setup:97 - Listening on 127.0.0.1:8899
...[redacted]... [I] pool.setup:106 - Started 16 acceptors in threadless (local) mode
...[redacted]... [I] transport._log:1873 - Connected (version 2.0, client OpenSSH_7.6p1)
...[redacted]... [I] transport._log:1873 - Authentication (publickey) successful!
...[redacted]... [I] listener.setup:116 - SSH connection established to ip.address.or.domain.name:22...
...[redacted]... [I] listener.start_port_forward:91 - :8899 forwarding successful...

Make a HTTP proxy request on remote server and verify that response contains public IP address of localhost as origin:

❯ # On remote
❯ curl -x 127.0.0.1:8899 http://httpbin.org/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  "origin": "x.x.x.x, y.y.y.y",
  "url": "https://httpbin.org/get"
}

Also, verify that proxy.py logs on localhost contains remote IP as client IP.

access_log:328 - remote:52067 - GET httpbin.org:80

Proxy Local Requests Remotely

                        |
+------------+          |     +----------+
|   LOCAL    |          |     |  REMOTE  |
|   HOST     | === SSH =====> |  SERVER  |
+------------+          |     +----------+
                        |     :8899 proxy.py
                        |
                    FIREWALL
                 (allow tcp/22)

Not planned.

If you have a valid use case, kindly open an issue. You are always welcome to send contributions via pull-requests to add this functionality :)

To proxy local requests remotely, make use of Proxy Pool Plugin.

Embed proxy.py

Blocking Mode

Start proxy.py in embedded mode with default configuration by using proxy.main method. Example:

import proxy

if __name__ == '__main__':
  proxy.main()

Customize startup flags by passing them as kwargs:

import ipaddress
import proxy

if __name__ == '__main__':
  proxy.main(
    hostname=ipaddress.IPv6Address('::1'),
    port=8899
  )

Note that:

1. main is equivalent to starting proxy.py from command line.
2. main does not accept any args (only kwargs).
3. main will automatically consume any available sys.argv as args.
4. main will block until proxy.py shuts down.

Non-blocking Mode

Start proxy.py in non-blocking embedded mode with default configuration by using Proxy context manager: Example:

import proxy

if __name__ == '__main__':
  with proxy.Proxy() as p:
    # Uncomment the line below and
    # implement your app your logic here
    proxy.sleep_loop()

Note that:

1. Proxy is similar to main, except Proxy will not block.
2. Internally, Proxy is a context manager which will start proxy.py when called and will shut it down once the scope ends.
3. Unlike main, startup flags with Proxy can also be customized by using args and kwargs. e.g. Proxy(['--port', '8899']) or by using passing flags as kwargs e.g. Proxy(port=8899).
4. Unlike main, Proxy will not inspect sys.argv.

Ephemeral Port

Use --port=0 to bind proxy.py on a random port allocated by the kernel.

In embedded mode, you can access this port. Example:

import proxy

if __name__ == '__main__':
  with proxy.Proxy(port=0) as p:
    print(p.flags.port)
    proxy.sleep_loop()

flags.port will give you access to the random port allocated by the kernel.

Loading Plugins

Users can use --plugins flag multiple times to load multiple plugins. See Unable to load plugins if you are running into issues.

When using in embedded mode, you have a few more options. Example:

1. Provide a fully-qualified name of the plugin class as bytes to the proxy.main method or proxy.Proxy context manager.
2. Provide type instance of the plugin class. This is especially useful if you plan to define plugins at runtime.

Example, load a single plugin using --plugins flag:

import proxy

if __name__ == '__main__':
  proxy.main(plugins=['proxy.plugin.CacheResponsesPlugin'])

For simplicity, you can also pass the list of plugins as a keyword argument to proxy.main or the Proxy constructor.

Example:

import proxy
from proxy.plugin import FilterByUpstreamHostPlugin

if __name__ == '__main__':
  proxy.main(plugins=[
    b'proxy.plugin.CacheResponsesPlugin',
    FilterByUpstreamHostPlugin,
  ])

Unit testing with proxy.py

proxy.TestCase

To setup and tear down proxy.py for your Python unittest classes, simply use proxy.TestCase instead of unittest.TestCase. Example:

import proxy

class TestProxyPyEmbedded(proxy.TestCase):

    def test_my_application_with_proxy(self) -> None:
        self.assertTrue(True)

Note that:

1. proxy.TestCase overrides unittest.TestCase.run() method to setup and tear down proxy.py.
2. proxy.py server will listen on a random available port on the system. This random port is available as self.PROXY.flags.port within your test cases.
3. Only a single acceptor and worker is started by default (--num-workers 1 --num-acceptors 1) for faster setup and tear down.
4. Most importantly, proxy.TestCase also ensures proxy.py server is up and running before proceeding with execution of tests. By default, proxy.TestCase will wait for 10 seconds for proxy.py server to start, upon failure a TimeoutError exception will be raised.

Override startup flags

To override default startup flags, define a PROXY_PY_STARTUP_FLAGS variable in your test class. Example:

class TestProxyPyEmbedded(TestCase):

    PROXY_PY_STARTUP_FLAGS = [
        '--num-workers', '2',
        '--num-acceptors', '1',
        '--enable-web-server',
    ]

    def test_my_application_with_proxy(self) -> None:
        self.assertTrue(True)

See test_embed.py for full working example.

With unittest.TestCase

If for some reasons you are unable to directly use proxy.TestCase, then simply override unittest.TestCase.run yourself to setup and tear down proxy.py. Example:

import unittest
import proxy


class TestProxyPyEmbedded(unittest.TestCase):

    def test_my_application_with_proxy(self) -> None:
        self.assertTrue(True)

    def run(self, result: Optional[unittest.TestResult] = None) -> Any:
        with proxy.start([
                '--num-workers', '1',
                '--num-acceptors', '1',
                '--port', '... random port ...']):