Tutorial #6: Fix SSL Error in Python requests when proxying through Burp Suite

tl;dr — Two ways to fix: either disable SSL checking completely with verify=False (the dirty approach) or use verify=<path to cert> to enforce certificate check against a chosen certificate from a desired local location.

The Background

In Python3, we can make use of the requests library to invoke HTTP requests. It is a pretty neat library and easy to pick up for writing scripts for any penetration testing engagement. When doing so I wanted to proxy all my python scripts traffic through Burp Suite so that the traffic (the requests and responses) can be examined more closely and troubleshoot if necessary.

The main issue when proxying the python through Burp Proxy is the certification validation error as Burp Suite is a form of MitM (man-in-the-middle) proxy which uses its own self-signed certificate. The python script will find the Burp certificate as invalid, and it would refuse to establish the connection.

After searching through much on the internet including various Stack Overflow posts, the following are my attempts to fix such SSL validation errors to make the set-up work. The following diagram is the set up.

The python script will use Burp Suite as its proxy when communicating with a target website.

The following is a sample code of python script making a simple GET to http://www.google.com making use of the python requests library. This will not work as python will be unable to validate the proxy certificate. The proxies parameter in requests allows the HTTP request traffic to flow through a proxy. (e.g. 127.0.0.1:8080 is where my Burp proxy is listening)

import requests

proxies = {"http":"127.0.0.1:8080","https":"127.0.0.1:8080"}
url = 'https://www.google.com'

r = requests.get(url,proxies=proxies)
print("Response Status Code: " + str(r.status_code))
print("Response Length:" + str(len(r.content)))

When running the above-mentioned code, the requests refuses to establish connection and show errors like this

Approach 1: The Dirty Fix with verify=False

If I am looking for a quick and dirty fix, I can just put an additional parameter of verify with the value False when triggering a get request as shown in the code below. This will force python to ignore SSL check at all and just establish the HTTP connection.

This is it; it is fixed! But if you still want to fix this in a better way, keep reading the next the approach 2 below.

import requests

proxies = {"http":"127.0.0.1:8080","https":"127.0.0.1:8080"}
url = 'https://www.google.com'

r = requests.get(url,proxies=proxies,verify=False)
print("Response Status Code: " + str(r.status_code))
print("Response Length:" + str(len(r.content)))

Approach 2: Performing Cert Validation Against a Local Certificate

First, I need a copy of Burp public certificate for the python library to validate against. Burp has its own built-in chromium browser which is already configured to proxy through 127.0.0.1:8080. Either using that browser or any browser of my own that is already configured to connect to burp proxy, I can go to http://burp and download the CA cert under the CA Certificate menu.

Download CA cert under the CA Certificate by visiting http://burp in your Burp default browser or your local browser that is configure to the local Burp Suite proxy

By doing so, you will get hold of a Burp CA certificate typically named cacert.der file. To be able to use this in the python script, the format needs to be converted to .pem format. By using openssl (which is typically installed by default in Linux boxes but I use it through wsl), convert the .der to .pem as shown below.

Convert .der to .pem with openssl e.g. openssl x509 -inform der -in cacert.der -out burpca.pem

Then, use parameter verify again and put the file path value to the newly converted .pem as seen in the code below.

import requests

proxies = {"http":"127.0.0.1:8080","https":"127.0.0.1:8080"}
url = 'https://www.google.com'

r = requests.get(url,proxies=proxies,verify='C:\\temp\\burpca.pem')
print("Response Status Code: " + str(r.status_code))
print("Response Length:" + str(len(r.content)))

There will be no more SSL error when running the script

So that was the second way that I tried out to fix the SSL issue. You have been reading my post (written by a real human) rather than information by some ChatGPT or CoPilot! And I thank you for that :D.

Have a good day.