1. Overview
In my previous blog, I discussed how to enhance PostgreSQL TLS security with OCSP Stapling. In this blog, I will share a simple procedure for setting up a gdb debugging environment to dive into TLS connections and gain a better understanding of the OpenSSL APIs used in PostgreSQL.
2. Build OpenSSL with debugging symbols
First, check out the OpenSSL source code and switch to the release you want to use. In this example, I want to compile the PostgreSQL with the OpenSSL 3.0.2 release, so I switch to release tag openssl-3.0.2
.
$ git clone https://github.com/openssl/openssl.git
$ cd openssl
$ git checkout openssl-3.0.2
Second, compile OpenSSL with gdb enabled using the following options:
$ ./config --prefix=/tmp/ssl --openssldir=/tmp/ssl -d shared -g3 -ggdb -fno-inline -O0 -fno-omit-frame-pointer
$ make -j
$ make install
Compiling OpenSSL with debugging symbols can take a while, and you’d better set the prefix to somewhere so that this particular debug version can be persisted and reused.
3. Build PostgreSQL with debugging OpenSSL libraries
Again, first, check out the PostgreSQL source code and switch to the branch you are interested in. In this blog, I’ll just use the master branch for demo purposes:
$ git clone https://github.com/postgres/postgres.git
Second, compile PostgreSQL with debugging options enabled and specifically specify the OpenSSL source code and debugging libraries you want to link to:
$ ./configure --prefix=/tmp/pgapp --enable-tap-tests --enable-debug CC="gcc -std=gnu99" CFLAGS="-O0 -fno-omit-frame-pointer -I/tmp/openssl/include" LDFLAGS="-Wl,-rpath=/tmp/openssl" --with-openssl
make -j
make install
Here, you must specify LDFLAGS=”-Wl,-rpath=/tmp/openssl” so that PostgreSQL will link to the OpenSSL libraries as you specified; otherwise, it will link to the system ones. To double confirm the OpenSSL libraries linked with PostgreSQL are the ones you wanted, I typically run the following commands:
$ which postgres
/tmp/pgapp/bin/postgres
$ ldd /tmp/pgapp/bin/postgres
linux-vdso.so.1 (0x00007ffd8a3c1000)
libssl.so.3 => /tmp/openssl/libssl.so.3 (0x00007f6878dc6000)
libcrypto.so.3 => /tmp/openssl/libcrypto.so.3 (0x00007f6877c00000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f68781cf000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f68780e8000)
libicui18n.so.70 => /lib/x86_64-linux-gnu/libicui18n.so.70 (0x00007f6877800000)
libicuuc.so.70 => /lib/x86_64-linux-gnu/libicuuc.so.70 (0x00007f6877605000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6877200000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6878e75000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f6876e00000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f68780c8000)
libicudata.so.70 => /lib/x86_64-linux-gnu/libicudata.so.70 (0x00007f6875000000)
Here, check the libraries libssl.so.3 and libcrypto.so.3 to make sure PostgreSQL and psql link to the OpenSSL libraries you expected. In the above example, my PostgreSQL links to the new OpenSSL libraries just compiled and installed with debugging symbols.
4. Enjoy debugging
To debug the OpenSSL APIs during a TLS connection setup or init, you need to have some certificates. You can either follow the PostgreSQL documentation or my blog to set up your own certificates or follow below steps to use the pre-built certificates under the ssl regression test folder. Here are the steps:
$ initdb -D /tmp/pgdata
$ cd postgres/src/test/ssl/ssl
$ cp -pr root_ca.crt server-cn-only+server_ca.crt server-cn-only.key /tmp/pgdata/
$ chmod 600 /tmp/pgdata/server-cn-only.key
Then edit the postgresql.conf file to enable SSL and specify the certificate-related files. For example:
$ vim pgdata/postgresql.conf
ssl = on
ssl_ca_file = 'root_ca.crt'
ssl_cert_file = 'server-cn-only+server_ca.crt'
ssl_key_file = 'server-cn-only.key'
Here, the ssl has been enabled, the root CA certificate pointed to root_ca.crt
, and the entity certificate server-cn-only
with the intermediate CA server_ca
has been used as the PostgreSQL server’s certificate with the corresponding private key file.
Now, we are ready for gdb debugging the OpenSSL APIs.
$ gdb -args postgres -D pgdata
...
(gdb) b be_tls_init
Breakpoint 1 at 0x419bf2: file be-secure-openssl.c, line 95.
(gdb) b SSL_CTX_use_certificate_chain_file
Breakpoint 2 at 0xd9970
Breakpoint 1, be_tls_init (isServerStart=true) at be-secure-openssl.c:95
95 int ssl_ver_min = -1;
(gdb) c
Continuing.
Breakpoint 2, SSL_CTX_use_certificate_chain_file (ctx=0x5555561e9ba0, file=0x5555561ddc08 "server-cn-only+server_ca.crt") at ssl/ssl_rsa.c:534
534 return use_certificate_chain_file(ctx, NULL, file);
(gdb) s
use_certificate_chain_file (ctx=0x5555561e9ba0, ssl=0x0, file=0x5555561ddc08 "server-cn-only+server_ca.crt") at ssl/ssl_rsa.c:419
419 {
(gdb) l
414 * Read a file that contains our certificate in "PEM" format, possibly
415 * followed by a sequence of CA certificates that should be sent to the peer
416 * in the Certificate message.
417 */
418 static int use_certificate_chain_file(SSL_CTX *ctx, SSL *ssl, const char *file)
419 {
420 BIO *in;
421 int ret = 0;
422 X509 *x = NULL;
423 pem_password_cb *passwd_callback;
(gdb) p file
$2 = 0x5555561ddc08 "server-cn-only+server_ca.crt"
(gdb) pt ctx
type = struct ssl_ctx_st {
OSSL_LIB_CTX *libctx;
const SSL_METHOD *method;
struct stack_st_SSL_CIPHER *cipher_list;
struct stack_st_SSL_CIPHER *cipher_list_by_id;
struct stack_st_SSL_CIPHER *tls13_ciphersuites;
struct x509_store_st *cert_store;
In the above example, I enabled two breakpoints: be_tls_init
and SSL_CTX_use_certificate_chain_file
. be_tls_init
is used to ensure that PostgreSQL has been built with debugging symbols enabled, while SSL_CTX_use_certificate_chain_file
is used to dive into OpenSSL. Here, we can print out this complex object ctx
as one example”
With this debugging environment setup, you should be able to debug the psql client side to better understand the entire TLS process.
5. Summary
In this blog post, I have explained a simple process for setting up a gdb debugging environment to dig into OpenSSL, and I hope it can be helpful.

A software developer specialized in C/C++ programming with experience in hardware, firmware, software, database, network, and system architecture. Now, working in HighGo Software Inc, as a senior PostgreSQL architect.
Recent Comments