How to run TLS regression test in PostgreSQL

Enterprise PostgreSQL Solutions

Comments are off

How to run TLS regression test in PostgreSQL

1. Overview

In my previous blogs, I discussed Setting up a debugging environment in PostgreSQL to better understand OpenSSL APIs, Enhance PostgreSQL TLS Security with OCSP Stapling, and How to setup TLS connection for PostgreSQL. In this blog, I will share a simple procedure about How to run SSL/TLS regression tests in PostgreSQL.

2. Postgres regression tests

Whenever we want to add a new feature or make some fix, we should run the tests provided by PostgreSQL to make sure nothing is broken. If the new feature or the fix does not have corresponding tests, we should consider adding some to ensure it will not break in the future when other changes are made. PostgreSQL provides a comprehensive document) explaining all available tests. For example, if you want to run a quick regression test to check if any “core” feature may be broken, you can run make check for a temporary installation within the build tree or make installcheck against a running PostgreSQL server. make check and make installcheck will only test the built-in functionality of the PostgreSQL server. To run all tests for the modules that have been selected to be built, including the core tests, you need to run either make check-world or make installcheck-world corresponding to a temporary installation within the build tree or a running PostgreSQL server. However, some features still won’t be tested by running make check-world or make installcheck-world, such as the security-related features, ssl, kerberos, etc. One of the reasons is that these test cases need some special settings. To run a regression test for these features, as the document mentioned, you need to run the tests like make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance'.

3. Run SSL/TLS Regression Tests

To run the regression tests for SSL/TLS related features, first, you need to compile PostgreSQL with the SSL library enabled. For example, running the commands below will compile PostgreSQL with OpenSSL libraries:

git clone https://github.com/postgres/postgres.git
cd postgres
./configure --prefix=/tmp/pgapp --enable-tap-tests CC="gcc -std=gnu99" --with-openssl
make -j
make check-world

If make check-world is okay, then you can check if the SSL/TLS regression test has been performed by checking the log at src/test/ssl/tmp_check/log.

$ cat *
[11:27:05.488](0.021s) 1..0 # SKIP Potentially unsafe test SSL not enabled in PG_TEST_EXTRA
[11:27:05.584](0.011s) 1..0 # SKIP Potentially unsafe test SSL not enabled in PG_TEST_EXTRA
[11:27:05.671](0.011s) 1..0 # SKIP Potentially unsafe test SSL not enabled in PG_TEST_EXTRA

The logs indicate that all three SSL/TLS regression tests (001_ssltests.pl, 002_scram.pl, and 003_sslinfo.pl) have been skipped.

In other words, you may break some SSL/TLS features if your fix is related to SSL/TLS but you only run make check-world on the fix. To test SSL/TLS related features using existing regression test cases, you need to run the SSL/TLS tests using the commands below:

cd src/test/ssl
make check PG_TEST_EXTRA=ssl

The expected results are something like below:

... 
# +++ tap check in src/test/ssl +++
t/001_ssltests.pl .. ok     
t/002_scram.pl ..... ok    
t/003_sslinfo.pl ... ok    
All tests successful.
Files=3, Tests=247, 11 wallclock secs ( 0.05 usr  0.00 sys +  2.01 cusr  1.59 csys =  3.65 CPU)
Result: PASS

Then you should see the logs in src/test/ssl/tmp_check/log like below with a lot of more details.

$ ls -l
total 164
-rw------- 1 david david 82348 Apr 19 11:39 001_ssltests_primary.log
-rw------- 1 david david 12408 Apr 19 11:39 002_scram_primary.log
-rw------- 1 david david 16402 Apr 19 11:39 003_sslinfo_primary.log
-rw-rw-r-- 1 david david 30451 Apr 19 11:39 regress_log_001_ssltests
-rw-rw-r-- 1 david david  4921 Apr 19 11:39 regress_log_002_scram
-rw-rw-r-- 1 david david  3880 Apr 19 11:39 regress_log_003_sslinfo

If you look at the log carefully, you may find out that a lot of messages contain certificate like below.

$ cat tmp_check/log/* | grep certificate
...
[11:39:27.633](0.000s) ok 31 - cert root file that contains two certificates, order 2: no stderr
[11:39:27.717](0.027s) ok 36 - connect with sslcertmode=require fails without a client certificate
[11:39:27.717](0.000s) ok 37 - connect with sslcertmode=require fails without a client certificate: matches
[11:39:27.882](0.028s) ok 48 - mismatch between host name and server certificate sslmode=require
...

4. How to generate SSL/TLS certificates

Then you might come up with some questions like, “where are the certificates coming from?” and “how were the certificates generated?”

Actually, all the certificates were generated using the Makefile under src/test/ssl and committed to PostgreSQL as part of the test case. Typically, you don’t need to generate them again unless you have some changes which require new or different certificates. Then you can run the commands below:

make sslfiles-clean
make sslfiles

After that, if you run git status, then you should find that all the files under src/test/ssl/ssl have been changed. At this point, if you run the previous commands make check PG_TEST_EXTRA=ssl, then it should use the newly generated certificates for the regression test cases. The reason I want to mention the certificates here is that certificates play a very important role in the SSL/TLS regression test.

5. Summary

In this blog post, I have explained a simple process to run SSL/TLS related regression tests. You can run similar regression tests for Kerberos, LDAP, etc.