1. Overview
In my previous blogs, I discussed Setting up a debugging environment in PostgreSQL to better understand OpenSSL APIs, Enhance PostgreSQL TLS Security with OCSP Stapling, and How to setup TLS connection for PostgreSQL. In this blog, I will share a simple procedure about How to run SSL/TLS regression tests in PostgreSQL.
2. Postgres regression tests
Whenever we want to add a new feature or make some fix, we should run the tests provided by PostgreSQL to make sure nothing is broken. If the new feature or the fix does not have corresponding tests, we should consider adding some to ensure it will not break in the future when other changes are made. PostgreSQL provides a comprehensive document) explaining all available tests. For example, if you want to run a quick regression test to check if any “core” feature may be broken, you can run make check
for a temporary installation within the build tree or make installcheck
against a running PostgreSQL server. make check
and make installcheck
will only test the built-in functionality of the PostgreSQL server. To run all tests for the modules that have been selected to be built, including the core tests, you need to run either make check-world
or make installcheck-world
corresponding to a temporary installation within the build tree or a running PostgreSQL server. However, some features still won’t be tested by running make check-world
or make installcheck-world
, such as the security-related features, ssl, kerberos, etc. One of the reasons is that these test cases need some special settings. To run a regression test for these features, as the document mentioned, you need to run the tests like make check-world PG_TEST_EXTRA='kerberos ldap ssl load_balance'
.
3. Run SSL/TLS Regression Tests
To run the regression tests for SSL/TLS related features, first, you need to compile PostgreSQL with the SSL library enabled. For example, running the commands below will compile PostgreSQL with OpenSSL libraries:
git clone https://github.com/postgres/postgres.git
cd postgres
./configure --prefix=/tmp/pgapp --enable-tap-tests CC="gcc -std=gnu99" --with-openssl
make -j
make check-world
If make check-world
is okay, then you can check if the SSL/TLS regression test has been performed by checking the log at src/test/ssl/tmp_check/log
.
$ cat *
[11:27:05.488](0.021s) 1..0 # SKIP Potentially unsafe test SSL not enabled in PG_TEST_EXTRA
[11:27:05.584](0.011s) 1..0 # SKIP Potentially unsafe test SSL not enabled in PG_TEST_EXTRA
[11:27:05.671](0.011s) 1..0 # SKIP Potentially unsafe test SSL not enabled in PG_TEST_EXTRA
The logs indicate that all three SSL/TLS regression tests (001_ssltests.pl, 002_scram.pl, and 003_sslinfo.pl) have been skipped.
In other words, you may break some SSL/TLS features if your fix is related to SSL/TLS but you only run make check-world
on the fix. To test SSL/TLS related features using existing regression test cases, you need to run the SSL/TLS tests using the commands below:
cd src/test/ssl
make check PG_TEST_EXTRA=ssl
The expected results are something like below:
...
# +++ tap check in src/test/ssl +++
t/001_ssltests.pl .. ok
t/002_scram.pl ..... ok
t/003_sslinfo.pl ... ok
All tests successful.
Files=3, Tests=247, 11 wallclock secs ( 0.05 usr 0.00 sys + 2.01 cusr 1.59 csys = 3.65 CPU)
Result: PASS
Then you should see the logs in src/test/ssl/tmp_check/log
like below with a lot of more details.
$ ls -l
total 164
-rw------- 1 david david 82348 Apr 19 11:39 001_ssltests_primary.log
-rw------- 1 david david 12408 Apr 19 11:39 002_scram_primary.log
-rw------- 1 david david 16402 Apr 19 11:39 003_sslinfo_primary.log
-rw-rw-r-- 1 david david 30451 Apr 19 11:39 regress_log_001_ssltests
-rw-rw-r-- 1 david david 4921 Apr 19 11:39 regress_log_002_scram
-rw-rw-r-- 1 david david 3880 Apr 19 11:39 regress_log_003_sslinfo
If you look at the log carefully, you may find out that a lot of messages contain certificate
like below.
$ cat tmp_check/log/* | grep certificate
...
[11:39:27.633](0.000s) ok 31 - cert root file that contains two certificates, order 2: no stderr
[11:39:27.717](0.027s) ok 36 - connect with sslcertmode=require fails without a client certificate
[11:39:27.717](0.000s) ok 37 - connect with sslcertmode=require fails without a client certificate: matches
[11:39:27.882](0.028s) ok 48 - mismatch between host name and server certificate sslmode=require
...
4. How to generate SSL/TLS certificates
Then you might come up with some questions like, “where are the certificates coming from?” and “how were the certificates generated?”
Actually, all the certificates were generated using the Makefile
under src/test/ssl
and committed to PostgreSQL as part of the test case. Typically, you don’t need to generate them again unless you have some changes which require new or different certificates. Then you can run the commands below:
make sslfiles-clean
make sslfiles
After that, if you run git status
, then you should find that all the files under src/test/ssl/ssl
have been changed. At this point, if you run the previous commands make check PG_TEST_EXTRA=ssl
, then it should use the newly generated certificates for the regression test cases. The reason I want to mention the certificates here is that certificates play a very important role in the SSL/TLS regression test.
5. Summary
In this blog post, I have explained a simple process to run SSL/TLS related regression tests. You can run similar regression tests for Kerberos, LDAP, etc.

A software developer specialized in C/C++ programming with experience in hardware, firmware, software, database, network, and system architecture. Now, working in HighGo Software Inc, as a senior PostgreSQL architect.
Recent Comments