HighGo Software Inc. (Canada) is pleased to announce the first GitHub community release of MongoDB Logical Decoding plugin v1.0.6 (wal2mongo), which can be used to replicate PostgreSQL database changes to an output format that can be directly fed into the mongo client tool to achieve logical replication between PostgreSQla and MongoDB. Wal2mongo plugin is useful for a case where PostgreSQL is used as the main raw data source to collect data from outside world but MongoDB is used internally for data analytics purposes. Manual data migration between PostgreSQL and MongoDB poses a lot of potential problem and having a logical decoding plugin like wal2mongo can help reduce the data migration complexity between the two databases.

Wal2mongo plugin produces outputs that can be understood by MongoDB without further parsing. For this reason, it can be used with the pg_recvlogical front end tool to achieve a very simple logical replication between the two databases. Simply feed the outputs of pg_recvlogical to the mongo client tool like this.

pg_recvlogical -S slot -f - --start | mongo

Both source and binary releases (for CentOS6, 7) of wal2mongo plugin can be found here

Feature Highlight

• Rich support in data type translation between PostgreSQL and MongoDB
• Support complex data type translation such as arrays, XML, JSON, timestamp …etc
• Support database WAL changes caused by INSERT, UPDATE, and DELETE clauses.
• Support 2 output modes: regular output mode and transaction output mode (for multi-document transaction control)
• Support MongoDB version 3.6, 4.0 and above
• Support multi-publisher translation mode
• Support change stream identification using a combination of cluster name, database name and replication slot name such * that when multiple PostgreSQL servers using the same decoding plugin, the destination MongoDB server is able to distinguish the source of the change stream
• simple, lightweight and feature-rich.
• Can be paired with HighGo replication application to achieve a full automatic replication setup.

Road Map

The HighGo team is currently working on a logical replication application project that can be used with wal2mongo to achieve a fully automatic logical replication setup with enhanced control, security and performance in mind. This application will be available in the near future. Please check back often for updates. In the meantime, We will continue to improve the logical decoding performance and enhance wal2mongo functionalities, feature such as supporting the output in MongoDB extended JSON v2 format is definitely in our road map. Stay tuned.

Cary Huang

Cary is a Senior Software Developer in HighGo Software Canada with 8 years of industrial experience developing innovative software solutions in C/C++ in the field of smart grid & metering prior to joining HighGo. He holds a bachelor degree in Electrical Engineering from University of British Columnbia (UBC) in Vancouver in 2012 and has extensive hands-on experience in technologies such as: Advanced Networking, Network & Data security, Smart Metering Innovations, deployment management with Docker, Software Engineering Lifecycle, scalability, authentication, cryptography, PostgreSQL & non-relational database, web services, firewalls, embedded systems, RTOS, ARM, PKI, Cisco equipment, functional and Architecture Design.