The Rise of Serverless Database

Enterprise PostgreSQL Solutions

Comments are off

The Rise of Serverless Database

Introduction

Serverless architecture has been gaining popularity in the world of technology. With its promise of reduced infrastructure costs and increased scalability, serverless architecture has become a go-to solution for many companies, big and small. This popular architecture makes developers not to consider issues such as server management, scalability, backup, failover or anything infrastructure-related issues. Instead, they can focus on application development, improving productivity and CI/CD performance.

Serverless can also have cost benefits, because users only pay for the computing resources they actually use, instead of buying expensive servers and storage devices or renting virtual private servers. It offers scale-to-zero when there is no traffic and scale up when there is traffic.

Pros and Cons of serverless architecture

Pros

  • Enterprises do not need to manage backend infrastructure
  • Save team time, no need for a full server management team
  • Low operating costs and better development productivity
  • Optimized pay-by-use architecture to have cost savings

Cons

  • Lack of control, higher dependency
  • Cold start affects performance
  • Vendor lock-in may be a risk
  • Security concerns

Neon Serverless Database

Neon Serverless database is one of the current players in the serverless database space. It is built on top of PostgreSQL and achive the separation of compute and storage, which essentially turns PostgreSQL from a stateful to a stateless machine. The core components that make this possible are written in rust. This fundamental change is the main building block of bringing PostgreSQL serverless.

The basic architecture of Neon looks like this:

where the green and purple blocks form the storage part and the blue forms the compute part. The PostgreSQL nodes in the blue box are the compute nodes that can be started or stopped by an orchastrator like kubernetes based on load. It also supports multi-tenancy, which means we can have multiple compute nodes up to serve different tenants, each have a totally isolated and separated database spaces.

Changes done on the compute nodes generate WAL files, which are sent to safekeepers. Paxos consensus protocol is used to ensure majority of the safekeepers ackowledged the receipt of a WAL segment, before it passes the segment to pageserver where the actual WAL REDO takes place. Safekeeper is also responsible for do WAL backups on cloud storage (currently support minio and AWS S3).

Page server is responsible for serving page data to requesting compute nodes. It produces a page based on the LSN requested by compute nodes. Based on this, it could derive a new page from a historical page by apply WAL (REDO) on the page. It is also responsible for backing up layer files (page and WAL files needed to derive any page) to cloud storage.

The core components are open-source located here and it also provide free and paid tier of serverless experience. You can learn more about them here

conclusion

Serverless architecture is changing the game in database hosting. It offers a cost-effective and scalable solution for companies of all sizes. With Neon achieving separation of storage and compute and have them open-sourced; it could potentially attract other players to build more innovations around this innovation. I am sure, very soon, we will see more new PostgreSQL-based serverless solutions emerging in the market.