We are launching Pisanix v0.1.0, an open source solution for Database Mesh. From its inception, Pisanix embraced the core idea of Database Mesh: to build an efficient and programmable database governance experience for the cloud native era. This marks a turning point, making Database Mesh no longer mere theory, but a framework that can be implemented in real scenarios.

Pisanix is an open source solution for Database Mesh sponsored by SphereEx. Written in Go and Rust for the Kubernetes environment, it currently supports MySQL. It includes the following three components: Pisa-Controller, Pisa-Proxy and Pisa-Daemon (coming soon), which help users easily implement SQL-aware traffic governance based on the Database Mesh framework, runtime resource-oriented programmability, and database reliability engineering (DBRE) to facilitate database governance in the cloud native era.

Pisanix v0.1.0 Features

Pisanix v0.1.0 supports the following features:

• Pisa-Controller
  • Support the injection of Sidecar.
  • Read Kubernetes custom resources, such as VirtualDatabase, TrafficStrategy and DatabaseEndpoint, and convert them to Pisa-Proxy configuration.
• Pisa-Proxy
  • Support the deployment of Kubernetes Sidecar and obtain configuration from Pisa-Controller.
  • Support multi-port MySQL protocol proxy.
  • Support connection pool and basic load balancer.
  • Support Yacc-based SQL parser including , SELECT, INSERT, UPDATE, DELETE, PREPARE, EXECUTE, BEGIN, and SET.
  • Support Regex-based SQL circuit breaker.
  • Support SQL concurrency control: restrict the number of SQL executed at a given time based on the Regex.
  • Observability: support sql_processed_total and sql_processed_duration.
  • Support deployment in a non-Kubernetes environment as an independent component.

Introducing Pisanix

Pisanix adopts the classic deployment mode of separating the control plane and data plane. It supports unified database access declaration configuration, supports access resource limits to the programmable database, and supports standardized automatic database maintenance through various CustomResourceDefinition. It is designed to implement database governance in a cloud native environment.

Pisanix currently provides the following capabilities:

• SQL-Aware Traffic Governance: provide load balancing, auditing, access control, and observability of database traffic based on SQL parsing capability.
• Runtime Resource-Oriented Management: support configurable and programmable resource capabilities by means of Linux kernel.
• DBRE: it can design an engineer-centered product with database reliability as the principle, and establish a unified cloud interface on the database Pisa-Controller Pisa-Controller is the control plane of Pisanix and an indispensable component, implemented by Go language. It is responsible for the control of data plane components and is the entry point for all Pisanix configurations. It has the following functions:
• Sidecar injection: inject Pisa-Proxy container into the specified Pod with MutatingAdmissionWebhook.
• Pisa-Proxy Configuration delivery and indicator collection: obtain and convert service discovery, load balancer, traffic control, and SQL firewall rules in CRD format in the cluster, deliver them to Pisa-Proxy, and collect various indicators when Pisa-Proxy is running.
• Pisa-Daemon configuration conversion and indicator collection: obtain and convert traffic QoS rules in CRD format in the cluster, deliver them to Pisa-Daemon, and collect various indicators when Pisa-Daemon is running.

Pisa-Proxy Pisa-Proxy is a high-performance proxy and an indispensable component implemented by Rust. It is deployed in the same Pod with business applications in Sidecar mode. It monitors the MySQL protocol to obtain the traffic when applications access the database, and provides various governance capabilities:

• SQL traffic governance: implement load balancer strategies and circuit breaker through SQL parsing.
• Access control: achieve fine-grained permission control based on the relationship between users and data permission.
• SQL firewall: prevent high-risk SQL statements from being executed
• Observability: expose various database access indicators, such as throughput and latency. Pisa-Daemon (coming soon） Pisa-Daemon is a node-level data plane and is an optional component. Pisa-Daemon is deployed on every node in a Kubernetes cluster and provides functions such as resource management through the various capabilities of the host kernel.
• Runtime traffic governance: it provides QoS guarantee for database access traffic with eBPF and other technologies to reduce the interference between traffic.

Pisanix Examples

Database Mesh's governance framework relies on the following workloads:

• Virtual database: a database endpoint accessible to developers.
• Traffic strategy: governance strategies for database access traffic such as label router, data sharding, load balancer, concurrency control and circuit breaker.
• Access control: provides fine-grained access control based on specified rules, such as table level.
• Security statement: data security statements such as data encryption.
• Audit application: record applications’ operation behavior on the database, such as accessing the risk control system.
• Observability: configuration of database observability such as access traffic, running state, and performance indicators etc.
• Event bus: an event bus that accepts data changes.
• QoS statements: resource QoS metrics set to improve the overall SLO metrics of the database.
• Backup plan: perform database backups as scheduled tasks.
• Schema pipeline: manage database schema changes in code to improve the success rate of database DDL and DML changes.

Currently, Pisanix supports the following three CustomResourceDefinition: VirtualDatabase, TrafficStrategy and DatabaseEndpoint.

VirtualDatabase VirtualDatabase is the central concept for all database governance activities. VirtualDatabase represents an endpoint for database access for developers and a logical database for DBAs. The actual back-end data source, DatabaseEndpoint, and specified TrafficStrategy need to be provided for VirtualDatabase. Pisanix currently supports VirtualDatabase of the MySQL type. The configuration example is as follows:

apiVersion: core.database-mesh.io/v1alpha1
kind: VirtualDatabase
metadata:
  name: catalogue
  namespace: default
spec:
  services:
  - databaseMySQL:               # the virtual database type is MySQL
      db: socksdb                # virtual database: schema
      host: 127.0.0.1            # access address of virtual database
      port: 3306                 # access port of virtual database
      user: catalogue_user       # access username of virtual database
      password: default_password # access password of virtual database
    name: mysql                  # name of virtual database
    trafficStrategy: catalogue   # traffic strategy of specified virtual database

TrafficStrategy TrafficStrategy describes the governance policies for database access traffic, which are configured by DBAs, such as label-based request routing, SLA-driven load balancer, SQL circuit breaker, and concurrency control. Pisanix supports basic load balancer, regex-based SQL circuit breaker, and concurrency control. The configuration example is as follows:

apiVersion: core.database-mesh.io/v1alpha1
kind: TrafficStrategy
kind: TrafficStrategy
metadata:
  name: catalogue
  namespace: default
spec:
  loadBalance:                   # load balancer strategy of accessing virtual database
    simpleLoadBalance:           # load balancer strategy is simple load balancer
      kind: random               # load balancer strategy is random
  selector:
    matchLabels:                 # declare label selector to select a target DatabaseEndpoint
      source: catalogue

DatabaseEndpoint DatabaseEndpoint describes the actual database access endpoint, which can be a database instance deployed on the cloud and is a priority resource for DBAs. As an abstraction from a real database, DatabaseEndpoint can be consumed by multiple VirtualDatabase, meeting different needs of developers and DBAs. Pisanix supports DatabaseEndpoint of the MySQL type. The configuration example is as follows: source

apiVersion: core.database-mesh.io/v1alpha1
kind: DatabaseEndpoint
metadata:
  name: catalogue-db
  namespace: default
spec:
  database:
    MySQL:                        # data source type is MySQL
      db: socksdb                 # database is schema
      host: cataloguedb.codtynlacssn.rds.cn-northwest-1.amazonaws.com.cn # address of data source
      port: 3306                  # access port of data source
      user: root                  # access username of data source    
      password: fake_password     # access password of data 

Weaveworks' Socks-Shop simulates the user-facing part of an e-commerce website that sells socks. It is intended to aid the demonstration and testing of microservice and cloud native technologies and provide a real testable application. The project uses the following technologies or frameworks: Java Spring Boot, Go, Node.js and uses containers to build and deploy.

Taking the socks-shop microservice project as an example, its governance process with the Pisanix framework is as follows:

1. The developer will submit VirtualDatabase for Catalogue service to apply for the VirtualDatabase type.
2. The DBA creates the associated TrafficStrategy and DatabaseEndpoint for VirtualDatabase.
3. Add label pisanix.io/inject=enabled for the Catalogue service by the SRE, enable the injection of Sidecar, and then configure the Catalogue database access information to the address in VirtualDatabase and start the application. After the configuration is completed, access the front-end service for verification.

Pisanix's design concept: Database Mesh

The article Database Mesh 2.0: Database Governance in a Cloud Native Environment, describes how Database Mesh 2.0 provides a database-centric governance framework.

• Databases are first-class citizens. All abstractions are centered on database governance, such as access control, traffic governance, and observability.
• Engineer-oriented experience: developers can continue to develop through easy to use database declarations and definitions regardless of the location of the database. For operation and maintenance personnel and DBAs, it provides a variety of abstraction of database governance behavior to realize automatic database reliability engineering.
• Cloud native: built and oriented towards cloud native, it is suitable to various cloud environments with an open ecosystem and implementation mechanism. Vendor lock-in is effectively eliminated. For more details, refer to Database Mesh's official website.

Outlook

Pisanix is just getting started. In the future, it will focus on enhancing traffic governance ability, such as developing data sharding ability based on sharding key, audit application for data access behavior, QoS of runtime resources, etc. while continuously improving its performance and ease of use. Pisanix plans to open the plugin extension mechanism, allowing users to write policies to adapt to different business scenarios, putting Unix's design philosophy of "policy and mechanism separation" into practice.

Community

Pisanix is just getting started. The Pisanix welcomes you to join us and share your ideas and opinions. It can be with code or documentation, issues or pull requests. For those of you who are willing to engage in database governance, let's build the Pisanix community together. The community holds an online discussion every two weeks:

• Mailing list
• Biweekly meeting - English community (from February 27, 2022), on Wednesday 9:00 AM PST
• Biweekly meeting - Chinese community (from April 27, 2022), on Wednesday 9:00 PM GMT+8
• Slack
• Meeting minutes