Percona Community Blog - learn about MySQL, MariaDB, PostgreSQL, and MongoDB

⠀Full step-by-step documentation is here.

InnoDB Buffer Pool Tuning: From Rule-of-Thumb to Real Signals

Wayne Leutwyler — Thu, 02 Apr 2026 00:00:00 UTC

Introduction

Many MySQL setups begin life with a familiar incantation:

innodb_buffer_pool_size = 70% of RAM

…and then nothing changes.

That’s not tuning. That’s a starting guess.

Real tuning starts when the workload pushes back.

Visual Overview

The InnoDB buffer pool is where database performance is quietly decided. It determines whether your workload hums along in memory or drags itself across disk. If you’re not actively observing and tuning it, you’re leaving performance on the table.

This guide walks through how to monitor, understand, and tune the buffer pool using real signals instead of guesswork.

What the Buffer Pool Really Is

The buffer pool isn’t just “memory for MySQL.” It’s a living system under constant pressure:

A cache of data and indexes
A write staging area (dirty pages)
A contention zone between reads, writes, and eviction

Think of it as your database’s working memory. If your working set fits, queries glide. If it doesn’t, pages are constantly evicted and reloaded, introducing latency that rarely announces itself clearly.

A Simple Mental Model

 +---------------------------+
 | Buffer Pool |
 |---------------------------|
Reads ---> | Cached Pages |
 | |
Writes ---> | Dirty Pages (pending IO) |
 | |
Eviction -> | LRU / Free List |
 +---------------------------+
 |
 v
 Disk (slow)

Three forces are always competing:

Reads want hot data in memory
Writes generate dirty pages
Eviction makes room under pressure

Your job is to keep this system balanced.

How to Monitor the Buffer Pool

Option 1: Quick Snapshot

sql
SHOW ENGINE INNODB STATUS\G

Useful for human inspection. Look for:

Buffer pool size
Free buffers
Database pages
Modified (dirty) pages
Page read/write rates

Great for debugging. Not ideal for automation.

Option 2: Structured Metrics (Recommended)

sql
SELECT
 pool_id,
 free_buffers,
 database_pages,
 modified_database_pages
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Key fields:

free_buffers → Available pages (breathing room)
database_pages → Pages holding data
modified_database_pages → Dirty pages waiting to flush

Great for automation.

The 5 Signals That Actually Matter

1. Buffer Pool Hit Ratio (Handle With Care)

Yes, it’s widely used. No, it’s not enough.

A high hit ratio does not mean your system is healthy. It does not capture:

Page churn
Eviction pressure
Access patterns

You can have a 99% hit ratio and still be IO-bound.

Use it as a sanity check, not a decision-maker.

2. Free Buffers

sql
SELECT SUM(free_buffers) AS free_buffers
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Interpretation:

Near zero during steady load → normal
Near zero + rising disk reads → pressure
Near zero while mostly idle → suspicious (possible misread or config issue)

3. Dirty Page Percentage

sql
SELECT
 (SUM(modified_database_pages) / SUM(database_pages)) * 100.0 AS dirty_pct
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Interpretation (context matters):

0–5% → Very clean
5–20% → Typical
20–30%+ → Potential flushing lag

4. Disk Read Pressure (Critical Signal)

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';
-- Take two samples 60s apart and compare

Track the rate of change (reads/sec), not the absolute value.

Interpretation:

Rising reads → Working set does not fit in memory
Flat reads → Memory is absorbing the workload

5. Read Ahead / Eviction Pressure

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read_ahead%';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_evicted';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';

Interpretation:

Efficient read-ahead:
- read_ahead increases
- read_ahead_evicted remains low
Inefficient read-ahead (wasted IO):
- High read_ahead_evicted / read_ahead
- Indicates access patterns defeating prefetching
Buffer pool churn:
- pages_evicted rising
- buffer_pool_reads rising
- Indicates pages are evicted and re-read from disk
Healthy vs unhealthy eviction:
- High evictions + stable reads → normal turnover
- High evictions + rising reads → memory pressure

Focus on rates of change over time, not absolute values.

Detecting Thrashing

Thrashing is when the buffer pool constantly evicts and reloads pages.

Classic Symptoms

Low or zero free buffers
Increasing disk reads
Stable (but misleading) hit ratio
Spiky query latency

Visualizing Thrash

Time --->

Memory: [FULL][FULL][FULL][FULL]
Reads: ↑ ↑↑ ↑↑↑ ↑↑↑↑
Latency: - ^ ^^ ^^^
Evictions: ↑ ↑↑ ↑↑↑ ↑↑↑↑

If you see this pattern, your working set does not fit in memory.

Tuning the Buffer Pool

Step 1: Size It Intentionally

Instead of blindly assigning 70% of RAM:

Observe working set behavior
Monitor free buffers and reads
Increase gradually

Avoid starving the OS or filesystem cache.

Step 2: Tune Flushing Behavior

innodb_max_dirty_pages_pct = 75
innodb_io_capacity = 1000
innodb_io_capacity_max = 2000

Sustained IO spikes → increase innodb_io_capacity
Dirty pages climbing → flushing lag
Sudden stalls → checkpoint pressure

What they control:

innodb_io_capacity → Expected steady-state IO throughput
innodb_io_capacity_max → Burst flushing capacity
innodb_max_dirty_pages_pct → Threshold for aggressive flushing

⚠️ These values should reflect real hardware capability.

Step 3: Buffer Pool Instances:Reduce Contention

A practical, battle-tested guideline:

Use 1 instance per ~1GB of buffer pool, up to a reasonable limit.

Buffer Pool Instances: Reducing Contention

The buffer pool can be split into multiple instances, each managing its own internal structures. This helps reduce contention under high concurrency.

Without this, all threads compete for the same buffer pool internals. With multiple instances, that load is distributed.

When It Matters

Buffer pool instances only help when contention exists. You’ll see benefits if your system has:

High concurrency (many active threads)
CPU-bound workloads
Mutex contention in InnoDB

If your workload is primarily IO-bound, this setting will have little impact.

Sizing Guidelines

General guidance:

< 1GB buffer pool → 1 instance
1GB–8GB → 2–4 instances
8GB–64GB → 4–8 instances
64GB+ → 8–16 instances

Keep Instances Large Enough

Each instance needs enough memory to function efficiently.

Avoid going below ~1GB per instance.

If instances are too small:

LRU efficiency drops
Eviction becomes more aggressive
Cache locality suffers

Example

sql
innodb_buffer_pool_size = 32G
innodb_buffer_pool_instances = 8

This gives ~4GB per instance, which is well-balanced.

Common Mistakes

Increasing instances without evidence of contention
Matching instance count to CPU cores
Using many instances with a small buffer pool
Expecting this to fix IO bottlenecks

Step 4: Understand Resizing Behavior

Buffer pool resizing is online in modern MySQL versions, but:

It happens in chunks
Controlled by innodb_buffer_pool_chunk_size

Real-World Scenarios

Scenario 1: “Everything Looks Fine… But It’s Slow”

High hit ratio
Low free buffers
Rising disk reads

Cause: Working set barely fits

Fix: Increase buffer pool size gradually

If increasing the buffer pool size does not reduce disk reads, the problem is not memory.

Scenario 2: Write-Heavy Workload

Dirty pages increasing
Periodic IO spikes

Cause: Flushing cannot keep up

Fix:

Increase innodb_io_capacity
Adjust dirty page thresholds

Scenario 3: Sudden Latency Spikes

Sharp performance drops
Disk activity surges

Cause: Checkpoint pressure

Fix:

Improve IO capacity tuning
Reduce dirty page buildup

Practical Monitoring Queries

Buffer Pool Usage (MB)

sql
SELECT
 (SUM(database_pages) * 16) / 1024 AS mb_used
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Assumes default 16KB page size (innodb_page_size).

Dirty Page Percentage

sql
SELECT
 (modified_database_pages / database_pages) * 100 AS dirty_pct
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Free Buffer Check

sql
SELECT SUM(free_buffers) AS free_buffers
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Common Mistakes

Treating 70% as a rule instead of a starting point
Blindly trusting hit ratio
Ignoring disk read trends
Oversizing and starving the OS
Not tuning IO capacity
Leaving defaults in write-heavy systems

Quick Checklist

If you remember nothing else:

Reads increasing? → working set too big
Free buffers always ~0? → pressure
Dirty pages high? → flushing lag
Latency spiking? → checkpoint or IO saturation

Final Thoughts

The InnoDB buffer pool doesn’t fail loudly. It degrades quietly until your disk becomes the bottleneck.

By the time you notice, you’re debugging latency instead of preventing it.

Monitor the right signals, and you’ll see problems forming before users do.

That’s the difference between reacting to performance… and controlling it.

Running pgBackRest with pg_tde: A Practical Percona Walkthrough

Shahid Ullah — Tue, 10 Mar 2026 00:00:00 UTC

Not every PostgreSQL installation requires encryption at rest. However, for organizations mandating strict data protection and privacy standards, it is often non-negotiable. When security policies are this rigorous, you need a strategy that protects your data without sacrificing recoverability.

While Transparent Data Encryption (TDE) successfully locks down your data at rest, it raises a critical operational question: will your backup and restore workflow continue to work correctly with encrypted data? When deploying Transparent Data Encryption, it is essential to validate that pgBackRest can reliably back up and restore encrypted clusters.

This post walks through a complete, step-by-step configuration of pg_tde with pgBackRest on Debian/Ubuntu. We will explore how to optimize your backup performance, secure your repository, and most importantly verify that your encrypted backup restores work exactly as expected.

What is pg_tde?

Percona’s solution for transparent data encryption (pg_tde) is an open source, community driven extension that provides Transparent Data Encryption (TDE) for PostgreSQL. This extension allows data to be encrypted at the storage level without affecting application behavior.

Unlike full disk encryption, which exposes data once the system boots, TDE ensures that the actual database files remain encrypted at the file system level. This protects your data, dumps, and backups even if the operating system is compromised. Currently, it is bundled with Percona Server for PostgreSQL and available in Percona Distribution for PostgreSQL 17+.

Recent Percona releases make this combination more practical than ever. With WAL encryption now ready for production use, we need a backup strategy that respects data security. In this walkthrough, we will demonstrate how to pair it with pgBackRest to ensure fully recoverable, encrypted backups.

The Use Case

Imagine a team that wants strong security controls without changing application code. They need:

Data files encrypted at rest (tables and WAL)
Backups that are consistent, verifiable, and restorable
A setup that is easy to automate and explain

Percona Distribution for PostgreSQL plus pg_tde and pgBackRest closes the gaps where needed: pg_tde takes care of encryption, pgBackRest provides flexible backup/restore capabilities.

A Quick Note on Compatibility

At this moment pg_tde cannot be yet used with Community PostgreSQL as pg_tde relies on specific hooks in the PostgreSQL core. Percona Server for PostgreSQL includes these necessary core modifications, which is why we validate this setup using the Percona Distribution for PostgreSQL.

While pgBackRest is fully capable of managing TDE enabled clusters, there are specific constraints you must respect to ensure data safety:

No Asynchronous Archiving: pgBackRest asynchronous archiving is not supported with encrypted WALs. You must configure your archive command to handle WALs synchronously.
Restore Wrappers: Standard restore commands will not work for encrypted WALs. You must use the pg_tde_restore_encrypt utility to wrap your restore process.

This guide currently focuses on pgBackRest because it is the backup tool that has been tested and validated with pg_tde by the pg_tde community at this time. Other backup tools may also be viable and we are open to collaborating with other tool maintainers and their communities on a shared effort to validate and support pg_tde.

What You Will Build

By the end of this post you will have:

Percona Distribution for PostgreSQL installed with pg_tde and pgBackRest
A key directory and key providers created
pg_tde enabled in PostgreSQL
Encrypt tables and indexes with pg_tde (docs)
WAL encryption enabled
A pgBackRest stanza configured and a full backup completed
Verification that encrypted data is not readable on disk

Prerequisites

A host (or VM) on Debian/Ubuntu
Network access to Percona repositories
Root/Sudo: You need sudo access for installing packages and editing system configuration files in /etc
Postgres User: All database commands (psql, pgbackrest) run as the postgres system user
Postgres user is in the sudoer list to run sudo commands

Step 1: Install Percona Packages

We begin by installing the Percona release repository and enabling the correct PostgreSQL distribution. See the Percona Distribution for PostgreSQL installation guide for full details.

Debian / Ubuntu

bash
# Install repo helper
sudo apt-get update
sudo apt-get install -y wget gnupg2 lsb-release curl
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
sudo dpkg -i percona-release_latest.generic_all.deb

# Enable the repository for the major version selected
sudo percona-release setup ppg-18
sudo apt-get update

# Install the server, tde extension, and pgbackrest
sudo apt-get install -y percona-postgresql-18 percona-pg-tde18 percona-pgbackrest

# Verify Installation
psql --version

Step 2: Enable pg_tde and Create Keys

2.1 Configure shared_preload_libraries

Before we can use any of the encryption features, PostgreSQL needs to load the pg_tde library into memory at startup. You can do this by adding it to shared_preload_libraries.

You have two ways to handle this: the SQL way or the classic config file way.

Option A: SQL way (recommended)

The fastest way is using ALTER SYSTEM. This saves you from hunting through your file system for the right config file.

bash
# Set the library and restart to apply changes
psql -c "ALTER SYSTEM SET shared_preload_libraries = 'pg_tde';"
sudo systemctl restart postgresql

Option B: Manual config edit

If you prefer managing your config files manually, find your postgresql.conf and add the extension there.

bash
# Locate the config file
PG_CONF=$(psql -t -P format=unaligned -c "show config_file;")
# Open that file and find the 'shared_preload_libraries' line:
# shared_preload_libraries = 'pg_tde'

# Restart PostgreSQL to load the library
sudo systemctl restart postgresql

Note: Regardless of which method you choose, a full restart of the PostgreSQL service is required. A simple reload won’t work for shared libraries.

2.2 Create the Key Provider

Now that the extension is loaded, we need to tell pg_tde where to store its encryption keys.

For this tutorial, we will use the File Provider (storing keys in a local file). In production environments, storing encryption keys locally on the PostgreSQL server can introduce security risks. To enhance security, pg_tde supports integration with external Key Management Systems (KMS) through a Global Key Provider interface.

Note: While key files may be acceptable for local or testing environments, KMS integration is the recommended approach for production deployments.

bash
# 1. Create a secure directory for the keys
sudo mkdir -p /etc/postgresql/keys
sudo chown postgres:postgres /etc/postgresql/keys
sudo chmod 700 /etc/postgresql/keys

sql
-- 2. Connect to PostgreSQL to configure TDE
CREATE EXTENSION pg_tde;

-- 1. Define the global key provider
SELECT pg_tde_add_global_key_provider_file('global-file-provider', '/etc/postgresql/keys/tde-global.per');

-- 2. Create and Set the Principal Key
SELECT pg_tde_create_key_using_global_key_provider('global-master-key', 'global-file-provider');
SELECT pg_tde_set_default_key_using_global_key_provider('global-master-key', 'global-file-provider');

-- 3. Enable WAL encryption configuration
ALTER SYSTEM SET pg_tde.wal_encrypt = 'on';

bash
# 4. Restart PostgreSQL to apply the encryption settings fully
sudo systemctl restart postgresql

Step 3: Configure pgBackRest

Next, configure pgBackRest by defining the repository and stanza settings. Before proceeding, it is important to understand how encryption is handled in this setup. pg_tde encrypts PostgreSQL data files on disk, protecting the live database. However, during WAL archiving, the pg_tde archive helper decrypts WAL records before passing them to pgBackRest. This means backups and archived WAL would be stored unencrypted unless repository encryption is enabled. To ensure backup data remains protected at rest, we enable pgBackRest repository encryption in this configuration. We configure the repository with a cipher type and key. Encryption is performed client side, ensuring data is secure before it is written to the repository.

Note on Compression: In many pgBackRest deployments, compression is enabled to reduce backup size. However, when using pg_tde, database pages are already encrypted before pgBackRest processes them. Encryption randomizes the data blocks, making traditional compression algorithms such as gzip ineffective. For this reason, compression is disabled to avoid unnecessary CPU overhead.

bash
# Create the configuration file
# /etc/pgbackrest.conf
# In pg1-path use data directory path accordingly
sudo bash -c "cat < /etc/pgbackrest.conf
[demo]
pg1-path=/var/lib/postgresql/18/main

[global]
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2
log-level-console=info
start-fast=y

# PERFORMANCE: Encrypted data doesn't compress. We save CPU by disabling it.
compress-type=none

# SECURITY: Since the helper sends decrypted data, we MUST encrypt the repo.
repo1-cipher-type=aes-256-cbc
repo1-cipher-pass=Ifw7O0kTvdU5127L1gu8q3xVfWM61kl/NruTxQFWf9xP8A63Tg2IggRR9LUL9yJd
# TDE REQUIREMENT:
# Asynchronous archiving is NOT supported with pg_tde.
EOF"

Secure the Configuration File

Because this file contains your repository’s master passphrase, you must restrict access so only the postgres user can read it.

bash
sudo chown postgres:postgres /etc/pgbackrest.conf
sudo chmod 600 /etc/pgbackrest.conf

bash
# Create the repository directory
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown postgres:postgres /var/lib/pgbackrest

Understanding the Settings

pg1-path: The data directory path to be backed up
repo1-path: The directory where backups will be stored
repo1-retention-full: Keep only two full backups
start-fast=y: Forces a checkpoint immediately when a backup starts. Without this, the backup would wait for the next scheduled checkpoint.
compress-type=none: By skipping compression, we eliminate unnecessary CPU overhead since compressing encrypted data blocks yields almost zero storage benefit.
repo1-cipher-type and repo1-cipher-pass: Enable pgBackRest repository encryption. While pg_tde protects the live database files, these settings ensure that backup files and archived WAL stored in the repository are also encrypted at rest using AES-256

Step 4: Wire pgBackRest into PostgreSQL Archiving

pg_tde encrypts WAL files on disk. To allow pgBackRest to archive them correctly, we must decrypt them on the fly using the pg_tde_archive_decrypt wrapper.

Now, configure the archive_command. This command tells PostgreSQL to pipe the WAL file through the decryption wrapper before handing it off to pgBackRest.

sql
ALTER SYSTEM SET wal_level = 'replica';
ALTER SYSTEM SET max_wal_senders = 4;
ALTER SYSTEM SET archive_mode = 'on';
ALTER SYSTEM SET archive_command = '/usr/lib/postgresql/18/bin/pg_tde_archive_decrypt %f %p "pgbackrest --config=/etc/pgbackrest.conf --stanza=demo archive-push %%p"';

bash
# Restart PostgreSQL (adjust service name if needed, e.g., postgresql-18)
sudo systemctl restart postgresql

Step 5: Validate Encryption on Disk

Let’s verify that pg_tde is actually doing its job. We will create two tables, one standard and one encrypted and then inspect the raw files on disk to see the difference.

sql
-- 1. Create data: One clear text, one encrypted
CREATE TABLE IF NOT EXISTS clear_table (id INT, secret_info TEXT);
CREATE TABLE IF NOT EXISTS crypt_table (id INT, secret_info TEXT) USING tde_heap;

-- Scenario A: The "Flushed" Data (Testing the Heap)
-- We insert data and force a CHECKPOINT to push it to the .rel files.
INSERT INTO clear_table (id, secret_info) VALUES (1, 'FIND_ME_EASILY_123');
INSERT INTO crypt_table (id, secret_info) VALUES (1, 'HIDDEN_FROM_DISK_456');
CHECKPOINT;

-- Scenario B: The "In-Flight" Data (Testing the WAL)
-- We insert data but do NOT checkpoint. This data exists only in the WAL.
INSERT INTO clear_table VALUES (2, 'VISIBLE_IN_WAL_789');
INSERT INTO crypt_table VALUES (2, 'HIDDEN_IN_WAL_000');

-- Force a WAL switch so the archive helper processes the segment immediately
SELECT pg_switch_wal();

-- Verify TDE encryption status
-- For non-encrypted tables, this must return 'f' (false)
SELECT pg_tde_is_encrypted('clear_table');

-- For encrypted table, this must return 't' (true)
SELECT pg_tde_is_encrypted('crypt_table');

bash
# 2. Locate the files on disk
DATA_DIR=$(psql -t -P format=unaligned -c "show data_directory;")
CLEAR_FILE=$(psql -t -P format=unaligned -c "SELECT pg_relation_filepath('clear_table');")
CRYPT_FILE=$(psql -t -P format=unaligned -c "SELECT pg_relation_filepath('crypt_table');")
LATEST_WAL=$(ls -t ${DATA_DIR}/pg_wal | head -n 1)

# 3. Grep for the secret strings
echo "--- CHECKING DATA FILES ---"
echo "Checking Clear Table (Should Match):"
grep -a "FIND_ME_EASILY_123" "${DATA_DIR}/${CLEAR_FILE}" && echo " -> FOUND: Clear text is visible!"

echo "Checking Encrypted Table (Should FAIL):"
grep -a "HIDDEN_FROM_DISK_456" "${DATA_DIR}/${CRYPT_FILE}" || echo " -> CLEAN: Encrypted text was NOT found."

echo -e "\n--- CHECKING THE WAL (In-Flight) ---"
grep -a "VISIBLE_IN_WAL_789" "${DATA_DIR}/pg_wal/${LATEST_WAL}" && echo " -> FOUND: WAL contains clear text."
grep -a "HIDDEN_IN_WAL_000" "${DATA_DIR}/pg_wal/${LATEST_WAL}" || echo " -> CLEAN: WAL is successfully encrypted."

# View the first few bytes of the WAL to see the "scrambled" nature
hexdump -C "${DATA_DIR}/pg_wal/${LATEST_WAL}" | head -n 20

Step 6: Initialize the Stanza and Run a Full Backup

With the archive command configured, we can now initialize the stanza and run our first backup.

bash
pgbackrest --stanza=demo stanza-create
pgbackrest --stanza=demo --type=full backup
pgbackrest --stanza=demo info

Step 7: Run Backup Integrity Tests

pgBackRest has a built-in verify command that checks the integrity of the files in the backup repo.

pgbackrest --stanza=demo verify

7.1 Verify Repository Encryption

We will now search the pgBackRest repository for the same “secret” strings we used earlier. Because we configured repo1-cipher-type=aes-256-cbc, these should be completely invisible to grep.

bash
# Define the repository path
REPO_DIR="/var/lib/pgbackrest/backup/demo"

echo "--- SCANNING BACKUP REPOSITORY ---"

# Search for the 'Flushed' secret
sudo grep -r -a "HIDDEN_ON_DISK_456" "${REPO_DIR}" || echo " -> CLEAN: Permanent data is encrypted in the repo."

# Search for the 'In-Flight' WAL secret
# (This is the critical test for the archive helper/re-encryption flow)
sudo grep -r -a "HIDDEN_IN_WAL_000" "${REPO_DIR}" || echo " -> CLEAN: WAL data is encrypted in the repo."

# To be 100% sure we are not just failing to find the strings because of a typo, check the file type of a backup manifest or data block. Pick a random file from the repository

TARGET_FILE=$(find ${REPO_DIR} -type f -name "*.bundle" -o -name "*.gz" | head -n 1)

# Run the 'file' command
sudo file -s "$TARGET_FILE"

Expected result: It should return data. If it returns PostgreSQL or ASCII text, encryption is not active.

Step 8: Restore (pg_tde-aware)

Restoring an encrypted cluster requires us to reverse the process. Since our backups are stored decrypted by pgBackRest, we use the pg_tde_restore_encrypt wrapper to re-encrypt the WAL files as they are written back to disk.

8.1 Stop the Service

sudo systemctl stop postgresql

8.2 Simulate Data Loss

The following command wipes the current data directory clean, ensuring we are restoring into a fresh environment.

find /var/lib/postgresql/18/main -mindepth 1 -delete

8.3 Restore from Backup

bash
pgbackrest --stanza=demo restore --recovery-option=restore_command='/usr/lib/postgresql/18/bin/pg_tde_restore_encrypt %f %p "pgbackrest --stanza=demo archive-get %%f %%p"'

8.4 Configure the Restore Command

We used --recovery-option in the restore command. This option writes the correct restore_command for this recovery run and keeps the configuration in one place.

pg_tde_restore_encrypt is the required wrapper for pg_tde WAL restore: pgBackRest reads WALs from the repository in plain form, and this tool re-encrypts them as PostgreSQL writes them back to disk so the restored cluster remains encrypted.

8.5 Start PostgreSQL

sudo systemctl start postgresql

Step 9: Run Verification Tests

After restoring and starting the PostgreSQL server successfully, verify that the data was restored properly and also make sure that encrypted data can be retrieved.

sql
-- Verify data integrity (sample rows)
SELECT * FROM clear_table LIMIT 5;
SELECT * FROM crypt_table LIMIT 5;

-- Verify TDE encryption status
-- For non-encrypted tables, this must return 'f' (false)
SELECT pg_tde_is_encrypted('clear_table');

-- For encrypted table, this must return 't' (true)
SELECT pg_tde_is_encrypted('crypt_table');

Wrap-up

Security often comes at the cost of operational complexity, but it doesn’t have to compromise recoverability. By pairing Percona’s solution for transparent data encryption (pg_tde) with pgBackRest, you can established a strategy that satisfies both security auditors and operations teams: your data is transparently encrypted on disk to meet strict compliance standards, while your backups remain consistent, verifiable, and easy to restore.

While this walkthrough used a local file provider for simplicity it is highly discouraged to do so for any production or otherwise serious use cases. For this particular scenario, the focus was supposed to be on backup, please let us know if some similar articles about Key Management System (KMS) configuration is what you would be interested in.

As you progress from this blog post to a production deployment, we recommend exploring a dedicated KMS solution to further harden your architecture against unauthorized access.

Finally, be aware that to support archiving, the pg_tde wrapper decrypts WAL files before sending them to the repository. This means your backup repository currently holds unencrypted data. To close this security gap in production, you must ensure that encryption is enabled at the backup repository level so that your backups remain just as secure as your live database.

Remember: While a backup is running, you should not change any WAL encryption settings, including:

Global key provider operations (creating or changing)
WAL encryption keys (creating or changing)
The pg_tde.wal_encrypt setting

The reason is that standbys or standalone clusters created from backups taken during these changes may fail to start during WAL replay and can also lead to corruption of encrypted data (tables, indexes, and other relations).

I Built an AI That Impersonates Me on Slack, and It Was Disturbingly Easy

Kai Wagner — Mon, 09 Mar 2026 10:00:00 UTC

I spend a lot of time in Slack. Most people in tech do. It’s where a lot of “work” happens such as quick questions, async decisions, the “hey can you look at this?” threads that never seem to end. It feels personal. You think you know who’s on the other end.

So, a few days back, I just asked myself, what would it actually take to have an AI respond to my DMs, pretending to be me?

Turns out: a few hours, some TypeScript, and a token already sitting on my machine.

The overall design idea

The bot polls your Direct Messages (DMs) in Slack silently in the background using your real desktop token, no Slack admin approval, no OAuth app setup, no review process needed. Getting that token is straightforward: Slack’s desktop app stores your session in browser local storage, so one DevTools command gives you API access equivalent to the app itself.

From there, each incoming DM is sent to the model with recent thread context plus a persona prompt built from your past conversations. The model returns structured output: reply text, emoji reaction, or silence. A lightweight rate limiter spaces requests to stay within free-tier constraints.

In practice, a few conversation samples are enough for the model to mirror tone, vocabulary, and punctuation style. It also handles attachments and image messages sensibly, not just raw metadata.

I wanted it to pretend it’s human, so I implemented constraints as to how it behaves. With these in place, if asked whether it’s an AI, it deflects with casual confusion. To ensure that the bot does not get into sensitive topics like salary or politics these get redirected to “let’s talk in person.”. The bot is also explicitly limited to DMs. Group channels are hard-blocked in code.

The first version worked in roughly two hours. The remaining time went into handling real-world rough edges such as rate limits, image handling, a 200-DM pagination ceiling, and Slack emoji-name validation.

flowchart LR
slack(["Slack"])
bot["Bot running locally"]
ai(["Claude / Ollama"])
persona[/"Your writing samples"/]
slack -->|"incoming DMs"| bot
persona --> bot
bot <-->|"generate reply in your voice"| ai
bot -->|"reply as you"| slack

The Uncomfortable Part

Here’s what stuck with me after building this.

Slack feels safe. It’s behind your company SSO. It’s where people share things they wouldn’t put in an email. With this bot excercise and realizing how easy this is, I felt I’ve broken something that felt secure. Was this even a morally correct thing to do overall? So far I felt safe, but now should I start to question messages I get on Slack the same as I do with some documents or links in emails? Overall I’m still undecided on how to think about the outcome of the experiment. While I’m excited, I’m also scared that I’ve broken something deeper.

What I built here is, if you strip out the friendly framing: a system that reads every DM to a user, replies under their name in their tone, actively deflects if you try to verify whether it’s human, and does all of this indefinitely and silently from a laptop running in the background. If I fed this bot with enough background information and history, I’m almost certain, it could go unnotice for quite a long time. So the moral delimma between curiosity and ethical boundaries and the urge to inform people about it is real.

I added ethical guardrails, but those are prompt instructions. They exist because I chose to write them. Someone building this without my good intent, simply wouldn’t have them. Yes, I hear you, this is getting a litte scary at times.

This conversation has happened, without me ever touching the keyboard….yes, Zsolt was aware!

“Easy” Is Relative, But Not By Much

Core functionality (polling DMs, calling the API, posting replies) was working in under two hours, as stated above. Why do I repeat myself? Because it’s scary…

The tooling: Bun, a modern TypeScript runtime that made setup trivial. Anthropic’s SDK, clean API, takes a system prompt and a conversation and returns structured JSON. Slack’s own API, well-documented and permissive with desktop tokens.

No specialised knowledge needed. Anyone motivated enough could reproduce this easily. Someone who does this professionally could build something considerably more capable, and that’s precisely where it gets more uncomfortable.

That’s how the CLI output looks like

slack-bot$ bun run bot
$ bun run src/index.ts
[slack-bot] Running | mode: allowlist | backend: claude | review: off
[slack-bot] My user ID: U03A3PZHK5X
[slack-bot] Mode: allowlist | Allowlist: U03QTQQHZFX, U83651WSX
[slack-bot] Polling every 15s...
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]
[slack-bot] New DM from U83651WSH — generating reply...
[slack-bot] Claude call — est. ~933 input tokens
[slack-bot] Claude tokens: 1049 in / 8 out
[slack-bot] Ignoring message from U83651WSX (AI chose no response)
[slack-bot] Handled message from U83651WSX
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]
[slack-bot] New DM from U83651WSX — generating reply...
[slack-bot] Claude call — est. ~944 input tokens
[slack-bot] Claude tokens: 1059 in / 23 out
[slack-bot] Handled message from U83651WSX
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]
[slack-bot] New DM from U83651WSX — generating reply...
[slack-bot] Claude call — est. ~976 input tokens
[slack-bot] Claude tokens: 1085 in / 36 out
[slack-bot] Handled message from U83651WSX
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]

That’s how the CLI helper and options look like

slack-bot$ bun run bot --help
$ bun run src/index.ts --help
Usage: slack-bot [options] [command]

Personal Slack bot that replies as you

Options:
 -V, --version output the version number
 --mode  Response mode: auto | away | allowlist | manual
 --review Enable review mode (approve before sending)
 --no-review Disable review mode
 --allow  Add user to allowlist (Slack user ID)
 --interval  Poll interval in seconds
 --backend  AI backend: claude | ollama
 --config  Path to config file (default: "config.json")
 -h, --help display help for command

Commands:
 context Manage active context
 check-user [options]  Check whether a user's DM channel is found and reachable

What It Looks Like Without the Constraints

What I built runs against Claude’s API with free-tier rate limits, small context window, a handful of persona examples, a throttle on message volume. Those constraints are real and also completely trivially removable.

You can run the same thing with a local model, Llama 3, Mistral, take your pick from the open-weight models available on consumer hardware today, and it changes significantly.

No rate limits. Every message gets answered immediately, without the 12-second pause between API calls. Response timing becomes indistinguishable from a fast typist.
No token budget. Instead of a few hundred tokens of context, you can feed it your entire Slack history. Months, years of it. Every thread, every in-joke, every project reference. The model doesn’t just match your writing style, it knows what you’ve been working on, what you said about the Q3 roadmap in October, what you think about your manager.
No API calls leaving your machine. Nothing logged externally. Invisible from a network perspective.

With a large enough context window (Llama 3.1 supports 128k tokens, roughly 100,000 words), the last few months fit. “Remember what we decided on Thursday?” doesn’t expose it anymore, because it actually has that conversation in its context.

Seeing articles like that make me wonder, how many people are out there already, doing exactly that as we speak…or do we?

A Few Things Worth Knowing

This isn’t a call to panic. But it’s probably worth stopping for a second and questioning more what is happening around is.

For anything that actually matters, financial, personal, strategic, verify out-of-band. A quick voice note or phone call costs almost nothing and resolves almost everything - at least until the video part also improves even further. I know people don’t like phone calls, especially in the developer ecosystem, but maybe we should reconsider this nowadays?

Unusual patterns are worth noticing. Response timing that’s too consistent. Answers that are slightly generic when you’d expect specific. Deflection where you’d expect directness. None of these are proof of anything individually, but they’re worth filing away.

The safe-space feeling Slack gives you is a product of habit, not architecture. Slack’s security model protects your data from outsiders. It doesn’t protect you from someone who has authenticated as themselves and is quietly running a process in the background. In the past this would be only a consideration for man-in-the middle attacks, nowadays it also may be a consideration for other cases as I have demonstrated.

Specific questions still help, for now. “Remind me what we decided on Thursday?” trips up a system with limited context. But that window is closing as context windows grow.

Why did I do it?

I built this to see if it was possible. It was, faster than I expected, with tools that are widely available. The version I built in an evening is convincing enough for routine exchanges. A version with local inference and full conversation history would be convincing for most exchanges, including ones where you’re actively looking for tells.

That gap between “afternoon project” and “genuinely hard to detect” is smaller than people assume and it’s shrinking. Better models, larger context windows, cheaper hardware, each of these individually makes impersonation easier; together they compound.

The signals we relied up to now to establish trust in digital communication: name, avatar, writing style, shared history, plausible timing. These signals are all reproducible now, with effort that ranges from an afternoon to a weekend depending on how convincing you want to be.

My grandma always used to say, that history repeats itself and you just have to wait long enough until “old” becomes “new and modern” again. Maybe simple things like code words is something to reconsider in this context, as a last chance to not get tricked.

So please be a little curious about who you’re actually talking to and maybe agree on a code word in case you’re in doubts. And every now and then, just call them…which reminds me, that this might be worth another evening project research ;-).

For the first time the source of a project of mine is not in my repository, as I still fight my inner fight with ethics.

PostgreSQL 18 OIDC Authentication with Ping Identity using pg_oidc_validator

Mohit Joshi — Wed, 04 Mar 2026 00:00:00 UTC

PostgreSQL 18 introduced native OAuth 2.0 authentication support, marking an important step towards modern, centralized identity-based access control. However, since every identity provider implements OpenID Connect (OIDC) slightly differently, PostgreSQL delegates token validation to external validator libraries. This is where Percona’s pg_oidc_validator extension comes in - it bridges PostgreSQL with any OIDC-compliant Identity Provider.

There are several identity and access management (IAM) solutions available today that enable Single Sign-On (SSO) using OAuth 2.0 and OpenID Connect. In an earlier blog by my colleague Zsolt, OIDC in PostgreSQL: With Keycloak, he demonstrated how PostgreSQL 18 can be integrated with Keycloak using pg_oidc_validator. In this post, we explore the same concept using Ping Identity (PingOne).

Ping Identity is widely used in enterprise environments for identity and access management. If you are in such an environment, integrating PostgreSQL directly with Ping Identity can provide access control.

This blog is intended for PostgreSQL users, DBAs and customers who want to evaluate or deploy OIDC authentication using pg_oidc_validator. The goal is to provide a practical step-by-step walk-through to get a working setup.

We will cover the following topics as part of this blog:

Setting up PingOne environment
Install PostgreSQL 18 from Packages
Configure PostgreSQL for OAuth/OIDC authentication
Test login using an OIDC flow

Setting up PingOne environment

Register a new account
Fill in the required details to complete the profile
The next step is to sign in to your Ping Identity account
Upon successful Sign-in, we will see Ping Identity Administrator Console. In the left navigation panel, click on Environments -> Environments + (marked in red).
Provide an environment name and click on Finish
Once the environment is created, click on Manage environment.
In the left navigation panel, click on Applications -> Applications -> click on Applications +. Fill the application name, select the application type as Device Authorization and click on Save.
Upon successful creation, we will see generated Client ID and Issuer ID. The Issuer ID can be copied from under the Connection Details section. Enable the toggle so that the application is Active.
Once the application is successfully created, we need to add a client scope. In the left navigation panel, click on Applications -> Resources -> OpenID Connect. You will see a section called Scopes under which there is a + Add Scope button.
Upon clicking the Add scope button, we need to fill the Scope name and click on Save. In our example, we are creating a scope called pgscope
Now, let’s assign the custom scope we created to our client application. In the left navigation panel, click on Applications -> Applications. Select the application postgres which we created previously and click on Resource Access
From the list of available scopes, select the custom scope we created and click on Save. This will assign the scope to our application.
The next step is to add a new user. In the left navigation panel, click on Directory -> Users and click on Users + sign.

Fill the username field. For our exercise, we are creating a user called employees. In some identity providers (IdPs), it is possible to customize tokens and control how certain claims (including the sub - subject claim) are generated or mapped. However, it is important to note that while Ping Identity allows customization of the ID token, the access token claims (including sub) for the default OpenID Connect resource cannot be customized. The value of sub in the access token is generated and managed internally by PingOne and cannot be altered, mapped, or derived from another attribute (such as email or username). As a result, PostgreSQL must be configured to work with the sub value exactly as issued in the access token by PingOne.
Enable the user by turning the toggle “ON” and set a password.

Install PostgreSQL 18 from Packages

Since OAuth support is only available starting with PostgreSQL 18, we need a PostgreSQL server of at least this version.In the guide, we will install PostgreSQL 18 using Percona’s official packages.

Note:

Ensure that the percona-release is already installed and configured on your system. You can refer to the official Percona documentation for setup instructions.
For this exercise, the steps are demonstrated on Ubuntu 24.04

Enable the PostgreSQL 18 repository

sudo percona-release enable-only ppg-18.2 release
sudo apt update

Install PostgreSQL 18

sudo apt install -y percona-postgresql-18

Install OAuth Support for libpq

sudo apt install libpq-oauth

Configure PostgreSQL for OAuth/OIDC authentication

Install pg_oidc_validator

sudo apt install percona-pg-oidc-validator18

Setting Up a Sample Use Case for OIDC-Based Access

Imagine the following use case:

A company regularly generates promotional discount codes and stores them in a table called dcode inside a database named promo
A new discount code is generated and added to this table every day.
The company wants all employees to be able to access the latest code whenever needed.
For simplicity in this demonstration, employees retrieve the code by connecting to the database and querying the table directly.
To avoid managing individual database accounts for every employee, access is not tied to separate user credentials.
Instead, authentication to the database is handled through the company’s SSO system, allowing employees to connect using their existing corporate identity.

---
config:
theme: neutral
---
architecture-beta
group company_network(cloud)[Company Network]
service employee(user)[Employee] in company_network
service sso(server)[Company SSO PingIdentity] in company_network
service promo_db(database)[Promo Database] in company_network
service code_gen(server)[Daily Code Generator] in company_network
code_gen:B --> T:promo_db
employee:R --> L:sso
sso:R --> L:promo_db

Let’s connect to PostgreSQL:

sudo -u postgres psql

Create a database:

sql
CREATE DATABASE promo;
\c promo

Add a table to store discount codes:

sql
CREATE TABLE dcode (code varchar(10), GENERATED_AT TIMESTAMP default now());
INSERT INTO dcode (code) VALUES ('SAVENOW');

Create the access user:

sql
CREATE ROLE employees WITH LOGIN;
GRANT SELECT ON dcode TO employees;

Configure OAuth access in PostgreSQL:

In order for users to connect using OAuth and authenticate through the Ping Identity server, we need to create an identity map and add an entry for such access on PostgreSQL’s authentication configuration file.

Edit the identity mapping configuration file and add an entry, which we will call oidc, mapping connections originated by a system user identified with a sub ID. For this exercise, we allow any string to match

bash
sudo vim /etc/postgresql/18/main/pg_ident.conf

# MAPNAME SYSTEM-USERNAME DATABASE-USERNAME
oidc /^(.*)$ employees

Next, edit the authentication configuration file. The configuration file pg_hba.conf acts as a sort of firewall for connections. With the below line, we are instructing PostgreSQL to allow all connections coming from any network that attempt to access the database promo as user employees using the new authentication method oauth. We are also indicating that the authentication provider is Ping Identity and the scope is openid.

bash
sudo vim /etc/postgresql/18/main/pg_hba.conf

# TYPE DATABASE USER ADDRESS METHOD
host promo employees 0.0.0.0/0 oauth issuer=https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/as scope=pgscope map=oidc

Note:

Place this entry after the existing local rules and just before the replication rules in pg_hba.conf. PostgreSQL evaluates pg_hba.conf from top to bottom, and the first matching rule is applied. Putting the OAuth rule earlier ensures that connections to database promo as user employees use OAuth instead of falling back to password authentication.
In production environments, restrict the IP range instead of using 0.0.0.0/0
The oauth_issuer must exactly match the Issuer ID from your PingOne environment. The Issuer URL is unique to each PingOne environment and contains your environment UUID. It typically follows this format: https://auth.pingone.com.au//as. Replace with the actual value from your PingOne environment. Do not copy the placeholder value directly.

Enabling the pg_oidc_validation extension

Edit the postgresql.conf and add below lines

bash
sudo vim /etc/postgresql/18/main/postgresql.conf

oauth_validator_libraries = 'pg_oidc_validator'

The validator uses the sub claim from the access token by default. Hence, we need not explicitly set pg_oidc_validator.authn_field=sub. The sub claim is defined by the OpenID Connect specification as a stable and unique identifier for a user within an identity provider (IdP). It is intended to uniquely represent a user and remain consistent across authentication sessions.

PostgreSQL does not interpret or transform this value. The validator extracts the configured claim and PostgreSQL compares it against a database role or an entry in pg_ident.conf. If the value does not match the expected role or mapping, authentication will fail, even if the token itself is valid.

In this setup with Ping Identity, only the sub claim can be used for authentication with the default OpenID Connect resource.

Reload the configuration

bash
sudo -u postgres psql
SELECT pg_reload_conf();

Monitor the server logs

sudo tail -f /var/log/postgresql/postgresql-18-main.log

Connecting to the database:

For this quick connection test, we use psql to connect to the promo database as the employees user, explicitly specifying the host IP along with the oauth_issuer and client_id.

psql 'host=127.0.0.1 user=employees dbname=promo oauth_issuer=https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/as oauth_client_id=1e892f71-09d7-4ed6-a534-0dc888d39c7c'

By connecting via the host IP address rather than the local socket, PostgreSQL treats this as a host-based connection, ensuring that the OAuth configuration in pg_hba.conf is applied. The authentication is handled by PostgreSQL’s authentication framework, which uses OIDC with Ping Identity as the identity provider to validate the token.

We will see a prompt on the console with the URL and activation code.You will notice an activation code XXXX-XXX. Example shown below:

Visit https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/device and enter the code: 7KK7-88DK

Upon clicking the URL, it will prompt you to log in with the employee's user, which we created during the PingOne environment setup.

Next, it will prompt you to enter the activation code.

Approve access for the application, and that’s it! The user has now been successfully authenticated via OIDC.

Return to the PostgreSQL prompt and you should see that the login to the promo database is successful. You can now query the dcode table to fetch the discount code.

bash
psql 'host=127.0.0.1 user=employees dbname=promo oauth_issuer=https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/as oauth_client_id=1e892f71-09d7-4ed6-a534-0dc888d39c7c'
Visit https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/device and enter the code: 7KK7-88DK
psql (18.2 - Percona Server for PostgreSQL 18.2.1)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: postgresql)
Type "help" for help.

promo=> select * from dcode;
 code | generated_at
---------+----------------------------
 SAVENOW | 2026-02-13 09:40:13.109801
(1 row)

With the steps shown in this guide, we now have a working end-to-end setup using OIDC authentication and device flow login. From here, the same model can be extended to real-world enterprise environments with tighter network restrictions and role mapping.

If you run into issues while setting up pg_oidc_validator or integrating PostgreSQL with Ping Identity, check the community forums first, chances are someone in the community may already have encountered a similar issue. If not, feel free to open a discussion or raise a request for help.

Hardening MySQL: Practical Security Strategies for DBAs

Wayne Leutwyler — Mon, 02 Mar 2026 00:00:00 UTC

MySQL Security Best Practices: A Practical Guide for Locking Down Your Database

Introduction

MySQL runs just about everywhere. I’ve seen it behind small personal projects, internal tools, SaaS platforms, and large enterprise systems handling serious transaction volume. When your database sits at the center of everything, it becomes part of your security perimeter whether you planned it that way or not. And that makes it a target.

Securing MySQL isn’t about flipping one magical setting and calling it done. It’s about layers. Tight access control. Encrypted connections. Clear visibility into what’s happening on the server. And operational discipline that doesn’t drift over time.

In this guide, I’m going to walk through practical MySQL security best practices that you can apply right away. These are the kinds of checks and hardening steps that reduce real risk in real environments, and help build a database platform that stays resilient under pressure.

1. Principle of Least Privilege

One of the most common security mistakes is over-granting privileges. Applications and users should have only the permissions they absolutely need.

Bad Practice

sql
GRANT ALL PRIVILEGES ON *.* TO 'appuser'@'10.%';

Better Approach

sql
GRANT SELECT, INSERT, UPDATE ON appdb.* TO 'appuser'@'10.%';

Recommendations

Avoid global privileges unless absolutely required
Restrict users by host whenever possible
Separate admin accounts from application accounts
Use different credentials for read-only vs write operations

Audit Existing Privileges

sql
SELECT user, host, Select_priv, Insert_priv, Update_priv, Delete_priv
FROM mysql.user;

2. Strong Authentication & Password Policies

Weak credentials remain one of the easiest attack vectors.

Enable Password Validation

component_validate_password is MySQL’s modern password policy engine. Think of it as a gatekeeper for credential quality. Every time someone tries to set or change a password, it checks whether that password meets your defined security standards before letting it in.

It replaces the older validate_password plugin with a component-based architecture that is more flexible and better aligned with MySQL 8.x design.

sql
INSTALL COMPONENT 'file://component_validate_password';

What It Does

When enabled, it enforces rules such as:

Minimum password length
Required mix of character types
Dictionary file checks
Strength scoring

If a password fails policy, the statement is rejected before the credential is stored.

Why It Matters

Weak passwords remain one of the most common entry points in database breaches. This component reduces risk by enforcing baseline credential hygiene automatically, instead of relying on developer discipline.

Recommended Policies

Minimum length: 14+ characters
Require mixed case, numbers, and symbols
Enable dictionary checks
Enable username checks

Remove Anonymous Accounts

Find Anonymous Users

Anonymous users have an empty User field.

sql
SELECT user, host FROM mysql.user WHERE user='';

If you see rows returned, those are anonymous accounts.

Drop Anonymous Users

In modern MySQL versions:

sql
DROP USER ''@'localhost';
DROP USER ''@'%';

Adjust the Host value based on what your query returned.

Why This Matters

Anonymous users:

Allow login without credentials
May have default privileges in some distributions
Increase the attack surface unnecessarily

In hardened environments, there should be zero accounts with an empty username. Every identity should be explicit, accountable, and least-privileged.

3. Encryption Everywhere

Encryption protects data both in transit and at rest.

Enable Transparent Data Encryption (TDE)

See my January 13 post for a deep dive into Transparent Data Encryption: Configuring the Component Keyring in Percona Server and PXC 8.4

Enable TLS for Connections

sql
require_secure_transport=ON

Verify SSL Usage

sql
SHOW STATUS LIKE 'Ssl_cipher';

Encryption Areas to Consider

Client-server connections
Replication channels
Backups and snapshot storage
Disk-level encryption

4. Patch Management & Version Hygiene

Running outdated MySQL versions is equivalent to leaving known vulnerabilities exposed.

Maintenance Strategy

Track vendor security advisories
Apply minor updates regularly
Test patches in staging before production rollout
Avoid unsupported MySQL versions

Check Version

sql
SELECT VERSION();

5. Logging, Auditing, and Monitoring

Security without visibility is blind defense, enable Audit Logging.

1. audit_log Plugin (Legacy Model)

Installation

sql
INSTALL PLUGIN audit_log SONAME 'audit_log.so';

Verify

sql
SHOW PLUGINS LIKE 'audit%';

2. audit_log_filter Component (Modern Model)

Introduced in MySQL 8 to provide a more flexible and granular alternative to the older plugin model.

Installation

sql
INSTALL COMPONENT 'file://component_audit_log_filter';

Verify

sql
SELECT * FROM mysql.component;

Architecture Difference

Instead of a single global policy, you create:

Filters (define what to log)
Users assigned to filters

It’s granular and rule-driven.

Auditing Key Events

Failed logins
Privilege changes
Schema modifications
Unusual query activity

References:

Useful Metrics

sql
SHOW GLOBAL STATUS LIKE 'Aborted_connects';
SHOW GLOBAL STATUS LIKE 'Connections';

6. Secure Configuration Hardening

A secure baseline configuration reduces risk from common attack patterns.

Recommended Settings

ini
local_infile=OFF
secure_file_priv=/var/lib/mysql-files
sql_mode="STRICT_ALL_TABLES"
secure-log-path=/var/log/mysql

Why These Matter

Prevent arbitrary file imports
Reduce filesystem abuse
Restrict data export/import locations

7. Backup Security

Backups often contain everything an attacker wants.

Backup Best Practices

Encrypt backups
Restrict filesystem permissions
Store offsite copies securely
Rotate backup credentials
Verify restore procedures regularly

Example Permission Check

ls -l /backup/mysql

8. Replication & Cluster Security

Replication is not just a data distribution feature. It is a persistent, privileged communication channel between servers. If misconfigured, it can become a lateral movement pathway inside your infrastructure. Treat every replication link as a trusted but tightly controlled corridor.

Principle: Replication Is a Privileged Service Account

Replication users require elevated capabilities. They must be isolated, tightly scoped, and monitored like any other service identity.

Secure Replication Users

sql
CREATE USER 'repl'@'10.%'
 IDENTIFIED BY 'strongpassword'
 REQUIRE SSL;

GRANT REPLICATION REPLICA ON *.* TO 'repl'@'10.%';

Hardening considerations:

Restrict host patterns as narrowly as possible. Avoid % whenever feasible.
Require SSL or X.509 certificate authentication.
Enforce strong password policies or use a secrets manager.
Disable interactive login capability if applicable.

Encrypt Replication Traffic

Replication traffic may include sensitive row data, DDL statements, and metadata. Always encrypt it.

At minimum:

Enable require_secure_transport=ON
Configure TLS certificates on source and replica
Set replication channel to use SSL:

sql
CHANGE REPLICATION SOURCE TO
 SOURCE_SSL=1,
 SOURCE_SSL_CA='/path/ca.pem',
 SOURCE_SSL_CERT='/path/client-cert.pem',
 SOURCE_SSL_KEY='/path/client-key.pem';

For MySQL Group Replication or InnoDB Cluster:

Enable group communication SSL
Validate certificate identity
Use dedicated replication networks

Binary Log and Relay Log Protection

Replication relies on binary logs. Protect them.

Set binlog_encryption=ON
Set relay_log_info_repository=TABLE
Restrict filesystem access to log directories
Monitor log retention policies

Compromised binary logs can reveal historical data changes.

9. Continuous Security Reviews

Security is not a one-time checklist. Regular audits help catch configuration drift and evolving threats.

Suggested Review Cadence

Weekly: failed login review
Monthly: privilege audits
Quarterly: configuration review
Semiannually: full security assessment

Security Checklist Summary

Area	Key Action
Access Control	Least privilege grants
Authentication	Strong password policies
Encryption	TLS + encrypted storage
Updates	Regular patching
Monitoring	Audit logging enabled
Configuration	Harden defaults
Backups	Encrypt and protect
Replication	Secure replication users

Final Thoughts

Strong MySQL security doesn’t come from one feature or one tool. It comes from layers working together. Hardened configuration. Tight, intentional privilege design. Encryption everywhere it makes sense. And monitoring that actually gets reviewed instead of just written to disk.

In my experience, the strongest environments aren’t the ones trying to be unbreakable. They’re the ones built to detect, contain, and respond. Every layer should either reduce blast radius or increase visibility. If an attacker gets through one control, the next one slows them down. And while they’re slowing down, your logging and monitoring should already be telling you something isn’t right.

That’s what a mature security posture looks like in practice.

A reponsible role for AI in Open Source projects?

Alastair Turner — Thu, 26 Feb 2026 14:00:00 UTC

AI-driven pressure on open source maintainers, reviewers and, even, contributors, has been very much in the news lately. Nobody needs another set of edited highlights on the theme from me. For a Postgres-specific view, and insight on how low quality AI outputs affect contributors, Tomas Vondra published a great post on his blog recently, which referenced an interesting talk by Robert Haas at PGConf.dev in Montreal last year. I won’t rehash the content here, they’re both quite quick reads and well worth the time.

The key points which got me thinking about the rest of this piece are:

Low quality AI reviews are also a time sink, not just AI slop patches
AI is best at code review, so that happens soonest, but conceptual review should happen first

Today also marks the release of an out-of-cycle of Postgres. The release is happening because a number of security fixes which were introduced in the releases two weeks ago were found to cause functional and performance regressions. On the upside, this shows that the longer, public patch review process does find issues which otherwise go unnoticed. On the whole, fewer out-of-cycle releases, with the friction they create for the maintainers and users would, however, be better.

The link between these two tracks of thought is review. Reviewing the security patches included in any Postgres release cycle is clearly different. What if it’s different in a way which suited AI assistance?

We know that AI coding agents can contribute to this process, because one just has. Unfortunately, a little bit late. Some of the issues fixed in today’s out-of-cycle release were identified when a contributor opened up his agentic AI coding assistant one morning and asked it to tell him about the new patches. Some of those patches were newly visible to most contributors, after being pushed to the Git source control system for Postgres, because they were security patches which had not been discussed on the public mailing lists. You may hear the issues which drove these fixes referred to by the initialism for the Common Vulnerabilities and Exposures program which records these security reports (CVEs).

It should go without saying that this is not just a case of: download Claude Code or Gemini Code Assist, install, add value and attain glory. There is definitely work involved in getting value out of these tools: Setting up a project specific context is important, particularly for a long standing project like Postgres, which has some long standing coding styles Asking the right question, known in the AI-assisted paradigm as prompting, will also require quite some understanding of the problem domain Human judgement on the quality of the output. At its most basic, this is a decision about whether to turn the coding assistant’s output into a review, bin the output, or investigate further.

The first two of those, the purely technical ones, are very unlikely to happen on the first attempt to use agentic AI as a coding assistant. This is a new class of tool everyone needs to learn. Perversely, people who have more of that third quality, that judgement, are more likely to walk away from their early experiments because those experiments are not delivering value. People skipping the third area of work is how we have ended up with AI being regarded as a burden on open source projects. This isn’t new, long before we got Large Language Models involved in code analysis, there were deterministic static analysis tools, and their false positives have been heaping noise on the signal in patch review processes for years.

In his presentation which I linked above, Robert lists four key, high level principles for patch review:

Start With The Big Picture
Consistency Is Critical
Be a Pessimist
Apply a Spirit of Maintainership

The middle two of those principles, consistency and pessimism, are the two most amenable to AI interventions, and the two most important for reviewing security patches.

When a piece of code has been committed, the big picture decisions and the decisions in the spirit of maintainership have already been made. If the code is later found to expose Postgres users to security issues, reviewing the solution comes down to all the possible angles of pessimism about the new code’s possible impact. It is in this area where Claude Code not just identified a regression in the case above, but also created a small reproducer to trigger the regression.

Crafting and reviewing security patches is a beast of a task. The leading experts on all matters Postgres are writing and reviewing these patches, but the group is small, because any information about potential exploits needs to be kept under tight control. Timelines for a fix might also be quite tight. Both factors limit opportunities for collaboration and multi-stage review. In these conditions, an extra pair of AI eyes (A-Eyes?) on the patches may just be very useful.

Set up, prompted, and reviewed by those who really know the problem domain, AI coding agents could provide valuable pointers on issues and regressions to consider when reviewing security patches. In that small bubble, this would not be a source of empty workload, but a source of extra review without having to coordinate more people in a small team. AI assistance will probably not provide time savings or quality gains on the first security patch it’s used to review, maybe not even on the seventh. After some setup effort, and a human learning a bit about improving its setup, an Agentic AI assistant (or two, with different configuration on different providers) may well be able to avoid another out-of-cycle release. That would be a major win.

Meet Percona at KubeCon + CloudNativeCon Europe 2026

Ege Güneş — Wed, 25 Feb 2026 12:00:00 UTC

The Percona team is heading to KubeCon + CloudNativeCon Europe in Amsterdam, and we’d love to meet you in person!

You can find us at Booth 790. This is a great chance to talk with engineers working on Percona Operators.

We will be there to discuss:

Running MySQL, PostgreSQL, and MongoDB on Kubernetes
Production-ready HA setups
Backup and PITR strategies
Multi-cluster and multi-region deployments
Operators roadmap and upcoming features
Real-world troubleshooting stories

If you’re running Percona Operators in production (or just getting started), we’d love to hear your feedback and learn about your challenges.

If you’re just curious (or even suspicious) about running databases on Kubernetes, we’d love to talk and answer your questions.

Admission Tickets - 20% Off for Our Community

We have a 20% discount code available for Percona community members.
If you’re planning to attend and don’t have a ticket yet, drop a comment or message us and we’ll share the details.

Schedule a Meeting

Want dedicated time with our engineers?
Drop a comment here or reach out directly. We’re happy to schedule a meeting during the event.

See you in Amsterdam!

PostgreSQL coffee break: version upgrade related reindexing - reasons

Jan Wieremjewicz — Wed, 25 Feb 2026 11:00:00 UTC

During FOSDEM I had a chance to join a presentation by Alexander Sosna of GitLab on the backup procedure he has followed with his team to decrease the downtime during major upgrades. I highly recommend the talk, definitely worth watching!

I loved the presentation, especially since a similar procedure is what we recommend our users as well. Unfortunately it’s not for everyone: it’s applicable ONLY when you can stop DDL operations on your database cluster for some time. As you can imagine that’s not a case for every deployment.

This limitation got me into some very engaging discussions during my fav part of any conference, the so called “hallway track”. Networking and discussions after the talks and on the conference corridors are why I like going to such events. I’ve heard some stories of nasty surprises coming out of the unexpected re-indexing after an upgrade.

It made me wonder, how many professionals have moved to PostgreSQL from other areas of expertise and are lacking information about the index rebuilds after upgrades may be necessary. How often are DevOps engineers or SREs, neither fluent in PostgreSQL nor experienced with databases, tasked with maintaining database infrastructure?

In the meantime I figured out that a short coffee time read is what I want to aim at. No super deep dives, rather food for thought and inspiring more of those engaging discussions I like so much about the conferences.

So let’s get to it. This week I want to go through some basic facts before diving into more challenging topics in the coming weeks.

What are collations

In general, a collation defines how values are ordered and compared. In databases, it most commonly applies to text. It’s often not a trivial thing to determine how strings are sorted, whether two values are considered equal, and how things like upper vs lower case or special characters are treated. Just like alphabetical order helps us organize words in everyday life, collations define the rules the database uses when comparing and sorting character data. This allows us to sort structures like strings. Numbers are even simpler.

A good visualization of how significant collations are is when you try to imagine determining whether one item is before another if they come from different alphabets. Where do you position country specific characters in such a sorting order?

Lets look at a set of example data:

Bob
Anna
Zoë
Álvaro

When using an English-like collation the sorting would look like this as Á is treated like A . This is visible in Collation 1:

Anna
Álvaro
Bob
Zoë

Though with a collation changed so that Á is sorted separately before A . This is visible in Collation 2:

Álvaro
Anna
Bob
Zoë

It’s clearly visible that collation defines how text is sorted and compared.

What are indexes

If you’re not a database professional, you may not be familiar with what an index is. Think of it as of a structure that speeds up the search. In an old school library you had to check the index of authors to find the book you’ve been looking for. Similarly in databases when knowing what the data is going to be searched for, we introduce indexes.

A common use of indexes is to enforce uniqueness of values. Primary keys and unique constraints automatically create unique indexes to ensure that no duplicate values are inserted. In addition, users can create their own indexes to support specific query patterns. Indexes can be single or multi column, depending on the particular use cases and to help speed up specific queries. Let’s look at an example of physically unsorted data stored in heap

Users (unsorted data)
--------------------
[1] Zoë
[2] Álvaro
[3] Anna
[4] Bob

with an Index on name

Index (B-tree on name)
----------------------
Anna -> [3]
Bob -> [4]
Zoë -> [1]
Álvaro -> [2]

The index stores values in sorted order and points to their location in the table. Scanning through the list in order is fine for small number of items, as indexes get larger, there are opportunities to optimize this process. The most commonly used optimisation, very simplified, is to split the list and create a pointer which stores the maximum value in the left half of the list and the minimum value in the right half. As the lists get too large again, they are split again, creating a tree of these pointers.

Why re-indexing is needed

If the rules for comparing the strings are different at query time from what they were when the index was created, the search may fail in various ways. If the scan encounters a value which, by the rules at query time, is higher in the sort order than the value being searched for, it will conclude that the value being searched for is not in the list. Similarly, the search may follow a pointer to the wrong list of values. To make this more concrete, let’s look at a real world example from a glibc change in 2019. Before the change, a list of our values was sorted as follows

aa
a a
a-a
a+a

after the change, the list was sorted as follows:

a a
a+a
a-a
aa

If an index was built under the first set of rules and searched under the second, a search for the value ‘a a’ will fail, because the first item in the index (‘aa’) is higher in the sort order than the value being searched for. Failing to find a value which is in the list can cause issues with application behavior - like missing records or records which appear in some queries (when the index is not used) but not in others (where the index is used). It may allow the insertion of values that should now be considered equal under the new collation rules, effectively violating uniqueness expectations.

Since PostgreSQL 10, the server tracks the collation version used when an index was built and can detect when it no longer matches the system’s current collation version, which is why reindexing may suddenly become mandatory after an upgrade.

However, PostgreSQL server does not analyze your actual data to determine whether the collation change affects your stored values. It only detects that the collation provider version has changed. In some cases, this means reindexing is required even though the effective sort order of your specific data has not changed. Because PostgreSQL cannot reliably determine whether your specific dataset is affected, it treats the situation as potentially unsafe.

When do collations change?

There’s a number of scenarios when collations change:

OS / glibc upgrade (Linux) - PostgreSQL relies on the system glibc provided collations so if a version changes, the collation rules may change as well. The example above is taken from the upgrade to glibc 2.28 which is one PostgreSQL Community remembers due to the ripple effect it caused.
ICU library upgrade (for ICU collations) - if PostgreSQL deployment uses ICU collations, upgrading ICU library will affect these. While they are not tied to glibc these changes also happen.
Major PostgreSQL upgrade (in some cases) - if the new version uses a different collation provider behavior or updated ICU integration
Database restored on a system with different collation versions - logical dump/restore onto a host with different glibc / ICU versions can invalidate indexes.

When re-indexing is necessary

Looking at the above provided list an observation is quite immediate, that not every collation type is affected:

if the collation is not dependent on glibc / ICU then it won’t be affected. As such C/POSIX collations are immune to such issues.
Same truth sticks to the data types. The collation change problem affects only those indexes which are the text or character based. All other datatypes like all types of integers and floating points, date, timestamp, geometric data types or even vector data remain unaffected.

What’s also important is that even if a collation changed it’s not necessary it will affect a given index. Think of it this way, if your database does not use any language specific characters, chances are that collation change will not require re-index. Unfortunately you don’t know that until you look inside your data. In controlled environments, teams may assess the risk before scheduling reindexing, but from PostgreSQL’s perspective the index must be considered potentially inconsistent until rebuilt.

What’s next?

Next week we’ll look at what is the reality of the DBA team regarding upgrades

PostgreSQL minor release postponed in Q1’ 2026

Jan Wieremjewicz — Wed, 18 Feb 2026 11:00:00 UTC

In case you are awaiting the February PostgreSQL Community minor update released on plan on February 12 we want to make sure that our users and customers are up to date and aware of what to expect.

This scheduled PostgreSQL release was delivered by the PostgreSQL Community on time and came carrying 5 CVE fixes and over 65 bugs bug fixes.

Unfortunately shortly after, the release team announced that an additional out of cycle release is planned for February 26. This follow up release addresses two regressions identified in the February 12 update.

Because of this, we have decided not to ship a Percona Distribution for PostgreSQL build based on the February 12 release. Instead, we will wait for the February 26 Community update and base our release on that version once it becomes available from PGDG. This means also a delay in the release of Percona Operator for PostgreSQL that uses images based on our PostgreSQL releases.

Always look on the bright side

While this is a delay in release it comes with some benefits. For our users and customers, this means a cleaner upgrade path. Rather than releasing February 12 now and asking you to update again shortly after, we prefer to wait and deliver a single update that includes the fixes. Our goal is to make updates predictable and smooth for users of Percona Distribution for PostgreSQL, as well as extensions such as pg_tde and pg_stat_monitor. It should also allow you to carry less operational burden with the added maintenance that an extra update would require.

We appreciate how quickly the PostgreSQL Community identified and addressed the regressions. Open collaboration across the ecosystem, including reports and testing from many contributors, helps ensure PostgreSQL continues to improve for everyone.

Path forward

This is the third out of cycle release in the past year, following similar updates in November 2024 and February 2025. It highlights how responsive and diligent the PostgreSQL Community is when issues are identified. At the same time, it reminds us all how important continuous testing and collaboration are as PostgreSQL adoption continues to grow. Contributing to PostgreSQL, whether through testing, reporting, or development, is one of the best ways to help strengthen quality across the ecosystem.

Elephants keep ears open

As soon as the February 26 release is available and our builds are ready, we will share the update.

If you need to move forward with the February 12 version in the meantime, please reach out. We are happy to talk through your situation and help you assess what makes the most sense for your environment.

Our customers can contact us through Percona Support Services to receive the high quality assistance we are known for. We also encourage community users to reach out via the Percona Community Forums, where we will do our best to provide guidance based on the information you share.

Thank you for your trust and for being part of the Percona community.

Pre-FOSDEM & FOSDEM 2026, Community, Databases, and Open Source

Edith Puclla — Mon, 09 Feb 2026 10:00:00 UTC

This is a recap of Percona at preFosdem and Fosdem!

Before FOSDEM officially started, the database community gathered for MySQL Belgium Days (Pre-FOSDEM), a two-day event bringing together MySQL developers, DBAs, engineers, tool builders, and open-source enthusiasts. It was an excellent space for deep technical discussions, knowledge sharing, and reconnecting with the community, hosted by the amazing Frederic Descamps. The event featured strong participation from Percona and the wider MySQL ecosystem, with talks led by Peter Zaitsev, Marco Tusa, Fernando Laudares Camargos, Arunjith Aravindan, Vinicius Grippa, Pep Pla, and Yura Sorokin.

Find the recordings of the talks here.

Also, in Belgium same week, several other events took place, including PGDay-FOSDEM, MariaDB Day, and the MySQL Summit.

At PGDay, it was a pleasure to see the PostgreSQL community together; we had several participants representing us.

MariaDB Day. We had Peter Zeitsev presenting a talk titled “What MariaDB Community can learn from PostgreSQL?”

MySQL summit

During MySQL Days in Brussels, the community gathered for an in-person MySQL Summit focused on collaboration and strengthening the MySQL ecosystem, with open discussions around its present and future driven by community involvement.

MySQL RockStars 2026

The MySQL Rockstar Award is a recognition given by the MySQL Community Team at Oracle, together with previous award winners, to members of the MySQL community who have actively contributed to promoting MySQL during the past year. MySQL Legends are long-standing community members who have made a significant and lasting impact on the adoption, development, and evolution of MySQL over many years.

This year, the MySQL RockStars selected were:

Matthias Crauwels
Marco Tusa
Umesh Shastry
Ronald Bradford
Marcelo Altmann

Congratulations to all of them! You can find previous MySQL RockStars in this list

FOSDEM 2026, Percona booth

At the Percona booth, conversations focused on open-source databases and Kubernetes, including OpenEverest’s first-ever presence at FOSDEM. Around 40 Perconians were present, a great chance to finally meet many colleagues in person. As always, we had many visitors, and it was great to see that many already knew about Percona, while others were eager to learn more and explore what we do in the world of Open Source!

FOSDEM 2026, Databases DevRoom

FOSDEM 2026 officially kicked off at the ULB Solbosch Campus, bringing together thousands of open-source contributors from around the world. The Database DevRoom (UB2.252A) was packed with high-quality talks and discussions, co-led with Matthias Crauwels and Ray Paik, and

Find more details and some of the recordings here.

Celebrating 20 Years of Percona

This year, Percona celebrates its 20th anniversary. Throughout FOSDEM and Pre-FOSDEM events, it was inspiring to meet long-time users who have relied on Percona’s open-source solutions for years and shared their positive experiences. You can explore Percona’s 20-year journey here: 👉 https://percona20.com/ If you’ve had a great experience with Percona, you’re invited to share your story via the community survey.

See you at FOSDEM 2027!

PGDay and FOSDEM Report from Kai

Kai Wagner — Wed, 04 Feb 2026 10:00:00 UTC

The following thoughts and comments are completely my personal opinion and do not reflect my employers thoughts or beliefs. If you don’t like anything in this post, reach out to me directly, so I can ignore it ;-).

I’m currently on the train on my way back home from FOSDEM this year and man, I’m exhausted but also happy. Why? Because the PG and FOSDEM community is just crazily awesome. While it’s always too much of everything, it’s at the same time inspiring to see so many enthusiastic IT nerds in one place, discussing and working on what they love - technology and engineering challenges.

PGDay FOSDEM

It all started with the usual PGDay FOSDEM the day before FOSDEM. Just in case - this has been happening for over 15 years and if you read this as a little blame that you didn’t know about it, that’s absolutely correct, as you should. It’s been a great event as usual: around 150 Postgres enthusiasts collaborating with each other. There was a great set of talks (no recording available, so yes, just join next year to not miss anything), as well as the hallway track conversations.

I was able and accepted again as a volunteer helping to make the event happen. While you might think, what’s special about it, I cannot express my gratitude for being able to help in any way. I simply love it. I’m not a great coder and I’ve never been one. I’m the one that looks at his code from a year ago and questions his technical existence and overall abilities if I should rather do something without touching a keyboard. What I am very well capable of is helping and supporting events. So it was my pleasure and I hope you do feel inspired to do the same next year or at any future event, not only in the Postgres ecosystem but in general. I strongly believe in this: doing good things will get you good things back.

After the wrap up to the PGDay and a great community dinner to collaborate and discuss further, I simply fell completely tired asleep, as the next day and FOSDEM was already waiting.

FOSDEM Day 1

The next day started with volunteering at the Postgres booth. As usual, Saturday was simply crazy. The Postgres swag like hoodies, caps, mugs, shirts, etc. was almost ripped out of our living hands. We had people waiting in line just to be able to get some swag. That fact alone shows how Postgres is viewed outside of the internal PG ecosystem community. How many times I heard the sentence “Thanks a lot for the great work you do” or “Postgres just works.” Yeah, we can all argue about the details and scenarios, but what this is about is the overall ease of use. Not everyone has terabytes of data or the most complex HA and replication scenarios on this planet. Some just need a functional and boring database and, in the best case, open source - and we all know, looking at real open source, not single-vendor owned, Postgres is the king and here to stay.

After all of this, I switched clothes and helped at the Percona booth. This wasn’t any less interesting in comparison to the PG booth. How many people stepped by, asking about what we do or thanking us for our projects and that we remained open source even after all these years and so many other companies not withstanding the quick and easy money to go with open-core or closed offerings. That’s the reason I’m proud to be part of this company. We walk the talk, since 20y and we have no incentive ever changing it. Thanks to Peter Zaitsev and Peter Farkas aka P² - for those who know, just know.

Following that I had the pleassure of being the Slonik guide again. What is a Slonik guide you might ask? Slonik, the mascot of Postgres (big blue elephant), needs some help and guidance while walking throught the crowd, as you can barely see anything while inside the costome. As usual, Slonik is a celebraty. Everyone wants a picture and taking their chance to photograph Slonik in the “wild”. As you can see, even MySQL’s Sakila couldn’t resist and had to take a picture with Slonik.

If you’re wondering, like many others, why Slonik and why an Elephant? Click here for some nice written down history lesson

After an exciting but also energy-draining day, I enjoyed a Percona crew/team dinner at BrewDog, with some great conversations and good food. Fun Fact: Did you know that BrewDog is also open source?. I couldn’t stay too long - sorry about that - but I had another date. The famous Floor Drees organized in tradition another karaoke event that I couldn’t miss. As I couldn’t make it to earlier versions of it, I definitely wanted to join. What should I say apart from thanks, Floor, for this great tradition. Yes, I had a hard time talking the next day, but damn I had fun singing Swedish, Polish, German, and English songs - and yes, I most likely misunderstood all of them as usual.

Too many songs for my voice and maybe a “soft drink or two” later, I felt in my bed like a stone, and couldn’t really accept the fact that my alarm clock went off almost five minutes later (at least that’s how it felt to me).

FOSDEM Day 2

No whining helped, just getting up and making myself ready for Day 2 of FOSDEM, which started with another round of volunteering at the Postgres and Percona booths. Both basically matched the previous feedback, apart from a definitely dropped and less crowded space - seems I wasn’t the only one singing last night ;-).

With that, thanks a lot to everyone making this great FOSDEM happen. I’ll try now if the Deutsche Bahn restaurant actually works this time, as I need coffee, a big one, maybe two… See all of you next year again or at another event this year.

Stay on top of Postgres development without the inbox overwhelm. Explore hackorum.dev today and share your feedback with us.

Hackorum - A Forum-Style View of pg-hackers

Kai Wagner — Mon, 02 Feb 2026 00:00:00 UTC

Last year at pgconf.dev, there was a discussion about improving the user interface for the PostgreSQL hackers mailing list, which is the main communication channel for PostgreSQL core development. Based on that discussion, I want to share a small project we have been working on:

https://hackorum.dev/

Hackorum provides a read-only (for now) web view of the mailing list with a more forum-like presentation. It is a work-in-progress proof of concept, and we are primarily looking for feedback on whether this approach is useful and what we should improve next.

What Hackorum already does

Hackorum focuses on readability, navigation, and workflow improvements for people who follow pg-hackers. Some highlights:

Continuous mailing list synchronization: The site is subscribed to the list
Commitfest integration: See commitfest context next to threads - you directly know what’s the state of this commit/thread.
User profiles: Contributor/committer status from the main website
Statistics: Per-user and mailing lists insights
Easy download of attached patches: Including helper script for easy rebase and merge
Additional logged-in user features: Per-message read status, starring threads, tags, notes, mentions on messages and threads
Basic team support: Shared reading status, shared mentioned tags and notes - mention someone underneath an email an the person gets notified
Resend email: Integration from the official archive
Importing read status / tags via CSV files: To help migration from email-based workflows

What we plan next

Sending emails from the web UI: Initially via Gmail API for Google-authenticated users who authorize sending
Advanced search functionality
Integrating other mailing lists

If you want to take a look, just got to https://hackorum.dev/

The repository, including a simple dev setup, can be found here: https://github.com/hackorum-dev/hackorum

Is this useful? What is missing? What would you change? Bug reports, feature requests, and contributions are all welcome. https://github.com/hackorum-dev/hackorum/issues

Thanks for taking a look, and we appreciate any feedback.

Tuning MySQL for Performance: The Variables That Actually Matter

Wayne Leutwyler — Sun, 01 Feb 2026 00:00:00 UTC

There is a special kind of boredom that only database people know. The kind where you stare at a server humming along and think, surely there is something here I can tune. Good news: there is.

This post walks through the most important MySQL variables to tune for performance, why they matter, and when touching them helps versus when it quietly makes things worse. This is written with InnoDB-first workloads in mind, because let’s be honest, that’s almost everyone.

1. `innodb_buffer_pool_size`

Real metrics to watch

Before touching this variable, look at these:

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read%';

Key fields:

Innodb_buffer_pool_reads – physical reads from disk
Innodb_buffer_pool_read_requests – logical reads

Rule of thumb: If reads / read_requests > 1–2%, your buffer pool is too small.

Example graph

Plot Innodb_buffer_pool_reads over time. A healthy system shows a flat or gently rising line. Spikes that look like a city skyline usually mean memory pressure or a cold cache.

If MySQL performance had a crown jewel, this would be it.

What it does

The InnoDB buffer pool caches table data and indexes in memory. Reads served from RAM are fast. Reads from disk are… character building.

How to tune it

Dedicated DB server: 60–75% of system RAM
Shared server: be conservative and leave memory for the OS and other services

sql
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';

Pro tip

If your working set fits in the buffer pool, MySQL feels magical. If it doesn’t, no amount of query tuning will save you.

2. `innodb_buffer_pool_instances`

This one matters once memory gets big.

What it does

Splits the buffer pool into multiple instances to reduce internal mutex contention.

How to tune it

Only relevant if buffer pool is ≥ 1GB
Rule of thumb: 1 instance per 1–2GB, max 8

sql
SHOW VARIABLES LIKE 'innodb_buffer_pool_instances';

Gotcha

More is not always better. Too many instances wastes memory and can hurt performance.

3. `innodb_log_file_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Innodb_log%';

Pay attention to:

Innodb_log_waits
Innodb_log_write_requests

If Innodb_log_waits is non-zero, redo logs are too small for your write rate.

Example graph

Graph Innodb_log_waits as a rate per second. Ideally, this line hugs zero like it’s afraid of heights.

This variable controls how calmly MySQL handles write-heavy workloads.

What it does

Defines the size of redo logs. Larger logs mean fewer checkpoints and smoother writes.

How to tune it

OLTP workloads: 1–4GB total redo log is common
Large transactions benefit from larger logs

sql
SHOW VARIABLES LIKE 'innodb_log_file_size';

Warning

Changing this requires a restart. Plan accordingly or accept the wrath of your on-call future self.

4. `innodb_flush_log_at_trx_commit`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Innodb_os_log_fsyncs';

Switching from 1 to 2 often reduces fsyncs by orders of magnitude.

Example graph

Overlay two lines:

Transactions per second
Innodb_os_log_fsyncs per second

On busy systems, this graph alone can justify the change to skeptical auditors.

Performance versus durability, the eternal duel.

What it does

Controls how often redo logs are flushed to disk.

Common values

1 – Safest, slowest (flush every commit)
2 – Very popular compromise
0 – Fast, risky

sql
SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';

Reality check

For many production systems, 2 delivers massive performance gains with acceptable risk, especially with reliable storage.

5. `innodb_flush_method`

This decides how MySQL talks to your disks.

What it does

Controls whether MySQL uses OS cache or bypasses it.

Caveat

Some filesystems and older kernels behave differently. Always test.

6. `max_connections`

This is not a performance knob. It is a damage limiter.

What it does

Caps the number of concurrent client connections.

Why it matters

Each connection consumes memory. Too many and MySQL dies spectacularly.

sql
SHOW VARIABLES LIKE 'max_connections';

Advice

Set it realistically
Use connection pooling
Monitor Threads_connected

7. `thread_cache_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Threads%';

Key fields:

Threads_created
Connections

If Threads_created / Connections stays above a few percent, your cache is undersized.

Example graph

Graph Threads_created as a counter. A healthy system shows a curve that flattens over time, not a staircase.

Small change, measurable win.

What it does

Caches threads so MySQL doesn’t constantly create and destroy them.

How to tune

Watch:

sql
SHOW STATUS LIKE 'Threads_created';

If it keeps climbing, increase thread_cache_size.

8. `table_open_cache` and `table_definition_cache`

Metadata matters more than people expect.

What they do

Cache open tables and table definitions to avoid repeated filesystem access.

Symptoms of being too low

High Opened_tables
Metadata lock waits

sql
SHOW VARIABLES LIKE 'table_open_cache';
SHOW VARIABLES LIKE 'table_definition_cache';

9. `tmp_table_size` and `max_heap_table_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Created_tmp%';

Watch:

Created_tmp_tables
Created_tmp_disk_tables

If disk temp tables exceed 5–10% of total temp tables, queries are spilling to disk.

Example graph

Stacked area chart:

In-memory temp tables
Disk-based temp tables

Disk usage creeping upward usually points to reporting queries pretending to be OLTP.

Disk-based temp tables are silent performance killers.

What they do

Limit how large in-memory temp tables can grow.

How to tune

Set both to the same value:

ini
tmp_table_size=256M
max_heap_table_size=256M

Reality

This helps complex queries, but bad queries still need fixing.

10. `slow_query_log` and `long_query_time`

Not a performance variable, but a performance revelation.

Why it matters

You cannot tune what you cannot see.

ini
slow_query_log=ON
long_query_time=1

This turns guesswork into evidence.

A Note on Graphing These Metrics

You don’t need exotic tools. These work well:

performance_schema
sys schema views
Prometheus + mysqld_exporter
Percona Monitoring and Management (PMM)

Golden rule: Always graph rates, not raw counters.

Final Thoughts

Tuning MySQL is less about endless knobs and more about understanding pressure points:

Memory first
I/O second
Concurrency third

Most performance wins come from a handful of variables, not heroic config files full of folklore.

If you tune one thing today, make it the buffer pool. If you tune two, add redo logs. Everything else is refinement.

And if you’re bored again tomorrow, congratulations. You’re officially a database person.

OIDC in PostgreSQL: With Keycloak

Zsolt Parragi — Mon, 19 Jan 2026 00:00:00 UTC

We spent a long time, two blog posts to be specific, talking about OAuth/OIDC in theory. Now we’ll take a more practical look at the topic: how can we configure PostgreSQL with a popular open source identity provider, Keycloak, and our pg_oidc_validator plugin?

We’ll not only look at the PostgreSQL configuration part, but also discuss the environment requirements and setting up Keycloak.

Docker containers

If you are only interested in trying out a working demo installation, we have a ready-to-use Docker Compose configuration available in our GitHub repo. This setup includes a Keycloak instance, a PostgreSQL server, and a utility container that runs psql, all running in different containers, simulating different machines.

graph TB
subgraph Host["Host Machine"]
User["👤 User"]
Browser["🌐 Browser"]
end
subgraph DockerNetwork["Docker Network"]
Keycloak["🔐 Keycloak Container"]
PG["🗄️ PostgreSQL Container"]
PSQLClient["💻 psql Container"]
end
User --- Browser
User --- PSQLClient
Browser --- Keycloak
PSQLClient --- Keycloak
PSQLClient --- PG
PG --- Keycloak
style User fill:#e1f5ff
style Browser fill:#fff4e6
style Keycloak fill:#ffe6e6
style PG fill:#e6ffe6
style PSQLClient fill:#f0e6ff
style Host fill:#f5f5f5
style DockerNetwork fill:#e8f4f8

Warning: This is a demo environment, intended only for testing purposes. Do not use it in production.

Alternatively, if you are only interested in configuring PostgreSQL, you can use this setup to start up Keycloak, and only focus on the PostgreSQL related sections of this post.

Also note: while it is a ready-to-use configuration, with everything set up… it’s missing one bit: because it tries to do everything correctly, including running every service in a different container, we have to use hostnames; we can’t just use localhost everywhere. This means that the Keycloak service uses the keycloak hostname as its name, and the host OS needs to be able to resolve this to use the device authorization flow. In most operating systems, this requires editing the hosts file - detailed instructions are shown later.

Running Keycloak (with HTTPS)

Keycloak itself has a ready-to-use Docker image for trying it out. Executing it is quite simple, but this default setup results in an unsecure setup, which is not enough for us:

docker run -p 8080:8080 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:latest start-dev

The above command starts up a container with a freshly initialized provider, with an admin user and admin password, and exposes it on port 8080 (on every interface). This is a nice way to try out the UI and start discovering Keycloak, but it has some limitations:

Authentication/authorization has to be secure, and that means it has to use secure transport layers. While the OAuth standard doesn’t specify an explicit protocol, RFC 9700, which defines best practices, clearly showcases HTTPS everywhere.

But to use that, we first need certificates for the encrypted connection.

How do I get certificates?

Depending on the exact demo environment, we have two choices:

If the demo environment uses a public domain, the proper approach is to use a certificate signed by a trusted third party. There are free authorities like Let’s Encrypt or ZeroSSL for quick setups.
In the more likely case where the demo environment is private, we have to use self-signed certificates. The rest of the blog post will discuss this approach.

Generating a simple self-signed certificate is easy with OpenSSL. The following is a sample command that can run non-interactively, without asking additional questions - but again it has some environment dependency, the hostname, which we first have to figure out:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out crt.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN="

It generates a certificate that is valid for 10 years, for hostname. The hostname part is important: to enforce security, PostgreSQL validates the TLS certificate’s hostname against the issuer URL. If the hostname in the OAuth issuer URL and the certificate’s Common Name (or Subject Alternative Name) don’t match, it won’t proceed with the login.

If you plan to run Keycloak in a Docker container but run PostgreSQL directly on the host machine, Docker exposes the 8443 port used by Keycloak on localhost, both the browser used to complete the login process and PostgreSQL can refer to the issuer as https://localhost:8443/..., which means the hostname can be localhost:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out crt.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=localhost"

But if you intend to follow the Docker compose setup, where Keycloak and PostgreSQL are two separate containers, this no longer works: the port mapping only exposes the Keycloak service for the host, not for the PostgreSQL container. That container has to refer to it as “https://keycloak:8443/…”

Which means that in this scenario, we have to use CN=keycloak instead. Or alternatively, you can generate a certificate that includes both hostnames, this is what the docker compose example configuration does:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out crt.pem -sha256 -days 3650 -nodes \
 -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=keycloak" \
 -addext "subjectAltName=DNS:keycloak,DNS:localhost,IP:127.0.0.1"

Back to Keycloak

To run Keycloak with HTTPS, we have to use a different port, and specify the certificates

docker run -p 127.0.0.1:8443:8443 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin -e KC_HTTPS_CERTIFICATE_FILE=/keys/crt.pem -e KC_HTTPS_CERTIFICATE_KEY_FILE=/keys/key.pem -v /path/to/the/keys:/keys/ quay.io/keycloak/keycloak:latest start-dev

This command specifies two more environment variables, the filenames of the certificate and the private key, and mounts the directory containing them.

Note: Please note that we used 127.0.0.1:8443:8443 instead of simply 8443:8443 It is a good practice to not expose admin interfaces with default passwords publicly.

Trusting the certificates

Now that we have a running Keycloak instance with HTTPS certificates, we need to make sure our systems trust them.

When you open a browser and navigate to a website with a self-signed certificate, you’ll get a warning. After acknowledging the warning you can proceed and use the website normally.

Similarly, software using HTTPS for communications usually defaults to proper certificate verification, but often also allows administrators to either disable the certificate check – not safe in production, but useful for quick demos like this – or to manually specify a certificate authority used for verification.

Unfortunately at this point this isn’t the case for PostgreSQL, it doesn’t provide such options. The operating system has to trust the certificates on both the server and client host, otherwise it will refuse to complete the OIDC authentication flow.

If you used the approach with a public domain and generally trusted authority, this is not an issue. But if you generated a self-signed certificate instead, you won’t be able to authenticate unless you make the systems running PostgreSQL trust this certificate.

On most Linux systems, this is as simple as copying the certificate (crt.pem) to a specific directory, and running a system script that updates the trusted certificates. For example, on Ubuntu:

sudo cp crt.pem /usr/local/share/ca-certificates/keycloak-test.crt
sudo update-ca-certificates

Also, we shouldn’t forget that we can have up to 4 different systems:

the Keycloak server
the PostgreSQL Server
the system running the psql client
and another system running the browser which completes the device flow

This is the case with our Docker Compose example - 3 of these are containers, and the browser runs on the host machine. Browsers usually ignore certificates placed in the above folder. But that’s not an issue, you can acknowledge the warning and still use the website. The only part where trust matters is the libcurl library used by PostgreSQL, and that uses the certificates trusted by the system.

The host system only has to trust the certificate if it is also used to run PostgreSQL.

Note: Please do not add random certificates to your everyday OS, or at least remember to delete them later.

Recognizing the Keycloak host

Besides trusting the certificates, there’s another hostname-related configuration we need to address.

Even if you run PostgreSQL directly on the host machine, it is still possible to access Keycloak using the ‘https://keycloak’ URL instead of localhost – and if you do run the PostgreSQL server in a container, you have to use this form.

For this to work, all 3 systems that need to connect to Keycloak (the PostgreSQL server, the psql client, and the browser) have to recognize this hostname. When using docker compose, this hostname resolution will work directly in the other containers, but not on the host itself.

To make this work on the host, or on any other machine that requires it, you have to edit the hosts file:

On Linux/Mac:

bash
echo "127.0.0.1 keycloak" | sudo tee -a /etc/hosts

On Windows: Edit C:\Windows\System32\drivers\etc\hosts as Administrator and add:

127.0.0.1 keycloak

Can’t I just use localhost in the browser instead?

You might be wondering if there’s a shortcut here.

Even if the PostgreSQL container has to reference Keycloak as keycloak, your host still sees the exposed port as localhost:8443. So can you just use this in the browser instead, and complete the authentication that way, without editing the hosts file?

The answer is unfortunately no. There are two possible scenarios:

If Keycloak is configured with strict hostname, it will try to redirect the browser to “https://keycloak…” during the authorization process
If Keycloak is configured with dynamic hostname, it will complete the process with “https://localhost”, but it will also use “localhost” in the generated access tokens instead of “keycloak”. When our validator checks the token, it will notice this discrepancy and reject the login attempt.

Configuring your realm

With the Keycloak infrastructure setup out of the way, we can now focus on configuring Keycloak itself.

After you have your Keycloak instance up and running, it is time to open a browser and navigate to https://keycloak:8443. A quick login with “admin” and “admin”, and the browser already displays the admin UI with the default master realm.

A complete detailed introduction is out of scope for this blog post – the Keycloak documentation is much better for that – we will only try to explain the minimum required to set up a relatively simple, but secure configuration for our PostgreSQL instance. The steps we show here will be similar to what our demo setup also uses.

Let’s start by creating a new realm, under the “Manage realms” menu: realms are the main building blocks of isolation in Keycloak, storing users, clients, and everything, so it’s a good practice not to use the default one. In our example, we named our realm pgrealm. This name is important, as it will be included in the OAuth issuer URL.

After hitting “Create”, the new realm is automatically set as current, and we can continue configuring it.

Let’s continue by creating our “testuser” under “Users”. Select that the email is verified, fill the requested email, first name and last name fields, and hit create. If you miss some of these fields, Keycloak will ask you to complete them during the first login.

After the user is created, navigate to the “Credentials” tab on the displayed user admin page, and set a password. Also remove the checkbox from “Temporary”, unless you want to change it during the first login. In our example setup, we used “asdfasdf”. This is of course only appropriate for a quick demo setup, use a better one for anything else.

After creating our user, let’s create a client under “Clients” with the “Create Client” button. The first screen asks for a client ID – this will be required for the psql command – and a name and description – these will be displayed on the authorization page by the browser.

After clicking next, the next screen configures how OIDC should work exactly.

The first toggle, “Client authentication” can be both on and off – this controls if the client requires a secret, or only an ID. As we discussed earlier, psql is a public client, and while it can use a secret, it’s not really a secret. Adding a secret only adds complexity while not providing more security, as we also have to specify that during the connection call, but it is supported.

For the “authentication flow” we have to check “OAuth 2.0 Device Authorization Grant” to enable the device flow, and we can leave everything else on default.

The final third screen doesn’t require any changes – those are settings for HTTP-based flows, but we are using the device flow. If you are configuring PostgreSQL OIDC with a web application, of course you should fill these properly.

After creating our client, it’s also a good practice to make our user consent more explicit: as we tried to make this clear in earlier blog posts, this is the only line of defense with OAuth-based logins: administrators have to be very clear about displaying where the user logs in.

First, if you scroll down on the Client administration page displayed after creation, there’s a section about the consent screen. It’s a good practice to make this as explicit as possible, with a nice custom message for the users.

After saving this, we also should add a client scope using the menu item below “Clients”. The name attribute is what we’ll have to specify in our PostgreSQL configuration. The two toggles below are both required: “include in token scope” means that it will be included in the JWT; without that, the validator won’t be able to verify the presence of the scope. “Display on consent screen” means that this is a scope that should be explicitly displayed to the user during login.

(We’ll discuss what scopes are and why they matter in the next section.)

And with this, our basic Keycloak configuration is ready!

What is a scope?

The scope configuration in the previous section might seem confusing: why did we configure consent screen settings at multiple locations? What is this scope concept actually about?

To answer this question, we have to remember our earlier examples, where we discussed that PostgreSQL itself is not a client (application), but it is just something used by possibly multiple clients. In the real world, that client could be “psql”, or “EditorApp”, or anything else that uses PostgreSQL while not providing more security.

PostgreSQL could use a list of allowed clients for authorization – but it doesn’t do that. Instead it relies on another OAuth concept, scopes.

The OAuth scope is a mechanism intended to limit what a token is allowed to access. There are some generic scopes, such as “email” or “profile”. And there are many application specific scopes: cloud providers like Google or Azure use them to control which services a token is allowed to access. For example, if you grant something the ability to add entries to your calendar, it won’t be allowed to read your emails or location history.

The same way, you can think of scopes with PostgreSQL as “which database the user can access”. Database here could mean both a PostgreSQL instance, or just a single database in it – as OAuth is configured in pg_hba, it is possible to configure different required scopes for different databases (more about this later).

During the Keycloak configuration we recommended configuring an explicit consent screen for the client, but technically that part is outside of the database configuration. A database can be accessed by multiple clients, some of those might be created way later in time.

Even if somebody doesn’t configure that part, a scope with a required consent screen will be displayed, if the client requests it. And that is part of the database configuration – as we can configure PostgreSQL explicitly to require specific scopes in its configuration.

Configuring PostgreSQL

Now that we have a working, properly configured Keycloak server, it’s time to configure the PostgreSQL side of the equation. While we do have a working Docker example as part of the Docker Compose configuration, this is currently tricky to set up: there is no Docker image with our pg_oidc_validator.

You can check the relevant section of the compose configuration, but it requires a custom entry point and a short bash script.

In this blog post we’ll focus on the actual manual steps instead.

Required packages

On the server, we need the PostgreSQL 18 server packages installed, and also the pg_oidc_validator package. For the validator, we currently have downloadable deb and rpm packages on our GitHub releases page, and packages for SUSE can be found in the official SUSE packages.

On the client side, we need the PostgreSQL 18 client along with the OAuth client package – this is usually a separate package named libpq-oauth on most distributions and requires separate installation

Setting up a data directory

Let’s just assume that we have a data directory initialized somewhere:

initdb -D datadir

We will have to modify a few configuration files for OIDC to work.

Let’s start by enabling the validator in datadir/postgresql.conf:

oauth_validator_libraries = pg_oidc_validator
pg_oidc_validator.authn_field = email

The first line tells PostgreSQL to load the Percona pg_oidc_validator, and the second line is a specific configuration parameter for our validator – it tells it that we want to map Keycloak users to PostgreSQL users based on the email claim in the access token. This second line is optional; it defaults to the “sub” (subject) field, which identifies the user in most OIDC providers. However, Keycloak doesn’t allow us to customize the value of this field, and it returns a non-user-friendly identifier in it. In practice, it is clearer to use a more verbose field for mapping, such as email, so we’ll use that in our example.

After modifying this file, we can try to start the server. If all required packages are installed correctly, and there isn’t another PostgreSQL instance running on the default port, it should start without issues:

pg_ctl -D datadir start

With the server running, let’s create a matching PostgreSQL user for our testuser:

createuser testuser

Next, we have to tell PostgreSQL that we want to use the Keycloak instance we set up previously. Let’s add an entry to datadir/pg_hba.conf to reference the OIDC provider. Add this line to the beginning of the file, before the trust sections – since pg_hba is executed line by line, if the trust entries are before the OAuth entry, authentication will never reach that line:

host all all 0.0.0.0/0 oauth scope="email pgscope",issuer=https://keycloak:8443/realms/pgrealm,map=kcmap

Warning: In a real-world setup, you would have to remove all trust entries. The trust authentication method allows connections without any password verification and should never be used in production.

This entry adds the OAuth option for connections using IPv4. We specify that we require “pgscope”, which is the example scope with the custom consent screen we created earlier in the Keycloak configuration, and the “email” scope, which instructs Keycloak to include the user’s email address in the JWT (this will also be presented on the consent screen).

The latter is required because in this example we map our database users to the users on the identity provider using their email address – this will be configured in more detail in the “kcmap” referenced in the configuration line.

The only required parameter to OAuth is the “issuer”; everything else is optional. But providing required scopes is a good practice, and it is required for proper security, as we explained earlier. There are also other parameters not mentioned here, a full list is available in the PostgreSQL documentation.

One potentially important parameter is the “validator”. PostgreSQL allows multiple different pg_hba OAuth entries, and it also allows multiple different validators: the configuration parameter in postgresql.conf is called oauth_validator_libraries. If that parameter actually contains multiple libraries, the “validator” parameter becomes required for OAuth entries in pg_hba conf, and has to match an entry in the validator list. Otherwise, it is assumed that all OAuth entries use the one available validator.

The above means that the map setting is also optional. There are scenarios where it’s not needed, such as if the identity provider has a claim that directly contains user names matching PostgreSQL usernames, we could use that field instead and skip the manual mapping.

Following our example, let’s define an entry in datadir/pg_ident.conf for the kcmap. We only need a single entry for our testuser:

# MAPNAME SYSTEM-USERNAME DATABASE-USERNAME
kcmap testuser@example.com testuser

All that’s left is to reload our configuration:

pg_ctl -D datadir reload

Connecting to the database

With all the configuration in place, it’s time for the moment of truth: actually connecting to PostgreSQL using OIDC!

With the server properly configured, we are ready to connect to it using psql. To do so, we have to use a command similar to the following:

bin/psql -h 127.0.0.1 'dbname=postgres oauth_issuer=https://keycloak:8443/realms/pgrealm oauth_client_id=pgclient'

Note that we have to repeat the issuer URL here. It is always required, even if the PostgreSQL configuration only contains one OAuth issuer. This URL has to match exactly the issuer URL in a pg_hba line, and both should exactly match the issuer URL in the JWT. If there’s a mismatch anywhere, authentication will fail. This is why we had to take extra steps previously to make the Keycloak hostname available everywhere.

This is also the only place where we have to mention a client id – and if we configured an authenticated client in Keycloak, we also have to specify oauth_client_secret. The server can work with multiple clients, but now that we are actually starting up a PostgreSQL client, we can specify which OAuth client will we use to complete the authentication flow.

After we execute this command, psql will display the device authorization instructions, showing us a URL and a device code.

All we have to do is follow the instructions:

navigate to the specified URL
enter the device code
login with our testuser
confirm the consent screen

If you are doing all of this in a single session, don’t forget to log out of the admin user session before executing these steps, or use a different browser for it. We want to log in with the testuser, not with the admin – the admin user doesn’t have a mapping in our PostgreSQL configuration.

While we go through these steps, the psql client periodically polls Keycloak to see if the flow was completed on the OIDC provider side. Since this is a periodic polling, done every few seconds, we might have to wait a few seconds before we are logged in to an SQL session.

That’s all!

And just like that, we’ve successfully authenticated to PostgreSQL using OIDC!

With the psql command logged in, we’ve completed the full circle: from setting up Keycloak with proper certificates, through configuring realms and clients, to establishing a secure OIDC-authenticated PostgreSQL connection.

Hopefully these instructions were clear and everything worked on the first try. If you encountered issues along the way, don’t be discouraged: OAuth/OIDC is complex, there are many things that could go wrong. Since this is an important security feature, it has to fail if anything is even slightly wrong.

In our next post, we’ll focus on errors and failures: both to help diagnose possible errors with the OAuth flow in PostgreSQL, but also for reassurance: in an authentication setup not letting unauthorized people log in is just as important as successfully logging in somebody with the proper permissions. Stay tuned for our examples showcasing how pg_oidc_validator and PostgreSQL’s OAuth support can keep your server safe!

If you find any issues with our validator, or have comments / feature requests, please reach out to us in our Github page!

Configuring the Component Keyring in Percona Server and PXC 8.4

Wayne Leutwyler — Tue, 13 Jan 2026 00:00:00 UTC

Configuring the Component Keyring in Percona Server and PXC 8.4

(Or: how to make MySQL encryption boring, which is the goal)

Encryption is one of those things everyone agrees is important, right up until MySQL refuses to start and you’re staring at a JSON file wondering which brace ruined your evening.

With MySQL 8.4, encryption has firmly moved into the component world, and if you’re running Percona Server 8.4 or Percona XtraDB Cluster (PXC) 8.4, the supported path forward is the component_keyring_file component.

The good news: the setup is mostly identical for Percona Server and PXC.
The bad news: PXC 8.4.4 and 8.4.5 shipped with a bug that makes this less fun than it should be.

Let’s walk through a setup that works, keeps your keys locked down, and avoids the usual landmines.

Step 1: Tell MySQL Which Component to Load

Components are registered using JSON, not traditional MySQL configuration syntax. This is important, because MySQL will not politely warn you if you get it wrong. It will simply refuse to start.

Create the file:

sudo vi /usr/sbin/mysqld.my

Add:

json
{
 "components": "file://component_keyring_file"
}

Take a second to double-check the formatting. One missing quote here will cost you more time than you want to admit.

Now lock it down:

bash
sudo chown root:root /usr/sbin/mysqld.my
sudo chmod 644 /usr/sbin/mysqld.my

This is configuration, not data. MySQL only needs to read it.

Step 2: Prepare the Keyring Directory (Handle With Care)

This directory will hold encryption keys. Treat it accordingly.

bash
cd /var/lib
sudo mkdir mysql-keyring
sudo chown mysql:mysql mysql-keyring
sudo chmod 750 mysql-keyring

A simple rule that saves headaches:

mysql owns the keys
MySQL is allowed to access them
Nobody else gets any ideas

Step 3: Configure the Keyring Component Itself

Next, move to the plugin directory:

cd /usr/lib64/mysql/plugin

Create the component configuration file:

sudo vi component_keyring_file.cnf

Add:

json
{
 "path": "/var/lib/mysql-keyring/component_keyring_file",
 "read_only": true
}

This file tells MySQL where the keyring lives and ensures it can’t be casually modified at runtime.

Set ownership and permissions:

bash
sudo chown root:root component_keyring_file.cnf
sudo chmod 640 component_keyring_file.cnf

Again: configuration belongs to root. MySQL just reads it.

Step 4: The PXC 8.4.4 / 8.4.5 Bug (Yes, There’s One)

If you’re running Percona Server, you can skip this entire section and enjoy your day.

If you’re running Percona XtraDB Cluster 8.4.4 or 8.4.5, there is a known issue with plugin paths that prevents the component keyring from loading correctly. This was fixed in PXC 8.4.6.

If upgrading isn’t an option yet, you’ll need one of the following workarounds.

Option A: Create a Symlink (Preferred)

bash
sudo ln -s \
/usr/bin/pxc_extra/pxb-8.4/lib/lib64/xtrabackup/plugin \
/usr/bin/pxc_extra/pxb-8.4/lib/plugin

Option B: Copy the Plugin Directory

bash
sudo cp -ar \
/usr/bin/pxc_extra/pxb-8.4/lib/lib64/xtrabackup/plugin \
/usr/bin/pxc_extra/pxb-8.4/lib

If you’re on PXC 8.4.6 or newer, this problem is already behind you and you can safely pretend it never existed.

Step 5: Restart MySQL

Time for the moment of truth:

sudo systemctl restart mysql

Or mysqld, depending on your system.

If MySQL starts cleanly, you’re doing well. If not, go back and check your JSON files. It’s almost always the JSON.

Step 6: Verify the Keyring Is Actually Loaded

Never assume. Always verify.

sql
SELECT *
FROM performance_schema.keyring_component_status;

You should see the component_keyring_file listed and active. If it’s there, the keyring is live.

sql
+---------------------+-----------------------------------------------+
| STATUS_KEY | STATUS_VALUE |
+---------------------+-----------------------------------------------+
| Component_name | component_keyring_file |
| Author | Oracle Corporation |
| License | GPL |
| Implementation_name | component_keyring_file |
| Version | 1.0 |
| Component_status | Active |
| Data_file | /var/lib/mysql-keyring/component_keyring_file |
| Read_only | Yes |
+---------------------+-----------------------------------------------+
8 rows in set (0.00 sec)

A Note for Percona Server Users

Percona Server may still include legacy keyring plugins such as:

keyring_file
keyring_vault

Do not mix legacy keyring plugins with component keyrings. They come from different eras of MySQL design and do not coexist peacefully.

Choose one model. For MySQL 8.4 and forward, components are the future.

Additional Steps for Percona XtraDB Cluster (PXC)

Percona XtraDB Cluster introduces one critical difference compared to standalone Percona Server: the keyring file itself is not replicated by Galera. Only metadata and transactional state are replicated. The encryption keys remain node-local filesystem artifacts and must be handled deliberately.

Node 1: Establish the Authoritative Keyring

Choose a single node to initialize the keyring. This is typically Node1, but the choice itself is not important as long as you are consistent.

On this node:

Complete all previous steps in this document
Start MySQL successfully
Verify the keyring component is loaded:

sql
SELECT *
FROM performance_schema.keyring_component_status;

Once this node is running, the file below will be created and populated:

swift
/var/lib/mysql-keyring/component_keyring_file

This file becomes the authoritative source of encryption keys for the entire cluster.

Why the Keyring File Must Be Copied

PXC ensures that encrypted data remains readable on all nodes, but it does not distribute encryption keys themselves. Each node must have access to the same key material, or encrypted tablespaces will fail to open.

If a node starts without the correct keyring file, you may see:

Tablespace open failures
Startup errors related to encryption
Inconsistent behavior during SST or IST

This is expected behavior and not a bug.

Distribute the Keyring File to Other Nodes

On each remaining PXC node:

Ensure MySQL is stopped
Create the keyring directory if it does not exist:

bash
sudo mkdir -p /var/lib/mysql-keyring
sudo chown mysql:mysql /var/lib/mysql-keyring
sudo chmod 750 /var/lib/mysql-keyring

Securely copy the keyring file from Node1:

scp /var/lib/mysql-keyring/component_keyring_file node2:/var/lib/mysql-keyring/component_keyring_file

Important: Do not modify the file. Do not recreate it. Do not allow MySQL to generate a new one on secondary nodes.

Start MySQL on Each Node and Verify

After the keyring file is in place:

sudo systemctl start mysqld

Verify the component is active:

sql
SELECT * FROM performance_schema.keyring_component_status;

Each node should report the component_keyring_file as loaded and active.

At this point:

Encrypted tablespaces will open correctly
SST and IST operations will succeed
The cluster will behave consistently during restarts

Operational Notes and Best Practices

Treat the keyring file like a secret, not configuration
Restrict access to root only
Include the keyring file in your secure backup strategy
When provisioning new nodes, copy the keyring file before first startup
Never rotate or regenerate the keyring independently on individual nodes

If the keyring is lost and encrypted data exists, recovery is not possible.

Final Thoughts

This setup works reliably for:

Percona Server 8.4
Percona XtraDB Cluster 8.4
(with the known exception of 8.4.4–8.4.5)

Most failures come down to:

Treating JSON like a .cnf file
Loose ownership on sensitive files
Forgetting the PXC-specific workaround

Once those are handled, the component keyring fades into the background where it belongs. And when it comes to encryption, boring, quiet, and uneventful is exactly the outcome you want.

Open source, PostgreSQL, and risk mitigation in an era of acquisitions

Jan Wieremjewicz — Fri, 19 Dec 2025 11:00:00 UTC

As the year comes to a close and many of us start slowing down before the winter holidays, I find myself reflecting on patterns I’ve seen repeat, both as a customer and as someone working closely with the PostgreSQL ecosystem.

Looking back at software acquisitions over the past year, one might assume they only change logos. In reality, they often change roadmaps, priorities, and unfortunately all too often also the promises customers originally bought into.

I have experienced this more than once as a customer. Most recently, my team evaluated a product management tool that, shortly after being acquired, informed us we had a single month to migrate away. While I personally had not invested much time yet, colleagues had already moved documentation and workflows only to see that work become wasted effort. For those not familiar with product management, migrating all the product planning documentation is a huge effort and having it wasted is like erasing a month of your life.

One might say “business is business” and that customers can always take their money elsewhere. That may be true, but it does little to make customers feel safe or protected. In practice, it often leaves them confused about where accountability lies. Who should we blame when the original vendor no longer exists and the acquiring company is acting in its own strategic interest?

What struck me this year is how often these experiences echoed each other: across different tools, teams, and ecosystems. Unfortunately, I am now seeing similar patterns emerge in my professional focus area: PostgreSQL.

To be very clear, the PostgreSQL Community itself is driving an outstanding database, one that has been, and I am confident will remain, truly open source and community-developed. PostgreSQL as a project is healthy, stable, and thriving, and I hope it will thrive even more in the coming year.

That said, PostgreSQL is not just the database engine itself, but an entire ecosystem of vendors, extensions, and services built around it.

Over the past year, several important companies in the PostgreSQL ecosystem have been acquired and in conversations throughout the year this topic kept resurfacing. In recent prospective customer conversations, we increasingly hear from PostgreSQL users running mission-critical workloads on-premises who feel abandoned by their vendor. These organizations are being quietly nudged toward “modernization” paths that, in practice, resemble mandatory SaaS migration.

Having worked closely with MongoDB users for years, this pattern feels familiar. Since MongoDB Atlas became the primary strategic focus, many customers experienced similar pressure. What feels different, and what stood out to me most this year, in the PostgreSQL world is timing.

Teams often discover late in the renewal cycle that:

Their on-prem PostgreSQL deployment is no longer strategic for the vendor
Key components they depend on, which were never fully open source, are no longer available for renewal
The implied options are to “move to the cloud” or rapidly find an alternative, even when migration planning is complicated by lack of source availability

This puts PostgreSQL customers in a difficult position: accept architectural change under pressure, or scramble to replace a trusted vendor while the clock is already ticking.

While recent conversations often reference Crunchy Data following its acquisition by Snowflake, this is not about a single company. The broader pattern has repeated across the industry, including infrastructure significant projects involving MinIO, Bitnami, HashiCorp, and Redis.

This raises a fundamental question for the PostgreSQL ecosystem and infrastructure software in general:

When a critical infrastructure vendor is acquired or changes licensing, who advocates for the customers that cannot move?

Open source is not just a licensing model or philosophical ideal. It provides freedom of choice, deployment flexibility, and risk mitigation. Recent community responses such as OpenTofu, OpenBao, and Valkey demonstrate a growing maturity and ability of open source communities to organize when freedoms erode.

It is disappointing to see these dynamics emerge around PostgreSQL, even though the database itself remains one of the most open source and community governed projects in the industry.

It is disappointing to see similar dynamics affect PostgreSQL, a database often considered one of the most open source driven projects in the industry. When PostgreSQL derivatives are impacted, the shadow inevitably reaches upstream as well.

If this post reaches teams who were not yet aware of these shifts and gives them more time to plan going into the new year, it serves its purpose. If you are running PostgreSQL in environments where SaaS is not an option, now is the time to ask difficult questions before renewal conversations start, not after options disappear.

As we head into a new year, I expect these conversations to become even more common. Acquisitions, licensing changes and cloud first strategies are not slowing down, but neither is the need for predictability, transparency and choice in how PostgreSQL is deployed and supported.

Looking ahead to 2026, my hope is simple: more open source products, more stable and predictable service offerings around them, and fewer last-minute surprises for the teams who depend on them every day.

We Want to Hear from You

I am curious:

Have you seen similar shifts in PostgreSQL vendors post-acquisition?
What is preventing you from moving to PostgreSQL community builds or alternative support models?

Let’s talk. You can reach me via:

LinkedIn: https://www.linkedin.com/in/janwie/
Email: jan(dot)wieremjewicz(at)percona(dot)com

Open Source isn’t a strategy, it’s who we are!

We are always open to your feedback. You can reach us at:

Percona Community Forums https://forums.percona.com/c/postgresql/25
Via issues and discussions on Percona GitHub repositories

If there is an event we attend, focused on open source or not, don’t be a stranger, come chat with us in person!

Enhancing PostgreSQL OIDC with pg_oidc_validator

Jan Wieremjewicz — Wed, 17 Dec 2025 11:00:00 UTC

With PostgreSQL 18 introducing built-in OAuth 2.0 and OpenID Connect (OIDC) authentication, tools like pg_oidc_validator have become an essential part of the ecosystem by enabling server-side verification of OIDC tokens directly inside PostgreSQL. If you’re new to the topic, make sure to read our earlier posts explaining the underlying concepts and the need for external validators:

This release builds on the initial version announced in October and continues our mission to make OIDC adoption in PostgreSQL reliable, fast, and accessible for all users.

What’s New in This Release

This new iteration of pg_oidc_validator (v0.2) introduces two major improvements:

initial caching support, and
Debian/Ubuntu and RPM packages to simplify installation.

Most importantly, these improvements come directly from community feedback, ****whether during conversations at PGConf.EU in Riga, KubeCon US in Atlanta, or through GitHub and forums. Thank you for helping us shape this project!

Caching Support in pg_oidc_validator

OIDC token verification requires fetching issuer metadata and JWKS keysets from an external identity provider (IdP). Without caching, every PostgreSQL backend performing validation must re-fetch this data, increasing latency and putting unnecessary load on the IdP.

pg_oidc_validator v0.2 introduces a lightweight caching layer. This allows the validator to:

cache OIDC discovery documents and JWKS responses when permitted by the IdP,
use cached responses across PostgreSQL backends,
reduce outbound HTTP calls,
validate tokens at in-memory speeds, and
integrate cleanly with IdP key rotation.

This results in improved performance, reduced IdP load, and better scalability for deployments using Keycloak, Okta, Microsoft Entra ID, Ping Identity, or other OIDC providers.

A Note on Testing

The caching layer currently lacks full automated test coverage. This is because Keycloak does not allow caching for issuer or JWKS endpoints (Keycloak issue #15216), preventing us from validating caching behavior.

To address this, we plan to extend the test setup by placing an nginx proxy between PostgreSQL and Keycloak to simulate IdP responses that include cache-friendly headers.

Pre-Built Packages Now Available

Installing pg_oidc_validator is now easier than ever. We provide builds at the latest release page, where nightly builds are available for:

Debian / Ubuntu - tested on Ubuntu 24.04
RHEL / Oracle Linux / Rocky Linux - tested on OL8 and OL9

If you prefer building from source, instructions are available directly in the project’s README.

Try it, test it, tell us all about it!

As an open source project, pg_oidc_validator grows with your feedback. We want to hear about:

your deployment use cases,
performance characteristics,
integration challenges,
and features you’d like to see next.

You can reach us here:

Percona Community Forums:

https://forums.percona.com/c/postgresql/25
GitHub:
- Issues: https://github.com/Percona-Lab/pg_oidc_validator/issues
- Discussions: https://github.com/Percona-Lab/pg_oidc_validator/discussions

And of course, if you see Percona at an event, come talk to us at the booth!

What is New in Percona Toolkit 3.7.1

Sveta Smirnova — Wed, 17 Dec 2025 00:00:00 UTC

Percona Toolkit 3.7.1 has been released on Dec 17, 2025. The most important updates in this version are:

Finalized SSL/TLS support for MySQL
Added support for Debian 13 and Amazon Linux 2023
Fixed MariaDB support broken in version 3.7.0
Added options to skip certain collections in pt-k8s-debug-collector and pt-stalk
Documentation improvements
Other performance improvements

In this blog, I will outline the most significant changes. A full list of improvements and bug fixes can be found in the release notes.

SSL/TLS support for MySQL

Percona Toolkit historically did not have consistent SSL support. This was reported at https://perconadev.atlassian.net/browse/PT-191. In version 3.7.0, option s for DSN was introduced. This option instructs DBD::mysql to open a secure connection with the database. This version also adds command-line option --mysql-ssl and its short form -s to all tools. All other SSL/TLS-related options, such as ssl-ca, ssl-cert, ssl-cipher, etc, could be specified in the configuration file if necessary. This completes SSL/TLS support for MySQL. For more details and information check this blog post.

Supported Platforms Update

Percona repositories now have Percona Toolkit packages for Debian 13 and Amazon Linux 2023. To install them enable repository pt with the percona-release utility. Ubuntu Focal reached its EOL and support for this platform has been removed. More information on Percona repositories is available in the User Reference Manual.

Regression Bug Fixes

Recent major changes introducing MySQL 8.4 support missed ignore case modificator for the regular expression that checks if MySQL flavor is MariaDB. As a result, tools executed replication statements not compatible with MariaDB. Version 3.7.1 fixes the regular expression and re-adds MariaDB support back (PT-2451). Future versions of Percona Toolkit will have better MariaDB support, including MariaDB-specific versions of non-offensive replication commands.

Utility pt-sift stopped working, because dependent library alt_cmds.sh was not included (PT-2498). This was not found during previous release testing, because regression test for the tool was not run. Now this miss is fixed and the utility works properly again. Additionally, regression test is updated.

Helper utility version_cmp was written in some compiled language and source code for it was not available (PT-2469). This broke version checking on platforms not compatible with the unknown platform where the binary was originally compiled. Now this utility rewritten as a Bourne-Again shell script.

Modern MySQL Support

Percona Toolkit uses legacy MySQL syntax in many places to be compatible with older versions of MySQL. In other places, it misses modern MySQL diagnostic additions. This version makes first steps to improve this situation by adding such features as invisible index support in pt-duplicate-key-checker (PR-996) and performance_schema.threads collecton in pt-stalk (PT-1718). Currently, data from performance_schema.threads is collected along with the deprecated information_schema.processlist. In the future, support for information_schema.processlist will be deprecated, then removed.

Future versions of Percona Toolkit will have more modern MySQL diagnostic support.

Performance Improvements

pt-stalk now has new option, --skip-collection, that allows to skip one or more collections. Supported values for this option are: ps-locks-transactions, thread-variables, innodbstatus, lock-waits, mysqladmin, processlist, rocksdbstatus, transactions. To skip two or more collections, separate them with a comma. E.g., --skip-collection=processlist,innodbstatus. You will find more information at PT-2289 and in the User Reference Manual for pt-stalk.

pt-k8s-debug-collector introduces option -skip-pod-summary allowing to skip pod summary collections, such as pt-mysql-summary, pt-mongodb-summary, or pg_gather. Check PT-2453 and the User Reference Manual for pt-k8s-debug-collector.

Originally, tools output was always buffered. This is usually good for performance but you may want to disable this feature when need to see output of the tools faster. For example, if you run pt-archiver or pt-table-checksum on large table in Kubernetes, you won’t see progress (PT-2052) until the tool finishes. New option, --[no]buffer-stdout, allows to disable buffering when needed.

Incompatilbe change

Earlier, if --chunk-size was enabled for pt-online-schema-change, option --chunk-time was ignored. This caused situations when a user has to start with default automatic chunk size even if it was not effective for some tables, and wait when chunk size is adjusted in subsequent iterations. Alternatively, they had to guess fixed chunk size that implies time consuming try and error approach (PT-1423).

Starting from version 3.7.1, if both options --chunk-size and --chunk-time are specified, initial chunk size will be as specified by the option --chunk-size, but later it will be adjusted, so that the next query takes specified amount of time (in seconds) to execute.

Documentation Improvements

While working on this release we found undocumented featues such as --recursion-method=dsn support in pt-table-sync (PT-2470), broken man page for pt-secure-collect and other tools written in Go language (PT-1564), as well as minor documentation issues. Now all of them are fixed.

Community contributions

This release includes contributions from Community and Percona Engineers who do not actively work on the project. We want to thank:

Iwo Panowicz for option -skip-pod-summary in pt-k8s-debug-collector (PT-2453)
Matthew Boehm for invisible indexes support in pt-duplicate-key-checker (PR-996)
Nilnandan Joshi for collecting performance_schema.threads along with information_schema.processlist in pt-stalk (PT-1718) and fix for PT-2014 - pt-config-diff does not honor case insensitivity flag
Paweł Kudzia for the updated documentation of pt-query-digest (PR-953)
Maciej Dobrzanski for fixing PR-890 - pt-config-diff: MySQL truncates run-time variable values longer than 1024 characters
Marek Knappe for fixing PT-2418 - pt-online-schema-change 3.7.0 lost data when exe alter xxx rename column xxx and PT-2458 - remove-data-dir defaults to True
Yoann La Cancellera for his work on pt-galera-log-explainer
Nyele for restoring MariaDB support (PT-2465)
Taehyung Lim for fixing PT-2401 - pt-online-schema-change ’table does not exist’ on macos
Viktoras Agejevas for fixing PR-989 - Fix script crashing with precedence error in pt-online-schema-change
Hartley McGuire for fixing PT-2015 - pt-config-diff does not sort variable flags

MySQL Replication Best Practices: How to Keep Your Replicas Sane (and Your Nights Quiet)

Wayne Leutwyler — Wed, 03 Dec 2025 00:00:00 UTC

MySQL replication has been around forever, and yet… people still manage to set it up in ways that break at the worst possible moment. Even in 2025, you can get burned by tiny schema differences, missing primary keys, or one forgotten config flag. I’ve seen replicas drift so far out of sync they might as well live in a different universe.

This guide covers the practical best practices—the stuff real DBAs use every day to keep replication stable, predictable, and boring. (Boring is a compliment in database land.)

Always Use GTIDs. Yes, Always.

GTID-based replication is one of those features that people resist turning on, and then once they do, they never want to go back.

Why GTIDs?

Failover become sane
Reparenting replicas stops being a headache
Missing transactions are easy to detect

Your my.cnf should absolutely include:

gtid_mode=ON
enforce_gtid_consistency=ON
log_replica_updates=ON

Once GTIDs are enabled, do not mix in old-style replication. That path leads straight to confusion.

Use Row-Based Replication (RBR)

Statement-based replication is a nostalgia trip that nobody asked for. It breaks on:

NOW(), UUID(), and similar functions
Floating point differences
Collation mismatches
Triggers behaving differently

Just skip the pain and use:

binlog_format=ROW

RBR is slightly more verbose, but 100× more predictable. When something breaks, it’s never because you chose ROW.

Every Table Needs a Primary Key. No Exceptions.

If you take nothing else from this guide, take this:

Replication without primary keys is a bad time.

Row-based replication needs a way to find the row that changed. Without a PK (or at least a UNIQUE index), the server has to use every column as a lookup. That’s slow, error-prone, and sometimes impossible.

The usual symptoms:

Replication lag slowly creeping up
Replica doing full table scans on updates
Rows failing to apply
Errors like:

Error 1032: Can't find record in table

Save yourself hours of debugging and just make sure every table has a primary key.

Keep the Schema Identical Everywhere

Replication assumes that everyone’s using the same schema. MySQL will happily keep going even if your schemas don’t match—and then quietly drift out of sync.

Here are the practical ways to keep schemas aligned:

Approach A — mysqldump (most common)

Export schemas only:

mysqldump --no-data mydb > schema.sql

From both servers, then:

diff source-schema.sql replica-schema.sql

Approach B — information_schema metadata

This approach is great for automaton:

SELECT table_name, column_name, column_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema = 'mydb'
ORDER BY table_name, ordinal_position;

Execute this query on each server and diff the results. Update mydb to match the database whose schema metadata you want to examine.

Approach C — pt-table-checksum (data only)

This doesn’t compare schemas — it catches data drift. You should consider running it on a schedule such as:

high-change OLTP DBs run weekly or even daily
huge multi-TB DBs run quarterly
some sensitive systems avoid running it during peak hours

pt-table-checksum --replicate=percona.checksums

You can fix drift with:

pt-table-sync --execute --replicate=percona.checksums

Schema checks + data checks = safe replication.

Harden Your Binary Log Settings

Your binlogs are the backbone of replication. Treat them carefully.

sync_binlog=1
binlog_row_image=FULL
binlog_expire_logs_seconds=604800 # 7 days

sync_binlog=1 is the big one—without it, a crash can corrupt binlogs or the GTID position, and that leads to a very bad day.

Protect Your Replicas with super_read_only

Never allow accidental writes to replicas, in your my.cnf set:

read_only=ON
super_read_only=ON

super_read_only closes the loophole that even SUPER users could previously use to write to replicas.

Use a Dedicated Replication User

Give the minimal permissions:

CREATE USER 'repl'@'%' IDENTIFIED BY 'strong_password';
GRANT REPLICATION REPLICA ON *.* TO 'repl'@'%';

This user should do exactly one thing: replicate. Don’t reuse app users—you’re just begging for trouble.

Replication Lag: Watch It Like a Hawk

Seconds_Behind_Source lies more often than you’d expect. It’s okay for a quick glance but don’t rely on it.

Better options:

Performance Schema: replication_applier_status_by_worker
Percona Monitoring and Management (PMM)
Custom heartbeat tables
pt-heartbeat

Lag is one of the biggest causes of outages—monitor it continuously. Lag is usually the first sign something is wrong—catch it early.

Use Parallel Replication (But Don’t Overdo It)

If your primary has multiple writers or many concurrent transactions, in your my.cnf enable parallel workers:

replica_parallel_type=LOGICAL_CLOCK
replica_parallel_workers=4

4–8 workers is a sweet spot for most systems. More workers ≠ more speed; after a point it just increases memory footprint without real benefit.

But when it helps, it really helps—like cutting lag by 80–90%.

Use SSL Anywhere Outside the LAN

Replication traffic isn’t something you want exposed.

source_ssl=1
source_ssl_ca=/path/ca.pem

Earlier versions used the master_ssl_* variables, but the idea is the same: encrypt the connection when it leaves your trusted network.

Final Thoughts

MySQL replication can be rock-solid, but only if you follow a handful of rules that experienced DBAs know by heart:

Use GTIDs
Use RBR
Always have primary keys
Keep schemas aligned
Check for data drift
Harden binlog settings
Protect replicas from accidental writes
Monitor lag properly
Use parallel workers when appropriate
Encrypt connections over untrusted networks

Follow these, and your replicas will stay healthy, consistent, and (mostly) invisible—which is exactly how you want them.

Community Recap: Percona.Connect London 2025, Building the Future of Open Source Together

Edith Puclla — Tue, 02 Dec 2025 00:00:00 UTC

Percona.Connect London 2025 brought the open-source database community together for a half-day of learning and collaboration. The event focused on providing practical, technical insights for DBAs, DevOps engineers, and developers. The main takeaway was clear: Stability, Openness, and Automation are essential for modern, large-scale data infrastructure.

Top Discussions & Key Takeaways

1. The Rise of Valkey: A Truly Open Caching Alternative

Martin Visser, Valkey Technical Lead, explained the changes to the Redis license, the community needs a trusted, open-source replacement. Valkey was highlighted as the leading solution.

Valkey was started by former Redis contributors quickly after Redis removed its open source license in 2024.
It is a true open-source project governed under the Linux Foundation.
It offers enhancements like better memory efficiency, performance, and scalability.
In a recent Percona survey of 200 DBAs, Valkey was the most preferred alternative to Redis.

2. Running PostgreSQL in a Cloud Native context

Takis Stathopoulos, Enterprise Architect, presented on running PostgreSQL in a Cloud Native context, explaining how Kubernetes Operators simplify complex deployments.

Cloud Native vs. Cloud First: Cloud Native (Kubernetes) offers Portability and No vendor lock-in, allowing you to run the database consistently across different clouds and on-premise infrastructure.
Percona Operator for PostgreSQL: This tool automates crucial operations like setting up high availability (using Patroni), backups (using pgBackrest), and scaling.
When to use Cloud Native: It’s ideal for large, microservice-based applications and teams prioritizing portability and avoiding vendor lock-in.

3. Native PostgreSQL TDE is Here: Securing Data Simply

Alastair Turner, Postgres Community Advocate, introduced the new Native Transparent Data Encryption (TDE) for PostgreSQL.

4. The Future of MySQL: Vector Search & Binlog Server

Dennis Kittrell, MySQL Product Manager, discussed two key features planned for MySQL that address major operational and feature challenges.

MySQL Binlog Server MVP: This component aims to solve the problem of quick disaster recovery by acting as a stable, reliable replication source. It enables Precise Point-in-Time Recovery (PITR) using simple time or GTID coordinates.
Native Vector Support MVP: This feature allows users to eliminate the complexity of using a separate vector database. You can store, index, and search vector embeddings directly in MySQL, allowing you to combine vector searches with standard business logic in a single, transactional query

Our Community Focus

A common theme from the use cases was that while open source adoption is high, operational teams often lack the proper support and visibility.

Percona’s goal is to support the community by providing:

Stability when under heavy load or during maintenance.
Faster Troubleshooting with better monitoring and observability.
Safer Deployments through expert configuration and security support.

Thank you to everyone who joined us in London for a dynamic event. We hope the insights gained will help you with your open source database deployments.

The conversations continue in the Percona Community! You can reach out directly to the speakers:

Martin Visser (Valkey Technical Lead) LinkedIn
Dennis Kittrell (MySQL Product Manager) LinkedIn
Alastair Turner (Postgres Community Advocate) LinkedIn
Takis Stathopoulos (Enterprise Architect) LinkedIn
Andre Pons (Enterprise Sales Manager) LinkedIn

Join the Percona Community Conversation!

TDE is now available for PostgreSQL 18

Jan Wieremjewicz — Fri, 28 Nov 2025 11:00:00 UTC

Back in October, before PGConf.EU, I explained the issues impacting the prolonged wait for TDE in PostgreSQL 18. Explanations were needed as users were buzzing with anticipation, and they deserved to understand what caused the delays and what the roadmap looked like. In that blog post I have shared that due to one of the features newly added in 18.0, the Asynchronous IO (AIO), we have decided to give ourselves time until 18.1 has been released to provide a build with TDE. We wanted to ensure best quality of the solution and that takes time.

As planned in the PostgreSQL Development Group (PGDG) roadmap, on November 13, the Community released PostgreSQL 18.1, and Percona engineers managed not only to deliver TDE compatible with the PostgreSQL 18 changes, but to fully support them. We are also skipping PostgreSQL 18.0 entirely in our distribution. The first TDE enabled release will be Percona Distribution for PostgreSQL 18.1.1, aligning our builds directly with the first PostgreSQL 18 minor release. Here are some details on what’s new!

Percona PostgreSQL + TDE + AIO

Today, we’re proud to announce that Percona now ships PostgreSQL 18.1 with fully supported TDE and AIO from the very beginning.

No patching, no workarounds, no “experimental caveats.” Encryption-at-rest is not just compatible with AIO. It’s supported, integrated and ready for production deployments across the ecosystem, enabling hardened PostgreSQL environments to meet compliance requirements with confidence as beginning with PostgreSQL 18.1, Percona:

fully supports native TDE
ships AIO-enabled builds, aligned with Community PostgreSQL
provides production-ready packages for enterprise deployments

TDE matures and pg_tde returns home In 2026 we are planning significant investment to ensure that TDE continues to evolve for Community PostgreSQL. As part of this renewed focus, Percona is shifting pg_tde back to its dedicated home:

➡️ https://github.com/percona/pg_tde

This reflects pg_tde’s new role. With the growing number of pg_tde users, it can no longer be treated as a stopgap solution filling a gap in PostgreSQL. Instead, we want to approach it as a complementary, advanced encryption layer for the PostgreSQL 18+ era:

a place to deliver extended capabilities for data-at-rest encryption as soon as they are ready
a hub for integrations requested by the community
a safe space for feedback, comments, and questions to help drive the future of TDE

Releasing this new version brings some real improvements, especially on the KMS integration side.

KMS improvements

We expanded Key Management Service (KMS) capabilities based directly on user and customer feedback. Whether it came through GitHub, Percona Community Forums, events, or direct conversations — thank you! Your input shapes the roadmap.

pg_tde now works with Akeyless KMS

Akeyless is gaining traction among organizations implementing zero-trust security models. pg_tde now integrates cleanly with Akeyless, enabling robust key retrieval and lifecycle management across cloud and on-prem deployments using KMIP.

HashiCorp Vault & OpenBao Namespace Support

Vault and OpenBao users can now take advantage of namespaces when storing encryption keys

Why do namespaces matter?

Multi-tenancy: isolate key access for teams, environments, or applications
Security boundaries: each namespace can enforce its own authentication and audit policies
Cleaner CI/CD: dev/staging/prod can share a consistent key path structure
Delegation & separation of duties: security teams manage root policies, while application teams manage their own namespaces

In large organizations, namespace support isn’t just a convenience, it’s a requirement. OpenBao prioritized this early and released it back in May 2025. With pg_tde now supporting namespaces natively, PostgreSQL deployments gain enterprise-grade key management flexibility.

What Comes Next?

Our commitment remains unchanged: TDE must be a first-class, accessible, community-driven feature in PostgreSQL. Share your feedback and let us achieve this! We can build the future of an open source TDE solution for PostgreSQL together with full openness and transparency!

Winter may be coming, but no weather can stop us! Expect more from TDE soon:

Key length configurations are coming - expect to be able to use 256-bit keys with TDE soon!
extended KMS integrations - we’re already looking into cloud KMS support. Please share which ones you are using and what use cases you want supported!
improvements to pg_tde encryption targets - temporary files are next to be explored followed by system catalog
compatibility with Community PostgreSQL - we want to drive inclusion of any changes required by pg_tde in Community PostgreSQL so that the extension can run for users of all PostgreSQL builds!

⠀At the risk of repeating myself, the deserved highlight goes to the fact that we build based on your feedback, so don’t be strangers!

We Want to Hear from You

Tell us how you’re using PostgreSQL with TDE. Tell us what your security requirements look like. Tell us where tooling can be improved. You can reach us at:

Percona Community Forums https://forums.percona.com/c/postgresql/25
pg_tde GitHub Repo https://github.com/percona/pg_tde
Issues & Discussions
- https://github.com/percona/pg_tde/issues
- https://github.com/percona/pg_tde/discussions

Or, if you see us at an event, come chat with us in person!

The Right Tool for the Job

Wayne Leutwyler — Mon, 24 Nov 2025 00:00:00 UTC

When I first got into woodworking, my mentor shared a piece of advice that has stuck with me ever since: “Use the right tool for the job.” You wouldn’t reach for a belt sander to flatten a board when a planer can accomplish the task faster, cleaner, and with far better results.

The same principle applies in the world of database engineering. When working with MySQL or Percona Server, choosing the correct tool can be the difference between efficient diagnostics and unnecessary downtime.

In this post, I’ll highlight several of the most practical and commonly used utilities from the Percona Toolkit. While the toolkit includes many powerful commands, I’ll focus on the ones that provide the most value in day-to-day operations, troubleshooting, and gathering actionable details for support cases.

PT Summary

A Percona Toolkit utility that provides a concise, high-level overview of a system’s hardware, OS configuration and performance-related metrics. It’s designed to quickly capture the essential details needed for diagnostics or support cases—CPU, memory, disk layout, kernel parameters and more all in a single, easy-to-read report.

Example

Run pt-summary with no arguments to generate a full system summary. When possible, run it with sudo to allow the tool to collect additional details that require elevated privileges:

sudo pt-summary
# Percona Toolkit System Summary Report ######################
 Date | 2025-11-24 17:15:19 UTC (local TZ: EST -0500)
 Hostname | pi16gb
 Uptime | 41 days, 2:27, 4 users, load average: 0.00, 0.00, 0.00
 Platform | Linux
 Release | Debian GNU/Linux 12 (bookworm) (bookworm)
 Kernel | 6.12.47+rpt-rpi-2712
Architecture | CPU = 32-bit, OS = 64-bit
 Threading | NPTL 2.36
 SELinux | No SELinux detected
 Virtualized | No virtualization detected
# Processor ##################################################
 Processors | physical = 4, cores = 0, virtual = 4, hyperthreading = no
 Speeds |
 Models |
 Caches |
 Designation Configuration Size Associativity
 ========================= ============================== ======== ======================
# Memory #####################################################
 Total | 15.8G
 Free | 675.0M
 Used | physical = 5.3G, swap allocated = 512.0M, swap used = 0.0, virtual = 5.3G
 Shared | 44.7M
 Buffers | 10.6G
 Caches | 10.5G
 Dirty | 128 kB
 UsedRSS | 5.1G
 Swappiness | 60
 DirtyPolicy | 20, 10
 DirtyStatus | 0, 0
 Locator Size Speed Form Factor Type Type Detail
 ========= ======== ================= ============= ============= ===========
# Mounted Filesystems ########################################
 Filesystem Size Used Type Opts Mountpoint
 /dev/nvme0n1p1 510M 14%
 /dev/nvme0n1p2 458G 5%
 /dev/sda1 117G 16%
# Disk Schedulers And Queue Size #############################
 nvme0n1 | [none] 255
 sda | [mq-deadline] 60
# Disk Partitioning ##########################################
# Kernel Inode State #########################################
dentry-state | 107782 98346 45 0 32304 0
 file-nr | 3680 0 9223372036854775807
 inode-nr | 99614 20818
# LVM Volumes ################################################
Unable to collect information
# LVM Volume Groups ##########################################
Unable to collect information
# RAID Controller ############################################
 Controller | No RAID controller detected
# Network Config #############################################
 Controller | 00.0 Ethernet controller
 FIN Timeout | 60
 Port Range | 60999
# Interface Statistics #######################################
 interface rx_bytes rx_packets rx_errors tx_bytes tx_packets tx_errors
 ========= ========= ========== ========== ========== ========== ==========
 lo 6000000000 175000 0 6000000000 175000 0
 eth0 0 0 0 0 0 0
 wlan0 5000000000 30000000 0 15000000000 22500000 0
# Network Devices ############################################
 Device Speed Duplex
 ========= ========= =========
 eth0 Unknown! Unknown!
# Network Connections ########################################
 Connections from remote IP addresses
 192.168.1.91 1
 192.168.1.251 1
 2603 2
 Connections to local IP addresses
 192.168.1.145 2
 2603 2
 Connections to top 10 local ports
 3306 2
 6011:ef0:7260:::22 2
 States of connections
 ESTABLISHED 3
 LISTEN 6
 TIME_WAIT 1
# Top Processes ##############################################
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 95842 root 20 0 0 0 0 I 6.7 0.0 0:00.08 kworker+
 1 root 20 0 169520 13088 8672 S 0.0 0.1 0:18.56 systemd
 2 root 20 0 0 0 0 S 0.0 0.0 0:01.70 kthreadd
 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pool_wo+
 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 7 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
# Notable Processes ##########################################
 PID OOM COMMAND
 ? ? sshd doesn't appear to be running
# Simplified and fuzzy rounded vmstat (wait please) ##########
 procs ---swap-- -----io---- ---system---- --------cpu--------
 r b si so bi bo ir cs us sy il wa st
 2 0 0 0 1 6 100 150 0 0 100 0 0
 1 0 0 0 0 0 1750 3000 1 3 97 0 0
 1 0 0 0 0 0 250 400 0 0 100 0 0
 1 0 0 0 0 0 300 450 0 0 100 0 0
 1 0 0 0 0 0 300 450 0 0 100 0 0
# Memory management ##########################################
# The End ####################################################

Redirect output to a file.

pt-summary > server-summary.txt

PT MySQL Summary

A Percona Toolkit utility that collects and displays a concise overview of a MySQL or Percona Server instance, including key configuration settings, performance metrics, storage engine details, replication status, buffer pool usage, and important global variables. It provides a fast, structured snapshot of the database environment, making it ideal for troubleshooting, tuning, and preparing information for support teams.

Example

pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
 System time | 2025-11-24 17:45:54 UTC (local TZ: EST -0500)
# Instances ##################################################
 Port Data Directory Nice OOM Socket
 ===== ========================== ==== === ======
 3306 /data0/mysql/data/ 0 0 /usr/local/mysql/mysql.sock
# MySQL Executable ###########################################
 Path to executable | /usr/local/mysql/bin/mysqld
 Has symbols | Yes
# Report On Port 3306 ########################################
 User | wayne@localhost
 Time | 2025-11-24 12:45:54 (EST)
 Hostname | pi16gb
 Version | 8.4.6-6 Source distribution
 Built On | Linux aarch64
 Started | 2025-10-14 09:48 (up 41+02:57:35)
 Databases | 10
 Datadir | /data0/mysql/data/
 Processes | 2 connected, 2 running
 Replication | Is not a replica, has 1 replicas connected
 Pidfile | /usr/local/mysql/mysqld.pid (exists)
# Processlist ################################################

 Command COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 Binlog Dump GTID 1 1 3000000 3000000
 Query 1 1 0 0

 User COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 replication 1 1 3000000 3000000
 wayne 1 1 0 0

 Host COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 192.168.1.251 1 1 3000000 3000000
 localhost 1 1 0 0

 db COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 NULL 2 2 3000000 3000000

 State COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 init 1 1 0 0
 Source has sent all binlog to 1 1 3000000 3000000

# Status Counters (Wait 10 Seconds) ##########################
Variable Per day Per second 11 secs
Aborted_clients 1
Binlog_snapshot_position 350000 4
Binlog_cache_use 6000
Bytes_received 20000000 225 600
Bytes_sent 2250000000 25000 4000
[...]
Table_open_cache_misses 400
Table_open_cache_overflows 225
Threads_created 9
Uptime 90000 1 1
# Table cache ################################################
 Size | 1000
 Usage | 100%
# Key Percona Server features ################################
 Table & Index Stats | Disabled
 Multiple I/O Threads | Enabled
 Corruption Resilient | Enabled
 Durable Replication | Not Supported
 Import InnoDB Tables | Not Supported
 Fast Server Restarts | Not Supported
 Enhanced Logging | Disabled
 Replica Perf Logging | Disabled
 Response Time Hist. | Not Supported
 Smooth Flushing | Not Supported
 HandlerSocket NoSQL | Not Supported
 Fast Hash UDFs | Unknown
# Percona XtraDB Cluster #####################################
# Plugins ####################################################
 InnoDB compression | ACTIVE
# Schema #####################################################
Specify --databases or --all-databases to dump and summarize schemas
# Noteworthy Technologies ####################################
 SSL | Yes
 Explicit LOCK TABLES | No
 Delayed Insert | No
 XA Transactions | No
 NDB Cluster | No
 Prepared Statements | Yes
 Prepared statement count | 0
# InnoDB #####################################################
 Version | 8.4.6-6
 Buffer Pool Size | 8.0G
 Buffer Pool Fill | 30%
 Buffer Pool Dirty | 0%
 File Per Table | ON
 Page Size | 16k
 Log File Size | 2 * 48.0M = 96.0M
 Log Buffer Size | 64M
 Flush Method | O_DIRECT
 Flush Log At Commit | 2
 XA Support |
 Checksums |
 Doublewrite | ON
 R/W I/O Threads | 4 4
 I/O Capacity | 200
 Thread Concurrency | 4
 Concurrency Tickets | 5000
 Commit Concurrency | 0
 Txn Isolation Level | REPEATABLE-READ
 Adaptive Flushing | ON
 Adaptive Checkpoint |
 Checkpoint Age | 0
 InnoDB Queue | 0 queries inside InnoDB, 0 queries in queue
 Oldest Transaction | 0 Seconds
 History List Len | 1
 Read Views | 0
 Undo Log Entries | 0 transactions, 0 total undo, 0 max undo
 Pending I/O Reads | 0 buf pool reads, 0 normal AIO, 0 ibuf AIO, 0 preads
 Pending I/O Writes | 0 buf pool (0 LRU, 0 flush list, 0 page); 0 AIO, 0 sync, 0 log IO (0 log, 0 chkp); 1 pwrites
 Pending I/O Flushes | 0 buf pool, 0 log
 Transaction States | 3xnot started
# MyISAM #####################################################
 Key Cache | 8.0M
 Pct Used | 20%
 Unflushed | 0%
# Security ###################################################
 Users | 8 users, 0 anon, 0 w/o pw, 7 old pw
 Old Passwords |
# Encryption #################################################
No keyring plugins found
# Binary Logging #############################################
 Binlogs | 3
 Zero-Sized | 0
 Total Size | 437.9M
 binlog_format | ROW
 expire_logs_days |
 sync_binlog | 0
 server_id | 10
 binlog_do_db |
 binlog_ignore_db |
# Noteworthy Variables #######################################
 Auto-Inc Incr/Offset | 1/1
 default_storage_engine | InnoDB
 flush_time | 0
 init_connect |
 init_file |
 sql_mode | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
 join_buffer_size | 256k
 sort_buffer_size | 256k
 read_buffer_size | 128k
 read_rnd_buffer_size | 256k
 bulk_insert_buffer | 0.00
 max_heap_table_size | 16M
 tmp_table_size | 16M
 max_allowed_packet | 64M
 thread_stack | 1M
 log |
 log_error | /var/log/mysql/mysqld.log
 log_warnings |
 log_slow_queries |
log_queries_not_using_indexes | OFF
 log_replica_updates | ON
# Configuration File #########################################
 Config File | /etc/my.cnf

[mysqld]
character-set-server = utf8mb4
authentication_policy = '*'
port = 3306
socket = /usr/local/mysql/mysql.sock
pid-file = /usr/local/mysql/mysqld.pid
basedir = /usr/local/mysql/
datadir = /data0/mysql/data/
tmpdir = /data0/mysql/tmp/
general_log_file = /var/log/mysql/mysql-general.log
log-error = /var/log/mysql/mysqld.log
slow_query_log_file = /var/log/mysql/slow_query.log
[...]
innodb_data_home_dir = /data0/mysql/data/
innodb_log_group_home_dir = /data0/mysql/data/
innodb_temp_data_file_path = ../tmp/ibtmp1:12M:autoextend:max:8G
innodb_buffer_pool_size = 8G
innodb-redo-log-capacity = 2G
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_io_capacity = 200
innodb_buffer_pool_instances = 8
innodb_thread_concurrency = 4

# Memory management library ##################################
jemalloc is not enabled in mysql config for process with id 788
# The End ####################################################

Redirect output to file.

pt-mysql-summary > percona-server.txt

Capture both pt-summary and pt-mysql-summary into a single file.

pt-summary > percona-server-summary.txt
pt-mysql-summary >> percona-server-summary.txt

PT Online Schema Change

A Percona Toolkit utility that performs online ALTER TABLE operations by creating a shadow copy of the table, applying the schema change to that copy, and keeping it in sync with the original using triggers until it is ready to swap. This workflow minimizes locking and reduces downtime, allowing large production tables to be altered safely with minimal impact on applications. However, it’s important to remind users that long-running queries or transactions holding metadata locks (MDL) on the table will still block the final swap, potentially delaying completion of the schema change.

Examples

Adding a New Column

pt-online-schema-change \
 --alter "ADD COLUMN status TINYINT NOT NULL DEFAULT 0" \
 D=mydb,t=orders \
 --execute

This safely introduces a new column to a busy table without blocking reads or writes. The tool handles the copy, synchronization, and final table swap automatically.

Modifying a Column Type

pt-online-schema-change \
 --alter "MODIFY COLUMN price DECIMAL(10,2)" \
 D=shop,t=products \
 --execute

Changing column definitions—especially on large datasets—can be disruptive using standard SQL. With pt-osc, the migration happens online, keeping applications responsive throughout the operation.

Dropping an Unused Column

pt-online-schema-change \
 --alter "DROP COLUMN old_flag" \
 D=analytics,t=events \
 --execute

Column drops can require a full table rebuild, making them great candidates for pt-osc. This example removes a legacy column while avoiding table locks.

Adding an Index

pt-online-schema-change \
 --alter "ADD INDEX idx_user_id (user_id)" \
 D=app,t=logins \
 --execute

Index creation is another expensive operation for large tables. Here, pt-osc allows the index to be added online, improving performance without interrupting the application.

Changing a Primary Key

pt-online-schema-change \
 --alter "DROP PRIMARY KEY, ADD PRIMARY KEY(id, created_at)" \
 D=orders,t=order_items \
 --execute

Primary key modifications usually require a full table rewrite. pt-osc makes this process safer and easier on production systems by performing the change on a temporary shadow table.

Performing a Dry Run

pt-online-schema-change \
 --alter "ADD COLUMN test INT" \
 D=mydb,t=mytable \
 --dry-run

A dry run allows you to validate the plan and review the process without making any actual changes—a critical safeguard when preparing for production schema work.

Printing SQL Changes Before Execution

pt-online-schema-change \
 --alter "ADD COLUMN updated_at TIMESTAMP NULL" \
 D=crm,t=customers \
 --print \
 --execute

Using –print provides transparency into the SQL operations the tool will perform. This is particularly useful during code reviews or change-control processes.

PT Show Grants

A Percona Toolkit utility that extracts MySQL user accounts and privileges and outputs them as clean, executable CREATE USER and GRANT statements. It normalizes and orders the privileges for readability, making it valuable for auditing security, documenting access, migrating users between servers, or preparing accurate privilege information for support and compliance purposes.

Examples

Dump All Grants for All Users

The simplest and most common use case is generating a complete privilege snapshot:

pt-show-grants

This returns normalized CREATE USER and GRANT statements for every account in the instance. It’s ideal for audits, environment comparisons, and creating human-readable privilege reports.

Show Grants for a Specific User

If you want to inspect privileges for a single account, you can filter by user/host:

pt-show-grants --accounts='user@localhost'

This makes privilege debugging and user-level audits quick and targeted.

Export All Grants to a File

To create a reusable backup of every user account:

pt-show-grants > grants.sql

The resulting file is a set of CREATE USER and GRANT statements that can be restored simply by executing:

mysql < grants.sql

This is an excellent practice before server upgrades, user cleanup, or major permission changes.

Show Grants for Multiple Accounts

You can provide a comma-separated list of accounts to extract only what you need:

pt-show-grants --accounts='app@%,reporting@localhost,backup@localhost'

This is ideal for teams that manage groups of service accounts across environments.

Ignore Specific System Accounts

For cleanup scripts or custom inventory reports, skip built-in MySQL accounts:

pt-show-grants --ignore='mysql.sys@localhost,mysql.infoschema@localhost'

This focuses output on only the accounts relevant to your application.

Summary

This post highlights the importance of using the right tool for the job—both in woodworking and in database engineering. For MySQL and Percona Server environments, the Percona Toolkit offers a set of powerful utilities that simplify diagnostics, troubleshooting, schema changes, and security audits.

It introduces four key tools:

pt-summary – Generates a high-level report of system hardware, OS settings, filesystems, networking, and performance metrics. Useful for support cases and quick environment overviews.
pt-mysql-summary – Produces a structured snapshot of a MySQL instance, including configuration, performance counters, replication status, storage engine details, and important variables. Ideal for tuning and issue analysis.
pt-online-schema-change – Enables online ALTER TABLE operations by copying and syncing the table in the background, minimizing downtime. Several examples show how to add, drop, or modify columns and indexes safely.
pt-show-grants – Extracts all MySQL users and privileges into clean, reproducible CREATE USER and GRANT statements. Helpful for audits, migrations, backups, and security reviews.

Percona Operator for MySQL Is Now GA, More MySQL Options for the Community on Kubernetes

Edith Puclla — Wed, 19 Nov 2025 11:00:00 UTC

We’re excited to share that the new Percona Operator for MySQL (based on Percona Server for MySQL) is officially in General Availability (GA)!

This release introduces native MySQL Group Replication support for Kubernetes, providing our community with another open-source option for running reliable, consistent MySQL clusters at scale.

This is about more choices for the community. Each MySQL replication technology addresses different real-world needs, and now you can choose the one that best fits your workloads.

What This Means for the Community

With this release, Percona now supports two fully open-source MySQL Operators:

1. Percona Operator for MySQL (Percona Server for MySQL), New and GA

Group Replication (synchronous)
Asynchronous replication (Technical Preview)
Native MySQL experience
Auto-failover
Kubernetes-native design

2. Percona XtraDB Cluster Operator (PXC)

Galera-based synchronous replication
Strong high availability
Auto-failover
Battle-tested for mission-critical workloads

These Operators complement each other; they are not replacements. They give users the freedom to choose the right replication model for their business and technical priorities.

This GA release is a step in that direction, and we will continue publishing technical blog posts to explain when to use each Operator, how Group Replication works, and how this all fits into real-world Kubernetes environments.

Call for Community Testing and Feedback

Asynchronous replication is now available in Technical Preview, we invite you to:

Test it in your clusters
Share your feedback
Open GitHub issues
Contribute docs or examples

Your feedback will guide the next features we bring to the Operator.

Explore Percona Operator for MySQL:

OIDC in PostgreSQL: How It Works and Staying Secure

Zsolt Parragi — Mon, 17 Nov 2025 09:00:00 UTC

In the previous blog post about the topic, OAuth, OIDC and validators, we discussed basic terminologies to understand the differences between the protocols and how they relate to PostgreSQL.

In this second part, we’ll go one step further and see how OIDC works exactly in other software and in PostgreSQL, and what OAuthBearer is about. We also focus on the possible attacks and dangers in this flow with some examples to showcase why it’s important to use a properly configured secure provider and to teach our users not to just skim through the authorization process.

Can you keep a secret?

Even the original OAuth RFC was designed to work in many different situations, and later extensions made it even more generic to support more use cases. However, it also has to acknowledge that different setups have different requirements and limitations and, because of that, might require a different authorization flow.

In one of our previous examples, we used applications such as CloudStorage or EditorApp. We’ll continue to use them for a while for simplicity, but I want to emphasize that this example, even though it uses different names, is relevant and important for understanding how OAuth and OIDC work together with PostgreSQL.

For CloudStorage, we can safely assume it’s a traditional backend-frontend application, since it has to store the data somewhere on a backend.
For EditorApp, the answer can be different. Since it doesn’t store photos directly but retrieves/saves them on CloudStorage, and also doesn’t store users directly but retrieves them from the Provider using OIDC, it has no requirement for a backend. It is easily imaginable as a single page application, completely written in a modern JS framework, or even a downloadable version of it with Electron or a similar framework.

But why is this distinction important for us? Because we actually have to authenticate and authorize multiple actors here, not just users:

The applications, CloudStorage and EditorApp, have to trust the Provider to supply them with proper information about their users and their permissions.
The OAuth Provider has to identify that it’s speaking to CloudStorage or EditorApp to tell them what exactly they can do with their users.

OAuth or OIDC providers don’t just register users – they also register client applications, and these applications receive either a randomly generated or user-specified identifier (“username”), depending on the Provider in question. When we talk about authentication, that usually involves multiple credentials, such as a username and a password, not just a username. The OAuth standard also recognizes that passwords are good practice and lets client applications use them. This is called a client secret.

But are all applications equal in this sense? Can all of them keep secrets?

The answer to this question is, unfortunately, no. In our example, CloudStorage has a backend – this means it can keep its secret on the backend, never sending it to the frontend. This is how we usually treat our passwords, and this is what OAuth calls a confidential client.

EditorApp is, however, in a different situation. It’s an application purely implemented in HTML and JavaScript, without any backend code. Anybody can download its source, open it in a browser, and start using it. It’s not capable of hiding its secret. Even if it has a client secret – which is optional in OAuth exactly because of this situation – a user with sufficient knowledge can extract this secret from its code. And this is true even if it’s not a JavaScript application but rather a traditional desktop application written in a compiled language. As long as it doesn’t have a backend, a part hidden from its users, there’s no way to completely hide the secret. This is called a public client in OAuth.

Are you trying to log in to application “X”?

So how can the Provider make sure it’s talking to EditorApp and not to something else impersonating EditorApp? Without a secret, without the client being confidential, it can never be sure about this.

As long as we’re talking about web applications, which usually have URLs, it can try to minimize the chance of impersonation. Providers can make sure that the redirects done during the authorization flow go to URLs configured by the client administrators / owners. But even this can be circumvented, and in the case of desktop applications, it’s not usable.

This is why the Provider uses a different strategy. Since it can’t validate for itself that the thing claiming to be EditorApp is really EditorApp or something else, it informs the user. When the user clicks on the “Login with ” button, instead of silently authorizing the login, it first displays an information screen to the user. This is called the consent screen, and it has to display two things:

That user is trying to log in to application
That is requesting permissions to read and write

Along with this information, it has to ask the question: Is the above information correct? Is really trying to log in to ? Is okay with granting these permissions to ?

And only if the user, agrees can the request proceed.

Practical limitations

The above all sounds very nice and secure, but unfortunately we have to discuss some problems with it in practice.

Most importantly, users have a tendency to click “next, next, next” without properly reading, especially if they’re completing the same login flow for the 100th time. Even if a Provider implements the RFCs perfectly in the most secure way possible, it doesn’t help if the user just wants to get through the process quickly and doesn’t check the details about the application name and permissions properly.

Another issue is that providers don’t always properly follow the RFCs. For some providers, it’s configurable – administrators can choose when the consent screen is displayed. They can choose to only display it once when the user first logs into an application, to display it every time the user logs in to the application, or to never display it. Some providers allow these configuration settings for the permissions, for the application name, or both. Some providers simply never display permissions and always just say something like “be sure to trust this application ”.

Unfortunately, during our testing we found providers that in some cases completely skipped the consent screen, even when we tried to explicitly configure that it’s always required. Be skeptical about the authorization flow used by the Provider, verify that it’s secure enough, and don’t hesitate to change providers or report bug reports to the developers if something isn’t as it should be!

Limited devices

Everything we discussed above is nice, but PostgreSQL isn’t a website. Something using it might be one, and in that case, everything we discussed above applies – but to go back to the only currently supported client, psql, that’s a console application and can’t display a graphical web browser for login.

OAuth also thought about similar clients – not exactly console applications, but primarily devices that can’t display a browser and process a normal login flow to the Provider. These are called limited devices because the primary goal of this extension, RFC 8628, was to support specialized hardware where the user either can’t or doesn’t want to log in.

A typical example is a smart TV. While it can display a browser and a virtual keyboard, I wouldn’t want to type my password and log in to a TV using that. But these devices can be even more limited. The only requirement for them is that they should be able to display a verification code and instruct the user where (URL) to enter that verification code on another device, which is capable and secure enough to handle the normal password login process or where the user is already logged in.

Other than this indirection, the login process is similar. The user sees the code, opens the device login webpage on another device, enters the code, and then receives a similar consent screen as before. This consent screen states that device/application is trying to log in and requests permissions . The user then clicks the approve button, while in the background the device periodically checks for approval. Once approved, the authorization proceeds as normal on the limited device.

sequenceDiagram
participant Device as Limited Device
(psql)
participant Provider as OIDC Provider
participant Browser as User's Browser
(phone/laptop)
Device->>Provider: Request device code
Provider->>Device: Device code: ABC123
URL: provider.com/device
Note over Device: Display code and URL
to user
Browser->>Provider: Navigate to provider.com/device
Provider->>Browser: Show code entry form
Browser->>Provider: Enter code: ABC123
Note over Provider: Verify code is valid
Provider->>Browser: Show consent screen
"psql is requesting access"
Browser->>Provider: User clicks "Approve"
loop Polling every few seconds
Device->>Provider: Is code ABC123 approved?
Provider->>Device: Not yet...
end
Device->>Provider: Is code ABC123 approved?
Provider->>Device: Yes! Here's your access token
Note over Device: Token received,
authentication complete

While psql isn’t strictly a limited device, it’s in a similar situation. It’s possible that somebody is using it directly on a computer with a UI, with a web browser already logged in to the OIDC provider, but this is an unlikely scenario. More likely, the user has an SSH session open to another computer, running psql in it. Even if that remote computer has a graphical user interface and browser installed, displaying a login page on it wouldn’t help – the user wouldn’t see it. But it’s much more likely that it’s a console-only virtual machine / container somewhere, without a proper way to do a graphical login. In this sense, psql together with its environment is a limited device.

Before we talk about vulnerabilities and staying secure, it’s important to clarify something. OAuth/OIDC supports many different authentication workflows, including but not limited to the possibilities listed above. Support for specific flows also varies between vendors – the supported features and nuances vary from Provider to Provider.

But if a vendor supports a specific flow, that doesn’t mean it’s automatically usable by all clients. Every provider we tested so far lets the administrators configure which authorization flows they want to support. If a specific application only works with confidential clients, the best configuration is to disable anything else.

The PostgreSQL wire protocol (and server) doesn’t enforce the use of any specific flow, but a generic validator plugin also can’t assume that the server uses some specific configuration. This isn’t something the validator is capable of checking, as it gets executed on the server, after the authorization flow already concluded on the client side.

Strictly speaking, the server doesn’t even know the ID of the client (application), only the issuer URL.

And if anyone wants to be able to use the psql command, that only supports the limited device flow.

OAuthBearer

I already mentioned this in the previous blog post, and a few times in this one. It’s important to understand that the PostgreSQL project has multiple different roles in these authorization flows. The client and the server/validator are two different actors, and the client can be any client, not just libpq. It can be either a well-known third-party implementation of the wire protocol or a completely custom implementation created by an attacker with malicious intent.

From a security standpoint, the server/validator can’t trust the client or assume anything about the authentication process done by the client.

This design isn’t unique to PostgreSQL. Support for OAuth over non-HTTP protocols, using the SASL mechanism (which was already supported by PostgreSQL previously), is called OAuthBearer, RFC 7628, and it’s implemented by PostgreSQL similarly to how other software does it.

The idea of OAuthBearer is that the client using the non-HTTP service (in our case, PostgreSQL) creates an access token in some way. Then this client can use this access token to connect to PostgreSQL and possibly other services. Even if the client uses multiple services that all authenticate with OIDC, it only has to complete the OAuth flow once.

On the other side, the server doesn’t have to do anything else with OAuth other than validating that the token it received is correct and valid. It doesn’t have to deal with multiple flows. On the other hand, it can’t assume anything about the flow.

Can you please enter the code?

Why are the above details important, and why did I repeat the same description multiple times, worded slightly differently?

Because OIDC with public clients, even if implemented properly without mistakes, is vulnerable to the human factor. And in practice, most of the time, PostgreSQL and OIDC means using public clients.

There are multiple possible attack vectors that were previously used, and are still being used, to gain access to services, and there’s no way to fully secure against them. Both are enabled by the fact that with OAuthBearer, we have no control over the authentication process – we have to trust that the access token sent to us by the client was indeed created by the user with the intent to log in to this server.

One very simple “attack” against the device authentication flow is to ask the user nicely. Since the two parts of the process – requesting and using the device code, and verifying the device code – can happen at two different locations in a completely valid setup, there’s not much any software can do programmatically.

In this scenario, the attacker sends an email or calls the victim and comes up with some reason why the user has to go immediately to the device login page and enter the code. The reason is, of course, usually something completely unrelated, like “this is the code to join the meeting,” or “go to the website and enter this code to verify your account,” or anything similar. This is called device code phishing, and it’s made worse by the fact that some providers don’t display detailed enough consent screens during device code authentication, placing even careful users in vulnerable situations.

sequenceDiagram
participant Attacker
participant Provider as OIDC Provider
participant Victim as Victim's Browser
participant PG as PostgreSQL Database
Attacker->>Provider: Request device code
Provider->>Attacker: Code: XYZ789
URL: provider.com/device
Note over Attacker: Attacker now has
device code
Attacker->>Victim: Email: "Enter code XYZ789
at provider.com/device
to verify your account"
Victim->>Provider: Navigate to provider.com/device
Provider->>Victim: Enter device code
Victim->>Provider: Enter code: XYZ789
Note over Victim: User thinks they're
verifying their account
Provider->>Victim: Show consent screen
"psql requesting database access"
Victim->>Provider: Click "Approve"
(without reading carefully)
Provider->>Attacker: Access token granted!
Note over Attacker,PG: Attacker now has valid token
Attacker->>PG: Connect with stolen token
PG->>Attacker: Connection successful
Note over Attacker,PG: Attacker has full
database access

Unfortunately, this can’t be prevented as long as device code flow is enabled. It can be mitigated to some degree by educating users and making sure that the consent screen is always displayed and is very clear about the request details.

Client ID spoofing

Another easy OAuth/OIDC attack vector relies on the fact that we’re using a public client – meaning that either there isn’t a client secret, or even if there is one, it’s not really a secret. It’s easy to create a legitimate-looking website that uses OIDC for something else and starts an authorization flow… but in the background, it uses the same client credentials as the PostgreSQL client.

This again relies on either a non-securely configured consent screen or a careless user who doesn’t read the details about the request. And compared to the previous example, this doesn’t require the device flow – it can work with other flows too. In the world of LLMs, it’s very easy to create a valid-looking website tailor-made just for this purpose.

sequenceDiagram
participant Victim as User
participant Fake as Fake Website
"Photo Gallery"
participant Provider as OIDC Provider
participant PG as PostgreSQL Database
Victim->>Fake: Visit fake-photo-gallery.com
Fake->>Victim: "Login with your account"
Note over Fake: Uses REAL psql client ID!
(public, can't be hidden)
Victim->>Fake: Click "Login"
Fake->>Provider: OAuth request with
psql client ID
Provider->>Victim: Show consent screen
"psql requesting access"
Note over Victim: User thinks:
"I'm logging into
Photo Gallery"
Victim->>Provider: Click "Approve"
Provider->>Fake: Access token
Note over Fake: Fake site now has
database token!
Fake->>PG: Connect with token
PG->>Fake: Connection successful
Note over Fake,PG: Attacker has database access

Similarly to the previous example, there’s not much we can do to prevent this in the plugin. OAuth has extensions that aim to make the process more secure, such as PKCE (RFC 7636) or DPoP (RFC 9449), preventing specific situations, but none of those help with OAuthBearer. As the entire authorization process happens on the client side, the validator on the server can’t do anything but assume that if the access token is valid, then the user created it intentionally.

Educate your users!

Once more I’d like to emphasize that while we can’t prevent these attack vectors completely, it is possible to minimize the risk by teaching, both your users and administrators.

Teach your users to:

Never enter device codes unless you initiated the login process yourself
Always read the consent screen carefully, even if you’ve seen it before
Verify the application name matches what you’re trying to access
Be suspicious of unexpected emails or messages asking you to enter codes
When in doubt, don’t approve – contact your administrator instead

And your administrators to:

Understand that OIDC Provider selection and configuration isn’t just a checkbox item
If a provider can’t be configured in a secure way, it presents a security vulnerability to any client using it
Always enable and require consent screens for public client flows, for every login, not just for registration
Make sure the consent screen clearly displays the application name and requested permissions

These attacks aren’t theoretical – they’re actively used in the wild. The combination of OAuthBearer’s trust model and the inherent limitations of public clients means that proper provider configuration and user awareness are your primary defenses.

Next steps

Now that we understand how OIDC works with PostgreSQL and the security considerations involved, it’s time to see this in practice.

In the next blog post, we’ll walk through setting up Keycloak with PostgreSQL from scratch. We’ll cover both getting Keycloak running and configuring it securely for PostgreSQL authentication, even if you’ve never worked with Keycloak before. You’ll see exactly how to configure the settings we discussed here and how to test that everything works correctly.

Stay tuned for practical, hands-on setup instructions!

PGScorecard - PostgreSQL Compatibility Index

Kai Wagner — Thu, 13 Nov 2025 00:00:00 UTC

We’re excited to share that our recent test run using the Postgres Compatibility Index (PCI) achieved 100% compatibility.

The PCI was created to bring clarity to the often used but loosely defined term “PostgreSQL compatible.” As Mayur explains in his article The Making of ‘Postgres Is’, the goal is simple: to ensure that when a system claims to be compatible with PostgreSQL, it truly behaves like upstream PostgreSQL in practice. The PCI accomplishes this by running a comprehensive set of tests across features like data types, procedural functions, constraints, extensions, and more, and producing a measurable, transparent score. This gives users and vendors a reliable benchmark rather than relying on marketing claims.

Compatibility matters because many organizations rely on PostgreSQL variants, repackaged distributions, or vendor supported systems. They want the confidence that their schemas, tools, extensions, ORMs, client libraries, and workflows will continue to work as expected. A system that drifts from upstream PostgreSQL can introduce subtle risks such as, unsupported features, migration challenges, and vendor lock-in.

Achieving a perfect PCI score means our system supports the full baseline feature set as currently defined. It demonstrates that users can rely on the same behavior as community PostgreSQL, whether they are self-hosting, using a vendor-supported version, or integrating with existing tools. Importantly, it also shows that you can have a fully open-source system while still benefiting from vendor support, without compromising compatibility.

In a world where “PostgreSQL compatible” is often a vague claim, initiatives like the PCI provide the needed help with transparency and comparability. It helps to protect you from marketing claims in the PostgreSQL ecosystem, ensuring that tooling, and workflows continue to function reliably.

Special thanks to Mayur, whose initiative is helping define a clear, standardized framework for PostgreSQL compatibility.

The test run was done against Percona Server for PostgreSQL 17.6.1. Click for test result. Percona Server for PostgreSQL is a binary-compatible, open source drop-in replacement for PostgreSQL with the currently needed enhancements, to make Transparent Data Encryption (TDE) work.

MySQL Memory Usage: A Guide to Optimization

Roman Agabekov — Tue, 11 Nov 2025 00:00:00 UTC

Struggling with MySQL memory spikes? Knowing how and where memory is allocated can make all the difference in maintaining a fast, reliable database. From global buffers to session-specific allocations, understanding the details of MySQL’s memory management can help you optimize performance and avoid slowdowns. Let’s explore the core elements of MySQL memory usage with best practices for trimming excess in demanding environments.

How MySQL Uses Memory

MySQL dynamically manages memory across several areas to process queries, handle connections, and optimize performance. The two primary areas of memory usage include:

Global Buffers

These are shared by the entire MySQL server and include components like the InnoDB buffer pool, key buffer, and query cache. The InnoDB buffer pool is particularly memory-intensive, especially in data-heavy applications, as it stores frequently accessed data and indexes to speed up queries.

Connection (per thread) Buffers

When a client connects, MySQL allocates memory specifically for that session. This includes sort buffers, join buffers, and temporary table memory. The more concurrent connections you have, the more memory is consumed. Session buffers are critical to monitor in high-traffic environments.

Why MySQL Memory Usage Might Surge

Memory spikes in MySQL often result from specific scenarios or misconfigurations. Here are a few examples:

High Traffic with Large Connection Buffers: A surge in concurrent connections can quickly exhaust memory if sort or join buffers are set too large.
Complex Queries: Queries with large joins, subqueries, or extensive temporary table usage can temporarily allocate significant memory, especially when poorly optimized.
Oversized InnoDB Buffer Pool : Setting the InnoDB buffer pool size too large for the server’s available memory can trigger swapping, severely degrading database and server performance.
Large Temporary Tables : When temporary tables exceed the in-memory limit ( tmp_table_size ), they are written to disk, consuming additional resources and slowing down operations.
Inefficient Indexing : A lack of proper indexes forces MySQL to perform full table scans, increasing memory and CPU usage for even moderately complex queries.

Best Practices for Controlling MySQL Memory Usage

When you notice MySQL using more memory than expected, consider the following strategies:

1. Set Limits on Global Buffers

Configure innodb_buffer_pool_size to 60-70% of available memory for InnoDB-heavy workloads. For smaller workloads, scale it down to avoid overcommitting memory.
Keep innodb_log_buffer_size at a practical level (e.g., 16MB) unless write-heavy workloads demand more.
Adjust key_buffer_size for MyISAM tables, ensuring it remains proportionate to table usage to avoid unnecessary memory allocation.

2. Adjust Connection Buffer Sizes

Reduce sort_buffer_size and join_buffer_size to balance memory usage with query performance, especially in environments with high concurrency.
Optimize tmp_table_size and max_heap_table_size to control in-memory temporary table allocation and avoid excessive disk usage.

3. Fine-Tune Table Caches

Adjust table_open_cache to avoid bottlenecks while considering OS file descriptor limits.
Configure table_definition_cache to manage table metadata efficiently, especially in environments with many tables or foreign key relationships.

4. Control Thread Cache and Connection Limits

Use thread_cache_size to reuse threads effectively and reduce overhead from frequent thread creation.
Adjust thread_stack and net_buffer_length to suit your workload while keeping memory usage scalable.
Limit max_connections to a level appropriate for your workload, preventing excessive session buffers from overwhelming server memory.

5. Track Temporary Table Usage

Monitor temporary table usage and reduce memory pressure by optimizing queries that rely on GROUP BY, ORDER BY, or UNION.

6. Use MySQL Memory Calculator

Incorporate tools like the MySQL Memory Calculator by Releem to estimate memory usage. Input your MySQL configuration values, and the calculator will provide real-time insights into maximum memory usage. This prevents overcommitting your server’s memory and helps allocate resources effectively.

7. Monitor Query Performance

High-memory-consuming queries, such as those with large joins or sorts, queries without indexes, can affect memory usage. Use Releem’s Query Analytics and Optimization feature to determine inefficient queries and gain insights on further tuning opportunities.

Simplifying MySQL Memory Tuning with Releem

Releem takes the guesswork out of MySQL optimization by automatically analyzing your setup and suggesting configuration changes that align with your memory limits and performance needs. Whether you’re dealing with complex workloads or simply don’t have time for manual adjustments, Releem makes it easier to keep MySQL running smoothly.

A thread through my 2025 Postgres events

Alastair Turner — Mon, 10 Nov 2025 07:00:00 UTC

I recently got back from PostgreSQL Conference Europe in Riga, marking the end of my conference activities for 2025. The speakers were great. The audience, for the Extensions Showcase on Community Day on Tuesday and my Kubernetes from the database out talk, were great. The event team was great. The singing at karaoke was terrible, but it’s supposed to be.

After attending a good few events this year, starting with CERN PGDay in mid-January, I wanted to write something about more than just the most recent event. I see a common thread across presentations and sessions at a number of events over the year, that is, scale-out Postgres and particularly, its use in non-profit scientific environments.

The (beginning and) end users

Far fewer data processing challenges require pooling the resources of many physical servers these days, with servers getting bigger and storage faster. Scientific data analysis and managing large, complex scientific facilities still do. I saw three presentations on this: Rafal Kulaga, Antonin Kveton and Martin Zemko’s on managing CERN’s SCADA data; Daniel Krefl and Krzysztof Nienartowicz at CERN on how Sendai queries variable star data; and Jaoquim Oliveira in Riga on managing the European Space Agency’s (ESA’s) survey mission data.

I admit a fondness for ESA’s GAIA catalog dataset. After I was lucky enough to do a proof of concept project on joining it with other catalog data, it has provided significant intellectual interest. Don’t let me get started on the possible ways to optimise computationally expensive inequality joins on horribly skewed data, unless you really care about the problem. My interest in a dataset discussed in two of these talks is not why the thread connecting them is worth commenting on. All three presentations had a lot of content on selecting or developing database technologies for the work they were doing. That’s worth discussing a bit further.

Getting the details right

The thread of sharded, scale out, or Massively Parallel Processing (MPP) Postgres connects end user stories at my first event of the year and my last, along with stories of building this software at events in between. At PGConf.dev in Montreal David Wein gave a very condensed explanation of how AWS’s Aurora Limitless handles distributed snapshot isolation (watch the lightning talk at on YouTube), there was also an unconference session on handling the issue in core Postgres the next day. For an in-depth explanation of of what the distributed snapshot problem is and how it may be addressed, see Ants Aasma’s talk from PGConf.EU 2024

The organisations with the data are looking for open source software solutions and bumping into issues around open core licensing, project contribution breadth, project activity levels, project governance. The Postgres developer community is working on the knottiest of the problems in this space, trying to get it absolutely right. In the mean-time, various forks and extensions are delivering useful functionality for the owners of these big, complex datasets. Useful, but could do better

If this were working out for everyone, there wouldn’t be a story to tell. Sednai are building Potgres-XZ, which builds on TBase, which built on Postgres-XL. The ESAC Science Data Centre (ESDC) is facing a decision between two single-vendor projects, where one vendor doesn’t provide support for on-premises deployments. CERN procurement sought written assurances over license terms for TimescaleDB, since the CERN facilities organisation may be viewed as a service provider to their hosted scientific projects.

This pattern of licenses built specifically to avoid “AWS stealing our innovation/lunch/…”, (and it is always AWS set up as the bogeyman in these stories), is particularly unfortunate here, because it just isn’t true for Postgres. AWS, and Azure, employ big teams of community contributors to work on open source Postgres. The progress on statistics management, asynchronous IO, and vacuum in Postgres 18 are, among others, thanks to these teams’ efforts.

No matter how positive the involvement of the hyperscalers may be for Postgres, there are organisations who will prefer to run their own databases. On-premises hosting is a clear choice for organisations with big facilities capabilities, capital-centric budgeting, extreme requirements and predictable, always on workloads. Many of these organisations are publicly funded scientific projects. It would be great if there were broad-based open source solutions to meet their data management needs.

Doing better, together

At PGConf in Riga the Percona team took a few, early steps towards building a joint effort to deliver the components of such a solution. I hope that the big, open managers of structured scientific data (or their subcontractors, depending on their engagement model) and a few vendors can come together to build event data compression, columnar storage, and all the other bits which can be implemented as extensions.

The current Postgres extensions and forks for scale out systems were built on older versions of Postgres, so they had to build features which now exist in core Postgres. Their implementation of partitioning, for instance, differs subtly from the capabilities now available in modern Postgres. As feature-specific extensions can take over capabilities which are currently intertwined with sharding (like compression in Timescale or columnar storage in Citus), users will be less locked in to vertical stacks of features, some useful to them and some not. Simple sharding can then become a proxy (like pgDog), an automation on DDL on a gateway server or even a core Postgres feature.

Which leaves those special cases where moving data between shards during execution is key to performance. This is mattering less with ever bigger servers, improving Postgres parallelism and tools like DuckDB - but when it matters it still really matters. Here the sons of the ‘plum - CloudberryDB and WarehousePG, forked from Greenplum when it closed source - work their magic (hat tip to Jimmy Angelakos for the “the ‘plum” contraction). Managing that particular capability will always be a big, complex code base. If the patches carried to make it happen shrink as Postgres and extensions fill the gap, we’ll have a more sustainable route to all good database things being openly available.

OAuth, OIDC, validators, what is all this about?

Zsolt Parragi — Fri, 07 Nov 2025 10:00:00 UTC

Somebody might tell you, “let’s configure PostgreSQL 18 with OIDC, it should be simple, only takes a few minutes!” And that might be the case if you already have an OIDC provider set up and know all the details about the protocols, configurations, and possible issues. Or it might take much longer if you just open your favorite search engine and type “What is this OIDC stuff about?”

In this series of blog posts, I’ll try to help with this task. First, by clearing up all the terminology and details in this article. Later, I’ll provide vendor-specific setup instructions for some of the popular providers, using our fully open source pg_oidc_validator plugin.

OAuth 2.0, OIDC, what’s even the difference?

From news and other sources you might have heard that PostgreSQL 18 now has support for OIDC. But if you look at the PostgreSQL documentation about it, or the variables/configuration options PostgreSQL provides, it’s clear that is about OAuth everywhere.

People often use them interchangeably, because the two are closely related: OIDC is built on top of OAuth. However, they serve different purposes.

flowchart TB
OAuth[OAuth 2.0
Authorization Protocol]
OIDC[OpenID Connect OIDC
Authentication Layer]
OAuth --> OIDC
OAuthQ["Can user X access
resource Y?"]
OIDCq["Who is this user?"]
OAuth -.->|Answers| OAuthQ
OIDC -.->|Answers| OIDCq
style OAuth fill:#e3f2fd
style OIDC fill:#f3e5f5
style OAuthQ fill:#fff,stroke:#1976d2,stroke-dasharray: 5 5
style OIDCq fill:#fff,stroke:#7b1fa2,stroke-dasharray: 5 5

Authorization is not Authentication

The “Auth” in OAuth is about Authorization, not Authentication as people often believe – and specifically about remote, distributed authorization involving multiple applications. For example, somebody might store pictures on some cloud storage, and wants to use a photo editing application written and maintained by a different company. This EditorApp reaches out to the CloudStorage, and says “Can you provide me the pictures please?” After this question, CloudStorage has to figure out if it is allowed to do so or not.

OAuth was designed to handle situations like this – managing permissions in a complex online environment. However, EditorApp might also ask the question, “Hey, I want to display the name of the user I’m working with. Can you tell me who I am working with?” And now we’ve leaped into the context of Authentication. OIDC, or OpenID Connect, is a protocol built on top of OAuth to answer questions like this – to provide information about who the user is.

Which of them do we need with PostgreSQL?

The question with PostgreSQL and similar software using OAuth/OIDC is a bit different: we want to answer, “somebody is trying to log in – can I allow this login to proceed?”

Is this the right question, or am I oversimplifying things? Shouldn’t we be asking, “who is trying to log in?” Sometimes we do. The most common setup will likely ask both: “can this login proceed, and if yes, who is the user?”

But not necessarily always. It’s perfectly valid to configure a server to use this login flow only for administrators, where anybody who is allowed to use it gets associated with an internal admin account. In this scenario, we don’t care about the identity provided by the external provider: if the user has a valid access token, we treat them as our admin user and proceed accordingly.

To handle this specific workflow, OAuth is enough: all we have to do is check if the access token is allowed to access the PostgreSQL server, and if yes, the login can proceed as the admin user. This is however a limited, specific use case, not the generic scenario, where we also want to figure out who is the user on the provider side.

And there are also limitations on the server and client side:

When using our validator plugin, it supports the more generic scenario. It has to work with user identities from the provider, so it requires OIDC features. That’s why we called it pg_oidc_validator and not pg_oauth_validator.
While the server and wire protocol can work with any OAuth flow, currently the only client that implements a login mechanism using it is libpq (and with that, the psql command). And while the psql command also uses parameters with OAuth in their name, internally it relies on a feature called OIDC discovery – which, as the name suggests, is part of the OIDC standard, not OAuth.

To summarize: in practice, right now anyone who wants to log in to a PostgreSQL server using OAuth and psql has to use a provider that also supports the OIDC protocol. In practice this isn’t a restriction, since OIDC is commonly supported.

Why do we need this validator?

Why do we need to use a plugin to validate things? Many websites and apps implement login with OIDC providers, and they don’t need separate validators, so: Why can’t PostgreSQL do everything internally in the core?

To answer these questions, we have to understand that PostgreSQL in this context isn’t an application in itself – it’s a part of a complex infrastructure. Even libpq, mentioned above, isn’t strictly part of the picture. There are completely independent implementations of the PostgreSQL wire protocol, which can all implement the client-side flow completely differently.

This bigger system that happens to use PostgreSQL might also use other services that need authorization. The EditorApp in our earlier example might use PostgreSQL as its database. And that means, after authenticating the user, it has to talk to two different services:

CloudStorage, to access the photos on behalf of the user
PostgreSQL, to access the database on behalf of the user

Since users like seamless experiences, the developers of EditorApp want to log the user in only once, internally as part of their application, and then forward this information to both CloudStorage and PostgreSQL.

sequenceDiagram
participant User
participant EditorApp
participant Provider
participant CloudStorage
participant PostgreSQL
User->>EditorApp: Login
EditorApp->>Provider: Request OAuth/OIDC authentication
Provider->>User: Show login page
User->>Provider: Enter credentials
Provider->>EditorApp: Return access token
Note over EditorApp: User authenticated once,
token used for multiple services
EditorApp->>CloudStorage: Access photos (with token)
CloudStorage->>Provider: Validate token
Provider->>CloudStorage: Token valid
CloudStorage->>EditorApp: Return photos
EditorApp->>PostgreSQL: Connect to database (with token)
PostgreSQL->>PostgreSQL: Validator checks token
PostgreSQL->>EditorApp: Connection established

This showcases an important difference compared to “simple” websites and applications: PostgreSQL doesn’t own the entire authentication flow. It receives information – what the OAuth standard calls an access token – and needs to use this to complete its internal authentication/authorization flow. To do this, it has to validate the access token and answer the question: “was this token really created by the issuer I trust?”

There’s nothing new about this question. Big cloud providers like Google, Microsoft, and many others all do SSO (Single sign-on). If you log in to their webmail service, you can also open their cloud storage, calendar, or other services, and you’ll be similarly authenticated and authorized.

However, there is one very important difference in the above example compared to PostgreSQL: they all only work with their own users and their own tokens.

You might think this is still an easy problem: OAuth and OIDC are standards, so all we have to do is read the related parts about how validation works and implement our validation flow accordingly. Except that the standards don’t say anything about it.

These standards never define what an access token is. It can be something completely opaque, only interpretable by the original issuer. Or it can be something completely transparent – a proper JSON document with a digital signature that guarantees it was issued by the issuer, usually called a JWT (JSON Web Token).

Many OAuth providers implement JWT tokens, but not all of them. And even with JWT tokens, there are differences between providers. Since it’s not standardized, the content might differ between vendors. There’s also the question of signature validation, where some providers have differences and require special handling.

And this is where a plugin comes into the picture. The PostgreSQL maintainers decided they don’t want to add vendor-specific code into the core. Implementing JWT access tokens with the most commonly used structures could have been a solution, but that would have meant PostgreSQL had no way to support providers that were completely standards-compliant but implemented access tokens differently.

Even with JWT tokens, we have to think about token revocation. JWT tokens have a specified lifetime. When we validate them strictly based on the digital signature, we also have to check if we’re still within the allocated time. The protocols have a mechanism for refreshing these tokens so that users can stay logged in without repeating the process every time the tokens expire.

But sometimes administrators or the security team might discover a breach – that somebody got hold of an access token that’s still valid for hours or days – and decide to revoke it. They can’t change the tokens already circulating the network, but they can tell their authorization server to start rejecting the token, even if it’s still within its allowed lifetime.

If a validator relies solely on validating the signature of the token, it will still authorize users with these revoked tokens as long as their lifetime allows. If it performs a request to the server to check if the token is still usable, it might provide a more secure experience at the cost of additional HTTP requests. Alternatively, some providers allow applications to subscribe to broadcasts where they announce revocations – so instead of actively querying each token, they can keep a list of which tokens they aren’t allowed to authorize, as this list is typically short.

Unfortunately, with these questions we’ve again arrived at “vendor-specific” territory, where a validator has to work differently for different services. It’s also a user choice:

On a service where logins are rare but security is a top priority, it might make sense to do explicit queries every time.
But on a server processing thousands of login requests every second, these additional requests might slow things down too much. Administrators might decide not to do them, or to cache the results for some period instead.

With all these details and choices, it’s definitely better to leave the options open so everyone can select a validator suited for their specific needs.

What does pg_oidc_validator offer?

The Percona validator in its current form focuses on providers working with JWT tokens. We can’t guarantee it works with all providers using JWT tokens – as mentioned above, the exact format of these tokens isn’t standardized – but we implemented logic that can handle all the providers we tested, in some cases allowing customization with configuration variables.

In the future, we might add support for providers using opaque tokens or additional optional checks for token revocation, but these weren’t in scope for the first version.

If you try it out, and find any issues, or have any suggestions, miss some features, please provide us feedback on Github!

The future

OAuth support in PostgreSQL is new and only implements the bare minimum. Client and application support is also in its early days – adoption will take time. It can already be useful for some use cases, but it might still be too limited for others.

I can’t predict how the code will change in the server itself, but there are certainly many improvements that can be made. I also want to avoid confusion and clear up some things related to my previous examples – things that aren’t currently supported but might be in the future:

Even if a validator plugin checks for revocation, that check is only done during the login process. If a user is already logged in to PostgreSQL and the token is revoked, the user will stay logged in – the validator can’t evict them from the server.
Similarly, while the protocol has this concept of access token lifetime and how applications using the token can refresh it, this is currently ignored by PostgreSQL. Tokens are validated during the login process, and we check the validity, including the lifetime of the token, during that time. We don’t try to refresh these tokens later, and we don’t evict users when the token expires – currently there’s no infrastructure for that.
In the examples above, CloudStorage was able to decide which photos EditorApp can access. OAuth is about authorization, and it could be used for authorization within PostgreSQL – but this isn’t the case currently. Validators have to map access tokens to internal users present in the PostgreSQL database. Other than this, the information provided by the OAuth Provider has no interaction with the internal permissions within PostgreSQL (grants).
In the example above, we described a situation where an application used PostgreSQL together with other services using the same OAuth provider. While this is a possible use case, it’s also possible that an application only needs an OIDC login flow to access PostgreSQL. In that case, it would be possible to provide better security guarantees by integrating the login flow deeply into PostgreSQL, especially when using more recent OAuth/OIDC extensions such as PKCE (Proof Key for Code Exchange).

Similarly, our pg_oidc_validator is a pre-release prototype. While it already performs JWT-based authentication and authorization, it currently doesn’t support opaque token providers at all, has no internal caches, and doesn’t check for revoked tokens. Also, some of the features mentioned above would be better with integrated core support, but could technically also be implemented as part of a validator plugin. This would provide the additional benefit that users wouldn’t have to wait for newer PostgreSQL versions for the new feature – it would be enough to upgrade the validator plugin to a newer version.

So there are lots of improvement opportunities for both parts. Stay tuned and follow the changelogs!

Encryption support in PMM Dump

Sveta Smirnova — Thu, 30 Oct 2025 11:00:00 UTC

The pmm-dump client utility performs a logical backup of the performance metrics collected by the PMM Server and imports them into a different PMM Server instance. PMM Dump allows you to share monitoring data collected by your PMM server with the Percona Support team securely.

Up until now dumps, created by the tool, were not encrypted. It was possible to encrypt them after they are done but this required additional actions from the user.

Starting from the upcoming PMM Dump version 0.8.0-ga released on October 29, 2025, dumps are encrypted by default.

Key points

Dump files are encrypted by default with AES-256-based encryption.
An auto-generated password is produced for each encrypted dump; it is printed at the end of the export operation or can be written to a file with --pass-filepath.
You can provide a custom password with --pass.
Disable encryption with --no-encryption only when you understand the risks.
By default, for encrypted dumps, export logging to STDOUT is suppressed; use --no-just-key to override.

Why this matters

Encrypting PMM dumps prevents accidental exposure of monitoring and query data that may contain sensitive information (query text, hostnames, metrics). It brings PMM Dump in line with secure data-handling best practices and simplifies safe sharing with Percona Support.

Quick examples

Export (encryption enabled by default):

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --allow-insecure-certs
...
Password: ****************
$ ls pmm-dump-.tar.gz.enc

Provide a custom password:

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --pass='My$trongP@ss'

Save auto-generated password to file:

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --pass-filepath=/tmp/pmm-dump.pass

Disable encryption (not recommended):

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --no-encryption

Import an encrypted dump:

$ pmm-dump import --pmm-url='https://admin:admin@127.0.0.1' --allow-insecure-certs \
--dump-path=pmm-dump-1758017090.tar.gz.enc --pass='My$trongP@ss'

Decrypt an encrypted dump (if needed):

$ openssl enc -d -aes-256-ctr -pbkdf2 -in dump.tar.gz.enc -out dump.tar.gz

Recommendations

Prefer leaving encryption enabled.
Use --pass-filepath to store passwords securely rather than relying on terminal output.
Transfer encrypted archives over secure channels (SCP/SFTP) and share passwords via secure out-of-band channels.

Availability

Encryption support is included starting in the recent PMM Dump 0.8.0-ga release. Check your PMM Dump version (pmm-dump version) and the docs for exact version details.

Additional information

Say Hello to OIDC in PostgreSQL 18!

Jan Wieremjewicz — Wed, 22 Oct 2025 11:00:00 UTC

If you’ve ever wondered how to set up OpenID Connect (OIDC) authentication in PostgreSQL, the wait is almost over.

We’ve spent some time exploring what it would take to make OIDC easier and more reliable to use with PostgreSQL. And now, we’re happy to share the first results of that work.

Why OIDC, and why now?

We’ve spoken to some of our customers and noticed a trend of moving away from LDAP to OIDC. Our MongoDB product is already providing OIDC integration and the team working on PostgreSQL products saw an opportunity coming with PostgreSQL 18.

As some of you may have noticed PostgreSQL 18 has introduced improvements to authentication that include OAuth 2.0 support. It’s a small step from OAuth 2.0 to OIDC, which builds directly on top of it. So we set out to test PostgreSQL 18’s integration with one of the most common OIDC providers: Okta.

That’s when we discovered a missing piece. While PostgreSQL 18 includes the underlying support for OAuth 2.0, it still lacks a validator library required to successfully complete an OIDC configuration. So… we built it.

Closing the gap for enterprise needs

Percona is known for our Support, Managed, and Consulting services for open source databases. But while our services are commercial, our mission remains deeply open source.

We don’t differentiate users by the size of their wallets. We want everyone using open source databases to succeed, grow, and benefit from the same quality foundations that enterprises rely on. Services is what finances our open source investments, what we believe is truly honest and sustainable open source development.

OIDC is a great example of how real-world enterprise needs and community innovation come together. PostgreSQL 18 introduced OAuth thanks to community efforts, but OIDC that helps organizations handle compliance and scale was still not supported. With the work on OIDC validator we want to close the gap between these two while keeping the solution open source.

The power is in testing

It would be great if integrating with one OIDC provider meant it works with all of them. Unfortunately, real world implementations differ, sometimes significantly. That’s why compatibility testing matters. That’s also why often particular providers we find require independent approach and some custom handling in the code.

So far, we’ve done some preliminary tests of the library. It’s not production-ready yet. It’s a first release that’s shared with users and Community to get feedback while in the meantime our engineers will put it through rigorous testing regimen. We’ve so far tested the validator library compatibility with:

✅ Okta

✅ Ping Identity

✅ Keycloak

✅ Microsoft Entra ID (Azure AD)

We’re aware it’s not yet compatible with Google’s OIDC implementation, which has a few unique quirks, but that’s on our roadmap for future work. The broader the testing, the stronger the solution. This is where we hope the Community can join us.

Try the OIDC validator library now!

We’re excited to share that the first release of the OIDC validator library is now available for your feedback. The repository includes:

Basic setup instructions in the README,
Introductory documentation for getting started, and
Links to examples and test configurations.

More detailed guides, including how OIDC works under the hood and how to use it in real PostgreSQL deployments, as well as direct integration guides are coming soon in follow up blog posts and documentation articles.

We’d love your feedback

If you’re experimenting with PostgreSQL 18 or exploring modern authentication options, give the OIDC validator library a try and let us know what you think! Your input will help us make this capability more robust, portable, and enterprise-ready while keeping it open source and accessible to everyone.

Keep Calm - TDE for PostgreSQL 18 Is on Its Way!

Jan Wieremjewicz — Wed, 15 Oct 2025 11:00:00 UTC

If you’ve been following the buzz around PostgreSQL, you’ve probably already heard that database level open source data-at-rest encryption is now available thanks to the Transparent Data Encryption (TDE) extension available in the Percona Distribution for PostgreSQL. So naturally, the next question is:

Where’s Percona Distribution for PostgreSQL 18?

The short answer:

It’s coming.

The slightly longer one:

It’s taking a bit of time, for all the right reasons.

Why the delay?

We’ve been laser-focused on advancing pg_tde maturity, making it ready for your production workloads. We succeeded and now you can use Percona Distribution for PostgreSQL together with TDE on PROD! The next steps for us are: 1 Merging the changes to PostgreSQL 18. 2 Ensuring that the patches making TDE on PostgreSQL possible are accepted by PostgreSQL Community.

As part of both of these efforts, we took the time to thoroughly test TDE against PostgreSQL 18. During that process, we found that to use TDE on PostgreSQL 18 some extra work is needed to ensure the new Asynchronous I/O (AIO) feature works as designed. Since TDE extends the Storage Manager (SMGR), it needs to integrate smoothly with changes introduced by AIO and that’s something we focus on now and are definitely do not want to rush.

Percona stands for quality of services and products. Databases are the fundaments of the information systems, outside of security the most important values are stability and availability. To respect these values we chose not to push a major release just for the sake of timing, but instead to take the time to do things properly.

We’re planning to release TDE with the first minor patch of PostgreSQL 18, currently scheduled for November 13.

Taking our time responsibly

Most production users don’t deploy brand new major releases on day one. Typically before using a new major version on production we’ve seen a minimum on 1-2 minor releases for even the users we’ve seen fastest to adopt new versions.

That gives us a valuable window to make sure everything is rock-solid before it reaches your clusters. We’re using that time wisely: testing, aligning, and polishing so that when you upgrade to PostgreSQL 18 with Percona TDE, it’ll feel like it was part of the core all along.

Oh, and something new is brewing…

As PostgreSQL 18 introduces OAuth Authorization/Authentication we tried using it with OpenID Connect (OIDC). OIDC is an identity layer built on top of OAuth 2.0, adding user authentication to OAuth’s authorization capabilities.

We tried using OIDC from a number of providers and came to a conclusion that there’s currently no easy way of using it with PostgreSQL 18. The key missing component to make it work is a validator library that would allow to validate the identity tokens.

I’m happy to share what we started working on OIDC support for PostgreSQL, built on top of the OAuth support introduced in PG18. The first beta release of our OIDC validator library is coming soon, stay tuned for that!

How to help?

Easy, share your feedback!

Have you tried pg_tde? Tell us what worked, what didn’t, and how it felt in real use.
Looking for OIDC in PostgreSQL? We’d love your thoughts once the beta drops.
Using pg_stat_monitor? Share how you’re using it and helps us make it better.

We’re invested to making open source databases more secure, flexible and ready for whatever you throw at them one careful step at a time.

Audit Log Filters Part II

Wayne Leutwyler — Wed, 08 Oct 2025 00:00:00 UTC

In my first post on the MySQL 8.4 Audit Log Filter component, I covered how to install the component and configure a basic filter that captures all events. The Audit Log Filter framework offers a highly granular and configurable auditing mechanism, enabling administrators to log specific events based on criteria such as user, host, or event type. This selective approach enhances observability, supports compliance initiatives, and minimizes unnecessary logging overhead.

In this follow-up, we’ll take a deeper technical look at defining and optimizing audit log filters to capture only the most relevant database activities—delivering actionable audit data while significantly reducing noise and log volume.

Example 1

Audit all events:

SELECT audit_log_filter_set_filter('log_all_events', '{
 "filter": {"log": true}
}');

Once this filter is created and assigned to a user (for example, with SELECT audit_log_filter_set_user(’%’, ’log_all_events’);), every database event triggered by that user—or by all users if % is used—will be written to the audit log file.

In short:

This is the most permissive audit configuration possible. It’s typically used:

As a baseline test to verify that the audit log component is working.
In diagnostic or forensic scenarios where full visibility is required.

For production environments, however, it’s recommended to create more selective filters (e.g., by event class, command type, or user) to reduce log volume and improve performance. Which we will go into more detail in the upcoming examples.

Example 2

Log table access:

select audit_log_filter_set_filter('log_table_access', '{
 "filter": {
 "class": [
 { "name": "table_access" },
 { "name": "connection" },
 { "name": "general" }
 ]
 }
}');

Included Event Classes

table_access Logs events when MySQL reads from or writes to tables.
- Useful for tracking which users or applications are accessing specific tables.
- Helps in auditing data access patterns and detecting unauthorized data reads/writes.
connection Logs connection-related events such as user logins, logouts, and failed authentication attempts.
- Important for tracking session activity and security auditing.
general Logs general query execution events—like statements sent to the server (e.g., SELECT, INSERT, UPDATE, etc.).
- Useful for general SQL activity auditing.

What It Does Functionally

After this filter is defined and assigned to a user or host (for example, with SELECT audit_log_filter_set_user(’%’, ’log_table_access’);), MySQL will only log events that fall into one of these three classes.

All other event types—like administrative commands, stored program executions, or system-level actions—will be excluded from the audit log.

Why use this filter

This configuration strikes a balance between completeness and efficiency:

Captures key operational and access-related activity.
Avoids excessive log volume from irrelevant events.
Suitable for data access auditing, security monitoring, and compliance logging.

In short, log_table_access provides targeted visibility into table usage, connections, and general query activity—ideal for environments where tracking who accessed what data is more important than recording every internal event.

Example 3

SELECT audit_log_filter_set_filter('log_minimum', '{
 "filter": {
 "class":
 [ { "name": "connection" }, { "name": "table_access", "event": [
 { "name": "delete"}, { "name": "insert"}, { "name": "update"} ]
 } ]
 }
}');

Included Event Classes

“class”: “connection”
- Logs all connection-level events:
- connect: when a user logs in.
- disconnect: when a session ends.
- Failed logins and other connection-related actions.

Purpose: provides visibility into who connected, from where, and when.

“class”: “table_access” with “event”:
- Limits logging to specific table access events:
  - “delete” when rows are deleted.
  - “insert” when new rows are added.
  - “update” when existing rows are modified.
- Read operations (like SELECT) and metadata queries are excluded.

What It Does Functionally

Once assigned to a user or host (e.g. SELECT audit_log_filter_set_user(’%’, ’log_minimum’);), this filter will produce audit entries only when:

A user connects or disconnects from MySQL.
A user performs a DML (Data Manipulation Language) operation that changes data in a table.

All other events — such as simple SELECT queries, schema reads, or administrative commands — will be ignored.

Why Use This Filter

This is a minimalist, high-value audit configuration. It’s designed to:

Track security-relevant activity (connections and data changes).
Meet compliance requirements with low performance overhead.
Prevent excessive logging and disk usage.

In Short log_minimum is an efficient auditing strategy for production environments where you only need to know:

Who accessed the database, and
What data they changed.

It gives you essential accountability and change tracking without the overhead of logging every read or administrative event.

Example 4

SELECT audit_log_filter_set_filter('log_connections', '{
 "filter": {
 "class": [
 { "name": "connection",
 "event": [
 { "name": "connect"},
 { "name": "disconnect"}
 ]
 }
 ]
 }
}');

Included Event Classes

“class”: “connection”
- This class captures events related to user sessions and authentication.
“event”: [“connect”, “disconnect”]
- connect, Logged when a client establishes a connection to the MySQL server. Includes details like username, host, client program, IP address, and connection method.
- disconnect, Logged when that client session ends or times out. Useful for tracking session duration and identifying abnormal terminations.
Important Note on pre_authenticate Events
- The pre_authenticate events are not included in the examples above.
- These events occur before the MySQL server has received authentication information from the client—meaning no user account details are available at this stage of the connection lifecycle. Because of that, if a filter that includes pre_authenticate events is assigned to a specific user (rather than a wildcard like %) using audit_log_filter_set_user(), those events will not be filtered or logged.
- This behavior often leads to confusion, as users may expect pre_authenticate events to appear in user-specific logs. Several reports and support cases have been filed on this topic, but it is expected behavior due to the timing of authentication during connection initialization.

Why Use This Filter

This filter is particularly useful when you need to:

Monitor user logins and logouts without recording query activity.
Audit connection patterns (e.g., who connected, from where, and when).
Maintain minimal log size and low overhead.
Support security investigations or session tracking without performance impact.

In short the log_connections filter provides a focused, low-overhead auditing strategy that records only connection lifecycle events. It’s ideal for environments where you primarily need to know who connected to the database, when, and from where without capturing every SQL statement or table access.

Example 5

SELECT audit_log_filter_set_filter('log_full_table_access', '{
 "filter": {
 "class": [
 {
 "name": "connection",
 "event": [
 { "name": "connect"},
 { "name": "disconnect"}
 ]
 },
 {
 "name": "query",
 "event": [
 {
 "name": "start",
 "log": {
 "or": [
 { "field": { "name": "sql_command_id", "value": "select"} },
 { "field": { "name": "sql_command_id", "value": "insert"} },
 { "field": { "name": "sql_command_id", "value": "update"} },
 { "field": { "name": "sql_command_id", "value": "delete"} },
 { "field": { "name": "sql_command_id", "value": "truncate"} },
 { "field": { "name": "sql_command_id", "value": "create_table"} },
 { "field": { "name": "sql_command_id", "value": "alter_table"} },
 { "field": { "name": "sql_command_id", "value": "drop_table"} }
 ]
 }
 }
 ]
 }
 ]
 }
}');

This statement defines a MySQL Audit Log Filter called log_full_table_access, which is designed to capture both connection activity and all table-related SQL operations — including reads, writes, and schema changes. It provides broad visibility into how users interact with tables in the database while filtering out unrelated or low-value events.

Included Event Classes

After assigning it to users or hosts (e.g. SELECT audit_log_filter_set_user(’%’, ’log_full_table_access’);), MySQL will log:

Connection lifecycle events (connect, disconnect)
All DML statements (SELECT, INSERT, UPDATE, DELETE, TRUNCATE)
All DDL statements that create, modify, or remove tables (CREATE TABLE, ALTER TABLE, DROP TABLE)
A full list of SQL_COMMANDS can be obtained from:

SELECT NAME FROM performance_schema.setup_instruments WHERE NAME LIKE 'statement/sql/%' ORDER BY NAME;

+-----------------------------------------------+
| NAME |
+-----------------------------------------------+
| statement/sql/alter_db |
| statement/sql/alter_event |
| statement/sql/alter_function |
| statement/sql/alter_instance |
| statement/sql/alter_procedure |
| statement/sql/alter_resource_group |
| statement/sql/alter_server |
| statement/sql/alter_table |
| statement/sql/alter_tablespace |
| statement/sql/alter_user |
| statement/sql/alter_user_default_role |
| statement/sql/analyze |
| statement/sql/assign_to_keycache |
| statement/sql/begin |
| statement/sql/binlog |
| statement/sql/call_procedure |
| statement/sql/change_db |
| statement/sql/change_repl_filter |
| statement/sql/change_replication_source |
| statement/sql/check |
| statement/sql/checksum |
| statement/sql/commit |
| statement/sql/create_compression_dictionary |
[...]
+-----------------------------------------------+
168 rows in set (0.07 sec)

Everything else — administrative commands, stored procedure calls, replication control, etc. — will be excluded.

Why Use This Filter

This filter offers a comprehensive audit view of how users interact with data and schema structures — perfect for compliance, forensic analysis, or access accountability. It ensures that all table reads, writes, and structure changes are tracked without overwhelming the log with irrelevant internal events.

In Short, log_full_table_access provides a broad but targeted audit scope:

Tracks connections for user session context.
Logs all table-level operations, both data and schema-related.
Delivers complete visibility into how data is accessed and changed, making it ideal for security auditing and regulatory compliance scenarios.

Final Summary

The MySQL 8.4 Audit Log Filter component provides a powerful and flexible framework for controlling how database activity is captured and logged. By allowing administrators to define granular filters based on event class, event type, user, or host, it transforms auditing from an all-or-nothing process into a precisely tuned observability tool.

In this post, we explored a range of filter examples—from the most permissive (log_all_events) to more focused configurations like log_minimum, log_connections, and log_full_table_access. Each serves a different operational or compliance purpose:

log_all_events – Captures every event for baseline validation or forensic debugging.
log_table_access – Balances visibility and performance by logging table, connection, and general query activity.
log_minimum – Targets critical actions such as connections and data modifications, providing essential accountability with minimal overhead.
log_connections – Focuses solely on login and logout events, ideal for lightweight session auditing.
log_full_table_access – Delivers comprehensive insight into all table-level DML and DDL operations along with connection tracking, suitable for compliance and change auditing.

By tailoring filters to specific operational needs, administrators can significantly reduce log volume, improve performance, and focus on high-value security and compliance events. The result is a leaner, more informative audit log that provides actionable insight into how users and applications interact with your MySQL environment—without the burden of unnecessary data.

Reference

Special Thanks

Yura Sorokin for the collaboration that made this blog post possible.

Percona at PostgreSQL Conference Europe 2025

Edith Puclla — Fri, 03 Oct 2025 11:00:00 UTC

We’re proud to announce that Percona is a Platinum Sponsor of PostgreSQL Conference Europe (PGConf.EU) 2025, taking place October 21–24, 2025 in Riga, Latvia 🇱🇻 at the Radisson Blu Latvija Conference Center.

As a Platinum Sponsor, you can find Percona in a prime main exhibit floor location; a chance to connect directly with our PostgreSQL experts from around the world. Visitors stopping by our booth will be able to:

Enter raffles and win Percona & PostgreSQL SWAG
Watch live demos and learn from expert-led sessions
Explore open source PostgreSQL solutions built for scalability, performance, and security

As a leader in open source database management and services, Percona supports innovation, collaboration, and the sharing of knowledge that drives the open source ecosystem forward. By participating in PostgreSQL Conference Europe, Percona connects with developers, contributors, and industry leaders to discuss the latest trends, challenges, and advancements. We look forward to further showcasing Percona’s dedication to empowering organizations with robust, scalable, and secure open source database solutions.

About the event

This year’s conference is the 15th Annual PostgreSQL Conference Europe. The conference is organised by PostgreSQL Europe, with participation from most of the PostgreSQL user groups around Europe, and is intended to be an important meeting and cooperation point for users both in and out of Europe. PGConf.EU is a unique chance for European PostgreSQL users and developers to catch up, learn, build relationships, get to know each other and consolidate a real network of professionals that use and work with PostgreSQL.

Percona Speaker Agenda

Our experts are taking the stage across multiple tracks to share insights, hands-on experience, and forward-looking innovations:

Wednesday, October 22

What implementing pg_tde taught us about PostgreSQL

👤 Jan Wieremjewicz

⏰ 16:05 – 16:55 | Room: Alfa | Track: Community (45 minutes)

Thursday, October 23

Kubernetes from the Database Out

👤 Alastair Turner

⏰ 10:25 – 10:55 | Room: Omega 2 | Track: DBA (25 minutes)
TDE as an Extension: A Different Path for PostgreSQL Encryption

👤 Zsolt Parragi

⏰ 11:25 – 12:15 | Room: Beta | Track: Sponsors
Why PostgreSQL took the crown from MySQL and what lies ahead

👤 Peter Zaitsev

⏰ 17:20 – 17:35 | Room: Omega 1 | Track: Platinum Sponsor Keynotes

Friday, October 24

Lessons from two decades of hacking the proprietary value into open source databases

👤 Michal Nosek

⏰ 09:25 – 10:15 | Room: Beta | Track: Sponsors

See the full agenda here.

Community Leadership – Kai Wagner

We are also proud that Kai Wagner, Senior Engineering Manager for PostgreSQL at Percona, is serving on the PGConf.EU 2025 Selection Committee. Kai is a long-time open source contributor and speaker, actively involved in projects like Ceph, openATTIC, and PostgreSQL. As a community builder and PGConf Germany organizer, he helps shape the PostgreSQL ecosystem and ensure diverse, impactful content is represented at the conference.

Join us in Riga this October to connect, learn, and celebrate PostgreSQL with Percona and the wider open source community!

Audit Log Filter Component

Wayne Leutwyler — Thu, 18 Sep 2025 00:00:00 UTC

The audit log filter component in MySQL 8.4 provides administrators with a powerful mechanism for auditing database activity at a fine-grained level. While it offers significant flexibility—such as selectively logging events based on users, hosts, or event types—it can also be challenging to understand and configure correctly.

In this article, we will examine how the audit log filter component works, walk through its core concepts, and share practical tips for configuring and managing audit filters effectively. Our goal is to help you leverage this feature to improve observability, meet compliance requirements, and reduce unnecessary logging overhead.

Enabling Audit Log Filter

We will be using Percona Server 8.4.4 or higher in the examples below. First, we need to enable the audit log filter component. To install the audit log filter component, we need to run the following command:

mysql -u root -p < /usr/share/percona-server/mysql/share/audit_log_filter_linux_install.sql

Verify that the audit log filter component is enabled by running:

select * from mysql.component;
+--------------+--------------------+-----------------------------------+
| component_id | component_group_id | component_urn |
+--------------+--------------------+-----------------------------------+
| 2 | 1 | file://component_audit_log_filter |
+--------------+--------------------+-----------------------------------+
1 row in set (0.00 sec)

Installing the component creates two new tables in the mysql system database: audit_log_filter and audit_log_user. These tables store the audit log filter definitions and the user-to-filter mappings. Together, they are referred to as the audit log filter tables.

+------------------------------------------------------+
| Tables_in_mysql |
+------------------------------------------------------+
| audit_log_filter |
| audit_log_user |
+------------------------------------------------------+

Although the configuration is persisted in these tables, they are not usually modified directly with INSERT or UPDATE statements. Instead, MySQL provides built-in functions such as:

audit_log_filter_set_filter()
audit_log_filter_set_user()

To manage filter definitions and user assignments safely.

Configure the my.cnf file to define the desired audit log output format and specify the location of the audit.log file. In the example below, the log format is set to JSON, but other formats (e.g., NEW or OLD) can also be configured depending on your requirements. The audit log file can be written to any path accessible to the MySQL server process.

Example Changes:

[mysqld]
# auditlog
audit_log_filter.format=JSON
audit_log_filter.file=/var/lib/mysql/audit.log

Restart mysql server to apply the changes:

sudo systemctl restart mysqld

The audit log filter install is complete. Now we can start using the audit log filter component.

Creating Audit Log Filters

The audit log filter component in MySQL 8.4 provides fine-grained control over database auditing. Instead of logging all events indiscriminately, administrators can define audit log filters, which are rule sets that determine exactly which events should be captured and which should be excluded.

This allows you to:

Log only the activity relevant to security, compliance, or troubleshooting.
Reduce unnecessary noise and audit log volume.
Apply different filters to specific users, hosts, or accounts for tailored auditing.

Because filters can be customized and assigned at the user or host level, the audit log filter component offers both flexibility and efficiency, making it a powerful mechanism for monitoring and securing database activity while minimizing overhead.

Lets create a rule that will log all events:

SELECT audit_log_filter_set_filter('log_all_events', '{ "filter": {"log": true } }');

Lets assign the rule to the user:

SELECT audit_log_filter_set_user('%', 'log_all_events');
SELECT audit_log_filter_flush();

Now all users will have all their events logged.

Before proceeding, ensure that the jq utility is installed on your system. The installation commands provided in the examples below are compatible with both RHEL-based and Debian-based distributions.

RHEL builds

sudo yum install jq

Debian builds

sudo apt install jq

To validate that the audit log filter is functioning as expected, we can inspect the raw contents of the audit.log file. Since the log entries are in JSON format, using jq provides an efficient way to query and extract specific events. For example, to filter and display only connection-related events, run:

cat audit.log | jq '.[]|select(.class=="connection")'

This command streams the log file, parses the JSON structure, and returns only entries where the “class” field is equal to “connection”. This approach allows for targeted analysis, making it easier to verify filter behavior, troubleshoot issues, or monitor specific event classes without manually parsing large volumes of log data.

Example Output:

{
 "timestamp": "2025-07-24 07:18:00",
 "id": 27110,
 "class": "connection",
 "event": "connect",
 "connection_id": 415,
 "account": {
 "user": "wayne",
 "host": "localhost"
 },
 "login": {
 "user": "wayne",
 "os": "",
 "ip": "",
 "proxy": ""
 },
 "connection_data": {
 "connection_type": "socket",
 "status": 0,
 "db": ""
 },
 "connection_attributes": {
 "_pid": "717914",
 "_platform": "aarch64",
 "_os": "Linux",
 "_client_name": "libmysql",
 "os_user": "wayne",
 "_client_version": "8.4.5-5",
 "program_name": "mysql"
 }
}

I’ll cover more advanced filter configurations in a follow-up post—stay tuned for Part 2 of the Audit Log Filter Component series.

In summary, the audit log filter component in MySQL 8.4 provides administrators with a flexible and fine-grained approach to database auditing. By tailoring filters to specific users, hosts, and event types, you can ensure that only the most relevant activity is logged, making it easier to meet compliance requirements while reducing overhead. With proper configuration and careful use of filters, you can transform the audit log from a noisy data dump into a precise monitoring tool that strengthens both security and observability in your MySQL environment.

Encrypting PostgreSQL Tables with PG_TDE: Step-by-Step Guide for Beginners

Daniil Bazhenov — Mon, 01 Sep 2025 11:00:00 UTC

Let’s install and try out a new package for PostgreSQL — PG_TDE by Percona. This extension adds Transparent Data Encryption (TDE), a mechanism that allows data to be encrypted at the storage level without affecting application behavior.

TDE protects data in case of disk, dump, or backup compromise: everything stored at the file system level is encrypted. Meanwhile, the application continues to work with the data as usual — encryption and decryption happen transparently, without changes to SQL queries or logic.

In this post, we’ll deploy PostgreSQL with PG_TDE in Docker, configure keys, create an encrypted table, and verify that the data is truly protected — even if someone gains direct access to the files. Other installation methods are available in the documentation.

For simplicity and speed of testing, we’ll use a key provider with a keyring file (for dev/test only) — the easiest way to get started with PG_TDE. This approach is convenient for local development and demos, as it doesn’t require additional dependencies or infrastructure.

Installing PostgreSQL with PG_TDE via Docker

PG_TDE is part of the Percona Distribution for PostgreSQL — an enhanced PostgreSQL distribution that includes tools for monitoring, auditing, replication, and of course, Transparent Data Encryption. We’ll use the official Docker image, which already contains everything needed to run PG_TDE.

Let’s start by launching the container:

docker run --name pg-tde \
 -e POSTGRES_PASSWORD=secret \
 -e ENABLE_PG_TDE=1 \
 -p 5432:5432 \
 -d percona/percona-distribution-postgresql:17.5-3

Here:

POSTGRES_PASSWORD=secret — sets the superuser password
ENABLE_PG_TDE=1 — enables Transparent Data Encryption support
Port 5432 — standard for PostgreSQL

After launching, connect to PostgreSQL inside the container:

docker exec -it pg-tde psql

And activate the PG_TDE extension:

CREATE EXTENSION pg_tde;

Verify that PG_TDE is enabled and check its version using the SQL function:

SELECT pg_tde_version();

Here’s the result at the time of writing — I’m using PG_TDE 2.0

I also connected to PostgreSQL using pgAdmin, accessing via localhost, user postgres, and the password from POSTGRES_PASSWORD.

pgAdmin is very convenient for exploring the database via UI.

Now TDE is ready — we can create encrypted tables, manage keys, and verify how data is protected at the storage level.

Configuring Encryption Keys with PG_TDE

After installing and activating the pg_tde extension, the next step is to configure the keys that will be used for data encryption. In this example, we’re using a key provider based on a keyring file — a simple and fast way to start working with PG_TDE in development mode.

Important: The keyring file stores keys in unencrypted form and is intended for testing and development only. For production, it’s recommended to use an external key store such as HashiCorp Vault — we’ll cover that in the next post.

Add the key provider:

SELECT pg_tde_add_database_key_provider_file(
 'file-vault',
 '/tmp/pg_tde_local_keyring.per'
);

This command:

Registers a key provider named ‘file-vault’
Specifies that keys will be stored in the file located at /tmp/pg_tde_local_keyring.per
The file path can be anything — you can use a different location or filename, as long as PostgreSQL has access to it. If the file doesn’t exist, it will be created when the first key is generated

It’s important not to lose this file — it contains the encryption keys. If needed, it can be placed in a local folder and mounted into the container via Docker, but for this task, using a path inside the container is perfectly fine.

Create the key:

SELECT pg_tde_create_key_using_database_key_provider(
 'test-db-key',
 'file-vault'
);

We’re creating a key named test-db-key inside the specified key provider. This key will be used to encrypt tables.

Set the active key:

SELECT pg_tde_set_key_using_database_key_provider(
 'test-db-key',
 'file-vault'
);

This command sets test-db-key as the current active key, which will be used when creating new encrypted tables.

Let’s verify.

Retrieve key information using the command:

SELECT pg_tde_key_info();

Result:

postgres=# SELECT pg_tde_key_info();
 pg_tde_key_info
------------------------------------------------------------
 (test-db-key,file-vault,1,"2025-09-04 07:16:04.420629+00")
(1 row)

Check the contents of the keyring file — just for experimentation.

In a separate terminal tab, connect to the container:

docker exec -it pg-tde bash

Use hexdump to read the file since it contains binary data:

hexdump -C /tmp/pg_tde_local_keyring.per | head

This will output the first bytes of the file in hex format, allowing you to confirm that the file is not empty and the key is indeed recorded.

If the file is empty or unchanged — the key may not have been created or written.

Creating Tables with Data

Now that the encryption key is set, let’s create two tables: one with encryption enabled, and one regular. This will help visually demonstrate how PG_TDE works and how data storage differs.

Create the encrypted table by specifying USING tde_heap:

CREATE TABLE secure_data (
 id SERIAL PRIMARY KEY,
 secret TEXT,
 created_at DATE
) USING tde_heap;

Add a few rows:

INSERT INTO secure_data (secret, created_at) VALUES
 ('The launch code is hidden in plain sight.', '2025-09-01'),
 ('Trust no one. The truth is encrypted.', '2025-09-02'),
 ('The treasure lies beneath the third stone by the old oak.', '2025-09-03');

Create a similar table without encryption:

CREATE TABLE plain_data (
 id SERIAL PRIMARY KEY,
 secret TEXT,
 created_at DATE
);

Insert the same data:

INSERT INTO plain_data (secret, created_at) VALUES
 ('The launch code is hidden in plain sight.', '2025-09-01'),
 ('Trust no one. The truth is encrypted.', '2025-09-02'),
 ('The treasure lies beneath the third stone by the old oak.', '2025-09-03');

Now both tables contain identical rows, but one stores the data in encrypted form.

Let’s check the encryption status using PG_TDE.

Use the built-in function pg_tde_is_encrypted, which returns true (t) if the table uses encryption, and false (f) if not:

postgres=# SELECT pg_tde_is_encrypted('secure_data');
 pg_tde_is_encrypted
---------------------
 t
(1 row)

postgres=# SELECT pg_tde_is_encrypted('plain_data');
 pg_tde_is_encrypted
---------------------
 f
(1 row)

Excellent, now we can move on to comparison: let’s see how the data looks inside PostgreSQL — and how it’s stored on disk. This will help confirm that PG_TDE truly protects content, even if someone gains direct access to the files.

Comparing Tables: Encrypted vs Unencrypted

At the SQL query level, both tables — secure_data (encrypted) and plain_data (unencrypted) — look absolutely identical. This is important: PG_TDE does not affect SQL interface behavior, and the application continues to work with the data as usual.

postgres=# SELECT * FROM secure_data;
 id | secret | created_at
----+-----------------------------------------------------------+------------
 1 | The launch code is hidden in plain sight. | 2025-09-01
 2 | Trust no one. The truth is encrypted. | 2025-09-02
 3 | The treasure lies beneath the third stone by the old oak. | 2025-09-03
(3 rows)

postgres=# SELECT * FROM plain_data;
 id | secret | created_at
----+-----------------------------------------------------------+------------
 1 | The launch code is hidden in plain sight. | 2025-09-01
 2 | Trust no one. The truth is encrypted. | 2025-09-02
 3 | The treasure lies beneath the third stone by the old oak. | 2025-09-03
(3 rows)

Display in the application using pgAdmin

Data from the plain_data table:

Data from the encrypted table secure_data:

The pgAdmin screenshots also show no differences: data is displayed in readable form, regardless of whether the table is encrypted. This confirms that PG_TDE operates at the storage level, without affecting data handling logic.

As you can see, the result is identical: encryption is transparent to the user and application.

Comparing Table File Contents

Now let’s verify that PG_TDE truly encrypts data at the storage level. To do this, we’ll locate the physical table files and compare their contents using hexdump.

In PostgreSQL, each table is physically stored as one or more files in the base directory, inside the folder corresponding to the database. The file path looks like $PGDATA/base//

database_oid — unique database identifier (OID), retrieved from pg_database
relfilenode — table file identifier, retrieved from pg_class

Step 1: Retrieve Identifiers

Get the OID of the current database:

postgres=# SELECT oid, datname FROM pg_database WHERE datname = current_database();
 oid | datname
-----+----------
 5 | postgres
(1 row)

Get the relfilenode of the tables:

postgres=# SELECT relname, relfilenode FROM pg_class WHERE relname IN ('secure_data', 'plain_data');
 relname | relfilenode
-------------+-------------
 secure_data | 16433
 plain_data | 16442
(2 rows)

Now, knowing oid = 5 and the relfilenode of the tables, we can locate the table files at:

data/db/base/5/16433 — encrypted table secure_data
data/db/base/5/16442 — unencrypted table plain_data

Step 2: Read Table Files

Open a terminal and connect to the PostgreSQL container for bash commands:

docker exec -it pg-tde bash

Read data from the unencrypted table (plain_data):

hexdump -C data/db/base/5/16442 | head -n 20

The output shows that the data is readable directly from the file:

Read data from the encrypted table (secure_data):

hexdump -C data/db/base/5/16433 | head -n 20

The output shows that the contents are fully encrypted:

No readable strings — PG_TDE reliably encrypts data at the file system level.

Conclusion

In this post, we installed PostgreSQL with Transparent Data Encryption (PG_TDE) by Percona, created keys, encrypted a table, and verified that the data is truly protected during physical storage. PG_TDE by Percona provides transparent encryption: the application continues to work with the data as usual, while security is enforced at the file system level.

If you’re working with PostgreSQL and looking to strengthen your data protection, PG_TDE offers a great starting point. I invite you to experiment:

Create encrypted tables and perform a backup. Try restoring it on a different installation and see how key access is enforced.
Explore key rotation, switching the active key, or integrating alternative key providers.
Test how PG_TDE behaves in real-world scenarios: migration, failover, CI/CD pipelines

If you’ve already tried PG_TDE or plan to — share your experience and insights by posting in this blog or on the forum

In the next post, we’ll explore a more robust option — integrating PG_TDE with HashiCorp Vault, which enables centralized key management, rotation, and access control.

pg_tde can now encrypt your WAL on PROD!

Jan Wieremjewicz — Mon, 01 Sep 2025 11:00:00 UTC

Just recently, we announced the production-ready release of pg_tde, bringing open source Transparent Data Encryption (TDE) to PostgreSQL.
Now, I may have spoiled the fun a little with the title, but take a look at the word puzzle below—can you guess the announcement? Bear with me… and my sense of humor, which might be a bit too dry for some :)

Yes, it’s an elephant carrying logs to a safe. Because Write Ahead Log (WAL) deserves secure storage!

At the time of the TDE GA release, one of the most common questions was:

“How does pg_tde handle the data in Write Ahead Logs (WAL)?”

The short answer back then was:

“It can encrypt WAL data, but the feature is still in beta.”

I’m happy to share that this capability is now production ready as well! WAL encryption has reached General Availability (GA) and is included in the latest release of Percona Distribution for PostgreSQL 17.5.3.

How to use it?

To use WAL encryption, you’ll need both:

pg_tde extension
Percona Server for PostgreSQL (included in Percona Distribution)

Why? Because pg_tde relies on patches that extend PostgreSQL APIs, and these are not yet available in the upstream PostgreSQL Community releases. Our long-term goal is to ensure they are available in the Community PostgreSQL Server as well, but for now you’ll find them only in Percona Server. If you’ve already been running the GA version of TDE in Percona Distribution 17.5.2, the upgrade is straightforward. Assuming pg_tde is installed and initialized, enabling WAL encryption requires:

Configure the encryption key by:
- Setting up the key provider for WAL encryption
- Creating the principal key
Turning the WAL encryption on by running:
- ALTER SYSTEM SET pg_tde.wal_encrypt = on;
Restarting the server to apply the changes

Some known limitations of the first version

While WAL encryption is now GA, there are some first version limitations to be aware of:

Compatibility with other extensions: pg_tde has only been validated with the extensions shipped in Percona Distribution. Extensions that rely heavily on WAL (such as some backup, replication, or logical decoding tools) may not yet work correctly.
Backup and restore:
- WAL encryption has been tested and verified with PgBackRest, our recommended tool for backup and recovery.
- pg_basebackup works with WAL encryption when using streaming or none methods. While the -x fetch method is not yet supported for other methods there are some good practices we highly recommend.
- Other backup tools may not yet support WAL encryption. An example is Barman as the following Barman backup methods are not yet supported with WAL encryption:
  - Postgres method
  - Rsync method
  - Snapshot method
Native logical replication: tools like pg_createsubscriber are not yet supported with encrypted WAL. We are planning to enable them in one of the next releases. Do share feedback if you need them, it will help us prioritize.

See more details regarding the WAL encryption GA supported scope in our documentation. We’ll continue testing compatibility with widely used extensions and documenting what works out of the box versus what requires extra steps. Here, we rely on our community of users as well to let us help prioritize work. Let us know what would you want to see tested or you’ve already tested and need help in making work with WAL encryption.

Why this matters for regulated environments

STIG documentation proves that PostgreSQL is ready to tackle any government use cases already. Encryption of both data files and WAL is often a requirement when organizations work under strict compliance frameworks such as:

FIPS 140-2 / 140-3 – cryptographic module validation requirements for U.S. government use.
HIPAA – protection of electronic protected health information (ePHI).
PCI-DSS – encryption requirements for cardholder data at rest.
FedRAMP and other government security standards – full coverage of data at rest, including temporary and log files.

Many PostgreSQL distributions rely on filesystem-level encryption or external tooling to meet these standards. With PCI-DSS 4.0 this is no longer enough. With pg_tde, encryption is native to PostgreSQL internals, ****covering both tables and WAL, implemented with widely recognized cryptographic algorithms via OpenSSL, and integrated with enterprise grade Key Management Systems (KMS). This reduces operational complexity and helps align directly with compliance controls.

What does the future hold for TDE?

We’re far from done. The roadmap for TDE includes:

New capabilities:
- More cipher configuration options
- Expanded KMS support (beyond what we support today)
- Temporary files encryption
- Features requested from real-world deployments
Lifting current limitations:
- Broader backup/restore scenarios
- Compatibility with logical replication tooling such as pg_createsubscriber
- Validating more third-party extensions
Upstream contributions:
- Our long-term goal is to contribute the necessary server patches into Community PostgreSQL. This way, pg_tde can eventually run on any PostgreSQL release, not just Percona Server for PostgreSQL.

We’d love your feedback! If you’ve started using pg_tde or are planning to test it (especially with WAL encryption) let us know what works well, what could be improved or fixed, and what you’d like to see next. Your input directly helps us prioritize features, improve compatibility, and shape the future of TDE in PostgreSQL.

Share your experiences via our Community Forums, alternatively via GitHub Discussions or Issues!

pg_stat_monitor Needs You! Join the Feedback Phase

Jan Wieremjewicz — Wed, 13 Aug 2025 00:00:00 UTC

At Percona, we believe that great open source software is built with the Community, not just for it. As we plan the next iteration of pg_stat_monitor, our advanced PostgreSQL monitoring extension, we’re taking a closer look at the current feature set and how it aligns with real-world usage.

In open source, the community isn’t just a user base, it’s the most important stakeholder. While we set the vision, your feedback is the compass that guides us. Your experiences, bug reports, and feature requests are what validate our direction and keep us focused on what matters most. Without your active involvement, it’s impossible to build a tool that truly solves the problems you face every day. Your input ensures pg_stat_monitor evolves in a way that is both innovative and genuinely useful.

Your Feedback Is the Compass

Over time, pg_stat_monitor has grown beyond its initial query performance monitoring scope.

While many features have proven to be extremely useful (especially in use with Percona Monitoring and Management (PMM)), others may see limited adoption or give no evidence of any adoption at all. To ensure we’re investing in what really matters, we want to understand what you, the users and contributors, actually rely on day-to-day.

That’s why we’re kicking off a Community feedback phase

We’re especially interested in:

What features of pg_stat_monitor are critical for your workflows?
Are you using it together with PMM, CLI or in another way?
Are there parts of the extension that feel unnecessary or unclear?
What would you love to see in the next release?

This is just the beginning of a broader review and improvement effort, one that we want to run transparently and inclusively, true to our open source values.

Whether you’re a developer, DBA, or platform engineer using pg_stat_monitor directly or via tools like PMM, know that your input matters.

👉 Leave a comment below, or reach out on our Percona Community Forums or via GitHub. Let’s shape the future of pg_stat_monitor together.

Percona pg_tde: A Security Review Reveals Robust Encryption

Kai Wagner — Thu, 24 Jul 2025 00:00:00 UTC

At Percona, we are committed to providing robust and secure database solutions. We recently engaged Longterm Security for an in-depth review of our Transparent Data Encryption (TDE) feature for PostgreSQL, known as pg_tde. This comprehensive assessment included a cryptographic design evaluation, an application security review for coding errors, and extensive fuzz testing. We’re excited to share the key takeaways from this engagement, highlighting both pg_tde’s strengths and areas for continued improvement.

Strong Foundations in Compliance

A major highlight of the review is pg_tde’s excellent alignment with leading compliance requirements. Our TDE implementation is well-suited to meet the stringent demands of standards like PCI-DSS, HIPAA, GDPR, and SOC2. This is largely due to its adherence to best practices outlined by ECRYPT-CSA, BSI, and NIST, particularly concerning key reuse, master key management, initialization vectors, randomness generation, and various counter modes (CBC, CTR, GCM).

The sole exception identified for full compliance is CNSA (formerly NSA Suite B), which specifically mandates a 256-bit key size for AES. While pg_tde currently supports AES-128, which is widely considered future-proof and meets most compliance needs for data lifetimes beyond 10 years, adding AES-256 support remains a key recommendation to meet the demands of government and high-security environments.

While technical work continues on managing temporary files and potentially swappable memory, it’s important to note that compliance standards—such as PCI DSS—generally permit the use of temporary files for encrypting sensitive data like PAN. That said, we understand the practical risks involved and are actively working to mitigate them.

No Major Vulnerabilities Uncovered

After a comprehensive security engagement, we are pleased to report that no major vulnerabilities have been found within pg_tde. This outcome is a testament to the robust design principles and meticulous implementation that underpin pg_tde, validating our continuous investment in secure development.

The highest priority issue they found (LTS-1: Temporary Files Not Subject to Encryption) was actually something we already knew about and being on our roadmap, while we documented ways to lessen the risk. Our team is also actively tackling another known issue (LTS-3) where sensitive data or encryption keys might end up on your disk if your system uses “swap memory.” As of today, we do not recommend using SWAP, but we’re actively looking into making it work as well. The other issues they pointed out (LTS-4 through LTS-7) are more like helpful suggestions for making things even more robust, rather than actual vulnerabilities.

Here’s a quick overview of the findings:

Issue	Severity	Status	Title
LTS-1	Medium	Already Known	Temporary Files Not Subject to Encryption
LTS-2	Low	Reported	Disable Core Dumps when pg_tde is active
LTS-3.1	Low	Already Known	palloc memory may be swapped to disk, storing customer data
LTS-3.2	Low	Already Known	palloc memory may be swapped to disk, storing key material from pg_tde
LTS-4	Informational	Informative	Privileged postgres users with file read access may disclose encryption keys from memory
LTS-5	Informational	Reported	Keyring_vault.c: Invalid JSON response can lead to NULL dereference
LTS-6	Informational	Informative	Consider HKDF with Principal Key Usage
LTS-7	Informational	Reported	Document how to securely deploy keyring authentication credentials to systems without storing to disk
LTS-8	Informational	Reported	AES-GCM-SIV is available in very new OpenSSL versions as an alternative key wrapping algorithm

Always Improving: What’s Next for pg_tde around security?

The review also helped us pinpoint a few areas where we can make pg_tde even stronger, especially when it comes to handling memory and exploring advanced encryption techniques:

Tackling Swap Memory and Core Dumps: We’re working on ways to prevent sensitive data and encryption keys from potentially leaking to your disk through swap memory or “core dumps” (which happen when a program crashes). We recommend using encrypted swap partitions and turning off core dumps when pg_tde is active. We’re also looking into using special memory allocators that keep key material from being swapped out.
More Encryption Options: To meet even more compliance needs and give you more choices, we’re considering adding support for a wider range of encryption ciphers, including AES-CBC-256, AES-CTR-256, AES-GCM-256, ChaCha20-Poly1305, and AES-XTS-256. The 256-bit AES key sizes, in particular, will help us achieve CNSA compliance.
Smarter Key Management (HKDF with Principal Key Usage): To boost security even further and protect against theoretical key-reuse attacks, we’re evaluating the use of HKDF (HMAC-based Key Derivation Function). This would add an extra layer of protection by creating unique keys for different situations.
Securely Handling Your Keyring Credentials: It’s super important that your keyring authentication credentials (like Vault tokens or KMIP keys) are never stored in plain text on your disk. We’re putting together clear guidelines for secure deployment, recommending solutions that keep credentials only in memory or use hardware-backed keystores. We’ll also advise against putting credentials in command-line arguments.
Keeping an Eye on New Algorithms (AES-GCM-SIV): We’re aware of a newer encryption algorithm called AES-GCM-SIV, which is available in very recent OpenSSL versions. While it’s not yet FIPS-140-3 compliant and might be a bit slower, it offers better protection against certain types of attacks, and we’re definitely keeping it on our radar for future consideration.
Preparing for Post-Quantum Cryptography: The cryptographic landscape is evolving rapidly, with post-quantum encryption (PQC) algorithms on the horizon. As these new algorithms are expected to be more resource-intensive, pg_tde’s granular table-level encryption will be crucial for efficiently managing performance overheads, ensuring your data remains secure without compromising application responsiveness in a post-quantum world. We are actively monitoring these ciphers’ development and standardization.

Our Commitment to Security

The security review by Longterm Security has been incredibly valuable. It’s not only confirmed pg_tde’s strong capabilities but also given us a clear roadmap for future development. We’re really proud of how secure pg_tde is, which you can see in its readiness for compliance and the fact that no major vulnerabilities were found. Our ongoing efforts to address the identified issues, even the low-priority ones, show just how committed we are to constantly improving and providing you with the most secure database solutions out there.

We’re committed to building a secure and reliable data encryption solution, and your feedback plays a vital role in shaping its future. Try it out!

If you want to work directly with the code/test/build, take a look at the GitHub project. For issues, report them via Jira or go to our Forum to ask questions. For detailed deployment guidelines and best practices, take a look at our documentation. We’re looking forward to any feedback and collaboration around making PostgreSQL secure.

GitOps Journey: Part 4 – Observability and Monitoring with Coroot in Kubernetes

Daniil Bazhenov — Tue, 22 Jul 2025 00:01:00 UTC

Our PostgreSQL cluster is running, and the demo app is generating traffic — but we have no visibility into the health of the Kubernetes cluster, services, or applications.

What happens when disk space runs out? What if the database is under heavy load and needs scaling? What if errors are buried in application logs? How busy are the network and storage layers? What’s the actual cost of the infrastructure?

This is where Coroot comes in.

Coroot is an open-source observability platform that provides dashboards for profiling, logs, service maps, and resource usage — helping you track system health and diagnose issues quickly.

We’ll deploy it using Helm via ArgoCD, continuing with our GitOps workflow.

This is Part 4 in our series. Previously, we:

Set up ArgoCD and a GitHub repository for declarative manifests (Part 1)
Installed a PostgreSQL cluster using Percona Operator
Deployed a demo application to simulate traffic and interact with the database

All infrastructure is defined declaratively and deployed from the GitHub repository, following GitOps practices.

So far, we’ve explored cluster scaling, user management, and dynamic configuration — and now it’s time for observability.

We’ll install Coroot by following the official documentation for Kubernetes.

Steps ahead:

Install the Coroot Operator
Install the Coroot Community Edition

Let’s get started.

Project Structure

We already have a postgres/ directory for PostgreSQL manifests and an apps/ directory for ArgoCD applications.

We’ll preserve this layout and add a new coroot/ folder for clarity. You can use a different structure if preferred.

Create Manifest for Installing the Coroot Operator

The documentation recommends installing via Helm.
Since we use ArgoCD, we’ll create a manifest that installs via Helm.

Create file: coroot/operator.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot-operator
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://coroot.github.io/helm-charts
 chart: coroot-operator
 targetRevision: 0.4.2
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

Note: I’m using version 0.4.2, which was current at the time of writing.
To check available versions, use this GitHub link or Helm CLI:

helm repo add coroot https://coroot.github.io/helm-charts
helm repo update
helm search repo coroot-operator --versions

Create Manifest for Installing Coroot Community Edition

Create file: coroot/coroot.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://coroot.github.io/helm-charts
 chart: coroot-ce
 targetRevision: 0.3.1
 helm:
 parameters:
 - name: clickhouse.shards
 value: "2"
 - name: clickhouse.replicas
 value: "2"
 - name: service.type
 value: LoadBalancer
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true

This chart creates a minimal Coroot Custom Resource.
I’ve added service.type: LoadBalancer to expose a public IP.

If you don’t use LoadBalancer, you’ll need to forward the Coroot port after installation:

kubectl port-forward -n coroot service/coroot-coroot 8080:8080

Create ArgoCD Application Manifest

Since we manage our infrastructure via a GitHub repository, we need an ArgoCD Application that tracks changes in the coroot/ folder.

Create file: apps/argocd-coroot.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot-sync-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/percona-argocd-pg-coroot.git
 targetRevision: main
 path: coroot
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

This lightweight app will monitor the folder and apply updates automatically if a change is detected (e.g. chart version bump).

Define Chart Installation Order

We have two charts: operator.yaml and coroot.yaml, and the operator must be installed first.

Create coroot/kustomization.yaml to specify resource order:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - operator.yaml
 - coroot.yaml

Publish Manifests to GitHub

Check which files were changed:

git status

Add changes:

git add .

Verify staged files:

git status

Commit:

git commit -m "Installing Coroot Operator and Coroot with ArgoCD"

Push:

git push origin main

Apply ArgoCD Application

Deploy the ArgoCD app that installs Coroot from our GitHub repository:

kubectl apply -f apps/argocd-coroot.yaml -n argocd

Validate installation and sync:

We now see coroot, coroot-operator, and coroot-sync-app deployed.

Access Coroot UI

Since we deployed Coroot using LoadBalancer, retrieve its external IP:

kubectl get svc -n coroot

Open EXTERNAL-IP on port 8080.
For example: http://35.202.140.216:8080/

If you didn’t use LoadBalancer, run port-forward:

kubectl port-forward -n coroot service/coroot-coroot 8080:8080

Then visit http://localhost:8080

You’ll be prompted to set an admin password on first login.

Exploring Coroot UI

On the home page, we see a list of applications running in the cluster and resource usage.

I increased the load on the PostgreSQL cluster using the Demo App to test observability.

The PostgreSQL cluster dashboard offers several tabs:

CPU
Memory
Storage
Instances
Logs
Profiling
Tracing

Coroot displays a visual map of service interactions — showing which app connects to the PostgreSQL cluster.

The Profiling tab looks excellent and intuitive. Here’s the Demo App profiling view:

I also triggered an intentional error in the demo app.
Coroot correctly displayed it in both the home view and the app details page.

I especially liked the Logs and Costs sections in the sidebar — very well implemented.

First Incident: Storage Usage in PostgreSQL Turns Yellow

While exploring Coroot and the cluster, I increased the load on the PostgreSQL cluster using the Demo App.

After a short while, I noticed that the Postgres disk was full.

I opened the cluster details and went to the Storage tab.

By default, the cr.yaml file allocates just 1Gi of disk space — which is fine for a test setup.

Let’s increase disk size the GitOps way.

Increase Storage Size

Open the file postgres/cr.yaml and locate the section:

 dataVolumeClaimSpec:
# storageClassName: standard
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

Change storage from 1Gi to 5Gi.

Note: Backup volumes (pgBackRest) are also enabled by default and set to 1Gi.

 manual:
 repoName: repo1
 options:
 - --type=full
# initialDelaySeconds: 120
 repos:
 - name: repo1
 schedules:
 full: "0 0 * * 6"
# differential: "0 1 * * 1-6"
# incremental: "0 1 * * 1-6"
 volume:
 volumeClaimSpec:
# storageClassName: standard
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

Increase this storage to 5Gi as well.

Save changes to cr.yaml, then commit and push to the GitHub repository.

ArgoCD will automatically apply the changes. Pure GitOps magic.

Check the result in Coroot — everything looks great. The disk is increased to 5Gi and the issue is resolved.

Conclusion

We’ve installed and tested a solid monitoring tool, and it really makes a difference.

Across this 4-part series, we walked through the GitOps journey step by step:

Part 1 - Created a Kubernetes cluster, installed ArgoCD, and set up a GitHub repository.
Part 2 - Deployed a PostgreSQL cluster using Percona Operator for PostgreSQL.
Part 3 - Deployed a demo app via ArgoCD using Helm.
Installed and tested Coroot, an excellent open-source observability tool.

Managed the PG cluster through GitHub and ArgoCD — scaled replicas, created users, resized volumes, configured access, and more.

Thank you for reading — I hope this series was helpful.

The project files are available in my repository https://github.com/dbazhenov/percona-argocd-pg-coroot

I’d love to hear your questions, feedback, and suggestions for improvement.

GitOps Journey: Part 3 – Deploying a Load Generator and Connecting to PostgreSQL

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:50 UTC

We’ll deploy a demo application into the Kubernetes cluster using ArgoCD to simulate load on the PostgreSQL cluster.

This is a series of articles, in previous parts we:

Part 1 - Prepared the environment and installed ArgoCD and GitHub repository.
Part 2 - Installed Percona Operator for Postgres and created a Postgres cluster.

The application is a custom Go-based service that generates traffic for PostgreSQL, MongoDB, or MySQL.

It uses a dataset of GitHub repositories and pull requests, and mimics real-world operations like fetching, creating, updating, and deleting records.
Load intensity is configurable through a browser-based control panel.

We’ll install it using Helm, tracked and deployed via ArgoCD.

Reference repository: github-stat

Create the ArgoCD Application Manifest

Create a file named argocd-demo-app.yaml in the apps/ directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: demo-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/github-stat
 targetRevision: main
 path: k8s/helm
 destination:
 server: https://kubernetes.default.svc
 namespace: demo-app
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

This will install the Helm chart from
https://github.com/dbazhenov/github-stat/tree/main/k8s/helm

By default, the service is configured as LoadBalancer, making it accessible from the internet.

To switch to NodePort (if needed), override the Helm value:

source:
 helm:
 parameters:
 - name: controlPanelService.type
 value: NodePort

We’ll keep default settings in this example.

Push the Application Manifest to GitHub

Track and commit your changes:

git status

git add .

git commit -m "Installing Demo Application in ArgoCD by HELM"

git push origin main

Expected Git output:

➜ percona-argocd-pg-coroot git:(main) git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
 (use "git add ..." to include in what will be committed)
 apps/argocd-demo-app.yaml

nothing added to commit but untracked files present (use "git add" to track)
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git commit -m "Installing Demo Application in ArgoCD by HELM"
[main 03ce175] Installing Demo Application in ArgoCD by HELM
 1 file changed, 20 insertions(+)
 create mode 100644 apps/argocd-demo-app.yaml
➜ percona-argocd-pg-coroot git:(main) git push origin main
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 686 bytes | 686.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:dbazhenov/percona-argocd-pg-coroot.git
 6b2dc98..03ce175 main -> main
➜ percona-argocd-pg-coroot git:(main)

Apply the ArgoCD Application

Deploy the app via:

kubectl apply -f apps/argocd-demo-app.yaml -n argocd

ArgoCD will install the app and has started tracking the app’s HELM chart

Validate the Deployment

Confirm the app status in ArgoCD UI:

Check running pods:

kubectl get pods -n demo-app

Expected pods:

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n demo-app
NAME READY STATUS RESTARTS AGE
demo-app-dataset-6d886f67-j648w 1/1 Running 0 2m52s
demo-app-load-577cff97c9-d8j99 1/1 Running 0 2m52s
demo-app-valkey-74989c9bf7-gjp4x 1/1 Running 0 2m52s
demo-app-web-5b98d4c65c-xmkq9 1/1 Running 0 2m52s

demo-app-dataset - loads dataset
demo-app-load - generates traffic
demo-app-valkey - Redis-compatible DB backend
demo-app-web - UI dashboard

Open the Application Dashboard

Retrieve the external IP:

kubectl get svc -n demo-app

Find the EXTERNAL-IP of demo-app-web-service.

Sample output:

➜ percona-argocd-pg-coroot git:(main) kubectl get svc -n demo-app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-app-valkey-service ClusterIP 34.118.235.203  6379/TCP 4m59s
demo-app-web-service LoadBalancer 34.118.232.144 34.28.221.107 80:31308/TCP 4m59s

Access the app in your browser:

Navigate to the Settings tab to configure a PostgreSQL connection.

PostgreSQL Credentials Setup

Percona Operator has already (Application and system users):

Created schema and database cluster1
Created user cluster1
Stored credentials in cluster1-pguser-cluster1 secret

Extract the password:

kubectl get secret cluster1-pguser-cluster1 -n postgres-operator --template='{{.data.password | base64decode}}{{"\n"}}'

Let’s connect to the database from the Demo application using the given user and cluster1-pgbouncer.postgres-operator.svc host

In the Connection String field enter

user=cluster1 password='[PASSWORD]' dbname=cluster1 host=cluster1-pgbouncer.postgres-operator.svc port=5432

The connection has been successfully created, this is good.

To start generating the load, we need to import the Dataset using the Import Dataset button.

Dataset Import Error: Create Schema Denied

During import, the app tries to create a schema.
By default, pgBouncer limits user privileges, preventing this action.

Percona documentation suggests enabling proxy.pgBouncer.exposeSuperusers and creating a privileged user.

We’ll handle this via GitOps. It seems cool that we’ll be doing this with tracking in Git, as these are important settings and we shouldn’t forget about them and turn them off in the future.

Define a New PostgreSQL User

We will make changes to postgres/cr.yaml that will add a new user and also enable the proxy.pgBouncer.exposeSuperusers option.

In the postgres/cr.yaml file I found the users section, uncommented and added my user data.

In postgres/cr.yaml, add:

 users:
 - name: daniil
 databases:
 - demo
 options: "SUPERUSER"
 password:
 type: ASCII
 secretName: "daniil-credentials"

Note: In production, use scoped permissions like "LOGIN CREATE CREATEDB" rather than SUPERUSER.

I also found the proxy.pgBouncer.exposeSuperusers setting and set it to true

Update pgBouncer config:

 proxy:
 pgBouncer:
 replicas: 3
 image: docker.io/percona/percona-pgbouncer:1.24.1
 exposeSuperusers: true

Commit and push:

git status

git add .

git commit -m "Postgres cluster: Creating a new user and pgBouncer.exposeSuperusers"

git push origin main

After a couple of minutes, ArgoCD will synchronize the changes and Percona Operator will create the user and change the configuration.

Connect With the New User

Get the password:

kubectl get secret daniil-credentials -n postgres-operator --template='{{.data.password | base64decode}}{{"\n"}}'

Let’s replace Connection String in Demo application, I got the following string

user=daniil password='iKj:e[wT3*g]OF5+f' dbname=dataset host=cluster1-pgbouncer.postgres-operator.svc port=5432

Click the “Import Dataset” button and wait a few minutes until the import is in Done status in the Dataset tab.

Enable Load Generation

Activate the load generator:

Toggle Enable Load in the connection settings
Click Update Connection

Open the Load Generator Control Panel and adjust sliders and toggles as needed:

Conclusion

In this part, we:

Deployed a demo application via Helm in ArgoCD
Connected it to our PostgreSQL cluster
Managed PostgreSQL users and access via GitHub and GitOps
Imported a dataset and activated the traffic generator through the web UI

In Part 4, we’ll deploy Coroot for observability and profiling.
It’s an impressive tool for diagnosing behavior across services in the Kubernetes cluster.

GitOps Journey: Part 2 – Deploying PostgreSQL with GitOps and ArgoCD

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:30 UTC

We’re now ready to deploy PostgreSQL 17 using GitOps — with ArgoCD, GitHub, and the Percona Operator for PostgreSQL.

If you’re a DBA, developer, DevOps engineer, or engineering manager, this part focuses on GitOps in action: deploying and managing a real database cluster using declarative infrastructure.

In Part 1, we set up the Kubernetes environment and installed ArgoCD.
Now it’s time to define and launch the PostgreSQL cluster — fully versioned and synced through Git.

We’ll follow the official Percona Operator documentation and reference the GitHub repository to build out a production-grade setup.

Preparing the Environment

There are multiple ways to install the Percona Operator and create a PostgreSQL cluster.
We’ll use the simplest and most GitOps-friendly approach:

Deploy the operator using deploy/bundle.yaml
Deploy the cluster using deploy/cr.yaml

Source files:

https://github.com/percona/percona-postgresql-operator/blob/main/deploy/bundle.yaml
https://github.com/percona/percona-postgresql-operator/blob/main/deploy/cr.yaml

Project Structure

Repository structure can vary depending on your services and infrastructure scale.
For this series, we’ll use:

postgres/ → Contains all manifests related to PostgreSQL: the operator, clusters, backups
apps/ → Contains ArgoCD application manifests that track changes in the repository

You’re free to choose a different structure. Just ensure all paths are correctly referenced in ArgoCD.

Creating the Postgres Directory and Saving Manifests

You can manually download the files from GitHub or automate it via CLI:

mkdir postgres

curl -o postgres/bundle.yaml https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/bundle.yaml

curl -o postgres/cr.yaml https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/cr.yaml

You can also separately clone the operator repository and grab the necessary files from there.

Creating the ArgoCD Application Manifest

This ArgoCD application will track the postgres/ directory and automatically sync changes from GitHub.

Create the file: apps/argocd-postgres.yaml

Content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: postgres
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/percona-argocd-pg-coroot.git
 targetRevision: main
 path: postgres
 destination:
 server: https://kubernetes.default.svc
 namespace: postgres-operator
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true
 - ServerSideApply=true 

You can also create this app manually via the ArgoCD UI or CLI, but using a manifest aligns better with GitOps principles.

Double-check your:

repoURL → matches your GitHub repository
path → corresponds to your PostgreSQL manifest directory
namespace → targets the correct namespace for operator and cluster

Managing ArgoCD Sync Order with Waves

ArgoCD applies manifests based on sync-wave annotations.

The operator (bundle.yaml) should be applied first
The cluster (cr.yaml) comes second

Add these annotations:

In bundle.yaml:

metadata:
 annotations:
 argocd.argoproj.io/sync-wave: "1"

In cr.yaml:

metadata:
 name: cluster1
 annotations:
 argocd.argoproj.io/sync-wave: "5"

This ensures a stable deployment sequence.

Later in the series (e.g. when installing Coroot), we’ll use a more advanced method: defining sync order via kustomization.yaml.

Reviewing Cluster Configuration

Before applying the manifests, review and adjust your cluster settings in cr.yaml.

Key defaults:

name: cluster1 → Cluster name
postgresVersion: 17 → PostgreSQL version

To keep the cluster lightweight and test horizontal scaling later, reduce replicas to 1:

instances:
 - name: instance1
 replicas: 1

proxy:
 pgBouncer:
 replicas: 1

You can also configure resource limits, disk sizes, backups, and users in this file.

Publishing the Configuration to GitHub

Verify your repo status:

git status

Add files:

git add .

Review staged files:

git status

Expected result:

➜ percona-argocd-pg-coroot git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
 (use "git add ..." to include in what will be committed)
 apps/
 postgres/

nothing added to commit but untracked files present (use "git add" to track)
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
 (use "git restore --staged ..." to unstage)
 new file: apps/argocd-postgres.yaml
 new file: postgres/bundle.yaml
 new file: postgres/cr.yaml

Commit:

git commit -m "Initial configuration of a Postgres cluster using Percona Operator for Postgres and ArgoCD"

Push to GitHub:

git push origin main

Verify files are published correctly in the repository.

Applying the ArgoCD App Manifest

To initiate the deployment, apply the previously created manifest:

kubectl apply -f apps/argocd-postgres.yaml -n argocd

After a minute or two, the ArgoCD dashboard should display the synced PostgreSQL application and the deployed cluster.

Verifying Cluster Status

Check the running pods:

kubectl get pods -n postgres-operator

Expected results:

One PostgreSQL instance pod
One pgBouncer pod

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-5g98-5b29w 0/1 Completed 0 27m
cluster1-instance1-22vd-0 4/4 Running 0 28m
cluster1-pgbouncer-649b7cf845-fgs9l 2/2 Running 0 28m
cluster1-repo-host-0 2/2 Running 0 28m
percona-postgresql-operator-79f75d5f76-xjndr 1/1 Running 0 29m

Scaling the Cluster via GitOps

Let’s test the GitOps model by updating the cluster configuration to increase replicas to 3.

Edit postgres/cr.yaml:

 instances:
 - name: instance1
 replicas: 3

 proxy:
 pgBouncer:
 replicas: 3

Save the changes and push them:

git status

git add .

git commit -m "Postgres cluster: Horizontal scaling from 1 replica to 3"

git push origin main

ArgoCD will automatically detect and apply this update.

Expected results:

➜ percona-argocd-pg-coroot git:(main) git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
 (use "git add ..." to update what will be committed)
 (use "git restore ..." to discard changes in working directory)
 modified: postgres/cr.yaml

no changes added to commit (use "git add" and/or "git commit -a")
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git commit -m "Postgres cluster: Horizontal scaling from 1 replica to 3"
[main 6b2dc98] Postgres cluster: Horizontal scaling from 1 replica to 3
 1 file changed, 2 insertions(+), 2 deletions(-)
➜ percona-argocd-pg-coroot git:(main) git push origin main
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 435 bytes | 435.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:dbazhenov/percona-argocd-pg-coroot.git
 81ae9e8..6b2dc98 main -> main
➜ percona-argocd-pg-coroot git:(main)

Expected results on GitHub Repo:

Confirming the Update in ArgoCD

In the ArgoCD UI, you should now see the application synced to the latest commit with the updated replica count.

Verify pods:

kubectl get pods -n postgres-operator

Expected result — 3 PostgreSQL pods and 3 pgBouncer pods.

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-5g98-5b29w 0/1 Completed 0 38m
cluster1-instance1-22vd-0 4/4 Running 0 39m
cluster1-instance1-q2r4-0 4/4 Running 0 3m38s
cluster1-instance1-r4s2-0 4/4 Running 0 3m39s
cluster1-pgbouncer-649b7cf845-9cppx 2/2 Running 0 3m37s
cluster1-pgbouncer-649b7cf845-fgs9l 2/2 Running 0 39m
cluster1-pgbouncer-649b7cf845-tkf9z 2/2 Running 0 3m36s
cluster1-repo-host-0 2/2 Running 0 39m
percona-postgresql-operator-79f75d5f76-xjndr 1/1 Running 0 40m

What’s Next

We’ve successfully installed the Percona Operator for PostgreSQL and deployed a cluster using GitHub and ArgoCD.

We also verified GitOps functionality by scaling the cluster through Git-controlled configuration.
All changes are tracked, versioned, and declarative — a solid foundation for modern infrastructure management.

To continue experimenting:

Connect to the Cluster
Manage Users. Note: the default user does not have SUPERUSER privileges. If your app requires creating databases, you’ll need to configure appropriate roles.
Expose the Cluster. So you can connect from external clients or apps.

We’ll do exactly that in the next part — by deploying a demo application and connecting it to the database using GitOps.

GitOps Journey: Part 1 – Getting Started with ArgoCD and GitHub

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:10 UTC

Welcome to GitOps Journey — a hands-on guide to setting up infrastructure in Kubernetes using Git and automation.

GitOps has gained traction alongside Kubernetes, CI/CD, and declarative provisioning.
You’ve probably seen it mentioned in blog posts, tech talks, or conference slides — but what does it actually look like in practice?

We’ll start from scratch: prepare a cluster, deploy a PostgreSQL database, run a demo app, and set up observability — all managed via Git and GitHub using ArgoCD.

What We’ll Build

ArgoCD — syncs manifests from a GitHub repository to your cluster
PostgreSQL — a production-ready database using Percona Operator
Demo App — a real Go-based web app connected to the database
Coroot — an open-source tool for monitoring performance, logs, and service behavior

This series is for anyone new to GitOps or Kubernetes.
Each part includes clear steps, real-world YAML, and examples you can run yourself.

This is Part 1 of the GitOps Journey.
If you already have ArgoCD and a working Kubernetes cluster, you can skip ahead:

Part 2 – Deploying PostgreSQL with Percona Operator

Part 3 – Connecting a Real App to the Cluster

Part 4 – Observability with Coroot

Copilot assisted with formatting, Markdown structure, and translation.
All ideas, architecture decisions, and hands-on implementation were created by Daniil Bazhenov.

Otherwise, let’s start by preparing the cluster and setting up ArgoCD.

Creating a Kubernetes Cluster

I’ll be using Google Kubernetes Engine (GKE), but you can use AWS, DigitalOcean, or even run Minikube locally.

You’ll also need these CLI tools installed on your machine:

kubectl - The official CLI tool for Kubernetes — used to manage clusters, view resources, apply manifests, and more.
helm - A package manager for Kubernetes — lets you install complex apps using reusable charts (like PostgreSQL, monitoring tools, etc.)

I use the following command to create a cluster in GKE

gcloud container clusters create dbazhenov-demo \
 --project percona-product \
 --zone us-central1-a \
 --cluster-version 1.30 \
 --machine-type n1-standard-8 \
 --num-nodes=3

To delete the cluster:

gcloud container clusters delete dbazhenov-demo --zone us-central1-a

Note: This command doesn’t remove your LoadBalancers, so I prefer deleting them manually in Google Cloud’s web console to ensure no resources are left running post-experiment.

Here’s the resulting setup:

➜ community git:(blog_argocd_pg) ✗ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-dbazhenov-demo-default-pool-b1b48316-8nrj Ready  6m7s v1.30.12-gke.1279000
gke-dbazhenov-demo-default-pool-b1b48316-8v14 Ready  6m6s v1.30.12-gke.1279000
gke-dbazhenov-demo-default-pool-b1b48316-zg6z Ready  6m7s v1.30.12-gke.1279000
➜ community git:(blog_argocd_pg) ✗

Installing ArgoCD

We’ll begin with ArgoCD, the GitOps engine that will deploy:

PostgreSQL database cluster

A demo app to simulate real usage

Coroot for monitoring and profiling workloads

ArgoCD supports multiple deployment methods — we’ll experiment with different ones during this series.

Install ArgoCD (based on official docs):

Create namespace:

kubectl create namespace argocd

Deploy ArgoCD:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Check the pods:

kubectl get pods -n argocd

Expected output: ArgoCD components running (server, repo, redis, controllers, etc.)

➜ community git:(blog_argocd_pg) ✗ kubectl get pods -n argocd
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 0 57s
argocd-applicationset-controller-6d569f7895-89kgk 1/1 Running 0 64s
argocd-dex-server-5b44d67df9-p42z5 1/1 Running 0 62s
argocd-notifications-controller-5865dfbc8-gqzwt 1/1 Running 0 61s
argocd-redis-6bb7987874-99j59 1/1 Running 0 61s
argocd-repo-server-df8b9fd78-64czj 1/1 Running 0 60s
argocd-server-6d896f6785-82tf2 1/1 Running 0 59s

Access ArgoCD UI

You have two options (or more):

Port forwarding (local only)

kubectl port-forward svc/argocd-server -n argocd 8080:443

Internet-accessible LoadBalancer

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

I will use Load Balancer by executing the command above, you need to wait a few minutes to get the IP address.

Let’s get the IP address of the ArgoCD service in the EXTERNAL-IP field.

kubectl get svc argocd-server -n argocd

➜ community git:(blog_argocd_pg) ✗ kubectl get svc argocd-server -n argocd
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
argocd-server LoadBalancer 34.118.234.162 34.132.39.194 80:30549/TCP,443:32146/TCP 9m51s

Access the UI in your browser using the IP.

Getting Started with ArgoCD Login

Download Argo CD CLI

Install ArgoCD CLI (see instructions).

Get the initial password:

argocd admin initial-password -n argocd

ArgoCD recommends changing it to a new secure password, which we will do.

argocd login 34.132.39.194 --insecure

Authorize using initial-password and user admin and execute the password update command

argocd account update-password

All the steps to get and update your password are below.

➜ community git:(blog_argocd_pg) ✗ argocd admin initial-password -n argocd
0mxV6IVcF3qZDR-O

 This password must be only used for first time login. We strongly recommend you update the password using `argocd account update-password`.
➜ community git:(blog_argocd_pg) ✗ argocd login 34.132.39.194 --insecure
Username: admin
Password:
'admin:login' logged in successfully
Context '34.132.39.194' updated
➜ community git:(blog_argocd_pg) ✗ argocd account update-password
*** Enter password of currently logged in user (admin):
*** Enter new password for user admin:
*** Confirm new password for user admin:
Password updated
Context '34.132.39.194' updated
➜ community git:(blog_argocd_pg) ✗

Now log into the ArgoCD web UI using admin and your new password.

Welcome to the ArgoCD interface, we don’t have any applications right now, we will install them later.

Setting Up GitHub Repo

We’ll need a GitHub repo to store infrastructure manifests. ArgoCD will sync from this repo and apply changes.

Install git and create a GitHub account
Add your SSH key to your GitHub profile. GitHub SSH settings
Create a new GitHub repository

I recommend a public repo for this educational project — no secrets will be committed, and it simplifies ArgoCD setup. Plus, it earns you some green squares on GitHub. If you go with a private repo, make sure it’s properly linked in ArgoCD.

Clone the repo:

Clone the repository using an SSH address.

git clone git@github.com:dbazhenov/percona-argocd-pg-coroot.git

Navigate to the project directory

cd percona-argocd-pg-coroot

Expected output:

➜ gitops git clone git@github.com:dbazhenov/percona-argocd-pg-coroot.git
Cloning into 'percona-argocd-pg-coroot'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
Receiving objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
➜ gitops cd percona-argocd-pg-coroot
➜ percona-argocd-pg-coroot git:(main) ls
README.md
➜ percona-argocd-pg-coroot git:(main)

Summary

We’ve prepared everything to launch our GitOps-powered infrastructure:

Kubernetes cluster
ArgoCD deployed
GitHub repo ready

In the next posts, we’ll deploy the PostgreSQL cluster, the demo app, and add Coroot monitoring.

Stay tuned!

Using replicaSetHorizons in MongoDB

Ivan Groenewold — Tue, 22 Jul 2025 00:00:00 UTC

When running MongoDB replica sets in containerized environments like Docker or Kubernetes, making nodes reachable from inside the cluster as well as from external clients can be a challenge. To solve this problem, this post is going to explain the horizons feature of Percona Server for MongoDB.

Let’s start by looking at what happens behind the scenes when you connect to a replicaset URI.

Node auto-discovery

After connecting with a replset URI, the driver discovers the list of actual members by running the db.hello() command:

mongosh "mongodb://mongo1-internal:27017/?replicaSet=rs0"
rs0 [direct: primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877b5e18a13d54b752ff25c'),
 counter: Long('6')
 },
 hosts: [ 'mongo1-internal:27017', 'mongo2-internal:27017', 'mongo3-internal:27017' ],
...

The list of hosts returned contains the name of each member as you provided it to the rs.initialize() command.

The node identity crisis

The names are resolvable inside the same network, so all is well in this case. But what happens when connecting from outside?

Typically you would be using names like mongo1-external.mydomain.com that correctly point to the external IP addresses of the members. The problem is that after the initial connection is made, the driver will perform auto-discovery and try to connect to the names as reported by db.hello(). These are not resolvable from outside.

What if we connect by IP address directly? again, the driver will get the names from the list above, try to reach those and fail after the initial connection is made:

$ mongosh mongodb://user:pass@10.30.50.155:32768/?replicaSet=rs0
Current Mongosh Log ID: 6849eb15ba228be45a69e327
Connecting to: mongodb://@10.30.50.155:32768/?replicaSet=rs0&appName=mongosh+2.5.2
MongoNetworkError: getaddrinfo ENOTFOUND mongo1-internal

Even though mongo1-internal is not part of the connection string, the driver tries to reach it. So if the replica set members advertise their internal IPs or DNS names, clients outside can’t connect unless they can resolve that same name. We could work around that, but there’s another issue: the ports.

The port issue

In the containerized world, it is likely that you set up your containers to use default port 27017. However they might be mapped to a different external port, since you have to avoid port collisions (think about the case where containers are co-located in the same host).

We need a way for replica set members to identify themselves with different names and ports, depending on whether the client is in the same network or outside. A concept similar to split-brain DNS.

What is Horizons?

Horizons is a MongoDB feature that allows replica set members to advertise different identities depending on the client’s access context, such as internal versus external networks.

With this, you can make the same MongoDB replica set usable from:

Internal container network (using internal hostnames/IPs)
External applications (using public IPs or DNS names)

MongoDB’s horizons rely on Server Name Indication (SNI) during the TLS handshake to determine which hostname and port to advertise. At connection time, clients present the hostname they used, and MongoDB uses that to return the proper set of endpoints. For that reason TLS is required in order for horizons to work.

Let’s walk through an example.

Example Scenario: MongoDB Replica Set in Docker

You can run the following steps on your local machine to test the feature.

Get your certificates ready

Let’s start by creating the required CA and certificates using Cloudflare’s PKI and TLS toolkit.

Step 1: Create ca-csr.json

mkdir certs
cd certs
tee ca-csr.json <{
 "CN": "MyTestCA",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "US", "ST": "CA", "L": "SF", "O": "Acme", "OU": "MongoDB CA" }]
}
EOF

Generate the CA:

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

This creates:

ca.pem — CA certificate
ca-key.pem — CA private key

Step 2: Create server-csr.json for each server, specifying both internal and external names in the “hosts” section so that our certificate is valid for everything.

for i in 1 2 3; do
 name="mongo$i"

 tee "${name}-csr.json" <{
 "CN": "${name}",
 "hosts": ["${name}", "${name}.internal", "localhost", "127.0.0.1"],
 "key": { "algo": "rsa", "size": 2048 },
 "names": [
 { "O": "MongoDB", "OU": "Database", "L": "Internal", "ST": "DC", "C": "US" }
 ]
}
EOF
done

Step 3: Generate certificates using CFSSL

for i in 1 2 3; do
 name="mongo$i"

 cfssl gencert \
 -ca=ca.pem -ca-key=ca-key.pem \
 -config=<(cat <<'JSON'
{
 "signing": {
 "default": {
 "expiry": "8760h",
 "usages": [
 "signing",
 "key encipherment",
 "server auth",
 "client auth"
 ]
 }
 }
}
JSON
) "${name}-csr.json" | cfssljson -bare "${name}"
cat "${name}.pem" "${name}-key.pem" > "${name}-combined.pem"
done

cd ..

Resulting files:

mongo{1,2,3}.pem — cert for server
mongo{1,2,3}-key.pem, key for server
mongo{1,2,3}-combined.pem, both in a single file as expected by mongo

Docker Compose Setup

Create a file with docker compose configuration:

tee test-horizons.yml <name: horizons
services:
 mongo1:
 container_name: mongo1
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27017:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo1-combined.pem
 --tlsCAFile /certs/ca.pem
 mongo2:
 container_name: mongo2
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27018:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo2-combined.pem
 --tlsCAFile /certs/ca.pem
 mongo3:
 container_name: mongo3
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27019:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo3-combined.pem
 --tlsCAFile /certs/ca.pem
networks:
 default:
 driver: bridge
EOF

Here we are mapping our containers to ports 27017, 27018 and 27109 externally.

Now, start the services:

$ docker-compose -f test-horizons.yml up -d

Initiate the Replica Set with Horizons

Now let’s initiate the replica set with different host names and ports for external access.

Launch a shell into one of the containers:

$ docker exec -it mongo1 /bin/bash

Authenticate and initialize the replica set with this config:

$ mongosh --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsAllowInvalidCertificates
rs.initiate({
 _id: "rs0",
 members: [
 {
 _id: 0,
 host: "mongo1:27017",
 horizons: { external: "localhost:27017" }
 },
 {
 _id: 1,
 host: "mongo2:27017",
 horizons: { external: "localhost:27018" }
 },
 {
 _id: 2,
 host: "mongo3:27017",
 horizons: { external: "localhost:27019" }
 }
 ]
})

Note: The “horizon” field here maps the external context to a different address than the internal one. Since we are going to test connecting from the local machine directly to the containers, set the horizons to localhost and the mapped ports.

Connect from Inside Docker

Spin up a new containerized client, or use one of the existing MongoDB containers:

$ docker exec -it mongo1 mongosh --host rs0/mongo1:27017,mongo2:27017,mongo3:27017 --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsCAFile /certs/ca.pem
Current Mongosh Log ID: 6877deab6568339f46dfd9c4
Connecting to: mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongo1-combined.pem&tlsCAFile=%2Fcerts%2Fca.pem&appName=mongosh+2.5.0
Using MongoDB: 8.0.8-3
Using Mongosh: 2.5.0

rs0 [primary] test>

It connects using internal Docker hostnames.

Connect from Outside Docker

From your local machine:

$ mongosh "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsCAFile /certs/ca.pem
Current Mongosh Log ID: 6877defabc3f9a2d054a1296
Connecting to: mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=certs%2Fmongo1-combined.pem&tlsCAFile=certs%2Fca.pem&appName=mongosh+2.3.1
Using MongoDB: 8.0.8-3
Using Mongosh: 2.3.1

rs0 [primary] test>

Check the identities returned

As we have seen, MongoDB will resolve the external horizon names and connect successfully in both cases. You can verify the advertised hostnames and ports for the external connection:

rs0 [primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877de4c632adf89fb590f38'),
 counter: Long('6')
 },
 hosts: [ 'localhost:27017', 'localhost:27018', 'localhost:27019' ],
 setName: 'rs0',
 setVersion: 1,
 isWritablePrimary: true,
 secondary: false,
 primary: 'localhost:27017',
 me: 'localhost:27017',
...

Versus the internal case:

rs0 [primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877de4c632adf89fb590f38'),
 counter: Long('6')
 },
 hosts: [ 'mongo1:27017', 'mongo2:27017', 'mongo3:27017' ],
 setName: 'rs0',
 setVersion: 1,
 isWritablePrimary: true,
 secondary: false,
 primary: 'mongo1:27017',
 me: 'mongo1:27017',
...

Conclusion

The horizons feature in MongoDB is a powerful tool to bridge the gap between internal and external connectivity, especially in containerized or multi-network deployments.

Horizon also has following limitations:

Using horizons is only possible with TLS connections
Duplicating domain names in horizons is not allowed by MongoDB
Using IP addresses in horizons definitions is not allowed by MongoDB
Horizons should be set for all members of a replica set, or not set at all

This feature is not listed in the official MongoDB documentation for some reason, however it is available in both Percona Server for MongoDB and MongoDB Community Edition. Also, Kubernetes users rejoice! Percona Operator for MongoDB supports horizons since version 1.16.

Active-active replication - the hidden costs and complexities

Jan Wieremjewicz — Thu, 10 Jul 2025 00:00:00 UTC

In Part 1 of this series, we discussed what active-active databases are and identified some “good” reasons for considering them, primarily centered around extreme high availability and critical write availability during regional outages. Now, let’s turn our attention to the less compelling justifications and the substantial challenges that come with implementing such a setup.

What are “bad” reasons?

1. Scaling write throughput

Trying to scale your write capacity by deploying active-active across regions may sound like a clean horizontal solution, but it is rarely that simple. Write coordination, conflict resolution, and replication overhead introduce latency that defeats the purpose. If you are thinking about low latency writes between regions like Australia and the US, keep in mind that you will still be paying the round-trip cost, typically 150-200ms+, to maintain consistency. Physics don’t do any favors. Even if you have multiple primaries, unless you accept weaker consistency or potential conflicts, your writes will not scale linearly. In many real-world cases, throughput actually suffers compared to a well tuned primary replica setup. If your real goal is better throughput, you are usually better served by:

Regional read replicas
Sharding
Task queuing and eventual delegation

2. Performance

Performance is often used as a vague justification. But active-active is not a panaceum for general slowness. If the issue is database bottlenecks, reasons may be as basic as not enough work spent on data structuring, indexing, query tuning or scaling reads before jumping into multi primary deployments. Way too often we find that performance problems come not from scale limits, but from poor design and neglected maintenance. We are in 2025 and lessons like this classic one on fast inserts are still painfully relevant. If what you are really facing is application level latency, that is a different challenge. And even then, active-active with strong guarantees often adds more latency, not less.

3. Load balancing

You don’t need multi-primary to load balance. If your goal is spreading traffic more evenly, and reads are your bottleneck, there are simpler and much safer ways to get there. That’s what read replicas are for. If what you are looking for is distributing writes, please re-read point 1: synchronizing state across regions adds cost, complexity, and consistency challenges. Multi-master does not eliminate contention, it just moves it. In these situations you will have conflicts, some due to lag, some for other reasons. Resolution logic becomes critical. Conflicts need to be logged and their resolution auditable.

4. Because other systems did it

The fact that multi master was possible in other systems is not, on its own, a good reason to adopt it. If you have worked with Oracle RAC or GoldenGate, you likely remember the long nights of debugging conflicts, performance issues, the layers of failover logic, the dedicated teams just to keep things running smoothly. These systems made big promises, but keeping them stable in production was rarely simple or cheap. Copying those patterns means copying those problems. And trying to reproduce them with PostgreSQL, or some other modern databases, often misses the point. This is our chance to rethink how we build, not repeat the same mistakes. If you are going to pay the cost of active active, make sure you are doing it for the right reasons, not just because it looks like something you are used to.

The cost, the painful side of active-active

Let’s be very clear: active-active is not the shortcut it often appears to be. It’s not simply “HA but better.” It represents a fundamental system design change with significant, ongoing costs across engineering, operations, and product management. While it might be the only viable option in highly specific situations, by betting on extreme reliability, you are signing up for a hell of a lot of complexity.

Much of what makes active-active the wrong choice in typical scenarios comes down to these costs. So expect some repetition, that’s by design, because the price is the problem.

1. Conflict resolution is a problem of it’s own

Every multi-primary system has to answer one question: what happens when two nodes write to the same row at the same time? PostgreSQL doesn’t have conflict resolution built-in. You can bolt it on with tools like pgactive or pglogical2, but none of them are perfect, far from it.

The most common approaches applied:

Last write wins (based on timestamps).
Source priority (some nodes win conflicts).
Application-defined merge logic.

They all come with trade-offs. None of them are free.

Consider how other systems handle this is very often not conflict handling but conflict avoidance:

CockroachDB avoids exposing raw conflict handling to users by using strict serializability and sacrificing performance
Oracle RAC handles it via centralized locking over a shared-disk architecture (and even that causes headaches)
CouchDB, which embraces multi-master natively, makes conflict resolution a first-class application responsibility and warns users upfront

Unless your app is very carefully designed to dodge conflicts, say, by writing to disjoint subsets of data, you’re going to run into this. Probably when it’s already too late and your data’s out of sync.

2. Infrastructure & networking costs

Running active-active across regions increases your infra footprint and cost. Specifically:

More powerful machines or more of them – each node handles writes and replication
Higher network bandwidth usage – especially for busy systems or cross-region replication
Premium network setups – often needed for low-latency consistency
Increased complexity in monitoring – replication lag, node health, consistency checks

These costs aren’t just financial, they bleed into reliability and operational overhead too.

3. Operational and management overhead

Beyond setup, day-2 operations are where the real pain begins:

Deployments become dangerous – schema or app changes can break replication or trigger conflicts
Debugging is harder – tracing transactions across writeable nodes adds layers to incident response
Staffing costs rise – managing active-active needs deeper expertise in distributed systems
Auditing grows in importance – proving data consistency across nodes becomes a continuous task

Let’s be honest, what was hard before is now harder, and even the routine becomes risky.

Don’t kill the messenger

I love food. Cooking, eating, talking about it. Anyone I’ve worked with can vouch that I’m a foodie at heart. I like spicy food, really spicy. In my kitchen, you’ll find everything from Sichuan pepper to naga jolokia “ghost pepper”, from kala namak to asafoetida. These aren’t ingredients you throw into every dish. The right spice, at the wrong time, ruins everything. The same spice, used precisely, can turn something ordinary into magic.

What’s exactly how active-active works. It’s not bad by nature, it’s just very specific. You need to understand the dish, the eater, and the context. If you add ghost pepper to scrambled eggs because someone said it’s “cool,” you’re very likely going to regret it. Same goes for multi-primary setups.

Know what you’re in for. Use the right tool for the right reason. And don’t reach for active-active just because it sounds hot. (And if you do pull it off successfully, please send me your config. I owe you a beer.)

Percona Bug Report: June 2025

Aaditya Dubey — Mon, 30 Jun 2025 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs.

Percona Server/MySQL Bugs

PS-9823: mysql_migrate_keyring fails with PS Components.

The failure is triggered by a missing symbol, but the underlying cause is the way keyring components are built in Percona Server. When attempting to migrate keyring data (e.g., from Vault to File), the tool fails to load the Percona Server component .so files, making the migration process unusable.

Percona Server builds a reference to the my_free symbol, which is not properly resolved in the shared libraries. In contrast, upstream MySQL builds do not include this dependency.

This issue blocks both component-to-component and component-to-plugin keyring migrations, affecting users who rely on secure key management transitions.

Reported Affected Version/s: 8.0.x, 8.4.x
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9836: There is a regression issue with audit_log_filter.so compared to audit_log.so. The audit_log_filter, whether used as a plugin (8.0) or a component (8.0 and 8.4), shows a significant performance regression. When logging everything, QPS drops by over 70%. While configuring selective logging can reduce the impact, it still results in a 30–35% drop in QPS.

For this reason, moving to audit_log_filter in 8.0 is not recommended. Additionally, this should be taken into account when planning upgrades to 8.4, as audit logging can significantly impact performance. (audit_log is not available as a component—only as a plugin.)

Reported Affected Version/s: 8.0.42-33, 8.4.5-5
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9837: A crash occurs on replica nodes during parallel replication when an INSERT is executed on a secondary index that recently had a DELETE on the same key. The issue is caused by a race condition in the secondary index reuse logic, leading to an assertion failure (row0ins.cc:268).

This issue is more likely to occur under heavy write workloads, particularly when the application frequently performs DELETE followed by INSERT on the same keys. It only affects replica servers where replica_parallel_workers > 0 and slave_preserve_commit_order=ON.

Reported Affected Version/s: 8.0.36-28, 8.0.42-33
Upstream Bug: 118334
Workaround/Fix: The user can modify their logic to use UPDATE instead of DELETE followed by INSERT, which avoids triggering the crash path.
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9861: The audit_log_filter plugin cannot be installed when component_keyring_kmip is enabled with Fortanix DSM. While testing with component_keyring_kmip, we enabled the “Allow secrets with unknown operations” option in Fortanix, which allowed the audit log installation to proceed one step further. At this point, a secret is successfully created for the audit log, but MySQL crashes upon restart.

This issue is related to bug PS-9609 and still persists when using Fortanix DSM as the KMIP server.

Reported Affected Version/s: 8.0.42-33
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: The issue has been fixed, and the fix is expected in the upcoming release of Percona Server (PS).

PS-9914: After running ALTER TABLE … ENGINE=InnoDB to rebuild a large table (~10 million rows) with ROW_FORMAT=COMPRESSED, it was observed approximately a 50% drop in write-only workload throughput (measured via sysbench), despite a reduction in .ibd file size and no changes to table structure or indexes. The table had previously undergone heavy deletions (~50%), suggesting possible fragmentation prior to the rebuild.

Reported Affected Version/s: 8.0.37-29, 8.0.42-33
Upstream Bug: 118411
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9956: PS 8.4.4-4 with group replication crashes on Oracle Linux 9 during bootstrap or failover when the audit log filter component is enabled, but does not crash on Oracle Linux 8.

Reported Affected Version/s: 8.4.4-4
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation.

Percona Xtradb Cluster

PXC-4652: PXC 8.4 crashes with a SIGSEGV in unordered_map called from rpl_gtid_owned during high activity, while PXC 8.0 under the same workload and data remains stable; the crash occurs randomly during operations like COMMIT or INSERT.

Reported Affected Version/s: 8.4.3, 8.4.4
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: 8.4.5 – Pending Release

PXC-4684: An UPDATE query that joins two tables but modifies only one—e.g., UPDATE test.t2 JOIN test.t1 USING (i) SET t2.d = t2.d+1, t1.d = t1.d;—causes an MDL BF-BF conflict on other PXC nodes, even without triggers, as both tables are included in the Table_map_log_event.

Reported Affected Version/s: 8.0.41
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: 8.0.42 – Released | 8.4.5 – Pending Release

Percona Toolkit

PT-2418: In pt-online-schema-change 3.7.0, data was lost when executing the following SQL — the value of column col_2 was unexpectedly set to NULL:

Eg:

ALTER TABLE t RENAME COLUMN col_1 TO col_2;
MySQL version: 8.0+
pt-online-schema-change --no-version-check \
 h=127.0.0.1,u=root,p=xxx,P=xxx,D=sysbench,t=sbtest1 \
 --alter="RENAME COLUMN col_1 TO col_2" \
 --execute --statistics

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2419: pt-duplicate-key-checker Ignores DESC in Index Definitions. Users running pt-duplicate-key-checker regularly observed that a newly added composite index was being incorrectly flagged as a duplicate and removed.

Before:

KEY `idx_ts` (`ts`),
KEY `idx_ts_id` (`ts` DESC, `id`)

After:

KEY `idx_ts_id` (`ts`)

The tool appears to ignore the DESC direction in index definitions, leading to incorrect de-duplication. This behaviour may affect query plans and performance in setups relying on sort order.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2425: Case-Sensitive MariaDB Detection Causes Sync Failure in pt-table-sync. In pt-table-sync 3.7.0, a case-sensitive check for the MariaDB flavor ($vp->flavor() =~ m/maria/) fails because flavor() returns “MariaDB Server”, causing the condition to evaluate incorrectly. As a result, the tool looks for source_host and source_port in $source, while the actual keys are master_host and master_port, leading to failures or uninitialized value warnings.

The Error:

Use of uninitialized value in concatenation (.) or string at /usr/bin/pt-table-sync line 7086.

Manually updating the regex to m/maria/i resolves the issue. Similar case-sensitive checks appear elsewhere in the script and may require centralizing the MariaDB detection logic.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 3.7.1 - Not Yet Released

PT-2197: In pt-online-schema-change (version 3.7.0), attempting to run an ALTER operation results in the following error:

Use of uninitialized value in string eq at /usr/bin/pt-online-schema-change line 4321

This occurs even when replica connectivity in both directions is fully functional and multiple replicas are connected. Notably, the issue does not occur in version 3.5.1, where the operation succeeds as expected (with the expected increase in connections). Schema change automation breaks unexpectedly on newer versions despite a valid replication setup.

Reported Affected Version/s: 3.5.2, 3.6.0, 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2432: While pt-replica-find includes internal logic for handling replication channels, it currently lacks a corresponding --channel command-line option. Attempting to use it results in an error:

$ pt-replica-find --channel=foo
Unknown option: channel

This prevents users from specifying a replication channel directly, limiting the tool’s usability in multi-channel replication environments.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2448: pt-k8s-debug-collector should not collect secret details of pgbouncer

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2446: When attempting to run pt-table-checksum with Group Replication enabled, and the tool returns the following error:

Error checksumming table schema.table: DBD::mysql::st execute failed: The table does not comply with the requirements by an external plugin. [for Statement "DELETE FROM percona.checksums WHERE db = ? AND tbl = ?" with ParamValues: 0=' ', 1=' '] at /bin/pt-table-checksum line 11323.

It gets suspected that this is caused by the tool attempting to set @@binlog_format := ‘STATEMENT’, which is not supported under Group Replication.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM [Percona Monitoring and Management]

PMM-13994: pmm_agent shows disconnected status despite active metrics collection, After a temporary connectivity issue, pmm_agent continues to display a Disconnected status in pmm-admin list, even though connectivity has been restored and dashboards are populating correctly.

$ pmm-admin list
...
pmm_agent Disconnected
...

Reported Affected Version/s: 2.43.2, 2.44.1, 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Restarting the pmm_agent should fix the issue.
Fixed/Planned Version/s: 3.4.0 - Not Yet Released

PMM-13905: When adding both a MongoDB Cluster and a standalone MongoDB Replica Set (not part of the cluster) to the same PMM environment (e.g., “test”), the MongoDB ReplSet Summary dashboard does not allow viewing the standalone RS.

The “cluster” filter cannot be unselected, making it impossible to visualize replica sets that are not associated with a defined cluster. As a result, only RSs from the cluster are visible, while standalone RSs are excluded from the dashboard view.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Whenever possible, use separate environments when adding the cluster and standalone RS nodes in PMM (e.g., use “env1” for the cluster and “env2” for the standalone RS).
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13910: In the MongoDB Sharded Cluster Summary and Collections dashboards, several graphs fail to populate correctly. Specifically:

Top Hottest Collections by Read
Top Hottest Collections by Write

These graphs display only admin, config, and system collections, even when other collections are under heavy traffic. Additionally, the Collections dashboard shows no data across all graphs—except for the first one (Top 5 Databases By Size), which populates as expected.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13950: In both PMM 2 and PMM 3, with MySQL 5.7 and MySQL 8.0, the server_uuid is not being collected from MySQL’s global variables as expected. Despite being available via SHOW GLOBAL VARIABLES LIKE ‘server_uuid’;, the PMM agent fails to parse or capture this value.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13792: In PMM 2.44.0, the Advisor Insights incorrectly reports that journaling is not enabled for MongoDB 7.0.9-15, despite journaling being enabled by default in this version.

Attempts to explicitly enable journaling in the MongoDB config result in a startup warning:

The storage.journal.enabled option and the corresponding --journal and --nojournal command-line options have no effect in this version... Journaling is always enabled. Please remove those options from the config.

False alert may confuse users and lead to misconfiguration attempts that prevent MongoDB from starting.

Reported Affected Version/s: 2.44, 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

Percona Kubernetes Operator

K8SPG-772: In the Percona PostgreSQL Operator, a runtime panic occurs when CompletedAt is nil and not properly checked before dereferencing, leading to a segmentation fault:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0]

Stack trace:

github.com/percona/percona-postgresql-operator/percona/watcher.getLatestBackup
 .../wal.go:123

github.com/percona/percona-postgresql-operator/percona/watcher.WatchCommitTimestamps
 .../wal.go:65

The CompletedAt field is not validated before being accessed in getLatestBackup(), which causes a crash during WAL watcher execution.

This panic can crash the operator’s goroutine, interrupting WAL monitoring and potentially affecting backup or failover logic.

Reported Affected Version/s: 2.5.0, 2.6.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.7.0 - Pending Release

K8SPG-792: The upstream operator includes functionality that allows cluster or operator administrators to define default PostgreSQL images for each major version using environment variables. This enables users to create clusters without explicitly specifying spec.image, as the operator will automatically apply the predefined image.

However, a recently introduced Patroni version check does not align with this behavior. It introduces a hardcoded dependency on spec.image, effectively bypassing the default image mechanism and undermining the intended feature.

Reported Affected Version/s: 2.6.0
Upstream Bug: Not Applicable
Workaround/Fix: A possible workaround exists by manually setting the Patroni version through annotations, but this is not ideal and diminishes the convenience and flexibility originally provided.
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

K8SPXC-1651: While testing the Pod Scheduling Policy feature in Everest, we encountered a situation where a PXC database pod remained in the Pending state. This occurred because Kubernetes was unable to schedule the pod on any available node due to an affinity configuration mismatch.

However, even after updating the affinity rules in the PerconaXtraDBCluster object, the new configuration was not propagated to the pod, and it remained in the Pending state.

The fact that the Pod remains stuck in Pending even after affinity is changed or removed — and only a manual kubectl delete pod resolves it — indicates that the operator fails to reconcile affinity changes properly.

Note: This issue affects other operators as well, not just PXC.

Reported Affected Version/s: 1.17.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 1.20.0 - Yet to be released

K8SPXC-1648: The PVC size is rounded up to the nearest whole GiB value (e.g., 1.2Gi becomes 2Gi). When a storage resize operation is triggered, the operator deletes the existing StatefulSet (STS) and recreates it with the new requested PVC size.

However, if the new requested size rounds up to the same value as the original, the operator does not recreate the STS. Instead, it attempts to update the existing STS, which leads to the following error:

Note: This issue affects other operators as well, not just PXC.

Error: failed to deploy pxc: updatePod for pxc: failed to create or update sts:
update error: StatefulSet.apps "minimal-cluster-pxc" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'ordinals',
'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden

Reported Affected Version/s: 1.17.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 1.20.0 - Yet to be released

PBM [Percona Backup for MongoDB]

PBM-1499: Restore to Missing Backup Fails with Unclear Error in Restore Custom Resource Status.
When attempting to restore from a backup that does not exist in the main storage, the operator logs correctly report the failure:

define base backup: get backup metadata from storage: get from store: no such file

However, the status.error field in the PerconaServerMongoDBRestore custom resource only shows a generic message:

error: 'define base backup: %v'

This results in a misleading or unclear error message being surfaced to the user through the custom resource, even though the logs contain the full and accurate description of the issue.

Reported Affected Version/s: 2.8.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1502: In PBM 2.9.0, running pbm profile sync fails with the error:

Error:  or --all must be provided

This occurs even when a valid profile name is given, such as:

pbm profile sync azure-blob

The issue affects all defined profiles (azure-blob, gcp-cs, minio) and prevents syncing individual profiles. This appears to be a bug where the CLI fails to recognize the passed argument.

Reported Affected Version/s: 2.9.0
Upstream Bug: Not Applicable
Workaround/Fix: Use pbm profile sync --all instead
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1538: Backup is marked as successful, despite the oplog not being uploaded.

Reported Affected Version/s: 2.4.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1551: In a single-node PSMDB replica set with one PBM agent, PBM occasionally re-executes the last issued command, causing errors like:

active lock is present

This typically occurs when the database is under load.

Reported Affected Version/s: 2.9.1
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1553: When restoring a 33-shard physical backup from a mixed (MongoDB Enterprise + Percona) production cluster into a Percona-only test cluster, PBM intermittently fails during the “clean-up and reset replicaset config” stage. Some shards restore successfully, while others restore only partially or fail entirely.

Both clusters run MongoDB 6, with FCV set to 5.
Restore uses --replset-remapping due to different replica set names
Issue affects restores regardless of matching node count per shard.

The problem appears tied to the restore logic handling replica set configuration cleanup.

Reported Affected Version/s: 2.9.1
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1564: A user environment experiences repeated failures during incremental backups with the error:

[ERROR: cannot use the configured storage: source backup is stored on a different storage]

Full, base incremental, and logical backups succeed without issues. There is no indication of recent storage or configuration changes, and pbm status shows backup attempts occur close together.

The issue temporarily resolves after running pbm config --force-resync, suggesting a possible bug in storage metadata syncing or internal state handling.

Reported Affected Version/s: 2.8.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Geeks Go Peaks: Building Community Through Adventure and Challenge

Iuliia (Julie) Slobodian — Wed, 25 Jun 2025 00:00:00 UTC

We believe this is something every strong team aspires to: taking interactions beyond the office, building a sense of community outside of work, and getting to know each other beyond job titles. That’s the kind of value people seek when they come to us.

People want to feel alive, to experience new sensations, to see the world from a different perspective - and share that journey with fellow enthusiasts.

Join Our Adventure Community: Explore, Connect, Thrive

The Geeks Go Peaks community is all about bringing people together through unforgettable experiences. Whether you’re seeking a relaxed getaway with your partner or craving the thrill of a high-altitude expedition, we’ve got events for every adventurer. From laid-back activities like quad biking or ice dipping to group micro-adventures such as SUP boarding in central Berlin, caving, or rock climbing, there’s something for everyone. For the bold, we organize serious expeditions, including high-altitude mountain ascents. So far, we’ve summited nine peaks, each offering breathtaking views, and we’re eyeing Mount Kazbek in Georgia and Thorong Peak in Nepal next.

Our mission?

To build a vibrant community of explorers who value shared experiences and the discovery of the world’s cultural and natural treasures. These adventures do more than just create memories - they spark lifelong friendships and expand your personal and professional networks. Through our events, people find their best friends and continue exploring the world together. It’s a joy to bring such value to the world.

Why the IT Community?

The IT industry is full of driven individuals who spend long hours behind screens. At some point, many realize they need more - more movement, more nature, more challenges. Outdoor activities like running, climbing, obstacle races, diving, or windsurfing not only boost physical health but also recharge mental well-being. IT professionals are often high-achievers, eager to push their limits, whether that’s training for a marathon or summiting iconic peaks. We connect these like-minded adventurers, fostering a community where they can inspire and support each other.

“You can’t beat shared suffering as team building activity” – Peter Zaitsev (Founder at Percona, Altinity, FerretDB, Coroot Database)

What We’ve Done

Our past events showcase the diversity of our adventures:

Aconcagua, Argentina: A grueling climb to nearly 7,000 meters rewarded us with jaw-dropping views and a profound sense of achievement. “Aconcagua was the trip of a lifetime. I’d recommend it to anyone in good shape and hungry for a challenge. You won’t regret it!”
Caving in Budapest: An underground world surprised even locals, proving adventure is often closer than you think. “I’m from Hungary and never explored the caves before. Doing it with GGP was the perfect way to finally change that - I absolutely loved it!”
Rock Climbing in Red Rock Canyon, Nevada: First-timers conquered the walls, forging bonds while cheering each other on. “I couldn’t have asked for a better first climbing adventure. The group, the views - everything was awesome.”
Northern Lights in Lapland: Snowy forests and vibrant skies created magical nights and dreams of future trips. “This wasn’t my first GGP trip - we’re practically family now. It was magic seeing the lights every single day.”
Via Ferrata in the Dolomites, Italy; a descent through Vallée Blanche in France; local events with paddleboarding and rock climbing in North Carolina… and that’s just the beginning. We already have so much planned for this year and the next!

Upcoming Adventures

Our calendar is brimming with thrilling plans. In August, we’ll tackle the exhilarating ascent of Mount Kazbek. Then we’re off to hike under the midnight sun in Svalbard, an awe-inspiring land where untouched Arctic nature reigns and polar bears outnumber people.

We’ll ride the wild waters of the Gully River during dam release season, set sail on catamarans through the British Virgin Islands in February, and explore wintertime Alaska with freeride skiing, glacier ice climbing, and evenings in a Nordic sauna.

And that’s just the beginning - Nepal, Bolivia, the Grand Canyon, the volcanoes of Vanuatu, Peru, and so much more are waiting.

For those in the USA, we’ve got local adventures like Grand Canyon Rim-to-Rim hikes, quick one- and two-day hikes, paddleboarding and kayaking, cycling, fishing and yachting, rock climbing, and even diving. And events tied to conferences such as Re:Invent, ATO Raleigh, All Things Open RTP, and Mountain Data & Dev Conference in Utah.

And this fall, on September 21, we’re hosting a women’s charity run - CODE PINK RUN! All proceeds will go to the Pretty In Pink Foundation to support women in their fight against breast cancer. Join us! We’d love to see everyone at this celebration of female solidarity!

Stay Connected

We keep our community engaged via newsletters, Meetup, Slack, Telegram, and Strava - where we train together.. Our Slack platform lets members discuss training plans, gear, and expedition prep, making us more than just a club, but a true community united by shared passions.

Anyone can suggest an adventure they’d like to be part of! We’re always excited about exploring new countries and trying new activities, sporty or not. Our goal is to help people find like-minded adventurers to reach new heights and share unforgettable experiences together!

Join us to explore the world, meet incredible people, and create stories you’ll tell for years.

Let’s adventure together!

PostgreSQL active-active replication, do you really need it?

Jan Wieremjewicz — Wed, 18 Jun 2025 00:00:00 UTC

Before we start, what is active-active?

Active-active, also referred to as multi-primary, is a setup where multiple database nodes can accept writes at the same time and propagate those changes to the others. In comparison, regular streaming replication in PostgreSQL allows only one node (the primary) to accept writes. All other nodes (replicas) are read-only and follow changes.

In an active-active setup:

There is no single point of write.
Applications can write to any node.
The database needs a way to sort out conflicts when two nodes try to concurrently change the same data.

That last point is the hardest one. PostgreSQL was not designed for concurrent writes from multiple nodes; it’s not a distributed database and does not leverage proprietary dedicated storage capabilities. So, every multi-primary implementation has to solve the issue of conflicting concurrent writes somehow. Some resolve conflicts using timestamps or priorities. Some push conflict resolution to the application. Some avoid it altogether by writing to separate subsets of data.

While simple in concept, implementing an active-active configuration is challenging.

pgactive to the rescue?

Last week, Amazon open-sourced its active-active replication extension, pgactive (https://github.com/aws/pgactive). While the extension has been generally available on AWS RDS since October 2023, there are unfortunately not many stories about it being used in production available. To be fair, I was not able to find any 😟

We often see both users and customers come asking for active-active or multi-master. These terms, while different, are so often used as synonyms that we’ve come to expect that. So, though I understand that every multi-master is active-active but not necessarily the other way around, for the sake of clarity, if I use one or the other term throughout this post, they will refer to the same concept.

As it is an open-source extension now, it immediately raised my interest. It seems that it could cover this ask from users I often speak with about their pains and needs. As a product manager, when I hear an ask, I always try to understand the reasons—whether it is a requirement, a need, or actually a solution that addresses one. For multi-master, my strong opinion is that it is a solution.

Key question: do you need it?

I like the opening of the talk Johnathan Katz gave on PGConf Europe 2023 in Prague:

The first thing I always say on the journey to active active is: do you really need it? Because it definitely solves a lot of problems (…) but it’s very hard to manage.

That is exactly the first question I ask when I hear someone asking for active-active. We have seen teams introduce active-active replication for the wrong reasons. Here I have to pause. Yes, as database experts, we have strong opinions about what are the right reasons for using multi-master. It’s not a silver bullet. It’s not “cool infra.” And using it without a good reason tends to hurt for a long, long time.

So, what are the reasons to use active-active? I do not claim to be able to cover all scenarios, but I hope this post raises enough eyebrows and sparks enough discussion to eventually have solid reading material for anyone considering active-active that will help them make an informed decision.

What are “good” reasons?

These are some of the situations where active-active might actually make sense. While there may be more, here’s my top 5:

Business continuity across regions: extreme HA needs (99.999% uptime)

Just to remind everyone what 5 nines mean, I will refer you to this message:

uptime → max monthly downtime:

99% → 7.3 hours
99.9% → 44 minutes
99.99% → 4 minutes
99.999% → 26 seconds
99.9999% → 3 seconds
99.99999% → 1/4 second

where do you land?
how much time/money would you invest to add a 9?
— Ben Dicken (@BenjDicken) May 23, 2025

26 seconds of downtime a month, that’s 312 seconds a year. Yes, 5.2 minutes a year.

Now think about the cost of delivering that sort of reliability. I find this Wikipedia page surprisingly helpful in conveying how little time for maintenance and failures is left with enough nines added.

Consider what it would take to absorb failures across data centers or cloud regions without rejecting writes or failing over manually. Active-active can help here because failover becomes instant and transparent; write traffic just shifts to surviving nodes.

But again, the cost will match the ambition. Do you plan HA within the same server room with separate power and networking? Or are you aiming for full geographic separation, to stay online even during a country-wide outage? These decisions massively influence the architecture, and together with your uptime goals, they define the cost. At this level, every part of the solution should reflect real business needs, because every layer of complexity adds expense. You can’t overstate the value of planning and proper analysis when building systems like this.
Write availability during regional failures

If your business serves a global customer base and absolutely must accept writes in more than one region, for example, to maintain uptime guarantees or continue operating during a regional outage, then active-active might be the least painful of the painful options.

This is not about low latency. This is about keeping write traffic flowing even when something breaks. That includes:
- Cloud infrastructure outages, such as full region loss or core service failure from your cloud provider:
  - AWS us-east-1 outage in June 2023 affected 104 services. A Parametrix Insurance report estimated a 24-hour outage in this region could lead to $3.4 billion in direct revenue loss.
  - Google Cloud outage in June 2025 impacted Spotify, YouTube, Twitch, and others.
  - AWS S3 outage in 2017 was caused by an internal mistake and disrupted GitHub, Slack, and more.
    
    Joys of the @internetofshit - AWS goes down. So does my TV remote, my light controller, even my front gate. Yay for 2017.
    — Brian (@Hamster_Brian) February 28, 2017
- Name resolution and routing issues, such as DNS or BGP failures that take your services offline even when your backend is healthy:
  - Dyn DNS attack in 2016 brought down Twitter, Reddit, and Spotify.
  - Facebook DNS and BGP misconfiguration in 2021 made their domains unreachable and left millions of users in the dark.
  - That day, Twitter (rest in peace) greeted the internet:
All jokes aside, these are serious risks. If this kind of failure is unacceptable for your business, and you are willing to take on the operational weight and cost (we will get to that), active-active may be the right tool.

But be honest about what you are solving. If your system demands strong consistency, every transaction still needs coordination across nodes. For example, if a user in Australia writes to a local node, and the other node is in the United States, that write still involves a round trip to the United States before it can commit. That round trip adds latency, not removes it. While it may be 150-200ms on average for the Australia to USA round trip, it adds up with volume.

The real benefit of active-active here is not performance. It is write availability during failure. If your business cannot afford to reject writes when a region goes dark, and you are prepared for everything else that comes with this decision, this might be one of the rare cases where active-active makes sense.

Just be clear, what you are solving here is not distributed latency, but write continuity when something fails.
Migrating legacy architectures

If you’re part of an organization moving away from systems like Oracle RAC or GoldenGate, where distributed write semantics were either built-in or at least promised, you may face business or political pressure to deliver “the same thing” on PostgreSQL.

In these cases, active-active might be the shortest path to satisfying the checkbox. But it’s almost always a transitional compromise, not the destination. As any compromise, that’s not going to be all pleasant. The technically better (but less politically correct) move is usually to re-architect for clearer ownership of writes and better separation of concerns.

If you can push for that path, do it. If not, be aware of the cost you’re inheriting.
Application performance (not database performance)

In the end, what you are really trying to improve is not the database throughput, but the end-user experience. Active-active may be worth considering not for improving database internals, but for reducing perceived latency in globally distributed apps or smoothing responsiveness during network transitions.

In rare cases, this might justify active-active if the application can route users to their nearest region and issue local writes. But your app must be built for it. Deterministic conflict handling, idempotency, and careful session management are must-haves in such a case.

If your database is fast, but the user still feels lag because the write travels halfway across the planet, active-active might help. But this should be a last resort, not a default choice.
Local HA in disconnected or semi-connected environments

In edge computing, retail stores, ships, or military use cases, you might want each node to function independently to address intermittent connectivity. In such scenarios, you will still be able to write locally when the network is not available. When the network comes back, the changes are going to be synced. While conflict avoidance may be the strategy you go for, in the end, it’s going to become a cost of conflict resolution.

What’s next?

In the next blog post I will focus on the bad reasons to consider active-active replication and on the cost that should not be forgotten. Stay tuned!

What's new in PMM 3.2.0: Five major improvements you need to know

Catalina Adam — Tue, 03 Jun 2025 00:00:00 UTC

PMM 3.2.0 brings some long-awaited fixes and new capabilities. You can now install PMM Client on Amazon Linux 2023 with proper RPM packages, get complete MySQL 8.4 replication monitoring, and track MongoDB backups directly in PMM.

Here’s what’s most important in this release:

1. Native Amazon Linux 2023 support - no more workarounds

What’s new: If you’ve been running PMM Client on AL2023 and dealing with complex manual installations, those days are over. You can now install PMM Client through native RPM packages just like any other supported platform.

What this means for you: Streamlined setup means you can get your Amazon Linux 2023 environments monitored faster.

2. Complete MySQL 8.4 replication monitoring

What’s new: PMM now fully supports replication monitoring for MySQL 8.4, including key metrics like IO Thread status, SQL Thread status, and Replication Lag. MySQL 8.4 changed how these metrics are exposed, and earlier PMM versions couldn’t track them accurately.

What this means for you: With the upgraded MySQL Exporter (v0.17.2), you now get complete replication monitoring across all supported MySQL versions (5.7, 8.0, and 8.4) without any visibility gaps.

3. MongoDB backup monitoring dashboard

What’s new: The new PBM Details dashboard lets you monitor MongoDB backups directly in PMM using the PBM collector. Instead of switching between PMM and separate backup tools, you now get a real-time, unified view of backup activity across replica sets and sharded clusters.

What this means for you: Easily track backup status, configuration, size, duration, PITR status, and recent successful backups—all in one place. No more tool-hopping to stay on top of your backup operations.

4. Grafana 11.6 upgrade with enhanced capabilities

What’s new: PMM now ships with Grafana 11.6, delivering enhanced visualization capabilities and improved alerting workflows.

Key features include:

Alert state history for reviewing historical changes in alert statuses
Improved panel features and visualization actions
Simplified alert creation with better UI workflows
Recording rules for creating pre-computed metrics
Navigation bookmarks for quick dashboard access

What this means for you: These enhancements make your monitoring dashboards more interactive, your alerting more sophisticated, and your overall monitoring workflow more efficient.

5. Dramatically improved Query Analytics performance

What’s new: We’ve optimized QAN filter loading performance to reduce the number of processed rows by up to 95% in large environments.

What this means for you: Filters on the PMM Query Analytics page now load much faster, making the interface more responsive to improve your troubleshooting efficiency.

Additional improvements worth noting

Beyond these five major enhancements, PMM 3.2.0 also introduces:

Secure ClickHouse connections with authenticated credential support
MongoDB Feature Compatibility Version (FCV) panels for better cluster version visibility
Nomad integration laying groundwork for future extensibility
Numerous bug fixes improving stability across ProxySQL, PostgreSQL, and MySQL monitoring

Getting started with PMM 3.2.0

Ready to experience these improvements? Set up your PMM 3.2.0 instance using our quickstart guide or upgrade your existing installation following our migration documentation.

For existing users with external PostgreSQL databases, make sure to review the external PostgreSQL configuration migration guide before upgrading.

Questions or feedback? We’d love to hear from you! Connect with the Percona community through our forums or join the conversation on our community channels.

PostgreSQL 18 - Top Enterprise Features (fast read)

Jan Wieremjewicz — Mon, 26 May 2025 00:00:00 UTC

So the Beta1 is available for PostgreSQL 18 and while not all the features have to make it to GA, we can surely hope they do!

Taking a close look at what’s coming, here below is the selection of what excites me in particular:

1. OAuth 2.0 authentication support

→ Finally aligns with modern enterprise SSO and identity standards (e.g., Okta, Azure AD). A major win for security teams and regulatory compliance.

2. Logical replication from standbys now with conflict logging

→ Now you can replicate from replicas not only primary nodes and thanks to conflict logging troubleshooting issues moves closer to what the users have been asking for. It’s a big step toward robust, native, HA-friendly logical replication. Not yet there, but on the right path!

3. Asynchronous I/O (AIO)

→ Modern async reads improve performance, especially under heavy parallel workloads. Foundation for future IO improvements, and also a feature that scratches an itch for a lot of cloud deployments.

4. Faster & safer major upgrades

→ pg_upgrade enhancements like parallel upgrade checks (--jobs), safer upgrades (--swap), and planner stats carried forward = faster version adoption and smoother upgrades for large clusters.

5. Observability++

→ Enhanced EXPLAIN statement and pg_stat_io improvements enhance understanding and optimize I/O behavior across tables, indexes, and WAL. This reduces the need for external monitoring tools.

Other Notable Features

While these are the top mentions, the goodies are not limited to only these. Some smaller improvements are just as exciting. Looking at these, the top interesting ones for me are:

New extension_control_path Server Variable

→ Enables operators to manage PostgreSQL extensions via Kubernetes image volumes (Kubernetes 1.33 image volumes) without modifying the base image.

→ Huge win for immutable image strategies and GitOps-friendly operator design.
Easier online constraints management

→ Add new NOT NULL constraints without locking large tables using NOT VALID

→ Use NOT ENFORCED foreign keys and CHECK to model relationships without runtime overhead
Smarter index maintenance (bottom-up deletion)

→ Reduces bloat, lowers vacuum overhead.
SQL/JSON path support + JSON performance gains

→ Enables document-style querying at scale.
Better insights into queries and vacuum

→ EXPLAIN now includes buffer usage in subplans, triggers, and functions which helps spot slow parts

→ pg_stat_all_tables now tracks how much time vacuum and autovacuum take per table
Indexing improvements

→ Parallel GIN builds help speed up full-text and vector searches key for hybrid full-text and vector search

→ B-tree skip scans make range and selective queries faster

What enterprises would want in PostgreSQL 19+

There is already a lot to like in this release, but based on what we hear from users, customers, and our own teams, here is what is still high on the list. First up and this one’s close to home is that we’d love to see the Transparent Data Encryption (TDE) patches from Percona Server for PostgreSQL make their way upstream. That would allow users to benefit from pg_tde directly in Community PostgreSQL Server. The rest of the list is a mix of long standing asks and forward looking ideas. It is a wishlist for sure, but one we hope to help make real over time:

Built-in Logical Conflict Resolution Algorithms

→ Support for conflict-handling strategies (e.g., last-write-wins, column-level rules) would simplify bidirectional replication and eliminate the need for custom frameworks, opening the door for fully open-source multi-master setups.
Logical failover orchestration

→ Seamless promotion and failover in logical topologies, with less reliance on external tooling. This would be great from the perspective of both Kubernetes deployments as well as the ease of use for HA solutions out there.
Better integration with external auth systems

→ Automatic PostgreSQL user creation based on OAUTH/LDAP roles at login, reducing operational burden for large-scale identity management and central access control.
Pluggable or columnar storage support

→ Native support or better extension hooks for OLAP and hybrid workloads, closing the gap with cloud-native alternatives like Citus or Redshift.
Sharding to provide horizontal scaling

→ Transparent sharding is a highly desired capability that becomes critical as workloads scale. While not always needed on day one, having built-in sharding means teams can grow without reinventing the wheel. Lack of it makes horizontal scaling complex, requiring more expertise and introducing higher operational overhead for DBA teams.

OS Platform End of Life (EOL) Announcement for Ubuntu 20.04 LTS

Julia Vural — Thu, 22 May 2025 00:00:00 UTC

Ubuntu 20.04 LTS (Focal Fossa) is scheduled to reach its official end of life on May 31, 2025. In alignment with the upstream vendor’s lifecycle, we are also ending platform support for Ubuntu 20.04 for all our MySQL related product offerings. This date and others are published in advance on our Percona Release Lifecycle Overview page.

As part of our support policy, Percona will continue to provide advisory support for databases running on EOL platforms, but effective April 1, 2025, we have discontinued:

Delivery of new packages or binary builds
Distribution of hotfixes or bug fixes for Percona software on Ubuntu 20.04
OS-level support for issues not related to the database itself

However, all existing packages will remain available for download.

We are committed to ensuring a smooth and seamless experience for our users. Migrating to a supported operating system will ensure that you continue to receive security updates, bug fixes, and new features for our products.

We encourage you to take action promptly and plan your migration from Ubuntu Focal to a supported operating system. Each operating system vendor has different supported migration or upgrade paths to their next major release. Please contact us if you need assistance migrating your database to a different supported OS platform – we will be happy to assist you!

Note: If requested by a customer, we may provide updated Percona packages for up to six months beyond the Percona EOL date, as a courtesy grace period.

Percona Bug Report: April 2025

Aaditya Dubey — Mon, 19 May 2025 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs.

Percona Server/MySQL Bugs

PS-8846: The ALTER INSTANCE RELOAD TLS thread gets stuck. This happens in instances with a high new connections rate (>60/s) but not in all instances. Percona and Upstream did not fix it properly because the current implementation of ALTER INSTANCE RELOAD TLS requires all existing SSL connections to be closed. “A thread that executes ALTER INSTANCE RELOAD TLS tries to acquire an RCU Lock(Read-Copy-Update), waiting for the number of readers to become 0. In other words, when the server has a constant flow of new incoming SSL connections, the chances of acquiring this lock are pretty low.” Therefore, Percona and Oracle only partially fixed this; this fix should improve this behaviour.

Reported Affected Version/s: 8.0.32-24, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PS-9609: The audit_log_filter can’t be installed when the server is using component_keyring_kmip

Reported Affected Version/s: 8.0.39-30

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42-33

PS-9628: The binlog_encryption does not work with component_keyring_kmip

Reported Affected Version/s: 8.0.40-31

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42-33

PS-9664: With a very simple workload, MyRocks allocates a lot of memory and does not free it when the workload finishes. All instrumentation available either does not provide information about memory allocated or provides only part of it. As a result, users cannot predict how much RAM to install on the server that runs the MyRocks storage engine. With InnoDB same workload requires about 1.7G and frees about 0.5G once the job is finished.

Reported Affected Version/s: 8.0.39-30

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PS-9719: When changing binlog_transaction_dependency_tracking in high load workload, MySQL got a segmentation fault.

Reported Affected Version/s: 8.0.40, 8.0.41-32, 8.0.33

Upstream Bug: 117922

Workaround/Fix: “set global binlog_transaction_dependency_tracking = commit_order;”

Fixed/Planned Version/s: 8.0.42-33, 8.4.5-5

PS-9768: An unexpected duplicate error occurs when running a select query with a group by JSON data.

Reported Affected Version/s: 8.0.41-32, 8.4.4-4

Upstream Bug: 117927

Workaround/Fix: Rebuilding the table with “alter table db_name.table_name = rocksdb;” can fix the issue.

Fixed/Planned Version/s: To Be Determined

Percona Xtradb Cluster

PXC-4512: When DDLs run against tables with foreign key references when there is a write load simultaneously. The issue is typically triggered during pt-online-schema-change execution, and after a dozen or so iterations, random PXC nodes will terminate with MDL BF-BF conflict. Sometimes, the writer fails, and sometimes, the other nodes, but it can be reproducible with just the RENAME query.

Reported Affected Version/s: 8.0.33-25, 8.0.35-27, 8.0.36-28, 8.0.37-29, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Only a full write stop would help.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4648: After upgrading from 8.0.41 to 8.4.3, the node can’t join the group with the following error.

[ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.4.3/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():343: Group requested gcs_proto_ver: 5, max supported by this node: 4.Upgrade the node before joining this group.Need to abort.

The problem is that 8.0.41 and 8.4.4 use Galera 26.4.21. It introduced the GCS protocol version 5, 8.0.40 and 8.4.3 using Galera 26.4.20. So, the node that doesn’t understand protocol 5 tries to connect to a cluster that uses protocol 5.

It is fixed as a documented bug, which can be seen here.

Reported Affected Version/s: 8.4.3

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.4.4

PXC-4638: The binlog_utils_udf plugin fails to access binlog files correctly after an SST (State Snapshot Transfer) due to inconsistencies in the mysql-bin.index file. After an SST, the first entry in the mysql-bin.index file can be incorrectly formatted with a relative path, while subsequent entries use absolute paths. This inconsistency can prevent the binlog_utils_udf plugin from locating the correct binlog files.

The resulting mysql-bin.index content after SST might appear as:

mysql-bin.000010
/home/user/sandboxes/pxc_msb_8_0_40/node2/data/mysql-bin.000011
/home/user/sandboxes/pxc_msb_8_0_40/node2/data/mysql-bin.000012 

Reported Affected Version/s: 8.0.40, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Rewriting binlog.index and adding the “./” prefix to the first entry, and running flush binary logs resolved this issue.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-3576: Deploying a new installation using the setting lower_case_table_names=1 on the startup generates the following entry on the log:

[Warning] [MY-010324] [Server] 'db' entry 'percona_schema mysql.pxc.sst.role@localhost' had database in mixed case that has been forced to lowercase because lower_case_table_names is set. It will not be possible to remove this privilege using REVOKE.

Looking at the mysql.db, we can see the deployment was able to create the mysql.pxc.sst.role mapping an uppercase database name:

mysql> select * from mysql.db where user='mysql.pxc.sst.role';
+-----------+--------------------+--------------------+------+
| Host | Db | User | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Create_tmp_table_priv | Lock_tables_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Execute_priv | Event_priv | Trigger_priv |
+-----------+--------------------+--------------------+-------------+-------------+-------------+-------------+-------------+-----------+------------+-------------+
| localhost | PERCONA_SCHEMA | mysql.pxc.sst.role | N | N | N | N | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N |
| localhost | percona_schema | mysql.pxc.sst.role | N | N | N | N | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N |

This produces a duplicate entry for the percona_schema database and a warning message on the mysql.log.

Reported Affected Version/s: 8.0.21-12.1, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4657: When executing DML and DDL on the same table, the DML will get a deadlock error. If the DML does not change the data but matches, it won’t be replicated, for example.

mysql > UPDATE test.t SET d = d LIMIT 1;
Query OK, 0 rows affected (0.01 sec)
Rows matched: 1 Changed: 0 Warnings: 0

If the table contains a trigger, the DML does not get a deadlock error and will be replicated to other nodes.

CREATE TRIGGER `t_on_update`
AFTER UPDATE ON `t`
FOR EACH ROW
BEGIN
 INSERT INTO t_history (`d`)
 VALUES (NEW.`d`);
END

When other nodes apply the DML and DDL, the applier threads will get an MDL BF-BF conflict.

Reported Affected Version/s: 8.0.40, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Using a single applier thread will skip the bug, but it may introduce a performance issue, such as flow control.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4664: Converting thd->rli_slave to target type Slave_worker* causes a segmentation fault.

Reported Affected Version/s: 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

Percona Toolkit

PT-2392: pt-online-schema-change resume functionality doesn’t work with ADD INDEX

Reported Affected Version/s: 3.6.0, 3.7.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2322: The issue reports that pt-mysql-summary does not correctly detect and display the jemalloc memory management library, even when it is enabled. Despite jemalloc being loaded and visible in the process memory map (/proc//maps), the output from pt-mysql-summary is missing this information in some cases, unlike version 3.2.1, which correctly identifies and reports it.

Reported Affected Version/s: 3.5.6, 3.5.7

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2422: When using the –history option with pt-online-schema-change (pt-osc), the query responsible for updating the history entry with the new_table_name is not appropriately constrained by a primary or unique key. As a result, this UPDATE operation can inadvertently modify all entries in the history table, rather than just the intended row.

This behavior can lead to significant issues when running multiple schema change operations in parallel, as the history entries for different migrations may interfere with each other, causing data consistency problems.

Additionally, if a migration is paused and later resumed, this lack of key constraint can result in only a subset of the data being correctly copied to the new table, potentially leading to partial data loss or corruption when the final table swap occurs.

Reported Affected Version/s: 3.5.6, 3.5.7

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2442: Multiple security vulnerabilities have been identified in the latest version of Percona Toolkit, including:

CVE-2024-56171: Use-After-Free Vulnerability in libxml2

CVE-2024-12797: OpenSSL Raw Public Key Authentication Vulnerability

CVE-2022-37967: Windows Kerberos Elevation of Privilege Vulnerability

CVE-2025-24928: Stack-Based Buffer Overflow in libxml2

It is recommended that the associated advisories be reviewed and the necessary patches or upgrades be applied to mitigate the risk.

Reported Affected Version/s: 3.7.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.0-1

PMM (Percona Monitoring and Management)

PMM-13694: When using a non-default pg_stat_statements.max value, the calculated QPS displayed in QAN may be wrong.

Reported Affected Version/s: 2.38.0, 2.44.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

PMM-13807: pmm-agent crashed at query.Fingerprint due to query including a column named “value”

Reported Affected Version/s: 2.44.0, 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

PMM-13847: PMM 3.0 doesn’t support running on a different uid/gid in Kubernetes

Reported Affected Version/s: 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PMM-13984: Percona Monitoring and Management (PMM) version 3.1 with OVA image currently cannot be imported into VMware environments.

Reported Affected Version/s: 3.1.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PMM-12784: This error invalid GetActionRequest.ActionId: value length must be at least 1 runes. occurs in the QAN (Query Analytics) dashboard, indicating that the ActionId field in the GetActionRequest is empty or improperly formatted. This typically results from a missing or incorrectly populated Action ID parameter, which is required for retrieving query data.

Reported Affected Version/s: 2.33.0, 2.41.0, 2.44.0, 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

Percona XtraBackup

PXB-3421: XtraBackup fails when the –databases parameter contains a very long list of databases or has a large amount of whitespace before the actual database names. When the –databases parameter is provided with 1859 whitespace characters before a table name (e.g., db01.t1), XtraBackup crashes with a signal error. If the number of whitespace characters is reduced to 1858 or fewer, the backup proceeds successfully without error.

Reported Affected Version/s: 8.0.35-31

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PXB-3426: Using KMIP component causes double free of memory on error paths.

Reported Affected Version/s: 8.0.35-31

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.35-33, 8.4.0-3

PXB-3392: xtrabackup doesn’t pick up –innodb-log-group-home-dir config parameter

Reported Affected Version/s: 8.0.35-30, 8.0.35-31, 8.0.35-32

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

Percona Kubernetes Operator

K8SPG-703: When using ttlSecondsAfterFinished, there is a potential race condition where the backup jobs may be deleted before the Percona operator has had sufficient time to reconcile the perconapgbackups objects. This issue can occur even with relatively long timeouts like 1m, 5m, or 30m, not just extremely short intervals.

Reported Affected Version/s: 2.5.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.7.0

K8SPSMDB-1263: While creating a 1.3TB logical backup, the replica’s state changes to “errored” after approximately 16 hours with the message:

“failed to find CERTIFICATE”

despite the backup continuing to run and eventually completing. This raises concerns about the validity of the backup and the ability to restore from it without manually altering its status to “success.”

Reported Affected Version/s: 1.13.0, 1.14.0, 1.15.0, 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.20.0

K8SPSMDB-1292: When spec.tls.mode is set to requireTLS, physical backup restores fail with a “server selection timeout” error. This occurs because the operator cannot establish a secure connection to the MongoDB server, resulting in closed socket errors and the inability to disable Point-in-Time Recovery (PiTR).

Reported Affected Version/s: 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.21.0

K8SPSMDB-1294: When using MCS on GKE 1.30, an API version mismatch occurs, resulting in the error:

“no matches for kind ‘ServiceImport’ in version ’net.gke.io/v1alpha1’”

This indicates that the expected ServiceImport kind is not available in the specified API version, preventing proper service discovery.

Reported Affected Version/s: 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.20.0

K8SPSMDB-1336: Restoring a backup into a new Kubernetes cluster can lead to “Time monotonicity violation” errors on config servers and mongos, causing the pods to restart. This occurs when the restored chunk version timestamps are earlier than the expected timestamps in the new cluster, resulting in tripwire assertions and persistent crashes.

Reported Affected Version/s: 1.19.1

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

K8SPXC-1548: Failed to delete old backups on Google Cloud Storage

Reported Affected Version/s: 1.14.0, 1.15.1

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.18.0

PBM (Percona Backup for MongoDB)

PBM-1482: Selective Restore with replset-remapping hangs on oplog replay and doesn’t finish

Reported Affected Version/s: 2.5.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

PBM-1487: Error Location6493100 on mongos after successful logical restore or PITR

Reported Affected Version/s: 2.8.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

PBM-1531: PBM Restore getting randomly stuck

Reported Affected Version/s: 2.6.0, 2.7.0, 2.8.0, 2.9.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Progress report on pg_tde - GA extension is nearer every day!

Jan Wieremjewicz — Thu, 08 May 2025 00:00:00 UTC

Another week, another blogpost about the state of open source Transparent Data Encryption (TDE) for PostgreSQL.

First off, thank you for all the feedback shared so far!

Whether it’s reports about deployment issues with pg_tde, integration with KMS, missing features or gaps in our documentation, we truly appreciate it! Your input helps us build a better, more complete solution and to properly prioritize what’s next.

What’s the word?

We know many of you are eagerly waiting for a production-ready release of pg_tde, and today we’ve got some good news! You may remember the last release was a Release Candidate (RC), now we’re gearing up to launch RC2.

If you’re not familiar with the terminology, this simply means we’re one step closer to General Availability (GA).

Our nightly builds are always there if you want to keep up with the latest updates, but this new milestone release highlights the fixes to some pain points that our tests and your feedback helped uncover.

When to expect RC2?

This post is just a heads-up. The actual release is planned for early next week (after May 12, 2025). Keep an eye out.

In the meantime, feel free to check what’s coming in RC2. And if you’d like to benefit from these improvements right away, nightly builds are the way to go.

What’s new in RC2 compared to RC1?

Some recent documentation updates are already live, and we’d love to hear your thoughts on them. Don’t be a stranger, let us know how do you find them!

As for the code, here are the key changes:

KMS configuration improvements including:
- New parameter for passing a client certificate when configuring a KMIP provider
- Compatibility updates for key management systems (KMS)
  - Thales CypherTrust
  - Fortanix Data Security Manager
- Validation enforcement when adding key provider configurations
WAL improvements, hardening encryption in our beta WAL support
Security enhancements for multi-tenancy scenarios
Other updates like:
- Added pg_tde_verify_default_key() and pg_tde_default_key_info() functions
- Fixed support for logical replication

How to use nightly builds

In case you’re wondering what nightly builds are: think of them as automatically generated versions of the software with the latest changes, usually built overnight 😎. They’re useful for testing (especially for integration testing) and for those, like our developers, who want to work with the freshest code.

No elephants have been hurt to create our nightly builds! We rely solely on CI/CD automation!

You can find them in our experimental repo. Do note, they’re currently only available for x86_64 and a limited set of operating systems:

Setting Up and Monitoring MongoDB 8 Replica Sets with PMM 3 Using Docker: A Beginner-Friendly Guide

Daniil Bazhenov — Tue, 18 Mar 2025 00:00:00 UTC

This guide explains how to set up a MongoDB 8 Replica Set and monitor it using PMM 3, all within Docker. We’ll guide you through the steps to create a local environment, configure the necessary components, and connect them for effective monitoring and management.

The guide is written in detail for beginners. In the conclusion section there are ready configurations for the experienced.

The recent release of Percona Monitoring and Management 3 introduces several new features:

Upgraded Grafana version for an improved user experience.
Rootless containers for enhanced security.
ARM support for the pmm-client.
Monitoring capabilities for MongoDB 8, along with new dashboards.

This article is intended for developers and DBAs who want to experiment with these tools locally using Docker. We will cover the following steps to set everything up and test the functionality:

Launch the PMM 3 pmm-server for monitoring and open it in a browser.
Install MongoDB, starting with a standalone server, and then convert it into a Replica Set with three nodes. Percona Server for MongoDB images are used in this article.
Configure the pmm-client for MongoDB to send metrics to the pmm-server.
Explore the PMM 3 dashboards.

We will use Docker Compose to define and run multiple containers efficiently through a single docker-compose.yaml file.

If you’re ready to dive into the world of Dockerized database monitoring and management, let’s get started!

Step Zero: Preparation

To get started, you need a terminal to run Docker commands and a text editor to modify the docker-compose.yaml file.

You also need Docker installed on your system. If Docker is not installed, follow these instructions to set it up:

Docker Desktop: This application includes both Docker and Docker Compose. It is available for multiple operating systems. This guide uses Docker Desktop on macOS with an Apple Silicon ARM processor. Download Docker Desktop
Docker and Docker Compose (separately): If preferred, install Docker and Docker Compose individually. Use the following links for guidance:
- Download Docker for your OS
- Download Docker Compose

Command to Verify Installation:

Verify Docker Compose Installation: If you’re using Docker Desktop, Docker Compose is included. If you installed Docker Compose separately, you can verify the installation with:

docker-compose --version

This should return the version of Docker Compose installed.

sh
➜ community git:(main) ✗ docker-compose --version
Docker Compose version v2.21.0-desktop.1

Once Docker and Docker Compose are installed and verified, you are ready to move on to the next steps of deploying PMM 3 and MongoDB in Docker.

Project Directory

Create a directory where we’ll store the configuration and necessary files. For example, I created a directory named pmm-mongodb-setup and navigated into it.

bash
mkdir pmm-mongodb-setup
cd pmm-mongodb-setup

Step One: Starting PMM 3 Using Docker Compose

To start with, we will launch PMM 3, specifically the pmm-server, which we will later access via a browser.

Create the Docker Compose Configuration File

First, create a file named docker-compose.yaml in your project directory. Then, copy and paste the following configuration into the file to set up PMM 3:

yaml
version: '3'

services:
 pmm-server:
 image: percona/pmm-server:3
 platform: "linux/amd64" # Specifies that Docker should use the image for the amd64 architecture, which is necessary if the container doesn't support ARM and your host system is ARM (e.g., Mac with Apple Silicon).
 container_name: pmm-server
 ports:
 - 8080:80
 - 443:8443
 healthcheck: # Defines a command to check the container's health and sets the timing for executions and retries.
 test: ["CMD-SHELL", "curl -k -f -L https://pmm-server:8443 > /dev/null 2>&1 || exit 1"]
 interval: 30s
 timeout: 10s
 retries: 5

Explanation:

We define the first pmm-server service to use the percona/pmm-server:3 image

platform: This parameter ensures compatibility with ARM-based processors, such as my Mac with Apple Silicon, by instructing Docker to use the image for the amd64 architecture.

healthcheck: This parameter performs a check to confirm that the container has started successfully.

Start the PMM 3 Container

Save the docker-compose.yaml file, and then use the following command to start the PMM 3 container:

docker-compose up -d

This command will download the PMM 3 image if it is not already available locally and start the container in detached mode.

Expected result in the terminal:

shell
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 2/2
 ✔ Network pmm-mongodb-setup_default Created 0.1s
 ✔ Container pmm-server Started 0.9s
➜ pmm-mongodb-setup

Expected result in Docker Desktop:

Open PMM in a browser

Now you can open PMM in your browser at http://localhost. Use admin/admin as the username and password to log in. When prompted to change the password, skip this step by clicking the Skip button. Since this is a test setup, we need a simple password for other containers.

Now, PMM 3 is successfully running using Docker Compose.

Step Two: Starting MongoDB

We start by launching a standalone MongoDB service. This simple configuration allows us to quickly understand how it operates before moving on to a more advanced setup with a Replica Set.

To keep the database data persistent, even after a restart, a volume is used for MongoDB data storage. Add the following configuration to the docker-compose.yaml file under the pmm-server service:

yaml
 mongodb:
 image: "percona/percona-server-mongodb:8.0-multi"
 volumes:
 - mongodb-data:/data/db
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 ports:
 - "27017:27017"
 command: ["mongod", "--port", "27017", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 healthcheck:
 test: ["CMD-SHELL", "mongosh --eval 'db.adminCommand(\"ping\")' --quiet"]
 interval: 30s
 timeout: 10s
 retries: 5

volumes:
 mongodb-data: # MongoDB data storage volume

Explanation:

environment: Configures the root user’s credentials (username and password).

command: Sets additional parameters for MongoDB:

bind_ip_all: Allows external access to the database, for instance, through MongoDB Compass.

profile: Enables profiling settings to support Query Analytics (QAN) in PMM.

healthcheck: Ensures the container starts successfully by executing a health check command.

volumes:: Creates the specified volume for MongoDB data storage. Defines a Docker volume to store MongoDB data persistently.

Launching the Configuration

Save the updated docker-compose.yaml file and launch the MongoDB service by running the following command:

docker-compose up -d

Docker Compose checks the configuration and starts the MongoDB container.

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 3/3
 ✔ Volume "pmm-mongodb-setup_mongodb-data" Created 0.0s
 ✔ Container pmm-mongodb-setup-mongodb-1 Started 0.1s
 ✔ Container pmm-server Running 0.0s
➜ pmm-mongodb-setup

Verifying MongoDB in Docker Desktop:

Step Three: PMM Client

At this point, we have both the PMM Server, which is accessible in the browser, and MongoDB running. To transfer metrics from MongoDB to the PMM Server, a container with pmm-client needs to be started.

Add another service pmm-client to the docker-compose.yaml file, right after mongodb. Note that `volumes:`` must remain at the bottom of the file.

yaml
 pmm-client:
 image: percona/pmm-client:3
 container_name: pmm-client
 depends_on:
 pmm-server:
 condition: service_healthy
 mongodb:
 condition: service_healthy
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: admin
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 PMM_AGENT_SETUP: 1
 PMM_AGENT_SETUP_FORCE: 1
 PMM_AGENT_PRERUN_SCRIPT: >
 pmm-admin status --wait=10s &&
 pmm-admin add mongodb --username=databaseAdmin --password=password --host=mongodb --port=27017 --query-source=profiler

Explanation:

depends_on: Ensures that pmm-client starts only after pmm-server and mongodb have started successfully and passed the healthcheck.

PMM_AGENT_PRERUN_SCRIPT: The pmm-admin add mongodb command adds the MongoDB service to PMM for monitoring.

PMM_AGENT_SERVER_ADDRESS: Specifies the PMM Server address and uses port 8443.

PMM_AGENT_SERVER_USERNAME and PMM_AGENT_SERVER_PASSWORD: Update these values if you have changed the PMM login credentials.

Applying the Updated Configuration

Run the following command to apply the updated docker-compose.yaml configuration and start the pmm-client service:

docker-compose up -d

After running the command, you should see the pmm-client container start successfully:

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 3/3
 ✔ Container pmm-mongodb-setup-mongodb-1 Healthy 0.0s
 ✔ Container pmm-server Healthy 0.0s
 ✔ Container pmm-client Started 0.1s
➜ pmm-mongodb-setup

Open PMM in your browser. On the homepage, you should now see MongoDB listed as a monitored service:

Step Four: Convert a Standalone MongoDB to a Replica Set

A single MongoDB instance works well for development and testing. At this point, you can already connect to the database from your application or tools such as MongoDB Compass and run various NoSQL queries.

However, the goal is to deploy a Replica Set consisting of three replicas, which is recommended for production and operational setups. For now, we set up the Replica Set on a single machine with a single docker-compose, which is intended for testing and development purposes only.

Both the mongodb and pmm-client services need to be updated.

Stopping All Services

First, stop all the currently running services:

docker-compose down

Result:

sh
➜ pmm-mongodb-setup docker-compose down
[+] Running 4/4
 ✔ Container pmm-client Removed 0.3s
 ✔ Container pmm-mongodb-setup-mongodb-1 Removed 0.4s
 ✔ Container pmm-server Removed 4.5s
 ✔ Network pmm-mongodb-setup_default Removed

Generating a Key File

To run three MongoDB replicas that can securely communicate with each other, a key file is required.

Create a secrets folder next to the docker-compose.yaml file and generate the mongodb-keyfile:

sh
mkdir secrets
openssl rand -base64 128 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile

In this case, the mongodb-keyfile will be generated inside the secrets folder. If your operating system does not support these commands, manually create the secrets folder and add a mongodb-keyfile with the following content:

rVLhIK2PhZKGxysjwMR4t1OmNppqdAzEs408hrbzg95D146mn9YENixId6pvIGCA
Cy9hc1k6OKKabbv7Rm347NwSFxbdPPx0/jnaO80U/a6/mv0XqSmEl8wdR91b4jIm
d98LobplwRs4b7g9cnLMUAIULr0WG+J36NtKIA6q4eE=

Add three mongodb services for Replica Set

Remove the existing mongodb service from the docker-compose.yaml file and replace it with three Replica Set services:

yaml
 mongodb-rs101:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs101
 ports:
 - "27017:27017"
 command: ["mongod", "--port", "27017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-101:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

 mongodb-rs102:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs102
 ports:
 - "28017:28017"
 command: ["mongod", "--port", "28017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-102:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 28017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

 mongodb-rs103:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs103
 ports:
 - "29017:29017"
 command: ["mongod", "--port", "29017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-103:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 29017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

Key Points:

./secrets:/etc/secrets:ro: Mounts the key file from your disk into the container.

ports:: Each replica runs on a separate port since they are hosted on the same machine.

command::

--keyFile: Enables replication and uses the key file for secure communication between replicas.

--replSet: Defines a Replica set parameter named rs.

healthcheck:: Ensures that pmm-client starts only after the Replica Set is initialized.

Adding Volumes

Define three separate volumes for data storage:

yaml
volumes:
 mongodb-data-101:
 mongodb-data-102:
 mongodb-data-103:

Initializing the Replica Set

Add another service to initialize the Replica Set (after mongodb-rs103):

yaml
 mongodb-rs-init:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: rs-init
 depends_on:
 - mongodb-rs101
 - mongodb-rs102
 - mongodb-rs103
 entrypoint: [
 "sh", "-c",
 "until mongosh --host mongodb-rs101 --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'print(\"waited for connection\")'; do sleep 5; done && \
 mongosh --host mongodb-rs101 --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'config={\"_id\":\"rs\",\"members\":[{\"_id\":0,\"host\":\"mongodb-rs101:27017\"},{\"_id\":1,\"host\":\"mongodb-rs102:28017\"},{\"_id\":2,\"host\":\"mongodb-rs103:29017\"}],\"settings\":{\"keyFile\":\"/etc/secrets/mongodb-keyfile\"}};rs.initiate(config);'"
 ]
 volumes:
 - ./secrets:/etc/secrets:ro

This service connects to one of the replicas and initializes the Replica Set configuration.

Updating pmm-client

Finally, modify the pmm-client service to register all three Replica Set nodes:

yaml
 pmm-client:
 image: percona/pmm-client:3
 container_name: pmm-client
 depends_on:
 pmm-server:
 condition: service_healthy
 mongodb-rs101:
 condition: service_healthy
 mongodb-rs102:
 condition: service_healthy
 mongodb-rs103:
 condition: service_healthy
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: admin
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 PMM_AGENT_SETUP: 1
 PMM_AGENT_SETUP_FORCE: 1
 PMM_AGENT_PRERUN_SCRIPT: >
 pmm-admin status --wait=10s &&
 pmm-admin add mongodb --service-name=mongodb-rs101 --username=databaseAdmin --password=password --host=mongodb-rs101 --port=27017 --query-source=profiler &&
 pmm-admin add mongodb --service-name=mongodb-rs102 --username=databaseAdmin --password=password --host=mongodb-rs102 --port=28017 --query-source=profiler &&
 pmm-admin add mongodb --service-name=mongodb-rs103 --username=databaseAdmin --password=password --host=mongodb-rs103 --port=29017 --query-source=profiler

Explanation:

depends_on: Ensures pmm-client waits until all replicas and pmm-server are initialized.

PMM_AGENT_PRERUN_SCRIPT: Adds all three replicas to PMM monitoring with the pmm-admin add mongodb command.

Launching the Configuration

Start all services with:

docker-compose up -d

Expected Output:

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 10/10
 ✔ Network pmm-mongodb-setup_default Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-101" Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-102" Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-103" Created 0.0s
 ✔ Container mongodb-rs103 Healthy 0.1s
 ✔ Container pmm-server Healthy 1.1s
 ✔ Container mongodb-rs101 Healthy 0.1s
 ✔ Container mongodb-rs102 Healthy 0.1s
 ✔ Container rs-init Started 0.1s
 ✔ Container pmm-client Started 0.0s
➜ pmm-mongodb-setup

Expected Output in Docker Desktop:

Verifying in PMM

Open PMM and explore dashboards such as the MongoDB Replica Set Summary, which displays information about your replicas and various metrics:

For example, I experimented by restarting one of the MongoDB services in Docker Desktop. The Replica Set switched the Primary replica, and when I simulated a failure by stopping a service, the monitoring dashboard reflected this event:

Connecting to MongoDB and Useful Commands

Connecting to MongoDB

There are several ways to connect to MongoDB depending on your setup and tools:

Using Docker Desktop

If you are using Docker Desktop, you can select a container and open the Exec tab. This opens a terminal within the container.

From there, you can connect to MongoDB using the mongosh shell with the following command:
sh
```
mongosh --host localhost --port 27017 -u databaseAdmin -p password --authenticationDatabase admin
```
Using Docker CLI

If you’re running Docker without Docker Desktop, you can connect to the container using the following command:
bash
```
docker exec -it  bash
```
when you connect to the container, connect to MongoDB
sh
```
mongosh --host localhost --port 27017 -u databaseAdmin -p password --authenticationDatabase admin
```
Connecting from Applications or Tools

If you are connecting through an application or a tool like MongoDB Compass, you can use connection strings tailored to your setup:

Here are the MongoDB connection strings tailored for your use cases:
1. Standalone MongoDB: Connects directly to a single MongoDB instance.
  sh
  mongodb://databaseAdmin:password@localhost:27017
2. Primary Node: Forces a direct connection to the primary node in the Replica Set using the directConnection=true option.
  sh
  mongodb://databaseAdmin:password@localhost:27017/?directConnection=true
3. Full Replica Set: Lists all Replica Set members and enables automatic failover using replicaSet=rs
  sh
  mongodb://databaseAdmin:password@localhost:27017,localhost:28017,localhost:29017/?replicaSet=rs

Useful Commands

Here are some helpful commands for managing and troubleshooting your MongoDB setup:

Check the status of the Replica Set. After connecting to MongoDB, use the following command to retrieve the status of the Replica Set:
sh
```
rs.status()
```
Check MongoDB runtime configuration. Use this command to view the configuration options for the running MongoDB instance:
sh
```
db.adminCommand({ getCmdLineOpts: 1 })
```

Conclusion

In this article, we explored deploying MongoDB using Docker and Docker Compose. We covered both a Single Instance MongoDB for simple setups and a MongoDB Replica Set for high availability, while integrating them with Percona Monitoring and Management (PMM) for monitoring.

Here are the final configurations you can download:

Single Instance MongoDB + PMM 3: Link
Replica Set MongoDB + PMM 3: Link

Thank you for reading! I look forward to your comments and questions.

Join Us Online: Stream About Percona Toolkit for MySQL!

Sveta Smirnova — Fri, 14 Mar 2025 00:00:00 UTC

Are you passionate about databases and want to stay on top of the latest advancements in MySQL and Percona Toolkit? You’re in luck! We are excited to invite you to our upcoming online stream, where we’ll delve into some exciting changes and updates.

Date: March 27, 2025
Time: 13:30 GMT / 9:30 ET
Streaming Live on: LinkedIn and YouTube

About the Event

Our featured speaker, Sveta Smirnova, Principal Support Engineering Coordinator at Percona, will share her insights on the recent updates in MySQL, focusing on the removal of offensive replication statements like START/STOP SLAVE. As the maintainer of the Percona Toolkit, Sveta had to rewrite numerous tools and libraries to accommodate these changes, resulting in significant updates to 511 files.

This event is perfect for developers, DBAs, and anyone interested in databases. Sveta, an expert in MySQL and Percona Toolkit, is also an accomplished author and a speaker at international conferences on development and databases. During the stream, she will discuss the challenges she faced while renewing legacy code, including supporting SSL and handling the deprecation of certain tools.

We invite you to join the discussion and share your experiences, questions, and insights. Engage directly with Sveta and other participants to gain a deeper understanding of the Percona Toolkit and its latest enhancements.

Discussion Topics

Introduction to Percona Toolkit:
- History and evolution of Percona Toolkit.
- Overview of the most popular tools and their uses.
Changes in MySQL:
- Overview of new and legacy syntax.
- Implications of dropped offensive replication statements.
Adapting Percona Toolkit:
- Challenges in renewing legacy code.
- Solutions implemented, including SSL support and deprecation of certain tools.
Practical Insights:
- Detailed explanation of the changes made to 511 files.
- Fine-tuning and migration strategies for Percona Toolkit users.

Why Attend?

Insightful Content: Gain valuable knowledge on the latest changes in MySQL and how they impact Percona Toolkit.
Expert Guidance: Learn directly from Sveta Smirnova, an industry expert with extensive experience in database management.
Interactive Session: Have the opportunity to ask questions live and engage directly with the speaker.

How to Join

Tune in on March 27, 2025, at 13:30 GMT / 9:30 ET, and watch the live stream on LinkedIn and YouTube. Don’t miss this chance to enhance your understanding of MySQL and Percona Toolkit while gaining practical insights from one of the best in the field.

Mark your calendars, spread the word, and get ready for an informative session! We look forward to seeing you there!

If you have any questions or need further information, feel free to reach out to us.

📅 Add to Calendar
🔗 Join the LinkedIn Stream
🔗 Join the YouTube Stream

We can’t wait to see you at the event!

Percona Monitoring and Management 3 and rootless containers

Sergey Pronin — Wed, 19 Feb 2025 00:00:00 UTC

In today’s landscape, where security breaches are a constant concern, reducing potential attack vectors is a top priority for any organization. Percona Monitoring and Management (PMM) has established itself as a reliable solution for database performance monitoring. With the release of PMM version 3, Percona has significantly strengthened its security posture, notably by introducing support for rootless container deployments. This advancement directly addresses a crucial security challenge and enhances the overall robustness and reliability of PMM.

The inherent risks associated with root privileges are well-documented. While many applications, including those containerized, have historically relied on root access, this practice presents a substantial security vulnerability. In the event of a successful exploit, an attacker gains comprehensive control over the host system. This risk is further exacerbated in environments with outdated software or complex configurations. Essentially, while the root user offers extensive capabilities, it also represents a significant potential liability that should be carefully mitigated.

In this blog post we will look at the exact differences between PMM versions 2 and 3 and how they behave.

Enforcing Pod Security Standards

The Pod Security Standards define three different policies to broadly cover the security spectrum. These policies are cumulative and range from highly-permissive to highly-restrictive.

To enforce restrictions on a specific namespace we will create the following YAML manifest:

apiVersion: v1
kind: Namespace
metadata:
 name: secure-namespace
 labels:
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/warn: restricted
 pod-security.kubernetes.io/audit: restricted

This YAML creates a namespace named secure-namespace. The pod-security.kubernetes.io/enforce: restricted label instructs Kubernetes to deny any pods in this namespace that violate the “restricted” PSS profile. The warn and audit labels are also very useful for monitoring and testing before fully enforcing the restricted policy.

Deploy PMM

We will execute a series of deployments to demonstrate the difference between PMM2 and PMM3 behavior in insecure and secure environments. All files can be found in this github repository: spron-in/blog-data/pmm3-rootless

Regular namespace

PMM2

% kubectl apply -f 01.pmm2.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 45s

I can connect to my PMM2 server with a Service that I created.

PMM3

% kubectl apply -f 02.pmm3.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 20s

I can connect to my PMM3 server with a Service that I created.

Secure namespace

Let’s try to deploy both versions of Percona Monitoring and Management servers in a secure namespace. For both we will see the following:

% kubectl apply -f 01.pmm2.yaml -n secure-namespace
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "pmm-server" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "pmm-server" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "pmm-server" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "pmm-server" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
No resources found in secure-namespace namespace.

There are no Pods created and if you describe the StatefulSet, you are going to see a similar error:

Warning FailedCreate 5s (x15 over 87s) statefulset-controller create Pod pmm-server-0 in StatefulSet pmm-server failed error: pods "pmm-server-0" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "pmm-server" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "pmm-server" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "pmm-server" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "pmm-server" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The error will be the same for PMM2 and PMM3 manifests.

Secure namespace and security contexts

We are now going to apply Pod and Container Security Contexts to both manifests.

Under spec.containers we add everything that Kubernetes suggested us:

 - name: pmm-server
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop:
 - ALL
 runAsNonRoot: true
 seccompProfile:
 type: 'RuntimeDefault'

PMM2

% kubectl -n secure-namespace apply -f 03.pmm2-secure.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 0/1 Error 2 (15s ago) 28s

Even though the PMM2 server Pod can be created now, it is failing to start. If you check the logs, you are going to see the following:

% kubectl -n secure-namespace logs pmm-server-0
...
Error: Can't drop privilege as nonroot user
For help, use /usr/local/bin/supervisord -h

PMM3

% kubectl -n secure-namespace apply -f 04.pmm3-secure.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 32s

PMM3 starts just fine.

Helm

The recommended approach to deploy PMM3 in Kubernetes is via Helm. You can find our helm charts in percona/helm-charts github repository and more in our documentation.

To deploy PMM3 in a namespace or environment with strict security (like OpenShift), you need to pass similar security context parameters. You can find how in 05.pmm3-helm.yaml values manifest.

Then the deployment will look like this:

helm repo update
helm install pmm3 percona/pmm -f 05.pmm3-helm.yaml --namespace secure-namespace

Conclusion

PMM 3’s rootless design excels where PMM 2 falters in secure Kubernetes environments. Enforcing Pod Security Standards, we saw PMM 3 deploy successfully, while PMM 2 failed, even with security contexts. This highlights PMM 3’s enhanced security, crucial for modern, hardened deployments. Using Helm simplifies secure PMM 3 deployments, ensuring robust database monitoring without compromising security.

Try out Percona Monitoring and Management version 3, 100% open source database observability solution, and learn more about its enhancements.

Tell us what you think in our forum or let us know if you are looking for commercial support.

How to Use IAM Roles for Service Accounts (IRSA) with Percona Operator for MongoDB on AWS

Natalia Marukovich — Mon, 17 Feb 2025 00:00:00 UTC

Introduction

Percona Operator for MongoDB is an open-source solution designed to streamline and automate database operations within Kubernetes. It allows users to effortlessly deploy and manage highly available, enterprise-grade MongoDB clusters. The operator simplifies both initial deployment and setup, as well as ongoing management tasks like backups, restores, scaling, and upgrades, ensuring seamless database lifecycle management.

When running database workloads on Amazon EKS (Elastic Kubernetes Service), backup and restore processes often require access to AWS services like S3 for storage. A key challenge is ensuring these operations have secure, least-privileged access to AWS resources without relying on static credentials. Properly managing these permissions is crucial to maintaining data integrity, security, and compliance in automated backup and restore workflows.

IAM Roles for Service Accounts (IRSA) is the recommended approach to solve this problem. IRSA allows Kubernetes pods to securely assume IAM roles, eliminating the need for hardcoded credentials, long-lived AWS keys, or excessive permissions. Instead, it leverages OpenID Connect (OIDC) authentication, ensuring that only the right workloads get access to AWS services.
By implementing IRSA, you enhance the security posture of your Kubernetes workloads while simplifying IAM management. In this article, we’ll walk through how IRSA works, why it’s beneficial, and how to configure it properly for the Percona Operator for MongoDB in EKS clusters.

IRSA Installation and Configuration for Percona Operator for MongoDB

IRSA requires an OpenID Connect (OIDC) provider associated with your EKS cluster.
So, you should create an OIDC provider for your EKS cluster.

Creating an OIDC provider for your EKS cluster involves several steps. This setup allows your EKS cluster to use IAM roles for service accounts, which makes it possible to grant fine-grained IAM permissions to pods.

shell
# Check if OIDC is already set up:

aws eks describe-cluster --name  --query "cluster.identity.oidc.issuer" --output text

https://oidc.eks.eu-west-3.amazonaws.com/id/7AA1C67941083331A80382E464EB2F1F

# If it is not already set up, create an OIDC provider:

eksctl utils associate-iam-oidc-provider --region  --cluster  --approve

Here oidc-id is 7AA1C67941083331A80382E464EB2F1F. We will use it under role creation.

Create an IAM Policy to access s3 buckets. Substitute with your correct bucket name:

shell
# Define the required permissions in an IAM policy JSON file: 


cat s3-bucket-policy.json

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:*"
 ],
 "Resource": [
 "arn:aws:s3:::",
 "arn:aws:s3:::/*"
 ]
 }
 ]
}

# Create the IAM policy:

aws iam create-policy --policy-name  --policy-document file://s3-bucket-policy.json

Create an IAM Role and Attach the Policy:

shell
# Role example. Replace  with account id and  with cluster’s OIDC ID


cat role-trust-policy.json



{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/"
 },
 "Action": "sts:AssumeRoleWithWebIdentity",
 "Condition": {
 "StringEquals": {
 "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com"
 }
 }
 }
 ]
}

# Create role:

aws iam create-role --role-name  --assume-role-policy-document file://role-trust-policy.json --description "Allow access to s3 bucket"

Attach the policy to the role.

shell
# Please update ,  and  with the corresponding values.

aws iam attach-role-policy --role-name  --policy-arn arn:aws:iam:::policy/

Install the operator and deploy Percona Server for MongoDB in your EKS cluster (skip this step if you already have the operator and the database cluster installed).
To ensure proper functionality, we need to annotate both the operator service account (default: percona-server-mongodb-operator) and the cluster service account (default: default).

🔴 Warning: The cluster and operator won’t restart automatically; therefore, a manual restart is necessary to apply the changes.

shell
# Get service accounts:

$ kubectl get sa -n 
NAME SECRETS AGE
default 0 25m
percona-server-mongodb-operator 0 25m


# Get role_arn:

aws iam get-role --role-name  --query "Role.Arn" --output text

# Annotate service account. Please update role_arn with appropriate value.

kubectl annotate serviceaccount default \
 eks.amazonaws.com/role-arn=""

kubectl annotate serviceaccount percona-server-mongodb-operator \
 eks.amazonaws.com/role-arn=""

To verify that the settings have been applied, inspect service accounts and the environment variables in both the operator and replica set (RS/Config) pods. The variable AWS_ROLE_ARN should be properly set.

shell
# Check annotation in service account

$ kubectl get sa -n  percona-server-mongodb-operator -o yaml

$ kubectl get sa -n  default -o yaml

# Check the variable inside container

$ kubectl exec -ti  -n  bash

bash-5.1$ printenv | grep 'AWS_ROLE_ARN'
AWS_ROLE_ARN=arn:aws:iam::1111111111111:role/some-name-psmdb-access-s3-bucket

$ kubectl exec -ti  -n  bash

[mongodb@some-name-rs0-0 db]$ printenv | grep 'AWS_ROLE_ARN'
AWS_ROLE_ARN=arn:aws:iam::1111111111111:role/some-name-psmdb-access-s3-bucket

Configure the backup/restore settings as usual, but do not provide s3.credentialsSecret for the storage in deploy/cr.yaml. For detailed instructions please refer to Configure storage for backups.

shell
# backup section in cr.yaml example 
 storages:
 aws-s3:
 type: s3
 s3:
 region: 
 bucket: 

Conclusion

Using IAM Roles for Service Accounts (IRSA) in an Amazon EKS cluster is a best practice when running database operators in Kubernetes. By integrating IRSA, database operators—such as the Percona Server for MongoDB Operator—can securely access AWS services like S3 for backups without relying on static credentials.

IRSA enhances security by enforcing the principle of least privilege, ensuring that database operators in EKS have access only to the specific AWS resources they require. This approach reduces the risk of unauthorized access while also improving manageability by eliminating the need to store and rotate AWS credentials within Kubernetes secrets. By adopting IRSA in Percona Server for MongoDB Operator , organizations can create a more secure, scalable, and automated environment for managing MongoDB databases.

Join Percona for Google Summer of Code 2025 – Explore, Innovate, and Contribute!

Radoslaw Szulgo — Wed, 05 Feb 2025 00:00:00 UTC

Are you passionate about open-source databases, AI/ML, and security? Do you want to work on real-world projects that impact thousands of developers and enterprises worldwide? Percona is excited to invite students to participate in Google Summer of Code 2025 (GSoC) and help advance our cutting-edge open-source database solutions!

Why Contribute to Percona?

At Percona, we believe that open world is a better world! GSoC is an excellent opportunity to work with seasoned developers, gain hands-on experience, and contribute to powerful database tools used by businesses globally.

For 2025, we’re especially interested in projects that focus on AI/ML and security—two critical areas shaping the future of databases. Whether you’re passionate about automating database performance insights with AI or hardening security for mission-critical data, we have exciting challenges for you!

Percona mentors are going to help you realize your ideas or one of the ideas below. We invite you to contribute to products and projects such as:

Percona XtraDB Cluster
Percona Server for MySQL
Percona Distribution for PostgreSQL
Percona Operator for PostgreSQL
Percona Server for MongoDB
Percona Backup for MongoDB
Percona Operator for MongoDB
Percona Everest
Percona Monitoring and Management (PMM)
pg_tde: Transparent Database Encryption for PostgreSQL
As well as CI/CD related projects with the Percona Build Engineering.

Project Ideas for GSoC 2025

Below are some suggested project ideas categorized by Percona software:

Percona Distribution for PostgreSQL

Snapshot-based PostgreSQL backups

Database users are often very familiar with their storage provider’s storage snapshot capabilities. These snapshots are very handy and performant to use, hence their popularity among users. Backups for other databases (e.g., MongoDB) are often configured via this capability as it provides many performance benefits for large-scale data, especially on Cloud deployments. Having such technology supported across the backup solutions for multiple databases makes it possible to leverage this effectively for Percona Everest via the Percona Operators.

Comments from Crunchy on what needs to be glued together to get snapshots and pgBackRest working together better. Additionally, Timescale on how they use snapshots and pgBackRest together in their hosted managed service.

Deliverables: Have an API available to Percona Distribution for PostgreSQL to effectively use storage snapshots to create backups and restore from storage snapshot-based backups. Preferably, have it added to/complementary to the currently recommended solution of pgBackRest.

Have Percona Operator for PostgreSQL expose the storage snapshot-based backup/restore so that Percona Everest can leverage it.

Required/preferred skills: C++, PostgreSQL, Kubernetes Duration: 350 hours **Difficulty level: **Hard Mentors: @Andrew_Pogrebnoi, @Jan_Wieremjewicz Relevant repository and resources:

pgBackRest to Barman close gap improvements

PostgreSQL has two main backup tools: Barman and pgBackRest. Both are powerful backup and restore tools, each with its own strengths. pgBackRest is generally considered more advanced in terms of parallelism, performance, and flexibility. Barman does offer some advantages in certain areas. While pgBackRest is maintained by Community, Barman is a tool mainly maintained by one company and is less popular. Barman does have UX improvements over pgBackRest, especially for non-expert users:

direct WAL archiving with PostgreSQL’s built-in archive_command,
simpler backup and recovery process, especially for standby creation,
clearer logging and monitoring for backup integrity,
simpler configuration in small to medium deployments,
better native tools for cloud backups

It would be beneficial for Percona, which uses pgBackRest in the Percona Distribution for PostgreSQL, to have the backup tool close any functionality gaps in Barman. Percona customers sometimes use Barman and expect Percona to support it. Having a way to migrate off Barman to pgBackRest, reducing any potential friction for the users, would be beneficial.

Deliverables: Provide a close-gap set of improvements based on the list available in the description

Required/preferred skills: C++, PostgreSQL Duration: 350 hours Difficulty level: Hard Mentors: @Andrew_Pogrebnoi, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Tool to investigate PostgreSQL locks for dummies

Currently, there is no tool that allows users with low experience to detect and understand all types of locks on their PG database, which may lead to many issues in deployments not managed by expert PostgreSQL users. As described in the blog posts below, understanding how locks work is difficult:

Deliverables:

Detect ddl with mixed strong locks and others. i.e. allow to review locked PIDs as pg_locks will not work
Present all locks in a GUI
(Streched) have the GUI integrated in PMM

Required/preferred skills: C++, PostgreSQL Duration: 350 hours Difficulty level: Medium **Mentors: **Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Session continuity for PgBouncer for the zero downtime upgrades

Percona is looking to introduce zero downtime upgrades capability to the Percona Operator and later on to Percona Everest. The assumption is to base on pgBouncer and our HA solution utilizing the replica with the new database and a switch from the previous version to the new version.

Such a solution provides a zero downtime upgrade capability and a Rollback capability. To provide true zero downtime major upgrades for the current Percona Distribution for PostgreSQL, there needs to be an improvement that takes over the switching of sessions between the databases: the previous version and the new version

In the future, this tool should also make it possible to zero downtime and migrate to Everest.

Deliverables: Extend pgBouncer to ensure that the sessions can be switched between the databases without downtime for the users but only a potential performance drop

Required/preferred skills: C++, Kubernetes Duration: 175 hours Difficulty level: Hard **Mentors: **Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Percona Software for MongoDB

Interactive Shell Installer for Percona Software for MongoDB

This project aims to develop an interactive shell-based installer for Percona Server for MongoDB and Percona Backup for MongoDB. The installer will simplify the installation, configuration, and initial setup process, making it easy for users to deploy these open-source enterprise solutions efficiently. The primary goal is to enhance the user experience by reducing manual setup steps and ensuring proper configuration through guided prompts and automation.

Deliverables:

A command-line-based interactive installer script.
Automated dependency checks and installation.
Interactive prompts for configuration choices (e.g., authentication, replication, sharding).
Seamless installation of both Percona Server for MongoDB and Percona Backup for MongoDB.
Integration with package managers for major Linux distributions (Debian, Ubuntu, RHEL, CentOS).
Logging and validation mechanisms to ensure correct setup.
Documentation and user guide for the installer.

Required/preferred skills: C++ or Go

Duration: 350 hours

Difficulty level: Medium

Mentors: @radoslaw.szulgo

Relevant repository and resources: https://github.com/percona/percona-backup-mongodb

Percona Backup for MongoDB backup speed throttling

On large scale deployments, backups may significantly impact network performance - speficially network bandwidth may be heavily utilized, if the backup storage is fast, causing performance degradation of the database itself. Database reliability engineers would like to reduce the network load by slowing down physical backups with Percona Backup for MongoDB (PBM) configuration. The scope of the project is to implement a network bandwidth rate limiter and perform load testing showing the impact or rate limiting on backup time.

Deliverables: The expected outcome of this project is insurance that backup will not degrade network bandwidth impacting the database. As a result a participant needs to provide proposed code changes in a form of fork of PBM and a create a report with load test results.

Required/preferred skills: Go
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/percona-server-mongodb

Percona Backup for MongoDB Golang SDK

The project’s purpose is to extend capabilities and reduce maintenance in monitoring, managing, and automating backup and restores of MongoDB from Percona Monitoring and Management (PMM) tool. In the scope of the project there’s a migration from Percona Backup for MongoDB CLI to a dedicated PBM golang client library in PMM. The client library (aka SDK) has to be implemented and map all current CLI operations to Go API functions. As a stretch goal, the project can be extended into implementing a backup progress reporting using the created SDK.

Deliverables: The expected outcome of the project is the reduced maintenance of backup integration in PMM project and enabling extensibility of backup management. As a result of the project a new open-source SDK should be create

Required/preferred skills: Go
Duration: 350 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/pmm

CEPH Storage support in Percona Backup for MongoDB

Ceph is an open source Software Defined Storage(SDS) software that is massively scalable and reliable. It’s one of the most popular storage technologies in Kubernetes and Openshift. The project aims to enable users to store their Percona Backups for MongoDB data on a Ceph storage which would be very convenient as they wouldn’t need to manage other additional storages. The scope of the project includes building a workspace setup on Kubernetes and Percona Operator for MongoDB, research on current challanges using Ceph storage, and implement necessary changes to make it work in a performant way. At the end document the solution.

Deliverables: The project’s deliverables are technical documentation, Percona Operator for MongoDB changes, and instructions on setting up an environment with Ceph storage.

Required/preferred skills: Go, Kubernetes
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/percona-server-mongodb

BoostFS storage support in Percona Backup for MongoDB

Dell Data Domain Boost File System (BoostFS) provides a general file system interface to the DD Boost library, allowing standard backup applications to take advantage of DD Boost features. In this project we’d like to extend our open-source Percona Backup for MongoDB to leverage that storage technology to reduce backup and restore time - and the same time help users to reduce their Recovery Time Objective (RTO) and Recovery Point Objective (RPO). In the scope of the project there’s a preparation of the workspace setup with Percona Server for MongoDB, Percona Backup for MongoDB and mounted BoostFS disk volumes on Google Cloud Platform and documenting the architecture of the environment. Additionally, the project includes a research on how PBM works with that storage and implementing necessary changes to PBM to make it work. Finally, a simple benchmark should be performed that proves the performance boost.

Deliverables: It is expected that project delivers an architecture diagram of the testing environment in Google Cloud Platform, implementation of required changes to support BoostFS in PBM, and report incl. performance benchmark results and comparison to other storage systems.

OpenStack Swift storage support in Percona Backup for MongoDB

The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data with a simple API. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound. Swift is very convenient to use as a backup storage for MongoDB workloads running on OpenStack platform. The scope of the project includes building a workspace environment on Google Cloud Platform with OpenStack clusters and running there Percona Server for MongoDB. Then implementing required changes in Percona Backup for MongoDB to support Swift storage. Finally, performing benchmark tests and comparison to GCP native storages.

Deliverables: It is expected that project delivers an architecture diagram of the testing environment in Google Cloud Platform, implementation of required changes to support OpenStack Swift in PBM, and report incl. performance benchmark results and comparison to other storage systems.

Required/preferred skills: Go, GCP, OpenStack
Duration: 175 hours
Difficulty level: Medium
Mentors: @radoslaw.szulgo
Relevant repositories and resources:

Percona Server for MySQL and Percona XtraDB Cluster

Automating Code Merges with AI

The regular and manual merge from Oracle’s GitHub repository process is time-consuming, complex, and prone to errors, particularly due to merge conflicts. Careful attention is required to avoid introducing regressions into Percona’s open-source products. While this project is specific to Percona’s needs, it addresses a common challenge in open-source software development, as many projects rely on upstream repositories for their code. Therefore, the solution can be generalized and could benefit other open-source projects with similar code integration needs.

This GSoC project aims to develop an intelligent system using Artificial Intelligence to automate the MySQL fork merge process. Percona has been performing these merges for 18 years, accumulating a wealth of historical data (code changes, merge resolutions, conflict histories, test results) that can be leveraged to train an AI model.

The core objective is to create a tool that can:

Analyze upstream changes: Process and understand the changes introduced by Oracle in their MySQL repository.
Identify merge conflicts: Identify conflicts between upstream changes and Percona’s modifications.
Suggest merge resolutions: Propose solutions for resolving identified conflicts, drawing on patterns from historical merge data.
Automate merges: Automatically apply upstream changes with the suggested merge resolutions.
Learn and adapt: Continuously improve its performance and accuracy by learning from new merge data and feedback.

Deliverables:

Reduced merge time and effort: Automating the merge process will free up developer time for other critical tasks.
Improved merge accuracy: AI can potentially identify subtle conflicts that might be missed by manual review.
Faster release cycles: Streamlining the merge process will enable quicker releases of updated Percona products.
Open-source contribution: The resulting tool will be open-sourced, benefiting other projects that maintain forks of MySQL or similar databases. This problem is not unique to Percona; other open-source projects facing similar merging challenges can utilize this solution.

As a result of this project, you’re expected to deliver:

A working prototype of the AI-powered merge tool.
Well-documented code and training data.
Comprehensive test suite and evaluation results.
A report detailing the project’s methodology, findings, and future directions.

Required/preferred skills: Python, Machine Learning libraries and frameworks (e.g., TensorFlow, PyTorch, scikit-learn), C++, Git, database systems Duration: 350 hours Difficulty level: Hard Mentors: Julia Vural, Oleksiy Lukin Relevant repositories and resources:

Percona Everest

Easier Troubleshooting on database clusters in Percona Everest

The main goal is to provide tools to Percona Everest users to troubleshoot database clusters. This project will require the implementation of log collection, rotation, UI, and possibly an AI helper to analyze those logs. If users have a centralized log collection implemented, this tool needs to be able to integrate with it.

Deliverables:

Full user flow to support database cluster troubleshooting process (UI, backend, API, integrations).

Log Collection & Rotation System:
- Implement a mechanism to collect logs from Percona Everest-managed database clusters.
- Ensure efficient log rotation to manage storage and performance impact.
- Enable compatibility with external log aggregation tools (e.g., Elasticsearch, Grafana Loki, or OpenTelemetry)
User Interface for Log Access:
- Develop a UI within Percona Everest to allow users to view and analyze logs.
- Include search, filtering, and visualization options for better troubleshooting.
AI-Powered Log Analysis (Stretched scope)
- Explore AI-driven log analysis to provide users with insights, anomaly detection, and recommendations.
- Implement basic AI-assisted troubleshooting if feasible within the project timeline.
Documentation & Testing:
- Deliver user and developer documentation covering installation, usage, and troubleshooting.
- Include test cases and automation scripts to ensure system reliability.

Required/preferred skills: Kubernetes, Go, CI/CD Duration: 350 hours **Difficulty level: **Medium Mentors: @Diogo_Recharte, @Mayank_Shah Relevant repository and resources: https://github.com/percona/everest

Percona Everest RBAC policies management UI

Create a user interface to create and manage role-based access control policies

Deliverables:

Role-Based Access Control (RBAC) UI:
- Develop a user-friendly interface in Percona Everest to create, update, and manage RBAC policies.
- Implement role assignment and permission configuration for database clusters.
Documentation & Testing:
- Deliver comprehensive user and developer documentation.
- Include test cases and automation scripts to ensure reliability.

Required/preferred skills: Front-end, CI/CD tools Duration: 90 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Peter Szczepaniak Relevant repository and resources: https://github.com/percona/everest

Context sensitive help

The Percona Everest documentation contains valuable information, hints, and tips, but we lack a way to present relevant information to our users. This project aims to work with the UX and Docs teams to solve this problem.

Deliverables:

Implement a mechanism to display relevant documentation, hints, and tips based on the user’s current action or screen within Percona Everest.
Ensure seamless integration with the existing UI for a non-intrusive experience.
Enable contextual tooltips, pop-ups, or side panels that present relevant documentation without requiring users to leave the interface.
Support links to full documentation pages when needed.
Optionally, explore AI-driven suggestions based on user behavior and past queries.
Allow users to control the level of help they receive (e.g., enable/disable tips, adjust verbosity).
Provide user and developer documentation on how the system works and how to extend it.
Ensure thorough testing to validate the accuracy and relevance of displayed help content.

Required/preferred skills: Front-end, CI/CD tools Duration: 175 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Peter Szczepaniak Relevant repository and resources: https://github.com/percona/everest

Backups and restore timeline visualization

Databases are usually long-living services, and investigating issues with them is easier when you can see events like backups and restores of this service on a timeline.

Deliverables:

Develop a visual timeline within Percona Everest to display backup and restore events for database clusters.
Ensure the timeline is intuitive, zoomable, and supports different time ranges (e.g., last 24 hours, 7 days, custom range).
Retrieve and display backup and restore events from Percona Everest’s database and logs.
Include metadata such as timestamps, duration, status (success, failure), and associated users or processes.
Allow users to filter events by type (full backup, incremental backup, restore, etc.).
Enable color-coding or icons to differentiate event types at a glance.
Deliver comprehensive user and developer documentation.
Ensure automated tests for data accuracy, UI performance, and usability.

Refactor test automation using page object model

Our project currently has a functional end-to-end (E2E) UI test suite that ensures the stability and correctness of our application. However, the test suite does not follow the Page Object Model (POM) design pattern, making it harder to maintain, scale, and debug.

Deliverables:

Restructure existing test automation to follow the Page Object Model (POM) design pattern.
Ensure better separation of test logic and UI elements for improved maintainability.
Implement modular and reusable page object classes for different UI components and workflows.
Standardize naming conventions and best practices for test scripts.
Improve error handling and logging to make test failures easier to diagnose.
Ensure the refactored test suite runs efficiently in CI/CD pipelines.
Validate test performance improvements and maintain test coverage.

Required/preferred skills: Playwright, Typescript, Kubernetes Duration: 175 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Tomislav_Plavcic, Edith Puclla Relevant repository and resources: https://github.com/percona/everest

Percona Monitoring and Management (PMM)

Queryable backup and restore of Percona Server for MongoDB

The project aims to equip MongoDB Database Administrators with Percona Monitoring and Management (PMM) extension that allows to query data directly from a backup. It solves their pain when they want to inspect just a single document in a collection from a few Terabytes size backup. The time associated with downloading the snapshot, decompressing it, getting it running in a local MongoDB node, and finally running the query would be significant. Not only that, but there are obvious nontrivial costs — both monetary and operational — associated with having to quickly spin up new environments. In scope of the project there is a backend Go application server implementation to run on-demand ephemeral mongodb instance, load data from backup, and enable user to run a DB query from a UI interface.

Deliverables: It’s expected to deliver source code changes to PMM in form of PMM fork that extends PMM functionality for queryable MongoDB backup. Specifically, it is expected to prepare a solution design, implementation, unit and/or integration tests based on a Docker environment, and documentation.

Required/preferred skills: Go, MongoDB
Duration: 350 hours
Difficulty level: Medium
Mentors: @radoslaw.szulgo
Relevant repository:
- https://github.com/percona/pmm
- https://github.com/percona/percona-backup-mongodb

Design Engineering for PMM

Percona Monitoring and Management (PMM) is a long-standing open-source software, but its age also comes with some UX and UI debt. Facing new goals to help innovate PMM, the team is excited to look forward to starting “renovating the house” and swapping the GUI with a more modern one built in-house. We are looking for experts in design engineering to help with:

Making/refining/cataloging UI components that we will need for QA and production;
Creating functional prototypes ad hoc from written ideas or designs;
Convert old PMM pages like for like into new PMM pages (with new UI).

Deliverables:

Contributing with ideas to help make the code library more easy to contribute to;
Contributing to the code library with at least one new component;
Create at least one code prototype for one of the team’s ongoing ideas;
Convert at least one existing PMM functionality page UI into the new UI.

Required/preferred skills: CI/CD, Git, Storybook, React, MUI, Figma

Duration: 350 hours

Difficulty level: Medium

Mentor: @pedro.fernandes

PMM UI for PostgreSQL backups - create, restore, check, monitor

Backup management without UI is not an easy task for the users. Having a tool that could be a tool of choice for backup and restore management could provide a unification layer for multiple backup/restore tools as well as provide a very important value that’s often overlooked: backup monitoring.

As it turns out, many DBAs are often worried about the state of their backups and make it a daily routine task to check how the backups they have configured are. As the environments scale, this task becomes increasingly tiresome. Also, having to check whether the backups they create are effective makes it another routine task that needs either extra automation scripting to run a restore or a manual task.

Deliverables:

Extend PMM UI to add existing backups so that they can be monitored
Extend PMM UI to create backups
Create a tool to automate the backup testing (check whether the backups created are usable)
Extend PMM to monitor and alert on backup irregularities
Integrate the backup management with external schedulers like cron.

Required/preferred skills: C++, Go, PostgreSQL Duration: 350 hours Difficulty level: Hard Mentors: Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

LLM-Powered Test Scenario Generation for Open Source Contributions

Open-source projects thrive on community contributions, but ensuring that each pull request (PR) has adequate test coverage is a major challenge. Many PRs introduce changes without proper regression tests, leading to bugs and unstable releases.

This project aims to build an LLM-powered tool that analyzes code changes in GitHub PRs and automatically generates relevant test scenarios. Using an Open Source LLM (e.g., DeepSeek, Mistral, LLaMA), the system will identify impact areas, suggest missing test cases, and recommend regression tests based on past commits. The goal is to integrate this into GitHub workflows, enabling maintainers to quickly assess PR test coverage and guide contributors in writing better tests.

Deliverables:

The student will work on developing a system with the following features:

PR Analysis & Impact Assessment
Extract and analyze code diffs in pull requests.
Identify affected functions, dependencies, and modules.
Predict impact areas using a dependency graph.

Test Scenario Generation using LLMs

Use an Open Source LLM (DeepSeek, Mistral, etc.) to generate test cases.
Recommend unit tests, integration tests, and regression scenarios.
Compare new tests with existing ones to detect gaps in coverage.

GitHub Bot for Automated Suggestions

Implement a bot that comments on PRs with test recommendations.
Provide interactive feedback to contributors and maintainers.
Integrate with GitHub Actions for CI/CD automation.

Regression Test Identification

Identify existing test cases that need to be re-run.
Suggest additional tests based on historical PRs and past bug reports.

Evaluation Metrics & Benchmarking

Measure effectiveness by tracking missed bugs before/after integration.
Collect feedback from maintainers and contributors.

Future Scope:

Extend beyond GitHub to GitLab, Bitbucket, and other version control systems.
Support additional test types, such as security and performance tests.
Implement self-learning mechanisms to improve accuracy over time.

Required/preferred skills: Strong programming skills in Python or JavaScript, Experience with GitHub APIs & Pull Request Workflows, Understanding of Machine Learning / LLMs (DeepSeek, Mistral, LLaMA, etc.), Familiarity with Software Testing & QA Automation, Experience with CI/CD Pipelines & GitHub Actions (Bonus). Duration: 350 hours Difficulty level: Hard Mentor: Peter Sirotnak, @vasyl.yurkovych

Percona Build Engineering

SBOMs for Percona database software - MySQL, PostgreSQL, and MongoDB

A “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. An SBOM is a nested inventory, a list of ingredients that comprise software components. The project aims to adapt Percona’s build pipelines to generate SBOMs for Percona Software for MySQL, PostgreSQL, and MongoDB. This will enable organizations using Percona software to be more secure and avoid software supply chain vulnerabilities that were very harmful in late 2020 with the discovery of the Solar Winds cyberattack or later with the Log4j security flaw.

Deliverables: At the end of the project, a running staging pipeline in Jenkins and Trivy should produce complete SBOMs for Percona Server for MySQL, PostgreSQL, and MongoDB, Percona Backup for MongoDB, Percona Xtra Backup for MySQL. SBOMs are uploaded automatically to the Percona repository and are downloadable publicly. Additionally, technical documentation on how the process works is expected to be created.

Required/preferred skills: Jenkins, Trivy
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:

Evolving CI/CD: Automating Build, Test, and Release for Robust Software Delivery

Continuous Integration and Continuous Deployment (CI/CD) pipelines are the backbone of modern software development, ensuring rapid, reliable, and repeatable delivery. However, many pipelines still operate in fragmented stages, where builds and tests are automated, but releases remain a manual or semi-automated process.

This project aims to transform our CI/CD pipelines into a true end-to-end automated system, seamlessly integrating build, test, and release stages. By implementing best practices in CI/CD automation, we will ensure that only thoroughly tested software progresses to release, minimizing human intervention and reducing the risk of deployment failures.

Deliverables: The successful completion of this project will result in a fully automated and robust CI/CD pipeline that seamlessly integrates build, test, and release processes. The key outcomes will include:

Fully Automated CI/CD Pipeline

A redesigned pipeline where builds, testing, and releases are interconnected and automated. Code changes will automatically trigger builds, run tests, and, if successful, deploy releases without manual intervention. Comprehensive Test Integration

The pipeline will incorporate unit tests, integration tests, security scans, and other quality assurance mechanisms. Ensuring that faulty builds do not reach production by enforcing test-driven deployment. Automated Release Process

A mechanism that automatically releases software only if all tests pass. Versioning, tagging, and artifact management will be streamlined. The release process will be documented and configurable for different environments (e.g., staging, production). Infrastructure as Code (IaC) & Deployment Automation

Documentation & Guides

Clear technical documentation detailing the new pipeline’s workflow and configuration. A step-by-step guide for developers and DevOps engineers on how to use and extend the pipeline.

Required/preferred skills: CI/CDl like Jenkins, GitHub Actions, GitLab CI, or similar; Docker; Testing frameworks and automated deployment strategies; infrastructure as code (IaC) and cloud environments is a plus.
Duration: 350 hours
Difficulty level: Medium
Mentors: @Evgeniy_Patlan , @Vadim_Yalovets
Relevant repository: https://github.com/Percona-Lab/jenkins-pipelines

Build Automation for Open-Source Databases

Building and maintaining multiple database forks—such as MySQL, MongoDB, and PostgreSQL—often involves redundant build scripts, leading to inefficiencies, inconsistencies, and maintenance overhead. Currently, each database has its own set of build scripts despite sharing many common steps.

This project aims to develop a modular, extensible build system that allows for streamlined compilation and packaging of different database forks. The system will provide a flexible framework where users can select required modules, specify target OS distributions, and automate the build process with minimal configuration.

By implementing a plugin-based architecture, this modular builder will simplify cross-database maintenance, reduce duplication, and improve consistency across different builds.

Deliverables:

Modular Build Framework – A reusable, pluggable system that dynamically selects required modules for MySQL, MongoDB, and PostgreSQL builds.
Multi-OS Support – Automated builds for multiple Linux distributions (Debian, Ubuntu, CentOS, RHEL) with configurable OS selection.
Automated Package Creation – DEB and RPM package generation with standardized versioning and tagging.
Configurable & Scalable Builds – Easy customization of build parameters, allowing extension to new database forks or patches.
CI/CD Integration – Optional support for Jenkins, GitHub Actions, or GitLab CI to enable fully automated builds.
Comprehensive Documentation – User and developer guides with example configurations for quick adoption and extension.

Required/preferred skills: Bash/Python, CMake, Makefiles, Autotools, Linux and packaging (DEB/RPM), dependency management, CD/CD tools are a plus

Duration: 175 hours

Difficulty level: Medium

Mentors: @Evgeniy_Patlan , @Vadim_Yalovets

Relevant repositories:

More ideas are coming soon!

Suggest your ideas in the comments of the post or on the forum.

GSoC isn’t just about working on predefined ideas—it’s about innovation! If you have a project idea that aligns with Percona software, AI/ML, security, or database performance, submit your proposal, and our mentors will be happy to discuss it with you.

Do you have questions? Visit our Community Forum or join our chat channels to connect with potential mentors.

Ready to get started? See our Google Summer of Code 2025: Contribution guide.

See you in GSoC 2025!

Percona Operator for MongoDB 1.19: Remote Backups, Auto-Generated Passwords, and More!

Sergey Pronin — Fri, 31 Jan 2025 00:00:00 UTC

The latest release of the Percona Operator for MongoDB, version 1.19, is here. It brings a suite of enhancements designed to streamline your MongoDB deployments on Kubernetes. This release introduces a technical preview of remote file server backups, simplifies user management with auto-generated passwords, supports Percona Server for MongoDB 8.0, and includes numerous other improvements and bug fixes. Let’s dive into the details of what 1.19 has to offer.

Remote Backups with Network File System (Technical Preview)

Backing up your MongoDB data is crucial, and Percona Operator for MongoDB 1.19 introduces a powerful new option for backup storage: the filesystem type. This feature, currently in technical preview, allows you to leverage a remote file server, mounted locally as a sidecar volume, for your backups. This is particularly useful in environments with network restrictions that prevent the use of S3-compatible storage or for organizations using non-standard storage solutions that support the Network File System (NFS) protocol.

Setting Up Remote Backups

To use this new capability, you’ll need to add your remote storage as a sidecar volume within the replsets section of your Custom Resource (and configsvrReplSet for sharded clusters). Here’s how:

replsets:
 ...
 sidecarVolumes:
 - name: backup-nfs-vol
 nfs:
 server: "nfs-service.storage.svc.cluster.local"
 path: "/psmdb-my-cluster-name-rs0"
 ...

Then, configure the mount point and sidecar volume name in the backup.volumeMounts section:

YAML:

backup:
 ...
 volumeMounts:
 - mountPath: /mnt/nfs/
 name: backup-nfs-vol
 ...

Finally, set up a filesystem type storage in the backup.storages section, pointing it to the mount point:

YAML:

backup:
 enabled: true
 ...
 storages:
 backup-nfs:
 type: filesystem
 filesystem:
 path: /mnt/nfs/

See more in our documentation about this storage type.

Simplified User Management with Auto-Generated Passwords

Managing user credentials just got easier. Percona Operator for MongoDB 1.19 enhances declarative management of custom MongoDB users by adding the ability to generate passwords automatically. Now, when defining a new user in your deploy/cr.yaml file, you can omit the reference to an existing Secret containing the password, and the Operator will handle the generation for you:

YAML:

...
users:
 - name: my-user
 db: admin
 roles:
 - name: clusterAdmin
 db: admin
 - name: userAdminAnyDatabase
 db: admin

The Operator will create a Secret to store the generated password securely. It is important to note that the Secret will be created after the cluster is in the Ready state.

Get the user credentials: Find the Secret resource named -custom-user-secret Get the user password with this one-liner:

kubectl get secret my-cluster-name-custom-user-secret -o jsonpath='{.data.my-user}' | base64 -d

You can find more details on this automatically created Secret in our documentation.

Percona Server for MongoDB 8.0 Support

Staying up-to-date with the latest MongoDB versions is essential for performance and security. Percona Operator for MongoDB 1.19 now officially supports Percona Server for MongoDB 8.0, in addition to 6.0 and 7.0. This means you can leverage the latest features and improvements from MongoDB 8.0, combined with the enterprise-grade enhancements and open-source commitment of Percona Server for MongoDB.

Check out this blog post to learn more about the features in MongoDB 8.0.

Streamlined AWS S3 Access with IAM Roles for Service Accounts (IRSA)

Percona Operator for MongoDB 1.19 adds support for IAM Roles for Service Accounts (IRSA), simplifying secure access to AWS S3 for backups on Amazon EKS. IRSA lets you grant granular S3 permissions to specific Pods via their associated Kubernetes service accounts. This approach ensures that only the Pods that require S3 access receive it, adhering to the principle of least privilege. Furthermore, each Pod can only access credentials linked to its service account, providing strong credential isolation. For enhanced security, all S3 access is tracked through AWS CloudTrail, enabling comprehensive auditability. All of this happens without the need to manually manage and distribute AWS credentials.

Configuration Steps

Create an IAM Role: Define an IAM role with S3 access permissions. See AWS documentation.
Identify Service Accounts: The Operator uses percona-server-mongodb-operator and your cluster uses default (customizable in deploy/cr.yaml).

Annotate Service Accounts: Link the IAM role to both service accounts:

$ kubectl -n  annotate serviceaccount default eks.amazonaws.com/role-arn:  --overwrite
$ kubectl -n  annotate serviceaccount percona-server-mongodb-operator eks.amazonaws.com/role-arn:  --overwrite

Configure S3 Storage: Set up S3 storage in deploy/cr.yaml without s3.credentialsSecret. The Operator will use IRSA.

Important: IRSA credentials take precedence over IAM instance profiles, and S3 credentials in a Secret override both.

IRSA streamlines S3 access, enhancing security and manageability for your MongoDB backups on EKS. Learn more in our documentation.

Conclusion

Percona Operator for MongoDB 1.19 delivers a significant step forward in simplifying and automating the management of your MongoDB clusters on Kubernetes. With features like remote backups, auto-generated passwords, and support for Percona Server for MongoDB 8.0, this release empowers you to deploy, manage, and scale your databases with greater ease and efficiency.

We encourage you to explore the full release notes and try out the new features. As always, your feedback is invaluable to us. Please share your thoughts and contribute to the project on our GitHub repository or our Community Forum.

Percona Monitoring and Management 3.0.0-GA

Ondrej Patocka — Wed, 29 Jan 2025 00:00:00 UTC

We’re excited to announce the release of Percona Monitoring and Management (PMM) 3.0.0 GA.

The Percona Monitoring and Management (PMM) 3.0.0 release delivers major security and stability enhancements. Notable security improvements include rootless deployments and encryption of sensitive data, along with improved API authentication using Grafana service accounts. Deployment options have expanded with official ARM support and the ability to use Podman for rootless deployments, providing flexibility and better security. Additionally, the introduction of containerized architecture has increased stability, and a streamlined upgrade process ensures reliability and ease of maintenance.

User experience has been significantly improved with more flexible monitoring configurations and UI-based upgrades for Podman installations. This release also includes new features such as monitoring for MongoDB 8.0 and integration with Watchtower for automated container updates. These enhancements aim to provide users with a more secure, stable, and user-friendly monitoring and management experience.

Release notes

To see the full list of changes, check out the 3.0.0 GA Release Notes

Percona Monitoring and Management (PMM) 3.0.0 Release Notes:

Security Enhancements:
- Implementation of rootless deployments to enhance security.
- Encryption of sensitive data to ensure information confidentiality.
- Improved API authentication with Grafana service accounts, increasing access security.
Deployment Options:
- Official PMM Client ARM support, allowing the use of PMM on ARM architecture devices.
- Rootless deployments using Podman, providing flexibility and security.
- Support for deployments using Helm, Docker, Virtual Appliance, and Amazon AWS for various use cases.
Stability Improvements:
- Increased stability through containerized architecture, providing isolation and manageability.
- Streamlined upgrade process, reducing the risk of failures during updates and enhancing reliability.
User Experience:
- Flexible monitoring configurations, allowing users to tailor the system to their needs.
- UI-based upgrades for Podman installations, making the update process more convenient and intuitive.
New Features:
- Monitoring for MongoDB 8.0, ensuring support for the latest database versions.
- Integration with Watchtower for automated container updates, simplifying management and keeping the system up-to-date.

We invite you to install and try the new PMM 3.0

Quickstart guide

Get started with PMM

Multiple installation options

About PMM installation

MySQL 8.4 Support in Percona Toolkit 3.7.0

Sveta Smirnova — Mon, 06 Jan 2025 00:00:00 UTC

Percona Toolkit 3.7.0 has been released on Dec 23, 2024. The main feature of this release is MySQL 8.4 support.

In this blog, I will explain what has been changed. A full list of improvements and bug fixes can be found in the release notes.

TLDR;

Replication statements in 8.4 are fully supported by the Percona Toolkit
pt-slave-delay has been deprecated.
pt-slave-find has been renamed to pt-replica-find. The old name has been deprecated but exists in the repository as an alias of the pt-replica-find.
pt-slave-restart has been renamed to pt-replica-restart. Old name has been deprecated but exists in the repository as an alias of the pt-replica-restart.
Basic SSL support has been added to the tools where it was not working before (see https://perconadev.atlassian.net/browse/PT-191 ), and Percona Toolkit now supports caching_sha2_password and sha256_passwordauthentication plugins. Full implementation of https://perconadev.atlassian.net/browse/PT-191 is planned for the next version.

Replication Statements

MySQL 8.4 removed earlier deprecated offensive language, such as SLAVE or MASTER. This made tools written for earlier versions not compatible with the new version. Percona Toolkit was also affected, and I had to rewrite it.

However, Percona Toolkit should be able to run not only with MySQL 8.4 but also with older versions. So, the change was not a simple grep and replace of offensive words. It is not even possible for version MySQL 8.0 because new syntax was first introduced in 8.0.23 for the CHANGE REPLICATION SOURCE and START/STOP REPLICA commands. Earlier versions weren’t aware of this change.

Another challenge was the fact that I could replace all occurrences of the word SLAVE with REPLICA. Still, I could not do the same for the MASTER and SOURCE pairs because replication source-related commands are mapped differently:

Legacy syntax	Syntax without offensive words
`CHANGE MASTER`	`CHANGE REPLICATION SOURCE` (since 8.0.23)
`SHOW MASTER STATUS`	`SHOW BINARY LOG STATUS` (since 8.4.0)
`RESET MASTER`	`RESET BINARY LOGS[ AND GTIDS]` ( since 8.4.0)
`MASTER` in other commands	`SOURCE` (partially since 8.0.23, fully since 8.4)

So, I added selectors that use the correct command depending on the MySQL server version.

I intentionally implemented new syntax for version 8.4 only, so I do not have to check every single minor version of 8.0. I also did not implement new syntax for MariaDB. This may happen in the future.

However, all messages displayed to the user use the new syntax. If you rely on old syntax somewhere in your scripts, adjust them.

Internally, most of the functions were renamed to use the new syntax, but the important module lib/MasterSlave.pm kept its name.

Deprecated and Outdated Tools

As a result of this change, pt-slave-delay has been deprecated. The tool stays in the repository and works as before when connected to MySQL 8.0 or an earlier version. However, it refuses to work with MySQL 8.4. The tool will be removed in one of the future versions.

Tools pt-slave-find and pt-slave-restart were renamed to pt-replica-find and pt-replica-restart. Aliases with old names still exist, so you have time to change your scripts. However, expect that these aliases will be removed in one of the future versions as well.

Tool pt-variable-advisor has been updated to reflect current default values.

Basic SSL Support

Percona Toolkit did not have consistent SSL support: some of the tools were able to connect using SSL, and others did not. This was reported at https://perconadev.atlassian.net/browse/PT-191. In this version, I added option “s” for DSN that instructs DBD::mysql to open a secure connection with the database. As a result, Percona Toolkit now supports caching_sha2_password and sha256_password authentication plugins. But other SSL options are still missed. Full SSL support will be added in the next version.

Conclusion

Percona Toolkit fully supports MySQL 8.4. If you use pt-slave-find and pt-slave-restart, consider calling them by their new names pt-replica-find and pt-replica-restart. Tool pt-slave-delay has been deprecated and will be removed in future versions. Use built-in feature delayed replication instead.

Percona Monitoring and Management 3.0.0-Beta - Tech Preview

Ondrej Patocka — Mon, 02 Dec 2024 00:00:00 UTC

Percona Monitoring and Management 3.0.0 Beta - Tech Preview

We’re excited to announce the Tech Preview (Beta) release of Percona Monitoring and Management (PMM) 3.0.0-Beta.

This release is intended for testing environments only, as it’s not yet production-ready. The GA (General Availability) release will be available through standard channels in the upcoming months.

Release notes

To see the full list of changes, check out the 3.0.0-Beta - Tech Preview Release Notes

Installation options

PMM Server

Docker

Server: docker pull perconalab/pmm-server:3.0.0-beta
Docker installation guide

VM

PMM Client

Docker images

AMD 64 + ARM 64: docker pull perconalab/pmm-client:3.0.0-beta

Binary packages

Package Manager installation

Enable testing repository via Percona-release: percona-release enable pmm3-client testing
Install relevant pmm-client package using your system’s package manager

Percona Bug Report: October 2024

Aaditya Dubey — Mon, 25 Nov 2024 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-8057: When max_slowlog_size is set to above 4096, then it gets reset to 1073741824. This overwrites the slow log file path with a different file name, which becomes like node_name.log.000001. Due to this issue, your path defined at slow_query_log_file won`t be useful. This issue has started happening since MySQL Version 8.0.32.

E.g.:

MySQL 8.0.36 is running with the following set of configurations:

slow_query_log = ON
slow_query_log_file = /home/user/sandboxes/msb_ps8_0_36/data/slow
long_query_time = 10
max_slowlog_files = 2
max_slowlog_size = 510000000

Check the log_file path:

mysql [localhost:8036] {msandbox} ((none)) > show global variables like "%slow_query_log_file%";
+---------------------+------------------------------------------------------------+
| Variable_name | Value |
+---------------------+------------------------------------------------------------+
| slow_query_log_file | /home/adi/sandboxes/msb_ps8_0_36/data/localhost.log.000001 |
+---------------------+------------------------------------------------------------+
1 row in set (0.00 sec)

Interestingly, you will see that this file,/home/user/sandboxes/msb_ps8_0_36/data/localhost.log, 000001, was not even created.

user@localhost:~/sandboxes/msb_ps8_0_36/data$ ll | grep slow
-rw-r----- 1 adi adi 355518891 Aug 13 16:45 localhost-slow.log
-rw-r----- 1 adi adi 1079653421 Jul 30 18:13 localhost-slow.log.old
-rw-r----- 1 adi adi 255 Aug 13 16:48 slow
-rw-r----- 1 adi adi 255 Aug 13 16:45 slow.000001

After removing max_slowlog_size = 510000000

mysql [localhost:8036] {msandbox} ((none)) > show global variables like "%slow_query_log_file%";
+---------------------+--------------------------------------------+
| Variable_name | Value |
+---------------------+--------------------------------------------+
| slow_query_log_file | /home/adi/sandboxes/msb_ps8_0_36/data/slow |
+---------------------+--------------------------------------------+
1 row in set (0.01 sec)

Reported Affected Version/s: 5.7.36-39, 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.39-30, 8.4.2-2

Workaround/Fix: Use “set global slow_query_log_file =’’;”

PS-9214: INPLACE ALTER TABLE might fail with a duplicate key error if concurrent insertions occur; there have been many bugs reported here and in MySQL bugs regarding duplicate key errors while doing an online alter table operation on tables with primary and unique keys indexes. The bug is not as easy to reproduce but involves ONLY the primary key and includes an atomic sequence that cannot create a duplicate. It seems to be related to page splits/merges.

Reported Affected Version/s: 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.39-30, 8.4.2-2

Upstream Bug: 115511

Workaround/Fix: Use ALTER TABLE … ALGORITHM=COPY instead.

PS-9275: When querying based on a function, MySQL does not use the available functional index when using the LIKE operator, which results inconsistent query plans when functional Indexes are used.

E.g.:

CREATE TABLE `test` (
 `id` int NOT NULL AUTO_INCREMENT,
 `a` varchar(200) DEFAULT NULL,
 `test02` varchar(9) GENERATED ALWAYS AS (monthname(from_unixtime(`a`))) VIRTUAL,
 `!hidden!test01!0!0` varchar(9) GENERATED ALWAYS AS (monthname(from_unixtime(`a`))) VIRTUAL,
 PRIMARY KEY (`id`),
 KEY `test01` ((monthname(from_unixtime(`a`)))),
 KEY `test02` (`test02`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

mysql> explain select MONTHNAME(FROM_UNIXTIME(a)) from test WHERE MONTHNAME(FROM_UNIXTIME(a)) Like 'April%';
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| 1 | SIMPLE | test | NULL | ALL | NULL | NULL | NULL | NULL | 13 | 100.00 | Using where |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0,00 sec)

mysql> explain select MONTHNAME(FROM_UNIXTIME(a)) from test WHERE `!hidden!test01!0!0` Like 'April%';
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
| 1 | SIMPLE | test | NULL | range | test01 | test01 | 39 | NULL | 2 | 100.00 | Using where |
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0,00 sec)

Reported Affected Version/s: 8.0.36-28, 8.4.X

Upstream Bug: 104713

Workaround/Fix: Use the indexes created on virtual fields explicitly.

PS-9286: KMIP Component leaves keys in a pre-active state.

Reported Affected Version/s: 8.0.X, 8.3.0-1, 8.4.0-1

Fixed Version: PS 8.0.39-30, 8.4.2-2

PS-9314: The database crashed due to the SELECT statement. Since the JSON is invalid, the command should return ERROR 3146, an Invalid data type for JSON, but unfortunately, it crashed the instance with Signal 11 using JSON_TABLE.

Reported Affected Version/s: 8.0.36-28, 8.0.37-29, 8.0.39-30

Fixed Version: PS 8.0.39-30, 8.4.2-2

E.g.:

mysql> show global variables like 'version%';
+-------------------------+-----------------------------------------------------+
| Variable_name | Value |
+-------------------------+-----------------------------------------------------+
| version | 8.0.36-28 |
| version_comment | Percona Server (GPL), Release 28, Revision 47601f19 |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
| version_compile_zlib | 1.2.13 |
| version_suffix | |
+-------------------------+-----------------------------------------------------+
6 rows in set (0.01 sec)

mysql> SELECT ele AS domain FROM JSON_TABLE('["TEST'+(select load_file('test'))+'"]', "$[*]" COLUMNS (ele VARCHAR(70) PATH "$" )) AS json_elements ;
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
ERROR:
Can't connect to the server

PS-9369: The audit plugin causes memory exhaustion after a few days; disconnecting threads and disabling the audit plugin is undesirable. This workaround can not be used since it requires scheduling an application outage. Even when small, it’s a recurrent event.

Reported Affected Version/s: 8.0.37-29

Fixed Version: PS 8.0.40-31 [Yet to Release]

Percona Xtradb Cluster

PXC-4453: In 3 Node PXC cluster, node01 has active flow control(FC). Active FC blocks user sessions to insert a message into the channel queue (session waits on send monitor (conn->sm)); send monitor is blocked because FC is active. The idea behind the logic is that applier threads, when consuming messages from the queue conn->recv_q, should check if FC is active, and if the queue level is below conn->lower_limit, FC should be disabled, and the user connection thread waiting on the sending monitor should be woken up. In other words, disabling the FC signal is driven by the consumption of events from recv_q by applier threads.

In this case, it seems that recv_q is empty, but FC is active, so nothing can be added to recv_q. We have a vicious circle of some kind of deadlock, and due to this race condition, we are seeing cluster hangs.

Reported Affected Version/s: 5.7.25, 5.7.44, 8.0.36-28

Fixed Version: PXC 8.0.37-29, 8.4.0

PXC-4404: wsrep_preordered=ON causes protocol violations, which cause a node to crash when the group view changes on a cluster with a node acting as an async replica.

Reported Affected Version/s: 5.7.44-31

Workaround/Fix: Set wsrep_preordered=OFF; however, you may experience a delay in async replication.

Note: Option wsrep_preordered is deprecated in MySQL-wsrep: 8.0.19-26.3, MariaDB: 10.1.1

PXC-4362: The PXC node evicted when creating a function by the user doesn`t have the super privilege, and binary logging is enabled.

Reported Affected Version/s: 8.0.34-26

Fixed Version: PXC 8.0.36-28, 8.4.0

Workaround/Fix: Setting log_bin_trust_function_creators is the workaround. Note that log_bin_trust_function_creators is deprecated by MySQL 8.0.34 and will be removed in the future.

PXC-4365: PXC nodes leave clusters when the row size is too large and have more than 3 nvarchar columns.

Reported Affected Version/s: 8.0.35-27

Fixed Version: PXC 8.0.36-28, 8.3.0

Percona Toolkit

PT-2325: pt-table-sync does not produce the correct SQL statements to sync tables containing JSON columns properly.

E.g.:

pt-table-sync emits the following SQL:

DELETE FROM `test`.`test_to` WHERE `id`='2' AND `data`='{"baz": "quux"}' LIMIT 1;

INSERT INTO `test`.`test_to`(`id`, `data`) VALUES ('1', '{"foo": "bar"}');

The INSERT statement works fine, but the DELETE fails to delete the row with id=‘2’, because the AND data=’{“baz”: “quux”}’ portion of the WHERE clause will result in the query matching zero rows.

Verify the incorrect contents of the test_to table with the following:

# Examine the state of our test tables.
$ docker exec -it mysql_5_7_12_test mysql -utest -ptest -e "
 use test;
 select * from test_to;"

That should return the following output:

+------+-----------------+
| id | data |
+------+-----------------+
| 2 | {"baz": "quux"} |
| 1 | {"foo": "bar"} |
+------+-----------------+

Witness that the row with id=2 still exists in the table and was not deleted as it should have been. With JSON columns, the DELETE statement would need to look like this:

DELETE FROM `test`.`test_to`
WHERE `id`='2'
AND `data`=CAST('{"baz": "quux"}' AS JSON)
LIMIT 1;

Reported Affected Version/s: 3.5.7

PT-2329: During the run, pt-archiver will ignore columns that are camelCase during the insert, but it will get all the columns during select.

It could be confirmed by using a dry-run:

pt-archiver --source [...] --dest [...] --where "1=1" --statistics --progress=10000 --limit=1000 --no-delete --no-safe-auto-increment --no-check-columns --columns=addressLine1,addressLine2,city,state,postalCode,country,customerNumber --why-quit --skip-foreign-key-checks --dry-run

Here are the results:

SELECT /*!40001 SQL_NO_CACHE */ `addressLine1`,`addressLine2`,`city`,`state`,`postalCode`,`country`,`customerNumber`,`customernumber` FROM `classicmodels`.`customers` FORCE INDEX(`PRIMARY`) WHERE (1=1) ORDER BY `customernumber` LIMIT 1000

SELECT /*!40001 SQL_NO_CACHE */ `addressLine1`,`addressLine2`,`city`,`state`,`postalCode`,`country`,`customerNumber`,`customernumber` FROM `classicmodels`.`customers` FORCE INDEX(`PRIMARY`) WHERE (1=1) AND ((`customernumber` > ?)) ORDER BY `customernumber` LIMIT 1000

INSERT INTO `classicmodels`.`addresses`(`city`,`state`,`country`) VALUES (?,?,?)

Reported Affected Version/s: 3.5.7

Workaround/Fix: The solution is to include all columns in lowercase in the param –columns until the bug is fixed.

PT-2344: pt-config-diff compares mysqld options, but it fails if the [mysqld] section is in uppercase, even though that is a valid way of setting mysqld variables. Since [MYSQLD] is acceptable for MySQL, pt-config-diff should compare the options under that section.

Reported Affected Version/s: 3.5.7

Workaround/Fix: Use [mysqld] as lowercase until the bug is fixed.

PT-2355: Table data is lost if we accidentally resume a previously failed job that has null boundaries. pt-online-schema-change should not resume a job with empty boundaries.

Reported Affected Version/s: 3.6.0

Fixed Version: PT 3.7.1

Workaround/Fix: Do not run pt-online-schema-change with job id having null boundaries.

PT-2356: If you run pt-online-schema-change, which results in an error, then subsequent runs will create new tables that won`t be cleaned up.

Reported Affected Version/s: 3.6.0

PT-2349: pt-table-sync is failing to sync data from PXC to the async environment, and trigger errors include “WSREP detected deadlock/conflict and aborted the transaction.”

Reported Affected Version/s: 3.3.1, 3.5.2, 3.6.0

PT-1726: pt-query-digest is not distinguishing queries when an alias is used

Reported Affected Version/s: 3.6.0

E.g.:

Queries from slow query log:

Time: 2019-01-31T11:00:00.728957Z
# User@Host: sageone_ext_uk[sageone_ext_uk] @ [10.181.130.22] Id: 18714290
# Query_time: 2.709699 Lock_time: 0.000402 Rows_sent: 19 Rows_examined: 51011
use sageone_ext_uk;
SET timestamp=1548932400;
SELECT a,b,c from table1 as t1 where t1.a=3 and t1.b=5;

# Time: 2019-01-31T11:00:00.728957Z
# User@Host: sageone_ext_uk[sageone_ext_uk] @ [10.181.130.22] Id: 18714290
# Query_time: 2.709699 Lock_time: 0.000402 Rows_sent: 19 Rows_examined: 51011
use sageone_ext_uk;
SET timestamp=1548932400;
SELECT a,b,c from table1 as t1 where t1.a=3 and t1.c=5;

The fingerprints for the above queries are the same, which is incorrect behaviour:

select a,b,c from table? as t? where t?=? and t?=?

PT-2374: If we say –ignore=bob, every combination of the bob user will be ignored. This includes bob@localhost, bob@::1, bob@foobar, etc. But this is not the case. Only bob@% is ignored; pt-show-grants –ignore does not ignore all accounts.

Reported Affected Version/s: 3.6.0

PT-2375: When pt-table-sync is used on a table with a GENERATED AS column, it fails because we cannot REPLACE/INSERT values into a GENERATED column.

E.g.:

`requestStatus` tinyint(1) GENERATED ALWAYS AS (if((`provRequired` = 0),(`httpSyncStatus` not between 200 and 299),(`httpAsyncStatus` not between 200 and 299))) VIRTUAL,

ERROR 3105 (HY000): The value specified for generated column 'requestStatus' in table 'qqq' is not allowed.

The —-ignore-columns parameter specifically states that if a REPLACE/INSERT is needed, all columns will be used. Due to this, the pt-table-sync does not work with the generated columns.

Reported Affected Version/s: 3.6.0

PMM [Percona Monitoring and Management]

PMM-12013: If we add many RDS instances to the PMM server, say 200+, and Change the prom scrape.maxScrapeSize to the value that allows the VM to parse the reply from the exporter, then the metrics are gathered unreliably, there are gaps, and the exporter`s RSS feed goes to like 5GB for instance. This concludes that rds_exporter is unreliable for large deployments.

Reported Affected Version/s: 2.35.0

Fixed Version: PMM 3.0.0-Beta available as Tech Preview]

PMM-12161: In the Mongodb cluster summary page, Under QPS of Config Services dashboard, it is being clubbed configRS, mongoS and mongod servers. This results in too many configuration services under the QPS of the Config Services dashboard.

Reported Affected Version/s: 2.42.0

Fixed Version: PMM 3.1 [Yet to Release]

PMM-12993: In PMM, CPU metrics have a label “mode” to identify between CPU info: sys, iowait, nice, user, idle, etc. With 1 rds instance, the metric is perfectly fine. However, after adding more instances, the CPU metric is still collected, but the “mode” label is empty, which breaks the graphs in the Advanced Data Exploration dashboard.

Reported Affected Version/s: 2.41.1, 2.41.2

PMM-13148: If we run the queries without using the schema name, then we don`t see such queries in the QAN.

E.g.:

mysql [localhost:8036] {msandbox} ((none)) > update test.joinit set g=100,t="06:44:50" where i=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

Here, we can see we did not explicitly select the database name using the USE command and executed the query directly. This results in QAN not being able to capture such queries for analytics.

Reported Affected Version/s: 2.41.2

Workaround/Fix: Run queries with USE ; ;

PMM-13252: A 500 error message is returned while creating the role with the existing name.

E.g.:

Enable Access roles in PMM Settings
Open the Access role page and create a role with the name “Test. “
Try to create a new role with the name “Test. “

It returns with Internal server error 500:

in logs: msg="RPC /accesscontrol.v1beta1.AccessControlService/CreateRole done in 1.409839ms with unexpected error: pq: duplicate key value violates unique constraint "roles_title_key""

Reported Affected Version/s: 2.42.0

Workaround/Fix: It is expected to be fixed in PMM 3

PMM-13277: When we try to launch PMM using AWS AMI as mentioned in our docs. However, the AWS webpage works fine, and it logins, but every graph and details are blank with “Server error 502” The same can be seen in the log for Victoria metrics:

The following error will be seen:

2024-07-27T06:40:22.848Z panic /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/mergeset/part_header.go:88 FATAL: cannot read "/srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json": open /srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json: no such file or directory
panic: FATAL: cannot read "/srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json": open /srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json: no such file or directory

Reported Affected Version/s: 2.42.0

Fixed Version: PMM 2.43.0

Percona XtraBackup

PXB-3302: If the number of GTID sets is absolutely large on a MySQL instance, the output “GTID of the last change” in the Xtrabackup log is truncated compared to the full output in xtrabackup_binlog_info and xtrabackup_info. This can be an issue for external tools obtaining the GTID coordinates from the log as it would be impractical to get the coordinates from xtrabackup_binlog_info or xtrabackup_info on a large, compressed xbstream file.

Here is a snippet of a backup log:

2024-06-03T10:18:59.678581+08:00 0 [Note] [MY-011825] [Xtrabackup] MySQL binlog position: filename 'mysql-bin.000002', position '10197', GTID of the last change ** REDACTED **,9f18624b-214f-11ef-871f-b445068273a0:1,9f1bf5ae-214f-11ef-871f-b445068273a0:1,9f1f6fac-214f-11ef-871f-b445068273a0:1,9f231076-214f-11ef-871f-b445068273a0:1,9f26d153-214f-11ef-871f-b445068273a0:1,9f2a5fdd-214f-11ef-871f-b445068273a0:1,9f2df6e8-214f-11ef-871f-b445068273a0:1,9f318143-214f-11ef-871f-b445068273a0:1,9f353351-214f-11ef-871f-b445068273a0:1,9f38f96c-214f-11ef-871f-b445068273a0:1,9f3cdc53-214f-11ef-871f-b445068273a0:1,9f40fcc0-214f-11ef-871f-b445068273a0:1,9f44955a-214f-11ef-871f-b445068273a0:1,9f481188-214f-11ef-871f-b445068

Snippet of xtrabackup_binlog_info:

mysql-bin.000002 10197 ** REDACTED **,9fdd9048-214f-11ef-871f-b445068273a0:1,9fe138a3-214f-11ef-871f-b445068273a0:1,9fe4c3d8-214f-11ef-871f-b445068273a0:1,9fe82e39-214f-11ef-871f-b445068273a0:1

Reported Affected Version/s: 8.0.35-30

Fixed Version: PXB 8.4.0-1, 8.0.35-32

PXB-3283: When xtrabackup takes a backup and exports a tablespace, xtrabackup gets the wrong table definition from the ibd for tables that have changed the charset-collation in MySQL before backup.

Eg:

CREATE TABLE test.a (
 a datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_as_ci;

the collation_id is 8 (latin1_swedish_ci)

shell> ibd2sdi /var/lib/mysql/test/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 8

When MySQL converts the charset on a table, it converts the date and time data types columns in the ibd file but not the data dictionary cache. The collation in the ibd does not match that of the data dictionary.

ALTER TABLE test.a CONVERT TO CHARACTER SET utf8mb4 collate utf8mb4_unicode_ci;

The collation_id becomes 224 (utf8mb4_unicode_ci)

shell> ibd2sdi /var/lib/mysql/test/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 224

The collation_id of the copied table is 8 (latin1_swedish_ci)

create table xb.a like test.a;

shell> ibd2sdi /var/lib/mysql/xb/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 8

When xtrabackup exports the tablespace, the collation_id is 224 in ibd. Xtrabackup will write it to cfg metadata file.

When MySQL imports a tablespace, MySQL gets an error Column %s precise type mismatch because the collation_id of MySQL does not match that of xtrabackup.

Reported Affected Version/s: 8.0.35-30

Fixed Version: PXB 8.4.0-1, 8.0.35-31

PXB-2797: When importing a single table (IMPORT TABLESPACE) from a backup made using xtrabackup and the table contains a full-text index, the import process will error out with:

ERROR 1808 (HY000) at line 132: Schema mismatch (Index xxxxxx field xxxxxx is ascending which does not match metadata file which is descending)

Reported Affected Version/s: 8.0.28-20

Fixed Version: PXB 8.0.35-31

PXB-3210: PXB fails to build on macOS since 8.0.33-28 due to FIND_PROCPS()

CMake Error at cmake/procps.cmake:29 (MESSAGE):
 Cannot find proc/sysinfo.h or libproc2/meminfo.h in . You can pass it to
 CMake with -DPROCPS_INCLUDE_PATH= or install
 procps-devel/procps-ng-devel/libproc2-dev package
Call Stack (most recent call first):
 storage/innobase/xtrabackup/src/CMakeLists.txt:24 (FIND_PROCPS)

Reported Affected Version/s: 8.0.33-28, 8.0.34-29, 8.0.35-30

Fixed Version: PXB 8.0.35-31

PXB-3130: Performing upgrade from PS 8.0.30 -> PS 8.0.33 using PXB

Use PXB 8.0.30 on PS 8.0.30
Copy to new host
Prepare using PXB 8.0.33
Start PS 8.0.33

Which results in the Assertion:

I0825 22:33:01.738917 05155 ???:1] xtrabackup80-apply-log(stderr) - InnoDB: Assertion failure: log0recv.cc:4353:log.m_files.find(recovered_lsn) != log.m_files.end()

Reported Affected Version/s: 8.0.30

Fixed Version: PXB 8.0.35-31

Percona Kubernetes Operator

K8SPXC-1398: Scheduled PXC backup job pod fails to complete the process successfully in a random/sporadic fashion.

Error Returns As:

+ EXID_CODE=4
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessfull'
Terminating processProcess completed with error: /usr/bin/run_backup.sh: 4 (Interrupted system call)2024-05-03 09:39:08 [ERROR] Backup was finished unsuccessfull
+ exit 4

Reported Affected Version/s: 1.13.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

Note: Since we don`t have steps to reproduce the issue, it is hard to confirm whether the fix is working as expected. Please feel free to provide feedback or create a Jira if required.

K8SPXC-1397: The operator`s default configuration makes the cluster unusable if TDE (Transparent data encryption) is used; the entry point of the PXC container configures the parameter binlog_rotate_encryption_master_key_at_startup. As a workaround, binlog_rotate_encryption_master_key_at_startup should be disabled. However, it has security implications.

Reported Affected Version/s: 1.12.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

K8SPXC-1222: Upgrading Cluster Fails When Dataset Has Large Number Of Tables. When the operator replaces the first pod with one with the new version, it fails to start up and gets stuck in a loop that restarts every 120 seconds.

The problem looks like from pxc-entrypoint.sh:

for i in {120..0}; do
 if echo 'SELECT 1' | "${mysql[@]}" &>/dev/null; then
 break
 fi
 echo 'MySQL init process in progress...'
 sleep 1
 done

Reported Affected Version/s: 1.11.0, 1.12.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

Orchestrator

DISTMYSQL-406: Orchestrator 3.2.6-11 shows the MySQLOrchestratorPassword variable value in the error log and when accessing the web interface.

E.g.:

Create a MySQL and Orchestrator node

mysql -e "CREATE USER 'orchestrator_srv'@'%' IDENTIFIED BY 'orc_server_password'; GRANT ALL ON orchestrator.* TO 'orchestrator_srv'@'%';"

Configure Orchestrator to use node0 as MySQL backend database

vi /usr/local/orchestrator/orchestrator.conf.json

Add the following lines and remove sqlite options:

 "MySQLOrchestratorHost": "node_0_IP",
 "MySQLOrchestratorPort": 3306,
 "MySQLOrchestratorDatabase": "orchestrator",
 "MySQLOrchestratorUser": "orchestrator_srv",
 "MySQLOrchestratorPassword": "orc_server_password",

On node1, there are several messages showing the backend password:

Feb 28 23:26:03 XX-XX-node1 orchestrator[4262]: 2024-02-28 23:26:03 ERROR 2024-02-28 23:26:03 ERROR QueryRowsMap(orchestrator_srv:orc_server_password@tcp(10.124.33.138:3306)/orchestrator?timeout=1s&readTimeout=30s&rejectReadOnly=false&interpolateParams=true) select hostname, token, first_seen_active, last_seen_Active from active_node where anchor = 1: dial tcp 10.124.33.138:3306: connect: connection refused

Reported Affected Version/s: 8.0.36(PS)

Fixed Version: 8.4.0(PS)

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Run PMM Server with Docker instructions

About Percona:

As the only provider of distributions for all three of the most popular open source databases—PostgreSQL, MySQL, and MongoDB—Percona provides expertise, software, support, and services no matter the technology.

Whether its enabling developers or DBAs to realize value faster with tools, advice, and guidance, or making sure applications can scale and handle peak loads, Percona is here to help.

Percona is committed to being open source and preventing vendor lock-in. Percona contributes all changes to the upstream community for possible inclusion in future product releases.

Percona Monitoring and Management 2.43.0 Preview Release

Ondrej Patocka — Thu, 12 Sep 2024 00:00:00 UTC

Percona Monitoring and Management 2.43.0 Tech Preview Release

Hello everyone! Percona Monitoring and Management (PMM) 2.43.0 is now available as a Tech Preview Release.

We encourage you to try this PMM preview release in testing environments only, as these packages and images are not fully production-ready. The final version is expected to be released through the standard channels in the coming week.

To see the full list of changes, check out the PMM 2.43.0 Tech Preview Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.43.0-rc

PMM client package installation

Download AMD64 or Download ARM64 pmm2-client tarball for 2.43.0.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

Run PMM Server as a VM instructions

PMM2-Server-2.43.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-0db618c7da6e202f4

Timeline for Database on Kubernetes

Edith Puclla — Tue, 16 Jul 2024 00:00:00 UTC

The Evolution

Since its inception in June 2014, Kubernetes has dramatically transformed container orchestration, revolutionizing the management and scaling of applications. To mark its tenth anniversary, the Data on Kubernetes Community (DoKC) released an infographic showcasing key milestones and community contributions to the evolution of operators for managing stateful applications. This project was made possible by the collaboration of DoKC members Edith Puclla, Sergey Pronin, Robert Hodges, Gabriele Bartolini, Chris Malarky, Mark Kember, Paul Au, and Luciano Stabel.

Explore the infographic to see how Kubernetes has shaped the future of database management on Kubernetes.

Adoption and Impact

The CNCF says that 84% of organizations are using or considering Kubernetes, with 70% running stateful applications on it in production. The number of users and containers has grown, showing that more people are contributing, adopting cloud-native technologies, and finding new ways to use Kubernetes to manage stateful applications.

Looking Ahead

As we celebrate ten years of Kubernetes, the way databases are integrated keeps improving, thanks to community efforts and new technology. Percona Everest is an excellent example of this progress. It’s more than just a tool for databases; it represents the future of running databases on Kubernetes. It’s open-source and makes running any database on cloud-based Kubernetes clusters easy. If you want to try it, visit our Percona Everest GitHub Repository and give us a star if you like it. For feedback or comments, join the Percona Forum for Percona Everest discussion.

Percona Joins Community Over Code 2024 in Bratislava, Slovakia

Edith Puclla — Fri, 21 Jun 2024 00:00:00 UTC

Last week, I participated as a speaker for the first time at Community Over Code 2024.

Community Over Code

Community Over Code is a key principle at Apache, highlighting the importance of having a solid and collaborative community rather than just focusing on the code. While good code is essential, the community’s strength and resilience keep a project going and growing. I love how this is expressed in the “Apache Way.”

Renaming ApacheCon to “Community Over Code” reflects this idea, emphasizing the central role of community in Apache’s approach. This year, Community Over Code was in Bratislava, Slovakia. My talk was in the Community track, where I spoke about Outreachy Internship, the Apache Software Foundation, and Apache Airflow. My goal was to celebrate the success of the Outreachy program, which has surpassed 1,000 internships, and I am proud to be one of them.

I talked about the community, shared success stories, and provided clear examples, such as Bowrna Prabhakaran and Ephraim Anierobi, and shared my personal experience. I also explained how Outreachy, along with my mentors from Apache Airflow, Jarek Potiuk, and Elad Kalif, boosted my professional career in Open Source 03 years ago.

Working at Percona

Now I work for Percona, which has been recognized as one of Inc. Magazine’s 2024 Best Workplaces! I love my job and the amazing things we are building at Percona. Percona is an open-source software company that fully believes that open source should remain open throughout. This belief inspires me to work at Percona.

What We’re Focused on Now

If you’re interested in improving database performance on Kubernetes, you will love Percona Everest. It is our cloud-native solution for efficiently managing and running databases on Kubernetes. With a very user-friendly interface, you can run any database on any cloud provider or on-premises.

Here are some use cases where you might use Percona Everest.

Seeking ways to make the development of internal platforms easier and faster.
Looking for affordable alternatives to public Database-as-a-Service (DBaaS) offerings.
Building multi-cloud or hybrid cloud setups to meet data compliance needs for multi-regional businesses.
Wishing to leverage Kubernetes for its scalability and stability to run databases efficiently.
Transitioning from monolithic to microservices architecture to modernize legacy database infrastructure.

Any feedback is welcome on our Percona Everest forum, and if you like it, give us a start on GitHub: github.com/percona/everest.

Let's take a look at Percona Everest 1.0.0 RC

Daniil Bazhenov — Fri, 14 Jun 2024 00:00:00 UTC

Hi, the Percona Everest 1.0.0-rc1 release was published on GitHub.

Percona Everest is the first open source cloud-native platform for provisioning and managing PostgreSQL, MongoDB and MySQL database clusters.

I want to tell you how to install it so you can try it out.

RC builds aren’t meant for the general public; we don’t support upgrading from RC to stable versions. This means that this is only for testing and familiarizing yourself with the features. RC builds are not stable and are often buggy. There will be no upgrade. :)

To get started, you will need a Kubernetes cluster. Right now, Percona Everest is in Beta. Don’t use production clusters; use test clusters in the cloud like GKE or local in Minikube, k3d, or Kind.

I created a test cluster in GKE with the command:

gcloud container clusters create test-everest-rc --project percona-product --zone us-central1-a --cluster-version 1.27 --machine-type n1-standard-4 --num-nodes=3

Delete it after the test with the command:

gcloud container clusters delete test-everest-rc --zone us-central1-a

Now, we need Everest CLI for the RC version; download it from GitHub for your operating system.

I downloaded it, renamed it to everestctl, and copied it to a folder for experimentation.

Now, we need to make it executable

chmod +x ./everestctl

Let’s check that everestctl works and that we have the correct version.

./everestctl version

We can now install Percona Everest.

./everestctl install --version-metadata-url https://check-dev.percona.com

Note that I use the --version-metadata-url parameter https://check-dev.percona.com; this is required for RC builds.

During the installation process, you must set one or more namespaces and databases.

Once the installation is complete, the new user authentication feature is the first significant change. You will be offered two commands.

Command to retrieve the admin user password that was generated automatically during installation:

./everestctl accounts initial-admin-password

Command to set a new password:

./everestctl accounts set-password --username admin

Now that we know the admin user password, we can open Percona Everest in a browser. Run the following command to use kubectl port forwarding to connect to Percona Everest without exposing the service:

kubectl port-forward svc/everest 8080:8080 -n everest-system

Take a Clone it will last longer

Wayne Leutwyler — Sun, 02 Jun 2024 00:00:00 UTC

So cloning is a great subject. I mean we clone sheep, we can clone human organs in time we might be able to clone humans, but thats a topic for scientist and philosophers.

What is MySQL Replicaion:

The MySQL clone plugin can be used to replicate data from a MySQL server to another MySQL server, and it supports replication. The cloning process creates a physical snapshot of the data, including tables, schemas, tablespaces, and data dictionary metadata. It tracks replication coordinates from the source server and transfers them to the replica, which allows replication to begin at a consistent position in the replication stream. This data includes the binary log position (filename, offset) and the gtid_executed GTID set. The replication metadata repositories are also copied during the cloning operation.

The clone plugin that was released with MySQL Version 8.0.17. Its quick and very easy to setup. You can use it for so many different solutions. I’ve listed some common ones below, but I know that there are many more use cases.

Create a new replica.
Recover a replica which is out of sync with the primary.
Quickly deploy MySQL servers with data set already in place.

In this article I will cover the basics of setting up and running a clone from and existing MySQL server. I will be using MySQL version 8.0.36.

Prepare the Source Server

Install the clone plugin.
Create a clone user.

Install Clone plugin

source > INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Verify the clone plugin was installed.

source > show plugins;

Create Clone User

Now we need to create a user to run the clone process. I highly suggest you create a user for the cloning. Please don’t use an ID with full admin rights. Create a user with the least amount of privileges needed.

source > CREATE USER 'clone_user'@'%' IDENTIFIED BY 'S3k3rtPassWd';
source > GRANT BACKUP_ADMIN ON *.* TO 'clone_user'@'%';

Verify your new clone user.

source > show grants for clone_user;

Preparation on the source server is complete. Lets move on to the clone.

Prepare the Clone Server

We need to start by installing the clone plugin.
Then we need to define the source that clone will be based on.

Install plugin

clone > INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Verify the clone plugin was installed.

clone > show plugins;

Define Source Server

You can use the source host name or host IP address.

clone > SET GLOBAL clone_valid_donor_list='SOURCE_HOSTNAME:3306';

Start and monitor the cloning process

We are now ready to kick off the cloning.

clone > CLONE INSTANCE FROM clone_user@SOURCE_HOSTNAME:3306 IDENTIFIED BY 'S3k3rtPassWd';

Cloning should start now. If you have any issues, check your log files and very the steps above. Now that the cloning is running we can monitor the cloning process using this command.

clone > SELECT * FROM performance_schema.clone_progress\G

As you can see in the output below, that the cloning has completed DROP DATA and is now working on FILE COPY.

Observations

Just recently I worked with a customer who moved a 5.6TB database from a Galera Cluster to standard MySQL replication. The data was moved from the source to replicas in two different locations. Timings are detailed below.

Source to Replica in same location: 5.6TB in approxmently 50 mins.
- 112GB per minute.
Source to Replica in different geographic locations: 5.6TB in approxmently 1 hour and 45 mins.
- 53.33GB per minute.

MySQL Clone Plugin’s robust performance and efficiency in managing large data transfers, making it an excellent tool for environments requiring quick and reliable data replication.

Summary

The MySQL Clone Plugin offers several benefits, including:

Fast Data Copying:
- Enables rapid cloning of MySQL instances, facilitating quick data replication and environment setup.
Consistent Data State:
- Ensures data consistency during cloning, avoiding issues that can arise from manual copying or inconsistent states.
Reduced Downtime:
- Minimizes downtime during cloning operations, crucial for maintaining service availability.
Ease of Use:
- Simplifies the cloning process through straightforward commands, reducing the complexity for database administrators.
Automated Cloning Process:
- Automates many steps involved in the cloning process, reducing the potential for human error and increasing efficiency.

Reference

The Clone Plugin

Image by kjpargeter on Freepik

Learn PostgreSQL and SQL Quickly and Free

David Stokes — Mon, 20 May 2024 00:00:00 UTC

Learn PostgreSQL and SQL Quickly and Free

Want to learn PostgreSQL? What if you could learn PostgreSQL without having to install the database, load data, find sample data, and it was free? PGExamples.com is what you are looking for.

First, let me state that I have no connection to the website or its author. I discovered PgExercises.com by accident and was impressed by its quality and completeness. I am so impressed that I am exploring a video series on it.

Learn SQL. Learn PostgreSQL.

PGexercises.com is based on increasingly complex exercises based on a provided dataset. As you progress, the questions get trickier. You do not have to download and load the available data into a server to do the exercises.

There are seven groups of exercises - Basic, Joins and Subqueries, Modifying Data, Aggregates, Date, String, and Recursive. The author recommends Learning SQL by Alan Beaulieu as a reference text. Any essential book, Structured Query Language, will serve you well.

First Exercise

The first exercise asks you to retrieve all the data from a specific table.

At the top of the screen is an entity relationship map of the tables in the database. The table requested in the exercise is named cd.facilities.

If you use the hint button (next to the Your Answer line) you will see the following:

This should be enough of a clue to allow you, with the aid of your SQL reference, to find the answer. Enter your query and click on the Eun Query button.

The green check means that the answer is correct. But what if you did not get it correct. Scroll down and select the Answers and Discussion button.

Not only do you see the answer to the question, but there is a detailed analysis of the answer.

Video Series?

I am so impressed wth these exercises that the Percona Community Team is in discussion of creating a series of videos on them. Not only that, on of my colleagues who is learning PostgreSQL is volunteering to work through the exercises so that the series will feel like a study group. And I will comment on the query, possible alternatives when they exist, and help you work through the ‘rough spots’.

If you want to work along, please do. Let us know where you struggle or the places we are redundant. Out goal is to provide a first class learning experience to help people learn SQL and PostgreSQL.

Release Roundup May 15, 2024

David Quilty — Wed, 15 May 2024 00:00:00 UTC

Percona software releases and updates April 30 - May 15, 2024.

Percona is a leading provider of unbiased, performance-first, open source database solutions that allow organizations to easily, securely, and affordably maintain business agility, minimize risks, and stay competitive, free from vendor lock-in. Percona software is designed for peak performance, uncompromised security, limitless scalability, and disaster-proofed availability.

Our Release Roundups showcase the latest Percona software updates, tools, and features to help you manage and deploy our software. It offers highlights, critical information, links to the full release notes, and direct links to the software or service itself to download.

Today’s post includes releases and updates that have been released since April 29, 2024. Take a look.

Percona Distribution for MongoDB 6.0.15

Percona Distribution for MongoDB 6.0.15 was released on April 30, 2024. It is a freely available MongoDB database alternative that gives you a single solution that combines enterprise components from the open source community, designed and tested to work together. Please see the release notes for a full list of improvements and bug fixes.

Download Percona Distribution for MongoDB 6.0.15

Percona Server for MongoDB 6.0.15-12

On April 30, 2024, we released Percona Server for MongoDB 6.0.15-12. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 6.0.15. It is based on MongoDB 6.0.15 Community Edition and supports the upstream protocols and drivers. Please see the release notes for a full list of improvements and bug fixes.

Download Percona Server for MongoDB 6.0.15-12

Percona XtraDB Cluster 5.7.44-31.65.2

Percona XtraDB Cluster 5.7.44-31.65.2 was released on May 2, 2024. This release is part of MySQL 5.7 Post-EOL Support from Percona, and the fixes are available to MySQL 5.7 Post-EOL Support from Percona customers.

That’s it for this roundup, and be sure to follow us on Twitter to stay up-to-date on the most recent releases! Percona is a leader in providing best-of-breed enterprise-class support, consulting, managed services, training, and software for MySQL, MongoDB, PostgreSQL, MariaDB, and other open source databases in on-premises and cloud environments and is trusted by global brands to unify, monitor, manage, secure, and optimize their database environments.

How to Provision a MongoDB Cluster in Kubernetes with Percona Everest Summary

Edith Puclla — Thu, 02 May 2024 00:00:00 UTC

Kubernetes continues evolving, and the complexity of deploying and managing databases within the ecosystem is a topic of considerable discussion and importance these days. This article summarizes a detailed discussion between Piotr Szczepaniak and Diogo Recharte, who offer insights and live demonstrations to simplify database operations on Kubernetes with a new technology for cloud-native applications: Percona Everest. If you want to watch the full video, check out How to Provision a MongoDB Cluster in Kubernetes Webinar.

Peter mentions that Initially, people were doubtful about using virtual machines for databases, just like they were skeptical about Kubernetes. However, the topic brings together many people who run databases on containers to share their use cases and new discussions at events like Data on Kubernetes Day at Kubecon.

The introduction of StatefulSets and Persistent Volumes has altered the perception of Kubernetes from being purely ephemeral to being capable of handling persistent data. This change is important for database applications that require data retention over time.

The Kubernetes ecosystem is rapidly expanding. This growth is thanks to its open-source nature and the continuous addition of new functionalities, such as support for specialized hardware like GPUs, which are crucial for AI and machine learning applications.

Peter also mentioned that its complexity is the main barrier to Kubernetes adoption for databases. Organizations often need help with the layer added by Kubernetes on top of database management. Also, failure in initial attempts to integrate Kubernetes can discourage organizations from further attempts, primarily due to a lack of internal expertise.

Benefits of Database as a Service (DBaaS)

DBaaS significantly reduces the time required for database provisioning, which is particularly useful in organizations needing rapid deployment. Public and private DBaaS solutions offer scalability, which is crucial for handling varying workloads and organizational growth without compromising performance.

Private vs. Public DBaaS

Private DBaaS offers more extensive customization options and control over databases, which is essential for companies with specific needs that public solutions cannot meet.

Data security and compliance with regulations are more manageable in a private DBaaS because it operate within the company’s internal infrastructure.

Demo to Deploying MongoDB on Kubernetes

Diogo presented a demo of deploying a MongoDB database using Percona’s Everest platform on Kubernetes, where he showed how to handle daily operations and disaster recovery scenarios efficiently. Watch the Percona Everest Demo on YouTube

The session explained how Kubernetes operators and custom resources help manage databases more easily. They do this by simplifying complex processes and automating regular tasks.

Some questions that users asked in this presentation are:

How do we handle the PV when the Pods go down?

The PV will remain in place; this is a standard functionality of a stateful set. After the Pod goes down, the replacement Pod will attach to the PVC, which is standard behavior for a stateful set.

What happens if the node in Kubernetes goes down?

It depends on the storage layer that you have configured in your cluster. If the storage class you are using is tied to that node, then placing it on a new node will provision a new one, and some reconciliation will occur within the database itself.

What is the current state of Percona Everest?

Percona Everest is currently in a Beta stage, and Percona aims to release a GA version. The project is fully open source, and anyone can join our project on GitHub. We appreciate feedback from the community.

Do you want to send us feedback or contribute in this cool project? We are completely open-source, you can visit Percona Everest on GitHub.

Using ProxySQL Query Mirroring to test query performance on a new cluster

Isobel Smith — Wed, 01 May 2024 00:00:00 UTC

ProxySQL is an SQL aware proxy, which gives DBA’s fine grained control over clients’ access to the MySQL cluster. A key part of our DBA team’s process in testing and preparing for major MySQL version upgrades is comparing query plans using ProxySQL query mirroring. This feature allows us to mirror queries to another cluster / host, by configuring query rules. What makes mirroring particularly useful is the ability to selectively mirror queries based on the query digest, or client user. Results from the queries that are mirrored are not returned to the client, and are sent to /dev/null.

Before configuring ProxySQL for Query Mirroring, ensure that the clients that you want to mirror the queries for, are able to connect to both the current, and the new cluster. You should also ensure that the ProxySQL monitor can connect to the new cluster, otherwise ProxySQL will mark the new hosts as offline, and the queries will not be mirrored there.

To set up query mirroring in ProxySQL:

In order to set up query mirroring, you need to add the new hosts into the mysql_server table in ProxySQL. This is how the current mysql_servers table looks, before we add the new host that we want to mirror the queries to:

MySQL> SELECT hostgroup_id, hostname FROM mysql_servers;
+--------------+--------------+
| hostgroup_id | hostname |
+--------------+--------------+
| 10 | 10.12.0.123 |
| 20 | 10.12.0.123 |
| 20 | 10.16.0.456 |
| 20 | 10.16.0.789 |
+--------------+--------------+
4 rows in set (0.01 sec)

It is important to choose a hostgroup_id that is not yet in use. You can double check the currently configured host groups in the mysql hostgroups table, as you do not want to inadvertently add the mirror hosts into the production traffic!

MySQL> select * from mysql_replication_hostgroups;
+------------------+------------------+------------+----------------------+
| writer_hostgroup | reader_hostgroup | check_type | comment |
+------------------+------------------+------------+----------------------+
| 10 | 20 | read_only | Async Cluster |
+------------------+------------------+------------+----------------------+
1 row in set (0.00 sec)

Please note, in our example, we are using async replication, so we check the mysql_replication_hostgroups table, but the hostgroups table you need to check, depends on the cluster architecture you are using:

async replica clusters check the mysql_replications_hostgroups table.
galera clusters check the mysql_galera_hostgroups table
group replication check the mysql_group_replication_hostgroups table.

We are using hostgroup 10 for the writer hostgroup, and hostgroup 20 for the reader. For this example, we will choose 100 for the mirror hostgroup_id. Once you have decided on an unused hostgroup ID, add the new clusters’ nodes to the mysql_servers table in ProxySQL.

MySQL> INSERT INTO mysql_servers(host, hostgroup, comment) VALUES ("10.12.0.987", 100, "mirror_cluster");
LOAD MYSQL SERVERS TO RUN;
SAVE MYSQL SERVERS TO DISK;

The mysql_servers table will now include the new host:

MySQL> SELECT hostgroup_id, hostname FROM mysql_servers;
+--------------+--------------+
| hostgroup_id | hostname |
+--------------+--------------+
| 10 | 10.12.0.123 |
| 20 | 10.12.0.123 |
| 20 | 10.16.0.456 |
| 20 | 10.16.0.789 |
| 100 | 10.12.0.987 |
+--------------+--------------+
4 rows in set (0.01 sec)

In order to enable query mirroring, you need to update the mirror_hostgroup column in the mysql_query_rules table. When mirroring is not enabled, the value of the mirror_hostgroup column is NULL. Our query rules before enabling query mirroring are defined as:

MySQL> select rule_id, username, match_digest, destination_hostgroup, mirror_hostgroup from mysql_query_rules;
+---------+------------------------+---------------------+-----------------------+------------------+
| rule_id | username | match_digest | destination_hostgroup | mirror_hostgroup |
+---------+------------------------+---------------------+-----------------------+------------------+
| 1 | myApplicationUser | ^SELECT.*FOR UPDATE | 10 | NULL |
| 2 | myApplicationUser | ^SELECT | 20 | NULL |
+---------+------------------------+---------------------+-----------------------+------------------+
2 rows in set (0.00 sec)

To enable mirroring, we just need to update the mirror_hostgroup. For this example, we will mirror all the SELECT queries made by myApplicationUser:

UPDATE mysql_query_rules SET mirror_hostgroup = 100 where rule_id=2;
LOAD mysql query rules TO RUN;
SAVE mysql query rules TO DISK;

The rules should now be updated:

MySQL> select rule_id, username, match_digest, destination_hostgroup, mirror_hostgroup from mysql_query_rules;
+---------+-----------------------+---------------------+-----------------------+------------------+
| rule_id | username | match_digest | destination_hostgroup | mirror_hostgroup |
+---------+-----------------------+---------------------+-----------------------+------------------+
| 1 | myApplicationUser | ^SELECT.*FOR UPDATE | 10 | NULL |
| 2 | myApplicationUser | ^SELECT | 20 | 100 |
+---------+-----------------------+---------------------+-----------------------+------------------+
2 rows in set (0.00 sec)

The incoming queries that match the query rule, (in our example above, this is all queries as matching the regular expression ‘^SELECT’, for myApplicationUser, excluding queries matching ‘^SELECT.*FOR UPDATE’), will now be mirrored to the new cluster. You can verify this by checking the MySQL processlist on the new cluster.

The stats_mysql_query_digest table on ProxySQL holds statistics for the queries that are being processed by ProxySQL. To use the stats_mysql_query_digest table, the global variables mysql-commands_stats and mysql-query_digests must be set to true, which is the default.

Comparing query performance between two clusters

Query the stats_mysql_query_digest table to compare the performance per query between the current and the new cluster:

MySQL> select
 (b.count_star+a.count_star)/2 as count,
 cast(round(((b.sum_time + 0.0)/(b.count_star + 0.0))/((a.sum_time + 0.0)/(a.count_star + 0.0)),2)*100 as int) as percent,
 cast(round(((b.sum_time + 0.0)/(b.count_star + 0.0))/((a.sum_time + 0.0)/(a.count_star + 0.0)),2)*100 as int)*(b.count_star+a.count_star)/2 as load ,
 substr(a.digest_text,1,150)
from
 stats_mysql_query_digest a
inner join
 stats_mysql_query_digest b on
 a.digest = b.digest
where
 a.hostgroup = 10
 and b.hostgroup = 100
order by
 percent ASC;

In this example, the current production cluster has hostgroup 10, and the new mirror cluster was assigned hostgroup 100. The queries with a percentage above 100 are the queries that perform slower on the new cluster, and may be worth investigating, while the queries with a percentage below 100 are more performant on the new cluster. To investigate queries, you can compare the EXPLAIN plan of the query on the current and the new cluster. We use PMM Query Analytics to compare query analytics and the explain plan of the queries on the two separate clusters.

It is worth noting, that you should allow enough time for the MySQL buffer pool to get filled, before checking the stats_mysql_query_digest table. Otherwise, the query times on the new cluster can be skewed, as the active dataset may not yet be in memory (whereas on the current cluster it might be). Also, keep in mind that if you are mirroring only a subset of queries, the load on the new cluster will be different to the current cluster, and could affect the query performance on the new cluster, so that they appear significantly faster. Checking the execution plan of the query to see whether it has changed, is therefore more important than looking at overall load.

To conclude, using query mirroring to test queries on a new system, before making the migration, allows you to compare latency and query plans per normalised query, and proactively detect any necessary alterations before switching live traffic to the new cluster.

Release Roundup April 30, 2024

David Quilty — Tue, 30 Apr 2024 00:00:00 UTC

Percona software releases and updates April 17 - April 30, 2024.

Today’s post includes releases and updates that have been released since April 15, 2024. Take a look.

Percona Distribution for MySQL 8.3.0-1 (PS-based variant)

Percona Distribution for MySQL 8.3.0-1 (PS-based variant) was released on April 16, 2024. This release is based on Percona Server for MySQL 8.3.0-1 and merges the MySQL 8.3 code base. It introduces the following changes:

Percona updates the Binary Log UDFs to make them compatible with new tagged GTIDs (Global Transaction Identifiers).
PS-9044: Adds the following variables to MyRocks:
Changes the default values for the following variables:
- rocksdb_compaction_sequential_deletes from 0 to 14999
- rocksdb_compaction_sequential_deletes_count_sd from OFF to ON
- rocksdb_compaction_sequential_deletes_window from 0 to 15000
- rocksdb_force_flush_memtable_now from ON to OFF
- rocksdb_large_prefix from OFF to ON

Download Percona Distribution for MySQL 8.3.0-1 (PS-based variant)

Percona Server for MySQL 8.3

On April 16, 2024, we released Percona Server for MySQL 8.3. It includes all the features and bug fixes available in the MySQL 8.3 Community Edition in addition to enterprise-grade features developed by Percona. This release merges the MySQL 8.3 code base. Within this merge, Percona updates the Binary Log UDFs to make them compatible with new tagged GTIDs (Global Transaction Identifiers).

Download Percona Server for MySQL 8.3

Percona Distribution for MongoDB 7.0.8

Percona Distribution for MongoDB 7.0.8 was released on April 24, 2024. It is a freely available MongoDB database alternative that gives you a single solution that combines enterprise components from the open source community, designed and tested to work together. Bug fixes and improvements provided by MongoDB are included in Percona Distribution for MongoDB. Note: a number of issues with sharded multi-document transactions in sharded clusters of 2 or more shards have been identified that result in returning incorrect results and missing reads and writes. The issues occur when the transactions’ metadata is being concurrently modified by using the following operations: moveChunk, moveRange, movePrimary, renameCollection, drop, and reshardCollection. Please check the release notes for further information.

Download Percona Distribution for MongoDB 7.0.8

Percona Server for MongoDB 7.0.8-5

Percona Server for MongoDB 7.0.8-5 was released on April 24, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 7.0.8. A number of issues with sharded multi-document transactions in sharded clusters of 2 or more shards have been identified that result in returning incorrect results and missing reads and writes. The issues occur when the transactions’ metadata is being concurrently modified by using the following operations: moveChunk, moveRange, movePrimary, renameCollection, drop, and reshardCollection. Please check the release notes for further information.

Download Percona Server for MongoDB 7.0.8-5

Percona Bug Report: April 2024

Aaditya Dubey — Mon, 29 Apr 2024 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-9092: A query over an InnoDB table that uses a backward scan over the index occasionally might return incorrect/incomplete results when changes to the table (for example, DELETEs in another or even the same connection followed by asynchronous purge) cause concurrent B-tree page merges.

Reported Affected Version/s: 5.7.44, 8.0.35, 8.0.36

Upstream Bug: 114248

Workaround/Fix: Use descending indexes for the primary key. E.g.:

CREATE TABLE bugTest.testTable (key int unsigned, version bigint unsigned, rowmarker char(3) not null default 'aaa', value MEDIUMBLOB, PRIMARY KEY (key DESC, version DESC)) Engine=InnoDB;

PS-9107: A delete/insert into the secondary index is being change-buffered. This causes an insert into the ‘ibuf’ tree. There is a limit on the maximum size of the ibuf. So, on every ibuf insert, there is a compaction of ibuf tree (ibuf_contract). As part of ibuf_contract, we randomly open an ibuf page and apply the ibuf entries to the actual secondary index pages. After applying these ibuf entries from ibuf index, the ibuf tree goes on merging pages (optimistic vs pessimistic btree operations). To do this ibuf pessimistic delete on the tree, we save the cursor position, commit mtr and do a restore. This restore does a search and position again on the ibuf entry we were processing earlier, causing [MY-013183] [InnoDB] Assertion failure: ibuf0ibuf.cc:3833:ib::fatal triggered thread.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.37-29 [Yet to Release]

Upstream Bug: 114135

Workaround/Fix: Disable innodb_change_buffering

PS-9115: MySQL crashes due to getting a native index from get_mutex_cond in group replication, and before the crash, the following set of warnings/Errors is generated:

[Warning] [MY-011630] [Repl] Plugin group_replication reported: 'Due to a plugin error, some transactions were unable to be certified and will now rollback.'

[ERROR] [MY-011631] [Repl] Plugin group_replication reported: 'Error when trying to unblock non certified or consistent transactions. Check for consistency errors when restarting the service'

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is in progress and expected to be included in an upcoming release of Percona Servers.

Workaround/Fix: We can not guarantee that the bug will be avoided 100%, but shutting down/stopping group replication off-hours when the workload recedes should prevent the situation.

PS-9121: InnoDB updates the primary index but not the spatial index, which eventually corrupts the spatial index. The MySQL server crashes with “[ERROR] [MY-013183] [InnoDB] Assertion failure: row0ins.cc:268:!cursor->index->is_committed().” The update query changes the data from point(0.0000000000000099,0) to point(0.00000000000001, 0). When Innodb updates the record containing a spatial index, It updates the clustered index. This issue can be repeated using the following set of SQL statements.

CREATE TABLE a

 (

 id INT PRIMARY KEY,

 a GEOMETRY NOT NULL,

 SPATIAL KEY (a)

 )

ENGINE=InnoDB;

INSERT INTO a VALUES (1,POINT(0.0000000000000099, 0));

UPDATE a SET a = Point(0.00000000000001, 0);

DELETE FROM a WHERE id = 1;

INSERT INTO a VALUES (1,POINT(0.0000000000000099, 0));

Reported Affected Version/s: 8.0.36-28, 8.X [Innovative Release]

Fixed Version: The fix is in progress and expected to be included in an upcoming release of Percona Servers.

Upstream Bug: 114252

Workaround/Fix: Please consider resetting the shape to a different value from the new values.

PS-9109: The Percona server’s slow query rate is not accurately logged, which is controlled via the Log_slow_rate_type and Log_slow_rate_limit variables. Due to this issue, the slow query log records every query regardless of these variables’ values.

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

Percona Xtradb Cluster

PXC-4380: In a large cluster, for example, 15 nodes, in case one node has network issues and disconnects/re-connects/disconnects, the cluster might be sent to a non-primary state if the evs.install_timeout is reached.

Reported Affected Version/s: 5.7.42-31-65, 8.0.33-25, 8.0.35-27

Fixed Version: In such a big cluster, reaching consensus between nodes takes more time, so evs.install_timeout has to be adjusted. We can also configure the cluster to evict unresponsive nodes by setting evs.auto_evict=1. So, after investigation, we found that no fix is required, as everything works as expected/designed.

Workaround/Fix: Increasing the evs.install_timeout might fix the issue. The maximum value is 15S, which can still be reached depending on cluster size and how bad the flapping is.

The default value for evs.install_timeout is evs.inactive_timeout/2.

The minimum value is evs.join_retrans_period

The maximum time is evs.inactive_timeout + 1

So, If we keep defaults:

evs.join_retrans_period = 1s

evs.inactive_timeout = 15s

evs.install_timeout (default) = 7.5s

evs.install_timeout(min) = 1s

evs.install_timeout(max) = 16s

Please note that 15s is not a hard limit and is determined by evs.inactive_timeout

We have set wsrep_provider_options=“evs.install_timeout=PT15S” for all nodes in the test environment and are now unable to reproduce the issue. So, we are on some timeout boundaries, and the above configuration parameters were introduced to fine-tune in such environments.

PXC-4367: Innodb semaphore wait timeout failure seen after upgrade from 8.0.34 to 8.0.35. This issue is a possible side effect of this patch, Where PXC node acts as the async replica to some master and in parallel, the row is updated on PXC node and via replication, the PXC node hangs. To avoid the issue, upgrading to PXC 8.0.36 is recommended.

Reported Affected Version/s: 8.0.35-27

Fixed Version: 8.0.36-28

PXC-4363: Concurrent CREATE and DROP USER queries on different nodes lead to permanent lock.

| 16 | system user | db1 | Query | 411 | Waiting for table metadata lock | drop user IF EXISTS `msandbox_rw11`@`localhost` | 411380 | 0 | 

These queries cannot be cancelled or killed, and nodes refuse to be gracefully restarted. The shutdown is stuck with this forever. Only the forcible service kill helps restore the cluster.

2024-01-18T17:49:24.470428Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

PXC-4399: FLUSH TABLES during writes to the table with unique keys stall the cluster node; due to the stall, it’s not possible to abort/kill any of the above connections. The node sends a permanent flow control pause. The only way to get out of this stall is to kill the node.

Reported Affected Version/s: 8.0.33-25, 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

PXC-4418: In MySQL, the optimizer creates a temp table definition with indexes, where 2 of them have the same name (, , ). When the query is executed, a temp table is created, then MySql tries to access . In InnoDB, we search for index by name (dict_table_get_index_on_name()), which returns the wrong . Then row_sel_convert_mysql_key_to_innobase() crashes as structures are not aligned. Please note this bug affects only very complicated queries with many JOINs and subqueries. To repeat this bug, the internal temporary table needs to be created at least for two parts of the query.

Reported Affected Version/s: 8.0.32-24, 8.0.34-26

Fixed Version: 8.0.36-28

Percona Toolkit

PT-2190:The pt-show-grants use SHOW CREATE USER command to obtain grants from the MySQL server. By default, this query returns values as they are stored in the mysql.user table. When using caching_sha256_password, such hash of the password could contain a special character. Therefore, it would not be possible to use output printed by pt-show-grants to re-create users in the database. Since version 3.6.0, pt-show-grants checks if it runs against MySQL version 8.0.17 or higher and sets session option print_identified_as_hex to true before running SHOW CREATE USER command. This allows to print commands that could be used to re-create users.

Reported Affected Version/s: 3.5.1

Fixed Version: 3.6.0 [It is expected to be released soon]

PT-2215: pt-table-sync does not recognize the privileges in roles for MariaDB

Reported Affected Version/s: 3.5.2

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PT-2316: pt-config-diff with –pid option is broken with “Can’t locate object method “make_PID_file” via package “Daemon” at /usr/bin/pt-config-diff line 5522” on Ubuntu 20.04

Reported Affected Version/s: 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PT-2314: pt-online-schema-change fails due to duplicate constraint names when it attempts to make a table copy for alteration.

Reported Affected Version/s: 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

Workaround/Fix: Do not duplicate constraints name.

PT-2322: pt-mysql-summary does not detect jemalloc when installed as systemd.

Reported Affected Version/s: 3.5.6, 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PMM [Percona Monitoring and Management]

PMM-11583: On MySQL 8.0, innodb_redo_log_capacity supersedes innodb_log_files_in_group and innodb_log_file_size, which eventually breaks InnoDB Logging graphs. Due to this issue, the user can not determine whether the combined InnoDB redo log file size has to be increased or not.

Reported Affected Version/s: 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-13017: For certain db.collection.find(query, projection, options) queries, the Explain tab for QAN returns an error message saying, “error decoding key command: invalid JSON input; expected value for 64-bit integer.” Please note this issue specifically affects MongoDB monitoring.

Reported Affected Version/s: 2.35.0, 2.37.1, 2.41.1

Fixed Version: The fix is expected to be included in an upcoming release of PMM

PMM-12522: When adding data relatively large chunks of data to MongoDB sharded cluster, pmm-agent log starts flooded with “level=error msg="cannot create metric for changelog… & level=error msg="Failed to get database names:…” which eventually shows MongoS as disconnected in PMM UI (PMM-Inventory/Services).

Reported Affected Version/s: 2.39.0, 2.40.0, 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-12880: pmm-admin –tls-skip-verify does not work when x509 authentication is used.

Reported Affected Version/s: 2.41.0

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-12989: PMM agent logs flooded with wrong log entries when monitoring auth-enabled arbiters. Please note this issue specifically affects MongoDB monitoring.

Reported Affected Version/s: 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

Percona XtraBackup

PXB-3251: When PXB fails to load the encryption key, the xtrabackup_logfile is still created in the target dir. This causes a second attempt at running PXB to fail with a new error. The xtrabackup_logfile file contains data needed to run the –prepare process. The bigger this file is, the longer the –prepare process will take to finish. So, PXB should not create any files on disk until the encryption key is loaded.

Reported Affected Version/s: 8.0.35-30

Fixed Version: The fix is expected to be included in future releases of PXB

Percona Kubernetes Operator

K8SPG-496: When a PostgreSQL Database is set to a paused state via spec, the operator waits until all backups for the Database finish. After the backups finish, the PostgreSQL Database shall be paused, which is not happening.

Reported Affected Version/s: 2.3.0

Fixed Version: 2.3.1

K8SPG-494: High vulnerabilities found for pgbackrest, Postgres & pgbouncer package.

For pgbackrest and postgres: CVE-2023-38408

For pgbouncer: CVE-2023-32067

Reported Affected Version/s: 2.3.0

Fixed Version: 2.3.1

K8SPG-521: The upgrade path described in the documentation leads to disabled built-in extensions(pg_stat_monitor, pg_audit).

Reported Affected Version/s: 2.3.1

Fixed Version: 2.4.0 [It is expected to be released soon]

K8SPG-522: Cluster is broken if PG_VERSION file is missing during the upgrade from 2.2.0 to 2.3.1.

Reported Affected Version/s: 2.3.1

Fixed Version: 2.4.0 [It is expected to be released soon]

Workaround/Fix: In order to fix the issue, please do the following:

Create PG_VERSION with contents 14 in DB instance pod.

echo 14 > /pgdata/pg14/PG_VERSION
Apply crd and rbac

kubectl apply --force-conflicts --server-side -f crd.yaml

kubectl -n operatornew apply --force-conflicts --server-side -f rbac.yaml
Restart the operator deployment.

kubectl -n operatornew rollout restart deployment percona-postgresql-operator

At this moment, patronictl show-config starts to enlist extensions.

$ kubectl -n operatornew exec cluster1-instance1-gsvs-0 -it -- bash

Defaulted container "database" out of: database, replication-cert-copy, postgres-startup (init), nss-wrapper-init (init)

bash-4.4$ patronictl show-config

loop_wait: 10

postgresql:

 parameters:

 archive_command: pgbackrest --stanza=db archive-push "%p"

 archive_mode: 'on'

 archive_timeout: 60s

 huge_pages: 'off'

 jit: 'off'

 password_encryption: scram-sha-256

 pg_stat_monitor.pgsm_query_max_len: '2048'

 restore_command: pgbackrest --stanza=db archive-get %f "%p"

shared_preload_libraries: pg_stat_monitor,pgaudit

Now extensions appear in the \dx output without pod restarts, but all Postgresql servers will be restarted:

K8SPG-547: The pgbackrest container can’t use pgbackrest 2.50. This is because pgbackrest 2.50 requires libssh2.so.1, which requires epel. Without that fix, microdnf installs pgbackrest 2.48, which creates inconsistency with the Postgresql container.

Reported Affected Version/s: 2.2.0

Fixed Version: 2.4.0 [It is expected to be released soon]

Workaround/Fix: This patch can be used until it is fixed.

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References: