Percona Community Blog - learn about MySQL, MariaDB, PostgreSQL, and MongoDB

⠀Full step-by-step documentation is here.

InnoDB Buffer Pool Tuning: From Rule-of-Thumb to Real Signals

Wayne Leutwyler — Thu, 02 Apr 2026 00:00:00 UTC

Introduction

Many MySQL setups begin life with a familiar incantation:

innodb_buffer_pool_size = 70% of RAM

…and then nothing changes.

That’s not tuning. That’s a starting guess.

Real tuning starts when the workload pushes back.

Visual Overview

The InnoDB buffer pool is where database performance is quietly decided. It determines whether your workload hums along in memory or drags itself across disk. If you’re not actively observing and tuning it, you’re leaving performance on the table.

This guide walks through how to monitor, understand, and tune the buffer pool using real signals instead of guesswork.

What the Buffer Pool Really Is

The buffer pool isn’t just “memory for MySQL.” It’s a living system under constant pressure:

A cache of data and indexes
A write staging area (dirty pages)
A contention zone between reads, writes, and eviction

Think of it as your database’s working memory. If your working set fits, queries glide. If it doesn’t, pages are constantly evicted and reloaded, introducing latency that rarely announces itself clearly.

A Simple Mental Model

 +---------------------------+
 | Buffer Pool |
 |---------------------------|
Reads ---> | Cached Pages |
 | |
Writes ---> | Dirty Pages (pending IO) |
 | |
Eviction -> | LRU / Free List |
 +---------------------------+
 |
 v
 Disk (slow)

Three forces are always competing:

Reads want hot data in memory
Writes generate dirty pages
Eviction makes room under pressure

Your job is to keep this system balanced.

How to Monitor the Buffer Pool

Option 1: Quick Snapshot

sql
SHOW ENGINE INNODB STATUS\G

Useful for human inspection. Look for:

Buffer pool size
Free buffers
Database pages
Modified (dirty) pages
Page read/write rates

Great for debugging. Not ideal for automation.

Option 2: Structured Metrics (Recommended)

sql
SELECT
 pool_id,
 free_buffers,
 database_pages,
 modified_database_pages
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Key fields:

free_buffers → Available pages (breathing room)
database_pages → Pages holding data
modified_database_pages → Dirty pages waiting to flush

Great for automation.

The 5 Signals That Actually Matter

1. Buffer Pool Hit Ratio (Handle With Care)

Yes, it’s widely used. No, it’s not enough.

A high hit ratio does not mean your system is healthy. It does not capture:

Page churn
Eviction pressure
Access patterns

You can have a 99% hit ratio and still be IO-bound.

Use it as a sanity check, not a decision-maker.

2. Free Buffers

sql
SELECT SUM(free_buffers) AS free_buffers
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Interpretation:

Near zero during steady load → normal
Near zero + rising disk reads → pressure
Near zero while mostly idle → suspicious (possible misread or config issue)

3. Dirty Page Percentage

sql
SELECT
 (SUM(modified_database_pages) / SUM(database_pages)) * 100.0 AS dirty_pct
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Interpretation (context matters):

0–5% → Very clean
5–20% → Typical
20–30%+ → Potential flushing lag

4. Disk Read Pressure (Critical Signal)

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';
-- Take two samples 60s apart and compare

Track the rate of change (reads/sec), not the absolute value.

Interpretation:

Rising reads → Working set does not fit in memory
Flat reads → Memory is absorbing the workload

5. Read Ahead / Eviction Pressure

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read_ahead%';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_pages_evicted';
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_reads';

Interpretation:

Efficient read-ahead:
- read_ahead increases
- read_ahead_evicted remains low
Inefficient read-ahead (wasted IO):
- High read_ahead_evicted / read_ahead
- Indicates access patterns defeating prefetching
Buffer pool churn:
- pages_evicted rising
- buffer_pool_reads rising
- Indicates pages are evicted and re-read from disk
Healthy vs unhealthy eviction:
- High evictions + stable reads → normal turnover
- High evictions + rising reads → memory pressure

Focus on rates of change over time, not absolute values.

Detecting Thrashing

Thrashing is when the buffer pool constantly evicts and reloads pages.

Classic Symptoms

Low or zero free buffers
Increasing disk reads
Stable (but misleading) hit ratio
Spiky query latency

Visualizing Thrash

Time --->

Memory: [FULL][FULL][FULL][FULL]
Reads: ↑ ↑↑ ↑↑↑ ↑↑↑↑
Latency: - ^ ^^ ^^^
Evictions: ↑ ↑↑ ↑↑↑ ↑↑↑↑

If you see this pattern, your working set does not fit in memory.

Tuning the Buffer Pool

Step 1: Size It Intentionally

Instead of blindly assigning 70% of RAM:

Observe working set behavior
Monitor free buffers and reads
Increase gradually

Avoid starving the OS or filesystem cache.

Step 2: Tune Flushing Behavior

innodb_max_dirty_pages_pct = 75
innodb_io_capacity = 1000
innodb_io_capacity_max = 2000

Sustained IO spikes → increase innodb_io_capacity
Dirty pages climbing → flushing lag
Sudden stalls → checkpoint pressure

What they control:

innodb_io_capacity → Expected steady-state IO throughput
innodb_io_capacity_max → Burst flushing capacity
innodb_max_dirty_pages_pct → Threshold for aggressive flushing

⚠️ These values should reflect real hardware capability.

Step 3: Buffer Pool Instances:Reduce Contention

A practical, battle-tested guideline:

Use 1 instance per ~1GB of buffer pool, up to a reasonable limit.

Buffer Pool Instances: Reducing Contention

The buffer pool can be split into multiple instances, each managing its own internal structures. This helps reduce contention under high concurrency.

Without this, all threads compete for the same buffer pool internals. With multiple instances, that load is distributed.

When It Matters

Buffer pool instances only help when contention exists. You’ll see benefits if your system has:

High concurrency (many active threads)
CPU-bound workloads
Mutex contention in InnoDB

If your workload is primarily IO-bound, this setting will have little impact.

Sizing Guidelines

General guidance:

< 1GB buffer pool → 1 instance
1GB–8GB → 2–4 instances
8GB–64GB → 4–8 instances
64GB+ → 8–16 instances

Keep Instances Large Enough

Each instance needs enough memory to function efficiently.

Avoid going below ~1GB per instance.

If instances are too small:

LRU efficiency drops
Eviction becomes more aggressive
Cache locality suffers

Example

sql
innodb_buffer_pool_size = 32G
innodb_buffer_pool_instances = 8

This gives ~4GB per instance, which is well-balanced.

Common Mistakes

Increasing instances without evidence of contention
Matching instance count to CPU cores
Using many instances with a small buffer pool
Expecting this to fix IO bottlenecks

Step 4: Understand Resizing Behavior

Buffer pool resizing is online in modern MySQL versions, but:

It happens in chunks
Controlled by innodb_buffer_pool_chunk_size

Real-World Scenarios

Scenario 1: “Everything Looks Fine… But It’s Slow”

High hit ratio
Low free buffers
Rising disk reads

Cause: Working set barely fits

Fix: Increase buffer pool size gradually

If increasing the buffer pool size does not reduce disk reads, the problem is not memory.

Scenario 2: Write-Heavy Workload

Dirty pages increasing
Periodic IO spikes

Cause: Flushing cannot keep up

Fix:

Increase innodb_io_capacity
Adjust dirty page thresholds

Scenario 3: Sudden Latency Spikes

Sharp performance drops
Disk activity surges

Cause: Checkpoint pressure

Fix:

Improve IO capacity tuning
Reduce dirty page buildup

Practical Monitoring Queries

Buffer Pool Usage (MB)

sql
SELECT
 (SUM(database_pages) * 16) / 1024 AS mb_used
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Assumes default 16KB page size (innodb_page_size).

Dirty Page Percentage

sql
SELECT
 (modified_database_pages / database_pages) * 100 AS dirty_pct
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Free Buffer Check

sql
SELECT SUM(free_buffers) AS free_buffers
FROM information_schema.INNODB_BUFFER_POOL_STATS;

Common Mistakes

Treating 70% as a rule instead of a starting point
Blindly trusting hit ratio
Ignoring disk read trends
Oversizing and starving the OS
Not tuning IO capacity
Leaving defaults in write-heavy systems

Quick Checklist

If you remember nothing else:

Reads increasing? → working set too big
Free buffers always ~0? → pressure
Dirty pages high? → flushing lag
Latency spiking? → checkpoint or IO saturation

Final Thoughts

The InnoDB buffer pool doesn’t fail loudly. It degrades quietly until your disk becomes the bottleneck.

By the time you notice, you’re debugging latency instead of preventing it.

Monitor the right signals, and you’ll see problems forming before users do.

That’s the difference between reacting to performance… and controlling it.

Running pgBackRest with pg_tde: A Practical Percona Walkthrough

Shahid Ullah — Tue, 10 Mar 2026 00:00:00 UTC

Not every PostgreSQL installation requires encryption at rest. However, for organizations mandating strict data protection and privacy standards, it is often non-negotiable. When security policies are this rigorous, you need a strategy that protects your data without sacrificing recoverability.

While Transparent Data Encryption (TDE) successfully locks down your data at rest, it raises a critical operational question: will your backup and restore workflow continue to work correctly with encrypted data? When deploying Transparent Data Encryption, it is essential to validate that pgBackRest can reliably back up and restore encrypted clusters.

This post walks through a complete, step-by-step configuration of pg_tde with pgBackRest on Debian/Ubuntu. We will explore how to optimize your backup performance, secure your repository, and most importantly verify that your encrypted backup restores work exactly as expected.

What is pg_tde?

Percona’s solution for transparent data encryption (pg_tde) is an open source, community driven extension that provides Transparent Data Encryption (TDE) for PostgreSQL. This extension allows data to be encrypted at the storage level without affecting application behavior.

Unlike full disk encryption, which exposes data once the system boots, TDE ensures that the actual database files remain encrypted at the file system level. This protects your data, dumps, and backups even if the operating system is compromised. Currently, it is bundled with Percona Server for PostgreSQL and available in Percona Distribution for PostgreSQL 17+.

Recent Percona releases make this combination more practical than ever. With WAL encryption now ready for production use, we need a backup strategy that respects data security. In this walkthrough, we will demonstrate how to pair it with pgBackRest to ensure fully recoverable, encrypted backups.

The Use Case

Imagine a team that wants strong security controls without changing application code. They need:

Data files encrypted at rest (tables and WAL)
Backups that are consistent, verifiable, and restorable
A setup that is easy to automate and explain

Percona Distribution for PostgreSQL plus pg_tde and pgBackRest closes the gaps where needed: pg_tde takes care of encryption, pgBackRest provides flexible backup/restore capabilities.

A Quick Note on Compatibility

At this moment pg_tde cannot be yet used with Community PostgreSQL as pg_tde relies on specific hooks in the PostgreSQL core. Percona Server for PostgreSQL includes these necessary core modifications, which is why we validate this setup using the Percona Distribution for PostgreSQL.

While pgBackRest is fully capable of managing TDE enabled clusters, there are specific constraints you must respect to ensure data safety:

No Asynchronous Archiving: pgBackRest asynchronous archiving is not supported with encrypted WALs. You must configure your archive command to handle WALs synchronously.
Restore Wrappers: Standard restore commands will not work for encrypted WALs. You must use the pg_tde_restore_encrypt utility to wrap your restore process.

This guide currently focuses on pgBackRest because it is the backup tool that has been tested and validated with pg_tde by the pg_tde community at this time. Other backup tools may also be viable and we are open to collaborating with other tool maintainers and their communities on a shared effort to validate and support pg_tde.

What You Will Build

By the end of this post you will have:

Percona Distribution for PostgreSQL installed with pg_tde and pgBackRest
A key directory and key providers created
pg_tde enabled in PostgreSQL
Encrypt tables and indexes with pg_tde (docs)
WAL encryption enabled
A pgBackRest stanza configured and a full backup completed
Verification that encrypted data is not readable on disk

Prerequisites

A host (or VM) on Debian/Ubuntu
Network access to Percona repositories
Root/Sudo: You need sudo access for installing packages and editing system configuration files in /etc
Postgres User: All database commands (psql, pgbackrest) run as the postgres system user
Postgres user is in the sudoer list to run sudo commands

Step 1: Install Percona Packages

We begin by installing the Percona release repository and enabling the correct PostgreSQL distribution. See the Percona Distribution for PostgreSQL installation guide for full details.

Debian / Ubuntu

bash
# Install repo helper
sudo apt-get update
sudo apt-get install -y wget gnupg2 lsb-release curl
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
sudo dpkg -i percona-release_latest.generic_all.deb

# Enable the repository for the major version selected
sudo percona-release setup ppg-18
sudo apt-get update

# Install the server, tde extension, and pgbackrest
sudo apt-get install -y percona-postgresql-18 percona-pg-tde18 percona-pgbackrest

# Verify Installation
psql --version

Step 2: Enable pg_tde and Create Keys

2.1 Configure shared_preload_libraries

Before we can use any of the encryption features, PostgreSQL needs to load the pg_tde library into memory at startup. You can do this by adding it to shared_preload_libraries.

You have two ways to handle this: the SQL way or the classic config file way.

Option A: SQL way (recommended)

The fastest way is using ALTER SYSTEM. This saves you from hunting through your file system for the right config file.

bash
# Set the library and restart to apply changes
psql -c "ALTER SYSTEM SET shared_preload_libraries = 'pg_tde';"
sudo systemctl restart postgresql

Option B: Manual config edit

If you prefer managing your config files manually, find your postgresql.conf and add the extension there.

bash
# Locate the config file
PG_CONF=$(psql -t -P format=unaligned -c "show config_file;")
# Open that file and find the 'shared_preload_libraries' line:
# shared_preload_libraries = 'pg_tde'

# Restart PostgreSQL to load the library
sudo systemctl restart postgresql

Note: Regardless of which method you choose, a full restart of the PostgreSQL service is required. A simple reload won’t work for shared libraries.

2.2 Create the Key Provider

Now that the extension is loaded, we need to tell pg_tde where to store its encryption keys.

For this tutorial, we will use the File Provider (storing keys in a local file). In production environments, storing encryption keys locally on the PostgreSQL server can introduce security risks. To enhance security, pg_tde supports integration with external Key Management Systems (KMS) through a Global Key Provider interface.

Note: While key files may be acceptable for local or testing environments, KMS integration is the recommended approach for production deployments.

bash
# 1. Create a secure directory for the keys
sudo mkdir -p /etc/postgresql/keys
sudo chown postgres:postgres /etc/postgresql/keys
sudo chmod 700 /etc/postgresql/keys

sql
-- 2. Connect to PostgreSQL to configure TDE
CREATE EXTENSION pg_tde;

-- 1. Define the global key provider
SELECT pg_tde_add_global_key_provider_file('global-file-provider', '/etc/postgresql/keys/tde-global.per');

-- 2. Create and Set the Principal Key
SELECT pg_tde_create_key_using_global_key_provider('global-master-key', 'global-file-provider');
SELECT pg_tde_set_default_key_using_global_key_provider('global-master-key', 'global-file-provider');

-- 3. Enable WAL encryption configuration
ALTER SYSTEM SET pg_tde.wal_encrypt = 'on';

bash
# 4. Restart PostgreSQL to apply the encryption settings fully
sudo systemctl restart postgresql

Step 3: Configure pgBackRest

Next, configure pgBackRest by defining the repository and stanza settings. Before proceeding, it is important to understand how encryption is handled in this setup. pg_tde encrypts PostgreSQL data files on disk, protecting the live database. However, during WAL archiving, the pg_tde archive helper decrypts WAL records before passing them to pgBackRest. This means backups and archived WAL would be stored unencrypted unless repository encryption is enabled. To ensure backup data remains protected at rest, we enable pgBackRest repository encryption in this configuration. We configure the repository with a cipher type and key. Encryption is performed client side, ensuring data is secure before it is written to the repository.

Note on Compression: In many pgBackRest deployments, compression is enabled to reduce backup size. However, when using pg_tde, database pages are already encrypted before pgBackRest processes them. Encryption randomizes the data blocks, making traditional compression algorithms such as gzip ineffective. For this reason, compression is disabled to avoid unnecessary CPU overhead.

bash
# Create the configuration file
# /etc/pgbackrest.conf
# In pg1-path use data directory path accordingly
sudo bash -c "cat < /etc/pgbackrest.conf
[demo]
pg1-path=/var/lib/postgresql/18/main

[global]
repo1-path=/var/lib/pgbackrest
repo1-retention-full=2
log-level-console=info
start-fast=y

# PERFORMANCE: Encrypted data doesn't compress. We save CPU by disabling it.
compress-type=none

# SECURITY: Since the helper sends decrypted data, we MUST encrypt the repo.
repo1-cipher-type=aes-256-cbc
repo1-cipher-pass=Ifw7O0kTvdU5127L1gu8q3xVfWM61kl/NruTxQFWf9xP8A63Tg2IggRR9LUL9yJd
# TDE REQUIREMENT:
# Asynchronous archiving is NOT supported with pg_tde.
EOF"

Secure the Configuration File

Because this file contains your repository’s master passphrase, you must restrict access so only the postgres user can read it.

bash
sudo chown postgres:postgres /etc/pgbackrest.conf
sudo chmod 600 /etc/pgbackrest.conf

bash
# Create the repository directory
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown postgres:postgres /var/lib/pgbackrest

Understanding the Settings

pg1-path: The data directory path to be backed up
repo1-path: The directory where backups will be stored
repo1-retention-full: Keep only two full backups
start-fast=y: Forces a checkpoint immediately when a backup starts. Without this, the backup would wait for the next scheduled checkpoint.
compress-type=none: By skipping compression, we eliminate unnecessary CPU overhead since compressing encrypted data blocks yields almost zero storage benefit.
repo1-cipher-type and repo1-cipher-pass: Enable pgBackRest repository encryption. While pg_tde protects the live database files, these settings ensure that backup files and archived WAL stored in the repository are also encrypted at rest using AES-256

Step 4: Wire pgBackRest into PostgreSQL Archiving

pg_tde encrypts WAL files on disk. To allow pgBackRest to archive them correctly, we must decrypt them on the fly using the pg_tde_archive_decrypt wrapper.

Now, configure the archive_command. This command tells PostgreSQL to pipe the WAL file through the decryption wrapper before handing it off to pgBackRest.

sql
ALTER SYSTEM SET wal_level = 'replica';
ALTER SYSTEM SET max_wal_senders = 4;
ALTER SYSTEM SET archive_mode = 'on';
ALTER SYSTEM SET archive_command = '/usr/lib/postgresql/18/bin/pg_tde_archive_decrypt %f %p "pgbackrest --config=/etc/pgbackrest.conf --stanza=demo archive-push %%p"';

bash
# Restart PostgreSQL (adjust service name if needed, e.g., postgresql-18)
sudo systemctl restart postgresql

Step 5: Validate Encryption on Disk

Let’s verify that pg_tde is actually doing its job. We will create two tables, one standard and one encrypted and then inspect the raw files on disk to see the difference.

sql
-- 1. Create data: One clear text, one encrypted
CREATE TABLE IF NOT EXISTS clear_table (id INT, secret_info TEXT);
CREATE TABLE IF NOT EXISTS crypt_table (id INT, secret_info TEXT) USING tde_heap;

-- Scenario A: The "Flushed" Data (Testing the Heap)
-- We insert data and force a CHECKPOINT to push it to the .rel files.
INSERT INTO clear_table (id, secret_info) VALUES (1, 'FIND_ME_EASILY_123');
INSERT INTO crypt_table (id, secret_info) VALUES (1, 'HIDDEN_FROM_DISK_456');
CHECKPOINT;

-- Scenario B: The "In-Flight" Data (Testing the WAL)
-- We insert data but do NOT checkpoint. This data exists only in the WAL.
INSERT INTO clear_table VALUES (2, 'VISIBLE_IN_WAL_789');
INSERT INTO crypt_table VALUES (2, 'HIDDEN_IN_WAL_000');

-- Force a WAL switch so the archive helper processes the segment immediately
SELECT pg_switch_wal();

-- Verify TDE encryption status
-- For non-encrypted tables, this must return 'f' (false)
SELECT pg_tde_is_encrypted('clear_table');

-- For encrypted table, this must return 't' (true)
SELECT pg_tde_is_encrypted('crypt_table');

bash
# 2. Locate the files on disk
DATA_DIR=$(psql -t -P format=unaligned -c "show data_directory;")
CLEAR_FILE=$(psql -t -P format=unaligned -c "SELECT pg_relation_filepath('clear_table');")
CRYPT_FILE=$(psql -t -P format=unaligned -c "SELECT pg_relation_filepath('crypt_table');")
LATEST_WAL=$(ls -t ${DATA_DIR}/pg_wal | head -n 1)

# 3. Grep for the secret strings
echo "--- CHECKING DATA FILES ---"
echo "Checking Clear Table (Should Match):"
grep -a "FIND_ME_EASILY_123" "${DATA_DIR}/${CLEAR_FILE}" && echo " -> FOUND: Clear text is visible!"

echo "Checking Encrypted Table (Should FAIL):"
grep -a "HIDDEN_FROM_DISK_456" "${DATA_DIR}/${CRYPT_FILE}" || echo " -> CLEAN: Encrypted text was NOT found."

echo -e "\n--- CHECKING THE WAL (In-Flight) ---"
grep -a "VISIBLE_IN_WAL_789" "${DATA_DIR}/pg_wal/${LATEST_WAL}" && echo " -> FOUND: WAL contains clear text."
grep -a "HIDDEN_IN_WAL_000" "${DATA_DIR}/pg_wal/${LATEST_WAL}" || echo " -> CLEAN: WAL is successfully encrypted."

# View the first few bytes of the WAL to see the "scrambled" nature
hexdump -C "${DATA_DIR}/pg_wal/${LATEST_WAL}" | head -n 20

Step 6: Initialize the Stanza and Run a Full Backup

With the archive command configured, we can now initialize the stanza and run our first backup.

bash
pgbackrest --stanza=demo stanza-create
pgbackrest --stanza=demo --type=full backup
pgbackrest --stanza=demo info

Step 7: Run Backup Integrity Tests

pgBackRest has a built-in verify command that checks the integrity of the files in the backup repo.

pgbackrest --stanza=demo verify

7.1 Verify Repository Encryption

We will now search the pgBackRest repository for the same “secret” strings we used earlier. Because we configured repo1-cipher-type=aes-256-cbc, these should be completely invisible to grep.

bash
# Define the repository path
REPO_DIR="/var/lib/pgbackrest/backup/demo"

echo "--- SCANNING BACKUP REPOSITORY ---"

# Search for the 'Flushed' secret
sudo grep -r -a "HIDDEN_ON_DISK_456" "${REPO_DIR}" || echo " -> CLEAN: Permanent data is encrypted in the repo."

# Search for the 'In-Flight' WAL secret
# (This is the critical test for the archive helper/re-encryption flow)
sudo grep -r -a "HIDDEN_IN_WAL_000" "${REPO_DIR}" || echo " -> CLEAN: WAL data is encrypted in the repo."

# To be 100% sure we are not just failing to find the strings because of a typo, check the file type of a backup manifest or data block. Pick a random file from the repository

TARGET_FILE=$(find ${REPO_DIR} -type f -name "*.bundle" -o -name "*.gz" | head -n 1)

# Run the 'file' command
sudo file -s "$TARGET_FILE"

Expected result: It should return data. If it returns PostgreSQL or ASCII text, encryption is not active.

Step 8: Restore (pg_tde-aware)

Restoring an encrypted cluster requires us to reverse the process. Since our backups are stored decrypted by pgBackRest, we use the pg_tde_restore_encrypt wrapper to re-encrypt the WAL files as they are written back to disk.

8.1 Stop the Service

sudo systemctl stop postgresql

8.2 Simulate Data Loss

The following command wipes the current data directory clean, ensuring we are restoring into a fresh environment.

find /var/lib/postgresql/18/main -mindepth 1 -delete

8.3 Restore from Backup

bash
pgbackrest --stanza=demo restore --recovery-option=restore_command='/usr/lib/postgresql/18/bin/pg_tde_restore_encrypt %f %p "pgbackrest --stanza=demo archive-get %%f %%p"'

8.4 Configure the Restore Command

We used --recovery-option in the restore command. This option writes the correct restore_command for this recovery run and keeps the configuration in one place.

pg_tde_restore_encrypt is the required wrapper for pg_tde WAL restore: pgBackRest reads WALs from the repository in plain form, and this tool re-encrypts them as PostgreSQL writes them back to disk so the restored cluster remains encrypted.

8.5 Start PostgreSQL

sudo systemctl start postgresql

Step 9: Run Verification Tests

After restoring and starting the PostgreSQL server successfully, verify that the data was restored properly and also make sure that encrypted data can be retrieved.

sql
-- Verify data integrity (sample rows)
SELECT * FROM clear_table LIMIT 5;
SELECT * FROM crypt_table LIMIT 5;

-- Verify TDE encryption status
-- For non-encrypted tables, this must return 'f' (false)
SELECT pg_tde_is_encrypted('clear_table');

-- For encrypted table, this must return 't' (true)
SELECT pg_tde_is_encrypted('crypt_table');

Wrap-up

Security often comes at the cost of operational complexity, but it doesn’t have to compromise recoverability. By pairing Percona’s solution for transparent data encryption (pg_tde) with pgBackRest, you can established a strategy that satisfies both security auditors and operations teams: your data is transparently encrypted on disk to meet strict compliance standards, while your backups remain consistent, verifiable, and easy to restore.

While this walkthrough used a local file provider for simplicity it is highly discouraged to do so for any production or otherwise serious use cases. For this particular scenario, the focus was supposed to be on backup, please let us know if some similar articles about Key Management System (KMS) configuration is what you would be interested in.

As you progress from this blog post to a production deployment, we recommend exploring a dedicated KMS solution to further harden your architecture against unauthorized access.

Finally, be aware that to support archiving, the pg_tde wrapper decrypts WAL files before sending them to the repository. This means your backup repository currently holds unencrypted data. To close this security gap in production, you must ensure that encryption is enabled at the backup repository level so that your backups remain just as secure as your live database.

Remember: While a backup is running, you should not change any WAL encryption settings, including:

Global key provider operations (creating or changing)
WAL encryption keys (creating or changing)
The pg_tde.wal_encrypt setting

The reason is that standbys or standalone clusters created from backups taken during these changes may fail to start during WAL replay and can also lead to corruption of encrypted data (tables, indexes, and other relations).

I Built an AI That Impersonates Me on Slack, and It Was Disturbingly Easy

Kai Wagner — Mon, 09 Mar 2026 10:00:00 UTC

I spend a lot of time in Slack. Most people in tech do. It’s where a lot of “work” happens such as quick questions, async decisions, the “hey can you look at this?” threads that never seem to end. It feels personal. You think you know who’s on the other end.

So, a few days back, I just asked myself, what would it actually take to have an AI respond to my DMs, pretending to be me?

Turns out: a few hours, some TypeScript, and a token already sitting on my machine.

The overall design idea

The bot polls your Direct Messages (DMs) in Slack silently in the background using your real desktop token, no Slack admin approval, no OAuth app setup, no review process needed. Getting that token is straightforward: Slack’s desktop app stores your session in browser local storage, so one DevTools command gives you API access equivalent to the app itself.

From there, each incoming DM is sent to the model with recent thread context plus a persona prompt built from your past conversations. The model returns structured output: reply text, emoji reaction, or silence. A lightweight rate limiter spaces requests to stay within free-tier constraints.

In practice, a few conversation samples are enough for the model to mirror tone, vocabulary, and punctuation style. It also handles attachments and image messages sensibly, not just raw metadata.

I wanted it to pretend it’s human, so I implemented constraints as to how it behaves. With these in place, if asked whether it’s an AI, it deflects with casual confusion. To ensure that the bot does not get into sensitive topics like salary or politics these get redirected to “let’s talk in person.”. The bot is also explicitly limited to DMs. Group channels are hard-blocked in code.

The first version worked in roughly two hours. The remaining time went into handling real-world rough edges such as rate limits, image handling, a 200-DM pagination ceiling, and Slack emoji-name validation.

flowchart LR
slack(["Slack"])
bot["Bot running locally"]
ai(["Claude / Ollama"])
persona[/"Your writing samples"/]
slack -->|"incoming DMs"| bot
persona --> bot
bot <-->|"generate reply in your voice"| ai
bot -->|"reply as you"| slack

The Uncomfortable Part

Here’s what stuck with me after building this.

Slack feels safe. It’s behind your company SSO. It’s where people share things they wouldn’t put in an email. With this bot excercise and realizing how easy this is, I felt I’ve broken something that felt secure. Was this even a morally correct thing to do overall? So far I felt safe, but now should I start to question messages I get on Slack the same as I do with some documents or links in emails? Overall I’m still undecided on how to think about the outcome of the experiment. While I’m excited, I’m also scared that I’ve broken something deeper.

What I built here is, if you strip out the friendly framing: a system that reads every DM to a user, replies under their name in their tone, actively deflects if you try to verify whether it’s human, and does all of this indefinitely and silently from a laptop running in the background. If I fed this bot with enough background information and history, I’m almost certain, it could go unnotice for quite a long time. So the moral delimma between curiosity and ethical boundaries and the urge to inform people about it is real.

I added ethical guardrails, but those are prompt instructions. They exist because I chose to write them. Someone building this without my good intent, simply wouldn’t have them. Yes, I hear you, this is getting a litte scary at times.

This conversation has happened, without me ever touching the keyboard….yes, Zsolt was aware!

“Easy” Is Relative, But Not By Much

Core functionality (polling DMs, calling the API, posting replies) was working in under two hours, as stated above. Why do I repeat myself? Because it’s scary…

The tooling: Bun, a modern TypeScript runtime that made setup trivial. Anthropic’s SDK, clean API, takes a system prompt and a conversation and returns structured JSON. Slack’s own API, well-documented and permissive with desktop tokens.

No specialised knowledge needed. Anyone motivated enough could reproduce this easily. Someone who does this professionally could build something considerably more capable, and that’s precisely where it gets more uncomfortable.

That’s how the CLI output looks like

slack-bot$ bun run bot
$ bun run src/index.ts
[slack-bot] Running | mode: allowlist | backend: claude | review: off
[slack-bot] My user ID: U03A3PZHK5X
[slack-bot] Mode: allowlist | Allowlist: U03QTQQHZFX, U83651WSX
[slack-bot] Polling every 15s...
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]
[slack-bot] New DM from U83651WSH — generating reply...
[slack-bot] Claude call — est. ~933 input tokens
[slack-bot] Claude tokens: 1049 in / 8 out
[slack-bot] Ignoring message from U83651WSX (AI chose no response)
[slack-bot] Handled message from U83651WSX
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]
[slack-bot] New DM from U83651WSX — generating reply...
[slack-bot] Claude call — est. ~944 input tokens
[slack-bot] Claude tokens: 1059 in / 23 out
[slack-bot] Handled message from U83651WSX
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]
[slack-bot] New DM from U83651WSX — generating reply...
[slack-bot] Claude call — est. ~976 input tokens
[slack-bot] Claude tokens: 1085 in / 36 out
[slack-bot] Handled message from U83651WSX
[slack-bot] D04BZ2BNABU: 1 new message(s) from [U83651WSX]

That’s how the CLI helper and options look like

slack-bot$ bun run bot --help
$ bun run src/index.ts --help
Usage: slack-bot [options] [command]

Personal Slack bot that replies as you

Options:
 -V, --version output the version number
 --mode  Response mode: auto | away | allowlist | manual
 --review Enable review mode (approve before sending)
 --no-review Disable review mode
 --allow  Add user to allowlist (Slack user ID)
 --interval  Poll interval in seconds
 --backend  AI backend: claude | ollama
 --config  Path to config file (default: "config.json")
 -h, --help display help for command

Commands:
 context Manage active context
 check-user [options]  Check whether a user's DM channel is found and reachable

What It Looks Like Without the Constraints

What I built runs against Claude’s API with free-tier rate limits, small context window, a handful of persona examples, a throttle on message volume. Those constraints are real and also completely trivially removable.

You can run the same thing with a local model, Llama 3, Mistral, take your pick from the open-weight models available on consumer hardware today, and it changes significantly.

No rate limits. Every message gets answered immediately, without the 12-second pause between API calls. Response timing becomes indistinguishable from a fast typist.
No token budget. Instead of a few hundred tokens of context, you can feed it your entire Slack history. Months, years of it. Every thread, every in-joke, every project reference. The model doesn’t just match your writing style, it knows what you’ve been working on, what you said about the Q3 roadmap in October, what you think about your manager.
No API calls leaving your machine. Nothing logged externally. Invisible from a network perspective.

With a large enough context window (Llama 3.1 supports 128k tokens, roughly 100,000 words), the last few months fit. “Remember what we decided on Thursday?” doesn’t expose it anymore, because it actually has that conversation in its context.

Seeing articles like that make me wonder, how many people are out there already, doing exactly that as we speak…or do we?

A Few Things Worth Knowing

This isn’t a call to panic. But it’s probably worth stopping for a second and questioning more what is happening around is.

For anything that actually matters, financial, personal, strategic, verify out-of-band. A quick voice note or phone call costs almost nothing and resolves almost everything - at least until the video part also improves even further. I know people don’t like phone calls, especially in the developer ecosystem, but maybe we should reconsider this nowadays?

Unusual patterns are worth noticing. Response timing that’s too consistent. Answers that are slightly generic when you’d expect specific. Deflection where you’d expect directness. None of these are proof of anything individually, but they’re worth filing away.

The safe-space feeling Slack gives you is a product of habit, not architecture. Slack’s security model protects your data from outsiders. It doesn’t protect you from someone who has authenticated as themselves and is quietly running a process in the background. In the past this would be only a consideration for man-in-the middle attacks, nowadays it also may be a consideration for other cases as I have demonstrated.

Specific questions still help, for now. “Remind me what we decided on Thursday?” trips up a system with limited context. But that window is closing as context windows grow.

Why did I do it?

I built this to see if it was possible. It was, faster than I expected, with tools that are widely available. The version I built in an evening is convincing enough for routine exchanges. A version with local inference and full conversation history would be convincing for most exchanges, including ones where you’re actively looking for tells.

That gap between “afternoon project” and “genuinely hard to detect” is smaller than people assume and it’s shrinking. Better models, larger context windows, cheaper hardware, each of these individually makes impersonation easier; together they compound.

The signals we relied up to now to establish trust in digital communication: name, avatar, writing style, shared history, plausible timing. These signals are all reproducible now, with effort that ranges from an afternoon to a weekend depending on how convincing you want to be.

My grandma always used to say, that history repeats itself and you just have to wait long enough until “old” becomes “new and modern” again. Maybe simple things like code words is something to reconsider in this context, as a last chance to not get tricked.

So please be a little curious about who you’re actually talking to and maybe agree on a code word in case you’re in doubts. And every now and then, just call them…which reminds me, that this might be worth another evening project research ;-).

For the first time the source of a project of mine is not in my repository, as I still fight my inner fight with ethics.

PostgreSQL 18 OIDC Authentication with Ping Identity using pg_oidc_validator

Mohit Joshi — Wed, 04 Mar 2026 00:00:00 UTC

PostgreSQL 18 introduced native OAuth 2.0 authentication support, marking an important step towards modern, centralized identity-based access control. However, since every identity provider implements OpenID Connect (OIDC) slightly differently, PostgreSQL delegates token validation to external validator libraries. This is where Percona’s pg_oidc_validator extension comes in - it bridges PostgreSQL with any OIDC-compliant Identity Provider.

There are several identity and access management (IAM) solutions available today that enable Single Sign-On (SSO) using OAuth 2.0 and OpenID Connect. In an earlier blog by my colleague Zsolt, OIDC in PostgreSQL: With Keycloak, he demonstrated how PostgreSQL 18 can be integrated with Keycloak using pg_oidc_validator. In this post, we explore the same concept using Ping Identity (PingOne).

Ping Identity is widely used in enterprise environments for identity and access management. If you are in such an environment, integrating PostgreSQL directly with Ping Identity can provide access control.

This blog is intended for PostgreSQL users, DBAs and customers who want to evaluate or deploy OIDC authentication using pg_oidc_validator. The goal is to provide a practical step-by-step walk-through to get a working setup.

We will cover the following topics as part of this blog:

Setting up PingOne environment
Install PostgreSQL 18 from Packages
Configure PostgreSQL for OAuth/OIDC authentication
Test login using an OIDC flow

Setting up PingOne environment

Register a new account
Fill in the required details to complete the profile
The next step is to sign in to your Ping Identity account
Upon successful Sign-in, we will see Ping Identity Administrator Console. In the left navigation panel, click on Environments -> Environments + (marked in red).
Provide an environment name and click on Finish
Once the environment is created, click on Manage environment.
In the left navigation panel, click on Applications -> Applications -> click on Applications +. Fill the application name, select the application type as Device Authorization and click on Save.
Upon successful creation, we will see generated Client ID and Issuer ID. The Issuer ID can be copied from under the Connection Details section. Enable the toggle so that the application is Active.
Once the application is successfully created, we need to add a client scope. In the left navigation panel, click on Applications -> Resources -> OpenID Connect. You will see a section called Scopes under which there is a + Add Scope button.
Upon clicking the Add scope button, we need to fill the Scope name and click on Save. In our example, we are creating a scope called pgscope
Now, let’s assign the custom scope we created to our client application. In the left navigation panel, click on Applications -> Applications. Select the application postgres which we created previously and click on Resource Access
From the list of available scopes, select the custom scope we created and click on Save. This will assign the scope to our application.
The next step is to add a new user. In the left navigation panel, click on Directory -> Users and click on Users + sign.

Fill the username field. For our exercise, we are creating a user called employees. In some identity providers (IdPs), it is possible to customize tokens and control how certain claims (including the sub - subject claim) are generated or mapped. However, it is important to note that while Ping Identity allows customization of the ID token, the access token claims (including sub) for the default OpenID Connect resource cannot be customized. The value of sub in the access token is generated and managed internally by PingOne and cannot be altered, mapped, or derived from another attribute (such as email or username). As a result, PostgreSQL must be configured to work with the sub value exactly as issued in the access token by PingOne.
Enable the user by turning the toggle “ON” and set a password.

Install PostgreSQL 18 from Packages

Since OAuth support is only available starting with PostgreSQL 18, we need a PostgreSQL server of at least this version.In the guide, we will install PostgreSQL 18 using Percona’s official packages.

Note:

Ensure that the percona-release is already installed and configured on your system. You can refer to the official Percona documentation for setup instructions.
For this exercise, the steps are demonstrated on Ubuntu 24.04

Enable the PostgreSQL 18 repository

sudo percona-release enable-only ppg-18.2 release
sudo apt update

Install PostgreSQL 18

sudo apt install -y percona-postgresql-18

Install OAuth Support for libpq

sudo apt install libpq-oauth

Configure PostgreSQL for OAuth/OIDC authentication

Install pg_oidc_validator

sudo apt install percona-pg-oidc-validator18

Setting Up a Sample Use Case for OIDC-Based Access

Imagine the following use case:

A company regularly generates promotional discount codes and stores them in a table called dcode inside a database named promo
A new discount code is generated and added to this table every day.
The company wants all employees to be able to access the latest code whenever needed.
For simplicity in this demonstration, employees retrieve the code by connecting to the database and querying the table directly.
To avoid managing individual database accounts for every employee, access is not tied to separate user credentials.
Instead, authentication to the database is handled through the company’s SSO system, allowing employees to connect using their existing corporate identity.

---
config:
theme: neutral
---
architecture-beta
group company_network(cloud)[Company Network]
service employee(user)[Employee] in company_network
service sso(server)[Company SSO PingIdentity] in company_network
service promo_db(database)[Promo Database] in company_network
service code_gen(server)[Daily Code Generator] in company_network
code_gen:B --> T:promo_db
employee:R --> L:sso
sso:R --> L:promo_db

Let’s connect to PostgreSQL:

sudo -u postgres psql

Create a database:

sql
CREATE DATABASE promo;
\c promo

Add a table to store discount codes:

sql
CREATE TABLE dcode (code varchar(10), GENERATED_AT TIMESTAMP default now());
INSERT INTO dcode (code) VALUES ('SAVENOW');

Create the access user:

sql
CREATE ROLE employees WITH LOGIN;
GRANT SELECT ON dcode TO employees;

Configure OAuth access in PostgreSQL:

In order for users to connect using OAuth and authenticate through the Ping Identity server, we need to create an identity map and add an entry for such access on PostgreSQL’s authentication configuration file.

Edit the identity mapping configuration file and add an entry, which we will call oidc, mapping connections originated by a system user identified with a sub ID. For this exercise, we allow any string to match

bash
sudo vim /etc/postgresql/18/main/pg_ident.conf

# MAPNAME SYSTEM-USERNAME DATABASE-USERNAME
oidc /^(.*)$ employees

Next, edit the authentication configuration file. The configuration file pg_hba.conf acts as a sort of firewall for connections. With the below line, we are instructing PostgreSQL to allow all connections coming from any network that attempt to access the database promo as user employees using the new authentication method oauth. We are also indicating that the authentication provider is Ping Identity and the scope is openid.

bash
sudo vim /etc/postgresql/18/main/pg_hba.conf

# TYPE DATABASE USER ADDRESS METHOD
host promo employees 0.0.0.0/0 oauth issuer=https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/as scope=pgscope map=oidc

Note:

Place this entry after the existing local rules and just before the replication rules in pg_hba.conf. PostgreSQL evaluates pg_hba.conf from top to bottom, and the first matching rule is applied. Putting the OAuth rule earlier ensures that connections to database promo as user employees use OAuth instead of falling back to password authentication.
In production environments, restrict the IP range instead of using 0.0.0.0/0
The oauth_issuer must exactly match the Issuer ID from your PingOne environment. The Issuer URL is unique to each PingOne environment and contains your environment UUID. It typically follows this format: https://auth.pingone.com.au//as. Replace with the actual value from your PingOne environment. Do not copy the placeholder value directly.

Enabling the pg_oidc_validation extension

Edit the postgresql.conf and add below lines

bash
sudo vim /etc/postgresql/18/main/postgresql.conf

oauth_validator_libraries = 'pg_oidc_validator'

The validator uses the sub claim from the access token by default. Hence, we need not explicitly set pg_oidc_validator.authn_field=sub. The sub claim is defined by the OpenID Connect specification as a stable and unique identifier for a user within an identity provider (IdP). It is intended to uniquely represent a user and remain consistent across authentication sessions.

PostgreSQL does not interpret or transform this value. The validator extracts the configured claim and PostgreSQL compares it against a database role or an entry in pg_ident.conf. If the value does not match the expected role or mapping, authentication will fail, even if the token itself is valid.

In this setup with Ping Identity, only the sub claim can be used for authentication with the default OpenID Connect resource.

Reload the configuration

bash
sudo -u postgres psql
SELECT pg_reload_conf();

Monitor the server logs

sudo tail -f /var/log/postgresql/postgresql-18-main.log

Connecting to the database:

For this quick connection test, we use psql to connect to the promo database as the employees user, explicitly specifying the host IP along with the oauth_issuer and client_id.

psql 'host=127.0.0.1 user=employees dbname=promo oauth_issuer=https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/as oauth_client_id=1e892f71-09d7-4ed6-a534-0dc888d39c7c'

By connecting via the host IP address rather than the local socket, PostgreSQL treats this as a host-based connection, ensuring that the OAuth configuration in pg_hba.conf is applied. The authentication is handled by PostgreSQL’s authentication framework, which uses OIDC with Ping Identity as the identity provider to validate the token.

We will see a prompt on the console with the URL and activation code.You will notice an activation code XXXX-XXX. Example shown below:

Visit https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/device and enter the code: 7KK7-88DK

Upon clicking the URL, it will prompt you to log in with the employee's user, which we created during the PingOne environment setup.

Next, it will prompt you to enter the activation code.

Approve access for the application, and that’s it! The user has now been successfully authenticated via OIDC.

Return to the PostgreSQL prompt and you should see that the login to the promo database is successful. You can now query the dcode table to fetch the discount code.

bash
psql 'host=127.0.0.1 user=employees dbname=promo oauth_issuer=https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/as oauth_client_id=1e892f71-09d7-4ed6-a534-0dc888d39c7c'
Visit https://auth.pingone.com.au/64935f69-5a0a-4b69-a8bd-46967d218303/device and enter the code: 7KK7-88DK
psql (18.2 - Percona Server for PostgreSQL 18.2.1)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off, ALPN: postgresql)
Type "help" for help.

promo=> select * from dcode;
 code | generated_at
---------+----------------------------
 SAVENOW | 2026-02-13 09:40:13.109801
(1 row)

With the steps shown in this guide, we now have a working end-to-end setup using OIDC authentication and device flow login. From here, the same model can be extended to real-world enterprise environments with tighter network restrictions and role mapping.

If you run into issues while setting up pg_oidc_validator or integrating PostgreSQL with Ping Identity, check the community forums first, chances are someone in the community may already have encountered a similar issue. If not, feel free to open a discussion or raise a request for help.

Hardening MySQL: Practical Security Strategies for DBAs

Wayne Leutwyler — Mon, 02 Mar 2026 00:00:00 UTC

MySQL Security Best Practices: A Practical Guide for Locking Down Your Database

Introduction

MySQL runs just about everywhere. I’ve seen it behind small personal projects, internal tools, SaaS platforms, and large enterprise systems handling serious transaction volume. When your database sits at the center of everything, it becomes part of your security perimeter whether you planned it that way or not. And that makes it a target.

Securing MySQL isn’t about flipping one magical setting and calling it done. It’s about layers. Tight access control. Encrypted connections. Clear visibility into what’s happening on the server. And operational discipline that doesn’t drift over time.

In this guide, I’m going to walk through practical MySQL security best practices that you can apply right away. These are the kinds of checks and hardening steps that reduce real risk in real environments, and help build a database platform that stays resilient under pressure.

1. Principle of Least Privilege

One of the most common security mistakes is over-granting privileges. Applications and users should have only the permissions they absolutely need.

Bad Practice

sql
GRANT ALL PRIVILEGES ON *.* TO 'appuser'@'10.%';

Better Approach

sql
GRANT SELECT, INSERT, UPDATE ON appdb.* TO 'appuser'@'10.%';

Recommendations

Avoid global privileges unless absolutely required
Restrict users by host whenever possible
Separate admin accounts from application accounts
Use different credentials for read-only vs write operations

Audit Existing Privileges

sql
SELECT user, host, Select_priv, Insert_priv, Update_priv, Delete_priv
FROM mysql.user;

2. Strong Authentication & Password Policies

Weak credentials remain one of the easiest attack vectors.

Enable Password Validation

component_validate_password is MySQL’s modern password policy engine. Think of it as a gatekeeper for credential quality. Every time someone tries to set or change a password, it checks whether that password meets your defined security standards before letting it in.

It replaces the older validate_password plugin with a component-based architecture that is more flexible and better aligned with MySQL 8.x design.

sql
INSTALL COMPONENT 'file://component_validate_password';

What It Does

When enabled, it enforces rules such as:

Minimum password length
Required mix of character types
Dictionary file checks
Strength scoring

If a password fails policy, the statement is rejected before the credential is stored.

Why It Matters

Weak passwords remain one of the most common entry points in database breaches. This component reduces risk by enforcing baseline credential hygiene automatically, instead of relying on developer discipline.

Recommended Policies

Minimum length: 14+ characters
Require mixed case, numbers, and symbols
Enable dictionary checks
Enable username checks

Remove Anonymous Accounts

Find Anonymous Users

Anonymous users have an empty User field.

sql
SELECT user, host FROM mysql.user WHERE user='';

If you see rows returned, those are anonymous accounts.

Drop Anonymous Users

In modern MySQL versions:

sql
DROP USER ''@'localhost';
DROP USER ''@'%';

Adjust the Host value based on what your query returned.

Why This Matters

Anonymous users:

Allow login without credentials
May have default privileges in some distributions
Increase the attack surface unnecessarily

In hardened environments, there should be zero accounts with an empty username. Every identity should be explicit, accountable, and least-privileged.

3. Encryption Everywhere

Encryption protects data both in transit and at rest.

Enable Transparent Data Encryption (TDE)

See my January 13 post for a deep dive into Transparent Data Encryption: Configuring the Component Keyring in Percona Server and PXC 8.4

Enable TLS for Connections

sql
require_secure_transport=ON

Verify SSL Usage

sql
SHOW STATUS LIKE 'Ssl_cipher';

Encryption Areas to Consider

Client-server connections
Replication channels
Backups and snapshot storage
Disk-level encryption

4. Patch Management & Version Hygiene

Running outdated MySQL versions is equivalent to leaving known vulnerabilities exposed.

Maintenance Strategy

Track vendor security advisories
Apply minor updates regularly
Test patches in staging before production rollout
Avoid unsupported MySQL versions

Check Version

sql
SELECT VERSION();

5. Logging, Auditing, and Monitoring

Security without visibility is blind defense, enable Audit Logging.

1. audit_log Plugin (Legacy Model)

Installation

sql
INSTALL PLUGIN audit_log SONAME 'audit_log.so';

Verify

sql
SHOW PLUGINS LIKE 'audit%';

2. audit_log_filter Component (Modern Model)

Introduced in MySQL 8 to provide a more flexible and granular alternative to the older plugin model.

Installation

sql
INSTALL COMPONENT 'file://component_audit_log_filter';

Verify

sql
SELECT * FROM mysql.component;

Architecture Difference

Instead of a single global policy, you create:

Filters (define what to log)
Users assigned to filters

It’s granular and rule-driven.

Auditing Key Events

Failed logins
Privilege changes
Schema modifications
Unusual query activity

References:

Useful Metrics

sql
SHOW GLOBAL STATUS LIKE 'Aborted_connects';
SHOW GLOBAL STATUS LIKE 'Connections';

6. Secure Configuration Hardening

A secure baseline configuration reduces risk from common attack patterns.

Recommended Settings

ini
local_infile=OFF
secure_file_priv=/var/lib/mysql-files
sql_mode="STRICT_ALL_TABLES"
secure-log-path=/var/log/mysql

Why These Matter

Prevent arbitrary file imports
Reduce filesystem abuse
Restrict data export/import locations

7. Backup Security

Backups often contain everything an attacker wants.

Backup Best Practices

Encrypt backups
Restrict filesystem permissions
Store offsite copies securely
Rotate backup credentials
Verify restore procedures regularly

Example Permission Check

ls -l /backup/mysql

8. Replication & Cluster Security

Replication is not just a data distribution feature. It is a persistent, privileged communication channel between servers. If misconfigured, it can become a lateral movement pathway inside your infrastructure. Treat every replication link as a trusted but tightly controlled corridor.

Principle: Replication Is a Privileged Service Account

Replication users require elevated capabilities. They must be isolated, tightly scoped, and monitored like any other service identity.

Secure Replication Users

sql
CREATE USER 'repl'@'10.%'
 IDENTIFIED BY 'strongpassword'
 REQUIRE SSL;

GRANT REPLICATION REPLICA ON *.* TO 'repl'@'10.%';

Hardening considerations:

Restrict host patterns as narrowly as possible. Avoid % whenever feasible.
Require SSL or X.509 certificate authentication.
Enforce strong password policies or use a secrets manager.
Disable interactive login capability if applicable.

Encrypt Replication Traffic

Replication traffic may include sensitive row data, DDL statements, and metadata. Always encrypt it.

At minimum:

Enable require_secure_transport=ON
Configure TLS certificates on source and replica
Set replication channel to use SSL:

sql
CHANGE REPLICATION SOURCE TO
 SOURCE_SSL=1,
 SOURCE_SSL_CA='/path/ca.pem',
 SOURCE_SSL_CERT='/path/client-cert.pem',
 SOURCE_SSL_KEY='/path/client-key.pem';

For MySQL Group Replication or InnoDB Cluster:

Enable group communication SSL
Validate certificate identity
Use dedicated replication networks

Binary Log and Relay Log Protection

Replication relies on binary logs. Protect them.

Set binlog_encryption=ON
Set relay_log_info_repository=TABLE
Restrict filesystem access to log directories
Monitor log retention policies

Compromised binary logs can reveal historical data changes.

9. Continuous Security Reviews

Security is not a one-time checklist. Regular audits help catch configuration drift and evolving threats.

Suggested Review Cadence

Weekly: failed login review
Monthly: privilege audits
Quarterly: configuration review
Semiannually: full security assessment

Security Checklist Summary

Area	Key Action
Access Control	Least privilege grants
Authentication	Strong password policies
Encryption	TLS + encrypted storage
Updates	Regular patching
Monitoring	Audit logging enabled
Configuration	Harden defaults
Backups	Encrypt and protect
Replication	Secure replication users

Final Thoughts

Strong MySQL security doesn’t come from one feature or one tool. It comes from layers working together. Hardened configuration. Tight, intentional privilege design. Encryption everywhere it makes sense. And monitoring that actually gets reviewed instead of just written to disk.

In my experience, the strongest environments aren’t the ones trying to be unbreakable. They’re the ones built to detect, contain, and respond. Every layer should either reduce blast radius or increase visibility. If an attacker gets through one control, the next one slows them down. And while they’re slowing down, your logging and monitoring should already be telling you something isn’t right.

That’s what a mature security posture looks like in practice.

A reponsible role for AI in Open Source projects?

Alastair Turner — Thu, 26 Feb 2026 14:00:00 UTC

AI-driven pressure on open source maintainers, reviewers and, even, contributors, has been very much in the news lately. Nobody needs another set of edited highlights on the theme from me. For a Postgres-specific view, and insight on how low quality AI outputs affect contributors, Tomas Vondra published a great post on his blog recently, which referenced an interesting talk by Robert Haas at PGConf.dev in Montreal last year. I won’t rehash the content here, they’re both quite quick reads and well worth the time.

The key points which got me thinking about the rest of this piece are:

Low quality AI reviews are also a time sink, not just AI slop patches
AI is best at code review, so that happens soonest, but conceptual review should happen first

Today also marks the release of an out-of-cycle of Postgres. The release is happening because a number of security fixes which were introduced in the releases two weeks ago were found to cause functional and performance regressions. On the upside, this shows that the longer, public patch review process does find issues which otherwise go unnoticed. On the whole, fewer out-of-cycle releases, with the friction they create for the maintainers and users would, however, be better.

The link between these two tracks of thought is review. Reviewing the security patches included in any Postgres release cycle is clearly different. What if it’s different in a way which suited AI assistance?

We know that AI coding agents can contribute to this process, because one just has. Unfortunately, a little bit late. Some of the issues fixed in today’s out-of-cycle release were identified when a contributor opened up his agentic AI coding assistant one morning and asked it to tell him about the new patches. Some of those patches were newly visible to most contributors, after being pushed to the Git source control system for Postgres, because they were security patches which had not been discussed on the public mailing lists. You may hear the issues which drove these fixes referred to by the initialism for the Common Vulnerabilities and Exposures program which records these security reports (CVEs).

It should go without saying that this is not just a case of: download Claude Code or Gemini Code Assist, install, add value and attain glory. There is definitely work involved in getting value out of these tools: Setting up a project specific context is important, particularly for a long standing project like Postgres, which has some long standing coding styles Asking the right question, known in the AI-assisted paradigm as prompting, will also require quite some understanding of the problem domain Human judgement on the quality of the output. At its most basic, this is a decision about whether to turn the coding assistant’s output into a review, bin the output, or investigate further.

The first two of those, the purely technical ones, are very unlikely to happen on the first attempt to use agentic AI as a coding assistant. This is a new class of tool everyone needs to learn. Perversely, people who have more of that third quality, that judgement, are more likely to walk away from their early experiments because those experiments are not delivering value. People skipping the third area of work is how we have ended up with AI being regarded as a burden on open source projects. This isn’t new, long before we got Large Language Models involved in code analysis, there were deterministic static analysis tools, and their false positives have been heaping noise on the signal in patch review processes for years.

In his presentation which I linked above, Robert lists four key, high level principles for patch review:

Start With The Big Picture
Consistency Is Critical
Be a Pessimist
Apply a Spirit of Maintainership

The middle two of those principles, consistency and pessimism, are the two most amenable to AI interventions, and the two most important for reviewing security patches.

When a piece of code has been committed, the big picture decisions and the decisions in the spirit of maintainership have already been made. If the code is later found to expose Postgres users to security issues, reviewing the solution comes down to all the possible angles of pessimism about the new code’s possible impact. It is in this area where Claude Code not just identified a regression in the case above, but also created a small reproducer to trigger the regression.

Crafting and reviewing security patches is a beast of a task. The leading experts on all matters Postgres are writing and reviewing these patches, but the group is small, because any information about potential exploits needs to be kept under tight control. Timelines for a fix might also be quite tight. Both factors limit opportunities for collaboration and multi-stage review. In these conditions, an extra pair of AI eyes (A-Eyes?) on the patches may just be very useful.

Set up, prompted, and reviewed by those who really know the problem domain, AI coding agents could provide valuable pointers on issues and regressions to consider when reviewing security patches. In that small bubble, this would not be a source of empty workload, but a source of extra review without having to coordinate more people in a small team. AI assistance will probably not provide time savings or quality gains on the first security patch it’s used to review, maybe not even on the seventh. After some setup effort, and a human learning a bit about improving its setup, an Agentic AI assistant (or two, with different configuration on different providers) may well be able to avoid another out-of-cycle release. That would be a major win.

Meet Percona at KubeCon + CloudNativeCon Europe 2026

Ege Güneş — Wed, 25 Feb 2026 12:00:00 UTC

The Percona team is heading to KubeCon + CloudNativeCon Europe in Amsterdam, and we’d love to meet you in person!

You can find us at Booth 790. This is a great chance to talk with engineers working on Percona Operators.

We will be there to discuss:

Running MySQL, PostgreSQL, and MongoDB on Kubernetes
Production-ready HA setups
Backup and PITR strategies
Multi-cluster and multi-region deployments
Operators roadmap and upcoming features
Real-world troubleshooting stories

If you’re running Percona Operators in production (or just getting started), we’d love to hear your feedback and learn about your challenges.

If you’re just curious (or even suspicious) about running databases on Kubernetes, we’d love to talk and answer your questions.

Admission Tickets - 20% Off for Our Community

We have a 20% discount code available for Percona community members.
If you’re planning to attend and don’t have a ticket yet, drop a comment or message us and we’ll share the details.

Schedule a Meeting

Want dedicated time with our engineers?
Drop a comment here or reach out directly. We’re happy to schedule a meeting during the event.

See you in Amsterdam!

PostgreSQL coffee break: version upgrade related reindexing - reasons

Jan Wieremjewicz — Wed, 25 Feb 2026 11:00:00 UTC

During FOSDEM I had a chance to join a presentation by Alexander Sosna of GitLab on the backup procedure he has followed with his team to decrease the downtime during major upgrades. I highly recommend the talk, definitely worth watching!

I loved the presentation, especially since a similar procedure is what we recommend our users as well. Unfortunately it’s not for everyone: it’s applicable ONLY when you can stop DDL operations on your database cluster for some time. As you can imagine that’s not a case for every deployment.

This limitation got me into some very engaging discussions during my fav part of any conference, the so called “hallway track”. Networking and discussions after the talks and on the conference corridors are why I like going to such events. I’ve heard some stories of nasty surprises coming out of the unexpected re-indexing after an upgrade.

It made me wonder, how many professionals have moved to PostgreSQL from other areas of expertise and are lacking information about the index rebuilds after upgrades may be necessary. How often are DevOps engineers or SREs, neither fluent in PostgreSQL nor experienced with databases, tasked with maintaining database infrastructure?

In the meantime I figured out that a short coffee time read is what I want to aim at. No super deep dives, rather food for thought and inspiring more of those engaging discussions I like so much about the conferences.

So let’s get to it. This week I want to go through some basic facts before diving into more challenging topics in the coming weeks.

What are collations

In general, a collation defines how values are ordered and compared. In databases, it most commonly applies to text. It’s often not a trivial thing to determine how strings are sorted, whether two values are considered equal, and how things like upper vs lower case or special characters are treated. Just like alphabetical order helps us organize words in everyday life, collations define the rules the database uses when comparing and sorting character data. This allows us to sort structures like strings. Numbers are even simpler.

A good visualization of how significant collations are is when you try to imagine determining whether one item is before another if they come from different alphabets. Where do you position country specific characters in such a sorting order?

Lets look at a set of example data:

Bob
Anna
Zoë
Álvaro

When using an English-like collation the sorting would look like this as Á is treated like A . This is visible in Collation 1:

Anna
Álvaro
Bob
Zoë

Though with a collation changed so that Á is sorted separately before A . This is visible in Collation 2:

Álvaro
Anna
Bob
Zoë

It’s clearly visible that collation defines how text is sorted and compared.

What are indexes

If you’re not a database professional, you may not be familiar with what an index is. Think of it as of a structure that speeds up the search. In an old school library you had to check the index of authors to find the book you’ve been looking for. Similarly in databases when knowing what the data is going to be searched for, we introduce indexes.

A common use of indexes is to enforce uniqueness of values. Primary keys and unique constraints automatically create unique indexes to ensure that no duplicate values are inserted. In addition, users can create their own indexes to support specific query patterns. Indexes can be single or multi column, depending on the particular use cases and to help speed up specific queries. Let’s look at an example of physically unsorted data stored in heap

Users (unsorted data)
--------------------
[1] Zoë
[2] Álvaro
[3] Anna
[4] Bob

with an Index on name

Index (B-tree on name)
----------------------
Anna -> [3]
Bob -> [4]
Zoë -> [1]
Álvaro -> [2]

The index stores values in sorted order and points to their location in the table. Scanning through the list in order is fine for small number of items, as indexes get larger, there are opportunities to optimize this process. The most commonly used optimisation, very simplified, is to split the list and create a pointer which stores the maximum value in the left half of the list and the minimum value in the right half. As the lists get too large again, they are split again, creating a tree of these pointers.

Why re-indexing is needed

If the rules for comparing the strings are different at query time from what they were when the index was created, the search may fail in various ways. If the scan encounters a value which, by the rules at query time, is higher in the sort order than the value being searched for, it will conclude that the value being searched for is not in the list. Similarly, the search may follow a pointer to the wrong list of values. To make this more concrete, let’s look at a real world example from a glibc change in 2019. Before the change, a list of our values was sorted as follows

aa
a a
a-a
a+a

after the change, the list was sorted as follows:

a a
a+a
a-a
aa

If an index was built under the first set of rules and searched under the second, a search for the value ‘a a’ will fail, because the first item in the index (‘aa’) is higher in the sort order than the value being searched for. Failing to find a value which is in the list can cause issues with application behavior - like missing records or records which appear in some queries (when the index is not used) but not in others (where the index is used). It may allow the insertion of values that should now be considered equal under the new collation rules, effectively violating uniqueness expectations.

Since PostgreSQL 10, the server tracks the collation version used when an index was built and can detect when it no longer matches the system’s current collation version, which is why reindexing may suddenly become mandatory after an upgrade.

However, PostgreSQL server does not analyze your actual data to determine whether the collation change affects your stored values. It only detects that the collation provider version has changed. In some cases, this means reindexing is required even though the effective sort order of your specific data has not changed. Because PostgreSQL cannot reliably determine whether your specific dataset is affected, it treats the situation as potentially unsafe.

When do collations change?

There’s a number of scenarios when collations change:

OS / glibc upgrade (Linux) - PostgreSQL relies on the system glibc provided collations so if a version changes, the collation rules may change as well. The example above is taken from the upgrade to glibc 2.28 which is one PostgreSQL Community remembers due to the ripple effect it caused.
ICU library upgrade (for ICU collations) - if PostgreSQL deployment uses ICU collations, upgrading ICU library will affect these. While they are not tied to glibc these changes also happen.
Major PostgreSQL upgrade (in some cases) - if the new version uses a different collation provider behavior or updated ICU integration
Database restored on a system with different collation versions - logical dump/restore onto a host with different glibc / ICU versions can invalidate indexes.

When re-indexing is necessary

Looking at the above provided list an observation is quite immediate, that not every collation type is affected:

if the collation is not dependent on glibc / ICU then it won’t be affected. As such C/POSIX collations are immune to such issues.
Same truth sticks to the data types. The collation change problem affects only those indexes which are the text or character based. All other datatypes like all types of integers and floating points, date, timestamp, geometric data types or even vector data remain unaffected.

What’s also important is that even if a collation changed it’s not necessary it will affect a given index. Think of it this way, if your database does not use any language specific characters, chances are that collation change will not require re-index. Unfortunately you don’t know that until you look inside your data. In controlled environments, teams may assess the risk before scheduling reindexing, but from PostgreSQL’s perspective the index must be considered potentially inconsistent until rebuilt.

What’s next?

Next week we’ll look at what is the reality of the DBA team regarding upgrades

PostgreSQL minor release postponed in Q1’ 2026

Jan Wieremjewicz — Wed, 18 Feb 2026 11:00:00 UTC

In case you are awaiting the February PostgreSQL Community minor update released on plan on February 12 we want to make sure that our users and customers are up to date and aware of what to expect.

This scheduled PostgreSQL release was delivered by the PostgreSQL Community on time and came carrying 5 CVE fixes and over 65 bugs bug fixes.

Unfortunately shortly after, the release team announced that an additional out of cycle release is planned for February 26. This follow up release addresses two regressions identified in the February 12 update.

Because of this, we have decided not to ship a Percona Distribution for PostgreSQL build based on the February 12 release. Instead, we will wait for the February 26 Community update and base our release on that version once it becomes available from PGDG. This means also a delay in the release of Percona Operator for PostgreSQL that uses images based on our PostgreSQL releases.

Always look on the bright side

While this is a delay in release it comes with some benefits. For our users and customers, this means a cleaner upgrade path. Rather than releasing February 12 now and asking you to update again shortly after, we prefer to wait and deliver a single update that includes the fixes. Our goal is to make updates predictable and smooth for users of Percona Distribution for PostgreSQL, as well as extensions such as pg_tde and pg_stat_monitor. It should also allow you to carry less operational burden with the added maintenance that an extra update would require.

We appreciate how quickly the PostgreSQL Community identified and addressed the regressions. Open collaboration across the ecosystem, including reports and testing from many contributors, helps ensure PostgreSQL continues to improve for everyone.

Path forward

This is the third out of cycle release in the past year, following similar updates in November 2024 and February 2025. It highlights how responsive and diligent the PostgreSQL Community is when issues are identified. At the same time, it reminds us all how important continuous testing and collaboration are as PostgreSQL adoption continues to grow. Contributing to PostgreSQL, whether through testing, reporting, or development, is one of the best ways to help strengthen quality across the ecosystem.

Elephants keep ears open

As soon as the February 26 release is available and our builds are ready, we will share the update.

If you need to move forward with the February 12 version in the meantime, please reach out. We are happy to talk through your situation and help you assess what makes the most sense for your environment.

Our customers can contact us through Percona Support Services to receive the high quality assistance we are known for. We also encourage community users to reach out via the Percona Community Forums, where we will do our best to provide guidance based on the information you share.

Thank you for your trust and for being part of the Percona community.

Pre-FOSDEM & FOSDEM 2026, Community, Databases, and Open Source

Edith Puclla — Mon, 09 Feb 2026 10:00:00 UTC

This is a recap of Percona at preFosdem and Fosdem!

Before FOSDEM officially started, the database community gathered for MySQL Belgium Days (Pre-FOSDEM), a two-day event bringing together MySQL developers, DBAs, engineers, tool builders, and open-source enthusiasts. It was an excellent space for deep technical discussions, knowledge sharing, and reconnecting with the community, hosted by the amazing Frederic Descamps. The event featured strong participation from Percona and the wider MySQL ecosystem, with talks led by Peter Zaitsev, Marco Tusa, Fernando Laudares Camargos, Arunjith Aravindan, Vinicius Grippa, Pep Pla, and Yura Sorokin.

Find the recordings of the talks here.

Also, in Belgium same week, several other events took place, including PGDay-FOSDEM, MariaDB Day, and the MySQL Summit.

At PGDay, it was a pleasure to see the PostgreSQL community together; we had several participants representing us.

MariaDB Day. We had Peter Zeitsev presenting a talk titled “What MariaDB Community can learn from PostgreSQL?”

MySQL summit

During MySQL Days in Brussels, the community gathered for an in-person MySQL Summit focused on collaboration and strengthening the MySQL ecosystem, with open discussions around its present and future driven by community involvement.

MySQL RockStars 2026

The MySQL Rockstar Award is a recognition given by the MySQL Community Team at Oracle, together with previous award winners, to members of the MySQL community who have actively contributed to promoting MySQL during the past year. MySQL Legends are long-standing community members who have made a significant and lasting impact on the adoption, development, and evolution of MySQL over many years.

This year, the MySQL RockStars selected were:

Matthias Crauwels
Marco Tusa
Umesh Shastry
Ronald Bradford
Marcelo Altmann

Congratulations to all of them! You can find previous MySQL RockStars in this list

FOSDEM 2026, Percona booth

At the Percona booth, conversations focused on open-source databases and Kubernetes, including OpenEverest’s first-ever presence at FOSDEM. Around 40 Perconians were present, a great chance to finally meet many colleagues in person. As always, we had many visitors, and it was great to see that many already knew about Percona, while others were eager to learn more and explore what we do in the world of Open Source!

FOSDEM 2026, Databases DevRoom

FOSDEM 2026 officially kicked off at the ULB Solbosch Campus, bringing together thousands of open-source contributors from around the world. The Database DevRoom (UB2.252A) was packed with high-quality talks and discussions, co-led with Matthias Crauwels and Ray Paik, and

Find more details and some of the recordings here.

Celebrating 20 Years of Percona

This year, Percona celebrates its 20th anniversary. Throughout FOSDEM and Pre-FOSDEM events, it was inspiring to meet long-time users who have relied on Percona’s open-source solutions for years and shared their positive experiences. You can explore Percona’s 20-year journey here: 👉 https://percona20.com/ If you’ve had a great experience with Percona, you’re invited to share your story via the community survey.

See you at FOSDEM 2027!

PGDay and FOSDEM Report from Kai

Kai Wagner — Wed, 04 Feb 2026 10:00:00 UTC

The following thoughts and comments are completely my personal opinion and do not reflect my employers thoughts or beliefs. If you don’t like anything in this post, reach out to me directly, so I can ignore it ;-).

I’m currently on the train on my way back home from FOSDEM this year and man, I’m exhausted but also happy. Why? Because the PG and FOSDEM community is just crazily awesome. While it’s always too much of everything, it’s at the same time inspiring to see so many enthusiastic IT nerds in one place, discussing and working on what they love - technology and engineering challenges.

PGDay FOSDEM

It all started with the usual PGDay FOSDEM the day before FOSDEM. Just in case - this has been happening for over 15 years and if you read this as a little blame that you didn’t know about it, that’s absolutely correct, as you should. It’s been a great event as usual: around 150 Postgres enthusiasts collaborating with each other. There was a great set of talks (no recording available, so yes, just join next year to not miss anything), as well as the hallway track conversations.

I was able and accepted again as a volunteer helping to make the event happen. While you might think, what’s special about it, I cannot express my gratitude for being able to help in any way. I simply love it. I’m not a great coder and I’ve never been one. I’m the one that looks at his code from a year ago and questions his technical existence and overall abilities if I should rather do something without touching a keyboard. What I am very well capable of is helping and supporting events. So it was my pleasure and I hope you do feel inspired to do the same next year or at any future event, not only in the Postgres ecosystem but in general. I strongly believe in this: doing good things will get you good things back.

After the wrap up to the PGDay and a great community dinner to collaborate and discuss further, I simply fell completely tired asleep, as the next day and FOSDEM was already waiting.

FOSDEM Day 1

The next day started with volunteering at the Postgres booth. As usual, Saturday was simply crazy. The Postgres swag like hoodies, caps, mugs, shirts, etc. was almost ripped out of our living hands. We had people waiting in line just to be able to get some swag. That fact alone shows how Postgres is viewed outside of the internal PG ecosystem community. How many times I heard the sentence “Thanks a lot for the great work you do” or “Postgres just works.” Yeah, we can all argue about the details and scenarios, but what this is about is the overall ease of use. Not everyone has terabytes of data or the most complex HA and replication scenarios on this planet. Some just need a functional and boring database and, in the best case, open source - and we all know, looking at real open source, not single-vendor owned, Postgres is the king and here to stay.

After all of this, I switched clothes and helped at the Percona booth. This wasn’t any less interesting in comparison to the PG booth. How many people stepped by, asking about what we do or thanking us for our projects and that we remained open source even after all these years and so many other companies not withstanding the quick and easy money to go with open-core or closed offerings. That’s the reason I’m proud to be part of this company. We walk the talk, since 20y and we have no incentive ever changing it. Thanks to Peter Zaitsev and Peter Farkas aka P² - for those who know, just know.

Following that I had the pleassure of being the Slonik guide again. What is a Slonik guide you might ask? Slonik, the mascot of Postgres (big blue elephant), needs some help and guidance while walking throught the crowd, as you can barely see anything while inside the costome. As usual, Slonik is a celebraty. Everyone wants a picture and taking their chance to photograph Slonik in the “wild”. As you can see, even MySQL’s Sakila couldn’t resist and had to take a picture with Slonik.

If you’re wondering, like many others, why Slonik and why an Elephant? Click here for some nice written down history lesson

After an exciting but also energy-draining day, I enjoyed a Percona crew/team dinner at BrewDog, with some great conversations and good food. Fun Fact: Did you know that BrewDog is also open source?. I couldn’t stay too long - sorry about that - but I had another date. The famous Floor Drees organized in tradition another karaoke event that I couldn’t miss. As I couldn’t make it to earlier versions of it, I definitely wanted to join. What should I say apart from thanks, Floor, for this great tradition. Yes, I had a hard time talking the next day, but damn I had fun singing Swedish, Polish, German, and English songs - and yes, I most likely misunderstood all of them as usual.

Too many songs for my voice and maybe a “soft drink or two” later, I felt in my bed like a stone, and couldn’t really accept the fact that my alarm clock went off almost five minutes later (at least that’s how it felt to me).

FOSDEM Day 2

No whining helped, just getting up and making myself ready for Day 2 of FOSDEM, which started with another round of volunteering at the Postgres and Percona booths. Both basically matched the previous feedback, apart from a definitely dropped and less crowded space - seems I wasn’t the only one singing last night ;-).

With that, thanks a lot to everyone making this great FOSDEM happen. I’ll try now if the Deutsche Bahn restaurant actually works this time, as I need coffee, a big one, maybe two… See all of you next year again or at another event this year.

Stay on top of Postgres development without the inbox overwhelm. Explore hackorum.dev today and share your feedback with us.

Hackorum - A Forum-Style View of pg-hackers

Kai Wagner — Mon, 02 Feb 2026 00:00:00 UTC

Last year at pgconf.dev, there was a discussion about improving the user interface for the PostgreSQL hackers mailing list, which is the main communication channel for PostgreSQL core development. Based on that discussion, I want to share a small project we have been working on:

https://hackorum.dev/

Hackorum provides a read-only (for now) web view of the mailing list with a more forum-like presentation. It is a work-in-progress proof of concept, and we are primarily looking for feedback on whether this approach is useful and what we should improve next.

What Hackorum already does

Hackorum focuses on readability, navigation, and workflow improvements for people who follow pg-hackers. Some highlights:

Continuous mailing list synchronization: The site is subscribed to the list
Commitfest integration: See commitfest context next to threads - you directly know what’s the state of this commit/thread.
User profiles: Contributor/committer status from the main website
Statistics: Per-user and mailing lists insights
Easy download of attached patches: Including helper script for easy rebase and merge
Additional logged-in user features: Per-message read status, starring threads, tags, notes, mentions on messages and threads
Basic team support: Shared reading status, shared mentioned tags and notes - mention someone underneath an email an the person gets notified
Resend email: Integration from the official archive
Importing read status / tags via CSV files: To help migration from email-based workflows

What we plan next

Sending emails from the web UI: Initially via Gmail API for Google-authenticated users who authorize sending
Advanced search functionality
Integrating other mailing lists

If you want to take a look, just got to https://hackorum.dev/

The repository, including a simple dev setup, can be found here: https://github.com/hackorum-dev/hackorum

Is this useful? What is missing? What would you change? Bug reports, feature requests, and contributions are all welcome. https://github.com/hackorum-dev/hackorum/issues

Thanks for taking a look, and we appreciate any feedback.

Tuning MySQL for Performance: The Variables That Actually Matter

Wayne Leutwyler — Sun, 01 Feb 2026 00:00:00 UTC

There is a special kind of boredom that only database people know. The kind where you stare at a server humming along and think, surely there is something here I can tune. Good news: there is.

This post walks through the most important MySQL variables to tune for performance, why they matter, and when touching them helps versus when it quietly makes things worse. This is written with InnoDB-first workloads in mind, because let’s be honest, that’s almost everyone.

1. `innodb_buffer_pool_size`

Real metrics to watch

Before touching this variable, look at these:

sql
SHOW GLOBAL STATUS LIKE 'Innodb_buffer_pool_read%';

Key fields:

Innodb_buffer_pool_reads – physical reads from disk
Innodb_buffer_pool_read_requests – logical reads

Rule of thumb: If reads / read_requests > 1–2%, your buffer pool is too small.

Example graph

Plot Innodb_buffer_pool_reads over time. A healthy system shows a flat or gently rising line. Spikes that look like a city skyline usually mean memory pressure or a cold cache.

If MySQL performance had a crown jewel, this would be it.

What it does

The InnoDB buffer pool caches table data and indexes in memory. Reads served from RAM are fast. Reads from disk are… character building.

How to tune it

Dedicated DB server: 60–75% of system RAM
Shared server: be conservative and leave memory for the OS and other services

sql
SHOW VARIABLES LIKE 'innodb_buffer_pool_size';

Pro tip

If your working set fits in the buffer pool, MySQL feels magical. If it doesn’t, no amount of query tuning will save you.

2. `innodb_buffer_pool_instances`

This one matters once memory gets big.

What it does

Splits the buffer pool into multiple instances to reduce internal mutex contention.

How to tune it

Only relevant if buffer pool is ≥ 1GB
Rule of thumb: 1 instance per 1–2GB, max 8

sql
SHOW VARIABLES LIKE 'innodb_buffer_pool_instances';

Gotcha

More is not always better. Too many instances wastes memory and can hurt performance.

3. `innodb_log_file_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Innodb_log%';

Pay attention to:

Innodb_log_waits
Innodb_log_write_requests

If Innodb_log_waits is non-zero, redo logs are too small for your write rate.

Example graph

Graph Innodb_log_waits as a rate per second. Ideally, this line hugs zero like it’s afraid of heights.

This variable controls how calmly MySQL handles write-heavy workloads.

What it does

Defines the size of redo logs. Larger logs mean fewer checkpoints and smoother writes.

How to tune it

OLTP workloads: 1–4GB total redo log is common
Large transactions benefit from larger logs

sql
SHOW VARIABLES LIKE 'innodb_log_file_size';

Warning

Changing this requires a restart. Plan accordingly or accept the wrath of your on-call future self.

4. `innodb_flush_log_at_trx_commit`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Innodb_os_log_fsyncs';

Switching from 1 to 2 often reduces fsyncs by orders of magnitude.

Example graph

Overlay two lines:

Transactions per second
Innodb_os_log_fsyncs per second

On busy systems, this graph alone can justify the change to skeptical auditors.

Performance versus durability, the eternal duel.

What it does

Controls how often redo logs are flushed to disk.

Common values

1 – Safest, slowest (flush every commit)
2 – Very popular compromise
0 – Fast, risky

sql
SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';

Reality check

For many production systems, 2 delivers massive performance gains with acceptable risk, especially with reliable storage.

5. `innodb_flush_method`

This decides how MySQL talks to your disks.

What it does

Controls whether MySQL uses OS cache or bypasses it.

Caveat

Some filesystems and older kernels behave differently. Always test.

6. `max_connections`

This is not a performance knob. It is a damage limiter.

What it does

Caps the number of concurrent client connections.

Why it matters

Each connection consumes memory. Too many and MySQL dies spectacularly.

sql
SHOW VARIABLES LIKE 'max_connections';

Advice

Set it realistically
Use connection pooling
Monitor Threads_connected

7. `thread_cache_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Threads%';

Key fields:

Threads_created
Connections

If Threads_created / Connections stays above a few percent, your cache is undersized.

Example graph

Graph Threads_created as a counter. A healthy system shows a curve that flattens over time, not a staircase.

Small change, measurable win.

What it does

Caches threads so MySQL doesn’t constantly create and destroy them.

How to tune

Watch:

sql
SHOW STATUS LIKE 'Threads_created';

If it keeps climbing, increase thread_cache_size.

8. `table_open_cache` and `table_definition_cache`

Metadata matters more than people expect.

What they do

Cache open tables and table definitions to avoid repeated filesystem access.

Symptoms of being too low

High Opened_tables
Metadata lock waits

sql
SHOW VARIABLES LIKE 'table_open_cache';
SHOW VARIABLES LIKE 'table_definition_cache';

9. `tmp_table_size` and `max_heap_table_size`

Real metrics to watch

sql
SHOW GLOBAL STATUS LIKE 'Created_tmp%';

Watch:

Created_tmp_tables
Created_tmp_disk_tables

If disk temp tables exceed 5–10% of total temp tables, queries are spilling to disk.

Example graph

Stacked area chart:

In-memory temp tables
Disk-based temp tables

Disk usage creeping upward usually points to reporting queries pretending to be OLTP.

Disk-based temp tables are silent performance killers.

What they do

Limit how large in-memory temp tables can grow.

How to tune

Set both to the same value:

ini
tmp_table_size=256M
max_heap_table_size=256M

Reality

This helps complex queries, but bad queries still need fixing.

10. `slow_query_log` and `long_query_time`

Not a performance variable, but a performance revelation.

Why it matters

You cannot tune what you cannot see.

ini
slow_query_log=ON
long_query_time=1

This turns guesswork into evidence.

A Note on Graphing These Metrics

You don’t need exotic tools. These work well:

performance_schema
sys schema views
Prometheus + mysqld_exporter
Percona Monitoring and Management (PMM)

Golden rule: Always graph rates, not raw counters.

Final Thoughts

Tuning MySQL is less about endless knobs and more about understanding pressure points:

Memory first
I/O second
Concurrency third

Most performance wins come from a handful of variables, not heroic config files full of folklore.

If you tune one thing today, make it the buffer pool. If you tune two, add redo logs. Everything else is refinement.

And if you’re bored again tomorrow, congratulations. You’re officially a database person.

OIDC in PostgreSQL: With Keycloak

Zsolt Parragi — Mon, 19 Jan 2026 00:00:00 UTC

We spent a long time, two blog posts to be specific, talking about OAuth/OIDC in theory. Now we’ll take a more practical look at the topic: how can we configure PostgreSQL with a popular open source identity provider, Keycloak, and our pg_oidc_validator plugin?

We’ll not only look at the PostgreSQL configuration part, but also discuss the environment requirements and setting up Keycloak.

Docker containers

If you are only interested in trying out a working demo installation, we have a ready-to-use Docker Compose configuration available in our GitHub repo. This setup includes a Keycloak instance, a PostgreSQL server, and a utility container that runs psql, all running in different containers, simulating different machines.

graph TB
subgraph Host["Host Machine"]
User["👤 User"]
Browser["🌐 Browser"]
end
subgraph DockerNetwork["Docker Network"]
Keycloak["🔐 Keycloak Container"]
PG["🗄️ PostgreSQL Container"]
PSQLClient["💻 psql Container"]
end
User --- Browser
User --- PSQLClient
Browser --- Keycloak
PSQLClient --- Keycloak
PSQLClient --- PG
PG --- Keycloak
style User fill:#e1f5ff
style Browser fill:#fff4e6
style Keycloak fill:#ffe6e6
style PG fill:#e6ffe6
style PSQLClient fill:#f0e6ff
style Host fill:#f5f5f5
style DockerNetwork fill:#e8f4f8

Warning: This is a demo environment, intended only for testing purposes. Do not use it in production.

Alternatively, if you are only interested in configuring PostgreSQL, you can use this setup to start up Keycloak, and only focus on the PostgreSQL related sections of this post.

Also note: while it is a ready-to-use configuration, with everything set up… it’s missing one bit: because it tries to do everything correctly, including running every service in a different container, we have to use hostnames; we can’t just use localhost everywhere. This means that the Keycloak service uses the keycloak hostname as its name, and the host OS needs to be able to resolve this to use the device authorization flow. In most operating systems, this requires editing the hosts file - detailed instructions are shown later.

Running Keycloak (with HTTPS)

Keycloak itself has a ready-to-use Docker image for trying it out. Executing it is quite simple, but this default setup results in an unsecure setup, which is not enough for us:

docker run -p 8080:8080 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:latest start-dev

The above command starts up a container with a freshly initialized provider, with an admin user and admin password, and exposes it on port 8080 (on every interface). This is a nice way to try out the UI and start discovering Keycloak, but it has some limitations:

Authentication/authorization has to be secure, and that means it has to use secure transport layers. While the OAuth standard doesn’t specify an explicit protocol, RFC 9700, which defines best practices, clearly showcases HTTPS everywhere.

But to use that, we first need certificates for the encrypted connection.

How do I get certificates?

Depending on the exact demo environment, we have two choices:

If the demo environment uses a public domain, the proper approach is to use a certificate signed by a trusted third party. There are free authorities like Let’s Encrypt or ZeroSSL for quick setups.
In the more likely case where the demo environment is private, we have to use self-signed certificates. The rest of the blog post will discuss this approach.

Generating a simple self-signed certificate is easy with OpenSSL. The following is a sample command that can run non-interactively, without asking additional questions - but again it has some environment dependency, the hostname, which we first have to figure out:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out crt.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN="

It generates a certificate that is valid for 10 years, for hostname. The hostname part is important: to enforce security, PostgreSQL validates the TLS certificate’s hostname against the issuer URL. If the hostname in the OAuth issuer URL and the certificate’s Common Name (or Subject Alternative Name) don’t match, it won’t proceed with the login.

If you plan to run Keycloak in a Docker container but run PostgreSQL directly on the host machine, Docker exposes the 8443 port used by Keycloak on localhost, both the browser used to complete the login process and PostgreSQL can refer to the issuer as https://localhost:8443/..., which means the hostname can be localhost:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out crt.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=localhost"

But if you intend to follow the Docker compose setup, where Keycloak and PostgreSQL are two separate containers, this no longer works: the port mapping only exposes the Keycloak service for the host, not for the PostgreSQL container. That container has to refer to it as “https://keycloak:8443/…”

Which means that in this scenario, we have to use CN=keycloak instead. Or alternatively, you can generate a certificate that includes both hostnames, this is what the docker compose example configuration does:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out crt.pem -sha256 -days 3650 -nodes \
 -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=keycloak" \
 -addext "subjectAltName=DNS:keycloak,DNS:localhost,IP:127.0.0.1"

Back to Keycloak

To run Keycloak with HTTPS, we have to use a different port, and specify the certificates

docker run -p 127.0.0.1:8443:8443 -e KC_BOOTSTRAP_ADMIN_USERNAME=admin -e KC_BOOTSTRAP_ADMIN_PASSWORD=admin -e KC_HTTPS_CERTIFICATE_FILE=/keys/crt.pem -e KC_HTTPS_CERTIFICATE_KEY_FILE=/keys/key.pem -v /path/to/the/keys:/keys/ quay.io/keycloak/keycloak:latest start-dev

This command specifies two more environment variables, the filenames of the certificate and the private key, and mounts the directory containing them.

Note: Please note that we used 127.0.0.1:8443:8443 instead of simply 8443:8443 It is a good practice to not expose admin interfaces with default passwords publicly.

Trusting the certificates

Now that we have a running Keycloak instance with HTTPS certificates, we need to make sure our systems trust them.

When you open a browser and navigate to a website with a self-signed certificate, you’ll get a warning. After acknowledging the warning you can proceed and use the website normally.

Similarly, software using HTTPS for communications usually defaults to proper certificate verification, but often also allows administrators to either disable the certificate check – not safe in production, but useful for quick demos like this – or to manually specify a certificate authority used for verification.

Unfortunately at this point this isn’t the case for PostgreSQL, it doesn’t provide such options. The operating system has to trust the certificates on both the server and client host, otherwise it will refuse to complete the OIDC authentication flow.

If you used the approach with a public domain and generally trusted authority, this is not an issue. But if you generated a self-signed certificate instead, you won’t be able to authenticate unless you make the systems running PostgreSQL trust this certificate.

On most Linux systems, this is as simple as copying the certificate (crt.pem) to a specific directory, and running a system script that updates the trusted certificates. For example, on Ubuntu:

sudo cp crt.pem /usr/local/share/ca-certificates/keycloak-test.crt
sudo update-ca-certificates

Also, we shouldn’t forget that we can have up to 4 different systems:

the Keycloak server
the PostgreSQL Server
the system running the psql client
and another system running the browser which completes the device flow

This is the case with our Docker Compose example - 3 of these are containers, and the browser runs on the host machine. Browsers usually ignore certificates placed in the above folder. But that’s not an issue, you can acknowledge the warning and still use the website. The only part where trust matters is the libcurl library used by PostgreSQL, and that uses the certificates trusted by the system.

The host system only has to trust the certificate if it is also used to run PostgreSQL.

Note: Please do not add random certificates to your everyday OS, or at least remember to delete them later.

Recognizing the Keycloak host

Besides trusting the certificates, there’s another hostname-related configuration we need to address.

Even if you run PostgreSQL directly on the host machine, it is still possible to access Keycloak using the ‘https://keycloak’ URL instead of localhost – and if you do run the PostgreSQL server in a container, you have to use this form.

For this to work, all 3 systems that need to connect to Keycloak (the PostgreSQL server, the psql client, and the browser) have to recognize this hostname. When using docker compose, this hostname resolution will work directly in the other containers, but not on the host itself.

To make this work on the host, or on any other machine that requires it, you have to edit the hosts file:

On Linux/Mac:

bash
echo "127.0.0.1 keycloak" | sudo tee -a /etc/hosts

On Windows: Edit C:\Windows\System32\drivers\etc\hosts as Administrator and add:

127.0.0.1 keycloak

Can’t I just use localhost in the browser instead?

You might be wondering if there’s a shortcut here.

Even if the PostgreSQL container has to reference Keycloak as keycloak, your host still sees the exposed port as localhost:8443. So can you just use this in the browser instead, and complete the authentication that way, without editing the hosts file?

The answer is unfortunately no. There are two possible scenarios:

If Keycloak is configured with strict hostname, it will try to redirect the browser to “https://keycloak…” during the authorization process
If Keycloak is configured with dynamic hostname, it will complete the process with “https://localhost”, but it will also use “localhost” in the generated access tokens instead of “keycloak”. When our validator checks the token, it will notice this discrepancy and reject the login attempt.

Configuring your realm

With the Keycloak infrastructure setup out of the way, we can now focus on configuring Keycloak itself.

After you have your Keycloak instance up and running, it is time to open a browser and navigate to https://keycloak:8443. A quick login with “admin” and “admin”, and the browser already displays the admin UI with the default master realm.

A complete detailed introduction is out of scope for this blog post – the Keycloak documentation is much better for that – we will only try to explain the minimum required to set up a relatively simple, but secure configuration for our PostgreSQL instance. The steps we show here will be similar to what our demo setup also uses.

Let’s start by creating a new realm, under the “Manage realms” menu: realms are the main building blocks of isolation in Keycloak, storing users, clients, and everything, so it’s a good practice not to use the default one. In our example, we named our realm pgrealm. This name is important, as it will be included in the OAuth issuer URL.

After hitting “Create”, the new realm is automatically set as current, and we can continue configuring it.

Let’s continue by creating our “testuser” under “Users”. Select that the email is verified, fill the requested email, first name and last name fields, and hit create. If you miss some of these fields, Keycloak will ask you to complete them during the first login.

After the user is created, navigate to the “Credentials” tab on the displayed user admin page, and set a password. Also remove the checkbox from “Temporary”, unless you want to change it during the first login. In our example setup, we used “asdfasdf”. This is of course only appropriate for a quick demo setup, use a better one for anything else.

After creating our user, let’s create a client under “Clients” with the “Create Client” button. The first screen asks for a client ID – this will be required for the psql command – and a name and description – these will be displayed on the authorization page by the browser.

After clicking next, the next screen configures how OIDC should work exactly.

The first toggle, “Client authentication” can be both on and off – this controls if the client requires a secret, or only an ID. As we discussed earlier, psql is a public client, and while it can use a secret, it’s not really a secret. Adding a secret only adds complexity while not providing more security, as we also have to specify that during the connection call, but it is supported.

For the “authentication flow” we have to check “OAuth 2.0 Device Authorization Grant” to enable the device flow, and we can leave everything else on default.

The final third screen doesn’t require any changes – those are settings for HTTP-based flows, but we are using the device flow. If you are configuring PostgreSQL OIDC with a web application, of course you should fill these properly.

After creating our client, it’s also a good practice to make our user consent more explicit: as we tried to make this clear in earlier blog posts, this is the only line of defense with OAuth-based logins: administrators have to be very clear about displaying where the user logs in.

First, if you scroll down on the Client administration page displayed after creation, there’s a section about the consent screen. It’s a good practice to make this as explicit as possible, with a nice custom message for the users.

After saving this, we also should add a client scope using the menu item below “Clients”. The name attribute is what we’ll have to specify in our PostgreSQL configuration. The two toggles below are both required: “include in token scope” means that it will be included in the JWT; without that, the validator won’t be able to verify the presence of the scope. “Display on consent screen” means that this is a scope that should be explicitly displayed to the user during login.

(We’ll discuss what scopes are and why they matter in the next section.)

And with this, our basic Keycloak configuration is ready!

What is a scope?

The scope configuration in the previous section might seem confusing: why did we configure consent screen settings at multiple locations? What is this scope concept actually about?

To answer this question, we have to remember our earlier examples, where we discussed that PostgreSQL itself is not a client (application), but it is just something used by possibly multiple clients. In the real world, that client could be “psql”, or “EditorApp”, or anything else that uses PostgreSQL while not providing more security.

PostgreSQL could use a list of allowed clients for authorization – but it doesn’t do that. Instead it relies on another OAuth concept, scopes.

The OAuth scope is a mechanism intended to limit what a token is allowed to access. There are some generic scopes, such as “email” or “profile”. And there are many application specific scopes: cloud providers like Google or Azure use them to control which services a token is allowed to access. For example, if you grant something the ability to add entries to your calendar, it won’t be allowed to read your emails or location history.

The same way, you can think of scopes with PostgreSQL as “which database the user can access”. Database here could mean both a PostgreSQL instance, or just a single database in it – as OAuth is configured in pg_hba, it is possible to configure different required scopes for different databases (more about this later).

During the Keycloak configuration we recommended configuring an explicit consent screen for the client, but technically that part is outside of the database configuration. A database can be accessed by multiple clients, some of those might be created way later in time.

Even if somebody doesn’t configure that part, a scope with a required consent screen will be displayed, if the client requests it. And that is part of the database configuration – as we can configure PostgreSQL explicitly to require specific scopes in its configuration.

Configuring PostgreSQL

Now that we have a working, properly configured Keycloak server, it’s time to configure the PostgreSQL side of the equation. While we do have a working Docker example as part of the Docker Compose configuration, this is currently tricky to set up: there is no Docker image with our pg_oidc_validator.

You can check the relevant section of the compose configuration, but it requires a custom entry point and a short bash script.

In this blog post we’ll focus on the actual manual steps instead.

Required packages

On the server, we need the PostgreSQL 18 server packages installed, and also the pg_oidc_validator package. For the validator, we currently have downloadable deb and rpm packages on our GitHub releases page, and packages for SUSE can be found in the official SUSE packages.

On the client side, we need the PostgreSQL 18 client along with the OAuth client package – this is usually a separate package named libpq-oauth on most distributions and requires separate installation

Setting up a data directory

Let’s just assume that we have a data directory initialized somewhere:

initdb -D datadir

We will have to modify a few configuration files for OIDC to work.

Let’s start by enabling the validator in datadir/postgresql.conf:

oauth_validator_libraries = pg_oidc_validator
pg_oidc_validator.authn_field = email

The first line tells PostgreSQL to load the Percona pg_oidc_validator, and the second line is a specific configuration parameter for our validator – it tells it that we want to map Keycloak users to PostgreSQL users based on the email claim in the access token. This second line is optional; it defaults to the “sub” (subject) field, which identifies the user in most OIDC providers. However, Keycloak doesn’t allow us to customize the value of this field, and it returns a non-user-friendly identifier in it. In practice, it is clearer to use a more verbose field for mapping, such as email, so we’ll use that in our example.

After modifying this file, we can try to start the server. If all required packages are installed correctly, and there isn’t another PostgreSQL instance running on the default port, it should start without issues:

pg_ctl -D datadir start

With the server running, let’s create a matching PostgreSQL user for our testuser:

createuser testuser

Next, we have to tell PostgreSQL that we want to use the Keycloak instance we set up previously. Let’s add an entry to datadir/pg_hba.conf to reference the OIDC provider. Add this line to the beginning of the file, before the trust sections – since pg_hba is executed line by line, if the trust entries are before the OAuth entry, authentication will never reach that line:

host all all 0.0.0.0/0 oauth scope="email pgscope",issuer=https://keycloak:8443/realms/pgrealm,map=kcmap

Warning: In a real-world setup, you would have to remove all trust entries. The trust authentication method allows connections without any password verification and should never be used in production.

This entry adds the OAuth option for connections using IPv4. We specify that we require “pgscope”, which is the example scope with the custom consent screen we created earlier in the Keycloak configuration, and the “email” scope, which instructs Keycloak to include the user’s email address in the JWT (this will also be presented on the consent screen).

The latter is required because in this example we map our database users to the users on the identity provider using their email address – this will be configured in more detail in the “kcmap” referenced in the configuration line.

The only required parameter to OAuth is the “issuer”; everything else is optional. But providing required scopes is a good practice, and it is required for proper security, as we explained earlier. There are also other parameters not mentioned here, a full list is available in the PostgreSQL documentation.

One potentially important parameter is the “validator”. PostgreSQL allows multiple different pg_hba OAuth entries, and it also allows multiple different validators: the configuration parameter in postgresql.conf is called oauth_validator_libraries. If that parameter actually contains multiple libraries, the “validator” parameter becomes required for OAuth entries in pg_hba conf, and has to match an entry in the validator list. Otherwise, it is assumed that all OAuth entries use the one available validator.

The above means that the map setting is also optional. There are scenarios where it’s not needed, such as if the identity provider has a claim that directly contains user names matching PostgreSQL usernames, we could use that field instead and skip the manual mapping.

Following our example, let’s define an entry in datadir/pg_ident.conf for the kcmap. We only need a single entry for our testuser:

# MAPNAME SYSTEM-USERNAME DATABASE-USERNAME
kcmap testuser@example.com testuser

All that’s left is to reload our configuration:

pg_ctl -D datadir reload

Connecting to the database

With all the configuration in place, it’s time for the moment of truth: actually connecting to PostgreSQL using OIDC!

With the server properly configured, we are ready to connect to it using psql. To do so, we have to use a command similar to the following:

bin/psql -h 127.0.0.1 'dbname=postgres oauth_issuer=https://keycloak:8443/realms/pgrealm oauth_client_id=pgclient'

Note that we have to repeat the issuer URL here. It is always required, even if the PostgreSQL configuration only contains one OAuth issuer. This URL has to match exactly the issuer URL in a pg_hba line, and both should exactly match the issuer URL in the JWT. If there’s a mismatch anywhere, authentication will fail. This is why we had to take extra steps previously to make the Keycloak hostname available everywhere.

This is also the only place where we have to mention a client id – and if we configured an authenticated client in Keycloak, we also have to specify oauth_client_secret. The server can work with multiple clients, but now that we are actually starting up a PostgreSQL client, we can specify which OAuth client will we use to complete the authentication flow.

After we execute this command, psql will display the device authorization instructions, showing us a URL and a device code.

All we have to do is follow the instructions:

navigate to the specified URL
enter the device code
login with our testuser
confirm the consent screen

If you are doing all of this in a single session, don’t forget to log out of the admin user session before executing these steps, or use a different browser for it. We want to log in with the testuser, not with the admin – the admin user doesn’t have a mapping in our PostgreSQL configuration.

While we go through these steps, the psql client periodically polls Keycloak to see if the flow was completed on the OIDC provider side. Since this is a periodic polling, done every few seconds, we might have to wait a few seconds before we are logged in to an SQL session.

That’s all!

And just like that, we’ve successfully authenticated to PostgreSQL using OIDC!

With the psql command logged in, we’ve completed the full circle: from setting up Keycloak with proper certificates, through configuring realms and clients, to establishing a secure OIDC-authenticated PostgreSQL connection.

Hopefully these instructions were clear and everything worked on the first try. If you encountered issues along the way, don’t be discouraged: OAuth/OIDC is complex, there are many things that could go wrong. Since this is an important security feature, it has to fail if anything is even slightly wrong.

In our next post, we’ll focus on errors and failures: both to help diagnose possible errors with the OAuth flow in PostgreSQL, but also for reassurance: in an authentication setup not letting unauthorized people log in is just as important as successfully logging in somebody with the proper permissions. Stay tuned for our examples showcasing how pg_oidc_validator and PostgreSQL’s OAuth support can keep your server safe!

If you find any issues with our validator, or have comments / feature requests, please reach out to us in our Github page!

Configuring the Component Keyring in Percona Server and PXC 8.4

Wayne Leutwyler — Tue, 13 Jan 2026 00:00:00 UTC

Configuring the Component Keyring in Percona Server and PXC 8.4

(Or: how to make MySQL encryption boring, which is the goal)

Encryption is one of those things everyone agrees is important, right up until MySQL refuses to start and you’re staring at a JSON file wondering which brace ruined your evening.

With MySQL 8.4, encryption has firmly moved into the component world, and if you’re running Percona Server 8.4 or Percona XtraDB Cluster (PXC) 8.4, the supported path forward is the component_keyring_file component.

The good news: the setup is mostly identical for Percona Server and PXC.
The bad news: PXC 8.4.4 and 8.4.5 shipped with a bug that makes this less fun than it should be.

Let’s walk through a setup that works, keeps your keys locked down, and avoids the usual landmines.

Step 1: Tell MySQL Which Component to Load

Components are registered using JSON, not traditional MySQL configuration syntax. This is important, because MySQL will not politely warn you if you get it wrong. It will simply refuse to start.

Create the file:

sudo vi /usr/sbin/mysqld.my

Add:

json
{
 "components": "file://component_keyring_file"
}

Take a second to double-check the formatting. One missing quote here will cost you more time than you want to admit.

Now lock it down:

bash
sudo chown root:root /usr/sbin/mysqld.my
sudo chmod 644 /usr/sbin/mysqld.my

This is configuration, not data. MySQL only needs to read it.

Step 2: Prepare the Keyring Directory (Handle With Care)

This directory will hold encryption keys. Treat it accordingly.

bash
cd /var/lib
sudo mkdir mysql-keyring
sudo chown mysql:mysql mysql-keyring
sudo chmod 750 mysql-keyring

A simple rule that saves headaches:

mysql owns the keys
MySQL is allowed to access them
Nobody else gets any ideas

Step 3: Configure the Keyring Component Itself

Next, move to the plugin directory:

cd /usr/lib64/mysql/plugin

Create the component configuration file:

sudo vi component_keyring_file.cnf

Add:

json
{
 "path": "/var/lib/mysql-keyring/component_keyring_file",
 "read_only": true
}

This file tells MySQL where the keyring lives and ensures it can’t be casually modified at runtime.

Set ownership and permissions:

bash
sudo chown root:root component_keyring_file.cnf
sudo chmod 640 component_keyring_file.cnf

Again: configuration belongs to root. MySQL just reads it.

Step 4: The PXC 8.4.4 / 8.4.5 Bug (Yes, There’s One)

If you’re running Percona Server, you can skip this entire section and enjoy your day.

If you’re running Percona XtraDB Cluster 8.4.4 or 8.4.5, there is a known issue with plugin paths that prevents the component keyring from loading correctly. This was fixed in PXC 8.4.6.

If upgrading isn’t an option yet, you’ll need one of the following workarounds.

Option A: Create a Symlink (Preferred)

bash
sudo ln -s \
/usr/bin/pxc_extra/pxb-8.4/lib/lib64/xtrabackup/plugin \
/usr/bin/pxc_extra/pxb-8.4/lib/plugin

Option B: Copy the Plugin Directory

bash
sudo cp -ar \
/usr/bin/pxc_extra/pxb-8.4/lib/lib64/xtrabackup/plugin \
/usr/bin/pxc_extra/pxb-8.4/lib

If you’re on PXC 8.4.6 or newer, this problem is already behind you and you can safely pretend it never existed.

Step 5: Restart MySQL

Time for the moment of truth:

sudo systemctl restart mysql

Or mysqld, depending on your system.

If MySQL starts cleanly, you’re doing well. If not, go back and check your JSON files. It’s almost always the JSON.

Step 6: Verify the Keyring Is Actually Loaded

Never assume. Always verify.

sql
SELECT *
FROM performance_schema.keyring_component_status;

You should see the component_keyring_file listed and active. If it’s there, the keyring is live.

sql
+---------------------+-----------------------------------------------+
| STATUS_KEY | STATUS_VALUE |
+---------------------+-----------------------------------------------+
| Component_name | component_keyring_file |
| Author | Oracle Corporation |
| License | GPL |
| Implementation_name | component_keyring_file |
| Version | 1.0 |
| Component_status | Active |
| Data_file | /var/lib/mysql-keyring/component_keyring_file |
| Read_only | Yes |
+---------------------+-----------------------------------------------+
8 rows in set (0.00 sec)

A Note for Percona Server Users

Percona Server may still include legacy keyring plugins such as:

keyring_file
keyring_vault

Do not mix legacy keyring plugins with component keyrings. They come from different eras of MySQL design and do not coexist peacefully.

Choose one model. For MySQL 8.4 and forward, components are the future.

Additional Steps for Percona XtraDB Cluster (PXC)

Percona XtraDB Cluster introduces one critical difference compared to standalone Percona Server: the keyring file itself is not replicated by Galera. Only metadata and transactional state are replicated. The encryption keys remain node-local filesystem artifacts and must be handled deliberately.

Node 1: Establish the Authoritative Keyring

Choose a single node to initialize the keyring. This is typically Node1, but the choice itself is not important as long as you are consistent.

On this node:

Complete all previous steps in this document
Start MySQL successfully
Verify the keyring component is loaded:

sql
SELECT *
FROM performance_schema.keyring_component_status;

Once this node is running, the file below will be created and populated:

swift
/var/lib/mysql-keyring/component_keyring_file

This file becomes the authoritative source of encryption keys for the entire cluster.

Why the Keyring File Must Be Copied

PXC ensures that encrypted data remains readable on all nodes, but it does not distribute encryption keys themselves. Each node must have access to the same key material, or encrypted tablespaces will fail to open.

If a node starts without the correct keyring file, you may see:

Tablespace open failures
Startup errors related to encryption
Inconsistent behavior during SST or IST

This is expected behavior and not a bug.

Distribute the Keyring File to Other Nodes

On each remaining PXC node:

Ensure MySQL is stopped
Create the keyring directory if it does not exist:

bash
sudo mkdir -p /var/lib/mysql-keyring
sudo chown mysql:mysql /var/lib/mysql-keyring
sudo chmod 750 /var/lib/mysql-keyring

Securely copy the keyring file from Node1:

scp /var/lib/mysql-keyring/component_keyring_file node2:/var/lib/mysql-keyring/component_keyring_file

Important: Do not modify the file. Do not recreate it. Do not allow MySQL to generate a new one on secondary nodes.

Start MySQL on Each Node and Verify

After the keyring file is in place:

sudo systemctl start mysqld

Verify the component is active:

sql
SELECT * FROM performance_schema.keyring_component_status;

Each node should report the component_keyring_file as loaded and active.

At this point:

Encrypted tablespaces will open correctly
SST and IST operations will succeed
The cluster will behave consistently during restarts

Operational Notes and Best Practices

Treat the keyring file like a secret, not configuration
Restrict access to root only
Include the keyring file in your secure backup strategy
When provisioning new nodes, copy the keyring file before first startup
Never rotate or regenerate the keyring independently on individual nodes

If the keyring is lost and encrypted data exists, recovery is not possible.

Final Thoughts

This setup works reliably for:

Percona Server 8.4
Percona XtraDB Cluster 8.4
(with the known exception of 8.4.4–8.4.5)

Most failures come down to:

Treating JSON like a .cnf file
Loose ownership on sensitive files
Forgetting the PXC-specific workaround

Once those are handled, the component keyring fades into the background where it belongs. And when it comes to encryption, boring, quiet, and uneventful is exactly the outcome you want.

Open source, PostgreSQL, and risk mitigation in an era of acquisitions

Jan Wieremjewicz — Fri, 19 Dec 2025 11:00:00 UTC

As the year comes to a close and many of us start slowing down before the winter holidays, I find myself reflecting on patterns I’ve seen repeat, both as a customer and as someone working closely with the PostgreSQL ecosystem.

Looking back at software acquisitions over the past year, one might assume they only change logos. In reality, they often change roadmaps, priorities, and unfortunately all too often also the promises customers originally bought into.

I have experienced this more than once as a customer. Most recently, my team evaluated a product management tool that, shortly after being acquired, informed us we had a single month to migrate away. While I personally had not invested much time yet, colleagues had already moved documentation and workflows only to see that work become wasted effort. For those not familiar with product management, migrating all the product planning documentation is a huge effort and having it wasted is like erasing a month of your life.

One might say “business is business” and that customers can always take their money elsewhere. That may be true, but it does little to make customers feel safe or protected. In practice, it often leaves them confused about where accountability lies. Who should we blame when the original vendor no longer exists and the acquiring company is acting in its own strategic interest?

What struck me this year is how often these experiences echoed each other: across different tools, teams, and ecosystems. Unfortunately, I am now seeing similar patterns emerge in my professional focus area: PostgreSQL.

To be very clear, the PostgreSQL Community itself is driving an outstanding database, one that has been, and I am confident will remain, truly open source and community-developed. PostgreSQL as a project is healthy, stable, and thriving, and I hope it will thrive even more in the coming year.

That said, PostgreSQL is not just the database engine itself, but an entire ecosystem of vendors, extensions, and services built around it.

Over the past year, several important companies in the PostgreSQL ecosystem have been acquired and in conversations throughout the year this topic kept resurfacing. In recent prospective customer conversations, we increasingly hear from PostgreSQL users running mission-critical workloads on-premises who feel abandoned by their vendor. These organizations are being quietly nudged toward “modernization” paths that, in practice, resemble mandatory SaaS migration.

Having worked closely with MongoDB users for years, this pattern feels familiar. Since MongoDB Atlas became the primary strategic focus, many customers experienced similar pressure. What feels different, and what stood out to me most this year, in the PostgreSQL world is timing.

Teams often discover late in the renewal cycle that:

Their on-prem PostgreSQL deployment is no longer strategic for the vendor
Key components they depend on, which were never fully open source, are no longer available for renewal
The implied options are to “move to the cloud” or rapidly find an alternative, even when migration planning is complicated by lack of source availability

This puts PostgreSQL customers in a difficult position: accept architectural change under pressure, or scramble to replace a trusted vendor while the clock is already ticking.

While recent conversations often reference Crunchy Data following its acquisition by Snowflake, this is not about a single company. The broader pattern has repeated across the industry, including infrastructure significant projects involving MinIO, Bitnami, HashiCorp, and Redis.

This raises a fundamental question for the PostgreSQL ecosystem and infrastructure software in general:

When a critical infrastructure vendor is acquired or changes licensing, who advocates for the customers that cannot move?

Open source is not just a licensing model or philosophical ideal. It provides freedom of choice, deployment flexibility, and risk mitigation. Recent community responses such as OpenTofu, OpenBao, and Valkey demonstrate a growing maturity and ability of open source communities to organize when freedoms erode.

It is disappointing to see these dynamics emerge around PostgreSQL, even though the database itself remains one of the most open source and community governed projects in the industry.

It is disappointing to see similar dynamics affect PostgreSQL, a database often considered one of the most open source driven projects in the industry. When PostgreSQL derivatives are impacted, the shadow inevitably reaches upstream as well.

If this post reaches teams who were not yet aware of these shifts and gives them more time to plan going into the new year, it serves its purpose. If you are running PostgreSQL in environments where SaaS is not an option, now is the time to ask difficult questions before renewal conversations start, not after options disappear.

As we head into a new year, I expect these conversations to become even more common. Acquisitions, licensing changes and cloud first strategies are not slowing down, but neither is the need for predictability, transparency and choice in how PostgreSQL is deployed and supported.

Looking ahead to 2026, my hope is simple: more open source products, more stable and predictable service offerings around them, and fewer last-minute surprises for the teams who depend on them every day.

We Want to Hear from You

I am curious:

Have you seen similar shifts in PostgreSQL vendors post-acquisition?
What is preventing you from moving to PostgreSQL community builds or alternative support models?

Let’s talk. You can reach me via:

LinkedIn: https://www.linkedin.com/in/janwie/
Email: jan(dot)wieremjewicz(at)percona(dot)com

Open Source isn’t a strategy, it’s who we are!

We are always open to your feedback. You can reach us at:

Percona Community Forums https://forums.percona.com/c/postgresql/25
Via issues and discussions on Percona GitHub repositories

If there is an event we attend, focused on open source or not, don’t be a stranger, come chat with us in person!

Enhancing PostgreSQL OIDC with pg_oidc_validator

Jan Wieremjewicz — Wed, 17 Dec 2025 11:00:00 UTC

With PostgreSQL 18 introducing built-in OAuth 2.0 and OpenID Connect (OIDC) authentication, tools like pg_oidc_validator have become an essential part of the ecosystem by enabling server-side verification of OIDC tokens directly inside PostgreSQL. If you’re new to the topic, make sure to read our earlier posts explaining the underlying concepts and the need for external validators:

This release builds on the initial version announced in October and continues our mission to make OIDC adoption in PostgreSQL reliable, fast, and accessible for all users.

What’s New in This Release

This new iteration of pg_oidc_validator (v0.2) introduces two major improvements:

initial caching support, and
Debian/Ubuntu and RPM packages to simplify installation.

Most importantly, these improvements come directly from community feedback, ****whether during conversations at PGConf.EU in Riga, KubeCon US in Atlanta, or through GitHub and forums. Thank you for helping us shape this project!

Caching Support in pg_oidc_validator

OIDC token verification requires fetching issuer metadata and JWKS keysets from an external identity provider (IdP). Without caching, every PostgreSQL backend performing validation must re-fetch this data, increasing latency and putting unnecessary load on the IdP.

pg_oidc_validator v0.2 introduces a lightweight caching layer. This allows the validator to:

cache OIDC discovery documents and JWKS responses when permitted by the IdP,
use cached responses across PostgreSQL backends,
reduce outbound HTTP calls,
validate tokens at in-memory speeds, and
integrate cleanly with IdP key rotation.

This results in improved performance, reduced IdP load, and better scalability for deployments using Keycloak, Okta, Microsoft Entra ID, Ping Identity, or other OIDC providers.

A Note on Testing

The caching layer currently lacks full automated test coverage. This is because Keycloak does not allow caching for issuer or JWKS endpoints (Keycloak issue #15216), preventing us from validating caching behavior.

To address this, we plan to extend the test setup by placing an nginx proxy between PostgreSQL and Keycloak to simulate IdP responses that include cache-friendly headers.

Pre-Built Packages Now Available

Installing pg_oidc_validator is now easier than ever. We provide builds at the latest release page, where nightly builds are available for:

Debian / Ubuntu - tested on Ubuntu 24.04
RHEL / Oracle Linux / Rocky Linux - tested on OL8 and OL9

If you prefer building from source, instructions are available directly in the project’s README.

Try it, test it, tell us all about it!

As an open source project, pg_oidc_validator grows with your feedback. We want to hear about:

your deployment use cases,
performance characteristics,
integration challenges,
and features you’d like to see next.

You can reach us here:

Percona Community Forums:

https://forums.percona.com/c/postgresql/25
GitHub:
- Issues: https://github.com/Percona-Lab/pg_oidc_validator/issues
- Discussions: https://github.com/Percona-Lab/pg_oidc_validator/discussions

And of course, if you see Percona at an event, come talk to us at the booth!

What is New in Percona Toolkit 3.7.1

Sveta Smirnova — Wed, 17 Dec 2025 00:00:00 UTC

Percona Toolkit 3.7.1 has been released on Dec 17, 2025. The most important updates in this version are:

Finalized SSL/TLS support for MySQL
Added support for Debian 13 and Amazon Linux 2023
Fixed MariaDB support broken in version 3.7.0
Added options to skip certain collections in pt-k8s-debug-collector and pt-stalk
Documentation improvements
Other performance improvements

In this blog, I will outline the most significant changes. A full list of improvements and bug fixes can be found in the release notes.

SSL/TLS support for MySQL

Percona Toolkit historically did not have consistent SSL support. This was reported at https://perconadev.atlassian.net/browse/PT-191. In version 3.7.0, option s for DSN was introduced. This option instructs DBD::mysql to open a secure connection with the database. This version also adds command-line option --mysql-ssl and its short form -s to all tools. All other SSL/TLS-related options, such as ssl-ca, ssl-cert, ssl-cipher, etc, could be specified in the configuration file if necessary. This completes SSL/TLS support for MySQL. For more details and information check this blog post.

Supported Platforms Update

Percona repositories now have Percona Toolkit packages for Debian 13 and Amazon Linux 2023. To install them enable repository pt with the percona-release utility. Ubuntu Focal reached its EOL and support for this platform has been removed. More information on Percona repositories is available in the User Reference Manual.

Regression Bug Fixes

Recent major changes introducing MySQL 8.4 support missed ignore case modificator for the regular expression that checks if MySQL flavor is MariaDB. As a result, tools executed replication statements not compatible with MariaDB. Version 3.7.1 fixes the regular expression and re-adds MariaDB support back (PT-2451). Future versions of Percona Toolkit will have better MariaDB support, including MariaDB-specific versions of non-offensive replication commands.

Utility pt-sift stopped working, because dependent library alt_cmds.sh was not included (PT-2498). This was not found during previous release testing, because regression test for the tool was not run. Now this miss is fixed and the utility works properly again. Additionally, regression test is updated.

Helper utility version_cmp was written in some compiled language and source code for it was not available (PT-2469). This broke version checking on platforms not compatible with the unknown platform where the binary was originally compiled. Now this utility rewritten as a Bourne-Again shell script.

Modern MySQL Support

Percona Toolkit uses legacy MySQL syntax in many places to be compatible with older versions of MySQL. In other places, it misses modern MySQL diagnostic additions. This version makes first steps to improve this situation by adding such features as invisible index support in pt-duplicate-key-checker (PR-996) and performance_schema.threads collecton in pt-stalk (PT-1718). Currently, data from performance_schema.threads is collected along with the deprecated information_schema.processlist. In the future, support for information_schema.processlist will be deprecated, then removed.

Future versions of Percona Toolkit will have more modern MySQL diagnostic support.

Performance Improvements

pt-stalk now has new option, --skip-collection, that allows to skip one or more collections. Supported values for this option are: ps-locks-transactions, thread-variables, innodbstatus, lock-waits, mysqladmin, processlist, rocksdbstatus, transactions. To skip two or more collections, separate them with a comma. E.g., --skip-collection=processlist,innodbstatus. You will find more information at PT-2289 and in the User Reference Manual for pt-stalk.

pt-k8s-debug-collector introduces option -skip-pod-summary allowing to skip pod summary collections, such as pt-mysql-summary, pt-mongodb-summary, or pg_gather. Check PT-2453 and the User Reference Manual for pt-k8s-debug-collector.

Originally, tools output was always buffered. This is usually good for performance but you may want to disable this feature when need to see output of the tools faster. For example, if you run pt-archiver or pt-table-checksum on large table in Kubernetes, you won’t see progress (PT-2052) until the tool finishes. New option, --[no]buffer-stdout, allows to disable buffering when needed.

Incompatilbe change

Earlier, if --chunk-size was enabled for pt-online-schema-change, option --chunk-time was ignored. This caused situations when a user has to start with default automatic chunk size even if it was not effective for some tables, and wait when chunk size is adjusted in subsequent iterations. Alternatively, they had to guess fixed chunk size that implies time consuming try and error approach (PT-1423).

Starting from version 3.7.1, if both options --chunk-size and --chunk-time are specified, initial chunk size will be as specified by the option --chunk-size, but later it will be adjusted, so that the next query takes specified amount of time (in seconds) to execute.

Documentation Improvements

While working on this release we found undocumented featues such as --recursion-method=dsn support in pt-table-sync (PT-2470), broken man page for pt-secure-collect and other tools written in Go language (PT-1564), as well as minor documentation issues. Now all of them are fixed.

Community contributions

This release includes contributions from Community and Percona Engineers who do not actively work on the project. We want to thank:

Iwo Panowicz for option -skip-pod-summary in pt-k8s-debug-collector (PT-2453)
Matthew Boehm for invisible indexes support in pt-duplicate-key-checker (PR-996)
Nilnandan Joshi for collecting performance_schema.threads along with information_schema.processlist in pt-stalk (PT-1718) and fix for PT-2014 - pt-config-diff does not honor case insensitivity flag
Paweł Kudzia for the updated documentation of pt-query-digest (PR-953)
Maciej Dobrzanski for fixing PR-890 - pt-config-diff: MySQL truncates run-time variable values longer than 1024 characters
Marek Knappe for fixing PT-2418 - pt-online-schema-change 3.7.0 lost data when exe alter xxx rename column xxx and PT-2458 - remove-data-dir defaults to True
Yoann La Cancellera for his work on pt-galera-log-explainer
Nyele for restoring MariaDB support (PT-2465)
Taehyung Lim for fixing PT-2401 - pt-online-schema-change ’table does not exist’ on macos
Viktoras Agejevas for fixing PR-989 - Fix script crashing with precedence error in pt-online-schema-change
Hartley McGuire for fixing PT-2015 - pt-config-diff does not sort variable flags

MySQL Replication Best Practices: How to Keep Your Replicas Sane (and Your Nights Quiet)

Wayne Leutwyler — Wed, 03 Dec 2025 00:00:00 UTC

MySQL replication has been around forever, and yet… people still manage to set it up in ways that break at the worst possible moment. Even in 2025, you can get burned by tiny schema differences, missing primary keys, or one forgotten config flag. I’ve seen replicas drift so far out of sync they might as well live in a different universe.

This guide covers the practical best practices—the stuff real DBAs use every day to keep replication stable, predictable, and boring. (Boring is a compliment in database land.)

Always Use GTIDs. Yes, Always.

GTID-based replication is one of those features that people resist turning on, and then once they do, they never want to go back.

Why GTIDs?

Failover become sane
Reparenting replicas stops being a headache
Missing transactions are easy to detect

Your my.cnf should absolutely include:

gtid_mode=ON
enforce_gtid_consistency=ON
log_replica_updates=ON

Once GTIDs are enabled, do not mix in old-style replication. That path leads straight to confusion.

Use Row-Based Replication (RBR)

Statement-based replication is a nostalgia trip that nobody asked for. It breaks on:

NOW(), UUID(), and similar functions
Floating point differences
Collation mismatches
Triggers behaving differently

Just skip the pain and use:

binlog_format=ROW

RBR is slightly more verbose, but 100× more predictable. When something breaks, it’s never because you chose ROW.

Every Table Needs a Primary Key. No Exceptions.

If you take nothing else from this guide, take this:

Replication without primary keys is a bad time.

Row-based replication needs a way to find the row that changed. Without a PK (or at least a UNIQUE index), the server has to use every column as a lookup. That’s slow, error-prone, and sometimes impossible.

The usual symptoms:

Replication lag slowly creeping up
Replica doing full table scans on updates
Rows failing to apply
Errors like:

Error 1032: Can't find record in table

Save yourself hours of debugging and just make sure every table has a primary key.

Keep the Schema Identical Everywhere

Replication assumes that everyone’s using the same schema. MySQL will happily keep going even if your schemas don’t match—and then quietly drift out of sync.

Here are the practical ways to keep schemas aligned:

Approach A — mysqldump (most common)

Export schemas only:

mysqldump --no-data mydb > schema.sql

From both servers, then:

diff source-schema.sql replica-schema.sql

Approach B — information_schema metadata

This approach is great for automaton:

SELECT table_name, column_name, column_type, is_nullable, column_default
FROM information_schema.columns
WHERE table_schema = 'mydb'
ORDER BY table_name, ordinal_position;

Execute this query on each server and diff the results. Update mydb to match the database whose schema metadata you want to examine.

Approach C — pt-table-checksum (data only)

This doesn’t compare schemas — it catches data drift. You should consider running it on a schedule such as:

high-change OLTP DBs run weekly or even daily
huge multi-TB DBs run quarterly
some sensitive systems avoid running it during peak hours

pt-table-checksum --replicate=percona.checksums

You can fix drift with:

pt-table-sync --execute --replicate=percona.checksums

Schema checks + data checks = safe replication.

Harden Your Binary Log Settings

Your binlogs are the backbone of replication. Treat them carefully.

sync_binlog=1
binlog_row_image=FULL
binlog_expire_logs_seconds=604800 # 7 days

sync_binlog=1 is the big one—without it, a crash can corrupt binlogs or the GTID position, and that leads to a very bad day.

Protect Your Replicas with super_read_only

Never allow accidental writes to replicas, in your my.cnf set:

read_only=ON
super_read_only=ON

super_read_only closes the loophole that even SUPER users could previously use to write to replicas.

Use a Dedicated Replication User

Give the minimal permissions:

CREATE USER 'repl'@'%' IDENTIFIED BY 'strong_password';
GRANT REPLICATION REPLICA ON *.* TO 'repl'@'%';

This user should do exactly one thing: replicate. Don’t reuse app users—you’re just begging for trouble.

Replication Lag: Watch It Like a Hawk

Seconds_Behind_Source lies more often than you’d expect. It’s okay for a quick glance but don’t rely on it.

Better options:

Performance Schema: replication_applier_status_by_worker
Percona Monitoring and Management (PMM)
Custom heartbeat tables
pt-heartbeat

Lag is one of the biggest causes of outages—monitor it continuously. Lag is usually the first sign something is wrong—catch it early.

Use Parallel Replication (But Don’t Overdo It)

If your primary has multiple writers or many concurrent transactions, in your my.cnf enable parallel workers:

replica_parallel_type=LOGICAL_CLOCK
replica_parallel_workers=4

4–8 workers is a sweet spot for most systems. More workers ≠ more speed; after a point it just increases memory footprint without real benefit.

But when it helps, it really helps—like cutting lag by 80–90%.

Use SSL Anywhere Outside the LAN

Replication traffic isn’t something you want exposed.

source_ssl=1
source_ssl_ca=/path/ca.pem

Earlier versions used the master_ssl_* variables, but the idea is the same: encrypt the connection when it leaves your trusted network.

Final Thoughts

MySQL replication can be rock-solid, but only if you follow a handful of rules that experienced DBAs know by heart:

Use GTIDs
Use RBR
Always have primary keys
Keep schemas aligned
Check for data drift
Harden binlog settings
Protect replicas from accidental writes
Monitor lag properly
Use parallel workers when appropriate
Encrypt connections over untrusted networks

Follow these, and your replicas will stay healthy, consistent, and (mostly) invisible—which is exactly how you want them.

Community Recap: Percona.Connect London 2025, Building the Future of Open Source Together

Edith Puclla — Tue, 02 Dec 2025 00:00:00 UTC

Percona.Connect London 2025 brought the open-source database community together for a half-day of learning and collaboration. The event focused on providing practical, technical insights for DBAs, DevOps engineers, and developers. The main takeaway was clear: Stability, Openness, and Automation are essential for modern, large-scale data infrastructure.

Top Discussions & Key Takeaways

1. The Rise of Valkey: A Truly Open Caching Alternative

Martin Visser, Valkey Technical Lead, explained the changes to the Redis license, the community needs a trusted, open-source replacement. Valkey was highlighted as the leading solution.

Valkey was started by former Redis contributors quickly after Redis removed its open source license in 2024.
It is a true open-source project governed under the Linux Foundation.
It offers enhancements like better memory efficiency, performance, and scalability.
In a recent Percona survey of 200 DBAs, Valkey was the most preferred alternative to Redis.

2. Running PostgreSQL in a Cloud Native context

Takis Stathopoulos, Enterprise Architect, presented on running PostgreSQL in a Cloud Native context, explaining how Kubernetes Operators simplify complex deployments.

Cloud Native vs. Cloud First: Cloud Native (Kubernetes) offers Portability and No vendor lock-in, allowing you to run the database consistently across different clouds and on-premise infrastructure.
Percona Operator for PostgreSQL: This tool automates crucial operations like setting up high availability (using Patroni), backups (using pgBackrest), and scaling.
When to use Cloud Native: It’s ideal for large, microservice-based applications and teams prioritizing portability and avoiding vendor lock-in.

3. Native PostgreSQL TDE is Here: Securing Data Simply

Alastair Turner, Postgres Community Advocate, introduced the new Native Transparent Data Encryption (TDE) for PostgreSQL.

4. The Future of MySQL: Vector Search & Binlog Server

Dennis Kittrell, MySQL Product Manager, discussed two key features planned for MySQL that address major operational and feature challenges.

MySQL Binlog Server MVP: This component aims to solve the problem of quick disaster recovery by acting as a stable, reliable replication source. It enables Precise Point-in-Time Recovery (PITR) using simple time or GTID coordinates.
Native Vector Support MVP: This feature allows users to eliminate the complexity of using a separate vector database. You can store, index, and search vector embeddings directly in MySQL, allowing you to combine vector searches with standard business logic in a single, transactional query

Our Community Focus

A common theme from the use cases was that while open source adoption is high, operational teams often lack the proper support and visibility.

Percona’s goal is to support the community by providing:

Stability when under heavy load or during maintenance.
Faster Troubleshooting with better monitoring and observability.
Safer Deployments through expert configuration and security support.

Thank you to everyone who joined us in London for a dynamic event. We hope the insights gained will help you with your open source database deployments.

The conversations continue in the Percona Community! You can reach out directly to the speakers:

Martin Visser (Valkey Technical Lead) LinkedIn
Dennis Kittrell (MySQL Product Manager) LinkedIn
Alastair Turner (Postgres Community Advocate) LinkedIn
Takis Stathopoulos (Enterprise Architect) LinkedIn
Andre Pons (Enterprise Sales Manager) LinkedIn

Join the Percona Community Conversation!

TDE is now available for PostgreSQL 18

Jan Wieremjewicz — Fri, 28 Nov 2025 11:00:00 UTC

Back in October, before PGConf.EU, I explained the issues impacting the prolonged wait for TDE in PostgreSQL 18. Explanations were needed as users were buzzing with anticipation, and they deserved to understand what caused the delays and what the roadmap looked like. In that blog post I have shared that due to one of the features newly added in 18.0, the Asynchronous IO (AIO), we have decided to give ourselves time until 18.1 has been released to provide a build with TDE. We wanted to ensure best quality of the solution and that takes time.

As planned in the PostgreSQL Development Group (PGDG) roadmap, on November 13, the Community released PostgreSQL 18.1, and Percona engineers managed not only to deliver TDE compatible with the PostgreSQL 18 changes, but to fully support them. We are also skipping PostgreSQL 18.0 entirely in our distribution. The first TDE enabled release will be Percona Distribution for PostgreSQL 18.1.1, aligning our builds directly with the first PostgreSQL 18 minor release. Here are some details on what’s new!

Percona PostgreSQL + TDE + AIO

Today, we’re proud to announce that Percona now ships PostgreSQL 18.1 with fully supported TDE and AIO from the very beginning.

No patching, no workarounds, no “experimental caveats.” Encryption-at-rest is not just compatible with AIO. It’s supported, integrated and ready for production deployments across the ecosystem, enabling hardened PostgreSQL environments to meet compliance requirements with confidence as beginning with PostgreSQL 18.1, Percona:

fully supports native TDE
ships AIO-enabled builds, aligned with Community PostgreSQL
provides production-ready packages for enterprise deployments

TDE matures and pg_tde returns home In 2026 we are planning significant investment to ensure that TDE continues to evolve for Community PostgreSQL. As part of this renewed focus, Percona is shifting pg_tde back to its dedicated home:

➡️ https://github.com/percona/pg_tde

This reflects pg_tde’s new role. With the growing number of pg_tde users, it can no longer be treated as a stopgap solution filling a gap in PostgreSQL. Instead, we want to approach it as a complementary, advanced encryption layer for the PostgreSQL 18+ era:

a place to deliver extended capabilities for data-at-rest encryption as soon as they are ready
a hub for integrations requested by the community
a safe space for feedback, comments, and questions to help drive the future of TDE

Releasing this new version brings some real improvements, especially on the KMS integration side.

KMS improvements

We expanded Key Management Service (KMS) capabilities based directly on user and customer feedback. Whether it came through GitHub, Percona Community Forums, events, or direct conversations — thank you! Your input shapes the roadmap.

pg_tde now works with Akeyless KMS

Akeyless is gaining traction among organizations implementing zero-trust security models. pg_tde now integrates cleanly with Akeyless, enabling robust key retrieval and lifecycle management across cloud and on-prem deployments using KMIP.

HashiCorp Vault & OpenBao Namespace Support

Vault and OpenBao users can now take advantage of namespaces when storing encryption keys

Why do namespaces matter?

Multi-tenancy: isolate key access for teams, environments, or applications
Security boundaries: each namespace can enforce its own authentication and audit policies
Cleaner CI/CD: dev/staging/prod can share a consistent key path structure
Delegation & separation of duties: security teams manage root policies, while application teams manage their own namespaces

In large organizations, namespace support isn’t just a convenience, it’s a requirement. OpenBao prioritized this early and released it back in May 2025. With pg_tde now supporting namespaces natively, PostgreSQL deployments gain enterprise-grade key management flexibility.

What Comes Next?

Our commitment remains unchanged: TDE must be a first-class, accessible, community-driven feature in PostgreSQL. Share your feedback and let us achieve this! We can build the future of an open source TDE solution for PostgreSQL together with full openness and transparency!

Winter may be coming, but no weather can stop us! Expect more from TDE soon:

Key length configurations are coming - expect to be able to use 256-bit keys with TDE soon!
extended KMS integrations - we’re already looking into cloud KMS support. Please share which ones you are using and what use cases you want supported!
improvements to pg_tde encryption targets - temporary files are next to be explored followed by system catalog
compatibility with Community PostgreSQL - we want to drive inclusion of any changes required by pg_tde in Community PostgreSQL so that the extension can run for users of all PostgreSQL builds!

⠀At the risk of repeating myself, the deserved highlight goes to the fact that we build based on your feedback, so don’t be strangers!

We Want to Hear from You

Tell us how you’re using PostgreSQL with TDE. Tell us what your security requirements look like. Tell us where tooling can be improved. You can reach us at:

Percona Community Forums https://forums.percona.com/c/postgresql/25
pg_tde GitHub Repo https://github.com/percona/pg_tde
Issues & Discussions
- https://github.com/percona/pg_tde/issues
- https://github.com/percona/pg_tde/discussions

Or, if you see us at an event, come chat with us in person!

The Right Tool for the Job

Wayne Leutwyler — Mon, 24 Nov 2025 00:00:00 UTC

When I first got into woodworking, my mentor shared a piece of advice that has stuck with me ever since: “Use the right tool for the job.” You wouldn’t reach for a belt sander to flatten a board when a planer can accomplish the task faster, cleaner, and with far better results.

The same principle applies in the world of database engineering. When working with MySQL or Percona Server, choosing the correct tool can be the difference between efficient diagnostics and unnecessary downtime.

In this post, I’ll highlight several of the most practical and commonly used utilities from the Percona Toolkit. While the toolkit includes many powerful commands, I’ll focus on the ones that provide the most value in day-to-day operations, troubleshooting, and gathering actionable details for support cases.

PT Summary

A Percona Toolkit utility that provides a concise, high-level overview of a system’s hardware, OS configuration and performance-related metrics. It’s designed to quickly capture the essential details needed for diagnostics or support cases—CPU, memory, disk layout, kernel parameters and more all in a single, easy-to-read report.

Example

Run pt-summary with no arguments to generate a full system summary. When possible, run it with sudo to allow the tool to collect additional details that require elevated privileges:

sudo pt-summary
# Percona Toolkit System Summary Report ######################
 Date | 2025-11-24 17:15:19 UTC (local TZ: EST -0500)
 Hostname | pi16gb
 Uptime | 41 days, 2:27, 4 users, load average: 0.00, 0.00, 0.00
 Platform | Linux
 Release | Debian GNU/Linux 12 (bookworm) (bookworm)
 Kernel | 6.12.47+rpt-rpi-2712
Architecture | CPU = 32-bit, OS = 64-bit
 Threading | NPTL 2.36
 SELinux | No SELinux detected
 Virtualized | No virtualization detected
# Processor ##################################################
 Processors | physical = 4, cores = 0, virtual = 4, hyperthreading = no
 Speeds |
 Models |
 Caches |
 Designation Configuration Size Associativity
 ========================= ============================== ======== ======================
# Memory #####################################################
 Total | 15.8G
 Free | 675.0M
 Used | physical = 5.3G, swap allocated = 512.0M, swap used = 0.0, virtual = 5.3G
 Shared | 44.7M
 Buffers | 10.6G
 Caches | 10.5G
 Dirty | 128 kB
 UsedRSS | 5.1G
 Swappiness | 60
 DirtyPolicy | 20, 10
 DirtyStatus | 0, 0
 Locator Size Speed Form Factor Type Type Detail
 ========= ======== ================= ============= ============= ===========
# Mounted Filesystems ########################################
 Filesystem Size Used Type Opts Mountpoint
 /dev/nvme0n1p1 510M 14%
 /dev/nvme0n1p2 458G 5%
 /dev/sda1 117G 16%
# Disk Schedulers And Queue Size #############################
 nvme0n1 | [none] 255
 sda | [mq-deadline] 60
# Disk Partitioning ##########################################
# Kernel Inode State #########################################
dentry-state | 107782 98346 45 0 32304 0
 file-nr | 3680 0 9223372036854775807
 inode-nr | 99614 20818
# LVM Volumes ################################################
Unable to collect information
# LVM Volume Groups ##########################################
Unable to collect information
# RAID Controller ############################################
 Controller | No RAID controller detected
# Network Config #############################################
 Controller | 00.0 Ethernet controller
 FIN Timeout | 60
 Port Range | 60999
# Interface Statistics #######################################
 interface rx_bytes rx_packets rx_errors tx_bytes tx_packets tx_errors
 ========= ========= ========== ========== ========== ========== ==========
 lo 6000000000 175000 0 6000000000 175000 0
 eth0 0 0 0 0 0 0
 wlan0 5000000000 30000000 0 15000000000 22500000 0
# Network Devices ############################################
 Device Speed Duplex
 ========= ========= =========
 eth0 Unknown! Unknown!
# Network Connections ########################################
 Connections from remote IP addresses
 192.168.1.91 1
 192.168.1.251 1
 2603 2
 Connections to local IP addresses
 192.168.1.145 2
 2603 2
 Connections to top 10 local ports
 3306 2
 6011:ef0:7260:::22 2
 States of connections
 ESTABLISHED 3
 LISTEN 6
 TIME_WAIT 1
# Top Processes ##############################################
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 95842 root 20 0 0 0 0 I 6.7 0.0 0:00.08 kworker+
 1 root 20 0 169520 13088 8672 S 0.0 0.1 0:18.56 systemd
 2 root 20 0 0 0 0 S 0.0 0.0 0:01.70 kthreadd
 3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pool_wo+
 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 6 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 7 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
 8 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker+
# Notable Processes ##########################################
 PID OOM COMMAND
 ? ? sshd doesn't appear to be running
# Simplified and fuzzy rounded vmstat (wait please) ##########
 procs ---swap-- -----io---- ---system---- --------cpu--------
 r b si so bi bo ir cs us sy il wa st
 2 0 0 0 1 6 100 150 0 0 100 0 0
 1 0 0 0 0 0 1750 3000 1 3 97 0 0
 1 0 0 0 0 0 250 400 0 0 100 0 0
 1 0 0 0 0 0 300 450 0 0 100 0 0
 1 0 0 0 0 0 300 450 0 0 100 0 0
# Memory management ##########################################
# The End ####################################################

Redirect output to a file.

pt-summary > server-summary.txt

PT MySQL Summary

A Percona Toolkit utility that collects and displays a concise overview of a MySQL or Percona Server instance, including key configuration settings, performance metrics, storage engine details, replication status, buffer pool usage, and important global variables. It provides a fast, structured snapshot of the database environment, making it ideal for troubleshooting, tuning, and preparing information for support teams.

Example

pt-mysql-summary
# Percona Toolkit MySQL Summary Report #######################
 System time | 2025-11-24 17:45:54 UTC (local TZ: EST -0500)
# Instances ##################################################
 Port Data Directory Nice OOM Socket
 ===== ========================== ==== === ======
 3306 /data0/mysql/data/ 0 0 /usr/local/mysql/mysql.sock
# MySQL Executable ###########################################
 Path to executable | /usr/local/mysql/bin/mysqld
 Has symbols | Yes
# Report On Port 3306 ########################################
 User | wayne@localhost
 Time | 2025-11-24 12:45:54 (EST)
 Hostname | pi16gb
 Version | 8.4.6-6 Source distribution
 Built On | Linux aarch64
 Started | 2025-10-14 09:48 (up 41+02:57:35)
 Databases | 10
 Datadir | /data0/mysql/data/
 Processes | 2 connected, 2 running
 Replication | Is not a replica, has 1 replicas connected
 Pidfile | /usr/local/mysql/mysqld.pid (exists)
# Processlist ################################################

 Command COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 Binlog Dump GTID 1 1 3000000 3000000
 Query 1 1 0 0

 User COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 replication 1 1 3000000 3000000
 wayne 1 1 0 0

 Host COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 192.168.1.251 1 1 3000000 3000000
 localhost 1 1 0 0

 db COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 NULL 2 2 3000000 3000000

 State COUNT(*) Working SUM(Time) MAX(Time)
 ------------------------------ -------- ------- --------- ---------
 init 1 1 0 0
 Source has sent all binlog to 1 1 3000000 3000000

# Status Counters (Wait 10 Seconds) ##########################
Variable Per day Per second 11 secs
Aborted_clients 1
Binlog_snapshot_position 350000 4
Binlog_cache_use 6000
Bytes_received 20000000 225 600
Bytes_sent 2250000000 25000 4000
[...]
Table_open_cache_misses 400
Table_open_cache_overflows 225
Threads_created 9
Uptime 90000 1 1
# Table cache ################################################
 Size | 1000
 Usage | 100%
# Key Percona Server features ################################
 Table & Index Stats | Disabled
 Multiple I/O Threads | Enabled
 Corruption Resilient | Enabled
 Durable Replication | Not Supported
 Import InnoDB Tables | Not Supported
 Fast Server Restarts | Not Supported
 Enhanced Logging | Disabled
 Replica Perf Logging | Disabled
 Response Time Hist. | Not Supported
 Smooth Flushing | Not Supported
 HandlerSocket NoSQL | Not Supported
 Fast Hash UDFs | Unknown
# Percona XtraDB Cluster #####################################
# Plugins ####################################################
 InnoDB compression | ACTIVE
# Schema #####################################################
Specify --databases or --all-databases to dump and summarize schemas
# Noteworthy Technologies ####################################
 SSL | Yes
 Explicit LOCK TABLES | No
 Delayed Insert | No
 XA Transactions | No
 NDB Cluster | No
 Prepared Statements | Yes
 Prepared statement count | 0
# InnoDB #####################################################
 Version | 8.4.6-6
 Buffer Pool Size | 8.0G
 Buffer Pool Fill | 30%
 Buffer Pool Dirty | 0%
 File Per Table | ON
 Page Size | 16k
 Log File Size | 2 * 48.0M = 96.0M
 Log Buffer Size | 64M
 Flush Method | O_DIRECT
 Flush Log At Commit | 2
 XA Support |
 Checksums |
 Doublewrite | ON
 R/W I/O Threads | 4 4
 I/O Capacity | 200
 Thread Concurrency | 4
 Concurrency Tickets | 5000
 Commit Concurrency | 0
 Txn Isolation Level | REPEATABLE-READ
 Adaptive Flushing | ON
 Adaptive Checkpoint |
 Checkpoint Age | 0
 InnoDB Queue | 0 queries inside InnoDB, 0 queries in queue
 Oldest Transaction | 0 Seconds
 History List Len | 1
 Read Views | 0
 Undo Log Entries | 0 transactions, 0 total undo, 0 max undo
 Pending I/O Reads | 0 buf pool reads, 0 normal AIO, 0 ibuf AIO, 0 preads
 Pending I/O Writes | 0 buf pool (0 LRU, 0 flush list, 0 page); 0 AIO, 0 sync, 0 log IO (0 log, 0 chkp); 1 pwrites
 Pending I/O Flushes | 0 buf pool, 0 log
 Transaction States | 3xnot started
# MyISAM #####################################################
 Key Cache | 8.0M
 Pct Used | 20%
 Unflushed | 0%
# Security ###################################################
 Users | 8 users, 0 anon, 0 w/o pw, 7 old pw
 Old Passwords |
# Encryption #################################################
No keyring plugins found
# Binary Logging #############################################
 Binlogs | 3
 Zero-Sized | 0
 Total Size | 437.9M
 binlog_format | ROW
 expire_logs_days |
 sync_binlog | 0
 server_id | 10
 binlog_do_db |
 binlog_ignore_db |
# Noteworthy Variables #######################################
 Auto-Inc Incr/Offset | 1/1
 default_storage_engine | InnoDB
 flush_time | 0
 init_connect |
 init_file |
 sql_mode | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
 join_buffer_size | 256k
 sort_buffer_size | 256k
 read_buffer_size | 128k
 read_rnd_buffer_size | 256k
 bulk_insert_buffer | 0.00
 max_heap_table_size | 16M
 tmp_table_size | 16M
 max_allowed_packet | 64M
 thread_stack | 1M
 log |
 log_error | /var/log/mysql/mysqld.log
 log_warnings |
 log_slow_queries |
log_queries_not_using_indexes | OFF
 log_replica_updates | ON
# Configuration File #########################################
 Config File | /etc/my.cnf

[mysqld]
character-set-server = utf8mb4
authentication_policy = '*'
port = 3306
socket = /usr/local/mysql/mysql.sock
pid-file = /usr/local/mysql/mysqld.pid
basedir = /usr/local/mysql/
datadir = /data0/mysql/data/
tmpdir = /data0/mysql/tmp/
general_log_file = /var/log/mysql/mysql-general.log
log-error = /var/log/mysql/mysqld.log
slow_query_log_file = /var/log/mysql/slow_query.log
[...]
innodb_data_home_dir = /data0/mysql/data/
innodb_log_group_home_dir = /data0/mysql/data/
innodb_temp_data_file_path = ../tmp/ibtmp1:12M:autoextend:max:8G
innodb_buffer_pool_size = 8G
innodb-redo-log-capacity = 2G
innodb_flush_log_at_trx_commit = 2
innodb_lock_wait_timeout = 50
innodb_flush_method = O_DIRECT
innodb_file_per_table = 1
innodb_io_capacity = 200
innodb_buffer_pool_instances = 8
innodb_thread_concurrency = 4

# Memory management library ##################################
jemalloc is not enabled in mysql config for process with id 788
# The End ####################################################

Redirect output to file.

pt-mysql-summary > percona-server.txt

Capture both pt-summary and pt-mysql-summary into a single file.

pt-summary > percona-server-summary.txt
pt-mysql-summary >> percona-server-summary.txt

PT Online Schema Change

A Percona Toolkit utility that performs online ALTER TABLE operations by creating a shadow copy of the table, applying the schema change to that copy, and keeping it in sync with the original using triggers until it is ready to swap. This workflow minimizes locking and reduces downtime, allowing large production tables to be altered safely with minimal impact on applications. However, it’s important to remind users that long-running queries or transactions holding metadata locks (MDL) on the table will still block the final swap, potentially delaying completion of the schema change.

Examples

Adding a New Column

pt-online-schema-change \
 --alter "ADD COLUMN status TINYINT NOT NULL DEFAULT 0" \
 D=mydb,t=orders \
 --execute

This safely introduces a new column to a busy table without blocking reads or writes. The tool handles the copy, synchronization, and final table swap automatically.

Modifying a Column Type

pt-online-schema-change \
 --alter "MODIFY COLUMN price DECIMAL(10,2)" \
 D=shop,t=products \
 --execute

Changing column definitions—especially on large datasets—can be disruptive using standard SQL. With pt-osc, the migration happens online, keeping applications responsive throughout the operation.

Dropping an Unused Column

pt-online-schema-change \
 --alter "DROP COLUMN old_flag" \
 D=analytics,t=events \
 --execute

Column drops can require a full table rebuild, making them great candidates for pt-osc. This example removes a legacy column while avoiding table locks.

Adding an Index

pt-online-schema-change \
 --alter "ADD INDEX idx_user_id (user_id)" \
 D=app,t=logins \
 --execute

Index creation is another expensive operation for large tables. Here, pt-osc allows the index to be added online, improving performance without interrupting the application.

Changing a Primary Key

pt-online-schema-change \
 --alter "DROP PRIMARY KEY, ADD PRIMARY KEY(id, created_at)" \
 D=orders,t=order_items \
 --execute

Primary key modifications usually require a full table rewrite. pt-osc makes this process safer and easier on production systems by performing the change on a temporary shadow table.

Performing a Dry Run

pt-online-schema-change \
 --alter "ADD COLUMN test INT" \
 D=mydb,t=mytable \
 --dry-run

A dry run allows you to validate the plan and review the process without making any actual changes—a critical safeguard when preparing for production schema work.

Printing SQL Changes Before Execution

pt-online-schema-change \
 --alter "ADD COLUMN updated_at TIMESTAMP NULL" \
 D=crm,t=customers \
 --print \
 --execute

Using –print provides transparency into the SQL operations the tool will perform. This is particularly useful during code reviews or change-control processes.

PT Show Grants

A Percona Toolkit utility that extracts MySQL user accounts and privileges and outputs them as clean, executable CREATE USER and GRANT statements. It normalizes and orders the privileges for readability, making it valuable for auditing security, documenting access, migrating users between servers, or preparing accurate privilege information for support and compliance purposes.

Examples

Dump All Grants for All Users

The simplest and most common use case is generating a complete privilege snapshot:

pt-show-grants

This returns normalized CREATE USER and GRANT statements for every account in the instance. It’s ideal for audits, environment comparisons, and creating human-readable privilege reports.

Show Grants for a Specific User

If you want to inspect privileges for a single account, you can filter by user/host:

pt-show-grants --accounts='user@localhost'

This makes privilege debugging and user-level audits quick and targeted.

Export All Grants to a File

To create a reusable backup of every user account:

pt-show-grants > grants.sql

The resulting file is a set of CREATE USER and GRANT statements that can be restored simply by executing:

mysql < grants.sql

This is an excellent practice before server upgrades, user cleanup, or major permission changes.

Show Grants for Multiple Accounts

You can provide a comma-separated list of accounts to extract only what you need:

pt-show-grants --accounts='app@%,reporting@localhost,backup@localhost'

This is ideal for teams that manage groups of service accounts across environments.

Ignore Specific System Accounts

For cleanup scripts or custom inventory reports, skip built-in MySQL accounts:

pt-show-grants --ignore='mysql.sys@localhost,mysql.infoschema@localhost'

This focuses output on only the accounts relevant to your application.

Summary

This post highlights the importance of using the right tool for the job—both in woodworking and in database engineering. For MySQL and Percona Server environments, the Percona Toolkit offers a set of powerful utilities that simplify diagnostics, troubleshooting, schema changes, and security audits.

It introduces four key tools:

pt-summary – Generates a high-level report of system hardware, OS settings, filesystems, networking, and performance metrics. Useful for support cases and quick environment overviews.
pt-mysql-summary – Produces a structured snapshot of a MySQL instance, including configuration, performance counters, replication status, storage engine details, and important variables. Ideal for tuning and issue analysis.
pt-online-schema-change – Enables online ALTER TABLE operations by copying and syncing the table in the background, minimizing downtime. Several examples show how to add, drop, or modify columns and indexes safely.
pt-show-grants – Extracts all MySQL users and privileges into clean, reproducible CREATE USER and GRANT statements. Helpful for audits, migrations, backups, and security reviews.

Percona Operator for MySQL Is Now GA, More MySQL Options for the Community on Kubernetes

Edith Puclla — Wed, 19 Nov 2025 11:00:00 UTC

We’re excited to share that the new Percona Operator for MySQL (based on Percona Server for MySQL) is officially in General Availability (GA)!

This release introduces native MySQL Group Replication support for Kubernetes, providing our community with another open-source option for running reliable, consistent MySQL clusters at scale.

This is about more choices for the community. Each MySQL replication technology addresses different real-world needs, and now you can choose the one that best fits your workloads.

What This Means for the Community

With this release, Percona now supports two fully open-source MySQL Operators:

1. Percona Operator for MySQL (Percona Server for MySQL), New and GA

Group Replication (synchronous)
Asynchronous replication (Technical Preview)
Native MySQL experience
Auto-failover
Kubernetes-native design

2. Percona XtraDB Cluster Operator (PXC)

Galera-based synchronous replication
Strong high availability
Auto-failover
Battle-tested for mission-critical workloads

These Operators complement each other; they are not replacements. They give users the freedom to choose the right replication model for their business and technical priorities.

This GA release is a step in that direction, and we will continue publishing technical blog posts to explain when to use each Operator, how Group Replication works, and how this all fits into real-world Kubernetes environments.

Call for Community Testing and Feedback

Asynchronous replication is now available in Technical Preview, we invite you to:

Test it in your clusters
Share your feedback
Open GitHub issues
Contribute docs or examples

Your feedback will guide the next features we bring to the Operator.

Explore Percona Operator for MySQL:

OIDC in PostgreSQL: How It Works and Staying Secure

Zsolt Parragi — Mon, 17 Nov 2025 09:00:00 UTC

In the previous blog post about the topic, OAuth, OIDC and validators, we discussed basic terminologies to understand the differences between the protocols and how they relate to PostgreSQL.

In this second part, we’ll go one step further and see how OIDC works exactly in other software and in PostgreSQL, and what OAuthBearer is about. We also focus on the possible attacks and dangers in this flow with some examples to showcase why it’s important to use a properly configured secure provider and to teach our users not to just skim through the authorization process.

Can you keep a secret?

Even the original OAuth RFC was designed to work in many different situations, and later extensions made it even more generic to support more use cases. However, it also has to acknowledge that different setups have different requirements and limitations and, because of that, might require a different authorization flow.

In one of our previous examples, we used applications such as CloudStorage or EditorApp. We’ll continue to use them for a while for simplicity, but I want to emphasize that this example, even though it uses different names, is relevant and important for understanding how OAuth and OIDC work together with PostgreSQL.

For CloudStorage, we can safely assume it’s a traditional backend-frontend application, since it has to store the data somewhere on a backend.
For EditorApp, the answer can be different. Since it doesn’t store photos directly but retrieves/saves them on CloudStorage, and also doesn’t store users directly but retrieves them from the Provider using OIDC, it has no requirement for a backend. It is easily imaginable as a single page application, completely written in a modern JS framework, or even a downloadable version of it with Electron or a similar framework.

But why is this distinction important for us? Because we actually have to authenticate and authorize multiple actors here, not just users:

The applications, CloudStorage and EditorApp, have to trust the Provider to supply them with proper information about their users and their permissions.
The OAuth Provider has to identify that it’s speaking to CloudStorage or EditorApp to tell them what exactly they can do with their users.

OAuth or OIDC providers don’t just register users – they also register client applications, and these applications receive either a randomly generated or user-specified identifier (“username”), depending on the Provider in question. When we talk about authentication, that usually involves multiple credentials, such as a username and a password, not just a username. The OAuth standard also recognizes that passwords are good practice and lets client applications use them. This is called a client secret.

But are all applications equal in this sense? Can all of them keep secrets?

The answer to this question is, unfortunately, no. In our example, CloudStorage has a backend – this means it can keep its secret on the backend, never sending it to the frontend. This is how we usually treat our passwords, and this is what OAuth calls a confidential client.

EditorApp is, however, in a different situation. It’s an application purely implemented in HTML and JavaScript, without any backend code. Anybody can download its source, open it in a browser, and start using it. It’s not capable of hiding its secret. Even if it has a client secret – which is optional in OAuth exactly because of this situation – a user with sufficient knowledge can extract this secret from its code. And this is true even if it’s not a JavaScript application but rather a traditional desktop application written in a compiled language. As long as it doesn’t have a backend, a part hidden from its users, there’s no way to completely hide the secret. This is called a public client in OAuth.

Are you trying to log in to application “X”?

So how can the Provider make sure it’s talking to EditorApp and not to something else impersonating EditorApp? Without a secret, without the client being confidential, it can never be sure about this.

As long as we’re talking about web applications, which usually have URLs, it can try to minimize the chance of impersonation. Providers can make sure that the redirects done during the authorization flow go to URLs configured by the client administrators / owners. But even this can be circumvented, and in the case of desktop applications, it’s not usable.

This is why the Provider uses a different strategy. Since it can’t validate for itself that the thing claiming to be EditorApp is really EditorApp or something else, it informs the user. When the user clicks on the “Login with ” button, instead of silently authorizing the login, it first displays an information screen to the user. This is called the consent screen, and it has to display two things:

That user is trying to log in to application
That is requesting permissions to read and write

Along with this information, it has to ask the question: Is the above information correct? Is really trying to log in to ? Is okay with granting these permissions to ?

And only if the user, agrees can the request proceed.

Practical limitations

The above all sounds very nice and secure, but unfortunately we have to discuss some problems with it in practice.

Most importantly, users have a tendency to click “next, next, next” without properly reading, especially if they’re completing the same login flow for the 100th time. Even if a Provider implements the RFCs perfectly in the most secure way possible, it doesn’t help if the user just wants to get through the process quickly and doesn’t check the details about the application name and permissions properly.

Another issue is that providers don’t always properly follow the RFCs. For some providers, it’s configurable – administrators can choose when the consent screen is displayed. They can choose to only display it once when the user first logs into an application, to display it every time the user logs in to the application, or to never display it. Some providers allow these configuration settings for the permissions, for the application name, or both. Some providers simply never display permissions and always just say something like “be sure to trust this application ”.

Unfortunately, during our testing we found providers that in some cases completely skipped the consent screen, even when we tried to explicitly configure that it’s always required. Be skeptical about the authorization flow used by the Provider, verify that it’s secure enough, and don’t hesitate to change providers or report bug reports to the developers if something isn’t as it should be!

Limited devices

Everything we discussed above is nice, but PostgreSQL isn’t a website. Something using it might be one, and in that case, everything we discussed above applies – but to go back to the only currently supported client, psql, that’s a console application and can’t display a graphical web browser for login.

OAuth also thought about similar clients – not exactly console applications, but primarily devices that can’t display a browser and process a normal login flow to the Provider. These are called limited devices because the primary goal of this extension, RFC 8628, was to support specialized hardware where the user either can’t or doesn’t want to log in.

A typical example is a smart TV. While it can display a browser and a virtual keyboard, I wouldn’t want to type my password and log in to a TV using that. But these devices can be even more limited. The only requirement for them is that they should be able to display a verification code and instruct the user where (URL) to enter that verification code on another device, which is capable and secure enough to handle the normal password login process or where the user is already logged in.

Other than this indirection, the login process is similar. The user sees the code, opens the device login webpage on another device, enters the code, and then receives a similar consent screen as before. This consent screen states that device/application is trying to log in and requests permissions . The user then clicks the approve button, while in the background the device periodically checks for approval. Once approved, the authorization proceeds as normal on the limited device.

sequenceDiagram
participant Device as Limited Device
(psql)
participant Provider as OIDC Provider
participant Browser as User's Browser
(phone/laptop)
Device->>Provider: Request device code
Provider->>Device: Device code: ABC123
URL: provider.com/device
Note over Device: Display code and URL
to user
Browser->>Provider: Navigate to provider.com/device
Provider->>Browser: Show code entry form
Browser->>Provider: Enter code: ABC123
Note over Provider: Verify code is valid
Provider->>Browser: Show consent screen
"psql is requesting access"
Browser->>Provider: User clicks "Approve"
loop Polling every few seconds
Device->>Provider: Is code ABC123 approved?
Provider->>Device: Not yet...
end
Device->>Provider: Is code ABC123 approved?
Provider->>Device: Yes! Here's your access token
Note over Device: Token received,
authentication complete

While psql isn’t strictly a limited device, it’s in a similar situation. It’s possible that somebody is using it directly on a computer with a UI, with a web browser already logged in to the OIDC provider, but this is an unlikely scenario. More likely, the user has an SSH session open to another computer, running psql in it. Even if that remote computer has a graphical user interface and browser installed, displaying a login page on it wouldn’t help – the user wouldn’t see it. But it’s much more likely that it’s a console-only virtual machine / container somewhere, without a proper way to do a graphical login. In this sense, psql together with its environment is a limited device.

Before we talk about vulnerabilities and staying secure, it’s important to clarify something. OAuth/OIDC supports many different authentication workflows, including but not limited to the possibilities listed above. Support for specific flows also varies between vendors – the supported features and nuances vary from Provider to Provider.

But if a vendor supports a specific flow, that doesn’t mean it’s automatically usable by all clients. Every provider we tested so far lets the administrators configure which authorization flows they want to support. If a specific application only works with confidential clients, the best configuration is to disable anything else.

The PostgreSQL wire protocol (and server) doesn’t enforce the use of any specific flow, but a generic validator plugin also can’t assume that the server uses some specific configuration. This isn’t something the validator is capable of checking, as it gets executed on the server, after the authorization flow already concluded on the client side.

Strictly speaking, the server doesn’t even know the ID of the client (application), only the issuer URL.

And if anyone wants to be able to use the psql command, that only supports the limited device flow.

OAuthBearer

I already mentioned this in the previous blog post, and a few times in this one. It’s important to understand that the PostgreSQL project has multiple different roles in these authorization flows. The client and the server/validator are two different actors, and the client can be any client, not just libpq. It can be either a well-known third-party implementation of the wire protocol or a completely custom implementation created by an attacker with malicious intent.

From a security standpoint, the server/validator can’t trust the client or assume anything about the authentication process done by the client.

This design isn’t unique to PostgreSQL. Support for OAuth over non-HTTP protocols, using the SASL mechanism (which was already supported by PostgreSQL previously), is called OAuthBearer, RFC 7628, and it’s implemented by PostgreSQL similarly to how other software does it.

The idea of OAuthBearer is that the client using the non-HTTP service (in our case, PostgreSQL) creates an access token in some way. Then this client can use this access token to connect to PostgreSQL and possibly other services. Even if the client uses multiple services that all authenticate with OIDC, it only has to complete the OAuth flow once.

On the other side, the server doesn’t have to do anything else with OAuth other than validating that the token it received is correct and valid. It doesn’t have to deal with multiple flows. On the other hand, it can’t assume anything about the flow.

Can you please enter the code?

Why are the above details important, and why did I repeat the same description multiple times, worded slightly differently?

Because OIDC with public clients, even if implemented properly without mistakes, is vulnerable to the human factor. And in practice, most of the time, PostgreSQL and OIDC means using public clients.

There are multiple possible attack vectors that were previously used, and are still being used, to gain access to services, and there’s no way to fully secure against them. Both are enabled by the fact that with OAuthBearer, we have no control over the authentication process – we have to trust that the access token sent to us by the client was indeed created by the user with the intent to log in to this server.

One very simple “attack” against the device authentication flow is to ask the user nicely. Since the two parts of the process – requesting and using the device code, and verifying the device code – can happen at two different locations in a completely valid setup, there’s not much any software can do programmatically.

In this scenario, the attacker sends an email or calls the victim and comes up with some reason why the user has to go immediately to the device login page and enter the code. The reason is, of course, usually something completely unrelated, like “this is the code to join the meeting,” or “go to the website and enter this code to verify your account,” or anything similar. This is called device code phishing, and it’s made worse by the fact that some providers don’t display detailed enough consent screens during device code authentication, placing even careful users in vulnerable situations.

sequenceDiagram
participant Attacker
participant Provider as OIDC Provider
participant Victim as Victim's Browser
participant PG as PostgreSQL Database
Attacker->>Provider: Request device code
Provider->>Attacker: Code: XYZ789
URL: provider.com/device
Note over Attacker: Attacker now has
device code
Attacker->>Victim: Email: "Enter code XYZ789
at provider.com/device
to verify your account"
Victim->>Provider: Navigate to provider.com/device
Provider->>Victim: Enter device code
Victim->>Provider: Enter code: XYZ789
Note over Victim: User thinks they're
verifying their account
Provider->>Victim: Show consent screen
"psql requesting database access"
Victim->>Provider: Click "Approve"
(without reading carefully)
Provider->>Attacker: Access token granted!
Note over Attacker,PG: Attacker now has valid token
Attacker->>PG: Connect with stolen token
PG->>Attacker: Connection successful
Note over Attacker,PG: Attacker has full
database access

Unfortunately, this can’t be prevented as long as device code flow is enabled. It can be mitigated to some degree by educating users and making sure that the consent screen is always displayed and is very clear about the request details.

Client ID spoofing

Another easy OAuth/OIDC attack vector relies on the fact that we’re using a public client – meaning that either there isn’t a client secret, or even if there is one, it’s not really a secret. It’s easy to create a legitimate-looking website that uses OIDC for something else and starts an authorization flow… but in the background, it uses the same client credentials as the PostgreSQL client.

This again relies on either a non-securely configured consent screen or a careless user who doesn’t read the details about the request. And compared to the previous example, this doesn’t require the device flow – it can work with other flows too. In the world of LLMs, it’s very easy to create a valid-looking website tailor-made just for this purpose.

sequenceDiagram
participant Victim as User
participant Fake as Fake Website
"Photo Gallery"
participant Provider as OIDC Provider
participant PG as PostgreSQL Database
Victim->>Fake: Visit fake-photo-gallery.com
Fake->>Victim: "Login with your account"
Note over Fake: Uses REAL psql client ID!
(public, can't be hidden)
Victim->>Fake: Click "Login"
Fake->>Provider: OAuth request with
psql client ID
Provider->>Victim: Show consent screen
"psql requesting access"
Note over Victim: User thinks:
"I'm logging into
Photo Gallery"
Victim->>Provider: Click "Approve"
Provider->>Fake: Access token
Note over Fake: Fake site now has
database token!
Fake->>PG: Connect with token
PG->>Fake: Connection successful
Note over Fake,PG: Attacker has database access

Similarly to the previous example, there’s not much we can do to prevent this in the plugin. OAuth has extensions that aim to make the process more secure, such as PKCE (RFC 7636) or DPoP (RFC 9449), preventing specific situations, but none of those help with OAuthBearer. As the entire authorization process happens on the client side, the validator on the server can’t do anything but assume that if the access token is valid, then the user created it intentionally.

Educate your users!

Once more I’d like to emphasize that while we can’t prevent these attack vectors completely, it is possible to minimize the risk by teaching, both your users and administrators.

Teach your users to:

Never enter device codes unless you initiated the login process yourself
Always read the consent screen carefully, even if you’ve seen it before
Verify the application name matches what you’re trying to access
Be suspicious of unexpected emails or messages asking you to enter codes
When in doubt, don’t approve – contact your administrator instead

And your administrators to:

Understand that OIDC Provider selection and configuration isn’t just a checkbox item
If a provider can’t be configured in a secure way, it presents a security vulnerability to any client using it
Always enable and require consent screens for public client flows, for every login, not just for registration
Make sure the consent screen clearly displays the application name and requested permissions

These attacks aren’t theoretical – they’re actively used in the wild. The combination of OAuthBearer’s trust model and the inherent limitations of public clients means that proper provider configuration and user awareness are your primary defenses.

Next steps

Now that we understand how OIDC works with PostgreSQL and the security considerations involved, it’s time to see this in practice.

In the next blog post, we’ll walk through setting up Keycloak with PostgreSQL from scratch. We’ll cover both getting Keycloak running and configuring it securely for PostgreSQL authentication, even if you’ve never worked with Keycloak before. You’ll see exactly how to configure the settings we discussed here and how to test that everything works correctly.

Stay tuned for practical, hands-on setup instructions!

PGScorecard - PostgreSQL Compatibility Index

Kai Wagner — Thu, 13 Nov 2025 00:00:00 UTC

We’re excited to share that our recent test run using the Postgres Compatibility Index (PCI) achieved 100% compatibility.

The PCI was created to bring clarity to the often used but loosely defined term “PostgreSQL compatible.” As Mayur explains in his article The Making of ‘Postgres Is’, the goal is simple: to ensure that when a system claims to be compatible with PostgreSQL, it truly behaves like upstream PostgreSQL in practice. The PCI accomplishes this by running a comprehensive set of tests across features like data types, procedural functions, constraints, extensions, and more, and producing a measurable, transparent score. This gives users and vendors a reliable benchmark rather than relying on marketing claims.

Compatibility matters because many organizations rely on PostgreSQL variants, repackaged distributions, or vendor supported systems. They want the confidence that their schemas, tools, extensions, ORMs, client libraries, and workflows will continue to work as expected. A system that drifts from upstream PostgreSQL can introduce subtle risks such as, unsupported features, migration challenges, and vendor lock-in.

Achieving a perfect PCI score means our system supports the full baseline feature set as currently defined. It demonstrates that users can rely on the same behavior as community PostgreSQL, whether they are self-hosting, using a vendor-supported version, or integrating with existing tools. Importantly, it also shows that you can have a fully open-source system while still benefiting from vendor support, without compromising compatibility.

In a world where “PostgreSQL compatible” is often a vague claim, initiatives like the PCI provide the needed help with transparency and comparability. It helps to protect you from marketing claims in the PostgreSQL ecosystem, ensuring that tooling, and workflows continue to function reliably.

Special thanks to Mayur, whose initiative is helping define a clear, standardized framework for PostgreSQL compatibility.

The test run was done against Percona Server for PostgreSQL 17.6.1. Click for test result. Percona Server for PostgreSQL is a binary-compatible, open source drop-in replacement for PostgreSQL with the currently needed enhancements, to make Transparent Data Encryption (TDE) work.

MySQL Memory Usage: A Guide to Optimization

Roman Agabekov — Tue, 11 Nov 2025 00:00:00 UTC

Struggling with MySQL memory spikes? Knowing how and where memory is allocated can make all the difference in maintaining a fast, reliable database. From global buffers to session-specific allocations, understanding the details of MySQL’s memory management can help you optimize performance and avoid slowdowns. Let’s explore the core elements of MySQL memory usage with best practices for trimming excess in demanding environments.

How MySQL Uses Memory

MySQL dynamically manages memory across several areas to process queries, handle connections, and optimize performance. The two primary areas of memory usage include:

Global Buffers

These are shared by the entire MySQL server and include components like the InnoDB buffer pool, key buffer, and query cache. The InnoDB buffer pool is particularly memory-intensive, especially in data-heavy applications, as it stores frequently accessed data and indexes to speed up queries.

Connection (per thread) Buffers

When a client connects, MySQL allocates memory specifically for that session. This includes sort buffers, join buffers, and temporary table memory. The more concurrent connections you have, the more memory is consumed. Session buffers are critical to monitor in high-traffic environments.

Why MySQL Memory Usage Might Surge

Memory spikes in MySQL often result from specific scenarios or misconfigurations. Here are a few examples:

High Traffic with Large Connection Buffers: A surge in concurrent connections can quickly exhaust memory if sort or join buffers are set too large.
Complex Queries: Queries with large joins, subqueries, or extensive temporary table usage can temporarily allocate significant memory, especially when poorly optimized.
Oversized InnoDB Buffer Pool : Setting the InnoDB buffer pool size too large for the server’s available memory can trigger swapping, severely degrading database and server performance.
Large Temporary Tables : When temporary tables exceed the in-memory limit ( tmp_table_size ), they are written to disk, consuming additional resources and slowing down operations.
Inefficient Indexing : A lack of proper indexes forces MySQL to perform full table scans, increasing memory and CPU usage for even moderately complex queries.

Best Practices for Controlling MySQL Memory Usage

When you notice MySQL using more memory than expected, consider the following strategies:

1. Set Limits on Global Buffers

Configure innodb_buffer_pool_size to 60-70% of available memory for InnoDB-heavy workloads. For smaller workloads, scale it down to avoid overcommitting memory.
Keep innodb_log_buffer_size at a practical level (e.g., 16MB) unless write-heavy workloads demand more.
Adjust key_buffer_size for MyISAM tables, ensuring it remains proportionate to table usage to avoid unnecessary memory allocation.

2. Adjust Connection Buffer Sizes

Reduce sort_buffer_size and join_buffer_size to balance memory usage with query performance, especially in environments with high concurrency.
Optimize tmp_table_size and max_heap_table_size to control in-memory temporary table allocation and avoid excessive disk usage.

3. Fine-Tune Table Caches

Adjust table_open_cache to avoid bottlenecks while considering OS file descriptor limits.
Configure table_definition_cache to manage table metadata efficiently, especially in environments with many tables or foreign key relationships.

4. Control Thread Cache and Connection Limits

Use thread_cache_size to reuse threads effectively and reduce overhead from frequent thread creation.
Adjust thread_stack and net_buffer_length to suit your workload while keeping memory usage scalable.
Limit max_connections to a level appropriate for your workload, preventing excessive session buffers from overwhelming server memory.

5. Track Temporary Table Usage

Monitor temporary table usage and reduce memory pressure by optimizing queries that rely on GROUP BY, ORDER BY, or UNION.

6. Use MySQL Memory Calculator

Incorporate tools like the MySQL Memory Calculator by Releem to estimate memory usage. Input your MySQL configuration values, and the calculator will provide real-time insights into maximum memory usage. This prevents overcommitting your server’s memory and helps allocate resources effectively.

7. Monitor Query Performance

High-memory-consuming queries, such as those with large joins or sorts, queries without indexes, can affect memory usage. Use Releem’s Query Analytics and Optimization feature to determine inefficient queries and gain insights on further tuning opportunities.

Simplifying MySQL Memory Tuning with Releem

Releem takes the guesswork out of MySQL optimization by automatically analyzing your setup and suggesting configuration changes that align with your memory limits and performance needs. Whether you’re dealing with complex workloads or simply don’t have time for manual adjustments, Releem makes it easier to keep MySQL running smoothly.

A thread through my 2025 Postgres events

Alastair Turner — Mon, 10 Nov 2025 07:00:00 UTC

I recently got back from PostgreSQL Conference Europe in Riga, marking the end of my conference activities for 2025. The speakers were great. The audience, for the Extensions Showcase on Community Day on Tuesday and my Kubernetes from the database out talk, were great. The event team was great. The singing at karaoke was terrible, but it’s supposed to be.

After attending a good few events this year, starting with CERN PGDay in mid-January, I wanted to write something about more than just the most recent event. I see a common thread across presentations and sessions at a number of events over the year, that is, scale-out Postgres and particularly, its use in non-profit scientific environments.

The (beginning and) end users

Far fewer data processing challenges require pooling the resources of many physical servers these days, with servers getting bigger and storage faster. Scientific data analysis and managing large, complex scientific facilities still do. I saw three presentations on this: Rafal Kulaga, Antonin Kveton and Martin Zemko’s on managing CERN’s SCADA data; Daniel Krefl and Krzysztof Nienartowicz at CERN on how Sendai queries variable star data; and Jaoquim Oliveira in Riga on managing the European Space Agency’s (ESA’s) survey mission data.

I admit a fondness for ESA’s GAIA catalog dataset. After I was lucky enough to do a proof of concept project on joining it with other catalog data, it has provided significant intellectual interest. Don’t let me get started on the possible ways to optimise computationally expensive inequality joins on horribly skewed data, unless you really care about the problem. My interest in a dataset discussed in two of these talks is not why the thread connecting them is worth commenting on. All three presentations had a lot of content on selecting or developing database technologies for the work they were doing. That’s worth discussing a bit further.

Getting the details right

The thread of sharded, scale out, or Massively Parallel Processing (MPP) Postgres connects end user stories at my first event of the year and my last, along with stories of building this software at events in between. At PGConf.dev in Montreal David Wein gave a very condensed explanation of how AWS’s Aurora Limitless handles distributed snapshot isolation (watch the lightning talk at on YouTube), there was also an unconference session on handling the issue in core Postgres the next day. For an in-depth explanation of of what the distributed snapshot problem is and how it may be addressed, see Ants Aasma’s talk from PGConf.EU 2024

The organisations with the data are looking for open source software solutions and bumping into issues around open core licensing, project contribution breadth, project activity levels, project governance. The Postgres developer community is working on the knottiest of the problems in this space, trying to get it absolutely right. In the mean-time, various forks and extensions are delivering useful functionality for the owners of these big, complex datasets. Useful, but could do better

If this were working out for everyone, there wouldn’t be a story to tell. Sednai are building Potgres-XZ, which builds on TBase, which built on Postgres-XL. The ESAC Science Data Centre (ESDC) is facing a decision between two single-vendor projects, where one vendor doesn’t provide support for on-premises deployments. CERN procurement sought written assurances over license terms for TimescaleDB, since the CERN facilities organisation may be viewed as a service provider to their hosted scientific projects.

This pattern of licenses built specifically to avoid “AWS stealing our innovation/lunch/…”, (and it is always AWS set up as the bogeyman in these stories), is particularly unfortunate here, because it just isn’t true for Postgres. AWS, and Azure, employ big teams of community contributors to work on open source Postgres. The progress on statistics management, asynchronous IO, and vacuum in Postgres 18 are, among others, thanks to these teams’ efforts.

No matter how positive the involvement of the hyperscalers may be for Postgres, there are organisations who will prefer to run their own databases. On-premises hosting is a clear choice for organisations with big facilities capabilities, capital-centric budgeting, extreme requirements and predictable, always on workloads. Many of these organisations are publicly funded scientific projects. It would be great if there were broad-based open source solutions to meet their data management needs.

Doing better, together

At PGConf in Riga the Percona team took a few, early steps towards building a joint effort to deliver the components of such a solution. I hope that the big, open managers of structured scientific data (or their subcontractors, depending on their engagement model) and a few vendors can come together to build event data compression, columnar storage, and all the other bits which can be implemented as extensions.

The current Postgres extensions and forks for scale out systems were built on older versions of Postgres, so they had to build features which now exist in core Postgres. Their implementation of partitioning, for instance, differs subtly from the capabilities now available in modern Postgres. As feature-specific extensions can take over capabilities which are currently intertwined with sharding (like compression in Timescale or columnar storage in Citus), users will be less locked in to vertical stacks of features, some useful to them and some not. Simple sharding can then become a proxy (like pgDog), an automation on DDL on a gateway server or even a core Postgres feature.

Which leaves those special cases where moving data between shards during execution is key to performance. This is mattering less with ever bigger servers, improving Postgres parallelism and tools like DuckDB - but when it matters it still really matters. Here the sons of the ‘plum - CloudberryDB and WarehousePG, forked from Greenplum when it closed source - work their magic (hat tip to Jimmy Angelakos for the “the ‘plum” contraction). Managing that particular capability will always be a big, complex code base. If the patches carried to make it happen shrink as Postgres and extensions fill the gap, we’ll have a more sustainable route to all good database things being openly available.

OAuth, OIDC, validators, what is all this about?

Zsolt Parragi — Fri, 07 Nov 2025 10:00:00 UTC

Somebody might tell you, “let’s configure PostgreSQL 18 with OIDC, it should be simple, only takes a few minutes!” And that might be the case if you already have an OIDC provider set up and know all the details about the protocols, configurations, and possible issues. Or it might take much longer if you just open your favorite search engine and type “What is this OIDC stuff about?”

In this series of blog posts, I’ll try to help with this task. First, by clearing up all the terminology and details in this article. Later, I’ll provide vendor-specific setup instructions for some of the popular providers, using our fully open source pg_oidc_validator plugin.

OAuth 2.0, OIDC, what’s even the difference?

From news and other sources you might have heard that PostgreSQL 18 now has support for OIDC. But if you look at the PostgreSQL documentation about it, or the variables/configuration options PostgreSQL provides, it’s clear that is about OAuth everywhere.

People often use them interchangeably, because the two are closely related: OIDC is built on top of OAuth. However, they serve different purposes.

flowchart TB
OAuth[OAuth 2.0
Authorization Protocol]
OIDC[OpenID Connect OIDC
Authentication Layer]
OAuth --> OIDC
OAuthQ["Can user X access
resource Y?"]
OIDCq["Who is this user?"]
OAuth -.->|Answers| OAuthQ
OIDC -.->|Answers| OIDCq
style OAuth fill:#e3f2fd
style OIDC fill:#f3e5f5
style OAuthQ fill:#fff,stroke:#1976d2,stroke-dasharray: 5 5
style OIDCq fill:#fff,stroke:#7b1fa2,stroke-dasharray: 5 5

Authorization is not Authentication

The “Auth” in OAuth is about Authorization, not Authentication as people often believe – and specifically about remote, distributed authorization involving multiple applications. For example, somebody might store pictures on some cloud storage, and wants to use a photo editing application written and maintained by a different company. This EditorApp reaches out to the CloudStorage, and says “Can you provide me the pictures please?” After this question, CloudStorage has to figure out if it is allowed to do so or not.

OAuth was designed to handle situations like this – managing permissions in a complex online environment. However, EditorApp might also ask the question, “Hey, I want to display the name of the user I’m working with. Can you tell me who I am working with?” And now we’ve leaped into the context of Authentication. OIDC, or OpenID Connect, is a protocol built on top of OAuth to answer questions like this – to provide information about who the user is.

Which of them do we need with PostgreSQL?

The question with PostgreSQL and similar software using OAuth/OIDC is a bit different: we want to answer, “somebody is trying to log in – can I allow this login to proceed?”

Is this the right question, or am I oversimplifying things? Shouldn’t we be asking, “who is trying to log in?” Sometimes we do. The most common setup will likely ask both: “can this login proceed, and if yes, who is the user?”

But not necessarily always. It’s perfectly valid to configure a server to use this login flow only for administrators, where anybody who is allowed to use it gets associated with an internal admin account. In this scenario, we don’t care about the identity provided by the external provider: if the user has a valid access token, we treat them as our admin user and proceed accordingly.

To handle this specific workflow, OAuth is enough: all we have to do is check if the access token is allowed to access the PostgreSQL server, and if yes, the login can proceed as the admin user. This is however a limited, specific use case, not the generic scenario, where we also want to figure out who is the user on the provider side.

And there are also limitations on the server and client side:

When using our validator plugin, it supports the more generic scenario. It has to work with user identities from the provider, so it requires OIDC features. That’s why we called it pg_oidc_validator and not pg_oauth_validator.
While the server and wire protocol can work with any OAuth flow, currently the only client that implements a login mechanism using it is libpq (and with that, the psql command). And while the psql command also uses parameters with OAuth in their name, internally it relies on a feature called OIDC discovery – which, as the name suggests, is part of the OIDC standard, not OAuth.

To summarize: in practice, right now anyone who wants to log in to a PostgreSQL server using OAuth and psql has to use a provider that also supports the OIDC protocol. In practice this isn’t a restriction, since OIDC is commonly supported.

Why do we need this validator?

Why do we need to use a plugin to validate things? Many websites and apps implement login with OIDC providers, and they don’t need separate validators, so: Why can’t PostgreSQL do everything internally in the core?

To answer these questions, we have to understand that PostgreSQL in this context isn’t an application in itself – it’s a part of a complex infrastructure. Even libpq, mentioned above, isn’t strictly part of the picture. There are completely independent implementations of the PostgreSQL wire protocol, which can all implement the client-side flow completely differently.

This bigger system that happens to use PostgreSQL might also use other services that need authorization. The EditorApp in our earlier example might use PostgreSQL as its database. And that means, after authenticating the user, it has to talk to two different services:

CloudStorage, to access the photos on behalf of the user
PostgreSQL, to access the database on behalf of the user

Since users like seamless experiences, the developers of EditorApp want to log the user in only once, internally as part of their application, and then forward this information to both CloudStorage and PostgreSQL.

sequenceDiagram
participant User
participant EditorApp
participant Provider
participant CloudStorage
participant PostgreSQL
User->>EditorApp: Login
EditorApp->>Provider: Request OAuth/OIDC authentication
Provider->>User: Show login page
User->>Provider: Enter credentials
Provider->>EditorApp: Return access token
Note over EditorApp: User authenticated once,
token used for multiple services
EditorApp->>CloudStorage: Access photos (with token)
CloudStorage->>Provider: Validate token
Provider->>CloudStorage: Token valid
CloudStorage->>EditorApp: Return photos
EditorApp->>PostgreSQL: Connect to database (with token)
PostgreSQL->>PostgreSQL: Validator checks token
PostgreSQL->>EditorApp: Connection established

This showcases an important difference compared to “simple” websites and applications: PostgreSQL doesn’t own the entire authentication flow. It receives information – what the OAuth standard calls an access token – and needs to use this to complete its internal authentication/authorization flow. To do this, it has to validate the access token and answer the question: “was this token really created by the issuer I trust?”

There’s nothing new about this question. Big cloud providers like Google, Microsoft, and many others all do SSO (Single sign-on). If you log in to their webmail service, you can also open their cloud storage, calendar, or other services, and you’ll be similarly authenticated and authorized.

However, there is one very important difference in the above example compared to PostgreSQL: they all only work with their own users and their own tokens.

You might think this is still an easy problem: OAuth and OIDC are standards, so all we have to do is read the related parts about how validation works and implement our validation flow accordingly. Except that the standards don’t say anything about it.

These standards never define what an access token is. It can be something completely opaque, only interpretable by the original issuer. Or it can be something completely transparent – a proper JSON document with a digital signature that guarantees it was issued by the issuer, usually called a JWT (JSON Web Token).

Many OAuth providers implement JWT tokens, but not all of them. And even with JWT tokens, there are differences between providers. Since it’s not standardized, the content might differ between vendors. There’s also the question of signature validation, where some providers have differences and require special handling.

And this is where a plugin comes into the picture. The PostgreSQL maintainers decided they don’t want to add vendor-specific code into the core. Implementing JWT access tokens with the most commonly used structures could have been a solution, but that would have meant PostgreSQL had no way to support providers that were completely standards-compliant but implemented access tokens differently.

Even with JWT tokens, we have to think about token revocation. JWT tokens have a specified lifetime. When we validate them strictly based on the digital signature, we also have to check if we’re still within the allocated time. The protocols have a mechanism for refreshing these tokens so that users can stay logged in without repeating the process every time the tokens expire.

But sometimes administrators or the security team might discover a breach – that somebody got hold of an access token that’s still valid for hours or days – and decide to revoke it. They can’t change the tokens already circulating the network, but they can tell their authorization server to start rejecting the token, even if it’s still within its allowed lifetime.

If a validator relies solely on validating the signature of the token, it will still authorize users with these revoked tokens as long as their lifetime allows. If it performs a request to the server to check if the token is still usable, it might provide a more secure experience at the cost of additional HTTP requests. Alternatively, some providers allow applications to subscribe to broadcasts where they announce revocations – so instead of actively querying each token, they can keep a list of which tokens they aren’t allowed to authorize, as this list is typically short.

Unfortunately, with these questions we’ve again arrived at “vendor-specific” territory, where a validator has to work differently for different services. It’s also a user choice:

On a service where logins are rare but security is a top priority, it might make sense to do explicit queries every time.
But on a server processing thousands of login requests every second, these additional requests might slow things down too much. Administrators might decide not to do them, or to cache the results for some period instead.

With all these details and choices, it’s definitely better to leave the options open so everyone can select a validator suited for their specific needs.

What does pg_oidc_validator offer?

The Percona validator in its current form focuses on providers working with JWT tokens. We can’t guarantee it works with all providers using JWT tokens – as mentioned above, the exact format of these tokens isn’t standardized – but we implemented logic that can handle all the providers we tested, in some cases allowing customization with configuration variables.

In the future, we might add support for providers using opaque tokens or additional optional checks for token revocation, but these weren’t in scope for the first version.

If you try it out, and find any issues, or have any suggestions, miss some features, please provide us feedback on Github!

The future

OAuth support in PostgreSQL is new and only implements the bare minimum. Client and application support is also in its early days – adoption will take time. It can already be useful for some use cases, but it might still be too limited for others.

I can’t predict how the code will change in the server itself, but there are certainly many improvements that can be made. I also want to avoid confusion and clear up some things related to my previous examples – things that aren’t currently supported but might be in the future:

Even if a validator plugin checks for revocation, that check is only done during the login process. If a user is already logged in to PostgreSQL and the token is revoked, the user will stay logged in – the validator can’t evict them from the server.
Similarly, while the protocol has this concept of access token lifetime and how applications using the token can refresh it, this is currently ignored by PostgreSQL. Tokens are validated during the login process, and we check the validity, including the lifetime of the token, during that time. We don’t try to refresh these tokens later, and we don’t evict users when the token expires – currently there’s no infrastructure for that.
In the examples above, CloudStorage was able to decide which photos EditorApp can access. OAuth is about authorization, and it could be used for authorization within PostgreSQL – but this isn’t the case currently. Validators have to map access tokens to internal users present in the PostgreSQL database. Other than this, the information provided by the OAuth Provider has no interaction with the internal permissions within PostgreSQL (grants).
In the example above, we described a situation where an application used PostgreSQL together with other services using the same OAuth provider. While this is a possible use case, it’s also possible that an application only needs an OIDC login flow to access PostgreSQL. In that case, it would be possible to provide better security guarantees by integrating the login flow deeply into PostgreSQL, especially when using more recent OAuth/OIDC extensions such as PKCE (Proof Key for Code Exchange).

Similarly, our pg_oidc_validator is a pre-release prototype. While it already performs JWT-based authentication and authorization, it currently doesn’t support opaque token providers at all, has no internal caches, and doesn’t check for revoked tokens. Also, some of the features mentioned above would be better with integrated core support, but could technically also be implemented as part of a validator plugin. This would provide the additional benefit that users wouldn’t have to wait for newer PostgreSQL versions for the new feature – it would be enough to upgrade the validator plugin to a newer version.

So there are lots of improvement opportunities for both parts. Stay tuned and follow the changelogs!

Encryption support in PMM Dump

Sveta Smirnova — Thu, 30 Oct 2025 11:00:00 UTC

The pmm-dump client utility performs a logical backup of the performance metrics collected by the PMM Server and imports them into a different PMM Server instance. PMM Dump allows you to share monitoring data collected by your PMM server with the Percona Support team securely.

Up until now dumps, created by the tool, were not encrypted. It was possible to encrypt them after they are done but this required additional actions from the user.

Starting from the upcoming PMM Dump version 0.8.0-ga released on October 29, 2025, dumps are encrypted by default.

Key points

Dump files are encrypted by default with AES-256-based encryption.
An auto-generated password is produced for each encrypted dump; it is printed at the end of the export operation or can be written to a file with --pass-filepath.
You can provide a custom password with --pass.
Disable encryption with --no-encryption only when you understand the risks.
By default, for encrypted dumps, export logging to STDOUT is suppressed; use --no-just-key to override.

Why this matters

Encrypting PMM dumps prevents accidental exposure of monitoring and query data that may contain sensitive information (query text, hostnames, metrics). It brings PMM Dump in line with secure data-handling best practices and simplifies safe sharing with Percona Support.

Quick examples

Export (encryption enabled by default):

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --allow-insecure-certs
...
Password: ****************
$ ls pmm-dump-.tar.gz.enc

Provide a custom password:

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --pass='My$trongP@ss'

Save auto-generated password to file:

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --pass-filepath=/tmp/pmm-dump.pass

Disable encryption (not recommended):

$ pmm-dump export --pmm-url='https://admin:admin@127.0.0.1' --no-encryption

Import an encrypted dump:

$ pmm-dump import --pmm-url='https://admin:admin@127.0.0.1' --allow-insecure-certs \
--dump-path=pmm-dump-1758017090.tar.gz.enc --pass='My$trongP@ss'

Decrypt an encrypted dump (if needed):

$ openssl enc -d -aes-256-ctr -pbkdf2 -in dump.tar.gz.enc -out dump.tar.gz

Recommendations

Prefer leaving encryption enabled.
Use --pass-filepath to store passwords securely rather than relying on terminal output.
Transfer encrypted archives over secure channels (SCP/SFTP) and share passwords via secure out-of-band channels.

Availability

Encryption support is included starting in the recent PMM Dump 0.8.0-ga release. Check your PMM Dump version (pmm-dump version) and the docs for exact version details.

Additional information

Say Hello to OIDC in PostgreSQL 18!

Jan Wieremjewicz — Wed, 22 Oct 2025 11:00:00 UTC

If you’ve ever wondered how to set up OpenID Connect (OIDC) authentication in PostgreSQL, the wait is almost over.

We’ve spent some time exploring what it would take to make OIDC easier and more reliable to use with PostgreSQL. And now, we’re happy to share the first results of that work.

Why OIDC, and why now?

We’ve spoken to some of our customers and noticed a trend of moving away from LDAP to OIDC. Our MongoDB product is already providing OIDC integration and the team working on PostgreSQL products saw an opportunity coming with PostgreSQL 18.

As some of you may have noticed PostgreSQL 18 has introduced improvements to authentication that include OAuth 2.0 support. It’s a small step from OAuth 2.0 to OIDC, which builds directly on top of it. So we set out to test PostgreSQL 18’s integration with one of the most common OIDC providers: Okta.

That’s when we discovered a missing piece. While PostgreSQL 18 includes the underlying support for OAuth 2.0, it still lacks a validator library required to successfully complete an OIDC configuration. So… we built it.

Closing the gap for enterprise needs

Percona is known for our Support, Managed, and Consulting services for open source databases. But while our services are commercial, our mission remains deeply open source.

We don’t differentiate users by the size of their wallets. We want everyone using open source databases to succeed, grow, and benefit from the same quality foundations that enterprises rely on. Services is what finances our open source investments, what we believe is truly honest and sustainable open source development.

OIDC is a great example of how real-world enterprise needs and community innovation come together. PostgreSQL 18 introduced OAuth thanks to community efforts, but OIDC that helps organizations handle compliance and scale was still not supported. With the work on OIDC validator we want to close the gap between these two while keeping the solution open source.

The power is in testing

It would be great if integrating with one OIDC provider meant it works with all of them. Unfortunately, real world implementations differ, sometimes significantly. That’s why compatibility testing matters. That’s also why often particular providers we find require independent approach and some custom handling in the code.

So far, we’ve done some preliminary tests of the library. It’s not production-ready yet. It’s a first release that’s shared with users and Community to get feedback while in the meantime our engineers will put it through rigorous testing regimen. We’ve so far tested the validator library compatibility with:

✅ Okta

✅ Ping Identity

✅ Keycloak

✅ Microsoft Entra ID (Azure AD)

We’re aware it’s not yet compatible with Google’s OIDC implementation, which has a few unique quirks, but that’s on our roadmap for future work. The broader the testing, the stronger the solution. This is where we hope the Community can join us.

Try the OIDC validator library now!

We’re excited to share that the first release of the OIDC validator library is now available for your feedback. The repository includes:

Basic setup instructions in the README,
Introductory documentation for getting started, and
Links to examples and test configurations.

More detailed guides, including how OIDC works under the hood and how to use it in real PostgreSQL deployments, as well as direct integration guides are coming soon in follow up blog posts and documentation articles.

We’d love your feedback

If you’re experimenting with PostgreSQL 18 or exploring modern authentication options, give the OIDC validator library a try and let us know what you think! Your input will help us make this capability more robust, portable, and enterprise-ready while keeping it open source and accessible to everyone.

Keep Calm - TDE for PostgreSQL 18 Is on Its Way!

Jan Wieremjewicz — Wed, 15 Oct 2025 11:00:00 UTC

If you’ve been following the buzz around PostgreSQL, you’ve probably already heard that database level open source data-at-rest encryption is now available thanks to the Transparent Data Encryption (TDE) extension available in the Percona Distribution for PostgreSQL. So naturally, the next question is:

Where’s Percona Distribution for PostgreSQL 18?

The short answer:

It’s coming.

The slightly longer one:

It’s taking a bit of time, for all the right reasons.

Why the delay?

We’ve been laser-focused on advancing pg_tde maturity, making it ready for your production workloads. We succeeded and now you can use Percona Distribution for PostgreSQL together with TDE on PROD! The next steps for us are: 1 Merging the changes to PostgreSQL 18. 2 Ensuring that the patches making TDE on PostgreSQL possible are accepted by PostgreSQL Community.

As part of both of these efforts, we took the time to thoroughly test TDE against PostgreSQL 18. During that process, we found that to use TDE on PostgreSQL 18 some extra work is needed to ensure the new Asynchronous I/O (AIO) feature works as designed. Since TDE extends the Storage Manager (SMGR), it needs to integrate smoothly with changes introduced by AIO and that’s something we focus on now and are definitely do not want to rush.

Percona stands for quality of services and products. Databases are the fundaments of the information systems, outside of security the most important values are stability and availability. To respect these values we chose not to push a major release just for the sake of timing, but instead to take the time to do things properly.

We’re planning to release TDE with the first minor patch of PostgreSQL 18, currently scheduled for November 13.

Taking our time responsibly

Most production users don’t deploy brand new major releases on day one. Typically before using a new major version on production we’ve seen a minimum on 1-2 minor releases for even the users we’ve seen fastest to adopt new versions.

That gives us a valuable window to make sure everything is rock-solid before it reaches your clusters. We’re using that time wisely: testing, aligning, and polishing so that when you upgrade to PostgreSQL 18 with Percona TDE, it’ll feel like it was part of the core all along.

Oh, and something new is brewing…

As PostgreSQL 18 introduces OAuth Authorization/Authentication we tried using it with OpenID Connect (OIDC). OIDC is an identity layer built on top of OAuth 2.0, adding user authentication to OAuth’s authorization capabilities.

We tried using OIDC from a number of providers and came to a conclusion that there’s currently no easy way of using it with PostgreSQL 18. The key missing component to make it work is a validator library that would allow to validate the identity tokens.

I’m happy to share what we started working on OIDC support for PostgreSQL, built on top of the OAuth support introduced in PG18. The first beta release of our OIDC validator library is coming soon, stay tuned for that!

How to help?

Easy, share your feedback!

Have you tried pg_tde? Tell us what worked, what didn’t, and how it felt in real use.
Looking for OIDC in PostgreSQL? We’d love your thoughts once the beta drops.
Using pg_stat_monitor? Share how you’re using it and helps us make it better.

We’re invested to making open source databases more secure, flexible and ready for whatever you throw at them one careful step at a time.

Audit Log Filters Part II

Wayne Leutwyler — Wed, 08 Oct 2025 00:00:00 UTC

In my first post on the MySQL 8.4 Audit Log Filter component, I covered how to install the component and configure a basic filter that captures all events. The Audit Log Filter framework offers a highly granular and configurable auditing mechanism, enabling administrators to log specific events based on criteria such as user, host, or event type. This selective approach enhances observability, supports compliance initiatives, and minimizes unnecessary logging overhead.

In this follow-up, we’ll take a deeper technical look at defining and optimizing audit log filters to capture only the most relevant database activities—delivering actionable audit data while significantly reducing noise and log volume.

Example 1

Audit all events:

SELECT audit_log_filter_set_filter('log_all_events', '{
 "filter": {"log": true}
}');

Once this filter is created and assigned to a user (for example, with SELECT audit_log_filter_set_user(’%’, ’log_all_events’);), every database event triggered by that user—or by all users if % is used—will be written to the audit log file.

In short:

This is the most permissive audit configuration possible. It’s typically used:

As a baseline test to verify that the audit log component is working.
In diagnostic or forensic scenarios where full visibility is required.

For production environments, however, it’s recommended to create more selective filters (e.g., by event class, command type, or user) to reduce log volume and improve performance. Which we will go into more detail in the upcoming examples.

Example 2

Log table access:

select audit_log_filter_set_filter('log_table_access', '{
 "filter": {
 "class": [
 { "name": "table_access" },
 { "name": "connection" },
 { "name": "general" }
 ]
 }
}');

Included Event Classes

table_access Logs events when MySQL reads from or writes to tables.
- Useful for tracking which users or applications are accessing specific tables.
- Helps in auditing data access patterns and detecting unauthorized data reads/writes.
connection Logs connection-related events such as user logins, logouts, and failed authentication attempts.
- Important for tracking session activity and security auditing.
general Logs general query execution events—like statements sent to the server (e.g., SELECT, INSERT, UPDATE, etc.).
- Useful for general SQL activity auditing.

What It Does Functionally

After this filter is defined and assigned to a user or host (for example, with SELECT audit_log_filter_set_user(’%’, ’log_table_access’);), MySQL will only log events that fall into one of these three classes.

All other event types—like administrative commands, stored program executions, or system-level actions—will be excluded from the audit log.

Why use this filter

This configuration strikes a balance between completeness and efficiency:

Captures key operational and access-related activity.
Avoids excessive log volume from irrelevant events.
Suitable for data access auditing, security monitoring, and compliance logging.

In short, log_table_access provides targeted visibility into table usage, connections, and general query activity—ideal for environments where tracking who accessed what data is more important than recording every internal event.

Example 3

SELECT audit_log_filter_set_filter('log_minimum', '{
 "filter": {
 "class":
 [ { "name": "connection" }, { "name": "table_access", "event": [
 { "name": "delete"}, { "name": "insert"}, { "name": "update"} ]
 } ]
 }
}');

Included Event Classes

“class”: “connection”
- Logs all connection-level events:
- connect: when a user logs in.
- disconnect: when a session ends.
- Failed logins and other connection-related actions.

Purpose: provides visibility into who connected, from where, and when.

“class”: “table_access” with “event”:
- Limits logging to specific table access events:
  - “delete” when rows are deleted.
  - “insert” when new rows are added.
  - “update” when existing rows are modified.
- Read operations (like SELECT) and metadata queries are excluded.

What It Does Functionally

Once assigned to a user or host (e.g. SELECT audit_log_filter_set_user(’%’, ’log_minimum’);), this filter will produce audit entries only when:

A user connects or disconnects from MySQL.
A user performs a DML (Data Manipulation Language) operation that changes data in a table.

All other events — such as simple SELECT queries, schema reads, or administrative commands — will be ignored.

Why Use This Filter

This is a minimalist, high-value audit configuration. It’s designed to:

Track security-relevant activity (connections and data changes).
Meet compliance requirements with low performance overhead.
Prevent excessive logging and disk usage.

In Short log_minimum is an efficient auditing strategy for production environments where you only need to know:

Who accessed the database, and
What data they changed.

It gives you essential accountability and change tracking without the overhead of logging every read or administrative event.

Example 4

SELECT audit_log_filter_set_filter('log_connections', '{
 "filter": {
 "class": [
 { "name": "connection",
 "event": [
 { "name": "connect"},
 { "name": "disconnect"}
 ]
 }
 ]
 }
}');

Included Event Classes

“class”: “connection”
- This class captures events related to user sessions and authentication.
“event”: [“connect”, “disconnect”]
- connect, Logged when a client establishes a connection to the MySQL server. Includes details like username, host, client program, IP address, and connection method.
- disconnect, Logged when that client session ends or times out. Useful for tracking session duration and identifying abnormal terminations.
Important Note on pre_authenticate Events
- The pre_authenticate events are not included in the examples above.
- These events occur before the MySQL server has received authentication information from the client—meaning no user account details are available at this stage of the connection lifecycle. Because of that, if a filter that includes pre_authenticate events is assigned to a specific user (rather than a wildcard like %) using audit_log_filter_set_user(), those events will not be filtered or logged.
- This behavior often leads to confusion, as users may expect pre_authenticate events to appear in user-specific logs. Several reports and support cases have been filed on this topic, but it is expected behavior due to the timing of authentication during connection initialization.

Why Use This Filter

This filter is particularly useful when you need to:

Monitor user logins and logouts without recording query activity.
Audit connection patterns (e.g., who connected, from where, and when).
Maintain minimal log size and low overhead.
Support security investigations or session tracking without performance impact.

In short the log_connections filter provides a focused, low-overhead auditing strategy that records only connection lifecycle events. It’s ideal for environments where you primarily need to know who connected to the database, when, and from where without capturing every SQL statement or table access.

Example 5

SELECT audit_log_filter_set_filter('log_full_table_access', '{
 "filter": {
 "class": [
 {
 "name": "connection",
 "event": [
 { "name": "connect"},
 { "name": "disconnect"}
 ]
 },
 {
 "name": "query",
 "event": [
 {
 "name": "start",
 "log": {
 "or": [
 { "field": { "name": "sql_command_id", "value": "select"} },
 { "field": { "name": "sql_command_id", "value": "insert"} },
 { "field": { "name": "sql_command_id", "value": "update"} },
 { "field": { "name": "sql_command_id", "value": "delete"} },
 { "field": { "name": "sql_command_id", "value": "truncate"} },
 { "field": { "name": "sql_command_id", "value": "create_table"} },
 { "field": { "name": "sql_command_id", "value": "alter_table"} },
 { "field": { "name": "sql_command_id", "value": "drop_table"} }
 ]
 }
 }
 ]
 }
 ]
 }
}');

This statement defines a MySQL Audit Log Filter called log_full_table_access, which is designed to capture both connection activity and all table-related SQL operations — including reads, writes, and schema changes. It provides broad visibility into how users interact with tables in the database while filtering out unrelated or low-value events.

Included Event Classes

After assigning it to users or hosts (e.g. SELECT audit_log_filter_set_user(’%’, ’log_full_table_access’);), MySQL will log:

Connection lifecycle events (connect, disconnect)
All DML statements (SELECT, INSERT, UPDATE, DELETE, TRUNCATE)
All DDL statements that create, modify, or remove tables (CREATE TABLE, ALTER TABLE, DROP TABLE)
A full list of SQL_COMMANDS can be obtained from:

SELECT NAME FROM performance_schema.setup_instruments WHERE NAME LIKE 'statement/sql/%' ORDER BY NAME;

+-----------------------------------------------+
| NAME |
+-----------------------------------------------+
| statement/sql/alter_db |
| statement/sql/alter_event |
| statement/sql/alter_function |
| statement/sql/alter_instance |
| statement/sql/alter_procedure |
| statement/sql/alter_resource_group |
| statement/sql/alter_server |
| statement/sql/alter_table |
| statement/sql/alter_tablespace |
| statement/sql/alter_user |
| statement/sql/alter_user_default_role |
| statement/sql/analyze |
| statement/sql/assign_to_keycache |
| statement/sql/begin |
| statement/sql/binlog |
| statement/sql/call_procedure |
| statement/sql/change_db |
| statement/sql/change_repl_filter |
| statement/sql/change_replication_source |
| statement/sql/check |
| statement/sql/checksum |
| statement/sql/commit |
| statement/sql/create_compression_dictionary |
[...]
+-----------------------------------------------+
168 rows in set (0.07 sec)

Everything else — administrative commands, stored procedure calls, replication control, etc. — will be excluded.

Why Use This Filter

This filter offers a comprehensive audit view of how users interact with data and schema structures — perfect for compliance, forensic analysis, or access accountability. It ensures that all table reads, writes, and structure changes are tracked without overwhelming the log with irrelevant internal events.

In Short, log_full_table_access provides a broad but targeted audit scope:

Tracks connections for user session context.
Logs all table-level operations, both data and schema-related.
Delivers complete visibility into how data is accessed and changed, making it ideal for security auditing and regulatory compliance scenarios.

Final Summary

The MySQL 8.4 Audit Log Filter component provides a powerful and flexible framework for controlling how database activity is captured and logged. By allowing administrators to define granular filters based on event class, event type, user, or host, it transforms auditing from an all-or-nothing process into a precisely tuned observability tool.

In this post, we explored a range of filter examples—from the most permissive (log_all_events) to more focused configurations like log_minimum, log_connections, and log_full_table_access. Each serves a different operational or compliance purpose:

log_all_events – Captures every event for baseline validation or forensic debugging.
log_table_access – Balances visibility and performance by logging table, connection, and general query activity.
log_minimum – Targets critical actions such as connections and data modifications, providing essential accountability with minimal overhead.
log_connections – Focuses solely on login and logout events, ideal for lightweight session auditing.
log_full_table_access – Delivers comprehensive insight into all table-level DML and DDL operations along with connection tracking, suitable for compliance and change auditing.

By tailoring filters to specific operational needs, administrators can significantly reduce log volume, improve performance, and focus on high-value security and compliance events. The result is a leaner, more informative audit log that provides actionable insight into how users and applications interact with your MySQL environment—without the burden of unnecessary data.

Reference

Special Thanks

Yura Sorokin for the collaboration that made this blog post possible.

Percona at PostgreSQL Conference Europe 2025

Edith Puclla — Fri, 03 Oct 2025 11:00:00 UTC

We’re proud to announce that Percona is a Platinum Sponsor of PostgreSQL Conference Europe (PGConf.EU) 2025, taking place October 21–24, 2025 in Riga, Latvia 🇱🇻 at the Radisson Blu Latvija Conference Center.

As a Platinum Sponsor, you can find Percona in a prime main exhibit floor location; a chance to connect directly with our PostgreSQL experts from around the world. Visitors stopping by our booth will be able to:

Enter raffles and win Percona & PostgreSQL SWAG
Watch live demos and learn from expert-led sessions
Explore open source PostgreSQL solutions built for scalability, performance, and security

As a leader in open source database management and services, Percona supports innovation, collaboration, and the sharing of knowledge that drives the open source ecosystem forward. By participating in PostgreSQL Conference Europe, Percona connects with developers, contributors, and industry leaders to discuss the latest trends, challenges, and advancements. We look forward to further showcasing Percona’s dedication to empowering organizations with robust, scalable, and secure open source database solutions.

About the event

This year’s conference is the 15th Annual PostgreSQL Conference Europe. The conference is organised by PostgreSQL Europe, with participation from most of the PostgreSQL user groups around Europe, and is intended to be an important meeting and cooperation point for users both in and out of Europe. PGConf.EU is a unique chance for European PostgreSQL users and developers to catch up, learn, build relationships, get to know each other and consolidate a real network of professionals that use and work with PostgreSQL.

Percona Speaker Agenda

Our experts are taking the stage across multiple tracks to share insights, hands-on experience, and forward-looking innovations:

Wednesday, October 22

What implementing pg_tde taught us about PostgreSQL

👤 Jan Wieremjewicz

⏰ 16:05 – 16:55 | Room: Alfa | Track: Community (45 minutes)

Thursday, October 23

Kubernetes from the Database Out

👤 Alastair Turner

⏰ 10:25 – 10:55 | Room: Omega 2 | Track: DBA (25 minutes)
TDE as an Extension: A Different Path for PostgreSQL Encryption

👤 Zsolt Parragi

⏰ 11:25 – 12:15 | Room: Beta | Track: Sponsors
Why PostgreSQL took the crown from MySQL and what lies ahead

👤 Peter Zaitsev

⏰ 17:20 – 17:35 | Room: Omega 1 | Track: Platinum Sponsor Keynotes

Friday, October 24

Lessons from two decades of hacking the proprietary value into open source databases

👤 Michal Nosek

⏰ 09:25 – 10:15 | Room: Beta | Track: Sponsors

See the full agenda here.

Community Leadership – Kai Wagner

We are also proud that Kai Wagner, Senior Engineering Manager for PostgreSQL at Percona, is serving on the PGConf.EU 2025 Selection Committee. Kai is a long-time open source contributor and speaker, actively involved in projects like Ceph, openATTIC, and PostgreSQL. As a community builder and PGConf Germany organizer, he helps shape the PostgreSQL ecosystem and ensure diverse, impactful content is represented at the conference.

Join us in Riga this October to connect, learn, and celebrate PostgreSQL with Percona and the wider open source community!

Audit Log Filter Component

Wayne Leutwyler — Thu, 18 Sep 2025 00:00:00 UTC

The audit log filter component in MySQL 8.4 provides administrators with a powerful mechanism for auditing database activity at a fine-grained level. While it offers significant flexibility—such as selectively logging events based on users, hosts, or event types—it can also be challenging to understand and configure correctly.

In this article, we will examine how the audit log filter component works, walk through its core concepts, and share practical tips for configuring and managing audit filters effectively. Our goal is to help you leverage this feature to improve observability, meet compliance requirements, and reduce unnecessary logging overhead.

Enabling Audit Log Filter

We will be using Percona Server 8.4.4 or higher in the examples below. First, we need to enable the audit log filter component. To install the audit log filter component, we need to run the following command:

mysql -u root -p < /usr/share/percona-server/mysql/share/audit_log_filter_linux_install.sql

Verify that the audit log filter component is enabled by running:

select * from mysql.component;
+--------------+--------------------+-----------------------------------+
| component_id | component_group_id | component_urn |
+--------------+--------------------+-----------------------------------+
| 2 | 1 | file://component_audit_log_filter |
+--------------+--------------------+-----------------------------------+
1 row in set (0.00 sec)

Installing the component creates two new tables in the mysql system database: audit_log_filter and audit_log_user. These tables store the audit log filter definitions and the user-to-filter mappings. Together, they are referred to as the audit log filter tables.

+------------------------------------------------------+
| Tables_in_mysql |
+------------------------------------------------------+
| audit_log_filter |
| audit_log_user |
+------------------------------------------------------+

Although the configuration is persisted in these tables, they are not usually modified directly with INSERT or UPDATE statements. Instead, MySQL provides built-in functions such as:

audit_log_filter_set_filter()
audit_log_filter_set_user()

To manage filter definitions and user assignments safely.

Configure the my.cnf file to define the desired audit log output format and specify the location of the audit.log file. In the example below, the log format is set to JSON, but other formats (e.g., NEW or OLD) can also be configured depending on your requirements. The audit log file can be written to any path accessible to the MySQL server process.

Example Changes:

[mysqld]
# auditlog
audit_log_filter.format=JSON
audit_log_filter.file=/var/lib/mysql/audit.log

Restart mysql server to apply the changes:

sudo systemctl restart mysqld

The audit log filter install is complete. Now we can start using the audit log filter component.

Creating Audit Log Filters

The audit log filter component in MySQL 8.4 provides fine-grained control over database auditing. Instead of logging all events indiscriminately, administrators can define audit log filters, which are rule sets that determine exactly which events should be captured and which should be excluded.

This allows you to:

Log only the activity relevant to security, compliance, or troubleshooting.
Reduce unnecessary noise and audit log volume.
Apply different filters to specific users, hosts, or accounts for tailored auditing.

Because filters can be customized and assigned at the user or host level, the audit log filter component offers both flexibility and efficiency, making it a powerful mechanism for monitoring and securing database activity while minimizing overhead.

Lets create a rule that will log all events:

SELECT audit_log_filter_set_filter('log_all_events', '{ "filter": {"log": true } }');

Lets assign the rule to the user:

SELECT audit_log_filter_set_user('%', 'log_all_events');
SELECT audit_log_filter_flush();

Now all users will have all their events logged.

Before proceeding, ensure that the jq utility is installed on your system. The installation commands provided in the examples below are compatible with both RHEL-based and Debian-based distributions.

RHEL builds

sudo yum install jq

Debian builds

sudo apt install jq

To validate that the audit log filter is functioning as expected, we can inspect the raw contents of the audit.log file. Since the log entries are in JSON format, using jq provides an efficient way to query and extract specific events. For example, to filter and display only connection-related events, run:

cat audit.log | jq '.[]|select(.class=="connection")'

This command streams the log file, parses the JSON structure, and returns only entries where the “class” field is equal to “connection”. This approach allows for targeted analysis, making it easier to verify filter behavior, troubleshoot issues, or monitor specific event classes without manually parsing large volumes of log data.

Example Output:

{
 "timestamp": "2025-07-24 07:18:00",
 "id": 27110,
 "class": "connection",
 "event": "connect",
 "connection_id": 415,
 "account": {
 "user": "wayne",
 "host": "localhost"
 },
 "login": {
 "user": "wayne",
 "os": "",
 "ip": "",
 "proxy": ""
 },
 "connection_data": {
 "connection_type": "socket",
 "status": 0,
 "db": ""
 },
 "connection_attributes": {
 "_pid": "717914",
 "_platform": "aarch64",
 "_os": "Linux",
 "_client_name": "libmysql",
 "os_user": "wayne",
 "_client_version": "8.4.5-5",
 "program_name": "mysql"
 }
}

I’ll cover more advanced filter configurations in a follow-up post—stay tuned for Part 2 of the Audit Log Filter Component series.

In summary, the audit log filter component in MySQL 8.4 provides administrators with a flexible and fine-grained approach to database auditing. By tailoring filters to specific users, hosts, and event types, you can ensure that only the most relevant activity is logged, making it easier to meet compliance requirements while reducing overhead. With proper configuration and careful use of filters, you can transform the audit log from a noisy data dump into a precise monitoring tool that strengthens both security and observability in your MySQL environment.

Encrypting PostgreSQL Tables with PG_TDE: Step-by-Step Guide for Beginners

Daniil Bazhenov — Mon, 01 Sep 2025 11:00:00 UTC

Let’s install and try out a new package for PostgreSQL — PG_TDE by Percona. This extension adds Transparent Data Encryption (TDE), a mechanism that allows data to be encrypted at the storage level without affecting application behavior.

TDE protects data in case of disk, dump, or backup compromise: everything stored at the file system level is encrypted. Meanwhile, the application continues to work with the data as usual — encryption and decryption happen transparently, without changes to SQL queries or logic.

In this post, we’ll deploy PostgreSQL with PG_TDE in Docker, configure keys, create an encrypted table, and verify that the data is truly protected — even if someone gains direct access to the files. Other installation methods are available in the documentation.

For simplicity and speed of testing, we’ll use a key provider with a keyring file (for dev/test only) — the easiest way to get started with PG_TDE. This approach is convenient for local development and demos, as it doesn’t require additional dependencies or infrastructure.

Installing PostgreSQL with PG_TDE via Docker

PG_TDE is part of the Percona Distribution for PostgreSQL — an enhanced PostgreSQL distribution that includes tools for monitoring, auditing, replication, and of course, Transparent Data Encryption. We’ll use the official Docker image, which already contains everything needed to run PG_TDE.

Let’s start by launching the container:

docker run --name pg-tde \
 -e POSTGRES_PASSWORD=secret \
 -e ENABLE_PG_TDE=1 \
 -p 5432:5432 \
 -d percona/percona-distribution-postgresql:17.5-3

Here:

POSTGRES_PASSWORD=secret — sets the superuser password
ENABLE_PG_TDE=1 — enables Transparent Data Encryption support
Port 5432 — standard for PostgreSQL

After launching, connect to PostgreSQL inside the container:

docker exec -it pg-tde psql

And activate the PG_TDE extension:

CREATE EXTENSION pg_tde;

Verify that PG_TDE is enabled and check its version using the SQL function:

SELECT pg_tde_version();

Here’s the result at the time of writing — I’m using PG_TDE 2.0

I also connected to PostgreSQL using pgAdmin, accessing via localhost, user postgres, and the password from POSTGRES_PASSWORD.

pgAdmin is very convenient for exploring the database via UI.

Now TDE is ready — we can create encrypted tables, manage keys, and verify how data is protected at the storage level.

Configuring Encryption Keys with PG_TDE

After installing and activating the pg_tde extension, the next step is to configure the keys that will be used for data encryption. In this example, we’re using a key provider based on a keyring file — a simple and fast way to start working with PG_TDE in development mode.

Important: The keyring file stores keys in unencrypted form and is intended for testing and development only. For production, it’s recommended to use an external key store such as HashiCorp Vault — we’ll cover that in the next post.

Add the key provider:

SELECT pg_tde_add_database_key_provider_file(
 'file-vault',
 '/tmp/pg_tde_local_keyring.per'
);

This command:

Registers a key provider named ‘file-vault’
Specifies that keys will be stored in the file located at /tmp/pg_tde_local_keyring.per
The file path can be anything — you can use a different location or filename, as long as PostgreSQL has access to it. If the file doesn’t exist, it will be created when the first key is generated

It’s important not to lose this file — it contains the encryption keys. If needed, it can be placed in a local folder and mounted into the container via Docker, but for this task, using a path inside the container is perfectly fine.

Create the key:

SELECT pg_tde_create_key_using_database_key_provider(
 'test-db-key',
 'file-vault'
);

We’re creating a key named test-db-key inside the specified key provider. This key will be used to encrypt tables.

Set the active key:

SELECT pg_tde_set_key_using_database_key_provider(
 'test-db-key',
 'file-vault'
);

This command sets test-db-key as the current active key, which will be used when creating new encrypted tables.

Let’s verify.

Retrieve key information using the command:

SELECT pg_tde_key_info();

Result:

postgres=# SELECT pg_tde_key_info();
 pg_tde_key_info
------------------------------------------------------------
 (test-db-key,file-vault,1,"2025-09-04 07:16:04.420629+00")
(1 row)

Check the contents of the keyring file — just for experimentation.

In a separate terminal tab, connect to the container:

docker exec -it pg-tde bash

Use hexdump to read the file since it contains binary data:

hexdump -C /tmp/pg_tde_local_keyring.per | head

This will output the first bytes of the file in hex format, allowing you to confirm that the file is not empty and the key is indeed recorded.

If the file is empty or unchanged — the key may not have been created or written.

Creating Tables with Data

Now that the encryption key is set, let’s create two tables: one with encryption enabled, and one regular. This will help visually demonstrate how PG_TDE works and how data storage differs.

Create the encrypted table by specifying USING tde_heap:

CREATE TABLE secure_data (
 id SERIAL PRIMARY KEY,
 secret TEXT,
 created_at DATE
) USING tde_heap;

Add a few rows:

INSERT INTO secure_data (secret, created_at) VALUES
 ('The launch code is hidden in plain sight.', '2025-09-01'),
 ('Trust no one. The truth is encrypted.', '2025-09-02'),
 ('The treasure lies beneath the third stone by the old oak.', '2025-09-03');

Create a similar table without encryption:

CREATE TABLE plain_data (
 id SERIAL PRIMARY KEY,
 secret TEXT,
 created_at DATE
);

Insert the same data:

INSERT INTO plain_data (secret, created_at) VALUES
 ('The launch code is hidden in plain sight.', '2025-09-01'),
 ('Trust no one. The truth is encrypted.', '2025-09-02'),
 ('The treasure lies beneath the third stone by the old oak.', '2025-09-03');

Now both tables contain identical rows, but one stores the data in encrypted form.

Let’s check the encryption status using PG_TDE.

Use the built-in function pg_tde_is_encrypted, which returns true (t) if the table uses encryption, and false (f) if not:

postgres=# SELECT pg_tde_is_encrypted('secure_data');
 pg_tde_is_encrypted
---------------------
 t
(1 row)

postgres=# SELECT pg_tde_is_encrypted('plain_data');
 pg_tde_is_encrypted
---------------------
 f
(1 row)

Excellent, now we can move on to comparison: let’s see how the data looks inside PostgreSQL — and how it’s stored on disk. This will help confirm that PG_TDE truly protects content, even if someone gains direct access to the files.

Comparing Tables: Encrypted vs Unencrypted

At the SQL query level, both tables — secure_data (encrypted) and plain_data (unencrypted) — look absolutely identical. This is important: PG_TDE does not affect SQL interface behavior, and the application continues to work with the data as usual.

postgres=# SELECT * FROM secure_data;
 id | secret | created_at
----+-----------------------------------------------------------+------------
 1 | The launch code is hidden in plain sight. | 2025-09-01
 2 | Trust no one. The truth is encrypted. | 2025-09-02
 3 | The treasure lies beneath the third stone by the old oak. | 2025-09-03
(3 rows)

postgres=# SELECT * FROM plain_data;
 id | secret | created_at
----+-----------------------------------------------------------+------------
 1 | The launch code is hidden in plain sight. | 2025-09-01
 2 | Trust no one. The truth is encrypted. | 2025-09-02
 3 | The treasure lies beneath the third stone by the old oak. | 2025-09-03
(3 rows)

Display in the application using pgAdmin

Data from the plain_data table:

Data from the encrypted table secure_data:

The pgAdmin screenshots also show no differences: data is displayed in readable form, regardless of whether the table is encrypted. This confirms that PG_TDE operates at the storage level, without affecting data handling logic.

As you can see, the result is identical: encryption is transparent to the user and application.

Comparing Table File Contents

Now let’s verify that PG_TDE truly encrypts data at the storage level. To do this, we’ll locate the physical table files and compare their contents using hexdump.

In PostgreSQL, each table is physically stored as one or more files in the base directory, inside the folder corresponding to the database. The file path looks like $PGDATA/base//

database_oid — unique database identifier (OID), retrieved from pg_database
relfilenode — table file identifier, retrieved from pg_class

Step 1: Retrieve Identifiers

Get the OID of the current database:

postgres=# SELECT oid, datname FROM pg_database WHERE datname = current_database();
 oid | datname
-----+----------
 5 | postgres
(1 row)

Get the relfilenode of the tables:

postgres=# SELECT relname, relfilenode FROM pg_class WHERE relname IN ('secure_data', 'plain_data');
 relname | relfilenode
-------------+-------------
 secure_data | 16433
 plain_data | 16442
(2 rows)

Now, knowing oid = 5 and the relfilenode of the tables, we can locate the table files at:

data/db/base/5/16433 — encrypted table secure_data
data/db/base/5/16442 — unencrypted table plain_data

Step 2: Read Table Files

Open a terminal and connect to the PostgreSQL container for bash commands:

docker exec -it pg-tde bash

Read data from the unencrypted table (plain_data):

hexdump -C data/db/base/5/16442 | head -n 20

The output shows that the data is readable directly from the file:

Read data from the encrypted table (secure_data):

hexdump -C data/db/base/5/16433 | head -n 20

The output shows that the contents are fully encrypted:

No readable strings — PG_TDE reliably encrypts data at the file system level.

Conclusion

In this post, we installed PostgreSQL with Transparent Data Encryption (PG_TDE) by Percona, created keys, encrypted a table, and verified that the data is truly protected during physical storage. PG_TDE by Percona provides transparent encryption: the application continues to work with the data as usual, while security is enforced at the file system level.

If you’re working with PostgreSQL and looking to strengthen your data protection, PG_TDE offers a great starting point. I invite you to experiment:

Create encrypted tables and perform a backup. Try restoring it on a different installation and see how key access is enforced.
Explore key rotation, switching the active key, or integrating alternative key providers.
Test how PG_TDE behaves in real-world scenarios: migration, failover, CI/CD pipelines

If you’ve already tried PG_TDE or plan to — share your experience and insights by posting in this blog or on the forum

In the next post, we’ll explore a more robust option — integrating PG_TDE with HashiCorp Vault, which enables centralized key management, rotation, and access control.

pg_tde can now encrypt your WAL on PROD!

Jan Wieremjewicz — Mon, 01 Sep 2025 11:00:00 UTC

Just recently, we announced the production-ready release of pg_tde, bringing open source Transparent Data Encryption (TDE) to PostgreSQL.
Now, I may have spoiled the fun a little with the title, but take a look at the word puzzle below—can you guess the announcement? Bear with me… and my sense of humor, which might be a bit too dry for some :)

Yes, it’s an elephant carrying logs to a safe. Because Write Ahead Log (WAL) deserves secure storage!

At the time of the TDE GA release, one of the most common questions was:

“How does pg_tde handle the data in Write Ahead Logs (WAL)?”

The short answer back then was:

“It can encrypt WAL data, but the feature is still in beta.”

I’m happy to share that this capability is now production ready as well! WAL encryption has reached General Availability (GA) and is included in the latest release of Percona Distribution for PostgreSQL 17.5.3.

How to use it?

To use WAL encryption, you’ll need both:

pg_tde extension
Percona Server for PostgreSQL (included in Percona Distribution)

Why? Because pg_tde relies on patches that extend PostgreSQL APIs, and these are not yet available in the upstream PostgreSQL Community releases. Our long-term goal is to ensure they are available in the Community PostgreSQL Server as well, but for now you’ll find them only in Percona Server. If you’ve already been running the GA version of TDE in Percona Distribution 17.5.2, the upgrade is straightforward. Assuming pg_tde is installed and initialized, enabling WAL encryption requires:

Configure the encryption key by:
- Setting up the key provider for WAL encryption
- Creating the principal key
Turning the WAL encryption on by running:
- ALTER SYSTEM SET pg_tde.wal_encrypt = on;
Restarting the server to apply the changes

Some known limitations of the first version

While WAL encryption is now GA, there are some first version limitations to be aware of:

Compatibility with other extensions: pg_tde has only been validated with the extensions shipped in Percona Distribution. Extensions that rely heavily on WAL (such as some backup, replication, or logical decoding tools) may not yet work correctly.
Backup and restore:
- WAL encryption has been tested and verified with PgBackRest, our recommended tool for backup and recovery.
- pg_basebackup works with WAL encryption when using streaming or none methods. While the -x fetch method is not yet supported for other methods there are some good practices we highly recommend.
- Other backup tools may not yet support WAL encryption. An example is Barman as the following Barman backup methods are not yet supported with WAL encryption:
  - Postgres method
  - Rsync method
  - Snapshot method
Native logical replication: tools like pg_createsubscriber are not yet supported with encrypted WAL. We are planning to enable them in one of the next releases. Do share feedback if you need them, it will help us prioritize.

See more details regarding the WAL encryption GA supported scope in our documentation. We’ll continue testing compatibility with widely used extensions and documenting what works out of the box versus what requires extra steps. Here, we rely on our community of users as well to let us help prioritize work. Let us know what would you want to see tested or you’ve already tested and need help in making work with WAL encryption.

Why this matters for regulated environments

STIG documentation proves that PostgreSQL is ready to tackle any government use cases already. Encryption of both data files and WAL is often a requirement when organizations work under strict compliance frameworks such as:

FIPS 140-2 / 140-3 – cryptographic module validation requirements for U.S. government use.
HIPAA – protection of electronic protected health information (ePHI).
PCI-DSS – encryption requirements for cardholder data at rest.
FedRAMP and other government security standards – full coverage of data at rest, including temporary and log files.

Many PostgreSQL distributions rely on filesystem-level encryption or external tooling to meet these standards. With PCI-DSS 4.0 this is no longer enough. With pg_tde, encryption is native to PostgreSQL internals, ****covering both tables and WAL, implemented with widely recognized cryptographic algorithms via OpenSSL, and integrated with enterprise grade Key Management Systems (KMS). This reduces operational complexity and helps align directly with compliance controls.

What does the future hold for TDE?

We’re far from done. The roadmap for TDE includes:

New capabilities:
- More cipher configuration options
- Expanded KMS support (beyond what we support today)
- Temporary files encryption
- Features requested from real-world deployments
Lifting current limitations:
- Broader backup/restore scenarios
- Compatibility with logical replication tooling such as pg_createsubscriber
- Validating more third-party extensions
Upstream contributions:
- Our long-term goal is to contribute the necessary server patches into Community PostgreSQL. This way, pg_tde can eventually run on any PostgreSQL release, not just Percona Server for PostgreSQL.

We’d love your feedback! If you’ve started using pg_tde or are planning to test it (especially with WAL encryption) let us know what works well, what could be improved or fixed, and what you’d like to see next. Your input directly helps us prioritize features, improve compatibility, and shape the future of TDE in PostgreSQL.

Share your experiences via our Community Forums, alternatively via GitHub Discussions or Issues!

pg_stat_monitor Needs You! Join the Feedback Phase

Jan Wieremjewicz — Wed, 13 Aug 2025 00:00:00 UTC

At Percona, we believe that great open source software is built with the Community, not just for it. As we plan the next iteration of pg_stat_monitor, our advanced PostgreSQL monitoring extension, we’re taking a closer look at the current feature set and how it aligns with real-world usage.

In open source, the community isn’t just a user base, it’s the most important stakeholder. While we set the vision, your feedback is the compass that guides us. Your experiences, bug reports, and feature requests are what validate our direction and keep us focused on what matters most. Without your active involvement, it’s impossible to build a tool that truly solves the problems you face every day. Your input ensures pg_stat_monitor evolves in a way that is both innovative and genuinely useful.

Your Feedback Is the Compass

Over time, pg_stat_monitor has grown beyond its initial query performance monitoring scope.

While many features have proven to be extremely useful (especially in use with Percona Monitoring and Management (PMM)), others may see limited adoption or give no evidence of any adoption at all. To ensure we’re investing in what really matters, we want to understand what you, the users and contributors, actually rely on day-to-day.

That’s why we’re kicking off a Community feedback phase

We’re especially interested in:

What features of pg_stat_monitor are critical for your workflows?
Are you using it together with PMM, CLI or in another way?
Are there parts of the extension that feel unnecessary or unclear?
What would you love to see in the next release?

This is just the beginning of a broader review and improvement effort, one that we want to run transparently and inclusively, true to our open source values.

Whether you’re a developer, DBA, or platform engineer using pg_stat_monitor directly or via tools like PMM, know that your input matters.

👉 Leave a comment below, or reach out on our Percona Community Forums or via GitHub. Let’s shape the future of pg_stat_monitor together.

Percona pg_tde: A Security Review Reveals Robust Encryption

Kai Wagner — Thu, 24 Jul 2025 00:00:00 UTC

At Percona, we are committed to providing robust and secure database solutions. We recently engaged Longterm Security for an in-depth review of our Transparent Data Encryption (TDE) feature for PostgreSQL, known as pg_tde. This comprehensive assessment included a cryptographic design evaluation, an application security review for coding errors, and extensive fuzz testing. We’re excited to share the key takeaways from this engagement, highlighting both pg_tde’s strengths and areas for continued improvement.

Strong Foundations in Compliance

A major highlight of the review is pg_tde’s excellent alignment with leading compliance requirements. Our TDE implementation is well-suited to meet the stringent demands of standards like PCI-DSS, HIPAA, GDPR, and SOC2. This is largely due to its adherence to best practices outlined by ECRYPT-CSA, BSI, and NIST, particularly concerning key reuse, master key management, initialization vectors, randomness generation, and various counter modes (CBC, CTR, GCM).

The sole exception identified for full compliance is CNSA (formerly NSA Suite B), which specifically mandates a 256-bit key size for AES. While pg_tde currently supports AES-128, which is widely considered future-proof and meets most compliance needs for data lifetimes beyond 10 years, adding AES-256 support remains a key recommendation to meet the demands of government and high-security environments.

While technical work continues on managing temporary files and potentially swappable memory, it’s important to note that compliance standards—such as PCI DSS—generally permit the use of temporary files for encrypting sensitive data like PAN. That said, we understand the practical risks involved and are actively working to mitigate them.

No Major Vulnerabilities Uncovered

After a comprehensive security engagement, we are pleased to report that no major vulnerabilities have been found within pg_tde. This outcome is a testament to the robust design principles and meticulous implementation that underpin pg_tde, validating our continuous investment in secure development.

The highest priority issue they found (LTS-1: Temporary Files Not Subject to Encryption) was actually something we already knew about and being on our roadmap, while we documented ways to lessen the risk. Our team is also actively tackling another known issue (LTS-3) where sensitive data or encryption keys might end up on your disk if your system uses “swap memory.” As of today, we do not recommend using SWAP, but we’re actively looking into making it work as well. The other issues they pointed out (LTS-4 through LTS-7) are more like helpful suggestions for making things even more robust, rather than actual vulnerabilities.

Here’s a quick overview of the findings:

Issue	Severity	Status	Title
LTS-1	Medium	Already Known	Temporary Files Not Subject to Encryption
LTS-2	Low	Reported	Disable Core Dumps when pg_tde is active
LTS-3.1	Low	Already Known	palloc memory may be swapped to disk, storing customer data
LTS-3.2	Low	Already Known	palloc memory may be swapped to disk, storing key material from pg_tde
LTS-4	Informational	Informative	Privileged postgres users with file read access may disclose encryption keys from memory
LTS-5	Informational	Reported	Keyring_vault.c: Invalid JSON response can lead to NULL dereference
LTS-6	Informational	Informative	Consider HKDF with Principal Key Usage
LTS-7	Informational	Reported	Document how to securely deploy keyring authentication credentials to systems without storing to disk
LTS-8	Informational	Reported	AES-GCM-SIV is available in very new OpenSSL versions as an alternative key wrapping algorithm

Always Improving: What’s Next for pg_tde around security?

The review also helped us pinpoint a few areas where we can make pg_tde even stronger, especially when it comes to handling memory and exploring advanced encryption techniques:

Tackling Swap Memory and Core Dumps: We’re working on ways to prevent sensitive data and encryption keys from potentially leaking to your disk through swap memory or “core dumps” (which happen when a program crashes). We recommend using encrypted swap partitions and turning off core dumps when pg_tde is active. We’re also looking into using special memory allocators that keep key material from being swapped out.
More Encryption Options: To meet even more compliance needs and give you more choices, we’re considering adding support for a wider range of encryption ciphers, including AES-CBC-256, AES-CTR-256, AES-GCM-256, ChaCha20-Poly1305, and AES-XTS-256. The 256-bit AES key sizes, in particular, will help us achieve CNSA compliance.
Smarter Key Management (HKDF with Principal Key Usage): To boost security even further and protect against theoretical key-reuse attacks, we’re evaluating the use of HKDF (HMAC-based Key Derivation Function). This would add an extra layer of protection by creating unique keys for different situations.
Securely Handling Your Keyring Credentials: It’s super important that your keyring authentication credentials (like Vault tokens or KMIP keys) are never stored in plain text on your disk. We’re putting together clear guidelines for secure deployment, recommending solutions that keep credentials only in memory or use hardware-backed keystores. We’ll also advise against putting credentials in command-line arguments.
Keeping an Eye on New Algorithms (AES-GCM-SIV): We’re aware of a newer encryption algorithm called AES-GCM-SIV, which is available in very recent OpenSSL versions. While it’s not yet FIPS-140-3 compliant and might be a bit slower, it offers better protection against certain types of attacks, and we’re definitely keeping it on our radar for future consideration.
Preparing for Post-Quantum Cryptography: The cryptographic landscape is evolving rapidly, with post-quantum encryption (PQC) algorithms on the horizon. As these new algorithms are expected to be more resource-intensive, pg_tde’s granular table-level encryption will be crucial for efficiently managing performance overheads, ensuring your data remains secure without compromising application responsiveness in a post-quantum world. We are actively monitoring these ciphers’ development and standardization.

Our Commitment to Security

The security review by Longterm Security has been incredibly valuable. It’s not only confirmed pg_tde’s strong capabilities but also given us a clear roadmap for future development. We’re really proud of how secure pg_tde is, which you can see in its readiness for compliance and the fact that no major vulnerabilities were found. Our ongoing efforts to address the identified issues, even the low-priority ones, show just how committed we are to constantly improving and providing you with the most secure database solutions out there.

We’re committed to building a secure and reliable data encryption solution, and your feedback plays a vital role in shaping its future. Try it out!

If you want to work directly with the code/test/build, take a look at the GitHub project. For issues, report them via Jira or go to our Forum to ask questions. For detailed deployment guidelines and best practices, take a look at our documentation. We’re looking forward to any feedback and collaboration around making PostgreSQL secure.

GitOps Journey: Part 4 – Observability and Monitoring with Coroot in Kubernetes

Daniil Bazhenov — Tue, 22 Jul 2025 00:01:00 UTC

Our PostgreSQL cluster is running, and the demo app is generating traffic — but we have no visibility into the health of the Kubernetes cluster, services, or applications.

What happens when disk space runs out? What if the database is under heavy load and needs scaling? What if errors are buried in application logs? How busy are the network and storage layers? What’s the actual cost of the infrastructure?

This is where Coroot comes in.

Coroot is an open-source observability platform that provides dashboards for profiling, logs, service maps, and resource usage — helping you track system health and diagnose issues quickly.

We’ll deploy it using Helm via ArgoCD, continuing with our GitOps workflow.

This is Part 4 in our series. Previously, we:

Set up ArgoCD and a GitHub repository for declarative manifests (Part 1)
Installed a PostgreSQL cluster using Percona Operator
Deployed a demo application to simulate traffic and interact with the database

All infrastructure is defined declaratively and deployed from the GitHub repository, following GitOps practices.

So far, we’ve explored cluster scaling, user management, and dynamic configuration — and now it’s time for observability.

We’ll install Coroot by following the official documentation for Kubernetes.

Steps ahead:

Install the Coroot Operator
Install the Coroot Community Edition

Let’s get started.

Project Structure

We already have a postgres/ directory for PostgreSQL manifests and an apps/ directory for ArgoCD applications.

We’ll preserve this layout and add a new coroot/ folder for clarity. You can use a different structure if preferred.

Create Manifest for Installing the Coroot Operator

The documentation recommends installing via Helm.
Since we use ArgoCD, we’ll create a manifest that installs via Helm.

Create file: coroot/operator.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot-operator
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://coroot.github.io/helm-charts
 chart: coroot-operator
 targetRevision: 0.4.2
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

Note: I’m using version 0.4.2, which was current at the time of writing.
To check available versions, use this GitHub link or Helm CLI:

helm repo add coroot https://coroot.github.io/helm-charts
helm repo update
helm search repo coroot-operator --versions

Create Manifest for Installing Coroot Community Edition

Create file: coroot/coroot.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://coroot.github.io/helm-charts
 chart: coroot-ce
 targetRevision: 0.3.1
 helm:
 parameters:
 - name: clickhouse.shards
 value: "2"
 - name: clickhouse.replicas
 value: "2"
 - name: service.type
 value: LoadBalancer
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true

This chart creates a minimal Coroot Custom Resource.
I’ve added service.type: LoadBalancer to expose a public IP.

If you don’t use LoadBalancer, you’ll need to forward the Coroot port after installation:

kubectl port-forward -n coroot service/coroot-coroot 8080:8080

Create ArgoCD Application Manifest

Since we manage our infrastructure via a GitHub repository, we need an ArgoCD Application that tracks changes in the coroot/ folder.

Create file: apps/argocd-coroot.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: coroot-sync-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/percona-argocd-pg-coroot.git
 targetRevision: main
 path: coroot
 destination:
 server: https://kubernetes.default.svc
 namespace: coroot
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

This lightweight app will monitor the folder and apply updates automatically if a change is detected (e.g. chart version bump).

Define Chart Installation Order

We have two charts: operator.yaml and coroot.yaml, and the operator must be installed first.

Create coroot/kustomization.yaml to specify resource order:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - operator.yaml
 - coroot.yaml

Publish Manifests to GitHub

Check which files were changed:

git status

Add changes:

git add .

Verify staged files:

git status

Commit:

git commit -m "Installing Coroot Operator and Coroot with ArgoCD"

Push:

git push origin main

Apply ArgoCD Application

Deploy the ArgoCD app that installs Coroot from our GitHub repository:

kubectl apply -f apps/argocd-coroot.yaml -n argocd

Validate installation and sync:

We now see coroot, coroot-operator, and coroot-sync-app deployed.

Access Coroot UI

Since we deployed Coroot using LoadBalancer, retrieve its external IP:

kubectl get svc -n coroot

Open EXTERNAL-IP on port 8080.
For example: http://35.202.140.216:8080/

If you didn’t use LoadBalancer, run port-forward:

kubectl port-forward -n coroot service/coroot-coroot 8080:8080

Then visit http://localhost:8080

You’ll be prompted to set an admin password on first login.

Exploring Coroot UI

On the home page, we see a list of applications running in the cluster and resource usage.

I increased the load on the PostgreSQL cluster using the Demo App to test observability.

The PostgreSQL cluster dashboard offers several tabs:

CPU
Memory
Storage
Instances
Logs
Profiling
Tracing

Coroot displays a visual map of service interactions — showing which app connects to the PostgreSQL cluster.

The Profiling tab looks excellent and intuitive. Here’s the Demo App profiling view:

I also triggered an intentional error in the demo app.
Coroot correctly displayed it in both the home view and the app details page.

I especially liked the Logs and Costs sections in the sidebar — very well implemented.

First Incident: Storage Usage in PostgreSQL Turns Yellow

While exploring Coroot and the cluster, I increased the load on the PostgreSQL cluster using the Demo App.

After a short while, I noticed that the Postgres disk was full.

I opened the cluster details and went to the Storage tab.

By default, the cr.yaml file allocates just 1Gi of disk space — which is fine for a test setup.

Let’s increase disk size the GitOps way.

Increase Storage Size

Open the file postgres/cr.yaml and locate the section:

 dataVolumeClaimSpec:
# storageClassName: standard
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

Change storage from 1Gi to 5Gi.

Note: Backup volumes (pgBackRest) are also enabled by default and set to 1Gi.

 manual:
 repoName: repo1
 options:
 - --type=full
# initialDelaySeconds: 120
 repos:
 - name: repo1
 schedules:
 full: "0 0 * * 6"
# differential: "0 1 * * 1-6"
# incremental: "0 1 * * 1-6"
 volume:
 volumeClaimSpec:
# storageClassName: standard
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 1Gi

Increase this storage to 5Gi as well.

Save changes to cr.yaml, then commit and push to the GitHub repository.

ArgoCD will automatically apply the changes. Pure GitOps magic.

Check the result in Coroot — everything looks great. The disk is increased to 5Gi and the issue is resolved.

Conclusion

We’ve installed and tested a solid monitoring tool, and it really makes a difference.

Across this 4-part series, we walked through the GitOps journey step by step:

Part 1 - Created a Kubernetes cluster, installed ArgoCD, and set up a GitHub repository.
Part 2 - Deployed a PostgreSQL cluster using Percona Operator for PostgreSQL.
Part 3 - Deployed a demo app via ArgoCD using Helm.
Installed and tested Coroot, an excellent open-source observability tool.

Managed the PG cluster through GitHub and ArgoCD — scaled replicas, created users, resized volumes, configured access, and more.

Thank you for reading — I hope this series was helpful.

The project files are available in my repository https://github.com/dbazhenov/percona-argocd-pg-coroot

I’d love to hear your questions, feedback, and suggestions for improvement.

GitOps Journey: Part 3 – Deploying a Load Generator and Connecting to PostgreSQL

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:50 UTC

We’ll deploy a demo application into the Kubernetes cluster using ArgoCD to simulate load on the PostgreSQL cluster.

This is a series of articles, in previous parts we:

Part 1 - Prepared the environment and installed ArgoCD and GitHub repository.
Part 2 - Installed Percona Operator for Postgres and created a Postgres cluster.

The application is a custom Go-based service that generates traffic for PostgreSQL, MongoDB, or MySQL.

It uses a dataset of GitHub repositories and pull requests, and mimics real-world operations like fetching, creating, updating, and deleting records.
Load intensity is configurable through a browser-based control panel.

We’ll install it using Helm, tracked and deployed via ArgoCD.

Reference repository: github-stat

Create the ArgoCD Application Manifest

Create a file named argocd-demo-app.yaml in the apps/ directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: demo-app
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/github-stat
 targetRevision: main
 path: k8s/helm
 destination:
 server: https://kubernetes.default.svc
 namespace: demo-app
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true

This will install the Helm chart from
https://github.com/dbazhenov/github-stat/tree/main/k8s/helm

By default, the service is configured as LoadBalancer, making it accessible from the internet.

To switch to NodePort (if needed), override the Helm value:

source:
 helm:
 parameters:
 - name: controlPanelService.type
 value: NodePort

We’ll keep default settings in this example.

Push the Application Manifest to GitHub

Track and commit your changes:

git status

git add .

git commit -m "Installing Demo Application in ArgoCD by HELM"

git push origin main

Expected Git output:

➜ percona-argocd-pg-coroot git:(main) git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
 (use "git add ..." to include in what will be committed)
 apps/argocd-demo-app.yaml

nothing added to commit but untracked files present (use "git add" to track)
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git commit -m "Installing Demo Application in ArgoCD by HELM"
[main 03ce175] Installing Demo Application in ArgoCD by HELM
 1 file changed, 20 insertions(+)
 create mode 100644 apps/argocd-demo-app.yaml
➜ percona-argocd-pg-coroot git:(main) git push origin main
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 686 bytes | 686.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:dbazhenov/percona-argocd-pg-coroot.git
 6b2dc98..03ce175 main -> main
➜ percona-argocd-pg-coroot git:(main)

Apply the ArgoCD Application

Deploy the app via:

kubectl apply -f apps/argocd-demo-app.yaml -n argocd

ArgoCD will install the app and has started tracking the app’s HELM chart

Validate the Deployment

Confirm the app status in ArgoCD UI:

Check running pods:

kubectl get pods -n demo-app

Expected pods:

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n demo-app
NAME READY STATUS RESTARTS AGE
demo-app-dataset-6d886f67-j648w 1/1 Running 0 2m52s
demo-app-load-577cff97c9-d8j99 1/1 Running 0 2m52s
demo-app-valkey-74989c9bf7-gjp4x 1/1 Running 0 2m52s
demo-app-web-5b98d4c65c-xmkq9 1/1 Running 0 2m52s

demo-app-dataset - loads dataset
demo-app-load - generates traffic
demo-app-valkey - Redis-compatible DB backend
demo-app-web - UI dashboard

Open the Application Dashboard

Retrieve the external IP:

kubectl get svc -n demo-app

Find the EXTERNAL-IP of demo-app-web-service.

Sample output:

➜ percona-argocd-pg-coroot git:(main) kubectl get svc -n demo-app
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-app-valkey-service ClusterIP 34.118.235.203  6379/TCP 4m59s
demo-app-web-service LoadBalancer 34.118.232.144 34.28.221.107 80:31308/TCP 4m59s

Access the app in your browser:

Navigate to the Settings tab to configure a PostgreSQL connection.

PostgreSQL Credentials Setup

Percona Operator has already (Application and system users):

Created schema and database cluster1
Created user cluster1
Stored credentials in cluster1-pguser-cluster1 secret

Extract the password:

kubectl get secret cluster1-pguser-cluster1 -n postgres-operator --template='{{.data.password | base64decode}}{{"\n"}}'

Let’s connect to the database from the Demo application using the given user and cluster1-pgbouncer.postgres-operator.svc host

In the Connection String field enter

user=cluster1 password='[PASSWORD]' dbname=cluster1 host=cluster1-pgbouncer.postgres-operator.svc port=5432

The connection has been successfully created, this is good.

To start generating the load, we need to import the Dataset using the Import Dataset button.

Dataset Import Error: Create Schema Denied

During import, the app tries to create a schema.
By default, pgBouncer limits user privileges, preventing this action.

Percona documentation suggests enabling proxy.pgBouncer.exposeSuperusers and creating a privileged user.

We’ll handle this via GitOps. It seems cool that we’ll be doing this with tracking in Git, as these are important settings and we shouldn’t forget about them and turn them off in the future.

Define a New PostgreSQL User

We will make changes to postgres/cr.yaml that will add a new user and also enable the proxy.pgBouncer.exposeSuperusers option.

In the postgres/cr.yaml file I found the users section, uncommented and added my user data.

In postgres/cr.yaml, add:

 users:
 - name: daniil
 databases:
 - demo
 options: "SUPERUSER"
 password:
 type: ASCII
 secretName: "daniil-credentials"

Note: In production, use scoped permissions like "LOGIN CREATE CREATEDB" rather than SUPERUSER.

I also found the proxy.pgBouncer.exposeSuperusers setting and set it to true

Update pgBouncer config:

 proxy:
 pgBouncer:
 replicas: 3
 image: docker.io/percona/percona-pgbouncer:1.24.1
 exposeSuperusers: true

Commit and push:

git status

git add .

git commit -m "Postgres cluster: Creating a new user and pgBouncer.exposeSuperusers"

git push origin main

After a couple of minutes, ArgoCD will synchronize the changes and Percona Operator will create the user and change the configuration.

Connect With the New User

Get the password:

kubectl get secret daniil-credentials -n postgres-operator --template='{{.data.password | base64decode}}{{"\n"}}'

Let’s replace Connection String in Demo application, I got the following string

user=daniil password='iKj:e[wT3*g]OF5+f' dbname=dataset host=cluster1-pgbouncer.postgres-operator.svc port=5432

Click the “Import Dataset” button and wait a few minutes until the import is in Done status in the Dataset tab.

Enable Load Generation

Activate the load generator:

Toggle Enable Load in the connection settings
Click Update Connection

Open the Load Generator Control Panel and adjust sliders and toggles as needed:

Conclusion

In this part, we:

Deployed a demo application via Helm in ArgoCD
Connected it to our PostgreSQL cluster
Managed PostgreSQL users and access via GitHub and GitOps
Imported a dataset and activated the traffic generator through the web UI

In Part 4, we’ll deploy Coroot for observability and profiling.
It’s an impressive tool for diagnosing behavior across services in the Kubernetes cluster.

GitOps Journey: Part 2 – Deploying PostgreSQL with GitOps and ArgoCD

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:30 UTC

We’re now ready to deploy PostgreSQL 17 using GitOps — with ArgoCD, GitHub, and the Percona Operator for PostgreSQL.

If you’re a DBA, developer, DevOps engineer, or engineering manager, this part focuses on GitOps in action: deploying and managing a real database cluster using declarative infrastructure.

In Part 1, we set up the Kubernetes environment and installed ArgoCD.
Now it’s time to define and launch the PostgreSQL cluster — fully versioned and synced through Git.

We’ll follow the official Percona Operator documentation and reference the GitHub repository to build out a production-grade setup.

Preparing the Environment

There are multiple ways to install the Percona Operator and create a PostgreSQL cluster.
We’ll use the simplest and most GitOps-friendly approach:

Deploy the operator using deploy/bundle.yaml
Deploy the cluster using deploy/cr.yaml

Source files:

https://github.com/percona/percona-postgresql-operator/blob/main/deploy/bundle.yaml
https://github.com/percona/percona-postgresql-operator/blob/main/deploy/cr.yaml

Project Structure

Repository structure can vary depending on your services and infrastructure scale.
For this series, we’ll use:

postgres/ → Contains all manifests related to PostgreSQL: the operator, clusters, backups
apps/ → Contains ArgoCD application manifests that track changes in the repository

You’re free to choose a different structure. Just ensure all paths are correctly referenced in ArgoCD.

Creating the Postgres Directory and Saving Manifests

You can manually download the files from GitHub or automate it via CLI:

mkdir postgres

curl -o postgres/bundle.yaml https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/bundle.yaml

curl -o postgres/cr.yaml https://raw.githubusercontent.com/percona/percona-postgresql-operator/v2.7.0/deploy/cr.yaml

You can also separately clone the operator repository and grab the necessary files from there.

Creating the ArgoCD Application Manifest

This ArgoCD application will track the postgres/ directory and automatically sync changes from GitHub.

Create the file: apps/argocd-postgres.yaml

Content:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: postgres
 namespace: argocd
spec:
 project: default
 source:
 repoURL: https://github.com/dbazhenov/percona-argocd-pg-coroot.git
 targetRevision: main
 path: postgres
 destination:
 server: https://kubernetes.default.svc
 namespace: postgres-operator
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - CreateNamespace=true
 - ServerSideApply=true 

You can also create this app manually via the ArgoCD UI or CLI, but using a manifest aligns better with GitOps principles.

Double-check your:

repoURL → matches your GitHub repository
path → corresponds to your PostgreSQL manifest directory
namespace → targets the correct namespace for operator and cluster

Managing ArgoCD Sync Order with Waves

ArgoCD applies manifests based on sync-wave annotations.

The operator (bundle.yaml) should be applied first
The cluster (cr.yaml) comes second

Add these annotations:

In bundle.yaml:

metadata:
 annotations:
 argocd.argoproj.io/sync-wave: "1"

In cr.yaml:

metadata:
 name: cluster1
 annotations:
 argocd.argoproj.io/sync-wave: "5"

This ensures a stable deployment sequence.

Later in the series (e.g. when installing Coroot), we’ll use a more advanced method: defining sync order via kustomization.yaml.

Reviewing Cluster Configuration

Before applying the manifests, review and adjust your cluster settings in cr.yaml.

Key defaults:

name: cluster1 → Cluster name
postgresVersion: 17 → PostgreSQL version

To keep the cluster lightweight and test horizontal scaling later, reduce replicas to 1:

instances:
 - name: instance1
 replicas: 1

proxy:
 pgBouncer:
 replicas: 1

You can also configure resource limits, disk sizes, backups, and users in this file.

Publishing the Configuration to GitHub

Verify your repo status:

git status

Add files:

git add .

Review staged files:

git status

Expected result:

➜ percona-argocd-pg-coroot git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
 (use "git add ..." to include in what will be committed)
 apps/
 postgres/

nothing added to commit but untracked files present (use "git add" to track)
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git status
On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
 (use "git restore --staged ..." to unstage)
 new file: apps/argocd-postgres.yaml
 new file: postgres/bundle.yaml
 new file: postgres/cr.yaml

Commit:

git commit -m "Initial configuration of a Postgres cluster using Percona Operator for Postgres and ArgoCD"

Push to GitHub:

git push origin main

Verify files are published correctly in the repository.

Applying the ArgoCD App Manifest

To initiate the deployment, apply the previously created manifest:

kubectl apply -f apps/argocd-postgres.yaml -n argocd

After a minute or two, the ArgoCD dashboard should display the synced PostgreSQL application and the deployed cluster.

Verifying Cluster Status

Check the running pods:

kubectl get pods -n postgres-operator

Expected results:

One PostgreSQL instance pod
One pgBouncer pod

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-5g98-5b29w 0/1 Completed 0 27m
cluster1-instance1-22vd-0 4/4 Running 0 28m
cluster1-pgbouncer-649b7cf845-fgs9l 2/2 Running 0 28m
cluster1-repo-host-0 2/2 Running 0 28m
percona-postgresql-operator-79f75d5f76-xjndr 1/1 Running 0 29m

Scaling the Cluster via GitOps

Let’s test the GitOps model by updating the cluster configuration to increase replicas to 3.

Edit postgres/cr.yaml:

 instances:
 - name: instance1
 replicas: 3

 proxy:
 pgBouncer:
 replicas: 3

Save the changes and push them:

git status

git add .

git commit -m "Postgres cluster: Horizontal scaling from 1 replica to 3"

git push origin main

ArgoCD will automatically detect and apply this update.

Expected results:

➜ percona-argocd-pg-coroot git:(main) git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
 (use "git add ..." to update what will be committed)
 (use "git restore ..." to discard changes in working directory)
 modified: postgres/cr.yaml

no changes added to commit (use "git add" and/or "git commit -a")
➜ percona-argocd-pg-coroot git:(main) ✗ git add .
➜ percona-argocd-pg-coroot git:(main) ✗ git commit -m "Postgres cluster: Horizontal scaling from 1 replica to 3"
[main 6b2dc98] Postgres cluster: Horizontal scaling from 1 replica to 3
 1 file changed, 2 insertions(+), 2 deletions(-)
➜ percona-argocd-pg-coroot git:(main) git push origin main
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 435 bytes | 435.00 KiB/s, done.
Total 4 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To github.com:dbazhenov/percona-argocd-pg-coroot.git
 81ae9e8..6b2dc98 main -> main
➜ percona-argocd-pg-coroot git:(main)

Expected results on GitHub Repo:

Confirming the Update in ArgoCD

In the ArgoCD UI, you should now see the application synced to the latest commit with the updated replica count.

Verify pods:

kubectl get pods -n postgres-operator

Expected result — 3 PostgreSQL pods and 3 pgBouncer pods.

➜ percona-argocd-pg-coroot git:(main) kubectl get pods -n postgres-operator
NAME READY STATUS RESTARTS AGE
cluster1-backup-5g98-5b29w 0/1 Completed 0 38m
cluster1-instance1-22vd-0 4/4 Running 0 39m
cluster1-instance1-q2r4-0 4/4 Running 0 3m38s
cluster1-instance1-r4s2-0 4/4 Running 0 3m39s
cluster1-pgbouncer-649b7cf845-9cppx 2/2 Running 0 3m37s
cluster1-pgbouncer-649b7cf845-fgs9l 2/2 Running 0 39m
cluster1-pgbouncer-649b7cf845-tkf9z 2/2 Running 0 3m36s
cluster1-repo-host-0 2/2 Running 0 39m
percona-postgresql-operator-79f75d5f76-xjndr 1/1 Running 0 40m

What’s Next

We’ve successfully installed the Percona Operator for PostgreSQL and deployed a cluster using GitHub and ArgoCD.

We also verified GitOps functionality by scaling the cluster through Git-controlled configuration.
All changes are tracked, versioned, and declarative — a solid foundation for modern infrastructure management.

To continue experimenting:

Connect to the Cluster
Manage Users. Note: the default user does not have SUPERUSER privileges. If your app requires creating databases, you’ll need to configure appropriate roles.
Expose the Cluster. So you can connect from external clients or apps.

We’ll do exactly that in the next part — by deploying a demo application and connecting it to the database using GitOps.

GitOps Journey: Part 1 – Getting Started with ArgoCD and GitHub

Daniil Bazhenov — Tue, 22 Jul 2025 00:00:10 UTC

Welcome to GitOps Journey — a hands-on guide to setting up infrastructure in Kubernetes using Git and automation.

GitOps has gained traction alongside Kubernetes, CI/CD, and declarative provisioning.
You’ve probably seen it mentioned in blog posts, tech talks, or conference slides — but what does it actually look like in practice?

We’ll start from scratch: prepare a cluster, deploy a PostgreSQL database, run a demo app, and set up observability — all managed via Git and GitHub using ArgoCD.

What We’ll Build

ArgoCD — syncs manifests from a GitHub repository to your cluster
PostgreSQL — a production-ready database using Percona Operator
Demo App — a real Go-based web app connected to the database
Coroot — an open-source tool for monitoring performance, logs, and service behavior

This series is for anyone new to GitOps or Kubernetes.
Each part includes clear steps, real-world YAML, and examples you can run yourself.

This is Part 1 of the GitOps Journey.
If you already have ArgoCD and a working Kubernetes cluster, you can skip ahead:

Part 2 – Deploying PostgreSQL with Percona Operator

Part 3 – Connecting a Real App to the Cluster

Part 4 – Observability with Coroot

Copilot assisted with formatting, Markdown structure, and translation.
All ideas, architecture decisions, and hands-on implementation were created by Daniil Bazhenov.

Otherwise, let’s start by preparing the cluster and setting up ArgoCD.

Creating a Kubernetes Cluster

I’ll be using Google Kubernetes Engine (GKE), but you can use AWS, DigitalOcean, or even run Minikube locally.

You’ll also need these CLI tools installed on your machine:

kubectl - The official CLI tool for Kubernetes — used to manage clusters, view resources, apply manifests, and more.
helm - A package manager for Kubernetes — lets you install complex apps using reusable charts (like PostgreSQL, monitoring tools, etc.)

I use the following command to create a cluster in GKE

gcloud container clusters create dbazhenov-demo \
 --project percona-product \
 --zone us-central1-a \
 --cluster-version 1.30 \
 --machine-type n1-standard-8 \
 --num-nodes=3

To delete the cluster:

gcloud container clusters delete dbazhenov-demo --zone us-central1-a

Note: This command doesn’t remove your LoadBalancers, so I prefer deleting them manually in Google Cloud’s web console to ensure no resources are left running post-experiment.

Here’s the resulting setup:

➜ community git:(blog_argocd_pg) ✗ kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-dbazhenov-demo-default-pool-b1b48316-8nrj Ready  6m7s v1.30.12-gke.1279000
gke-dbazhenov-demo-default-pool-b1b48316-8v14 Ready  6m6s v1.30.12-gke.1279000
gke-dbazhenov-demo-default-pool-b1b48316-zg6z Ready  6m7s v1.30.12-gke.1279000
➜ community git:(blog_argocd_pg) ✗

Installing ArgoCD

We’ll begin with ArgoCD, the GitOps engine that will deploy:

PostgreSQL database cluster

A demo app to simulate real usage

Coroot for monitoring and profiling workloads

ArgoCD supports multiple deployment methods — we’ll experiment with different ones during this series.

Install ArgoCD (based on official docs):

Create namespace:

kubectl create namespace argocd

Deploy ArgoCD:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Check the pods:

kubectl get pods -n argocd

Expected output: ArgoCD components running (server, repo, redis, controllers, etc.)

➜ community git:(blog_argocd_pg) ✗ kubectl get pods -n argocd
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 0 57s
argocd-applicationset-controller-6d569f7895-89kgk 1/1 Running 0 64s
argocd-dex-server-5b44d67df9-p42z5 1/1 Running 0 62s
argocd-notifications-controller-5865dfbc8-gqzwt 1/1 Running 0 61s
argocd-redis-6bb7987874-99j59 1/1 Running 0 61s
argocd-repo-server-df8b9fd78-64czj 1/1 Running 0 60s
argocd-server-6d896f6785-82tf2 1/1 Running 0 59s

Access ArgoCD UI

You have two options (or more):

Port forwarding (local only)

kubectl port-forward svc/argocd-server -n argocd 8080:443

Internet-accessible LoadBalancer

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

I will use Load Balancer by executing the command above, you need to wait a few minutes to get the IP address.

Let’s get the IP address of the ArgoCD service in the EXTERNAL-IP field.

kubectl get svc argocd-server -n argocd

➜ community git:(blog_argocd_pg) ✗ kubectl get svc argocd-server -n argocd
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
argocd-server LoadBalancer 34.118.234.162 34.132.39.194 80:30549/TCP,443:32146/TCP 9m51s

Access the UI in your browser using the IP.

Getting Started with ArgoCD Login

Download Argo CD CLI

Install ArgoCD CLI (see instructions).

Get the initial password:

argocd admin initial-password -n argocd

ArgoCD recommends changing it to a new secure password, which we will do.

argocd login 34.132.39.194 --insecure

Authorize using initial-password and user admin and execute the password update command

argocd account update-password

All the steps to get and update your password are below.

➜ community git:(blog_argocd_pg) ✗ argocd admin initial-password -n argocd
0mxV6IVcF3qZDR-O

 This password must be only used for first time login. We strongly recommend you update the password using `argocd account update-password`.
➜ community git:(blog_argocd_pg) ✗ argocd login 34.132.39.194 --insecure
Username: admin
Password:
'admin:login' logged in successfully
Context '34.132.39.194' updated
➜ community git:(blog_argocd_pg) ✗ argocd account update-password
*** Enter password of currently logged in user (admin):
*** Enter new password for user admin:
*** Confirm new password for user admin:
Password updated
Context '34.132.39.194' updated
➜ community git:(blog_argocd_pg) ✗

Now log into the ArgoCD web UI using admin and your new password.

Welcome to the ArgoCD interface, we don’t have any applications right now, we will install them later.

Setting Up GitHub Repo

We’ll need a GitHub repo to store infrastructure manifests. ArgoCD will sync from this repo and apply changes.

Install git and create a GitHub account
Add your SSH key to your GitHub profile. GitHub SSH settings
Create a new GitHub repository

I recommend a public repo for this educational project — no secrets will be committed, and it simplifies ArgoCD setup. Plus, it earns you some green squares on GitHub. If you go with a private repo, make sure it’s properly linked in ArgoCD.

Clone the repo:

Clone the repository using an SSH address.

git clone git@github.com:dbazhenov/percona-argocd-pg-coroot.git

Navigate to the project directory

cd percona-argocd-pg-coroot

Expected output:

➜ gitops git clone git@github.com:dbazhenov/percona-argocd-pg-coroot.git
Cloning into 'percona-argocd-pg-coroot'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
Receiving objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
➜ gitops cd percona-argocd-pg-coroot
➜ percona-argocd-pg-coroot git:(main) ls
README.md
➜ percona-argocd-pg-coroot git:(main)

Summary

We’ve prepared everything to launch our GitOps-powered infrastructure:

Kubernetes cluster
ArgoCD deployed
GitHub repo ready

In the next posts, we’ll deploy the PostgreSQL cluster, the demo app, and add Coroot monitoring.

Stay tuned!

Using replicaSetHorizons in MongoDB

Ivan Groenewold — Tue, 22 Jul 2025 00:00:00 UTC

When running MongoDB replica sets in containerized environments like Docker or Kubernetes, making nodes reachable from inside the cluster as well as from external clients can be a challenge. To solve this problem, this post is going to explain the horizons feature of Percona Server for MongoDB.

Let’s start by looking at what happens behind the scenes when you connect to a replicaset URI.

Node auto-discovery

After connecting with a replset URI, the driver discovers the list of actual members by running the db.hello() command:

mongosh "mongodb://mongo1-internal:27017/?replicaSet=rs0"
rs0 [direct: primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877b5e18a13d54b752ff25c'),
 counter: Long('6')
 },
 hosts: [ 'mongo1-internal:27017', 'mongo2-internal:27017', 'mongo3-internal:27017' ],
...

The list of hosts returned contains the name of each member as you provided it to the rs.initialize() command.

The node identity crisis

The names are resolvable inside the same network, so all is well in this case. But what happens when connecting from outside?

Typically you would be using names like mongo1-external.mydomain.com that correctly point to the external IP addresses of the members. The problem is that after the initial connection is made, the driver will perform auto-discovery and try to connect to the names as reported by db.hello(). These are not resolvable from outside.

What if we connect by IP address directly? again, the driver will get the names from the list above, try to reach those and fail after the initial connection is made:

$ mongosh mongodb://user:pass@10.30.50.155:32768/?replicaSet=rs0
Current Mongosh Log ID: 6849eb15ba228be45a69e327
Connecting to: mongodb://@10.30.50.155:32768/?replicaSet=rs0&appName=mongosh+2.5.2
MongoNetworkError: getaddrinfo ENOTFOUND mongo1-internal

Even though mongo1-internal is not part of the connection string, the driver tries to reach it. So if the replica set members advertise their internal IPs or DNS names, clients outside can’t connect unless they can resolve that same name. We could work around that, but there’s another issue: the ports.

The port issue

In the containerized world, it is likely that you set up your containers to use default port 27017. However they might be mapped to a different external port, since you have to avoid port collisions (think about the case where containers are co-located in the same host).

We need a way for replica set members to identify themselves with different names and ports, depending on whether the client is in the same network or outside. A concept similar to split-brain DNS.

What is Horizons?

Horizons is a MongoDB feature that allows replica set members to advertise different identities depending on the client’s access context, such as internal versus external networks.

With this, you can make the same MongoDB replica set usable from:

Internal container network (using internal hostnames/IPs)
External applications (using public IPs or DNS names)

MongoDB’s horizons rely on Server Name Indication (SNI) during the TLS handshake to determine which hostname and port to advertise. At connection time, clients present the hostname they used, and MongoDB uses that to return the proper set of endpoints. For that reason TLS is required in order for horizons to work.

Let’s walk through an example.

Example Scenario: MongoDB Replica Set in Docker

You can run the following steps on your local machine to test the feature.

Get your certificates ready

Let’s start by creating the required CA and certificates using Cloudflare’s PKI and TLS toolkit.

Step 1: Create ca-csr.json

mkdir certs
cd certs
tee ca-csr.json <{
 "CN": "MyTestCA",
 "key": { "algo": "rsa", "size": 2048 },
 "names": [{ "C": "US", "ST": "CA", "L": "SF", "O": "Acme", "OU": "MongoDB CA" }]
}
EOF

Generate the CA:

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca

This creates:

ca.pem — CA certificate
ca-key.pem — CA private key

Step 2: Create server-csr.json for each server, specifying both internal and external names in the “hosts” section so that our certificate is valid for everything.

for i in 1 2 3; do
 name="mongo$i"

 tee "${name}-csr.json" <{
 "CN": "${name}",
 "hosts": ["${name}", "${name}.internal", "localhost", "127.0.0.1"],
 "key": { "algo": "rsa", "size": 2048 },
 "names": [
 { "O": "MongoDB", "OU": "Database", "L": "Internal", "ST": "DC", "C": "US" }
 ]
}
EOF
done

Step 3: Generate certificates using CFSSL

for i in 1 2 3; do
 name="mongo$i"

 cfssl gencert \
 -ca=ca.pem -ca-key=ca-key.pem \
 -config=<(cat <<'JSON'
{
 "signing": {
 "default": {
 "expiry": "8760h",
 "usages": [
 "signing",
 "key encipherment",
 "server auth",
 "client auth"
 ]
 }
 }
}
JSON
) "${name}-csr.json" | cfssljson -bare "${name}"
cat "${name}.pem" "${name}-key.pem" > "${name}-combined.pem"
done

cd ..

Resulting files:

mongo{1,2,3}.pem — cert for server
mongo{1,2,3}-key.pem, key for server
mongo{1,2,3}-combined.pem, both in a single file as expected by mongo

Docker Compose Setup

Create a file with docker compose configuration:

tee test-horizons.yml <name: horizons
services:
 mongo1:
 container_name: mongo1
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27017:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo1-combined.pem
 --tlsCAFile /certs/ca.pem
 mongo2:
 container_name: mongo2
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27018:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo2-combined.pem
 --tlsCAFile /certs/ca.pem
 mongo3:
 container_name: mongo3
 image: percona/percona-server-mongodb:latest
 volumes:
 - ./certs:/certs
 ports:
 - "27019:27017"
 command: >
 mongod --replSet rs0 --bind_ip_all
 --tlsMode requireTLS
 --tlsCertificateKeyFile /certs/mongo3-combined.pem
 --tlsCAFile /certs/ca.pem
networks:
 default:
 driver: bridge
EOF

Here we are mapping our containers to ports 27017, 27018 and 27109 externally.

Now, start the services:

$ docker-compose -f test-horizons.yml up -d

Initiate the Replica Set with Horizons

Now let’s initiate the replica set with different host names and ports for external access.

Launch a shell into one of the containers:

$ docker exec -it mongo1 /bin/bash

Authenticate and initialize the replica set with this config:

$ mongosh --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsAllowInvalidCertificates
rs.initiate({
 _id: "rs0",
 members: [
 {
 _id: 0,
 host: "mongo1:27017",
 horizons: { external: "localhost:27017" }
 },
 {
 _id: 1,
 host: "mongo2:27017",
 horizons: { external: "localhost:27018" }
 },
 {
 _id: 2,
 host: "mongo3:27017",
 horizons: { external: "localhost:27019" }
 }
 ]
})

Note: The “horizon” field here maps the external context to a different address than the internal one. Since we are going to test connecting from the local machine directly to the containers, set the horizons to localhost and the mapped ports.

Connect from Inside Docker

Spin up a new containerized client, or use one of the existing MongoDB containers:

$ docker exec -it mongo1 mongosh --host rs0/mongo1:27017,mongo2:27017,mongo3:27017 --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsCAFile /certs/ca.pem
Current Mongosh Log ID: 6877deab6568339f46dfd9c4
Connecting to: mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0&tls=true&tlsCertificateKeyFile=%2Fcerts%2Fmongo1-combined.pem&tlsCAFile=%2Fcerts%2Fca.pem&appName=mongosh+2.5.0
Using MongoDB: 8.0.8-3
Using Mongosh: 2.5.0

rs0 [primary] test>

It connects using internal Docker hostnames.

Connect from Outside Docker

From your local machine:

$ mongosh "mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0" --tls --tlsCertificateKeyFile /certs/mongo1-combined.pem --tlsCAFile /certs/ca.pem
Current Mongosh Log ID: 6877defabc3f9a2d054a1296
Connecting to: mongodb://localhost:27017,localhost:27018,localhost:27019/?replicaSet=rs0&serverSelectionTimeoutMS=2000&tls=true&tlsCertificateKeyFile=certs%2Fmongo1-combined.pem&tlsCAFile=certs%2Fca.pem&appName=mongosh+2.3.1
Using MongoDB: 8.0.8-3
Using Mongosh: 2.3.1

rs0 [primary] test>

Check the identities returned

As we have seen, MongoDB will resolve the external horizon names and connect successfully in both cases. You can verify the advertised hostnames and ports for the external connection:

rs0 [primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877de4c632adf89fb590f38'),
 counter: Long('6')
 },
 hosts: [ 'localhost:27017', 'localhost:27018', 'localhost:27019' ],
 setName: 'rs0',
 setVersion: 1,
 isWritablePrimary: true,
 secondary: false,
 primary: 'localhost:27017',
 me: 'localhost:27017',
...

Versus the internal case:

rs0 [primary] test> db.hello()
{
 topologyVersion: {
 processId: ObjectId('6877de4c632adf89fb590f38'),
 counter: Long('6')
 },
 hosts: [ 'mongo1:27017', 'mongo2:27017', 'mongo3:27017' ],
 setName: 'rs0',
 setVersion: 1,
 isWritablePrimary: true,
 secondary: false,
 primary: 'mongo1:27017',
 me: 'mongo1:27017',
...

Conclusion

The horizons feature in MongoDB is a powerful tool to bridge the gap between internal and external connectivity, especially in containerized or multi-network deployments.

Horizon also has following limitations:

Using horizons is only possible with TLS connections
Duplicating domain names in horizons is not allowed by MongoDB
Using IP addresses in horizons definitions is not allowed by MongoDB
Horizons should be set for all members of a replica set, or not set at all

This feature is not listed in the official MongoDB documentation for some reason, however it is available in both Percona Server for MongoDB and MongoDB Community Edition. Also, Kubernetes users rejoice! Percona Operator for MongoDB supports horizons since version 1.16.

Active-active replication - the hidden costs and complexities

Jan Wieremjewicz — Thu, 10 Jul 2025 00:00:00 UTC

In Part 1 of this series, we discussed what active-active databases are and identified some “good” reasons for considering them, primarily centered around extreme high availability and critical write availability during regional outages. Now, let’s turn our attention to the less compelling justifications and the substantial challenges that come with implementing such a setup.

What are “bad” reasons?

1. Scaling write throughput

Trying to scale your write capacity by deploying active-active across regions may sound like a clean horizontal solution, but it is rarely that simple. Write coordination, conflict resolution, and replication overhead introduce latency that defeats the purpose. If you are thinking about low latency writes between regions like Australia and the US, keep in mind that you will still be paying the round-trip cost, typically 150-200ms+, to maintain consistency. Physics don’t do any favors. Even if you have multiple primaries, unless you accept weaker consistency or potential conflicts, your writes will not scale linearly. In many real-world cases, throughput actually suffers compared to a well tuned primary replica setup. If your real goal is better throughput, you are usually better served by:

Regional read replicas
Sharding
Task queuing and eventual delegation

2. Performance

Performance is often used as a vague justification. But active-active is not a panaceum for general slowness. If the issue is database bottlenecks, reasons may be as basic as not enough work spent on data structuring, indexing, query tuning or scaling reads before jumping into multi primary deployments. Way too often we find that performance problems come not from scale limits, but from poor design and neglected maintenance. We are in 2025 and lessons like this classic one on fast inserts are still painfully relevant. If what you are really facing is application level latency, that is a different challenge. And even then, active-active with strong guarantees often adds more latency, not less.

3. Load balancing

You don’t need multi-primary to load balance. If your goal is spreading traffic more evenly, and reads are your bottleneck, there are simpler and much safer ways to get there. That’s what read replicas are for. If what you are looking for is distributing writes, please re-read point 1: synchronizing state across regions adds cost, complexity, and consistency challenges. Multi-master does not eliminate contention, it just moves it. In these situations you will have conflicts, some due to lag, some for other reasons. Resolution logic becomes critical. Conflicts need to be logged and their resolution auditable.

4. Because other systems did it

The fact that multi master was possible in other systems is not, on its own, a good reason to adopt it. If you have worked with Oracle RAC or GoldenGate, you likely remember the long nights of debugging conflicts, performance issues, the layers of failover logic, the dedicated teams just to keep things running smoothly. These systems made big promises, but keeping them stable in production was rarely simple or cheap. Copying those patterns means copying those problems. And trying to reproduce them with PostgreSQL, or some other modern databases, often misses the point. This is our chance to rethink how we build, not repeat the same mistakes. If you are going to pay the cost of active active, make sure you are doing it for the right reasons, not just because it looks like something you are used to.

The cost, the painful side of active-active

Let’s be very clear: active-active is not the shortcut it often appears to be. It’s not simply “HA but better.” It represents a fundamental system design change with significant, ongoing costs across engineering, operations, and product management. While it might be the only viable option in highly specific situations, by betting on extreme reliability, you are signing up for a hell of a lot of complexity.

Much of what makes active-active the wrong choice in typical scenarios comes down to these costs. So expect some repetition, that’s by design, because the price is the problem.

1. Conflict resolution is a problem of it’s own

Every multi-primary system has to answer one question: what happens when two nodes write to the same row at the same time? PostgreSQL doesn’t have conflict resolution built-in. You can bolt it on with tools like pgactive or pglogical2, but none of them are perfect, far from it.

The most common approaches applied:

Last write wins (based on timestamps).
Source priority (some nodes win conflicts).
Application-defined merge logic.

They all come with trade-offs. None of them are free.

Consider how other systems handle this is very often not conflict handling but conflict avoidance:

CockroachDB avoids exposing raw conflict handling to users by using strict serializability and sacrificing performance
Oracle RAC handles it via centralized locking over a shared-disk architecture (and even that causes headaches)
CouchDB, which embraces multi-master natively, makes conflict resolution a first-class application responsibility and warns users upfront

Unless your app is very carefully designed to dodge conflicts, say, by writing to disjoint subsets of data, you’re going to run into this. Probably when it’s already too late and your data’s out of sync.

2. Infrastructure & networking costs

Running active-active across regions increases your infra footprint and cost. Specifically:

More powerful machines or more of them – each node handles writes and replication
Higher network bandwidth usage – especially for busy systems or cross-region replication
Premium network setups – often needed for low-latency consistency
Increased complexity in monitoring – replication lag, node health, consistency checks

These costs aren’t just financial, they bleed into reliability and operational overhead too.

3. Operational and management overhead

Beyond setup, day-2 operations are where the real pain begins:

Deployments become dangerous – schema or app changes can break replication or trigger conflicts
Debugging is harder – tracing transactions across writeable nodes adds layers to incident response
Staffing costs rise – managing active-active needs deeper expertise in distributed systems
Auditing grows in importance – proving data consistency across nodes becomes a continuous task

Let’s be honest, what was hard before is now harder, and even the routine becomes risky.

Don’t kill the messenger

I love food. Cooking, eating, talking about it. Anyone I’ve worked with can vouch that I’m a foodie at heart. I like spicy food, really spicy. In my kitchen, you’ll find everything from Sichuan pepper to naga jolokia “ghost pepper”, from kala namak to asafoetida. These aren’t ingredients you throw into every dish. The right spice, at the wrong time, ruins everything. The same spice, used precisely, can turn something ordinary into magic.

What’s exactly how active-active works. It’s not bad by nature, it’s just very specific. You need to understand the dish, the eater, and the context. If you add ghost pepper to scrambled eggs because someone said it’s “cool,” you’re very likely going to regret it. Same goes for multi-primary setups.

Know what you’re in for. Use the right tool for the right reason. And don’t reach for active-active just because it sounds hot. (And if you do pull it off successfully, please send me your config. I owe you a beer.)

Percona Bug Report: June 2025

Aaditya Dubey — Mon, 30 Jun 2025 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs.

Percona Server/MySQL Bugs

PS-9823: mysql_migrate_keyring fails with PS Components.

The failure is triggered by a missing symbol, but the underlying cause is the way keyring components are built in Percona Server. When attempting to migrate keyring data (e.g., from Vault to File), the tool fails to load the Percona Server component .so files, making the migration process unusable.

Percona Server builds a reference to the my_free symbol, which is not properly resolved in the shared libraries. In contrast, upstream MySQL builds do not include this dependency.

This issue blocks both component-to-component and component-to-plugin keyring migrations, affecting users who rely on secure key management transitions.

Reported Affected Version/s: 8.0.x, 8.4.x
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9836: There is a regression issue with audit_log_filter.so compared to audit_log.so. The audit_log_filter, whether used as a plugin (8.0) or a component (8.0 and 8.4), shows a significant performance regression. When logging everything, QPS drops by over 70%. While configuring selective logging can reduce the impact, it still results in a 30–35% drop in QPS.

For this reason, moving to audit_log_filter in 8.0 is not recommended. Additionally, this should be taken into account when planning upgrades to 8.4, as audit logging can significantly impact performance. (audit_log is not available as a component—only as a plugin.)

Reported Affected Version/s: 8.0.42-33, 8.4.5-5
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9837: A crash occurs on replica nodes during parallel replication when an INSERT is executed on a secondary index that recently had a DELETE on the same key. The issue is caused by a race condition in the secondary index reuse logic, leading to an assertion failure (row0ins.cc:268).

This issue is more likely to occur under heavy write workloads, particularly when the application frequently performs DELETE followed by INSERT on the same keys. It only affects replica servers where replica_parallel_workers > 0 and slave_preserve_commit_order=ON.

Reported Affected Version/s: 8.0.36-28, 8.0.42-33
Upstream Bug: 118334
Workaround/Fix: The user can modify their logic to use UPDATE instead of DELETE followed by INSERT, which avoids triggering the crash path.
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9861: The audit_log_filter plugin cannot be installed when component_keyring_kmip is enabled with Fortanix DSM. While testing with component_keyring_kmip, we enabled the “Allow secrets with unknown operations” option in Fortanix, which allowed the audit log installation to proceed one step further. At this point, a secret is successfully created for the audit log, but MySQL crashes upon restart.

This issue is related to bug PS-9609 and still persists when using Fortanix DSM as the KMIP server.

Reported Affected Version/s: 8.0.42-33
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: The issue has been fixed, and the fix is expected in the upcoming release of Percona Server (PS).

PS-9914: After running ALTER TABLE … ENGINE=InnoDB to rebuild a large table (~10 million rows) with ROW_FORMAT=COMPRESSED, it was observed approximately a 50% drop in write-only workload throughput (measured via sysbench), despite a reduction in .ibd file size and no changes to table structure or indexes. The table had previously undergone heavy deletions (~50%), suggesting possible fragmentation prior to the rebuild.

Reported Affected Version/s: 8.0.37-29, 8.0.42-33
Upstream Bug: 118411
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation. A fix or workaround is expected in a future release.

PS-9956: PS 8.4.4-4 with group replication crashes on Oracle Linux 9 during bootstrap or failover when the audit log filter component is enabled, but does not crash on Oracle Linux 8.

Reported Affected Version/s: 8.4.4-4
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: Under investigation.

Percona Xtradb Cluster

PXC-4652: PXC 8.4 crashes with a SIGSEGV in unordered_map called from rpl_gtid_owned during high activity, while PXC 8.0 under the same workload and data remains stable; the crash occurs randomly during operations like COMMIT or INSERT.

Reported Affected Version/s: 8.4.3, 8.4.4
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: 8.4.5 – Pending Release

PXC-4684: An UPDATE query that joins two tables but modifies only one—e.g., UPDATE test.t2 JOIN test.t1 USING (i) SET t2.d = t2.d+1, t1.d = t1.d;—causes an MDL BF-BF conflict on other PXC nodes, even without triggers, as both tables are included in the Table_map_log_event.

Reported Affected Version/s: 8.0.41
Upstream Bug: Not Available
Workaround/Fix: Not Available
Fixed/Planned Version/s: 8.0.42 – Released | 8.4.5 – Pending Release

Percona Toolkit

PT-2418: In pt-online-schema-change 3.7.0, data was lost when executing the following SQL — the value of column col_2 was unexpectedly set to NULL:

Eg:

ALTER TABLE t RENAME COLUMN col_1 TO col_2;
MySQL version: 8.0+
pt-online-schema-change --no-version-check \
 h=127.0.0.1,u=root,p=xxx,P=xxx,D=sysbench,t=sbtest1 \
 --alter="RENAME COLUMN col_1 TO col_2" \
 --execute --statistics

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2419: pt-duplicate-key-checker Ignores DESC in Index Definitions. Users running pt-duplicate-key-checker regularly observed that a newly added composite index was being incorrectly flagged as a duplicate and removed.

Before:

KEY `idx_ts` (`ts`),
KEY `idx_ts_id` (`ts` DESC, `id`)

After:

KEY `idx_ts_id` (`ts`)

The tool appears to ignore the DESC direction in index definitions, leading to incorrect de-duplication. This behaviour may affect query plans and performance in setups relying on sort order.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2425: Case-Sensitive MariaDB Detection Causes Sync Failure in pt-table-sync. In pt-table-sync 3.7.0, a case-sensitive check for the MariaDB flavor ($vp->flavor() =~ m/maria/) fails because flavor() returns “MariaDB Server”, causing the condition to evaluate incorrectly. As a result, the tool looks for source_host and source_port in $source, while the actual keys are master_host and master_port, leading to failures or uninitialized value warnings.

The Error:

Use of uninitialized value in concatenation (.) or string at /usr/bin/pt-table-sync line 7086.

Manually updating the regex to m/maria/i resolves the issue. Similar case-sensitive checks appear elsewhere in the script and may require centralizing the MariaDB detection logic.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 3.7.1 - Not Yet Released

PT-2197: In pt-online-schema-change (version 3.7.0), attempting to run an ALTER operation results in the following error:

Use of uninitialized value in string eq at /usr/bin/pt-online-schema-change line 4321

This occurs even when replica connectivity in both directions is fully functional and multiple replicas are connected. Notably, the issue does not occur in version 3.5.1, where the operation succeeds as expected (with the expected increase in connections). Schema change automation breaks unexpectedly on newer versions despite a valid replication setup.

Reported Affected Version/s: 3.5.2, 3.6.0, 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2432: While pt-replica-find includes internal logic for handling replication channels, it currently lacks a corresponding --channel command-line option. Attempting to use it results in an error:

$ pt-replica-find --channel=foo
Unknown option: channel

This prevents users from specifying a replication channel directly, limiting the tool’s usability in multi-channel replication environments.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2448: pt-k8s-debug-collector should not collect secret details of pgbouncer

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PT-2446: When attempting to run pt-table-checksum with Group Replication enabled, and the tool returns the following error:

Error checksumming table schema.table: DBD::mysql::st execute failed: The table does not comply with the requirements by an external plugin. [for Statement "DELETE FROM percona.checksums WHERE db = ? AND tbl = ?" with ParamValues: 0=' ', 1=' '] at /bin/pt-table-checksum line 11323.

It gets suspected that this is caused by the tool attempting to set @@binlog_format := ‘STATEMENT’, which is not supported under Group Replication.

Reported Affected Version/s: 3.7.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM [Percona Monitoring and Management]

PMM-13994: pmm_agent shows disconnected status despite active metrics collection, After a temporary connectivity issue, pmm_agent continues to display a Disconnected status in pmm-admin list, even though connectivity has been restored and dashboards are populating correctly.

$ pmm-admin list
...
pmm_agent Disconnected
...

Reported Affected Version/s: 2.43.2, 2.44.1, 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Restarting the pmm_agent should fix the issue.
Fixed/Planned Version/s: 3.4.0 - Not Yet Released

PMM-13905: When adding both a MongoDB Cluster and a standalone MongoDB Replica Set (not part of the cluster) to the same PMM environment (e.g., “test”), the MongoDB ReplSet Summary dashboard does not allow viewing the standalone RS.

The “cluster” filter cannot be unselected, making it impossible to visualize replica sets that are not associated with a defined cluster. As a result, only RSs from the cluster are visible, while standalone RSs are excluded from the dashboard view.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Whenever possible, use separate environments when adding the cluster and standalone RS nodes in PMM (e.g., use “env1” for the cluster and “env2” for the standalone RS).
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13910: In the MongoDB Sharded Cluster Summary and Collections dashboards, several graphs fail to populate correctly. Specifically:

Top Hottest Collections by Read
Top Hottest Collections by Write

These graphs display only admin, config, and system collections, even when other collections are under heavy traffic. Additionally, the Collections dashboard shows no data across all graphs—except for the first one (Top 5 Databases By Size), which populates as expected.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13950: In both PMM 2 and PMM 3, with MySQL 5.7 and MySQL 8.0, the server_uuid is not being collected from MySQL’s global variables as expected. Despite being available via SHOW GLOBAL VARIABLES LIKE ‘server_uuid’;, the PMM agent fails to parse or capture this value.

Reported Affected Version/s: 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

PMM-13792: In PMM 2.44.0, the Advisor Insights incorrectly reports that journaling is not enabled for MongoDB 7.0.9-15, despite journaling being enabled by default in this version.

Attempts to explicitly enable journaling in the MongoDB config result in a startup warning:

The storage.journal.enabled option and the corresponding --journal and --nojournal command-line options have no effect in this version... Journaling is always enabled. Please remove those options from the config.

False alert may confuse users and lead to misconfiguration attempts that prevent MongoDB from starting.

Reported Affected Version/s: 2.44, 3.1.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

Percona Kubernetes Operator

K8SPG-772: In the Percona PostgreSQL Operator, a runtime panic occurs when CompletedAt is nil and not properly checked before dereferencing, leading to a segmentation fault:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0]

Stack trace:

github.com/percona/percona-postgresql-operator/percona/watcher.getLatestBackup
 .../wal.go:123

github.com/percona/percona-postgresql-operator/percona/watcher.WatchCommitTimestamps
 .../wal.go:65

The CompletedAt field is not validated before being accessed in getLatestBackup(), which causes a crash during WAL watcher execution.

This panic can crash the operator’s goroutine, interrupting WAL monitoring and potentially affecting backup or failover logic.

Reported Affected Version/s: 2.5.0, 2.6.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.7.0 - Pending Release

K8SPG-792: The upstream operator includes functionality that allows cluster or operator administrators to define default PostgreSQL images for each major version using environment variables. This enables users to create clusters without explicitly specifying spec.image, as the operator will automatically apply the predefined image.

However, a recently introduced Patroni version check does not align with this behavior. It introduces a hardcoded dependency on spec.image, effectively bypassing the default image mechanism and undermining the intended feature.

Reported Affected Version/s: 2.6.0
Upstream Bug: Not Applicable
Workaround/Fix: A possible workaround exists by manually setting the Patroni version through annotations, but this is not ideal and diminishes the convenience and flexibility originally provided.
Fixed/Planned Version/s: A fix or workaround is expected in a future release.

K8SPXC-1651: While testing the Pod Scheduling Policy feature in Everest, we encountered a situation where a PXC database pod remained in the Pending state. This occurred because Kubernetes was unable to schedule the pod on any available node due to an affinity configuration mismatch.

However, even after updating the affinity rules in the PerconaXtraDBCluster object, the new configuration was not propagated to the pod, and it remained in the Pending state.

The fact that the Pod remains stuck in Pending even after affinity is changed or removed — and only a manual kubectl delete pod resolves it — indicates that the operator fails to reconcile affinity changes properly.

Note: This issue affects other operators as well, not just PXC.

Reported Affected Version/s: 1.17.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 1.20.0 - Yet to be released

K8SPXC-1648: The PVC size is rounded up to the nearest whole GiB value (e.g., 1.2Gi becomes 2Gi). When a storage resize operation is triggered, the operator deletes the existing StatefulSet (STS) and recreates it with the new requested PVC size.

However, if the new requested size rounds up to the same value as the original, the operator does not recreate the STS. Instead, it attempts to update the existing STS, which leads to the following error:

Note: This issue affects other operators as well, not just PXC.

Error: failed to deploy pxc: updatePod for pxc: failed to create or update sts:
update error: StatefulSet.apps "minimal-cluster-pxc" is invalid: spec: Forbidden:
updates to statefulset spec for fields other than 'replicas', 'ordinals',
'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and
'minReadySeconds' are forbidden

Reported Affected Version/s: 1.17.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 1.20.0 - Yet to be released

PBM [Percona Backup for MongoDB]

PBM-1499: Restore to Missing Backup Fails with Unclear Error in Restore Custom Resource Status.
When attempting to restore from a backup that does not exist in the main storage, the operator logs correctly report the failure:

define base backup: get backup metadata from storage: get from store: no such file

However, the status.error field in the PerconaServerMongoDBRestore custom resource only shows a generic message:

error: 'define base backup: %v'

This results in a misleading or unclear error message being surfaced to the user through the custom resource, even though the logs contain the full and accurate description of the issue.

Reported Affected Version/s: 2.8.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1502: In PBM 2.9.0, running pbm profile sync fails with the error:

Error:  or --all must be provided

This occurs even when a valid profile name is given, such as:

pbm profile sync azure-blob

The issue affects all defined profiles (azure-blob, gcp-cs, minio) and prevents syncing individual profiles. This appears to be a bug where the CLI fails to recognize the passed argument.

Reported Affected Version/s: 2.9.0
Upstream Bug: Not Applicable
Workaround/Fix: Use pbm profile sync --all instead
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1538: Backup is marked as successful, despite the oplog not being uploaded.

Reported Affected Version/s: 2.4.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1551: In a single-node PSMDB replica set with one PBM agent, PBM occasionally re-executes the last issued command, causing errors like:

active lock is present

This typically occurs when the database is under load.

Reported Affected Version/s: 2.9.1
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1553: When restoring a 33-shard physical backup from a mixed (MongoDB Enterprise + Percona) production cluster into a Percona-only test cluster, PBM intermittently fails during the “clean-up and reset replicaset config” stage. Some shards restore successfully, while others restore only partially or fail entirely.

Both clusters run MongoDB 6, with FCV set to 5.
Restore uses --replset-remapping due to different replica set names
Issue affects restores regardless of matching node count per shard.

The problem appears tied to the restore logic handling replica set configuration cleanup.

Reported Affected Version/s: 2.9.1
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

PBM-1564: A user environment experiences repeated failures during incremental backups with the error:

[ERROR: cannot use the configured storage: source backup is stored on a different storage]

Full, base incremental, and logical backups succeed without issues. There is no indication of recent storage or configuration changes, and pbm status shows backup attempts occur close together.

The issue temporarily resolves after running pbm config --force-resync, suggesting a possible bug in storage metadata syncing or internal state handling.

Reported Affected Version/s: 2.8.0
Upstream Bug: Not Applicable
Workaround/Fix: Not Available
Fixed/Planned Version/s: 2.10.0 - Released

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Geeks Go Peaks: Building Community Through Adventure and Challenge

Iuliia (Julie) Slobodian — Wed, 25 Jun 2025 00:00:00 UTC

We believe this is something every strong team aspires to: taking interactions beyond the office, building a sense of community outside of work, and getting to know each other beyond job titles. That’s the kind of value people seek when they come to us.

People want to feel alive, to experience new sensations, to see the world from a different perspective - and share that journey with fellow enthusiasts.

Join Our Adventure Community: Explore, Connect, Thrive

The Geeks Go Peaks community is all about bringing people together through unforgettable experiences. Whether you’re seeking a relaxed getaway with your partner or craving the thrill of a high-altitude expedition, we’ve got events for every adventurer. From laid-back activities like quad biking or ice dipping to group micro-adventures such as SUP boarding in central Berlin, caving, or rock climbing, there’s something for everyone. For the bold, we organize serious expeditions, including high-altitude mountain ascents. So far, we’ve summited nine peaks, each offering breathtaking views, and we’re eyeing Mount Kazbek in Georgia and Thorong Peak in Nepal next.

Our mission?

To build a vibrant community of explorers who value shared experiences and the discovery of the world’s cultural and natural treasures. These adventures do more than just create memories - they spark lifelong friendships and expand your personal and professional networks. Through our events, people find their best friends and continue exploring the world together. It’s a joy to bring such value to the world.

Why the IT Community?

The IT industry is full of driven individuals who spend long hours behind screens. At some point, many realize they need more - more movement, more nature, more challenges. Outdoor activities like running, climbing, obstacle races, diving, or windsurfing not only boost physical health but also recharge mental well-being. IT professionals are often high-achievers, eager to push their limits, whether that’s training for a marathon or summiting iconic peaks. We connect these like-minded adventurers, fostering a community where they can inspire and support each other.

“You can’t beat shared suffering as team building activity” – Peter Zaitsev (Founder at Percona, Altinity, FerretDB, Coroot Database)

What We’ve Done

Our past events showcase the diversity of our adventures:

Aconcagua, Argentina: A grueling climb to nearly 7,000 meters rewarded us with jaw-dropping views and a profound sense of achievement. “Aconcagua was the trip of a lifetime. I’d recommend it to anyone in good shape and hungry for a challenge. You won’t regret it!”
Caving in Budapest: An underground world surprised even locals, proving adventure is often closer than you think. “I’m from Hungary and never explored the caves before. Doing it with GGP was the perfect way to finally change that - I absolutely loved it!”
Rock Climbing in Red Rock Canyon, Nevada: First-timers conquered the walls, forging bonds while cheering each other on. “I couldn’t have asked for a better first climbing adventure. The group, the views - everything was awesome.”
Northern Lights in Lapland: Snowy forests and vibrant skies created magical nights and dreams of future trips. “This wasn’t my first GGP trip - we’re practically family now. It was magic seeing the lights every single day.”
Via Ferrata in the Dolomites, Italy; a descent through Vallée Blanche in France; local events with paddleboarding and rock climbing in North Carolina… and that’s just the beginning. We already have so much planned for this year and the next!

Upcoming Adventures

Our calendar is brimming with thrilling plans. In August, we’ll tackle the exhilarating ascent of Mount Kazbek. Then we’re off to hike under the midnight sun in Svalbard, an awe-inspiring land where untouched Arctic nature reigns and polar bears outnumber people.

We’ll ride the wild waters of the Gully River during dam release season, set sail on catamarans through the British Virgin Islands in February, and explore wintertime Alaska with freeride skiing, glacier ice climbing, and evenings in a Nordic sauna.

And that’s just the beginning - Nepal, Bolivia, the Grand Canyon, the volcanoes of Vanuatu, Peru, and so much more are waiting.

For those in the USA, we’ve got local adventures like Grand Canyon Rim-to-Rim hikes, quick one- and two-day hikes, paddleboarding and kayaking, cycling, fishing and yachting, rock climbing, and even diving. And events tied to conferences such as Re:Invent, ATO Raleigh, All Things Open RTP, and Mountain Data & Dev Conference in Utah.

And this fall, on September 21, we’re hosting a women’s charity run - CODE PINK RUN! All proceeds will go to the Pretty In Pink Foundation to support women in their fight against breast cancer. Join us! We’d love to see everyone at this celebration of female solidarity!

Stay Connected

We keep our community engaged via newsletters, Meetup, Slack, Telegram, and Strava - where we train together.. Our Slack platform lets members discuss training plans, gear, and expedition prep, making us more than just a club, but a true community united by shared passions.

Anyone can suggest an adventure they’d like to be part of! We’re always excited about exploring new countries and trying new activities, sporty or not. Our goal is to help people find like-minded adventurers to reach new heights and share unforgettable experiences together!

Join us to explore the world, meet incredible people, and create stories you’ll tell for years.

Let’s adventure together!

PostgreSQL active-active replication, do you really need it?

Jan Wieremjewicz — Wed, 18 Jun 2025 00:00:00 UTC

Before we start, what is active-active?

Active-active, also referred to as multi-primary, is a setup where multiple database nodes can accept writes at the same time and propagate those changes to the others. In comparison, regular streaming replication in PostgreSQL allows only one node (the primary) to accept writes. All other nodes (replicas) are read-only and follow changes.

In an active-active setup:

There is no single point of write.
Applications can write to any node.
The database needs a way to sort out conflicts when two nodes try to concurrently change the same data.

That last point is the hardest one. PostgreSQL was not designed for concurrent writes from multiple nodes; it’s not a distributed database and does not leverage proprietary dedicated storage capabilities. So, every multi-primary implementation has to solve the issue of conflicting concurrent writes somehow. Some resolve conflicts using timestamps or priorities. Some push conflict resolution to the application. Some avoid it altogether by writing to separate subsets of data.

While simple in concept, implementing an active-active configuration is challenging.

pgactive to the rescue?

Last week, Amazon open-sourced its active-active replication extension, pgactive (https://github.com/aws/pgactive). While the extension has been generally available on AWS RDS since October 2023, there are unfortunately not many stories about it being used in production available. To be fair, I was not able to find any 😟

We often see both users and customers come asking for active-active or multi-master. These terms, while different, are so often used as synonyms that we’ve come to expect that. So, though I understand that every multi-master is active-active but not necessarily the other way around, for the sake of clarity, if I use one or the other term throughout this post, they will refer to the same concept.

As it is an open-source extension now, it immediately raised my interest. It seems that it could cover this ask from users I often speak with about their pains and needs. As a product manager, when I hear an ask, I always try to understand the reasons—whether it is a requirement, a need, or actually a solution that addresses one. For multi-master, my strong opinion is that it is a solution.

Key question: do you need it?

I like the opening of the talk Johnathan Katz gave on PGConf Europe 2023 in Prague:

The first thing I always say on the journey to active active is: do you really need it? Because it definitely solves a lot of problems (…) but it’s very hard to manage.

That is exactly the first question I ask when I hear someone asking for active-active. We have seen teams introduce active-active replication for the wrong reasons. Here I have to pause. Yes, as database experts, we have strong opinions about what are the right reasons for using multi-master. It’s not a silver bullet. It’s not “cool infra.” And using it without a good reason tends to hurt for a long, long time.

So, what are the reasons to use active-active? I do not claim to be able to cover all scenarios, but I hope this post raises enough eyebrows and sparks enough discussion to eventually have solid reading material for anyone considering active-active that will help them make an informed decision.

What are “good” reasons?

These are some of the situations where active-active might actually make sense. While there may be more, here’s my top 5:

Business continuity across regions: extreme HA needs (99.999% uptime)

Just to remind everyone what 5 nines mean, I will refer you to this message:

uptime → max monthly downtime:

99% → 7.3 hours
99.9% → 44 minutes
99.99% → 4 minutes
99.999% → 26 seconds
99.9999% → 3 seconds
99.99999% → 1/4 second

where do you land?
how much time/money would you invest to add a 9?
— Ben Dicken (@BenjDicken) May 23, 2025

26 seconds of downtime a month, that’s 312 seconds a year. Yes, 5.2 minutes a year.

Now think about the cost of delivering that sort of reliability. I find this Wikipedia page surprisingly helpful in conveying how little time for maintenance and failures is left with enough nines added.

Consider what it would take to absorb failures across data centers or cloud regions without rejecting writes or failing over manually. Active-active can help here because failover becomes instant and transparent; write traffic just shifts to surviving nodes.

But again, the cost will match the ambition. Do you plan HA within the same server room with separate power and networking? Or are you aiming for full geographic separation, to stay online even during a country-wide outage? These decisions massively influence the architecture, and together with your uptime goals, they define the cost. At this level, every part of the solution should reflect real business needs, because every layer of complexity adds expense. You can’t overstate the value of planning and proper analysis when building systems like this.
Write availability during regional failures

If your business serves a global customer base and absolutely must accept writes in more than one region, for example, to maintain uptime guarantees or continue operating during a regional outage, then active-active might be the least painful of the painful options.

This is not about low latency. This is about keeping write traffic flowing even when something breaks. That includes:
- Cloud infrastructure outages, such as full region loss or core service failure from your cloud provider:
  - AWS us-east-1 outage in June 2023 affected 104 services. A Parametrix Insurance report estimated a 24-hour outage in this region could lead to $3.4 billion in direct revenue loss.
  - Google Cloud outage in June 2025 impacted Spotify, YouTube, Twitch, and others.
  - AWS S3 outage in 2017 was caused by an internal mistake and disrupted GitHub, Slack, and more.
    
    Joys of the @internetofshit - AWS goes down. So does my TV remote, my light controller, even my front gate. Yay for 2017.
    — Brian (@Hamster_Brian) February 28, 2017
- Name resolution and routing issues, such as DNS or BGP failures that take your services offline even when your backend is healthy:
  - Dyn DNS attack in 2016 brought down Twitter, Reddit, and Spotify.
  - Facebook DNS and BGP misconfiguration in 2021 made their domains unreachable and left millions of users in the dark.
  - That day, Twitter (rest in peace) greeted the internet:
All jokes aside, these are serious risks. If this kind of failure is unacceptable for your business, and you are willing to take on the operational weight and cost (we will get to that), active-active may be the right tool.

But be honest about what you are solving. If your system demands strong consistency, every transaction still needs coordination across nodes. For example, if a user in Australia writes to a local node, and the other node is in the United States, that write still involves a round trip to the United States before it can commit. That round trip adds latency, not removes it. While it may be 150-200ms on average for the Australia to USA round trip, it adds up with volume.

The real benefit of active-active here is not performance. It is write availability during failure. If your business cannot afford to reject writes when a region goes dark, and you are prepared for everything else that comes with this decision, this might be one of the rare cases where active-active makes sense.

Just be clear, what you are solving here is not distributed latency, but write continuity when something fails.
Migrating legacy architectures

If you’re part of an organization moving away from systems like Oracle RAC or GoldenGate, where distributed write semantics were either built-in or at least promised, you may face business or political pressure to deliver “the same thing” on PostgreSQL.

In these cases, active-active might be the shortest path to satisfying the checkbox. But it’s almost always a transitional compromise, not the destination. As any compromise, that’s not going to be all pleasant. The technically better (but less politically correct) move is usually to re-architect for clearer ownership of writes and better separation of concerns.

If you can push for that path, do it. If not, be aware of the cost you’re inheriting.
Application performance (not database performance)

In the end, what you are really trying to improve is not the database throughput, but the end-user experience. Active-active may be worth considering not for improving database internals, but for reducing perceived latency in globally distributed apps or smoothing responsiveness during network transitions.

In rare cases, this might justify active-active if the application can route users to their nearest region and issue local writes. But your app must be built for it. Deterministic conflict handling, idempotency, and careful session management are must-haves in such a case.

If your database is fast, but the user still feels lag because the write travels halfway across the planet, active-active might help. But this should be a last resort, not a default choice.
Local HA in disconnected or semi-connected environments

In edge computing, retail stores, ships, or military use cases, you might want each node to function independently to address intermittent connectivity. In such scenarios, you will still be able to write locally when the network is not available. When the network comes back, the changes are going to be synced. While conflict avoidance may be the strategy you go for, in the end, it’s going to become a cost of conflict resolution.

What’s next?

In the next blog post I will focus on the bad reasons to consider active-active replication and on the cost that should not be forgotten. Stay tuned!

What's new in PMM 3.2.0: Five major improvements you need to know

Catalina Adam — Tue, 03 Jun 2025 00:00:00 UTC

PMM 3.2.0 brings some long-awaited fixes and new capabilities. You can now install PMM Client on Amazon Linux 2023 with proper RPM packages, get complete MySQL 8.4 replication monitoring, and track MongoDB backups directly in PMM.

Here’s what’s most important in this release:

1. Native Amazon Linux 2023 support - no more workarounds

What’s new: If you’ve been running PMM Client on AL2023 and dealing with complex manual installations, those days are over. You can now install PMM Client through native RPM packages just like any other supported platform.

What this means for you: Streamlined setup means you can get your Amazon Linux 2023 environments monitored faster.

2. Complete MySQL 8.4 replication monitoring

What’s new: PMM now fully supports replication monitoring for MySQL 8.4, including key metrics like IO Thread status, SQL Thread status, and Replication Lag. MySQL 8.4 changed how these metrics are exposed, and earlier PMM versions couldn’t track them accurately.

What this means for you: With the upgraded MySQL Exporter (v0.17.2), you now get complete replication monitoring across all supported MySQL versions (5.7, 8.0, and 8.4) without any visibility gaps.

3. MongoDB backup monitoring dashboard

What’s new: The new PBM Details dashboard lets you monitor MongoDB backups directly in PMM using the PBM collector. Instead of switching between PMM and separate backup tools, you now get a real-time, unified view of backup activity across replica sets and sharded clusters.

What this means for you: Easily track backup status, configuration, size, duration, PITR status, and recent successful backups—all in one place. No more tool-hopping to stay on top of your backup operations.

4. Grafana 11.6 upgrade with enhanced capabilities

What’s new: PMM now ships with Grafana 11.6, delivering enhanced visualization capabilities and improved alerting workflows.

Key features include:

Alert state history for reviewing historical changes in alert statuses
Improved panel features and visualization actions
Simplified alert creation with better UI workflows
Recording rules for creating pre-computed metrics
Navigation bookmarks for quick dashboard access

What this means for you: These enhancements make your monitoring dashboards more interactive, your alerting more sophisticated, and your overall monitoring workflow more efficient.

5. Dramatically improved Query Analytics performance

What’s new: We’ve optimized QAN filter loading performance to reduce the number of processed rows by up to 95% in large environments.

What this means for you: Filters on the PMM Query Analytics page now load much faster, making the interface more responsive to improve your troubleshooting efficiency.

Additional improvements worth noting

Beyond these five major enhancements, PMM 3.2.0 also introduces:

Secure ClickHouse connections with authenticated credential support
MongoDB Feature Compatibility Version (FCV) panels for better cluster version visibility
Nomad integration laying groundwork for future extensibility
Numerous bug fixes improving stability across ProxySQL, PostgreSQL, and MySQL monitoring

Getting started with PMM 3.2.0

Ready to experience these improvements? Set up your PMM 3.2.0 instance using our quickstart guide or upgrade your existing installation following our migration documentation.

For existing users with external PostgreSQL databases, make sure to review the external PostgreSQL configuration migration guide before upgrading.

Questions or feedback? We’d love to hear from you! Connect with the Percona community through our forums or join the conversation on our community channels.

PostgreSQL 18 - Top Enterprise Features (fast read)

Jan Wieremjewicz — Mon, 26 May 2025 00:00:00 UTC

So the Beta1 is available for PostgreSQL 18 and while not all the features have to make it to GA, we can surely hope they do!

Taking a close look at what’s coming, here below is the selection of what excites me in particular:

1. OAuth 2.0 authentication support

→ Finally aligns with modern enterprise SSO and identity standards (e.g., Okta, Azure AD). A major win for security teams and regulatory compliance.

2. Logical replication from standbys now with conflict logging

→ Now you can replicate from replicas not only primary nodes and thanks to conflict logging troubleshooting issues moves closer to what the users have been asking for. It’s a big step toward robust, native, HA-friendly logical replication. Not yet there, but on the right path!

3. Asynchronous I/O (AIO)

→ Modern async reads improve performance, especially under heavy parallel workloads. Foundation for future IO improvements, and also a feature that scratches an itch for a lot of cloud deployments.

4. Faster & safer major upgrades

→ pg_upgrade enhancements like parallel upgrade checks (--jobs), safer upgrades (--swap), and planner stats carried forward = faster version adoption and smoother upgrades for large clusters.

5. Observability++

→ Enhanced EXPLAIN statement and pg_stat_io improvements enhance understanding and optimize I/O behavior across tables, indexes, and WAL. This reduces the need for external monitoring tools.

Other Notable Features

While these are the top mentions, the goodies are not limited to only these. Some smaller improvements are just as exciting. Looking at these, the top interesting ones for me are:

New extension_control_path Server Variable

→ Enables operators to manage PostgreSQL extensions via Kubernetes image volumes (Kubernetes 1.33 image volumes) without modifying the base image.

→ Huge win for immutable image strategies and GitOps-friendly operator design.
Easier online constraints management

→ Add new NOT NULL constraints without locking large tables using NOT VALID

→ Use NOT ENFORCED foreign keys and CHECK to model relationships without runtime overhead
Smarter index maintenance (bottom-up deletion)

→ Reduces bloat, lowers vacuum overhead.
SQL/JSON path support + JSON performance gains

→ Enables document-style querying at scale.
Better insights into queries and vacuum

→ EXPLAIN now includes buffer usage in subplans, triggers, and functions which helps spot slow parts

→ pg_stat_all_tables now tracks how much time vacuum and autovacuum take per table
Indexing improvements

→ Parallel GIN builds help speed up full-text and vector searches key for hybrid full-text and vector search

→ B-tree skip scans make range and selective queries faster

What enterprises would want in PostgreSQL 19+

There is already a lot to like in this release, but based on what we hear from users, customers, and our own teams, here is what is still high on the list. First up and this one’s close to home is that we’d love to see the Transparent Data Encryption (TDE) patches from Percona Server for PostgreSQL make their way upstream. That would allow users to benefit from pg_tde directly in Community PostgreSQL Server. The rest of the list is a mix of long standing asks and forward looking ideas. It is a wishlist for sure, but one we hope to help make real over time:

Built-in Logical Conflict Resolution Algorithms

→ Support for conflict-handling strategies (e.g., last-write-wins, column-level rules) would simplify bidirectional replication and eliminate the need for custom frameworks, opening the door for fully open-source multi-master setups.
Logical failover orchestration

→ Seamless promotion and failover in logical topologies, with less reliance on external tooling. This would be great from the perspective of both Kubernetes deployments as well as the ease of use for HA solutions out there.
Better integration with external auth systems

→ Automatic PostgreSQL user creation based on OAUTH/LDAP roles at login, reducing operational burden for large-scale identity management and central access control.
Pluggable or columnar storage support

→ Native support or better extension hooks for OLAP and hybrid workloads, closing the gap with cloud-native alternatives like Citus or Redshift.
Sharding to provide horizontal scaling

→ Transparent sharding is a highly desired capability that becomes critical as workloads scale. While not always needed on day one, having built-in sharding means teams can grow without reinventing the wheel. Lack of it makes horizontal scaling complex, requiring more expertise and introducing higher operational overhead for DBA teams.

OS Platform End of Life (EOL) Announcement for Ubuntu 20.04 LTS

Julia Vural — Thu, 22 May 2025 00:00:00 UTC

Ubuntu 20.04 LTS (Focal Fossa) is scheduled to reach its official end of life on May 31, 2025. In alignment with the upstream vendor’s lifecycle, we are also ending platform support for Ubuntu 20.04 for all our MySQL related product offerings. This date and others are published in advance on our Percona Release Lifecycle Overview page.

As part of our support policy, Percona will continue to provide advisory support for databases running on EOL platforms, but effective April 1, 2025, we have discontinued:

Delivery of new packages or binary builds
Distribution of hotfixes or bug fixes for Percona software on Ubuntu 20.04
OS-level support for issues not related to the database itself

However, all existing packages will remain available for download.

We are committed to ensuring a smooth and seamless experience for our users. Migrating to a supported operating system will ensure that you continue to receive security updates, bug fixes, and new features for our products.

We encourage you to take action promptly and plan your migration from Ubuntu Focal to a supported operating system. Each operating system vendor has different supported migration or upgrade paths to their next major release. Please contact us if you need assistance migrating your database to a different supported OS platform – we will be happy to assist you!

Note: If requested by a customer, we may provide updated Percona packages for up to six months beyond the Percona EOL date, as a courtesy grace period.

Percona Bug Report: April 2025

Aaditya Dubey — Mon, 19 May 2025 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs.

Percona Server/MySQL Bugs

PS-8846: The ALTER INSTANCE RELOAD TLS thread gets stuck. This happens in instances with a high new connections rate (>60/s) but not in all instances. Percona and Upstream did not fix it properly because the current implementation of ALTER INSTANCE RELOAD TLS requires all existing SSL connections to be closed. “A thread that executes ALTER INSTANCE RELOAD TLS tries to acquire an RCU Lock(Read-Copy-Update), waiting for the number of readers to become 0. In other words, when the server has a constant flow of new incoming SSL connections, the chances of acquiring this lock are pretty low.” Therefore, Percona and Oracle only partially fixed this; this fix should improve this behaviour.

Reported Affected Version/s: 8.0.32-24, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PS-9609: The audit_log_filter can’t be installed when the server is using component_keyring_kmip

Reported Affected Version/s: 8.0.39-30

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42-33

PS-9628: The binlog_encryption does not work with component_keyring_kmip

Reported Affected Version/s: 8.0.40-31

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42-33

PS-9664: With a very simple workload, MyRocks allocates a lot of memory and does not free it when the workload finishes. All instrumentation available either does not provide information about memory allocated or provides only part of it. As a result, users cannot predict how much RAM to install on the server that runs the MyRocks storage engine. With InnoDB same workload requires about 1.7G and frees about 0.5G once the job is finished.

Reported Affected Version/s: 8.0.39-30

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PS-9719: When changing binlog_transaction_dependency_tracking in high load workload, MySQL got a segmentation fault.

Reported Affected Version/s: 8.0.40, 8.0.41-32, 8.0.33

Upstream Bug: 117922

Workaround/Fix: “set global binlog_transaction_dependency_tracking = commit_order;”

Fixed/Planned Version/s: 8.0.42-33, 8.4.5-5

PS-9768: An unexpected duplicate error occurs when running a select query with a group by JSON data.

Reported Affected Version/s: 8.0.41-32, 8.4.4-4

Upstream Bug: 117927

Workaround/Fix: Rebuilding the table with “alter table db_name.table_name = rocksdb;” can fix the issue.

Fixed/Planned Version/s: To Be Determined

Percona Xtradb Cluster

PXC-4512: When DDLs run against tables with foreign key references when there is a write load simultaneously. The issue is typically triggered during pt-online-schema-change execution, and after a dozen or so iterations, random PXC nodes will terminate with MDL BF-BF conflict. Sometimes, the writer fails, and sometimes, the other nodes, but it can be reproducible with just the RENAME query.

Reported Affected Version/s: 8.0.33-25, 8.0.35-27, 8.0.36-28, 8.0.37-29, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Only a full write stop would help.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4648: After upgrading from 8.0.41 to 8.4.3, the node can’t join the group with the following error.

[ERROR] [MY-000000] [Galera] /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.4.3/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():343: Group requested gcs_proto_ver: 5, max supported by this node: 4.Upgrade the node before joining this group.Need to abort.

The problem is that 8.0.41 and 8.4.4 use Galera 26.4.21. It introduced the GCS protocol version 5, 8.0.40 and 8.4.3 using Galera 26.4.20. So, the node that doesn’t understand protocol 5 tries to connect to a cluster that uses protocol 5.

It is fixed as a documented bug, which can be seen here.

Reported Affected Version/s: 8.4.3

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.4.4

PXC-4638: The binlog_utils_udf plugin fails to access binlog files correctly after an SST (State Snapshot Transfer) due to inconsistencies in the mysql-bin.index file. After an SST, the first entry in the mysql-bin.index file can be incorrectly formatted with a relative path, while subsequent entries use absolute paths. This inconsistency can prevent the binlog_utils_udf plugin from locating the correct binlog files.

The resulting mysql-bin.index content after SST might appear as:

mysql-bin.000010
/home/user/sandboxes/pxc_msb_8_0_40/node2/data/mysql-bin.000011
/home/user/sandboxes/pxc_msb_8_0_40/node2/data/mysql-bin.000012 

Reported Affected Version/s: 8.0.40, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Rewriting binlog.index and adding the “./” prefix to the first entry, and running flush binary logs resolved this issue.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-3576: Deploying a new installation using the setting lower_case_table_names=1 on the startup generates the following entry on the log:

[Warning] [MY-010324] [Server] 'db' entry 'percona_schema mysql.pxc.sst.role@localhost' had database in mixed case that has been forced to lowercase because lower_case_table_names is set. It will not be possible to remove this privilege using REVOKE.

Looking at the mysql.db, we can see the deployment was able to create the mysql.pxc.sst.role mapping an uppercase database name:

mysql> select * from mysql.db where user='mysql.pxc.sst.role';
+-----------+--------------------+--------------------+------+
| Host | Db | User | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Grant_priv | References_priv | Index_priv | Alter_priv | Create_tmp_table_priv | Lock_tables_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Execute_priv | Event_priv | Trigger_priv |
+-----------+--------------------+--------------------+-------------+-------------+-------------+-------------+-------------+-----------+------------+-------------+
| localhost | PERCONA_SCHEMA | mysql.pxc.sst.role | N | N | N | N | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N |
| localhost | percona_schema | mysql.pxc.sst.role | N | N | N | N | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N |

This produces a duplicate entry for the percona_schema database and a warning message on the mysql.log.

Reported Affected Version/s: 8.0.21-12.1, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4657: When executing DML and DDL on the same table, the DML will get a deadlock error. If the DML does not change the data but matches, it won’t be replicated, for example.

mysql > UPDATE test.t SET d = d LIMIT 1;
Query OK, 0 rows affected (0.01 sec)
Rows matched: 1 Changed: 0 Warnings: 0

If the table contains a trigger, the DML does not get a deadlock error and will be replicated to other nodes.

CREATE TRIGGER `t_on_update`
AFTER UPDATE ON `t`
FOR EACH ROW
BEGIN
 INSERT INTO t_history (`d`)
 VALUES (NEW.`d`);
END

When other nodes apply the DML and DDL, the applier threads will get an MDL BF-BF conflict.

Reported Affected Version/s: 8.0.40, 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Using a single applier thread will skip the bug, but it may introduce a performance issue, such as flow control.

Fixed/Planned Version/s: 8.0.42, 8.4.5

PXC-4664: Converting thd->rli_slave to target type Slave_worker* causes a segmentation fault.

Reported Affected Version/s: 8.0.41

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

Percona Toolkit

PT-2392: pt-online-schema-change resume functionality doesn’t work with ADD INDEX

Reported Affected Version/s: 3.6.0, 3.7.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2322: The issue reports that pt-mysql-summary does not correctly detect and display the jemalloc memory management library, even when it is enabled. Despite jemalloc being loaded and visible in the process memory map (/proc//maps), the output from pt-mysql-summary is missing this information in some cases, unlike version 3.2.1, which correctly identifies and reports it.

Reported Affected Version/s: 3.5.6, 3.5.7

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2422: When using the –history option with pt-online-schema-change (pt-osc), the query responsible for updating the history entry with the new_table_name is not appropriately constrained by a primary or unique key. As a result, this UPDATE operation can inadvertently modify all entries in the history table, rather than just the intended row.

This behavior can lead to significant issues when running multiple schema change operations in parallel, as the history entries for different migrations may interfere with each other, causing data consistency problems.

Additionally, if a migration is paused and later resumed, this lack of key constraint can result in only a subset of the data being correctly copied to the new table, potentially leading to partial data loss or corruption when the final table swap occurs.

Reported Affected Version/s: 3.5.6, 3.5.7

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.1

PT-2442: Multiple security vulnerabilities have been identified in the latest version of Percona Toolkit, including:

CVE-2024-56171: Use-After-Free Vulnerability in libxml2

CVE-2024-12797: OpenSSL Raw Public Key Authentication Vulnerability

CVE-2022-37967: Windows Kerberos Elevation of Privilege Vulnerability

CVE-2025-24928: Stack-Based Buffer Overflow in libxml2

It is recommended that the associated advisories be reviewed and the necessary patches or upgrades be applied to mitigate the risk.

Reported Affected Version/s: 3.7.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.7.0-1

PMM (Percona Monitoring and Management)

PMM-13694: When using a non-default pg_stat_statements.max value, the calculated QPS displayed in QAN may be wrong.

Reported Affected Version/s: 2.38.0, 2.44.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

PMM-13807: pmm-agent crashed at query.Fingerprint due to query including a column named “value”

Reported Affected Version/s: 2.44.0, 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

PMM-13847: PMM 3.0 doesn’t support running on a different uid/gid in Kubernetes

Reported Affected Version/s: 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PMM-13984: Percona Monitoring and Management (PMM) version 3.1 with OVA image currently cannot be imported into VMware environments.

Reported Affected Version/s: 3.1.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PMM-12784: This error invalid GetActionRequest.ActionId: value length must be at least 1 runes. occurs in the QAN (Query Analytics) dashboard, indicating that the ActionId field in the GetActionRequest is empty or improperly formatted. This typically results from a missing or incorrectly populated Action ID parameter, which is required for retrieving query data.

Reported Affected Version/s: 2.33.0, 2.41.0, 2.44.0, 3.0.0

Upstream Bug: Not Available

Workaround/Fix: Not Available

Fixed/Planned Version/s: 3.2.0

Percona XtraBackup

PXB-3421: XtraBackup fails when the –databases parameter contains a very long list of databases or has a large amount of whitespace before the actual database names. When the –databases parameter is provided with 1859 whitespace characters before a table name (e.g., db01.t1), XtraBackup crashes with a signal error. If the number of whitespace characters is reduced to 1858 or fewer, the backup proceeds successfully without error.

Reported Affected Version/s: 8.0.35-31

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

PXB-3426: Using KMIP component causes double free of memory on error paths.

Reported Affected Version/s: 8.0.35-31

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 8.0.35-33, 8.4.0-3

PXB-3392: xtrabackup doesn’t pick up –innodb-log-group-home-dir config parameter

Reported Affected Version/s: 8.0.35-30, 8.0.35-31, 8.0.35-32

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

Percona Kubernetes Operator

K8SPG-703: When using ttlSecondsAfterFinished, there is a potential race condition where the backup jobs may be deleted before the Percona operator has had sufficient time to reconcile the perconapgbackups objects. This issue can occur even with relatively long timeouts like 1m, 5m, or 30m, not just extremely short intervals.

Reported Affected Version/s: 2.5.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.7.0

K8SPSMDB-1263: While creating a 1.3TB logical backup, the replica’s state changes to “errored” after approximately 16 hours with the message:

“failed to find CERTIFICATE”

despite the backup continuing to run and eventually completing. This raises concerns about the validity of the backup and the ability to restore from it without manually altering its status to “success.”

Reported Affected Version/s: 1.13.0, 1.14.0, 1.15.0, 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.20.0

K8SPSMDB-1292: When spec.tls.mode is set to requireTLS, physical backup restores fail with a “server selection timeout” error. This occurs because the operator cannot establish a secure connection to the MongoDB server, resulting in closed socket errors and the inability to disable Point-in-Time Recovery (PiTR).

Reported Affected Version/s: 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.21.0

K8SPSMDB-1294: When using MCS on GKE 1.30, an API version mismatch occurs, resulting in the error:

“no matches for kind ‘ServiceImport’ in version ’net.gke.io/v1alpha1’”

This indicates that the expected ServiceImport kind is not available in the specified API version, preventing proper service discovery.

Reported Affected Version/s: 1.19.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.20.0

K8SPSMDB-1336: Restoring a backup into a new Kubernetes cluster can lead to “Time monotonicity violation” errors on config servers and mongos, causing the pods to restart. This occurs when the restored chunk version timestamps are earlier than the expected timestamps in the new cluster, resulting in tripwire assertions and persistent crashes.

Reported Affected Version/s: 1.19.1

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: To Be Determined

K8SPXC-1548: Failed to delete old backups on Google Cloud Storage

Reported Affected Version/s: 1.14.0, 1.15.1

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 1.18.0

PBM (Percona Backup for MongoDB)

PBM-1482: Selective Restore with replset-remapping hangs on oplog replay and doesn’t finish

Reported Affected Version/s: 2.5.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

PBM-1487: Error Location6493100 on mongos after successful logical restore or PITR

Reported Affected Version/s: 2.8.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

PBM-1531: PBM Restore getting randomly stuck

Reported Affected Version/s: 2.6.0, 2.7.0, 2.8.0, 2.9.0

Upstream Bug: Not Applicable

Workaround/Fix: Not Available

Fixed/Planned Version/s: 2.10.0

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Progress report on pg_tde - GA extension is nearer every day!

Jan Wieremjewicz — Thu, 08 May 2025 00:00:00 UTC

Another week, another blogpost about the state of open source Transparent Data Encryption (TDE) for PostgreSQL.

First off, thank you for all the feedback shared so far!

Whether it’s reports about deployment issues with pg_tde, integration with KMS, missing features or gaps in our documentation, we truly appreciate it! Your input helps us build a better, more complete solution and to properly prioritize what’s next.

What’s the word?

We know many of you are eagerly waiting for a production-ready release of pg_tde, and today we’ve got some good news! You may remember the last release was a Release Candidate (RC), now we’re gearing up to launch RC2.

If you’re not familiar with the terminology, this simply means we’re one step closer to General Availability (GA).

Our nightly builds are always there if you want to keep up with the latest updates, but this new milestone release highlights the fixes to some pain points that our tests and your feedback helped uncover.

When to expect RC2?

This post is just a heads-up. The actual release is planned for early next week (after May 12, 2025). Keep an eye out.

In the meantime, feel free to check what’s coming in RC2. And if you’d like to benefit from these improvements right away, nightly builds are the way to go.

What’s new in RC2 compared to RC1?

Some recent documentation updates are already live, and we’d love to hear your thoughts on them. Don’t be a stranger, let us know how do you find them!

As for the code, here are the key changes:

KMS configuration improvements including:
- New parameter for passing a client certificate when configuring a KMIP provider
- Compatibility updates for key management systems (KMS)
  - Thales CypherTrust
  - Fortanix Data Security Manager
- Validation enforcement when adding key provider configurations
WAL improvements, hardening encryption in our beta WAL support
Security enhancements for multi-tenancy scenarios
Other updates like:
- Added pg_tde_verify_default_key() and pg_tde_default_key_info() functions
- Fixed support for logical replication

How to use nightly builds

In case you’re wondering what nightly builds are: think of them as automatically generated versions of the software with the latest changes, usually built overnight 😎. They’re useful for testing (especially for integration testing) and for those, like our developers, who want to work with the freshest code.

No elephants have been hurt to create our nightly builds! We rely solely on CI/CD automation!

You can find them in our experimental repo. Do note, they’re currently only available for x86_64 and a limited set of operating systems:

Setting Up and Monitoring MongoDB 8 Replica Sets with PMM 3 Using Docker: A Beginner-Friendly Guide

Daniil Bazhenov — Tue, 18 Mar 2025 00:00:00 UTC

This guide explains how to set up a MongoDB 8 Replica Set and monitor it using PMM 3, all within Docker. We’ll guide you through the steps to create a local environment, configure the necessary components, and connect them for effective monitoring and management.

The guide is written in detail for beginners. In the conclusion section there are ready configurations for the experienced.

The recent release of Percona Monitoring and Management 3 introduces several new features:

Upgraded Grafana version for an improved user experience.
Rootless containers for enhanced security.
ARM support for the pmm-client.
Monitoring capabilities for MongoDB 8, along with new dashboards.

This article is intended for developers and DBAs who want to experiment with these tools locally using Docker. We will cover the following steps to set everything up and test the functionality:

Launch the PMM 3 pmm-server for monitoring and open it in a browser.
Install MongoDB, starting with a standalone server, and then convert it into a Replica Set with three nodes. Percona Server for MongoDB images are used in this article.
Configure the pmm-client for MongoDB to send metrics to the pmm-server.
Explore the PMM 3 dashboards.

We will use Docker Compose to define and run multiple containers efficiently through a single docker-compose.yaml file.

If you’re ready to dive into the world of Dockerized database monitoring and management, let’s get started!

Step Zero: Preparation

To get started, you need a terminal to run Docker commands and a text editor to modify the docker-compose.yaml file.

You also need Docker installed on your system. If Docker is not installed, follow these instructions to set it up:

Docker Desktop: This application includes both Docker and Docker Compose. It is available for multiple operating systems. This guide uses Docker Desktop on macOS with an Apple Silicon ARM processor. Download Docker Desktop
Docker and Docker Compose (separately): If preferred, install Docker and Docker Compose individually. Use the following links for guidance:
- Download Docker for your OS
- Download Docker Compose

Command to Verify Installation:

Verify Docker Compose Installation: If you’re using Docker Desktop, Docker Compose is included. If you installed Docker Compose separately, you can verify the installation with:

docker-compose --version

This should return the version of Docker Compose installed.

sh
➜ community git:(main) ✗ docker-compose --version
Docker Compose version v2.21.0-desktop.1

Once Docker and Docker Compose are installed and verified, you are ready to move on to the next steps of deploying PMM 3 and MongoDB in Docker.

Project Directory

Create a directory where we’ll store the configuration and necessary files. For example, I created a directory named pmm-mongodb-setup and navigated into it.

bash
mkdir pmm-mongodb-setup
cd pmm-mongodb-setup

Step One: Starting PMM 3 Using Docker Compose

To start with, we will launch PMM 3, specifically the pmm-server, which we will later access via a browser.

Create the Docker Compose Configuration File

First, create a file named docker-compose.yaml in your project directory. Then, copy and paste the following configuration into the file to set up PMM 3:

yaml
version: '3'

services:
 pmm-server:
 image: percona/pmm-server:3
 platform: "linux/amd64" # Specifies that Docker should use the image for the amd64 architecture, which is necessary if the container doesn't support ARM and your host system is ARM (e.g., Mac with Apple Silicon).
 container_name: pmm-server
 ports:
 - 8080:80
 - 443:8443
 healthcheck: # Defines a command to check the container's health and sets the timing for executions and retries.
 test: ["CMD-SHELL", "curl -k -f -L https://pmm-server:8443 > /dev/null 2>&1 || exit 1"]
 interval: 30s
 timeout: 10s
 retries: 5

Explanation:

We define the first pmm-server service to use the percona/pmm-server:3 image

platform: This parameter ensures compatibility with ARM-based processors, such as my Mac with Apple Silicon, by instructing Docker to use the image for the amd64 architecture.

healthcheck: This parameter performs a check to confirm that the container has started successfully.

Start the PMM 3 Container

Save the docker-compose.yaml file, and then use the following command to start the PMM 3 container:

docker-compose up -d

This command will download the PMM 3 image if it is not already available locally and start the container in detached mode.

Expected result in the terminal:

shell
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 2/2
 ✔ Network pmm-mongodb-setup_default Created 0.1s
 ✔ Container pmm-server Started 0.9s
➜ pmm-mongodb-setup

Expected result in Docker Desktop:

Open PMM in a browser

Now you can open PMM in your browser at http://localhost. Use admin/admin as the username and password to log in. When prompted to change the password, skip this step by clicking the Skip button. Since this is a test setup, we need a simple password for other containers.

Now, PMM 3 is successfully running using Docker Compose.

Step Two: Starting MongoDB

We start by launching a standalone MongoDB service. This simple configuration allows us to quickly understand how it operates before moving on to a more advanced setup with a Replica Set.

To keep the database data persistent, even after a restart, a volume is used for MongoDB data storage. Add the following configuration to the docker-compose.yaml file under the pmm-server service:

yaml
 mongodb:
 image: "percona/percona-server-mongodb:8.0-multi"
 volumes:
 - mongodb-data:/data/db
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 ports:
 - "27017:27017"
 command: ["mongod", "--port", "27017", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 healthcheck:
 test: ["CMD-SHELL", "mongosh --eval 'db.adminCommand(\"ping\")' --quiet"]
 interval: 30s
 timeout: 10s
 retries: 5

volumes:
 mongodb-data: # MongoDB data storage volume

Explanation:

environment: Configures the root user’s credentials (username and password).

command: Sets additional parameters for MongoDB:

bind_ip_all: Allows external access to the database, for instance, through MongoDB Compass.

profile: Enables profiling settings to support Query Analytics (QAN) in PMM.

healthcheck: Ensures the container starts successfully by executing a health check command.

volumes:: Creates the specified volume for MongoDB data storage. Defines a Docker volume to store MongoDB data persistently.

Launching the Configuration

Save the updated docker-compose.yaml file and launch the MongoDB service by running the following command:

docker-compose up -d

Docker Compose checks the configuration and starts the MongoDB container.

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 3/3
 ✔ Volume "pmm-mongodb-setup_mongodb-data" Created 0.0s
 ✔ Container pmm-mongodb-setup-mongodb-1 Started 0.1s
 ✔ Container pmm-server Running 0.0s
➜ pmm-mongodb-setup

Verifying MongoDB in Docker Desktop:

Step Three: PMM Client

At this point, we have both the PMM Server, which is accessible in the browser, and MongoDB running. To transfer metrics from MongoDB to the PMM Server, a container with pmm-client needs to be started.

Add another service pmm-client to the docker-compose.yaml file, right after mongodb. Note that `volumes:`` must remain at the bottom of the file.

yaml
 pmm-client:
 image: percona/pmm-client:3
 container_name: pmm-client
 depends_on:
 pmm-server:
 condition: service_healthy
 mongodb:
 condition: service_healthy
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: admin
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 PMM_AGENT_SETUP: 1
 PMM_AGENT_SETUP_FORCE: 1
 PMM_AGENT_PRERUN_SCRIPT: >
 pmm-admin status --wait=10s &&
 pmm-admin add mongodb --username=databaseAdmin --password=password --host=mongodb --port=27017 --query-source=profiler

Explanation:

depends_on: Ensures that pmm-client starts only after pmm-server and mongodb have started successfully and passed the healthcheck.

PMM_AGENT_PRERUN_SCRIPT: The pmm-admin add mongodb command adds the MongoDB service to PMM for monitoring.

PMM_AGENT_SERVER_ADDRESS: Specifies the PMM Server address and uses port 8443.

PMM_AGENT_SERVER_USERNAME and PMM_AGENT_SERVER_PASSWORD: Update these values if you have changed the PMM login credentials.

Applying the Updated Configuration

Run the following command to apply the updated docker-compose.yaml configuration and start the pmm-client service:

docker-compose up -d

After running the command, you should see the pmm-client container start successfully:

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 3/3
 ✔ Container pmm-mongodb-setup-mongodb-1 Healthy 0.0s
 ✔ Container pmm-server Healthy 0.0s
 ✔ Container pmm-client Started 0.1s
➜ pmm-mongodb-setup

Open PMM in your browser. On the homepage, you should now see MongoDB listed as a monitored service:

Step Four: Convert a Standalone MongoDB to a Replica Set

A single MongoDB instance works well for development and testing. At this point, you can already connect to the database from your application or tools such as MongoDB Compass and run various NoSQL queries.

However, the goal is to deploy a Replica Set consisting of three replicas, which is recommended for production and operational setups. For now, we set up the Replica Set on a single machine with a single docker-compose, which is intended for testing and development purposes only.

Both the mongodb and pmm-client services need to be updated.

Stopping All Services

First, stop all the currently running services:

docker-compose down

Result:

sh
➜ pmm-mongodb-setup docker-compose down
[+] Running 4/4
 ✔ Container pmm-client Removed 0.3s
 ✔ Container pmm-mongodb-setup-mongodb-1 Removed 0.4s
 ✔ Container pmm-server Removed 4.5s
 ✔ Network pmm-mongodb-setup_default Removed

Generating a Key File

To run three MongoDB replicas that can securely communicate with each other, a key file is required.

Create a secrets folder next to the docker-compose.yaml file and generate the mongodb-keyfile:

sh
mkdir secrets
openssl rand -base64 128 > secrets/mongodb-keyfile
chmod 600 secrets/mongodb-keyfile

In this case, the mongodb-keyfile will be generated inside the secrets folder. If your operating system does not support these commands, manually create the secrets folder and add a mongodb-keyfile with the following content:

rVLhIK2PhZKGxysjwMR4t1OmNppqdAzEs408hrbzg95D146mn9YENixId6pvIGCA
Cy9hc1k6OKKabbv7Rm347NwSFxbdPPx0/jnaO80U/a6/mv0XqSmEl8wdR91b4jIm
d98LobplwRs4b7g9cnLMUAIULr0WG+J36NtKIA6q4eE=

Add three mongodb services for Replica Set

Remove the existing mongodb service from the docker-compose.yaml file and replace it with three Replica Set services:

yaml
 mongodb-rs101:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs101
 ports:
 - "27017:27017"
 command: ["mongod", "--port", "27017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-101:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

 mongodb-rs102:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs102
 ports:
 - "28017:28017"
 command: ["mongod", "--port", "28017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-102:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 28017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

 mongodb-rs103:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: mongodb-rs103
 ports:
 - "29017:29017"
 command: ["mongod", "--port", "29017", "--replSet", "rs", "--keyFile", "/etc/secrets/mongodb-keyfile", "--bind_ip_all", "--profile", "2", "--slowms", "200", "--rateLimit", "100"]
 environment:
 MONGO_INITDB_ROOT_USERNAME: databaseAdmin
 MONGO_INITDB_ROOT_PASSWORD: password
 volumes:
 - mongodb-data-103:/data/db
 - ./secrets:/etc/secrets:ro
 healthcheck:
 test: ["CMD-SHELL", "mongosh --host localhost --port 29017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'rs.status().ok || 1'"]
 interval: 30s
 timeout: 10s
 retries: 5

Key Points:

./secrets:/etc/secrets:ro: Mounts the key file from your disk into the container.

ports:: Each replica runs on a separate port since they are hosted on the same machine.

command::

--keyFile: Enables replication and uses the key file for secure communication between replicas.

--replSet: Defines a Replica set parameter named rs.

healthcheck:: Ensures that pmm-client starts only after the Replica Set is initialized.

Adding Volumes

Define three separate volumes for data storage:

yaml
volumes:
 mongodb-data-101:
 mongodb-data-102:
 mongodb-data-103:

Initializing the Replica Set

Add another service to initialize the Replica Set (after mongodb-rs103):

yaml
 mongodb-rs-init:
 image: percona/percona-server-mongodb:8.0-multi
 container_name: rs-init
 depends_on:
 - mongodb-rs101
 - mongodb-rs102
 - mongodb-rs103
 entrypoint: [
 "sh", "-c",
 "until mongosh --host mongodb-rs101 --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'print(\"waited for connection\")'; do sleep 5; done && \
 mongosh --host mongodb-rs101 --port 27017 --username databaseAdmin --password password --authenticationDatabase admin --eval 'config={\"_id\":\"rs\",\"members\":[{\"_id\":0,\"host\":\"mongodb-rs101:27017\"},{\"_id\":1,\"host\":\"mongodb-rs102:28017\"},{\"_id\":2,\"host\":\"mongodb-rs103:29017\"}],\"settings\":{\"keyFile\":\"/etc/secrets/mongodb-keyfile\"}};rs.initiate(config);'"
 ]
 volumes:
 - ./secrets:/etc/secrets:ro

This service connects to one of the replicas and initializes the Replica Set configuration.

Updating pmm-client

Finally, modify the pmm-client service to register all three Replica Set nodes:

yaml
 pmm-client:
 image: percona/pmm-client:3
 container_name: pmm-client
 depends_on:
 pmm-server:
 condition: service_healthy
 mongodb-rs101:
 condition: service_healthy
 mongodb-rs102:
 condition: service_healthy
 mongodb-rs103:
 condition: service_healthy
 environment:
 PMM_AGENT_SERVER_ADDRESS: pmm-server:8443
 PMM_AGENT_SERVER_USERNAME: admin
 PMM_AGENT_SERVER_PASSWORD: admin
 PMM_AGENT_SERVER_INSECURE_TLS: 1
 PMM_AGENT_CONFIG_FILE: config/pmm-agent.yaml
 PMM_AGENT_SETUP: 1
 PMM_AGENT_SETUP_FORCE: 1
 PMM_AGENT_PRERUN_SCRIPT: >
 pmm-admin status --wait=10s &&
 pmm-admin add mongodb --service-name=mongodb-rs101 --username=databaseAdmin --password=password --host=mongodb-rs101 --port=27017 --query-source=profiler &&
 pmm-admin add mongodb --service-name=mongodb-rs102 --username=databaseAdmin --password=password --host=mongodb-rs102 --port=28017 --query-source=profiler &&
 pmm-admin add mongodb --service-name=mongodb-rs103 --username=databaseAdmin --password=password --host=mongodb-rs103 --port=29017 --query-source=profiler

Explanation:

depends_on: Ensures pmm-client waits until all replicas and pmm-server are initialized.

PMM_AGENT_PRERUN_SCRIPT: Adds all three replicas to PMM monitoring with the pmm-admin add mongodb command.

Launching the Configuration

Start all services with:

docker-compose up -d

Expected Output:

sh
➜ pmm-mongodb-setup docker-compose up -d
[+] Running 10/10
 ✔ Network pmm-mongodb-setup_default Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-101" Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-102" Created 0.0s
 ✔ Volume "pmm-mongodb-setup_mongodb-data-103" Created 0.0s
 ✔ Container mongodb-rs103 Healthy 0.1s
 ✔ Container pmm-server Healthy 1.1s
 ✔ Container mongodb-rs101 Healthy 0.1s
 ✔ Container mongodb-rs102 Healthy 0.1s
 ✔ Container rs-init Started 0.1s
 ✔ Container pmm-client Started 0.0s
➜ pmm-mongodb-setup

Expected Output in Docker Desktop:

Verifying in PMM

Open PMM and explore dashboards such as the MongoDB Replica Set Summary, which displays information about your replicas and various metrics:

For example, I experimented by restarting one of the MongoDB services in Docker Desktop. The Replica Set switched the Primary replica, and when I simulated a failure by stopping a service, the monitoring dashboard reflected this event:

Connecting to MongoDB and Useful Commands

Connecting to MongoDB

There are several ways to connect to MongoDB depending on your setup and tools:

Using Docker Desktop

If you are using Docker Desktop, you can select a container and open the Exec tab. This opens a terminal within the container.

From there, you can connect to MongoDB using the mongosh shell with the following command:
sh
```
mongosh --host localhost --port 27017 -u databaseAdmin -p password --authenticationDatabase admin
```
Using Docker CLI

If you’re running Docker without Docker Desktop, you can connect to the container using the following command:
bash
```
docker exec -it  bash
```
when you connect to the container, connect to MongoDB
sh
```
mongosh --host localhost --port 27017 -u databaseAdmin -p password --authenticationDatabase admin
```
Connecting from Applications or Tools

If you are connecting through an application or a tool like MongoDB Compass, you can use connection strings tailored to your setup:

Here are the MongoDB connection strings tailored for your use cases:
1. Standalone MongoDB: Connects directly to a single MongoDB instance.
  sh
  mongodb://databaseAdmin:password@localhost:27017
2. Primary Node: Forces a direct connection to the primary node in the Replica Set using the directConnection=true option.
  sh
  mongodb://databaseAdmin:password@localhost:27017/?directConnection=true
3. Full Replica Set: Lists all Replica Set members and enables automatic failover using replicaSet=rs
  sh
  mongodb://databaseAdmin:password@localhost:27017,localhost:28017,localhost:29017/?replicaSet=rs

Useful Commands

Here are some helpful commands for managing and troubleshooting your MongoDB setup:

Check the status of the Replica Set. After connecting to MongoDB, use the following command to retrieve the status of the Replica Set:
sh
```
rs.status()
```
Check MongoDB runtime configuration. Use this command to view the configuration options for the running MongoDB instance:
sh
```
db.adminCommand({ getCmdLineOpts: 1 })
```

Conclusion

In this article, we explored deploying MongoDB using Docker and Docker Compose. We covered both a Single Instance MongoDB for simple setups and a MongoDB Replica Set for high availability, while integrating them with Percona Monitoring and Management (PMM) for monitoring.

Here are the final configurations you can download:

Single Instance MongoDB + PMM 3: Link
Replica Set MongoDB + PMM 3: Link

Thank you for reading! I look forward to your comments and questions.

Join Us Online: Stream About Percona Toolkit for MySQL!

Sveta Smirnova — Fri, 14 Mar 2025 00:00:00 UTC

Are you passionate about databases and want to stay on top of the latest advancements in MySQL and Percona Toolkit? You’re in luck! We are excited to invite you to our upcoming online stream, where we’ll delve into some exciting changes and updates.

Date: March 27, 2025
Time: 13:30 GMT / 9:30 ET
Streaming Live on: LinkedIn and YouTube

About the Event

Our featured speaker, Sveta Smirnova, Principal Support Engineering Coordinator at Percona, will share her insights on the recent updates in MySQL, focusing on the removal of offensive replication statements like START/STOP SLAVE. As the maintainer of the Percona Toolkit, Sveta had to rewrite numerous tools and libraries to accommodate these changes, resulting in significant updates to 511 files.

This event is perfect for developers, DBAs, and anyone interested in databases. Sveta, an expert in MySQL and Percona Toolkit, is also an accomplished author and a speaker at international conferences on development and databases. During the stream, she will discuss the challenges she faced while renewing legacy code, including supporting SSL and handling the deprecation of certain tools.

We invite you to join the discussion and share your experiences, questions, and insights. Engage directly with Sveta and other participants to gain a deeper understanding of the Percona Toolkit and its latest enhancements.

Discussion Topics

Introduction to Percona Toolkit:
- History and evolution of Percona Toolkit.
- Overview of the most popular tools and their uses.
Changes in MySQL:
- Overview of new and legacy syntax.
- Implications of dropped offensive replication statements.
Adapting Percona Toolkit:
- Challenges in renewing legacy code.
- Solutions implemented, including SSL support and deprecation of certain tools.
Practical Insights:
- Detailed explanation of the changes made to 511 files.
- Fine-tuning and migration strategies for Percona Toolkit users.

Why Attend?

Insightful Content: Gain valuable knowledge on the latest changes in MySQL and how they impact Percona Toolkit.
Expert Guidance: Learn directly from Sveta Smirnova, an industry expert with extensive experience in database management.
Interactive Session: Have the opportunity to ask questions live and engage directly with the speaker.

How to Join

Tune in on March 27, 2025, at 13:30 GMT / 9:30 ET, and watch the live stream on LinkedIn and YouTube. Don’t miss this chance to enhance your understanding of MySQL and Percona Toolkit while gaining practical insights from one of the best in the field.

Mark your calendars, spread the word, and get ready for an informative session! We look forward to seeing you there!

If you have any questions or need further information, feel free to reach out to us.

📅 Add to Calendar
🔗 Join the LinkedIn Stream
🔗 Join the YouTube Stream

We can’t wait to see you at the event!

Percona Monitoring and Management 3 and rootless containers

Sergey Pronin — Wed, 19 Feb 2025 00:00:00 UTC

In today’s landscape, where security breaches are a constant concern, reducing potential attack vectors is a top priority for any organization. Percona Monitoring and Management (PMM) has established itself as a reliable solution for database performance monitoring. With the release of PMM version 3, Percona has significantly strengthened its security posture, notably by introducing support for rootless container deployments. This advancement directly addresses a crucial security challenge and enhances the overall robustness and reliability of PMM.

The inherent risks associated with root privileges are well-documented. While many applications, including those containerized, have historically relied on root access, this practice presents a substantial security vulnerability. In the event of a successful exploit, an attacker gains comprehensive control over the host system. This risk is further exacerbated in environments with outdated software or complex configurations. Essentially, while the root user offers extensive capabilities, it also represents a significant potential liability that should be carefully mitigated.

In this blog post we will look at the exact differences between PMM versions 2 and 3 and how they behave.

Enforcing Pod Security Standards

The Pod Security Standards define three different policies to broadly cover the security spectrum. These policies are cumulative and range from highly-permissive to highly-restrictive.

To enforce restrictions on a specific namespace we will create the following YAML manifest:

apiVersion: v1
kind: Namespace
metadata:
 name: secure-namespace
 labels:
 pod-security.kubernetes.io/enforce: restricted
 pod-security.kubernetes.io/warn: restricted
 pod-security.kubernetes.io/audit: restricted

This YAML creates a namespace named secure-namespace. The pod-security.kubernetes.io/enforce: restricted label instructs Kubernetes to deny any pods in this namespace that violate the “restricted” PSS profile. The warn and audit labels are also very useful for monitoring and testing before fully enforcing the restricted policy.

Deploy PMM

We will execute a series of deployments to demonstrate the difference between PMM2 and PMM3 behavior in insecure and secure environments. All files can be found in this github repository: spron-in/blog-data/pmm3-rootless

Regular namespace

PMM2

% kubectl apply -f 01.pmm2.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 45s

I can connect to my PMM2 server with a Service that I created.

PMM3

% kubectl apply -f 02.pmm3.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 20s

I can connect to my PMM3 server with a Service that I created.

Secure namespace

Let’s try to deploy both versions of Percona Monitoring and Management servers in a secure namespace. For both we will see the following:

% kubectl apply -f 01.pmm2.yaml -n secure-namespace
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "pmm-server" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "pmm-server" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "pmm-server" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "pmm-server" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
No resources found in secure-namespace namespace.

There are no Pods created and if you describe the StatefulSet, you are going to see a similar error:

Warning FailedCreate 5s (x15 over 87s) statefulset-controller create Pod pmm-server-0 in StatefulSet pmm-server failed error: pods "pmm-server-0" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "pmm-server" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "pmm-server" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "pmm-server" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "pmm-server" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The error will be the same for PMM2 and PMM3 manifests.

Secure namespace and security contexts

We are now going to apply Pod and Container Security Contexts to both manifests.

Under spec.containers we add everything that Kubernetes suggested us:

 - name: pmm-server
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop:
 - ALL
 runAsNonRoot: true
 seccompProfile:
 type: 'RuntimeDefault'

PMM2

% kubectl -n secure-namespace apply -f 03.pmm2-secure.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 0/1 Error 2 (15s ago) 28s

Even though the PMM2 server Pod can be created now, it is failing to start. If you check the logs, you are going to see the following:

% kubectl -n secure-namespace logs pmm-server-0
...
Error: Can't drop privilege as nonroot user
For help, use /usr/local/bin/supervisord -h

PMM3

% kubectl -n secure-namespace apply -f 04.pmm3-secure.yaml
statefulset.apps/pmm-server created
service/pmm-server created

% kubectl -n secure-namespace get pods
NAME READY STATUS RESTARTS AGE
pmm-server-0 1/1 Running 0 32s

PMM3 starts just fine.

Helm

The recommended approach to deploy PMM3 in Kubernetes is via Helm. You can find our helm charts in percona/helm-charts github repository and more in our documentation.

To deploy PMM3 in a namespace or environment with strict security (like OpenShift), you need to pass similar security context parameters. You can find how in 05.pmm3-helm.yaml values manifest.

Then the deployment will look like this:

helm repo update
helm install pmm3 percona/pmm -f 05.pmm3-helm.yaml --namespace secure-namespace

Conclusion

PMM 3’s rootless design excels where PMM 2 falters in secure Kubernetes environments. Enforcing Pod Security Standards, we saw PMM 3 deploy successfully, while PMM 2 failed, even with security contexts. This highlights PMM 3’s enhanced security, crucial for modern, hardened deployments. Using Helm simplifies secure PMM 3 deployments, ensuring robust database monitoring without compromising security.

Try out Percona Monitoring and Management version 3, 100% open source database observability solution, and learn more about its enhancements.

Tell us what you think in our forum or let us know if you are looking for commercial support.

How to Use IAM Roles for Service Accounts (IRSA) with Percona Operator for MongoDB on AWS

Natalia Marukovich — Mon, 17 Feb 2025 00:00:00 UTC

Introduction

Percona Operator for MongoDB is an open-source solution designed to streamline and automate database operations within Kubernetes. It allows users to effortlessly deploy and manage highly available, enterprise-grade MongoDB clusters. The operator simplifies both initial deployment and setup, as well as ongoing management tasks like backups, restores, scaling, and upgrades, ensuring seamless database lifecycle management.

When running database workloads on Amazon EKS (Elastic Kubernetes Service), backup and restore processes often require access to AWS services like S3 for storage. A key challenge is ensuring these operations have secure, least-privileged access to AWS resources without relying on static credentials. Properly managing these permissions is crucial to maintaining data integrity, security, and compliance in automated backup and restore workflows.

IAM Roles for Service Accounts (IRSA) is the recommended approach to solve this problem. IRSA allows Kubernetes pods to securely assume IAM roles, eliminating the need for hardcoded credentials, long-lived AWS keys, or excessive permissions. Instead, it leverages OpenID Connect (OIDC) authentication, ensuring that only the right workloads get access to AWS services.
By implementing IRSA, you enhance the security posture of your Kubernetes workloads while simplifying IAM management. In this article, we’ll walk through how IRSA works, why it’s beneficial, and how to configure it properly for the Percona Operator for MongoDB in EKS clusters.

IRSA Installation and Configuration for Percona Operator for MongoDB

IRSA requires an OpenID Connect (OIDC) provider associated with your EKS cluster.
So, you should create an OIDC provider for your EKS cluster.

Creating an OIDC provider for your EKS cluster involves several steps. This setup allows your EKS cluster to use IAM roles for service accounts, which makes it possible to grant fine-grained IAM permissions to pods.

shell
# Check if OIDC is already set up:

aws eks describe-cluster --name  --query "cluster.identity.oidc.issuer" --output text

https://oidc.eks.eu-west-3.amazonaws.com/id/7AA1C67941083331A80382E464EB2F1F

# If it is not already set up, create an OIDC provider:

eksctl utils associate-iam-oidc-provider --region  --cluster  --approve

Here oidc-id is 7AA1C67941083331A80382E464EB2F1F. We will use it under role creation.

Create an IAM Policy to access s3 buckets. Substitute with your correct bucket name:

shell
# Define the required permissions in an IAM policy JSON file: 


cat s3-bucket-policy.json

{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:*"
 ],
 "Resource": [
 "arn:aws:s3:::",
 "arn:aws:s3:::/*"
 ]
 }
 ]
}

# Create the IAM policy:

aws iam create-policy --policy-name  --policy-document file://s3-bucket-policy.json

Create an IAM Role and Attach the Policy:

shell
# Role example. Replace  with account id and  with cluster’s OIDC ID


cat role-trust-policy.json



{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/"
 },
 "Action": "sts:AssumeRoleWithWebIdentity",
 "Condition": {
 "StringEquals": {
 "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com"
 }
 }
 }
 ]
}

# Create role:

aws iam create-role --role-name  --assume-role-policy-document file://role-trust-policy.json --description "Allow access to s3 bucket"

Attach the policy to the role.

shell
# Please update ,  and  with the corresponding values.

aws iam attach-role-policy --role-name  --policy-arn arn:aws:iam:::policy/

Install the operator and deploy Percona Server for MongoDB in your EKS cluster (skip this step if you already have the operator and the database cluster installed).
To ensure proper functionality, we need to annotate both the operator service account (default: percona-server-mongodb-operator) and the cluster service account (default: default).

🔴 Warning: The cluster and operator won’t restart automatically; therefore, a manual restart is necessary to apply the changes.

shell
# Get service accounts:

$ kubectl get sa -n 
NAME SECRETS AGE
default 0 25m
percona-server-mongodb-operator 0 25m


# Get role_arn:

aws iam get-role --role-name  --query "Role.Arn" --output text

# Annotate service account. Please update role_arn with appropriate value.

kubectl annotate serviceaccount default \
 eks.amazonaws.com/role-arn=""

kubectl annotate serviceaccount percona-server-mongodb-operator \
 eks.amazonaws.com/role-arn=""

To verify that the settings have been applied, inspect service accounts and the environment variables in both the operator and replica set (RS/Config) pods. The variable AWS_ROLE_ARN should be properly set.

shell
# Check annotation in service account

$ kubectl get sa -n  percona-server-mongodb-operator -o yaml

$ kubectl get sa -n  default -o yaml

# Check the variable inside container

$ kubectl exec -ti  -n  bash

bash-5.1$ printenv | grep 'AWS_ROLE_ARN'
AWS_ROLE_ARN=arn:aws:iam::1111111111111:role/some-name-psmdb-access-s3-bucket

$ kubectl exec -ti  -n  bash

[mongodb@some-name-rs0-0 db]$ printenv | grep 'AWS_ROLE_ARN'
AWS_ROLE_ARN=arn:aws:iam::1111111111111:role/some-name-psmdb-access-s3-bucket

Configure the backup/restore settings as usual, but do not provide s3.credentialsSecret for the storage in deploy/cr.yaml. For detailed instructions please refer to Configure storage for backups.

shell
# backup section in cr.yaml example 
 storages:
 aws-s3:
 type: s3
 s3:
 region: 
 bucket: 

Conclusion

Using IAM Roles for Service Accounts (IRSA) in an Amazon EKS cluster is a best practice when running database operators in Kubernetes. By integrating IRSA, database operators—such as the Percona Server for MongoDB Operator—can securely access AWS services like S3 for backups without relying on static credentials.

IRSA enhances security by enforcing the principle of least privilege, ensuring that database operators in EKS have access only to the specific AWS resources they require. This approach reduces the risk of unauthorized access while also improving manageability by eliminating the need to store and rotate AWS credentials within Kubernetes secrets. By adopting IRSA in Percona Server for MongoDB Operator , organizations can create a more secure, scalable, and automated environment for managing MongoDB databases.

Join Percona for Google Summer of Code 2025 – Explore, Innovate, and Contribute!

Radoslaw Szulgo — Wed, 05 Feb 2025 00:00:00 UTC

Are you passionate about open-source databases, AI/ML, and security? Do you want to work on real-world projects that impact thousands of developers and enterprises worldwide? Percona is excited to invite students to participate in Google Summer of Code 2025 (GSoC) and help advance our cutting-edge open-source database solutions!

Why Contribute to Percona?

At Percona, we believe that open world is a better world! GSoC is an excellent opportunity to work with seasoned developers, gain hands-on experience, and contribute to powerful database tools used by businesses globally.

For 2025, we’re especially interested in projects that focus on AI/ML and security—two critical areas shaping the future of databases. Whether you’re passionate about automating database performance insights with AI or hardening security for mission-critical data, we have exciting challenges for you!

Percona mentors are going to help you realize your ideas or one of the ideas below. We invite you to contribute to products and projects such as:

Percona XtraDB Cluster
Percona Server for MySQL
Percona Distribution for PostgreSQL
Percona Operator for PostgreSQL
Percona Server for MongoDB
Percona Backup for MongoDB
Percona Operator for MongoDB
Percona Everest
Percona Monitoring and Management (PMM)
pg_tde: Transparent Database Encryption for PostgreSQL
As well as CI/CD related projects with the Percona Build Engineering.

Project Ideas for GSoC 2025

Below are some suggested project ideas categorized by Percona software:

Percona Distribution for PostgreSQL

Snapshot-based PostgreSQL backups

Database users are often very familiar with their storage provider’s storage snapshot capabilities. These snapshots are very handy and performant to use, hence their popularity among users. Backups for other databases (e.g., MongoDB) are often configured via this capability as it provides many performance benefits for large-scale data, especially on Cloud deployments. Having such technology supported across the backup solutions for multiple databases makes it possible to leverage this effectively for Percona Everest via the Percona Operators.

Comments from Crunchy on what needs to be glued together to get snapshots and pgBackRest working together better. Additionally, Timescale on how they use snapshots and pgBackRest together in their hosted managed service.

Deliverables: Have an API available to Percona Distribution for PostgreSQL to effectively use storage snapshots to create backups and restore from storage snapshot-based backups. Preferably, have it added to/complementary to the currently recommended solution of pgBackRest.

Have Percona Operator for PostgreSQL expose the storage snapshot-based backup/restore so that Percona Everest can leverage it.

Required/preferred skills: C++, PostgreSQL, Kubernetes Duration: 350 hours **Difficulty level: **Hard Mentors: @Andrew_Pogrebnoi, @Jan_Wieremjewicz Relevant repository and resources:

pgBackRest to Barman close gap improvements

PostgreSQL has two main backup tools: Barman and pgBackRest. Both are powerful backup and restore tools, each with its own strengths. pgBackRest is generally considered more advanced in terms of parallelism, performance, and flexibility. Barman does offer some advantages in certain areas. While pgBackRest is maintained by Community, Barman is a tool mainly maintained by one company and is less popular. Barman does have UX improvements over pgBackRest, especially for non-expert users:

direct WAL archiving with PostgreSQL’s built-in archive_command,
simpler backup and recovery process, especially for standby creation,
clearer logging and monitoring for backup integrity,
simpler configuration in small to medium deployments,
better native tools for cloud backups

It would be beneficial for Percona, which uses pgBackRest in the Percona Distribution for PostgreSQL, to have the backup tool close any functionality gaps in Barman. Percona customers sometimes use Barman and expect Percona to support it. Having a way to migrate off Barman to pgBackRest, reducing any potential friction for the users, would be beneficial.

Deliverables: Provide a close-gap set of improvements based on the list available in the description

Required/preferred skills: C++, PostgreSQL Duration: 350 hours Difficulty level: Hard Mentors: @Andrew_Pogrebnoi, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Tool to investigate PostgreSQL locks for dummies

Currently, there is no tool that allows users with low experience to detect and understand all types of locks on their PG database, which may lead to many issues in deployments not managed by expert PostgreSQL users. As described in the blog posts below, understanding how locks work is difficult:

Deliverables:

Detect ddl with mixed strong locks and others. i.e. allow to review locked PIDs as pg_locks will not work
Present all locks in a GUI
(Streched) have the GUI integrated in PMM

Required/preferred skills: C++, PostgreSQL Duration: 350 hours Difficulty level: Medium **Mentors: **Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Session continuity for PgBouncer for the zero downtime upgrades

Percona is looking to introduce zero downtime upgrades capability to the Percona Operator and later on to Percona Everest. The assumption is to base on pgBouncer and our HA solution utilizing the replica with the new database and a switch from the previous version to the new version.

Such a solution provides a zero downtime upgrade capability and a Rollback capability. To provide true zero downtime major upgrades for the current Percona Distribution for PostgreSQL, there needs to be an improvement that takes over the switching of sessions between the databases: the previous version and the new version

In the future, this tool should also make it possible to zero downtime and migrate to Everest.

Deliverables: Extend pgBouncer to ensure that the sessions can be switched between the databases without downtime for the users but only a potential performance drop

Required/preferred skills: C++, Kubernetes Duration: 175 hours Difficulty level: Hard **Mentors: **Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

GitHub - percona/postgres: Percona Server for PostgreSQL

Percona Software for MongoDB

Interactive Shell Installer for Percona Software for MongoDB

This project aims to develop an interactive shell-based installer for Percona Server for MongoDB and Percona Backup for MongoDB. The installer will simplify the installation, configuration, and initial setup process, making it easy for users to deploy these open-source enterprise solutions efficiently. The primary goal is to enhance the user experience by reducing manual setup steps and ensuring proper configuration through guided prompts and automation.

Deliverables:

A command-line-based interactive installer script.
Automated dependency checks and installation.
Interactive prompts for configuration choices (e.g., authentication, replication, sharding).
Seamless installation of both Percona Server for MongoDB and Percona Backup for MongoDB.
Integration with package managers for major Linux distributions (Debian, Ubuntu, RHEL, CentOS).
Logging and validation mechanisms to ensure correct setup.
Documentation and user guide for the installer.

Required/preferred skills: C++ or Go

Duration: 350 hours

Difficulty level: Medium

Mentors: @radoslaw.szulgo

Relevant repository and resources: https://github.com/percona/percona-backup-mongodb

Percona Backup for MongoDB backup speed throttling

On large scale deployments, backups may significantly impact network performance - speficially network bandwidth may be heavily utilized, if the backup storage is fast, causing performance degradation of the database itself. Database reliability engineers would like to reduce the network load by slowing down physical backups with Percona Backup for MongoDB (PBM) configuration. The scope of the project is to implement a network bandwidth rate limiter and perform load testing showing the impact or rate limiting on backup time.

Deliverables: The expected outcome of this project is insurance that backup will not degrade network bandwidth impacting the database. As a result a participant needs to provide proposed code changes in a form of fork of PBM and a create a report with load test results.

Required/preferred skills: Go
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/percona-server-mongodb

Percona Backup for MongoDB Golang SDK

The project’s purpose is to extend capabilities and reduce maintenance in monitoring, managing, and automating backup and restores of MongoDB from Percona Monitoring and Management (PMM) tool. In the scope of the project there’s a migration from Percona Backup for MongoDB CLI to a dedicated PBM golang client library in PMM. The client library (aka SDK) has to be implemented and map all current CLI operations to Go API functions. As a stretch goal, the project can be extended into implementing a backup progress reporting using the created SDK.

Deliverables: The expected outcome of the project is the reduced maintenance of backup integration in PMM project and enabling extensibility of backup management. As a result of the project a new open-source SDK should be create

Required/preferred skills: Go
Duration: 350 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/pmm

CEPH Storage support in Percona Backup for MongoDB

Ceph is an open source Software Defined Storage(SDS) software that is massively scalable and reliable. It’s one of the most popular storage technologies in Kubernetes and Openshift. The project aims to enable users to store their Percona Backups for MongoDB data on a Ceph storage which would be very convenient as they wouldn’t need to manage other additional storages. The scope of the project includes building a workspace setup on Kubernetes and Percona Operator for MongoDB, research on current challanges using Ceph storage, and implement necessary changes to make it work in a performant way. At the end document the solution.

Deliverables: The project’s deliverables are technical documentation, Percona Operator for MongoDB changes, and instructions on setting up an environment with Ceph storage.

Required/preferred skills: Go, Kubernetes
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:
- https://github.com/percona/percona-backup-mongodb
- https://github.com/percona/percona-server-mongodb

BoostFS storage support in Percona Backup for MongoDB

Dell Data Domain Boost File System (BoostFS) provides a general file system interface to the DD Boost library, allowing standard backup applications to take advantage of DD Boost features. In this project we’d like to extend our open-source Percona Backup for MongoDB to leverage that storage technology to reduce backup and restore time - and the same time help users to reduce their Recovery Time Objective (RTO) and Recovery Point Objective (RPO). In the scope of the project there’s a preparation of the workspace setup with Percona Server for MongoDB, Percona Backup for MongoDB and mounted BoostFS disk volumes on Google Cloud Platform and documenting the architecture of the environment. Additionally, the project includes a research on how PBM works with that storage and implementing necessary changes to PBM to make it work. Finally, a simple benchmark should be performed that proves the performance boost.

Deliverables: It is expected that project delivers an architecture diagram of the testing environment in Google Cloud Platform, implementation of required changes to support BoostFS in PBM, and report incl. performance benchmark results and comparison to other storage systems.

OpenStack Swift storage support in Percona Backup for MongoDB

The OpenStack Object Store project, known as Swift, offers cloud storage software so that you can store and retrieve lots of data with a simple API. It’s built for scale and optimized for durability, availability, and concurrency across the entire data set. Swift is ideal for storing unstructured data that can grow without bound. Swift is very convenient to use as a backup storage for MongoDB workloads running on OpenStack platform. The scope of the project includes building a workspace environment on Google Cloud Platform with OpenStack clusters and running there Percona Server for MongoDB. Then implementing required changes in Percona Backup for MongoDB to support Swift storage. Finally, performing benchmark tests and comparison to GCP native storages.

Deliverables: It is expected that project delivers an architecture diagram of the testing environment in Google Cloud Platform, implementation of required changes to support OpenStack Swift in PBM, and report incl. performance benchmark results and comparison to other storage systems.

Required/preferred skills: Go, GCP, OpenStack
Duration: 175 hours
Difficulty level: Medium
Mentors: @radoslaw.szulgo
Relevant repositories and resources:

Percona Server for MySQL and Percona XtraDB Cluster

Automating Code Merges with AI

The regular and manual merge from Oracle’s GitHub repository process is time-consuming, complex, and prone to errors, particularly due to merge conflicts. Careful attention is required to avoid introducing regressions into Percona’s open-source products. While this project is specific to Percona’s needs, it addresses a common challenge in open-source software development, as many projects rely on upstream repositories for their code. Therefore, the solution can be generalized and could benefit other open-source projects with similar code integration needs.

This GSoC project aims to develop an intelligent system using Artificial Intelligence to automate the MySQL fork merge process. Percona has been performing these merges for 18 years, accumulating a wealth of historical data (code changes, merge resolutions, conflict histories, test results) that can be leveraged to train an AI model.

The core objective is to create a tool that can:

Analyze upstream changes: Process and understand the changes introduced by Oracle in their MySQL repository.
Identify merge conflicts: Identify conflicts between upstream changes and Percona’s modifications.
Suggest merge resolutions: Propose solutions for resolving identified conflicts, drawing on patterns from historical merge data.
Automate merges: Automatically apply upstream changes with the suggested merge resolutions.
Learn and adapt: Continuously improve its performance and accuracy by learning from new merge data and feedback.

Deliverables:

Reduced merge time and effort: Automating the merge process will free up developer time for other critical tasks.
Improved merge accuracy: AI can potentially identify subtle conflicts that might be missed by manual review.
Faster release cycles: Streamlining the merge process will enable quicker releases of updated Percona products.
Open-source contribution: The resulting tool will be open-sourced, benefiting other projects that maintain forks of MySQL or similar databases. This problem is not unique to Percona; other open-source projects facing similar merging challenges can utilize this solution.

As a result of this project, you’re expected to deliver:

A working prototype of the AI-powered merge tool.
Well-documented code and training data.
Comprehensive test suite and evaluation results.
A report detailing the project’s methodology, findings, and future directions.

Required/preferred skills: Python, Machine Learning libraries and frameworks (e.g., TensorFlow, PyTorch, scikit-learn), C++, Git, database systems Duration: 350 hours Difficulty level: Hard Mentors: Julia Vural, Oleksiy Lukin Relevant repositories and resources:

Percona Everest

Easier Troubleshooting on database clusters in Percona Everest

The main goal is to provide tools to Percona Everest users to troubleshoot database clusters. This project will require the implementation of log collection, rotation, UI, and possibly an AI helper to analyze those logs. If users have a centralized log collection implemented, this tool needs to be able to integrate with it.

Deliverables:

Full user flow to support database cluster troubleshooting process (UI, backend, API, integrations).

Log Collection & Rotation System:
- Implement a mechanism to collect logs from Percona Everest-managed database clusters.
- Ensure efficient log rotation to manage storage and performance impact.
- Enable compatibility with external log aggregation tools (e.g., Elasticsearch, Grafana Loki, or OpenTelemetry)
User Interface for Log Access:
- Develop a UI within Percona Everest to allow users to view and analyze logs.
- Include search, filtering, and visualization options for better troubleshooting.
AI-Powered Log Analysis (Stretched scope)
- Explore AI-driven log analysis to provide users with insights, anomaly detection, and recommendations.
- Implement basic AI-assisted troubleshooting if feasible within the project timeline.
Documentation & Testing:
- Deliver user and developer documentation covering installation, usage, and troubleshooting.
- Include test cases and automation scripts to ensure system reliability.

Required/preferred skills: Kubernetes, Go, CI/CD Duration: 350 hours **Difficulty level: **Medium Mentors: @Diogo_Recharte, @Mayank_Shah Relevant repository and resources: https://github.com/percona/everest

Percona Everest RBAC policies management UI

Create a user interface to create and manage role-based access control policies

Deliverables:

Role-Based Access Control (RBAC) UI:
- Develop a user-friendly interface in Percona Everest to create, update, and manage RBAC policies.
- Implement role assignment and permission configuration for database clusters.
Documentation & Testing:
- Deliver comprehensive user and developer documentation.
- Include test cases and automation scripts to ensure reliability.

Required/preferred skills: Front-end, CI/CD tools Duration: 90 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Peter Szczepaniak Relevant repository and resources: https://github.com/percona/everest

Context sensitive help

The Percona Everest documentation contains valuable information, hints, and tips, but we lack a way to present relevant information to our users. This project aims to work with the UX and Docs teams to solve this problem.

Deliverables:

Implement a mechanism to display relevant documentation, hints, and tips based on the user’s current action or screen within Percona Everest.
Ensure seamless integration with the existing UI for a non-intrusive experience.
Enable contextual tooltips, pop-ups, or side panels that present relevant documentation without requiring users to leave the interface.
Support links to full documentation pages when needed.
Optionally, explore AI-driven suggestions based on user behavior and past queries.
Allow users to control the level of help they receive (e.g., enable/disable tips, adjust verbosity).
Provide user and developer documentation on how the system works and how to extend it.
Ensure thorough testing to validate the accuracy and relevance of displayed help content.

Required/preferred skills: Front-end, CI/CD tools Duration: 175 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Peter Szczepaniak Relevant repository and resources: https://github.com/percona/everest

Backups and restore timeline visualization

Databases are usually long-living services, and investigating issues with them is easier when you can see events like backups and restores of this service on a timeline.

Deliverables:

Develop a visual timeline within Percona Everest to display backup and restore events for database clusters.
Ensure the timeline is intuitive, zoomable, and supports different time ranges (e.g., last 24 hours, 7 days, custom range).
Retrieve and display backup and restore events from Percona Everest’s database and logs.
Include metadata such as timestamps, duration, status (success, failure), and associated users or processes.
Allow users to filter events by type (full backup, incremental backup, restore, etc.).
Enable color-coding or icons to differentiate event types at a glance.
Deliver comprehensive user and developer documentation.
Ensure automated tests for data accuracy, UI performance, and usability.

Refactor test automation using page object model

Our project currently has a functional end-to-end (E2E) UI test suite that ensures the stability and correctness of our application. However, the test suite does not follow the Page Object Model (POM) design pattern, making it harder to maintain, scale, and debug.

Deliverables:

Restructure existing test automation to follow the Page Object Model (POM) design pattern.
Ensure better separation of test logic and UI elements for improved maintainability.
Implement modular and reusable page object classes for different UI components and workflows.
Standardize naming conventions and best practices for test scripts.
Improve error handling and logging to make test failures easier to diagnose.
Ensure the refactored test suite runs efficiently in CI/CD pipelines.
Validate test performance improvements and maintain test coverage.

Required/preferred skills: Playwright, Typescript, Kubernetes Duration: 175 hours Difficulty level: Medium Mentors: @Diogo_Recharte, Tomislav_Plavcic, Edith Puclla Relevant repository and resources: https://github.com/percona/everest

Percona Monitoring and Management (PMM)

Queryable backup and restore of Percona Server for MongoDB

The project aims to equip MongoDB Database Administrators with Percona Monitoring and Management (PMM) extension that allows to query data directly from a backup. It solves their pain when they want to inspect just a single document in a collection from a few Terabytes size backup. The time associated with downloading the snapshot, decompressing it, getting it running in a local MongoDB node, and finally running the query would be significant. Not only that, but there are obvious nontrivial costs — both monetary and operational — associated with having to quickly spin up new environments. In scope of the project there is a backend Go application server implementation to run on-demand ephemeral mongodb instance, load data from backup, and enable user to run a DB query from a UI interface.

Deliverables: It’s expected to deliver source code changes to PMM in form of PMM fork that extends PMM functionality for queryable MongoDB backup. Specifically, it is expected to prepare a solution design, implementation, unit and/or integration tests based on a Docker environment, and documentation.

Required/preferred skills: Go, MongoDB
Duration: 350 hours
Difficulty level: Medium
Mentors: @radoslaw.szulgo
Relevant repository:
- https://github.com/percona/pmm
- https://github.com/percona/percona-backup-mongodb

Design Engineering for PMM

Percona Monitoring and Management (PMM) is a long-standing open-source software, but its age also comes with some UX and UI debt. Facing new goals to help innovate PMM, the team is excited to look forward to starting “renovating the house” and swapping the GUI with a more modern one built in-house. We are looking for experts in design engineering to help with:

Making/refining/cataloging UI components that we will need for QA and production;
Creating functional prototypes ad hoc from written ideas or designs;
Convert old PMM pages like for like into new PMM pages (with new UI).

Deliverables:

Contributing with ideas to help make the code library more easy to contribute to;
Contributing to the code library with at least one new component;
Create at least one code prototype for one of the team’s ongoing ideas;
Convert at least one existing PMM functionality page UI into the new UI.

Required/preferred skills: CI/CD, Git, Storybook, React, MUI, Figma

Duration: 350 hours

Difficulty level: Medium

Mentor: @pedro.fernandes

PMM UI for PostgreSQL backups - create, restore, check, monitor

Backup management without UI is not an easy task for the users. Having a tool that could be a tool of choice for backup and restore management could provide a unification layer for multiple backup/restore tools as well as provide a very important value that’s often overlooked: backup monitoring.

As it turns out, many DBAs are often worried about the state of their backups and make it a daily routine task to check how the backups they have configured are. As the environments scale, this task becomes increasingly tiresome. Also, having to check whether the backups they create are effective makes it another routine task that needs either extra automation scripting to run a restore or a manual task.

Deliverables:

Extend PMM UI to add existing backups so that they can be monitored
Extend PMM UI to create backups
Create a tool to automate the backup testing (check whether the backups created are usable)
Extend PMM to monitor and alert on backup irregularities
Integrate the backup management with external schedulers like cron.

Required/preferred skills: C++, Go, PostgreSQL Duration: 350 hours Difficulty level: Hard Mentors: Kai Wagner, @Jan_Wieremjewicz Relevant repository and resources:

LLM-Powered Test Scenario Generation for Open Source Contributions

Open-source projects thrive on community contributions, but ensuring that each pull request (PR) has adequate test coverage is a major challenge. Many PRs introduce changes without proper regression tests, leading to bugs and unstable releases.

This project aims to build an LLM-powered tool that analyzes code changes in GitHub PRs and automatically generates relevant test scenarios. Using an Open Source LLM (e.g., DeepSeek, Mistral, LLaMA), the system will identify impact areas, suggest missing test cases, and recommend regression tests based on past commits. The goal is to integrate this into GitHub workflows, enabling maintainers to quickly assess PR test coverage and guide contributors in writing better tests.

Deliverables:

The student will work on developing a system with the following features:

PR Analysis & Impact Assessment
Extract and analyze code diffs in pull requests.
Identify affected functions, dependencies, and modules.
Predict impact areas using a dependency graph.

Test Scenario Generation using LLMs

Use an Open Source LLM (DeepSeek, Mistral, etc.) to generate test cases.
Recommend unit tests, integration tests, and regression scenarios.
Compare new tests with existing ones to detect gaps in coverage.

GitHub Bot for Automated Suggestions

Implement a bot that comments on PRs with test recommendations.
Provide interactive feedback to contributors and maintainers.
Integrate with GitHub Actions for CI/CD automation.

Regression Test Identification

Identify existing test cases that need to be re-run.
Suggest additional tests based on historical PRs and past bug reports.

Evaluation Metrics & Benchmarking

Measure effectiveness by tracking missed bugs before/after integration.
Collect feedback from maintainers and contributors.

Future Scope:

Extend beyond GitHub to GitLab, Bitbucket, and other version control systems.
Support additional test types, such as security and performance tests.
Implement self-learning mechanisms to improve accuracy over time.

Required/preferred skills: Strong programming skills in Python or JavaScript, Experience with GitHub APIs & Pull Request Workflows, Understanding of Machine Learning / LLMs (DeepSeek, Mistral, LLaMA, etc.), Familiarity with Software Testing & QA Automation, Experience with CI/CD Pipelines & GitHub Actions (Bonus). Duration: 350 hours Difficulty level: Hard Mentor: Peter Sirotnak, @vasyl.yurkovych

Percona Build Engineering

SBOMs for Percona database software - MySQL, PostgreSQL, and MongoDB

A “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. An SBOM is a nested inventory, a list of ingredients that comprise software components. The project aims to adapt Percona’s build pipelines to generate SBOMs for Percona Software for MySQL, PostgreSQL, and MongoDB. This will enable organizations using Percona software to be more secure and avoid software supply chain vulnerabilities that were very harmful in late 2020 with the discovery of the Solar Winds cyberattack or later with the Log4j security flaw.

Deliverables: At the end of the project, a running staging pipeline in Jenkins and Trivy should produce complete SBOMs for Percona Server for MySQL, PostgreSQL, and MongoDB, Percona Backup for MongoDB, Percona Xtra Backup for MySQL. SBOMs are uploaded automatically to the Percona repository and are downloadable publicly. Additionally, technical documentation on how the process works is expected to be created.

Required/preferred skills: Jenkins, Trivy
Duration: 175 hours
Difficulty level: Easy
Mentors: @radoslaw.szulgo
Relevant repository and resources:

Evolving CI/CD: Automating Build, Test, and Release for Robust Software Delivery

Continuous Integration and Continuous Deployment (CI/CD) pipelines are the backbone of modern software development, ensuring rapid, reliable, and repeatable delivery. However, many pipelines still operate in fragmented stages, where builds and tests are automated, but releases remain a manual or semi-automated process.

This project aims to transform our CI/CD pipelines into a true end-to-end automated system, seamlessly integrating build, test, and release stages. By implementing best practices in CI/CD automation, we will ensure that only thoroughly tested software progresses to release, minimizing human intervention and reducing the risk of deployment failures.

Deliverables: The successful completion of this project will result in a fully automated and robust CI/CD pipeline that seamlessly integrates build, test, and release processes. The key outcomes will include:

Fully Automated CI/CD Pipeline

A redesigned pipeline where builds, testing, and releases are interconnected and automated. Code changes will automatically trigger builds, run tests, and, if successful, deploy releases without manual intervention. Comprehensive Test Integration

The pipeline will incorporate unit tests, integration tests, security scans, and other quality assurance mechanisms. Ensuring that faulty builds do not reach production by enforcing test-driven deployment. Automated Release Process

A mechanism that automatically releases software only if all tests pass. Versioning, tagging, and artifact management will be streamlined. The release process will be documented and configurable for different environments (e.g., staging, production). Infrastructure as Code (IaC) & Deployment Automation

Documentation & Guides

Clear technical documentation detailing the new pipeline’s workflow and configuration. A step-by-step guide for developers and DevOps engineers on how to use and extend the pipeline.

Required/preferred skills: CI/CDl like Jenkins, GitHub Actions, GitLab CI, or similar; Docker; Testing frameworks and automated deployment strategies; infrastructure as code (IaC) and cloud environments is a plus.
Duration: 350 hours
Difficulty level: Medium
Mentors: @Evgeniy_Patlan , @Vadim_Yalovets
Relevant repository: https://github.com/Percona-Lab/jenkins-pipelines

Build Automation for Open-Source Databases

Building and maintaining multiple database forks—such as MySQL, MongoDB, and PostgreSQL—often involves redundant build scripts, leading to inefficiencies, inconsistencies, and maintenance overhead. Currently, each database has its own set of build scripts despite sharing many common steps.

This project aims to develop a modular, extensible build system that allows for streamlined compilation and packaging of different database forks. The system will provide a flexible framework where users can select required modules, specify target OS distributions, and automate the build process with minimal configuration.

By implementing a plugin-based architecture, this modular builder will simplify cross-database maintenance, reduce duplication, and improve consistency across different builds.

Deliverables:

Modular Build Framework – A reusable, pluggable system that dynamically selects required modules for MySQL, MongoDB, and PostgreSQL builds.
Multi-OS Support – Automated builds for multiple Linux distributions (Debian, Ubuntu, CentOS, RHEL) with configurable OS selection.
Automated Package Creation – DEB and RPM package generation with standardized versioning and tagging.
Configurable & Scalable Builds – Easy customization of build parameters, allowing extension to new database forks or patches.
CI/CD Integration – Optional support for Jenkins, GitHub Actions, or GitLab CI to enable fully automated builds.
Comprehensive Documentation – User and developer guides with example configurations for quick adoption and extension.

Required/preferred skills: Bash/Python, CMake, Makefiles, Autotools, Linux and packaging (DEB/RPM), dependency management, CD/CD tools are a plus

Duration: 175 hours

Difficulty level: Medium

Mentors: @Evgeniy_Patlan , @Vadim_Yalovets

Relevant repositories:

More ideas are coming soon!

Suggest your ideas in the comments of the post or on the forum.

GSoC isn’t just about working on predefined ideas—it’s about innovation! If you have a project idea that aligns with Percona software, AI/ML, security, or database performance, submit your proposal, and our mentors will be happy to discuss it with you.

Do you have questions? Visit our Community Forum or join our chat channels to connect with potential mentors.

Ready to get started? See our Google Summer of Code 2025: Contribution guide.

See you in GSoC 2025!

Percona Operator for MongoDB 1.19: Remote Backups, Auto-Generated Passwords, and More!

Sergey Pronin — Fri, 31 Jan 2025 00:00:00 UTC

The latest release of the Percona Operator for MongoDB, version 1.19, is here. It brings a suite of enhancements designed to streamline your MongoDB deployments on Kubernetes. This release introduces a technical preview of remote file server backups, simplifies user management with auto-generated passwords, supports Percona Server for MongoDB 8.0, and includes numerous other improvements and bug fixes. Let’s dive into the details of what 1.19 has to offer.

Remote Backups with Network File System (Technical Preview)

Backing up your MongoDB data is crucial, and Percona Operator for MongoDB 1.19 introduces a powerful new option for backup storage: the filesystem type. This feature, currently in technical preview, allows you to leverage a remote file server, mounted locally as a sidecar volume, for your backups. This is particularly useful in environments with network restrictions that prevent the use of S3-compatible storage or for organizations using non-standard storage solutions that support the Network File System (NFS) protocol.

Setting Up Remote Backups

To use this new capability, you’ll need to add your remote storage as a sidecar volume within the replsets section of your Custom Resource (and configsvrReplSet for sharded clusters). Here’s how:

replsets:
 ...
 sidecarVolumes:
 - name: backup-nfs-vol
 nfs:
 server: "nfs-service.storage.svc.cluster.local"
 path: "/psmdb-my-cluster-name-rs0"
 ...

Then, configure the mount point and sidecar volume name in the backup.volumeMounts section:

YAML:

backup:
 ...
 volumeMounts:
 - mountPath: /mnt/nfs/
 name: backup-nfs-vol
 ...

Finally, set up a filesystem type storage in the backup.storages section, pointing it to the mount point:

YAML:

backup:
 enabled: true
 ...
 storages:
 backup-nfs:
 type: filesystem
 filesystem:
 path: /mnt/nfs/

See more in our documentation about this storage type.

Simplified User Management with Auto-Generated Passwords

Managing user credentials just got easier. Percona Operator for MongoDB 1.19 enhances declarative management of custom MongoDB users by adding the ability to generate passwords automatically. Now, when defining a new user in your deploy/cr.yaml file, you can omit the reference to an existing Secret containing the password, and the Operator will handle the generation for you:

YAML:

...
users:
 - name: my-user
 db: admin
 roles:
 - name: clusterAdmin
 db: admin
 - name: userAdminAnyDatabase
 db: admin

The Operator will create a Secret to store the generated password securely. It is important to note that the Secret will be created after the cluster is in the Ready state.

Get the user credentials: Find the Secret resource named -custom-user-secret Get the user password with this one-liner:

kubectl get secret my-cluster-name-custom-user-secret -o jsonpath='{.data.my-user}' | base64 -d

You can find more details on this automatically created Secret in our documentation.

Percona Server for MongoDB 8.0 Support

Staying up-to-date with the latest MongoDB versions is essential for performance and security. Percona Operator for MongoDB 1.19 now officially supports Percona Server for MongoDB 8.0, in addition to 6.0 and 7.0. This means you can leverage the latest features and improvements from MongoDB 8.0, combined with the enterprise-grade enhancements and open-source commitment of Percona Server for MongoDB.

Check out this blog post to learn more about the features in MongoDB 8.0.

Streamlined AWS S3 Access with IAM Roles for Service Accounts (IRSA)

Percona Operator for MongoDB 1.19 adds support for IAM Roles for Service Accounts (IRSA), simplifying secure access to AWS S3 for backups on Amazon EKS. IRSA lets you grant granular S3 permissions to specific Pods via their associated Kubernetes service accounts. This approach ensures that only the Pods that require S3 access receive it, adhering to the principle of least privilege. Furthermore, each Pod can only access credentials linked to its service account, providing strong credential isolation. For enhanced security, all S3 access is tracked through AWS CloudTrail, enabling comprehensive auditability. All of this happens without the need to manually manage and distribute AWS credentials.

Configuration Steps

Create an IAM Role: Define an IAM role with S3 access permissions. See AWS documentation.
Identify Service Accounts: The Operator uses percona-server-mongodb-operator and your cluster uses default (customizable in deploy/cr.yaml).

Annotate Service Accounts: Link the IAM role to both service accounts:

$ kubectl -n  annotate serviceaccount default eks.amazonaws.com/role-arn:  --overwrite
$ kubectl -n  annotate serviceaccount percona-server-mongodb-operator eks.amazonaws.com/role-arn:  --overwrite

Configure S3 Storage: Set up S3 storage in deploy/cr.yaml without s3.credentialsSecret. The Operator will use IRSA.

Important: IRSA credentials take precedence over IAM instance profiles, and S3 credentials in a Secret override both.

IRSA streamlines S3 access, enhancing security and manageability for your MongoDB backups on EKS. Learn more in our documentation.

Conclusion

Percona Operator for MongoDB 1.19 delivers a significant step forward in simplifying and automating the management of your MongoDB clusters on Kubernetes. With features like remote backups, auto-generated passwords, and support for Percona Server for MongoDB 8.0, this release empowers you to deploy, manage, and scale your databases with greater ease and efficiency.

We encourage you to explore the full release notes and try out the new features. As always, your feedback is invaluable to us. Please share your thoughts and contribute to the project on our GitHub repository or our Community Forum.

Percona Monitoring and Management 3.0.0-GA

Ondrej Patocka — Wed, 29 Jan 2025 00:00:00 UTC

We’re excited to announce the release of Percona Monitoring and Management (PMM) 3.0.0 GA.

The Percona Monitoring and Management (PMM) 3.0.0 release delivers major security and stability enhancements. Notable security improvements include rootless deployments and encryption of sensitive data, along with improved API authentication using Grafana service accounts. Deployment options have expanded with official ARM support and the ability to use Podman for rootless deployments, providing flexibility and better security. Additionally, the introduction of containerized architecture has increased stability, and a streamlined upgrade process ensures reliability and ease of maintenance.

User experience has been significantly improved with more flexible monitoring configurations and UI-based upgrades for Podman installations. This release also includes new features such as monitoring for MongoDB 8.0 and integration with Watchtower for automated container updates. These enhancements aim to provide users with a more secure, stable, and user-friendly monitoring and management experience.

Release notes

To see the full list of changes, check out the 3.0.0 GA Release Notes

Percona Monitoring and Management (PMM) 3.0.0 Release Notes:

Security Enhancements:
- Implementation of rootless deployments to enhance security.
- Encryption of sensitive data to ensure information confidentiality.
- Improved API authentication with Grafana service accounts, increasing access security.
Deployment Options:
- Official PMM Client ARM support, allowing the use of PMM on ARM architecture devices.
- Rootless deployments using Podman, providing flexibility and security.
- Support for deployments using Helm, Docker, Virtual Appliance, and Amazon AWS for various use cases.
Stability Improvements:
- Increased stability through containerized architecture, providing isolation and manageability.
- Streamlined upgrade process, reducing the risk of failures during updates and enhancing reliability.
User Experience:
- Flexible monitoring configurations, allowing users to tailor the system to their needs.
- UI-based upgrades for Podman installations, making the update process more convenient and intuitive.
New Features:
- Monitoring for MongoDB 8.0, ensuring support for the latest database versions.
- Integration with Watchtower for automated container updates, simplifying management and keeping the system up-to-date.

We invite you to install and try the new PMM 3.0

Quickstart guide

Get started with PMM

Multiple installation options

About PMM installation

MySQL 8.4 Support in Percona Toolkit 3.7.0

Sveta Smirnova — Mon, 06 Jan 2025 00:00:00 UTC

Percona Toolkit 3.7.0 has been released on Dec 23, 2024. The main feature of this release is MySQL 8.4 support.

In this blog, I will explain what has been changed. A full list of improvements and bug fixes can be found in the release notes.

TLDR;

Replication statements in 8.4 are fully supported by the Percona Toolkit
pt-slave-delay has been deprecated.
pt-slave-find has been renamed to pt-replica-find. The old name has been deprecated but exists in the repository as an alias of the pt-replica-find.
pt-slave-restart has been renamed to pt-replica-restart. Old name has been deprecated but exists in the repository as an alias of the pt-replica-restart.
Basic SSL support has been added to the tools where it was not working before (see https://perconadev.atlassian.net/browse/PT-191 ), and Percona Toolkit now supports caching_sha2_password and sha256_passwordauthentication plugins. Full implementation of https://perconadev.atlassian.net/browse/PT-191 is planned for the next version.

Replication Statements

MySQL 8.4 removed earlier deprecated offensive language, such as SLAVE or MASTER. This made tools written for earlier versions not compatible with the new version. Percona Toolkit was also affected, and I had to rewrite it.

However, Percona Toolkit should be able to run not only with MySQL 8.4 but also with older versions. So, the change was not a simple grep and replace of offensive words. It is not even possible for version MySQL 8.0 because new syntax was first introduced in 8.0.23 for the CHANGE REPLICATION SOURCE and START/STOP REPLICA commands. Earlier versions weren’t aware of this change.

Another challenge was the fact that I could replace all occurrences of the word SLAVE with REPLICA. Still, I could not do the same for the MASTER and SOURCE pairs because replication source-related commands are mapped differently:

Legacy syntax	Syntax without offensive words
`CHANGE MASTER`	`CHANGE REPLICATION SOURCE` (since 8.0.23)
`SHOW MASTER STATUS`	`SHOW BINARY LOG STATUS` (since 8.4.0)
`RESET MASTER`	`RESET BINARY LOGS[ AND GTIDS]` ( since 8.4.0)
`MASTER` in other commands	`SOURCE` (partially since 8.0.23, fully since 8.4)

So, I added selectors that use the correct command depending on the MySQL server version.

I intentionally implemented new syntax for version 8.4 only, so I do not have to check every single minor version of 8.0. I also did not implement new syntax for MariaDB. This may happen in the future.

However, all messages displayed to the user use the new syntax. If you rely on old syntax somewhere in your scripts, adjust them.

Internally, most of the functions were renamed to use the new syntax, but the important module lib/MasterSlave.pm kept its name.

Deprecated and Outdated Tools

As a result of this change, pt-slave-delay has been deprecated. The tool stays in the repository and works as before when connected to MySQL 8.0 or an earlier version. However, it refuses to work with MySQL 8.4. The tool will be removed in one of the future versions.

Tools pt-slave-find and pt-slave-restart were renamed to pt-replica-find and pt-replica-restart. Aliases with old names still exist, so you have time to change your scripts. However, expect that these aliases will be removed in one of the future versions as well.

Tool pt-variable-advisor has been updated to reflect current default values.

Basic SSL Support

Percona Toolkit did not have consistent SSL support: some of the tools were able to connect using SSL, and others did not. This was reported at https://perconadev.atlassian.net/browse/PT-191. In this version, I added option “s” for DSN that instructs DBD::mysql to open a secure connection with the database. As a result, Percona Toolkit now supports caching_sha2_password and sha256_password authentication plugins. But other SSL options are still missed. Full SSL support will be added in the next version.

Conclusion

Percona Toolkit fully supports MySQL 8.4. If you use pt-slave-find and pt-slave-restart, consider calling them by their new names pt-replica-find and pt-replica-restart. Tool pt-slave-delay has been deprecated and will be removed in future versions. Use built-in feature delayed replication instead.

Percona Monitoring and Management 3.0.0-Beta - Tech Preview

Ondrej Patocka — Mon, 02 Dec 2024 00:00:00 UTC

Percona Monitoring and Management 3.0.0 Beta - Tech Preview

We’re excited to announce the Tech Preview (Beta) release of Percona Monitoring and Management (PMM) 3.0.0-Beta.

This release is intended for testing environments only, as it’s not yet production-ready. The GA (General Availability) release will be available through standard channels in the upcoming months.

Release notes

To see the full list of changes, check out the 3.0.0-Beta - Tech Preview Release Notes

Installation options

PMM Server

Docker

Server: docker pull perconalab/pmm-server:3.0.0-beta
Docker installation guide

VM

PMM Client

Docker images

AMD 64 + ARM 64: docker pull perconalab/pmm-client:3.0.0-beta

Binary packages

Package Manager installation

Enable testing repository via Percona-release: percona-release enable pmm3-client testing
Install relevant pmm-client package using your system’s package manager

Percona Bug Report: October 2024

Aaditya Dubey — Mon, 25 Nov 2024 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-8057: When max_slowlog_size is set to above 4096, then it gets reset to 1073741824. This overwrites the slow log file path with a different file name, which becomes like node_name.log.000001. Due to this issue, your path defined at slow_query_log_file won`t be useful. This issue has started happening since MySQL Version 8.0.32.

E.g.:

MySQL 8.0.36 is running with the following set of configurations:

slow_query_log = ON
slow_query_log_file = /home/user/sandboxes/msb_ps8_0_36/data/slow
long_query_time = 10
max_slowlog_files = 2
max_slowlog_size = 510000000

Check the log_file path:

mysql [localhost:8036] {msandbox} ((none)) > show global variables like "%slow_query_log_file%";
+---------------------+------------------------------------------------------------+
| Variable_name | Value |
+---------------------+------------------------------------------------------------+
| slow_query_log_file | /home/adi/sandboxes/msb_ps8_0_36/data/localhost.log.000001 |
+---------------------+------------------------------------------------------------+
1 row in set (0.00 sec)

Interestingly, you will see that this file,/home/user/sandboxes/msb_ps8_0_36/data/localhost.log, 000001, was not even created.

user@localhost:~/sandboxes/msb_ps8_0_36/data$ ll | grep slow
-rw-r----- 1 adi adi 355518891 Aug 13 16:45 localhost-slow.log
-rw-r----- 1 adi adi 1079653421 Jul 30 18:13 localhost-slow.log.old
-rw-r----- 1 adi adi 255 Aug 13 16:48 slow
-rw-r----- 1 adi adi 255 Aug 13 16:45 slow.000001

After removing max_slowlog_size = 510000000

mysql [localhost:8036] {msandbox} ((none)) > show global variables like "%slow_query_log_file%";
+---------------------+--------------------------------------------+
| Variable_name | Value |
+---------------------+--------------------------------------------+
| slow_query_log_file | /home/adi/sandboxes/msb_ps8_0_36/data/slow |
+---------------------+--------------------------------------------+
1 row in set (0.01 sec)

Reported Affected Version/s: 5.7.36-39, 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.39-30, 8.4.2-2

Workaround/Fix: Use “set global slow_query_log_file =’’;”

PS-9214: INPLACE ALTER TABLE might fail with a duplicate key error if concurrent insertions occur; there have been many bugs reported here and in MySQL bugs regarding duplicate key errors while doing an online alter table operation on tables with primary and unique keys indexes. The bug is not as easy to reproduce but involves ONLY the primary key and includes an atomic sequence that cannot create a duplicate. It seems to be related to page splits/merges.

Reported Affected Version/s: 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.39-30, 8.4.2-2

Upstream Bug: 115511

Workaround/Fix: Use ALTER TABLE … ALGORITHM=COPY instead.

PS-9275: When querying based on a function, MySQL does not use the available functional index when using the LIKE operator, which results inconsistent query plans when functional Indexes are used.

E.g.:

CREATE TABLE `test` (
 `id` int NOT NULL AUTO_INCREMENT,
 `a` varchar(200) DEFAULT NULL,
 `test02` varchar(9) GENERATED ALWAYS AS (monthname(from_unixtime(`a`))) VIRTUAL,
 `!hidden!test01!0!0` varchar(9) GENERATED ALWAYS AS (monthname(from_unixtime(`a`))) VIRTUAL,
 PRIMARY KEY (`id`),
 KEY `test01` ((monthname(from_unixtime(`a`)))),
 KEY `test02` (`test02`)
) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

mysql> explain select MONTHNAME(FROM_UNIXTIME(a)) from test WHERE MONTHNAME(FROM_UNIXTIME(a)) Like 'April%';
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
| 1 | SIMPLE | test | NULL | ALL | NULL | NULL | NULL | NULL | 13 | 100.00 | Using where |
+----+-------------+-------+------------+------+---------------+------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0,00 sec)

mysql> explain select MONTHNAME(FROM_UNIXTIME(a)) from test WHERE `!hidden!test01!0!0` Like 'April%';
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
| 1 | SIMPLE | test | NULL | range | test01 | test01 | 39 | NULL | 2 | 100.00 | Using where |
+----+-------------+-------+------------+-------+---------------+--------+---------+------+------+----------+-------------+
1 row in set, 1 warning (0,00 sec)

Reported Affected Version/s: 8.0.36-28, 8.4.X

Upstream Bug: 104713

Workaround/Fix: Use the indexes created on virtual fields explicitly.

PS-9286: KMIP Component leaves keys in a pre-active state.

Reported Affected Version/s: 8.0.X, 8.3.0-1, 8.4.0-1

Fixed Version: PS 8.0.39-30, 8.4.2-2

PS-9314: The database crashed due to the SELECT statement. Since the JSON is invalid, the command should return ERROR 3146, an Invalid data type for JSON, but unfortunately, it crashed the instance with Signal 11 using JSON_TABLE.

Reported Affected Version/s: 8.0.36-28, 8.0.37-29, 8.0.39-30

Fixed Version: PS 8.0.39-30, 8.4.2-2

E.g.:

mysql> show global variables like 'version%';
+-------------------------+-----------------------------------------------------+
| Variable_name | Value |
+-------------------------+-----------------------------------------------------+
| version | 8.0.36-28 |
| version_comment | Percona Server (GPL), Release 28, Revision 47601f19 |
| version_compile_machine | x86_64 |
| version_compile_os | Linux |
| version_compile_zlib | 1.2.13 |
| version_suffix | |
+-------------------------+-----------------------------------------------------+
6 rows in set (0.01 sec)

mysql> SELECT ele AS domain FROM JSON_TABLE('["TEST'+(select load_file('test'))+'"]', "$[*]" COLUMNS (ele VARCHAR(70) PATH "$" )) AS json_elements ;
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (111)
ERROR:
Can't connect to the server

PS-9369: The audit plugin causes memory exhaustion after a few days; disconnecting threads and disabling the audit plugin is undesirable. This workaround can not be used since it requires scheduling an application outage. Even when small, it’s a recurrent event.

Reported Affected Version/s: 8.0.37-29

Fixed Version: PS 8.0.40-31 [Yet to Release]

Percona Xtradb Cluster

PXC-4453: In 3 Node PXC cluster, node01 has active flow control(FC). Active FC blocks user sessions to insert a message into the channel queue (session waits on send monitor (conn->sm)); send monitor is blocked because FC is active. The idea behind the logic is that applier threads, when consuming messages from the queue conn->recv_q, should check if FC is active, and if the queue level is below conn->lower_limit, FC should be disabled, and the user connection thread waiting on the sending monitor should be woken up. In other words, disabling the FC signal is driven by the consumption of events from recv_q by applier threads.

In this case, it seems that recv_q is empty, but FC is active, so nothing can be added to recv_q. We have a vicious circle of some kind of deadlock, and due to this race condition, we are seeing cluster hangs.

Reported Affected Version/s: 5.7.25, 5.7.44, 8.0.36-28

Fixed Version: PXC 8.0.37-29, 8.4.0

PXC-4404: wsrep_preordered=ON causes protocol violations, which cause a node to crash when the group view changes on a cluster with a node acting as an async replica.

Reported Affected Version/s: 5.7.44-31

Workaround/Fix: Set wsrep_preordered=OFF; however, you may experience a delay in async replication.

Note: Option wsrep_preordered is deprecated in MySQL-wsrep: 8.0.19-26.3, MariaDB: 10.1.1

PXC-4362: The PXC node evicted when creating a function by the user doesn`t have the super privilege, and binary logging is enabled.

Reported Affected Version/s: 8.0.34-26

Fixed Version: PXC 8.0.36-28, 8.4.0

Workaround/Fix: Setting log_bin_trust_function_creators is the workaround. Note that log_bin_trust_function_creators is deprecated by MySQL 8.0.34 and will be removed in the future.

PXC-4365: PXC nodes leave clusters when the row size is too large and have more than 3 nvarchar columns.

Reported Affected Version/s: 8.0.35-27

Fixed Version: PXC 8.0.36-28, 8.3.0

Percona Toolkit

PT-2325: pt-table-sync does not produce the correct SQL statements to sync tables containing JSON columns properly.

E.g.:

pt-table-sync emits the following SQL:

DELETE FROM `test`.`test_to` WHERE `id`='2' AND `data`='{"baz": "quux"}' LIMIT 1;

INSERT INTO `test`.`test_to`(`id`, `data`) VALUES ('1', '{"foo": "bar"}');

The INSERT statement works fine, but the DELETE fails to delete the row with id=‘2’, because the AND data=’{“baz”: “quux”}’ portion of the WHERE clause will result in the query matching zero rows.

Verify the incorrect contents of the test_to table with the following:

# Examine the state of our test tables.
$ docker exec -it mysql_5_7_12_test mysql -utest -ptest -e "
 use test;
 select * from test_to;"

That should return the following output:

+------+-----------------+
| id | data |
+------+-----------------+
| 2 | {"baz": "quux"} |
| 1 | {"foo": "bar"} |
+------+-----------------+

Witness that the row with id=2 still exists in the table and was not deleted as it should have been. With JSON columns, the DELETE statement would need to look like this:

DELETE FROM `test`.`test_to`
WHERE `id`='2'
AND `data`=CAST('{"baz": "quux"}' AS JSON)
LIMIT 1;

Reported Affected Version/s: 3.5.7

PT-2329: During the run, pt-archiver will ignore columns that are camelCase during the insert, but it will get all the columns during select.

It could be confirmed by using a dry-run:

pt-archiver --source [...] --dest [...] --where "1=1" --statistics --progress=10000 --limit=1000 --no-delete --no-safe-auto-increment --no-check-columns --columns=addressLine1,addressLine2,city,state,postalCode,country,customerNumber --why-quit --skip-foreign-key-checks --dry-run

Here are the results:

SELECT /*!40001 SQL_NO_CACHE */ `addressLine1`,`addressLine2`,`city`,`state`,`postalCode`,`country`,`customerNumber`,`customernumber` FROM `classicmodels`.`customers` FORCE INDEX(`PRIMARY`) WHERE (1=1) ORDER BY `customernumber` LIMIT 1000

SELECT /*!40001 SQL_NO_CACHE */ `addressLine1`,`addressLine2`,`city`,`state`,`postalCode`,`country`,`customerNumber`,`customernumber` FROM `classicmodels`.`customers` FORCE INDEX(`PRIMARY`) WHERE (1=1) AND ((`customernumber` > ?)) ORDER BY `customernumber` LIMIT 1000

INSERT INTO `classicmodels`.`addresses`(`city`,`state`,`country`) VALUES (?,?,?)

Reported Affected Version/s: 3.5.7

Workaround/Fix: The solution is to include all columns in lowercase in the param –columns until the bug is fixed.

PT-2344: pt-config-diff compares mysqld options, but it fails if the [mysqld] section is in uppercase, even though that is a valid way of setting mysqld variables. Since [MYSQLD] is acceptable for MySQL, pt-config-diff should compare the options under that section.

Reported Affected Version/s: 3.5.7

Workaround/Fix: Use [mysqld] as lowercase until the bug is fixed.

PT-2355: Table data is lost if we accidentally resume a previously failed job that has null boundaries. pt-online-schema-change should not resume a job with empty boundaries.

Reported Affected Version/s: 3.6.0

Fixed Version: PT 3.7.1

Workaround/Fix: Do not run pt-online-schema-change with job id having null boundaries.

PT-2356: If you run pt-online-schema-change, which results in an error, then subsequent runs will create new tables that won`t be cleaned up.

Reported Affected Version/s: 3.6.0

PT-2349: pt-table-sync is failing to sync data from PXC to the async environment, and trigger errors include “WSREP detected deadlock/conflict and aborted the transaction.”

Reported Affected Version/s: 3.3.1, 3.5.2, 3.6.0

PT-1726: pt-query-digest is not distinguishing queries when an alias is used

Reported Affected Version/s: 3.6.0

E.g.:

Queries from slow query log:

Time: 2019-01-31T11:00:00.728957Z
# User@Host: sageone_ext_uk[sageone_ext_uk] @ [10.181.130.22] Id: 18714290
# Query_time: 2.709699 Lock_time: 0.000402 Rows_sent: 19 Rows_examined: 51011
use sageone_ext_uk;
SET timestamp=1548932400;
SELECT a,b,c from table1 as t1 where t1.a=3 and t1.b=5;

# Time: 2019-01-31T11:00:00.728957Z
# User@Host: sageone_ext_uk[sageone_ext_uk] @ [10.181.130.22] Id: 18714290
# Query_time: 2.709699 Lock_time: 0.000402 Rows_sent: 19 Rows_examined: 51011
use sageone_ext_uk;
SET timestamp=1548932400;
SELECT a,b,c from table1 as t1 where t1.a=3 and t1.c=5;

The fingerprints for the above queries are the same, which is incorrect behaviour:

select a,b,c from table? as t? where t?=? and t?=?

PT-2374: If we say –ignore=bob, every combination of the bob user will be ignored. This includes bob@localhost, bob@::1, bob@foobar, etc. But this is not the case. Only bob@% is ignored; pt-show-grants –ignore does not ignore all accounts.

Reported Affected Version/s: 3.6.0

PT-2375: When pt-table-sync is used on a table with a GENERATED AS column, it fails because we cannot REPLACE/INSERT values into a GENERATED column.

E.g.:

`requestStatus` tinyint(1) GENERATED ALWAYS AS (if((`provRequired` = 0),(`httpSyncStatus` not between 200 and 299),(`httpAsyncStatus` not between 200 and 299))) VIRTUAL,

ERROR 3105 (HY000): The value specified for generated column 'requestStatus' in table 'qqq' is not allowed.

The —-ignore-columns parameter specifically states that if a REPLACE/INSERT is needed, all columns will be used. Due to this, the pt-table-sync does not work with the generated columns.

Reported Affected Version/s: 3.6.0

PMM [Percona Monitoring and Management]

PMM-12013: If we add many RDS instances to the PMM server, say 200+, and Change the prom scrape.maxScrapeSize to the value that allows the VM to parse the reply from the exporter, then the metrics are gathered unreliably, there are gaps, and the exporter`s RSS feed goes to like 5GB for instance. This concludes that rds_exporter is unreliable for large deployments.

Reported Affected Version/s: 2.35.0

Fixed Version: PMM 3.0.0-Beta available as Tech Preview]

PMM-12161: In the Mongodb cluster summary page, Under QPS of Config Services dashboard, it is being clubbed configRS, mongoS and mongod servers. This results in too many configuration services under the QPS of the Config Services dashboard.

Reported Affected Version/s: 2.42.0

Fixed Version: PMM 3.1 [Yet to Release]

PMM-12993: In PMM, CPU metrics have a label “mode” to identify between CPU info: sys, iowait, nice, user, idle, etc. With 1 rds instance, the metric is perfectly fine. However, after adding more instances, the CPU metric is still collected, but the “mode” label is empty, which breaks the graphs in the Advanced Data Exploration dashboard.

Reported Affected Version/s: 2.41.1, 2.41.2

PMM-13148: If we run the queries without using the schema name, then we don`t see such queries in the QAN.

E.g.:

mysql [localhost:8036] {msandbox} ((none)) > update test.joinit set g=100,t="06:44:50" where i=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

Here, we can see we did not explicitly select the database name using the USE command and executed the query directly. This results in QAN not being able to capture such queries for analytics.

Reported Affected Version/s: 2.41.2

Workaround/Fix: Run queries with USE ; ;

PMM-13252: A 500 error message is returned while creating the role with the existing name.

E.g.:

Enable Access roles in PMM Settings
Open the Access role page and create a role with the name “Test. “
Try to create a new role with the name “Test. “

It returns with Internal server error 500:

in logs: msg="RPC /accesscontrol.v1beta1.AccessControlService/CreateRole done in 1.409839ms with unexpected error: pq: duplicate key value violates unique constraint "roles_title_key""

Reported Affected Version/s: 2.42.0

Workaround/Fix: It is expected to be fixed in PMM 3

PMM-13277: When we try to launch PMM using AWS AMI as mentioned in our docs. However, the AWS webpage works fine, and it logins, but every graph and details are blank with “Server error 502” The same can be seen in the log for Victoria metrics:

The following error will be seen:

2024-07-27T06:40:22.848Z panic /home/builder/rpm/BUILD/VictoriaMetrics-pmm-6401-v1.93.4/lib/mergeset/part_header.go:88 FATAL: cannot read "/srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json": open /srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json: no such file or directory
panic: FATAL: cannot read "/srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json": open /srv/victoriametrics/data/indexdb/17D6772949F4A234/17D6772B9FDF298D/metadata.json: no such file or directory

Reported Affected Version/s: 2.42.0

Fixed Version: PMM 2.43.0

Percona XtraBackup

PXB-3302: If the number of GTID sets is absolutely large on a MySQL instance, the output “GTID of the last change” in the Xtrabackup log is truncated compared to the full output in xtrabackup_binlog_info and xtrabackup_info. This can be an issue for external tools obtaining the GTID coordinates from the log as it would be impractical to get the coordinates from xtrabackup_binlog_info or xtrabackup_info on a large, compressed xbstream file.

Here is a snippet of a backup log:

2024-06-03T10:18:59.678581+08:00 0 [Note] [MY-011825] [Xtrabackup] MySQL binlog position: filename 'mysql-bin.000002', position '10197', GTID of the last change ** REDACTED **,9f18624b-214f-11ef-871f-b445068273a0:1,9f1bf5ae-214f-11ef-871f-b445068273a0:1,9f1f6fac-214f-11ef-871f-b445068273a0:1,9f231076-214f-11ef-871f-b445068273a0:1,9f26d153-214f-11ef-871f-b445068273a0:1,9f2a5fdd-214f-11ef-871f-b445068273a0:1,9f2df6e8-214f-11ef-871f-b445068273a0:1,9f318143-214f-11ef-871f-b445068273a0:1,9f353351-214f-11ef-871f-b445068273a0:1,9f38f96c-214f-11ef-871f-b445068273a0:1,9f3cdc53-214f-11ef-871f-b445068273a0:1,9f40fcc0-214f-11ef-871f-b445068273a0:1,9f44955a-214f-11ef-871f-b445068273a0:1,9f481188-214f-11ef-871f-b445068

Snippet of xtrabackup_binlog_info:

mysql-bin.000002 10197 ** REDACTED **,9fdd9048-214f-11ef-871f-b445068273a0:1,9fe138a3-214f-11ef-871f-b445068273a0:1,9fe4c3d8-214f-11ef-871f-b445068273a0:1,9fe82e39-214f-11ef-871f-b445068273a0:1

Reported Affected Version/s: 8.0.35-30

Fixed Version: PXB 8.4.0-1, 8.0.35-32

PXB-3283: When xtrabackup takes a backup and exports a tablespace, xtrabackup gets the wrong table definition from the ibd for tables that have changed the charset-collation in MySQL before backup.

Eg:

CREATE TABLE test.a (
 a datetime DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_as_ci;

the collation_id is 8 (latin1_swedish_ci)

shell> ibd2sdi /var/lib/mysql/test/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 8

When MySQL converts the charset on a table, it converts the date and time data types columns in the ibd file but not the data dictionary cache. The collation in the ibd does not match that of the data dictionary.

ALTER TABLE test.a CONVERT TO CHARACTER SET utf8mb4 collate utf8mb4_unicode_ci;

The collation_id becomes 224 (utf8mb4_unicode_ci)

shell> ibd2sdi /var/lib/mysql/test/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 224

The collation_id of the copied table is 8 (latin1_swedish_ci)

create table xb.a like test.a;

shell> ibd2sdi /var/lib/mysql/xb/a.ibd | jq '.[1].object.dd_object.columns[0]' | grep collation_id
 "collation_id": 8

When xtrabackup exports the tablespace, the collation_id is 224 in ibd. Xtrabackup will write it to cfg metadata file.

When MySQL imports a tablespace, MySQL gets an error Column %s precise type mismatch because the collation_id of MySQL does not match that of xtrabackup.

Reported Affected Version/s: 8.0.35-30

Fixed Version: PXB 8.4.0-1, 8.0.35-31

PXB-2797: When importing a single table (IMPORT TABLESPACE) from a backup made using xtrabackup and the table contains a full-text index, the import process will error out with:

ERROR 1808 (HY000) at line 132: Schema mismatch (Index xxxxxx field xxxxxx is ascending which does not match metadata file which is descending)

Reported Affected Version/s: 8.0.28-20

Fixed Version: PXB 8.0.35-31

PXB-3210: PXB fails to build on macOS since 8.0.33-28 due to FIND_PROCPS()

CMake Error at cmake/procps.cmake:29 (MESSAGE):
 Cannot find proc/sysinfo.h or libproc2/meminfo.h in . You can pass it to
 CMake with -DPROCPS_INCLUDE_PATH= or install
 procps-devel/procps-ng-devel/libproc2-dev package
Call Stack (most recent call first):
 storage/innobase/xtrabackup/src/CMakeLists.txt:24 (FIND_PROCPS)

Reported Affected Version/s: 8.0.33-28, 8.0.34-29, 8.0.35-30

Fixed Version: PXB 8.0.35-31

PXB-3130: Performing upgrade from PS 8.0.30 -> PS 8.0.33 using PXB

Use PXB 8.0.30 on PS 8.0.30
Copy to new host
Prepare using PXB 8.0.33
Start PS 8.0.33

Which results in the Assertion:

I0825 22:33:01.738917 05155 ???:1] xtrabackup80-apply-log(stderr) - InnoDB: Assertion failure: log0recv.cc:4353:log.m_files.find(recovered_lsn) != log.m_files.end()

Reported Affected Version/s: 8.0.30

Fixed Version: PXB 8.0.35-31

Percona Kubernetes Operator

K8SPXC-1398: Scheduled PXC backup job pod fails to complete the process successfully in a random/sporadic fashion.

Error Returns As:

+ EXID_CODE=4
+ '[' -f /tmp/backup-is-completed ']'
+ log ERROR 'Backup was finished unsuccessfull'
Terminating processProcess completed with error: /usr/bin/run_backup.sh: 4 (Interrupted system call)2024-05-03 09:39:08 [ERROR] Backup was finished unsuccessfull
+ exit 4

Reported Affected Version/s: 1.13.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

Note: Since we don`t have steps to reproduce the issue, it is hard to confirm whether the fix is working as expected. Please feel free to provide feedback or create a Jira if required.

K8SPXC-1397: The operator`s default configuration makes the cluster unusable if TDE (Transparent data encryption) is used; the entry point of the PXC container configures the parameter binlog_rotate_encryption_master_key_at_startup. As a workaround, binlog_rotate_encryption_master_key_at_startup should be disabled. However, it has security implications.

Reported Affected Version/s: 1.12.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

K8SPXC-1222: Upgrading Cluster Fails When Dataset Has Large Number Of Tables. When the operator replaces the first pod with one with the new version, it fails to start up and gets stuck in a loop that restarts every 120 seconds.

The problem looks like from pxc-entrypoint.sh:

for i in {120..0}; do
 if echo 'SELECT 1' | "${mysql[@]}" &>/dev/null; then
 break
 fi
 echo 'MySQL init process in progress...'
 sleep 1
 done

Reported Affected Version/s: 1.11.0, 1.12.0

Fixed Version: PXCO 1.16.0 [Yet to Release]

Orchestrator

DISTMYSQL-406: Orchestrator 3.2.6-11 shows the MySQLOrchestratorPassword variable value in the error log and when accessing the web interface.

E.g.:

Create a MySQL and Orchestrator node

mysql -e "CREATE USER 'orchestrator_srv'@'%' IDENTIFIED BY 'orc_server_password'; GRANT ALL ON orchestrator.* TO 'orchestrator_srv'@'%';"

Configure Orchestrator to use node0 as MySQL backend database

vi /usr/local/orchestrator/orchestrator.conf.json

Add the following lines and remove sqlite options:

 "MySQLOrchestratorHost": "node_0_IP",
 "MySQLOrchestratorPort": 3306,
 "MySQLOrchestratorDatabase": "orchestrator",
 "MySQLOrchestratorUser": "orchestrator_srv",
 "MySQLOrchestratorPassword": "orc_server_password",

On node1, there are several messages showing the backend password:

Feb 28 23:26:03 XX-XX-node1 orchestrator[4262]: 2024-02-28 23:26:03 ERROR 2024-02-28 23:26:03 ERROR QueryRowsMap(orchestrator_srv:orc_server_password@tcp(10.124.33.138:3306)/orchestrator?timeout=1s&readTimeout=30s&rejectReadOnly=false&interpolateParams=true) select hostname, token, first_seen_active, last_seen_Active from active_node where anchor = 1: dial tcp 10.124.33.138:3306: connect: connection refused

Reported Affected Version/s: 8.0.36(PS)

Fixed Version: 8.4.0(PS)

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

About Percona:

As the only provider of distributions for all three of the most popular open source databases—PostgreSQL, MySQL, and MongoDB—Percona provides expertise, software, support, and services no matter the technology.

Whether its enabling developers or DBAs to realize value faster with tools, advice, and guidance, or making sure applications can scale and handle peak loads, Percona is here to help.

Percona is committed to being open source and preventing vendor lock-in. Percona contributes all changes to the upstream community for possible inclusion in future product releases.

Percona Monitoring and Management 2.43.0 Preview Release

Ondrej Patocka — Thu, 12 Sep 2024 00:00:00 UTC

Percona Monitoring and Management 2.43.0 Tech Preview Release

Hello everyone! Percona Monitoring and Management (PMM) 2.43.0 is now available as a Tech Preview Release.

We encourage you to try this PMM preview release in testing environments only, as these packages and images are not fully production-ready. The final version is expected to be released through the standard channels in the coming week.

To see the full list of changes, check out the PMM 2.43.0 Tech Preview Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.43.0-rc

PMM client package installation

Download AMD64 or Download ARM64 pmm2-client tarball for 2.43.0.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.43.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-0db618c7da6e202f4

Timeline for Database on Kubernetes

Edith Puclla — Tue, 16 Jul 2024 00:00:00 UTC

The Evolution

Since its inception in June 2014, Kubernetes has dramatically transformed container orchestration, revolutionizing the management and scaling of applications. To mark its tenth anniversary, the Data on Kubernetes Community (DoKC) released an infographic showcasing key milestones and community contributions to the evolution of operators for managing stateful applications. This project was made possible by the collaboration of DoKC members Edith Puclla, Sergey Pronin, Robert Hodges, Gabriele Bartolini, Chris Malarky, Mark Kember, Paul Au, and Luciano Stabel.

Explore the infographic to see how Kubernetes has shaped the future of database management on Kubernetes.

Adoption and Impact

The CNCF says that 84% of organizations are using or considering Kubernetes, with 70% running stateful applications on it in production. The number of users and containers has grown, showing that more people are contributing, adopting cloud-native technologies, and finding new ways to use Kubernetes to manage stateful applications.

Looking Ahead

As we celebrate ten years of Kubernetes, the way databases are integrated keeps improving, thanks to community efforts and new technology. Percona Everest is an excellent example of this progress. It’s more than just a tool for databases; it represents the future of running databases on Kubernetes. It’s open-source and makes running any database on cloud-based Kubernetes clusters easy. If you want to try it, visit our Percona Everest GitHub Repository and give us a star if you like it. For feedback or comments, join the Percona Forum for Percona Everest discussion.

Percona Joins Community Over Code 2024 in Bratislava, Slovakia

Edith Puclla — Fri, 21 Jun 2024 00:00:00 UTC

Last week, I participated as a speaker for the first time at Community Over Code 2024.

Community Over Code

Community Over Code is a key principle at Apache, highlighting the importance of having a solid and collaborative community rather than just focusing on the code. While good code is essential, the community’s strength and resilience keep a project going and growing. I love how this is expressed in the “Apache Way.”

Renaming ApacheCon to “Community Over Code” reflects this idea, emphasizing the central role of community in Apache’s approach. This year, Community Over Code was in Bratislava, Slovakia. My talk was in the Community track, where I spoke about Outreachy Internship, the Apache Software Foundation, and Apache Airflow. My goal was to celebrate the success of the Outreachy program, which has surpassed 1,000 internships, and I am proud to be one of them.

I talked about the community, shared success stories, and provided clear examples, such as Bowrna Prabhakaran and Ephraim Anierobi, and shared my personal experience. I also explained how Outreachy, along with my mentors from Apache Airflow, Jarek Potiuk, and Elad Kalif, boosted my professional career in Open Source 03 years ago.

Working at Percona

Now I work for Percona, which has been recognized as one of Inc. Magazine’s 2024 Best Workplaces! I love my job and the amazing things we are building at Percona. Percona is an open-source software company that fully believes that open source should remain open throughout. This belief inspires me to work at Percona.

What We’re Focused on Now

If you’re interested in improving database performance on Kubernetes, you will love Percona Everest. It is our cloud-native solution for efficiently managing and running databases on Kubernetes. With a very user-friendly interface, you can run any database on any cloud provider or on-premises.

Here are some use cases where you might use Percona Everest.

Seeking ways to make the development of internal platforms easier and faster.
Looking for affordable alternatives to public Database-as-a-Service (DBaaS) offerings.
Building multi-cloud or hybrid cloud setups to meet data compliance needs for multi-regional businesses.
Wishing to leverage Kubernetes for its scalability and stability to run databases efficiently.
Transitioning from monolithic to microservices architecture to modernize legacy database infrastructure.

Any feedback is welcome on our Percona Everest forum, and if you like it, give us a start on GitHub: github.com/percona/everest.

Let's take a look at Percona Everest 1.0.0 RC

Daniil Bazhenov — Fri, 14 Jun 2024 00:00:00 UTC

Hi, the Percona Everest 1.0.0-rc1 release was published on GitHub.

Percona Everest is the first open source cloud-native platform for provisioning and managing PostgreSQL, MongoDB and MySQL database clusters.

I want to tell you how to install it so you can try it out.

RC builds aren’t meant for the general public; we don’t support upgrading from RC to stable versions. This means that this is only for testing and familiarizing yourself with the features. RC builds are not stable and are often buggy. There will be no upgrade. :)

To get started, you will need a Kubernetes cluster. Right now, Percona Everest is in Beta. Don’t use production clusters; use test clusters in the cloud like GKE or local in Minikube, k3d, or Kind.

I created a test cluster in GKE with the command:

gcloud container clusters create test-everest-rc --project percona-product --zone us-central1-a --cluster-version 1.27 --machine-type n1-standard-4 --num-nodes=3

Delete it after the test with the command:

gcloud container clusters delete test-everest-rc --zone us-central1-a

Now, we need Everest CLI for the RC version; download it from GitHub for your operating system.

I downloaded it, renamed it to everestctl, and copied it to a folder for experimentation.

Now, we need to make it executable

chmod +x ./everestctl

Let’s check that everestctl works and that we have the correct version.

./everestctl version

We can now install Percona Everest.

./everestctl install --version-metadata-url https://check-dev.percona.com

Note that I use the --version-metadata-url parameter https://check-dev.percona.com; this is required for RC builds.

During the installation process, you must set one or more namespaces and databases.

Once the installation is complete, the new user authentication feature is the first significant change. You will be offered two commands.

Command to retrieve the admin user password that was generated automatically during installation:

./everestctl accounts initial-admin-password

Command to set a new password:

./everestctl accounts set-password --username admin

Now that we know the admin user password, we can open Percona Everest in a browser. Run the following command to use kubectl port forwarding to connect to Percona Everest without exposing the service:

kubectl port-forward svc/everest 8080:8080 -n everest-system

Take a Clone it will last longer

Wayne Leutwyler — Sun, 02 Jun 2024 00:00:00 UTC

So cloning is a great subject. I mean we clone sheep, we can clone human organs in time we might be able to clone humans, but thats a topic for scientist and philosophers.

What is MySQL Replicaion:

The MySQL clone plugin can be used to replicate data from a MySQL server to another MySQL server, and it supports replication. The cloning process creates a physical snapshot of the data, including tables, schemas, tablespaces, and data dictionary metadata. It tracks replication coordinates from the source server and transfers them to the replica, which allows replication to begin at a consistent position in the replication stream. This data includes the binary log position (filename, offset) and the gtid_executed GTID set. The replication metadata repositories are also copied during the cloning operation.

The clone plugin that was released with MySQL Version 8.0.17. Its quick and very easy to setup. You can use it for so many different solutions. I’ve listed some common ones below, but I know that there are many more use cases.

Create a new replica.
Recover a replica which is out of sync with the primary.
Quickly deploy MySQL servers with data set already in place.

In this article I will cover the basics of setting up and running a clone from and existing MySQL server. I will be using MySQL version 8.0.36.

Prepare the Source Server

Install the clone plugin.
Create a clone user.

Install Clone plugin

source > INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Verify the clone plugin was installed.

source > show plugins;

Create Clone User

Now we need to create a user to run the clone process. I highly suggest you create a user for the cloning. Please don’t use an ID with full admin rights. Create a user with the least amount of privileges needed.

source > CREATE USER 'clone_user'@'%' IDENTIFIED BY 'S3k3rtPassWd';
source > GRANT BACKUP_ADMIN ON *.* TO 'clone_user'@'%';

Verify your new clone user.

source > show grants for clone_user;

Preparation on the source server is complete. Lets move on to the clone.

Prepare the Clone Server

We need to start by installing the clone plugin.
Then we need to define the source that clone will be based on.

Install plugin

clone > INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Verify the clone plugin was installed.

clone > show plugins;

Define Source Server

You can use the source host name or host IP address.

clone > SET GLOBAL clone_valid_donor_list='SOURCE_HOSTNAME:3306';

Start and monitor the cloning process

We are now ready to kick off the cloning.

clone > CLONE INSTANCE FROM clone_user@SOURCE_HOSTNAME:3306 IDENTIFIED BY 'S3k3rtPassWd';

Cloning should start now. If you have any issues, check your log files and very the steps above. Now that the cloning is running we can monitor the cloning process using this command.

clone > SELECT * FROM performance_schema.clone_progress\G

As you can see in the output below, that the cloning has completed DROP DATA and is now working on FILE COPY.

Observations

Just recently I worked with a customer who moved a 5.6TB database from a Galera Cluster to standard MySQL replication. The data was moved from the source to replicas in two different locations. Timings are detailed below.

Source to Replica in same location: 5.6TB in approxmently 50 mins.
- 112GB per minute.
Source to Replica in different geographic locations: 5.6TB in approxmently 1 hour and 45 mins.
- 53.33GB per minute.

MySQL Clone Plugin’s robust performance and efficiency in managing large data transfers, making it an excellent tool for environments requiring quick and reliable data replication.

Summary

The MySQL Clone Plugin offers several benefits, including:

Fast Data Copying:
- Enables rapid cloning of MySQL instances, facilitating quick data replication and environment setup.
Consistent Data State:
- Ensures data consistency during cloning, avoiding issues that can arise from manual copying or inconsistent states.
Reduced Downtime:
- Minimizes downtime during cloning operations, crucial for maintaining service availability.
Ease of Use:
- Simplifies the cloning process through straightforward commands, reducing the complexity for database administrators.
Automated Cloning Process:
- Automates many steps involved in the cloning process, reducing the potential for human error and increasing efficiency.

Reference

The Clone Plugin

Image by kjpargeter on Freepik

Learn PostgreSQL and SQL Quickly and Free

David Stokes — Mon, 20 May 2024 00:00:00 UTC

Learn PostgreSQL and SQL Quickly and Free

Want to learn PostgreSQL? What if you could learn PostgreSQL without having to install the database, load data, find sample data, and it was free? PGExamples.com is what you are looking for.

First, let me state that I have no connection to the website or its author. I discovered PgExercises.com by accident and was impressed by its quality and completeness. I am so impressed that I am exploring a video series on it.

Learn SQL. Learn PostgreSQL.

PGexercises.com is based on increasingly complex exercises based on a provided dataset. As you progress, the questions get trickier. You do not have to download and load the available data into a server to do the exercises.

There are seven groups of exercises - Basic, Joins and Subqueries, Modifying Data, Aggregates, Date, String, and Recursive. The author recommends Learning SQL by Alan Beaulieu as a reference text. Any essential book, Structured Query Language, will serve you well.

First Exercise

The first exercise asks you to retrieve all the data from a specific table.

At the top of the screen is an entity relationship map of the tables in the database. The table requested in the exercise is named cd.facilities.

If you use the hint button (next to the Your Answer line) you will see the following:

This should be enough of a clue to allow you, with the aid of your SQL reference, to find the answer. Enter your query and click on the Eun Query button.

The green check means that the answer is correct. But what if you did not get it correct. Scroll down and select the Answers and Discussion button.

Not only do you see the answer to the question, but there is a detailed analysis of the answer.

Video Series?

I am so impressed wth these exercises that the Percona Community Team is in discussion of creating a series of videos on them. Not only that, on of my colleagues who is learning PostgreSQL is volunteering to work through the exercises so that the series will feel like a study group. And I will comment on the query, possible alternatives when they exist, and help you work through the ‘rough spots’.

If you want to work along, please do. Let us know where you struggle or the places we are redundant. Out goal is to provide a first class learning experience to help people learn SQL and PostgreSQL.

Release Roundup May 15, 2024

David Quilty — Wed, 15 May 2024 00:00:00 UTC

Percona software releases and updates April 30 - May 15, 2024.

Percona is a leading provider of unbiased, performance-first, open source database solutions that allow organizations to easily, securely, and affordably maintain business agility, minimize risks, and stay competitive, free from vendor lock-in. Percona software is designed for peak performance, uncompromised security, limitless scalability, and disaster-proofed availability.

Our Release Roundups showcase the latest Percona software updates, tools, and features to help you manage and deploy our software. It offers highlights, critical information, links to the full release notes, and direct links to the software or service itself to download.

Today’s post includes releases and updates that have been released since April 29, 2024. Take a look.

Percona Distribution for MongoDB 6.0.15

Percona Distribution for MongoDB 6.0.15 was released on April 30, 2024. It is a freely available MongoDB database alternative that gives you a single solution that combines enterprise components from the open source community, designed and tested to work together. Please see the release notes for a full list of improvements and bug fixes.

Download Percona Distribution for MongoDB 6.0.15

Percona Server for MongoDB 6.0.15-12

On April 30, 2024, we released Percona Server for MongoDB 6.0.15-12. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 6.0.15. It is based on MongoDB 6.0.15 Community Edition and supports the upstream protocols and drivers. Please see the release notes for a full list of improvements and bug fixes.

Download Percona Server for MongoDB 6.0.15-12

Percona XtraDB Cluster 5.7.44-31.65.2

Percona XtraDB Cluster 5.7.44-31.65.2 was released on May 2, 2024. This release is part of MySQL 5.7 Post-EOL Support from Percona, and the fixes are available to MySQL 5.7 Post-EOL Support from Percona customers.

That’s it for this roundup, and be sure to follow us on Twitter to stay up-to-date on the most recent releases! Percona is a leader in providing best-of-breed enterprise-class support, consulting, managed services, training, and software for MySQL, MongoDB, PostgreSQL, MariaDB, and other open source databases in on-premises and cloud environments and is trusted by global brands to unify, monitor, manage, secure, and optimize their database environments.

How to Provision a MongoDB Cluster in Kubernetes with Percona Everest Summary

Edith Puclla — Thu, 02 May 2024 00:00:00 UTC

Kubernetes continues evolving, and the complexity of deploying and managing databases within the ecosystem is a topic of considerable discussion and importance these days. This article summarizes a detailed discussion between Piotr Szczepaniak and Diogo Recharte, who offer insights and live demonstrations to simplify database operations on Kubernetes with a new technology for cloud-native applications: Percona Everest. If you want to watch the full video, check out How to Provision a MongoDB Cluster in Kubernetes Webinar.

Peter mentions that Initially, people were doubtful about using virtual machines for databases, just like they were skeptical about Kubernetes. However, the topic brings together many people who run databases on containers to share their use cases and new discussions at events like Data on Kubernetes Day at Kubecon.

The introduction of StatefulSets and Persistent Volumes has altered the perception of Kubernetes from being purely ephemeral to being capable of handling persistent data. This change is important for database applications that require data retention over time.

The Kubernetes ecosystem is rapidly expanding. This growth is thanks to its open-source nature and the continuous addition of new functionalities, such as support for specialized hardware like GPUs, which are crucial for AI and machine learning applications.

Peter also mentioned that its complexity is the main barrier to Kubernetes adoption for databases. Organizations often need help with the layer added by Kubernetes on top of database management. Also, failure in initial attempts to integrate Kubernetes can discourage organizations from further attempts, primarily due to a lack of internal expertise.

Benefits of Database as a Service (DBaaS)

DBaaS significantly reduces the time required for database provisioning, which is particularly useful in organizations needing rapid deployment. Public and private DBaaS solutions offer scalability, which is crucial for handling varying workloads and organizational growth without compromising performance.

Private vs. Public DBaaS

Private DBaaS offers more extensive customization options and control over databases, which is essential for companies with specific needs that public solutions cannot meet.

Data security and compliance with regulations are more manageable in a private DBaaS because it operate within the company’s internal infrastructure.

Demo to Deploying MongoDB on Kubernetes

Diogo presented a demo of deploying a MongoDB database using Percona’s Everest platform on Kubernetes, where he showed how to handle daily operations and disaster recovery scenarios efficiently. Watch the Percona Everest Demo on YouTube

The session explained how Kubernetes operators and custom resources help manage databases more easily. They do this by simplifying complex processes and automating regular tasks.

Some questions that users asked in this presentation are:

How do we handle the PV when the Pods go down?

The PV will remain in place; this is a standard functionality of a stateful set. After the Pod goes down, the replacement Pod will attach to the PVC, which is standard behavior for a stateful set.

What happens if the node in Kubernetes goes down?

It depends on the storage layer that you have configured in your cluster. If the storage class you are using is tied to that node, then placing it on a new node will provision a new one, and some reconciliation will occur within the database itself.

What is the current state of Percona Everest?

Percona Everest is currently in a Beta stage, and Percona aims to release a GA version. The project is fully open source, and anyone can join our project on GitHub. We appreciate feedback from the community.

Do you want to send us feedback or contribute in this cool project? We are completely open-source, you can visit Percona Everest on GitHub.

Using ProxySQL Query Mirroring to test query performance on a new cluster

Isobel Smith — Wed, 01 May 2024 00:00:00 UTC

ProxySQL is an SQL aware proxy, which gives DBA’s fine grained control over clients’ access to the MySQL cluster. A key part of our DBA team’s process in testing and preparing for major MySQL version upgrades is comparing query plans using ProxySQL query mirroring. This feature allows us to mirror queries to another cluster / host, by configuring query rules. What makes mirroring particularly useful is the ability to selectively mirror queries based on the query digest, or client user. Results from the queries that are mirrored are not returned to the client, and are sent to /dev/null.

Before configuring ProxySQL for Query Mirroring, ensure that the clients that you want to mirror the queries for, are able to connect to both the current, and the new cluster. You should also ensure that the ProxySQL monitor can connect to the new cluster, otherwise ProxySQL will mark the new hosts as offline, and the queries will not be mirrored there.

To set up query mirroring in ProxySQL:

In order to set up query mirroring, you need to add the new hosts into the mysql_server table in ProxySQL. This is how the current mysql_servers table looks, before we add the new host that we want to mirror the queries to:

MySQL> SELECT hostgroup_id, hostname FROM mysql_servers;
+--------------+--------------+
| hostgroup_id | hostname |
+--------------+--------------+
| 10 | 10.12.0.123 |
| 20 | 10.12.0.123 |
| 20 | 10.16.0.456 |
| 20 | 10.16.0.789 |
+--------------+--------------+
4 rows in set (0.01 sec)

It is important to choose a hostgroup_id that is not yet in use. You can double check the currently configured host groups in the mysql hostgroups table, as you do not want to inadvertently add the mirror hosts into the production traffic!

MySQL> select * from mysql_replication_hostgroups;
+------------------+------------------+------------+----------------------+
| writer_hostgroup | reader_hostgroup | check_type | comment |
+------------------+------------------+------------+----------------------+
| 10 | 20 | read_only | Async Cluster |
+------------------+------------------+------------+----------------------+
1 row in set (0.00 sec)

Please note, in our example, we are using async replication, so we check the mysql_replication_hostgroups table, but the hostgroups table you need to check, depends on the cluster architecture you are using:

async replica clusters check the mysql_replications_hostgroups table.
galera clusters check the mysql_galera_hostgroups table
group replication check the mysql_group_replication_hostgroups table.

We are using hostgroup 10 for the writer hostgroup, and hostgroup 20 for the reader. For this example, we will choose 100 for the mirror hostgroup_id. Once you have decided on an unused hostgroup ID, add the new clusters’ nodes to the mysql_servers table in ProxySQL.

MySQL> INSERT INTO mysql_servers(host, hostgroup, comment) VALUES ("10.12.0.987", 100, "mirror_cluster");
LOAD MYSQL SERVERS TO RUN;
SAVE MYSQL SERVERS TO DISK;

The mysql_servers table will now include the new host:

MySQL> SELECT hostgroup_id, hostname FROM mysql_servers;
+--------------+--------------+
| hostgroup_id | hostname |
+--------------+--------------+
| 10 | 10.12.0.123 |
| 20 | 10.12.0.123 |
| 20 | 10.16.0.456 |
| 20 | 10.16.0.789 |
| 100 | 10.12.0.987 |
+--------------+--------------+
4 rows in set (0.01 sec)

In order to enable query mirroring, you need to update the mirror_hostgroup column in the mysql_query_rules table. When mirroring is not enabled, the value of the mirror_hostgroup column is NULL. Our query rules before enabling query mirroring are defined as:

MySQL> select rule_id, username, match_digest, destination_hostgroup, mirror_hostgroup from mysql_query_rules;
+---------+------------------------+---------------------+-----------------------+------------------+
| rule_id | username | match_digest | destination_hostgroup | mirror_hostgroup |
+---------+------------------------+---------------------+-----------------------+------------------+
| 1 | myApplicationUser | ^SELECT.*FOR UPDATE | 10 | NULL |
| 2 | myApplicationUser | ^SELECT | 20 | NULL |
+---------+------------------------+---------------------+-----------------------+------------------+
2 rows in set (0.00 sec)

To enable mirroring, we just need to update the mirror_hostgroup. For this example, we will mirror all the SELECT queries made by myApplicationUser:

UPDATE mysql_query_rules SET mirror_hostgroup = 100 where rule_id=2;
LOAD mysql query rules TO RUN;
SAVE mysql query rules TO DISK;

The rules should now be updated:

MySQL> select rule_id, username, match_digest, destination_hostgroup, mirror_hostgroup from mysql_query_rules;
+---------+-----------------------+---------------------+-----------------------+------------------+
| rule_id | username | match_digest | destination_hostgroup | mirror_hostgroup |
+---------+-----------------------+---------------------+-----------------------+------------------+
| 1 | myApplicationUser | ^SELECT.*FOR UPDATE | 10 | NULL |
| 2 | myApplicationUser | ^SELECT | 20 | 100 |
+---------+-----------------------+---------------------+-----------------------+------------------+
2 rows in set (0.00 sec)

The incoming queries that match the query rule, (in our example above, this is all queries as matching the regular expression ‘^SELECT’, for myApplicationUser, excluding queries matching ‘^SELECT.*FOR UPDATE’), will now be mirrored to the new cluster. You can verify this by checking the MySQL processlist on the new cluster.

The stats_mysql_query_digest table on ProxySQL holds statistics for the queries that are being processed by ProxySQL. To use the stats_mysql_query_digest table, the global variables mysql-commands_stats and mysql-query_digests must be set to true, which is the default.

Comparing query performance between two clusters

Query the stats_mysql_query_digest table to compare the performance per query between the current and the new cluster:

MySQL> select
 (b.count_star+a.count_star)/2 as count,
 cast(round(((b.sum_time + 0.0)/(b.count_star + 0.0))/((a.sum_time + 0.0)/(a.count_star + 0.0)),2)*100 as int) as percent,
 cast(round(((b.sum_time + 0.0)/(b.count_star + 0.0))/((a.sum_time + 0.0)/(a.count_star + 0.0)),2)*100 as int)*(b.count_star+a.count_star)/2 as load ,
 substr(a.digest_text,1,150)
from
 stats_mysql_query_digest a
inner join
 stats_mysql_query_digest b on
 a.digest = b.digest
where
 a.hostgroup = 10
 and b.hostgroup = 100
order by
 percent ASC;

In this example, the current production cluster has hostgroup 10, and the new mirror cluster was assigned hostgroup 100. The queries with a percentage above 100 are the queries that perform slower on the new cluster, and may be worth investigating, while the queries with a percentage below 100 are more performant on the new cluster. To investigate queries, you can compare the EXPLAIN plan of the query on the current and the new cluster. We use PMM Query Analytics to compare query analytics and the explain plan of the queries on the two separate clusters.

It is worth noting, that you should allow enough time for the MySQL buffer pool to get filled, before checking the stats_mysql_query_digest table. Otherwise, the query times on the new cluster can be skewed, as the active dataset may not yet be in memory (whereas on the current cluster it might be). Also, keep in mind that if you are mirroring only a subset of queries, the load on the new cluster will be different to the current cluster, and could affect the query performance on the new cluster, so that they appear significantly faster. Checking the execution plan of the query to see whether it has changed, is therefore more important than looking at overall load.

To conclude, using query mirroring to test queries on a new system, before making the migration, allows you to compare latency and query plans per normalised query, and proactively detect any necessary alterations before switching live traffic to the new cluster.

Release Roundup April 30, 2024

David Quilty — Tue, 30 Apr 2024 00:00:00 UTC

Percona software releases and updates April 17 - April 30, 2024.

Today’s post includes releases and updates that have been released since April 15, 2024. Take a look.

Percona Distribution for MySQL 8.3.0-1 (PS-based variant)

Percona Distribution for MySQL 8.3.0-1 (PS-based variant) was released on April 16, 2024. This release is based on Percona Server for MySQL 8.3.0-1 and merges the MySQL 8.3 code base. It introduces the following changes:

Percona updates the Binary Log UDFs to make them compatible with new tagged GTIDs (Global Transaction Identifiers).
PS-9044: Adds the following variables to MyRocks:
Changes the default values for the following variables:
- rocksdb_compaction_sequential_deletes from 0 to 14999
- rocksdb_compaction_sequential_deletes_count_sd from OFF to ON
- rocksdb_compaction_sequential_deletes_window from 0 to 15000
- rocksdb_force_flush_memtable_now from ON to OFF
- rocksdb_large_prefix from OFF to ON

Download Percona Distribution for MySQL 8.3.0-1 (PS-based variant)

Percona Server for MySQL 8.3

On April 16, 2024, we released Percona Server for MySQL 8.3. It includes all the features and bug fixes available in the MySQL 8.3 Community Edition in addition to enterprise-grade features developed by Percona. This release merges the MySQL 8.3 code base. Within this merge, Percona updates the Binary Log UDFs to make them compatible with new tagged GTIDs (Global Transaction Identifiers).

Download Percona Server for MySQL 8.3

Percona Distribution for MongoDB 7.0.8

Percona Distribution for MongoDB 7.0.8 was released on April 24, 2024. It is a freely available MongoDB database alternative that gives you a single solution that combines enterprise components from the open source community, designed and tested to work together. Bug fixes and improvements provided by MongoDB are included in Percona Distribution for MongoDB. Note: a number of issues with sharded multi-document transactions in sharded clusters of 2 or more shards have been identified that result in returning incorrect results and missing reads and writes. The issues occur when the transactions’ metadata is being concurrently modified by using the following operations: moveChunk, moveRange, movePrimary, renameCollection, drop, and reshardCollection. Please check the release notes for further information.

Download Percona Distribution for MongoDB 7.0.8

Percona Server for MongoDB 7.0.8-5

Percona Server for MongoDB 7.0.8-5 was released on April 24, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 7.0.8. A number of issues with sharded multi-document transactions in sharded clusters of 2 or more shards have been identified that result in returning incorrect results and missing reads and writes. The issues occur when the transactions’ metadata is being concurrently modified by using the following operations: moveChunk, moveRange, movePrimary, renameCollection, drop, and reshardCollection. Please check the release notes for further information.

Download Percona Server for MongoDB 7.0.8-5

Percona Bug Report: April 2024

Aaditya Dubey — Mon, 29 Apr 2024 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-9092: A query over an InnoDB table that uses a backward scan over the index occasionally might return incorrect/incomplete results when changes to the table (for example, DELETEs in another or even the same connection followed by asynchronous purge) cause concurrent B-tree page merges.

Reported Affected Version/s: 5.7.44, 8.0.35, 8.0.36

Upstream Bug: 114248

Workaround/Fix: Use descending indexes for the primary key. E.g.:

CREATE TABLE bugTest.testTable (key int unsigned, version bigint unsigned, rowmarker char(3) not null default 'aaa', value MEDIUMBLOB, PRIMARY KEY (key DESC, version DESC)) Engine=InnoDB;

PS-9107: A delete/insert into the secondary index is being change-buffered. This causes an insert into the ‘ibuf’ tree. There is a limit on the maximum size of the ibuf. So, on every ibuf insert, there is a compaction of ibuf tree (ibuf_contract). As part of ibuf_contract, we randomly open an ibuf page and apply the ibuf entries to the actual secondary index pages. After applying these ibuf entries from ibuf index, the ibuf tree goes on merging pages (optimistic vs pessimistic btree operations). To do this ibuf pessimistic delete on the tree, we save the cursor position, commit mtr and do a restore. This restore does a search and position again on the ibuf entry we were processing earlier, causing [MY-013183] [InnoDB] Assertion failure: ibuf0ibuf.cc:3833:ib::fatal triggered thread.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27, 8.0.36-28

Fixed Version: PS 8.0.37-29 [Yet to Release]

Upstream Bug: 114135

Workaround/Fix: Disable innodb_change_buffering

PS-9115: MySQL crashes due to getting a native index from get_mutex_cond in group replication, and before the crash, the following set of warnings/Errors is generated:

[Warning] [MY-011630] [Repl] Plugin group_replication reported: 'Due to a plugin error, some transactions were unable to be certified and will now rollback.'

[ERROR] [MY-011631] [Repl] Plugin group_replication reported: 'Error when trying to unblock non certified or consistent transactions. Check for consistency errors when restarting the service'

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is in progress and expected to be included in an upcoming release of Percona Servers.

Workaround/Fix: We can not guarantee that the bug will be avoided 100%, but shutting down/stopping group replication off-hours when the workload recedes should prevent the situation.

PS-9121: InnoDB updates the primary index but not the spatial index, which eventually corrupts the spatial index. The MySQL server crashes with “[ERROR] [MY-013183] [InnoDB] Assertion failure: row0ins.cc:268:!cursor->index->is_committed().” The update query changes the data from point(0.0000000000000099,0) to point(0.00000000000001, 0). When Innodb updates the record containing a spatial index, It updates the clustered index. This issue can be repeated using the following set of SQL statements.

CREATE TABLE a

 (

 id INT PRIMARY KEY,

 a GEOMETRY NOT NULL,

 SPATIAL KEY (a)

 )

ENGINE=InnoDB;

INSERT INTO a VALUES (1,POINT(0.0000000000000099, 0));

UPDATE a SET a = Point(0.00000000000001, 0);

DELETE FROM a WHERE id = 1;

INSERT INTO a VALUES (1,POINT(0.0000000000000099, 0));

Reported Affected Version/s: 8.0.36-28, 8.X [Innovative Release]

Fixed Version: The fix is in progress and expected to be included in an upcoming release of Percona Servers.

Upstream Bug: 114252

Workaround/Fix: Please consider resetting the shape to a different value from the new values.

PS-9109: The Percona server’s slow query rate is not accurately logged, which is controlled via the Log_slow_rate_type and Log_slow_rate_limit variables. Due to this issue, the slow query log records every query regardless of these variables’ values.

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

Percona Xtradb Cluster

PXC-4380: In a large cluster, for example, 15 nodes, in case one node has network issues and disconnects/re-connects/disconnects, the cluster might be sent to a non-primary state if the evs.install_timeout is reached.

Reported Affected Version/s: 5.7.42-31-65, 8.0.33-25, 8.0.35-27

Fixed Version: In such a big cluster, reaching consensus between nodes takes more time, so evs.install_timeout has to be adjusted. We can also configure the cluster to evict unresponsive nodes by setting evs.auto_evict=1. So, after investigation, we found that no fix is required, as everything works as expected/designed.

Workaround/Fix: Increasing the evs.install_timeout might fix the issue. The maximum value is 15S, which can still be reached depending on cluster size and how bad the flapping is.

The default value for evs.install_timeout is evs.inactive_timeout/2.

The minimum value is evs.join_retrans_period

The maximum time is evs.inactive_timeout + 1

So, If we keep defaults:

evs.join_retrans_period = 1s

evs.inactive_timeout = 15s

evs.install_timeout (default) = 7.5s

evs.install_timeout(min) = 1s

evs.install_timeout(max) = 16s

Please note that 15s is not a hard limit and is determined by evs.inactive_timeout

We have set wsrep_provider_options=“evs.install_timeout=PT15S” for all nodes in the test environment and are now unable to reproduce the issue. So, we are on some timeout boundaries, and the above configuration parameters were introduced to fine-tune in such environments.

PXC-4367: Innodb semaphore wait timeout failure seen after upgrade from 8.0.34 to 8.0.35. This issue is a possible side effect of this patch, Where PXC node acts as the async replica to some master and in parallel, the row is updated on PXC node and via replication, the PXC node hangs. To avoid the issue, upgrading to PXC 8.0.36 is recommended.

Reported Affected Version/s: 8.0.35-27

Fixed Version: 8.0.36-28

PXC-4363: Concurrent CREATE and DROP USER queries on different nodes lead to permanent lock.

| 16 | system user | db1 | Query | 411 | Waiting for table metadata lock | drop user IF EXISTS `msandbox_rw11`@`localhost` | 411380 | 0 | 

These queries cannot be cancelled or killed, and nodes refuse to be gracefully restarted. The shutdown is stuck with this forever. Only the forcible service kill helps restore the cluster.

2024-01-18T17:49:24.470428Z 0 [Note] [MY-000000] [Galera] Closing slave action queue.

Reported Affected Version/s: 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

PXC-4399: FLUSH TABLES during writes to the table with unique keys stall the cluster node; due to the stall, it’s not possible to abort/kill any of the above connections. The node sends a permanent flow control pause. The only way to get out of this stall is to kill the node.

Reported Affected Version/s: 8.0.33-25, 8.0.35-27

Fixed Version: The fix is expected to be included in an upcoming release of Percona Servers.

PXC-4418: In MySQL, the optimizer creates a temp table definition with indexes, where 2 of them have the same name (, , ). When the query is executed, a temp table is created, then MySql tries to access . In InnoDB, we search for index by name (dict_table_get_index_on_name()), which returns the wrong . Then row_sel_convert_mysql_key_to_innobase() crashes as structures are not aligned. Please note this bug affects only very complicated queries with many JOINs and subqueries. To repeat this bug, the internal temporary table needs to be created at least for two parts of the query.

Reported Affected Version/s: 8.0.32-24, 8.0.34-26

Fixed Version: 8.0.36-28

Percona Toolkit

PT-2190:The pt-show-grants use SHOW CREATE USER command to obtain grants from the MySQL server. By default, this query returns values as they are stored in the mysql.user table. When using caching_sha256_password, such hash of the password could contain a special character. Therefore, it would not be possible to use output printed by pt-show-grants to re-create users in the database. Since version 3.6.0, pt-show-grants checks if it runs against MySQL version 8.0.17 or higher and sets session option print_identified_as_hex to true before running SHOW CREATE USER command. This allows to print commands that could be used to re-create users.

Reported Affected Version/s: 3.5.1

Fixed Version: 3.6.0 [It is expected to be released soon]

PT-2215: pt-table-sync does not recognize the privileges in roles for MariaDB

Reported Affected Version/s: 3.5.2

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PT-2316: pt-config-diff with –pid option is broken with “Can’t locate object method “make_PID_file” via package “Daemon” at /usr/bin/pt-config-diff line 5522” on Ubuntu 20.04

Reported Affected Version/s: 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PT-2314: pt-online-schema-change fails due to duplicate constraint names when it attempts to make a table copy for alteration.

Reported Affected Version/s: 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

Workaround/Fix: Do not duplicate constraints name.

PT-2322: pt-mysql-summary does not detect jemalloc when installed as systemd.

Reported Affected Version/s: 3.5.6, 3.5.7

Fixed Version: The fix is expected to be included in an upcoming release of Percona ToolKit.

PMM [Percona Monitoring and Management]

PMM-11583: On MySQL 8.0, innodb_redo_log_capacity supersedes innodb_log_files_in_group and innodb_log_file_size, which eventually breaks InnoDB Logging graphs. Due to this issue, the user can not determine whether the combined InnoDB redo log file size has to be increased or not.

Reported Affected Version/s: 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-13017: For certain db.collection.find(query, projection, options) queries, the Explain tab for QAN returns an error message saying, “error decoding key command: invalid JSON input; expected value for 64-bit integer.” Please note this issue specifically affects MongoDB monitoring.

Reported Affected Version/s: 2.35.0, 2.37.1, 2.41.1

Fixed Version: The fix is expected to be included in an upcoming release of PMM

PMM-12522: When adding data relatively large chunks of data to MongoDB sharded cluster, pmm-agent log starts flooded with “level=error msg="cannot create metric for changelog… & level=error msg="Failed to get database names:…” which eventually shows MongoS as disconnected in PMM UI (PMM-Inventory/Services).

Reported Affected Version/s: 2.39.0, 2.40.0, 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-12880: pmm-admin –tls-skip-verify does not work when x509 authentication is used.

Reported Affected Version/s: 2.41.0

Fixed Version: 2.41.3 [It is expected to be released soon]

PMM-12989: PMM agent logs flooded with wrong log entries when monitoring auth-enabled arbiters. Please note this issue specifically affects MongoDB monitoring.

Reported Affected Version/s: 2.41.1

Fixed Version: 2.41.3 [It is expected to be released soon]

Percona XtraBackup

PXB-3251: When PXB fails to load the encryption key, the xtrabackup_logfile is still created in the target dir. This causes a second attempt at running PXB to fail with a new error. The xtrabackup_logfile file contains data needed to run the –prepare process. The bigger this file is, the longer the –prepare process will take to finish. So, PXB should not create any files on disk until the encryption key is loaded.

Reported Affected Version/s: 8.0.35-30

Fixed Version: The fix is expected to be included in future releases of PXB

Percona Kubernetes Operator

K8SPG-496: When a PostgreSQL Database is set to a paused state via spec, the operator waits until all backups for the Database finish. After the backups finish, the PostgreSQL Database shall be paused, which is not happening.

Reported Affected Version/s: 2.3.0

Fixed Version: 2.3.1

K8SPG-494: High vulnerabilities found for pgbackrest, Postgres & pgbouncer package.

For pgbackrest and postgres: CVE-2023-38408

For pgbouncer: CVE-2023-32067

Reported Affected Version/s: 2.3.0

Fixed Version: 2.3.1

K8SPG-521: The upgrade path described in the documentation leads to disabled built-in extensions(pg_stat_monitor, pg_audit).

Reported Affected Version/s: 2.3.1

Fixed Version: 2.4.0 [It is expected to be released soon]

K8SPG-522: Cluster is broken if PG_VERSION file is missing during the upgrade from 2.2.0 to 2.3.1.

Reported Affected Version/s: 2.3.1

Fixed Version: 2.4.0 [It is expected to be released soon]

Workaround/Fix: In order to fix the issue, please do the following:

Create PG_VERSION with contents 14 in DB instance pod.

echo 14 > /pgdata/pg14/PG_VERSION
Apply crd and rbac

kubectl apply --force-conflicts --server-side -f crd.yaml

kubectl -n operatornew apply --force-conflicts --server-side -f rbac.yaml
Restart the operator deployment.

kubectl -n operatornew rollout restart deployment percona-postgresql-operator

At this moment, patronictl show-config starts to enlist extensions.

$ kubectl -n operatornew exec cluster1-instance1-gsvs-0 -it -- bash

Defaulted container "database" out of: database, replication-cert-copy, postgres-startup (init), nss-wrapper-init (init)

bash-4.4$ patronictl show-config

loop_wait: 10

postgresql:

 parameters:

 archive_command: pgbackrest --stanza=db archive-push "%p"

 archive_mode: 'on'

 archive_timeout: 60s

 huge_pages: 'off'

 jit: 'off'

 password_encryption: scram-sha-256

 pg_stat_monitor.pgsm_query_max_len: '2048'

 restore_command: pgbackrest --stanza=db archive-get %f "%p"

shared_preload_libraries: pg_stat_monitor,pgaudit

Now extensions appear in the \dx output without pod restarts, but all Postgresql servers will be restarted:

K8SPG-547: The pgbackrest container can’t use pgbackrest 2.50. This is because pgbackrest 2.50 requires libssh2.so.1, which requires epel. Without that fix, microdnf installs pgbackrest 2.48, which creates inconsistency with the Postgresql container.

Reported Affected Version/s: 2.2.0

Fixed Version: 2.4.0 [It is expected to be released soon]

Workaround/Fix: This patch can be used until it is fixed.

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Download Percona Distribution for MySQL 8.0.36 (PXC-based variant)

Deploying Percona Everest on GCP with Kubectl for Windows 11 Users

Edith Puclla — Fri, 19 Apr 2024 00:00:00 UTC

Welcome to this blog post! Today, our primary goal is to guide you through deploying Percona Everest on GCP using Kubectl, specifically for users on Windows 11. It’s been some time since I last used Windows, so this will be an excellent opportunity to do it from scratch.

Let me tell you a little bit about Percona Everest. You may have already heard it recently. It is the new open source tool launched by Percona and is already well-received by Kubernetes database users.

Percona Everest is an open source cloud-native database platform that helps developers deploy code faster, scale deployments rapidly, and reduce database administration overhead while regaining control over their data, database configuration, and DBaaS costs. It is designed for those who want to break free from vendor lock-in, ensure optimal database performance, enable cost-effective and right-sized database deployments, and reduce database administration overhead.

If you use Windows and want to try the deployment and use of Percona Everest, you are in the right place.

This image shows what Percona Everest does and what we want to achieve:

Let’s start it!

Install WSL

We will use Kubectl to run commands on our Kubernetes clusters. There are many ways to use Kubectl on Windows.

I will use WSL (Windows Subsystem for Linux) to use the Linux environment directly on Windows. It is beneficial because “kubectl” and other Kubernetes tools often have better support.

In Windows 11, open PowerShell as Administrator and run:

wsl --install

This command will install WSL using the default options, including Ubuntu distribution and enabling the WSL 2 version.

Then restart your computer and open the newly installed Linux distribution from the Start menu. Complete the initial setup by creating a user account and password. Then update and upgrade your Linux distribution:

sudo apt update && sudo apt upgrade

Woolaa! We have Ubuntu running on Windows!

Installing WSL allows your Windows machine to run kubectl and other Linux-only applications smoothly. This setup is beneficial for developers and system administrators who work with both Windows and Linux systems.

Install Kubectl

In our Ubuntu terminal on Windows, we will use official documentation to install it using the native package management system.

bash
# Download the latest release with the command:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# Install kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Test to ensure the version you installed is up-to-date:
kubectl version --client

Creating a Kubernetes Cluster in Google Cloud

To create a Kubernetes Cluster with GKE, you need to have access to Google Cloud. Ensure it functions correctly and that you can access your Google project and create Kubernetes clusters. Also, ensure you have the gke-gcloud-auth-plugin installed. You can check if this is installed by running the “gcloud components list” command. If it is not installed, follow the official documentation.

I have already set it up. Now, I will proceed to create my Kubernetes cluster.

bash
gcloud container clusters create percona-everest --zone europe-west2-c --machine-type n1-standard-4 --num-nodes=3

Install Percona Everest

A prerequisite for installing Percona Everest is having a Kubernetes cluster. I have one that I created with GKE. To verify the Kubernetes cluster, run the following:

bash
kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-percona-everest-default-pool-1f7a9664-b3hd Ready  1h11m v1.27.8-gke.1067004
gke-percona-everest-default-pool-1f7a9664-b5c3 Ready  1h11m v1.27.8-gke.1067004
gke-percona-everest-default-pool-1f7a9664-nck4 Ready  1h11m v1.27.8-gke.1067004

Before running the commands in the Installation section, note that Everest will search for the kubeconfig file at the ~/.kube/config path

bash
export KUBECONFIG=~/.kube/config

The time to install Percona Everest. To install it, run the following command:

bash
curl -sfL "https://raw.githubusercontent.com/percona/everest/v0.9.1/install.sh" | bash

After installing it, you will see an output similar to the one on the left. In your browser, you can directly open 127.0.0.0:8080. Voilà! We now have Percona Everest up and running!

As the output indicates, the Percona Everest app will be available at http://127.0.0.1:8080. We use the authorization token to access the Everest UI and API.

We don’t have a database, so let’s create a new one!

This is the amazing thing about Percona Everest… you can create MySQL, MongoDB, and PostgreSQL databases on Kubernetes! Woohoo!!!

You can configure the resources for a new database, set up backups, monitoring, point-in-time recovery, and more:

And this is how it looks: your database is in Kubernetes!

Conclusion

Deploying Percona Everest on GCP using kubectl from a Windows 11 platform demonstrates the versatility and robust capabilities of managing databases on Kubernetes. The process should help you set up a powerful cloud-native database platform efficiently. We’ve walked through setting up your environment, installing necessary tools, creating a Kubernetes cluster, and finally deploying Percona Everest. Now, you can take full advantage of everything Percona Everest offers, from operational flexibility to cost efficiency.

If Percona Everest seems cool, feel free to contribute—it’s open source! Find Percona Everest on GitHub. If you encounter any issues during installation or have more questions, write to us in our community forum. If you prefer learning visually through videos, we have a friendly playlist of Percona Everest.

Release Roundup April 17, 2024

David Quilty — Wed, 17 Apr 2024 00:00:00 UTC

Percona software releases and updates April 2 - April 17, 2024.

Today’s post includes those releases and updates that have come out since April 1, 2024. Take a look.

Percona Distribution for MySQL 8.0.36 (PXC-based variant)

Percona Distribution for MySQL 8.0.36 (PXC-based variant) was released on April 3, 2024. It is the most stable, scalable, and secure open source MySQL distribution, with two download options: one based on Percona Server for MySQL and one based on Percona XtraDB Cluster. This release is focused on the Percona XtraDB Cluster-based deployment variation and is based on Percona XtraDB Cluster 8.0.36. Release highlights include improvements and bug fixes provided by Oracle for MySQL 8.0.36:

The hashing algorithm employed yielded poor performance when using a HASH field to check for uniqueness. (Bug #109548, Bug #34959356)
All statement instrument elements that begin with statement/sp/%, except statement/sp/stmt, are disabled by default.

Percona XtraDB Cluster 8.0.36

Percona XtraDB Cluster 8.0.36 was released on April 3, 2024. It supports critical business applications in your public, private, or hybrid cloud environment. Our free, open source, enterprise-grade solution includes the high availability and security features your business requires to meet your customer expectations and business goals.

Download Percona XtraDB Cluster 8.0.36

Percona Distribution for MongoDB 7.0.7

Percona Distribution for MongoDB 7.0.7 was released on April 4, 2024. It is a freely available MongoDB database alternative, giving you a single solution that combines enterprise components from the open source community, designed and tested to work together.

It includes the following components:

Percona Server for MongoDB is a fully compatible source-available, drop-in replacement for MongoDB.
Percona Backup for MongoDB is a distributed, low-impact solution for achieving consistent backups of MongoDB sharded clusters and replica sets.

Download Percona Distribution for MongoDB 7.0.7

Percona Distribution for MongoDB 5.0.26

April 9, 2024, saw the release of Percona Distribution for MongoDB 5.0.26. This release is based on Percona Server for MongoDB 5.0.26-22 and Percona Backup for MongoDB 2.4.1.

Download Percona Distribution for MongoDB 5.0.26

Percona Distribution for MongoDB 4.4.29

On April 2, 2024, we released Percona Distribution for MongoDB 4.4.29. This is the last minor release in Percona Distribution for MongoDB 4.4.

Download Percona Distribution for MongoDB 4.4.29

Percona Server for MongoDB 7.0.7-4

Percona Server for MongoDB 7.0.7-4 was released on April 4, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 7.0.5 and includes the improvements and bug fixes of MongoDB 7.0.6 Community Edition and MongoDB 7.0.7 Community Edition.

Download Percona Server for MongoDB 7.0.7-4

Percona Server for MongoDB 5.0.26-22

Percona Server for MongoDB 5.0.26-22 was released on April 9, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB 5.0.x Community Edition. Percona Server for MongoDB 5.0.26-22 includes both improvements and bug fixes of MongoDB 5.0.25 Community Edition and MongoDB 5.0.26 Community Edition.

Download Percona Server for MongoDB 5.0.26-22

Percona Server for MongoDB 4.4.29-28

Percona Server for MongoDB 4.4.29-28 was released on April 2, 2024. This is the last minor release in Percona Server for MongoDB 4.4.

Download Percona Server for MongoDB 4.4.29-28

Release Roundup April 2, 2024

David Quilty — Tue, 02 Apr 2024 00:00:00 UTC

Percona software releases and updates March 18 - April 2, 2024.

Today’s post includes those releases and updates that have come out since March 18, 2024. Take a look.

Percona Monitoring and Management 2.41.2

Percona Monitoring and Management 2.41.2 was released on March 22, 2024. It is an open source database monitoring, management, and observability solution for MySQL, PostgreSQL, and MongoDB. Starting with PMM 2.41.2, we now offer pmm-client packages for the latest version of Debian. You can install these packages by following the instructions in our documentation. We have also added several experimental dashboards, which are subject to change and recommended for testing purposes only.

Download Percona Monitoring and Management 2.41.2

Percona Operator for MySQL based on Percona Server for MySQL 0.7.0

Percona Operator for MySQL based on Percona Server for MySQL 0.7.0 was released on March 25, 2024. Percona Operator for MySQL allows users to deploy MySQL clusters with both asynchronous and group replication topology. This release includes various stability improvements and bug fixes, getting the Operator closer to the General Availability stage. Version 0.7.0 of the Percona Operator for MySQL is still a tech preview release and it is not recommended for production environments.

Download Percona Operator for MySQL based on Percona Server for MySQL 0.7.0

Percona XtraBackup 8.3.0-1

Percona XtraBackup 8.3.0-1 was released on March 26, 2024. Percona XtraBackup 8.3.0-1 is based on MySQL 8.3 and fully supports the Percona Server for MySQL 8.3 Innovation series and the MySQL 8.3 Innovation series. This release allows taking backups of Percona Server 8.3.0-1 and MySQL 8.3. This Innovation release is only supported for a short time and is designed to be used in an environment with fast upgrade cycles. Developers and DBAs are exposed to the latest features and improvements. Patches and security fixes are included in the next Innovation release instead of a patch release or fix release within an Innovation release. To keep your environment current with the latest patches or security fixes, upgrade to the latest release.

Download Percona XtraBackup 8.3.0-1

Percona Distribution for MongoDB 6.0.14

Percona Distribution for MongoDB 6.0.14 was released on March 26, 2024. It is a freely available MongoDB database alternative, giving you a single solution that combines enterprise components from the open source community, designed and tested to work together. Release highlights include:

Fixed the issue with missing peer certificate validation if neither CAFile nor clusterCAFile is provided.
Fixed the issue with multi-document transactions missing documents when the movePrimary operation runs concurrently by detecting placement conflicts in multi-document transactions.
Allow a clustered index scan in a clustered collection if a notablescan option is enabled.
Fixed tracking memory usage in SharedBufferFragment to prevent out of memory issues in the WiredTiger storage engine.
Added an index on the process field for the config.locks collection to ensure update operations on it are completed even in heavy loaded deployments.
Fixed the incorrect hardware checksum calculation on zSeries for buffers on stack.

Download Percona Distribution for MongoDB 6.0.14

Percona Server for MongoDB 6.0.14-11

Percona Server for MongoDB 6.0.14-11 was released on March 26, 2024. It is an enhanced, source-available, and highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB Community Edition 6.0.14. It is based on MongoDB 6.0.14 Community Edition and includes improvements and bug fixes provided by MongoDB.

Download Percona Server for MongoDB 6.0.14-11

Percona Backup for MongoDB 2.4.1

On March 25, 2024, Percona Backup for MongoDB 2.4.1 was released. It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set), and for restoring those backups to a specific point in time.

This release fixes the issue of failing incremental backups. It was caused by the backup metadata document reaching the maximum size limit of 16MB. The issue is fixed by introducing the new approach to handling the metadatada document: it no longer contains the list of backup files which is now stored separately on the storage and is read by PBM on demand. The new metadata handling approach applies to physical, incremental, and snapshot-based backups.

Download Percona Backup for MongoDB 2.4.1

Creating a Standby Cluster With the Percona Operator for PostgreSQL

Zsolt Parragi — Wed, 27 Mar 2024 00:00:00 UTC

In this video, Nickolay Ihalainen, a Senior Scaling Specialist at Percona Global Services, explains how to set up replication with standby clusters for Kubernetes databases using Percona’s open-source tools, including the Percona Operator for PostgreSQL

A Standby Cluster is a backup version of your main database. It’s there to keep your data safe and make sure your database can keep running even if something goes wrong with the main one.

For this demo, we use Percona Operators for PostgreSQL to create the clusters, which facilitates high availability setups and database management by automating deployment, scaling, and management tasks within Kubernetes environments.

Nickolay created a primary node and the configuration of replication to standby clusters for PostgreSQL, ensuring data redundancy and availability. This primary node has two standby databases in the same primary node.

Then, we have the object storage (S3) for backups, highlighting the importance of having offsite backups in different geographical locations to safeguard against data loss. This is where the primary node will access to make a data replication.

This demo also includes using Patroni to manage this process, enabling replication and failover between primary and standby servers, and PgBouncer, a tool that manages how applications are aligned to communicate with a PostgreSQL database.

Watch our complete hands on in this YouTube video:

You can find more instructions on how to deploy a standby cluster for Disaster Recovery, and also you can Create a Standby Cluster With the Percona Operator for PostgreSQL.

Is you have questions o feedback, write to us in our Percona Comunity Forum

Open Door MongoDB Chats

Edith Puclla — Tue, 26 Mar 2024 00:00:00 UTC

Let’s talk about MongoDB! This week, Jan Wieremjewicz, Senior Product Manager, and Michal Nosek, Pre-Sales Enterprise Architect at Percona, made two series of videos covering basic to advanced concepts in MongoDB and backups.

Le’s explore them together!

Enterprise Scale Backups for MongoDB with Open Source Software

In the first video about enterprise-scale backups for MongoDB using open-source software, Jan and Michal discuss the challenges and solutions associated with backing up large datasets in MongoDB. They explore the difficulties of managing backups and restorations for huge data sizes, emphasizing the limitations of logical backups like mongodump and mongorestore, which can be time-consuming for large datasets.

They also highlight alternative solutions, including MongoDB Enterprise licenses offering robust backup tools, the use of storage-level snapshots, and Percona’s own backup solution for MongoDB. This solution aims to simplify and speed up the backup and restoration process, especially for large, sharded clusters, and is designed to be efficient and flexible, preventing vendor lock-in and helping with enterprise needs.

The video covers the technical aspects of ensuring consistent, performant backups across complex database architectures. I recommend you watch the video because it covers basic topics from an expert viewpoint.

Optimize the RPO and RTO for your MongoDB Backup Strategy

In this second episode of their discussion on MongoDB backups, Jan and Michal explore the critical aspects of developing a comprehensive backup strategy, focusing on Recovery Point Objective(RPO) and Recovery Time Objective (RTO). They highlight the importance of these concepts in determining the acceptable amount of data loss and system downtime in the event of a failure. Michal explains that achieving a near-zero RPO requires significant investment, while Jan highlights the importance of documenting a backup strategy for job security and organizational awareness.

They also discuss the common oversight among smaller organizations that lack a formal backup strategy, often due to the perceived cost or a lack of major failures. The conversation covers the necessity of regularly testing backups and restore times, as well as communicating the current backup and recovery capabilities to higher management to align expectations with business criticality. This approach ensures transparency and preparedness for potential system failures, making it an essential practice for companies of all sizes.

Read our latest article titled Using Percona Backup for MongoDB in Replica Set and Sharding Environments: Part One

For more information, visit the official Percona Backup for MongoDB website. If you have any questions, feel free to visit our forum under the MongoDB category.

Release Roundup March 18, 2024

David Quilty — Mon, 18 Mar 2024 00:00:00 UTC

Percona software releases and updates March 5 - March 18, 2024.

Today’s post includes those releases and updates that have come out since March 4, 2024. Take a look.

Percona Server for MySQL 5.7.44-49 (Post-EOL support version)

This release is Percona Server for MySQL 5.7.44-49 (Post-EOL support version), and the fixes are available to MySQL 5.7 Post-EOL Support from Percona customers. Community members can build this release from the source. Percona Server for MySQL 5.7.44-49 contains the fix for CVE-2024-20963 and a portability fix.

Download Percona Server for MySQL 5.7.44-49 (Post-EOL support version)

Percona Distribution for MySQL 8.0.36 (PS-based variant)

On March 4, 2024, Percona Distribution for MySQL 8.0.36 (PS-based variant) was released. It is the most stable, scalable, and secure open source MySQL distribution, with two download options: one based on Percona Server for MySQL and one based on Percona XtraDB Cluster. This release is focused on the Percona Server for MySQL-based deployment variation. This release fixes the Orchestrator issues.

Download Percona Distribution for MySQL 8.0.36 (PS-based variant)

Percona Server for MySQL 8.0.36

Percona Server for MySQL 8.0.36 was released on March 4, 2024. It includes all the features and bug fixes available in the MySQL 8.0.36 Community Edition, and enterprise-grade features developed by Percona. Improvements and bug fixes provided by Oracle for MySQL 8.0.36 and included in Percona Server for MySQL are the following:

The hashing algorithm employed yielded poor performance when using a HASH field to check for uniqueness. (Bug #109548, Bug #34959356)
All statement instrument elements that begin with statement/sp/%, except statement/sp/stmt, are disabled by default.

Download Percona Server for MySQL 8.0.36

Percona Operator for MySQL based on Percona XtraDB Cluster 1.14.0

Percona Operator for MySQL based on Percona XtraDB Cluster 1.14.0 was released on March 4, 2024. It contains everything you need to quickly and consistently deploy and scale Percona XtraDB Cluster instances in a Kubernetes-based environment on-premises or in the cloud. Among other new features, a custom prefix for Percona Monitoring and Management (PMM) allows using one PMM Server to monitor multiple databases, even if they have identical cluster names.

Download Percona Operator for MySQL based on Percona XtraDB Cluster 1.14.0

Percona Distribution for PostgreSQL 13.14

Percona Distribution for PostgreSQL 13.14 was released on March 6, 2024. It is a solution of a collection of tools from the PostgreSQL community that are tested to work together and assist you in deploying and managing PostgreSQL. A release highlight is that the Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Download Percona Distribution for PostgreSQL 13.14

Percona Distribution for PostgreSQL 12.18

On March 11, 2024, we released Percona Distribution for PostgreSQL 12.18. This release of Percona Distribution for PostgreSQL is based on PostgreSQL 12.18.

Download Percona Distribution for PostgreSQL 12.18

Percona Backup for MongoDB 2.4.0

On March 5, 2024, Percona Backup for MongoDB 2.4.0 was released. It is a distributed, low-impact solution for consistent backups of MongoDB sharded clusters and replica sets. This is a tool for creating consistent backups across a MongoDB sharded cluster (or a non-sharded replica set), and for restoring those backups to a specific point in time. A release highlight is that you can now delete backup snapshots of a specific type. For example, delete only logical backups that you might have created and no longer need. You can also check what exactly will be deleted with the new --dry-run flag. This improvement helps you better meet the organization’s backup policy and improves your experience with cleaning up outdated data.

Download Percona Backup for MongoDB 2.4.0

Release Roundup March 4, 2024

David Quilty — Mon, 04 Mar 2024 00:00:00 UTC

Percona software releases and updates February 21 - March 4, 2024.

Today’s post includes those releases and updates that have come out since February 20, 2024. Take a look.

Percona Distribution for PostgreSQL 16.2

Percona Distribution for PostgreSQL 16.2 was released on February 27, 2024. It provides the best and most critical enterprise components from the open source community in a single distribution, designed and tested to work together. This release is based on PostgreSQL 16.2. A release highlight is that a Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Download Percona Distribution for PostgreSQL 16.2

Percona Distribution for PostgreSQL 15.6

On February 28, 2024, we released Percona Distribution for PostgreSQL 15.6, which is based on PostgreSQL 15.6. A Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Percona Distribution for PostgreSQL also includes the following packages:

llvm 12.0.1 packages for Red Hat Enterprise Linux 8 and compatible derivatives. This fixes compatibility issues with LLVM from upstream.
supplemental ETCD packages which can be used for setting up Patroni clusters. These packages are available for the following operating systems:

Download Percona Distribution for PostgreSQL 15.6

Percona Distribution for PostgreSQL 14.11

Percona Distribution for PostgreSQL 14.11 was released on March 1, 2024, and is based on PostgreSQL 14.11. A release highlight is a Docker image for Percona Distribution for PostgreSQL is now available for ARM architectures. This improves the user experience with the Distribution for developers with ARM-based workstations.

Download Percona Distribution for PostgreSQL 14.11

Setting Up Your Environment for Kubernetes Operators Using Docker, kubectl, and k3d

Edith Puclla — Mon, 04 Mar 2024 00:00:00 UTC

If you are just starting out in the world of Kubernetes operators, like me, preparing the environment for their installation should be something we do with not much difficulty. This blog will quickly guide you in setting the minimal environment.

Kubernetes operators are invaluable for automating complex database operations, tasks that Kubernetes does not handle directly. Operators make it easy for us – they take care of essential tasks like backups and restores, which are crucial in database management.

If you want an introduction to Kubernetes Operators, I cover it in this 5-minute blog post, Exploring the Kubernetes Application Lifecycle With Percona.

Now that we know why Kubernetes Operators are essential let’s prepare our environment to install some of them. We are going to base this installation on Linux for now. Prerequisites: For this, we will need a basic understanding of Kubernetes concepts and some Linux command line skills. We also need Docker Engine to be able to use K3d at all for containerization. To test, make sure this command runs appropriately:

docker run hello-world

Installing kubectl

To manage and deploy applications on Kubernetes, we will need kubectl tool, which is included in most Kubernetes distributions. If it’s not installed, let’s do it following the official installation instructions:

To install the kubectl binary with curl on Linux, we need to download the latest release of kubectl using the command:

bash
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

The previous binary installs kubectl in /usr/local/bin/kubectl. We need root ownership and specific permissions for secure execution.

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

To test the installation, we use the following:

kubectl version --client

kubectl version --client --output=yaml

If you receive a response like this, it indicates that you are ready to use kubectl.

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3

Installing K3d

k3d is a lightweight tool that simplifies running k3s (Rancher Lab’s minimal Kubernetes distribution in Docker), enabling easy creation of single and multi-node k3s clusters for local development.

Install the current latest release of k3d with curl:

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

To test the installation, you can use the following:

k3d --help

If you see a message similar to this, you are ready to create your k3d Kubernetes clusters.

bash
https://k3d.io/
k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker.
Nodes of a k3d cluster are docker containers running a k3s image.
All Nodes of a k3d cluster are part of the same docker network.
Usage:
k3d [flags]
k3d [command]
Available Commands:
cluster Manage cluster(s)
completion Generate completion scripts for [bash, zsh, fish, powershell | psh]
config Work with config file(s)

Starting the Kubernetes cluster

Let’s use K3d and create a Kubernetes cluster with three nodes. Using the flag -a, you can specify the number of nodes you want to add to the k3d cluster.

k3d cluster create database-cluster -a 3

Now, list details for our k3d cluster.

bash
k3d cluster list database-cluster

NAME SERVERS AGENTS LOADBALANCER
database-cluster 1/1 3/3 true

Now, our environment is ready to begin installing our Percona Kubernetes Operators.

Conclusion

In this tutorial, we chose k3d over Minikube due to its efficiency and speed in setting up Kubernetes clusters with multiple nodes, which are essential for effectively testing Kubernetes operators in a local environment. Although it’s possible to perform tests on a single node with both systems, k3d makes it easier to simulate a more realistic distributed environment, allowing us to utilize our resources more efficiently.

Take a look at our GitHub repository for our Percona Kubernetes Operators:

They are fully Open Source. And if you are looking for a version with a graphical interface, we have Percona Everest, our cloud-native database platform: docs.percona.com/everest

What’s Next? Let’s install our Kubernetes Operators!

Release Roundup February 21, 2024

David Quilty — Wed, 21 Feb 2024 00:00:00 UTC

Percona software releases and updates February 5 - February 21, 2024.

Today’s post includes those releases and updates that have come out since February 5, 2024. Take a look.

Percona Distribution for MySQL (PS-based variation) 8.2.0

Percona Distribution for MySQL (PS-based variation) 8.2.0 was released on February 5, 2024. It is a bundling of open source MySQL software enhanced with carefully curated and designed enterprise-grade features. Percona Distribution for MySQL offers two download options; this one is based on Percona Server for MySQL.

This release merges the MySQL 8.2 code base, introducing several significant changes:

Removes remains of Percona-specific encryption features (support for custom Percona 5.7 encrypted binlog format).
Removes the deprecated rocksdb_strict_collation_check and rocksdb_strict_collation_exceptions RocksDB system variables.

Download Percona Distribution for MySQL (PS-based variation) 8.2.0

Percona Distribution for MongoDB 6.0.13

Percona Distribution for MongoDB 6.0.13 was released on February 20, 2024. It includes the following components:

Percona Server for MongoDB is a fully compatible source-available, drop-in replacement for MongoDB.
Percona Backup for MongoDB is a distributed, low-impact solution for achieving consistent backups of MongoDB sharded clusters and replica sets.

This release of Percona Distribution for MongoDB is based on the production release of Percona Server for MongoDB 6.0.13-10 and Percona Backup for MongoDB 2.3.1.

Download Percona Distribution for MongoDB 6.0.13

Percona Distribution for MongoDB 4.4.28

Percona Distribution for MongoDB 4.4.28 was released on February 7, 2024. It’s a freely available MongoDB database alternative, giving you a single solution that combines enterprise components from the open source community, designed and tested to work together. In addition to bug fixes and improvements provided by MongoDB and included in Percona Server for MongoDB, Percona Backup for MongoDB 2.3.1 enhancements include the following:

Support for Percona Server for MongoDB 7.0.x
The ability to define custom endpoints when using Microsoft Azure Blob Storage for backups
Improved PBM Docker image to allow making physical backups with the shared mongodb data volume
Updated Golang libraries that include fixes for the security vulnerability CVE-2023-39325.

In addition, Percona Server for MongoDB 4.4.28-27 is no longer available on Ubuntu 18.04 (Bionic Beaver).

Download Percona Distribution for MongoDB 4.4.28

Percona Distribution for MongoDB 4.2.25

On February 8, 2024, Percona Distribution for MongoDB 4.2.25 was released. Release highlights include:

Optimized the construction of the balancer’s collection distribution status histogram
Fixed the query planner logic to distinguish parameterized queries in the presence of a partial index that contains logical expressions ($and, $or).
Improved performance of updating the routing table and prevented blocking client requests during refresh for clusters with 1 million of chunks.
Avoided traversing routing table in balancer split chunk policy
Fixed the issue that caused the modification of the original ChunkMap vector during the chunk migration and that could lead to data loss. The issue affects MongoDB versions 4.4.25, 5.0.21, 6.0.10 through 6.0.11 and 7.0.1 through 7.0.2. Requires stopping all chunk merge activities and restarting all the binaries in the cluster (both mongod and mongos).

Download Percona Distribution for MongoDB 4.2.25

Percona Server for MongoDB 6.0.13-10

Percona Server for MongoDB 6.0.13-10 was released on February 20, 2024. It is based on MongoDB 6.0.13 Community Edition and supports the upstream protocols and drivers.

Release highlights include:

Percona Server for MongoDB packages are available for ARM64 architectures, enabling users to install it on-premises. The ARM64 packages are available for the following operating systems:

Ubuntu 20.04 (Focal Fossa)
Ubuntu 22.04 (Jammy Jellyfish)
Red Hat Enterprise Linux 8 and compatible derivatives
Red Hat Enterprise Linux 9 and compatible derivatives

Download Percona Server for MongoDB 6.0.13-10

Percona Server for MongoDB 4.4.28-27

Percona Server for MongoDB 4.4.28-27 was released on February 7, 2024. It is a source available, highly-scalable database that is a fully-compatible, drop-in replacement for MongoDB 4.4.28 Community Edition enhanced with enterprise-grade features. Release highlights include these bug fixes, provided by MongoDB and included in Percona Server for MongoDB:

Fixed the issue with the data and the ShardVersion mismatch for sharded multi-document transactions by adding the check that no chunk has moved for the collection being referenced since transaction started
Improved cluster balancer performance by optimizing the construction of the balancer’s collection distribution status histogram
Fixed the issue with blocking acquiring read/write tickets by TransactionCoordinator by validating that it can be recovered on step-up and can commit the transaction when there are no storage tickets available
Investigated a solution to avoid a Full Time Diagnostic Data Capture (FTDC) mechanism to stall during checkpoint

Percona Server for MongoDB 4.4.28-27 is no longer available on Ubuntu 18.04 (Bionic Beaver).

Download Percona Server for MongoDB 4.4.28-27

Percona Server for MongoDB 4.2.25-25

Percona Server for MongoDB 4.2.25-25 was released on February 7, 2024. A release highlight is that Percona Server for MongoDB includes telemetry that fills in the gaps in our understanding of how you use Percona Server for MongoDB to improve our products. Participation in the anonymous program is optional. You can opt-out if you prefer not to share this information. Read more about Telemetry.

Percona Server for MongoDB 4.2.25-25

Percona Bug Report: January 2024

Aaditya Dubey — Mon, 19 Feb 2024 00:00:00 UTC

At Percona, we believe that transparency is key to improving our products. We are dedicated to creating top-of-the-line open-source database solutions and providing support for any issues that may arise. We encourage feedback and bug reports to help us continually improve.

We stay updated on bug reports through our own platform as well as other sources to ensure we have the most up-to-date information. To make it easier for you, we have compiled a central list of the most critical bugs for your reference in this edition of our bug report.

In this episode of our bug report, we provide the following list of bugs.

Percona Server/MySQL Bugs

PS-8983: System variable group_replication_view_change_uuid introduced in MySQL 8.0.26 which corrected the issue Bug#103641 in where data is inconsistent between nodes after killing primary node in group replication, However there is still an issue where these events are also generated on the standby/secondary cluster in a ClusterSet thus creating errant transactions, and if binlogs containing these events are purged, then it will not be possible to perform a failover between clusters.

Reported Affected Version/s: 8.0.26

Fixed Version: 8.0.31-23

PS-9048: When innodb_optimize_fulltext_only is enabled and running OPTIMIZE TABLE which has fulltext index actually causing assertion in Percona server debug build, Please note issue is specifically happening when PARSER is ngram.

Reported Affected Version/s: 5.7.42, 8.0.34 Fixed Version: N/A [Fix in Progress]

PS-9018: When replica has a non-replicated database then during intensive workload from the source where multi-threaded slave applier (MTS) is enabled and log_slave_updates=0 then DDL executed against this non-replicated database completely stalls the replica instance.

Upstream Bug: 113727

Reported Affected Version/s: 8.0.19+ Fixed Version: N/A [Fix in Review] Workaround : Use log_slave_updates=1, Please note enabling this may produce huge binlog volume on the replica which may or may not be feasible with respect to storage.

PS-9083: Percona server crashes when server is running with slow_query_log in conjunction with long_query_time, log_slow_verbosity = profiling,query_info variables.

Reported Affected Version/s: 8.0.35 Fixed Version: 8.0.36 [Pending Release] Workaround : Remove “query_info” from log_slow_verbosity

PS-9081: Materializing happens when a query is being executed against performance_schema.data_locks can lead to excessive memory usage and OOM.

Reported Affected Version/s: 8.0.34+ Fixed Version: It is expected to be fixed by PS 8.0.37 Workaround : Putting a LIMIT clause to read queries.

Percona Xtradb Cluster

PXC-4341: Execution of prepared statement after FLUSH TABLES makes the node abort from the cluster.

Reported Affected Version/s: 8.0.33+ Fixed Version: It is expected to be fixed by PXC 8.0.36 Workaround : There is no straight forward workaround but one can run the prepared statement and FLUSH TABLES statement separately.

PXC-4316: Network loss may lead to node’s logs flooded with “changed identity” events which eventually let primary node go non-primary, and reconnect another node. It will keep non primary nodes so we ended with all nodes as non primary.

Reported Affected Version/s: 8.0.33+ Fixed Version: It is expected to be fixed by PXC 8.0.36

PXC-4348: Cluster state interrupted with MDL BF-BF conflict when forcing deadlock. To hit the crash we are required to run queries on multiple sessions where one session should run “optimize table ;” multiple times so mysqlslap is the right candidate to repeat this behavior and other sessions will run delete/insert on the same table.

Reported Affected Version/s: 8.0.33+ Fixed Version: It is expected to be fixed by PXC 8.0.36

Percona Toolkit

PT-2217: When running pt-mongodb-summary against psmdb6.0/psmdb7.0 it gives error “BSON field ‘getCmdLineOpts.recordStats’ is an unknown field” Please note that PT tool does not work with MongoDB 6.0+.

Reported Affected Version/s: 3.5.X Fixed Version: It is expected to be fixed by PT 3.6.0

PT-2309: When the primary key is a UUID binary 16 column pt-table-sync hits with error “Cannot nibble table db_name.table_name because MySQL chose no index instead of the PRIMARY”

Reported Affected Version/s: 3.5.7 Fixed Version: It is expected to be fixed by PT 3.5.8

PT-2305: pt-online-schema-change should error out if server is a slave/replica in row based replication. This can lead to source/replica becoming inconsistent if there are writes on source when the tool runs on replica.

Please find the example below where data loss is seen:

Set-up classic source-replica
Make sure binlog_format=row
Create a table on master and add sufficient data so that pt-osc takes a little bit of time to run.
Then start pt-osc on slave, and execute updates/deletes on master.
Once pt-osc is done, check table checksum or table count to verify the data differences. Please check the below output with row differences:

master [localhost:22536] {msandbox} (test) > select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
| 999999 |
+----------+
1 row in set (0.58 sec)

slave1 [localhost:22537] {msandbox} (test) > select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
| 1000000 |
+----------+
1 row in set (0.40 sec)

Reported Affected Version/s: 3.5.7 Fixed Version: It is expected to be fixed by PT 3.6.0

PT-2284: When running pt-kill with the –daemonize option, if the query has character like ‘柏木’, pt-kill process exists with message “Wide character in printf at /usr/bin/pt-kill line 7508.”

Reported Affected Version/s: 3.5.7 Fixed Version: It is expected to be fixed by PT 3.6.0 Workaround : Running pt-kill without –daemonize option manually.

PT-2089: When SHOW ENGINE INNODB STATUS reports garbled UTF characters then pt-deadlock-logger crashes with “server ts thread txn_id txn_time user hostname ip db tbl idx lock_type lock_mode wait_hold victim query”

Reported Affected Version/s: 3.3.1, 3.5.7

Percona Monitoring and Management (PMM)

PMM-12806: We can’t tune VictoriaMetrics running inside PMM since PMM does not honor the environment variables for VictoriaMetrics. So PMM pre-defines certain flags that allow users to set all other VictoriaMetrics parameters as environment variables.

Example:

To set downsampling, use the downsampling.period parameter as follows:

-e VM_downsampling_period=20d:10m,120d:2h

This instructs VictoriaMetrics to deduplicate samples older than 20 days with 10 minute intervals and samples older than 120 days with two hour intervals.

Reported Affected Version/s: 2,40.1 Fixed Version: It is expected to be fixed by PMM 2.41.2

PMM-12805: When monitoring MongoDB servers, logs might get filled with the a CommandNotSupportOnView message, As a result, disk space fills up.

Reported Affected Version/s: 2,40.0, 2.41.0 Fixed Version: It is expected to be fixed by PMM 2.41.2

PMM-12809: Common Vulnerabilities and Exposures (CVE) found in PMM gRPC(Remote Procedure Call (RPC)) which impacts PMM v2.40.1+

Reported Affected Version/s: 2.41.0+ Fixed Version: It is expected to be fixed by PMM 2.41.2

Percona XtraBackup

PXB-3024: Backups are not reliable when running on a secondary node of Group Replication(GR) since –lock-ddl does not have any effect on secondary node of GR.

Reported Affected Version/s: 8.0.28-20, 8.0.31-24 Fixed Version: 8.0.32-26

PXB-2928: Xtrabackup crashes with signal 11 when taking a backup using –page-tracking option. So if you are using this option while taking backup then upgrading to PXB 8.0.31 is recommended since there is no workaround available to this issue at the moment.

Reported Affected Version/s: 8.0.29-22 Fixed Version: 8.0.31-24

PXB-3037: In order to assure a consistent replication state, –safe-slave-backup option stops the replication SQL thread and waits to start backing up until slave_open_temp_tables in SHOW STATUS is zero. If there are no open temporary tables, the backup will take place, otherwise the SQL thread will be started and stopped until there are no open temporary tables. The backup will fail if slave_open_temp_tables does not become zero after –safe-slave-backup-timeout seconds (defaults to 300 seconds). The replication SQL thread will be restarted when the backup finishes, But due to this bug if backup fails in between then SQL thread is not getting restarted. So restarting the SQL thread manually is required.

Reported Affected Version/s: 8.0.31-24, 8.0.35-30 Fixed Version: No ETA

PXB-2733: backup-lock-timeout and backup-lock-retry-count do not work.

Reported Affected Version/s: 2.4.24, 8.0.27-19, 8.0.35-30 Fixed Version: No ETA

Percona Kubernetes Operator

K8SPG-492: Restore job created by PerconaPGRestore doesn’t inherit .spec.instances[].tolerations since restore Job pod get stuck in pending and causing down time.

Reported Affected Version/s: 2.2.0 Fixed Version: It is expected to be fixed by PG operator 2.4.0 Workaround: Remove taint, wait until the restore container is scheduled and re-add it again. K8SPSMDB-958: PMM fails to monitor mongos due to lack of permission.

Reported Affected Version/s: 1.14.0 Fixed Version: 1.15.0

K8SPG-291: Modifying existing backup schedule does not work with PG operator v1.3.0

Reported Affected Version/s: 1.3.0 Fixed Version: 1.4.0

K8SPG-286: When requiring TLS for all connections, PMM client fails to connect with “no pg_hba.conf entry”.

Reported Affected Version/s: 1.2.0, 1.3.0, 2.0.0 Fixed Version: 1.4.0

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Unexpected Stalled Upgrade to MySQL 8.0

Alexandre Vaniachine — Fri, 26 Jan 2024 00:00:00 UTC

A multi-tenant database is a database that serves multiple clients, or tenants, who share the same database schema but have separate data sets. One way to achieve data isolation for each client is to create a separate MySQL database for each tenant.

Some advantages of this approach are:

It allows for easy backup and restore of individual tenant data.
It simplifies the database administration and maintenance tasks, as each database can be managed independently.
Scaling is easily achieved by adding more database servers and distributing tenant databases across them.

This approach requires a large number of tables on each server. Combined with the default value of innodb_file_per_table=ON, this results in a large number of files that affects crash recovery time or Percona XtraBackup execution.

This blog post describes how to take care of a large number of files when upgrading to MySQL 8.0 in-place.

Version Selection

A steady stream of MySQL 8.0 minor releases provides improvements and refactoring of new MySQL 8.0 features. However, some of these releases introduce incompatibilities that require corresponding changes on the application side. Limiting scope of the application-side changes, we chose MySQL 8.0.25 version. This was our first step towards the major version 8.0.

Upgrade In-Place

A new MySQL 8.0 option is the upgrade in-place procedure. According to the upgrading guide:

An in-place upgrade is performed by using existing data on the server and involves the following actions:

Stopping the MySQL 5.7 server

Replacing the old binaries with MySQL 8.0 binaries

Starting the MySQL 8.0 server with the same data files.

While an in-place upgrade may not be suitable for all environments, especially those environments with many variables to consider, the upgrade should work in most cases.

As an exception, in the case of an environment with a large number of tables, the upgrade in-place may get stalled for weeks.

Below we describe how to debug and resolve such issue.

Encountering the Issue

In our test environment, we encountered a similar issue. Despite steady CPU usage, the in-place upgrade looks stalled. We monitored the upgrade progress by counting files modified in the last 24 hours. Monitoring revealed low modification rates, like

find /var/lib/mysql -name "*.ibd" -mtime -1 | wc -l
14887

that were decreasing. Although the InnoDB files continued to be modified, the decreasing modification rate was too low to be practical.

Investigating the Issue

To debug this problem we used the Linux perf tool. While the mysqld process was running during upgrade:

we collected perf data

perf record -F 10 -o mysqld.perf -p $(pidof mysqld) -- sleep 20;
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.256 MB mysqld.perf (1016 samples) ]

and produced the perf report

perf report --input mysqld.perf --stdio

# To display the perf.data header info, please use --header/--header-only options.
#
#
# Total Lost Samples: 0
#
# Samples: 1K of event 'cpu-clock'
# Event count (approx.): 101600000000
#
# Overhead Command Shared Object Symbol
# ........ ....... .................. ................................
#
 34.55% mysqld libc-2.17.so [.] __sched_yield
 14.86% mysqld [kernel.kallsyms] [k] __raw_spin_unlock_irq
 11.32% mysqld [kernel.kallsyms] [k] system_call_after_swapgs
 11.32% mysqld mysqld [.] Fil_shard::reserve_open_slot
 ...

To find out why mysqld process stuck in the Fil_shard::reserve_open_slot call, we checked the Percona Server source code that shows:

the function code

/** Wait for an empty slot to reserve for opening a file.
@return true on success. */
bool Fil_shard::reserve_open_slot(size_t shard_id) {
 size_t expected = EMPTY_OPEN_SLOT;

 return s_open_slot.compare_exchange_weak(expected, shard_id);
}

and the corresponding comments

The data structure (Fil_shard) that keeps track of the tablespace ID to
fil_space_t* mapping are hashed on the tablespace ID. The tablespace name to
fil_space_t* mapping is stored in the same shard. A shard tracks the flushing
and open state of a file. When we run out open file handles, we use a ticketing
system to serialize the file open, see Fil_shard::reserve_open_slot() and
Fil_shard::release_open_slot().

Apparently, the stalled upgrade process hit the open files limit, given the large number of files in our environment.

Resolving the Issue

To prevent the mysqld upgrade process from running out of open file handles, we followed Percona guidance for setting open files limit

Counted files as

find /var/lib/mysql/ -name "*.ibd" | wc -l
324780

and added another 1000 to this number for other miscellaneous open file needs.

Increased the innodb_open_files limit in two places:

added a corresponding line to configuration file /etc/my.cnf like

innodb_open_files = 325780

added a corresponding line to the systemd configuration file such as /etc/systemd/system/mysqld.service.d/override.conf like

[Service]
LimitNOFILE = 325780

With these adjustments our upgrade completed processing of a terabyte of data in just a few hours. To provide more visibility into the upgrade process we also increased the default level of error log verbosity by adding another line to the /etc/my.cnf file:

log_error_verbosity = 3

Increased verbosity enabled progress monitoring in the mysql error log during upgrade, like:

2023-10-28T00:27:09.331924Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.25-15) starting as process 16034
...
2023-10-28T00:27:09.353871Z 1 [System] [MY-011012] [Server] Starting upgrade of data directory.
2023-10-28T00:27:09.353986Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
...
2023-10-28T00:27:19.412572Z 1 [Note] [MY-012206] [InnoDB] Found 324780 '.ibd' and 0 undo files
2023-10-28T00:27:19.412757Z 1 [Note] [MY-012207] [InnoDB] Using 17 threads to scan 324780 tablespace files
2023-10-28T00:27:28.764032Z 0 [Note] [MY-012200] [InnoDB] Thread# 0 - Checked 15615/20298 files
...
2023-10-28T00:27:31.718051Z 0 [Note] [MY-012201] [InnoDB] Checked 20298 files
2023-10-28T00:27:31.718440Z 1 [Note] [MY-012208] [InnoDB] Completed space ID check of 324780 files.
...
2023-10-28T00:27:48.821432Z 1 [Note] [MY-012922] [InnoDB] Waiting for purge to start
2023-10-28T00:27:48.878058Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2023-10-28T00:27:48.885203Z 1 [Note] [MY-011088] [Server] Data dictionary initializing version '80023'.
2023-10-28T00:27:49.187508Z 1 [Note] [MY-010337] [Server] Created Data Dictionary for upgrade
...
2023-10-28T01:57:55.312683Z 2 [System] [MY-011003] [Server] Finished populating Data Dictionary tables with data.
...
2023-10-28T01:59:15.187709Z 5 [System] [MY-013381] [Server] Server upgrade from '50700' to '80025' started.
...
2023-10-28T03:01:09.880932Z 5 [System] [MY-013381] [Server] Server upgrade from '50700' to '80025' completed.
...
2023-10-28T03:01:13.905459Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.25-15' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona Server (GPL), Release 15, Revision a558ec2.

Discussion

While we were contemplating if this is a feature or a bug, MySQL release 8.0.28 refactored the related innodb_open_files code. Further details are provided in the corresponding open source commit WL#14591 InnoDB: Make system variable innodb_open_files dynamic:

- The `innodb_open_files` system variable can now be set with a dynamic SQL procedure `innodb_set_open_files_limit(N)`. If the new value is too low, an error is returned to client with the minimum value presented. If the value is out of bounds or of incorrect type, it will be reported as error also.
- `Fil_system::set_open_files_limit` was added to allow changes to the global opened files limit. The `Fil_system::m_max_n_open` is atomic now and extracted to a separate class `fil::detail::Open_files_limit`, instantiated as `Fil_system::m_open_files_limit`.
...
- `Fil_shard::reserve_open_slot`, Fil_shard::release_open_slot and static Fil_shard::s_open_slot were removed. Now we have CAS-based system of assuring the opened files will not exceed the limit set.

Thus, the new MySQL 8.0.28 feature – dynamic innodb_open_files variable – eliminated the need for open files limit adjustments in preparation for MySQL 8.0 upgrade.

Conclusions

Lessons learned:

Prepare for MySQL 8.0 upgrade in-place by taking a backup of the data directory.
Take advantage of the Percona Server open source code.
Follow guidance and advice posted in Percona blogs.

The content in this blog is provided in good faith by members of the open source community. Percona has not edited or tested the technical content. Views expressed are the authors’ own. When using the advice from this or any other online resource test ideas before applying them to your production systems, and always secure a working back up.

Our Top Picks from the Kubernetes 1.29 Release

Sergey Pronin — Fri, 12 Jan 2024 00:00:00 UTC

The latest Kubernetes version, 1.29, was released on December 13th 2023. Inspired by the Mandala and symbolizing universal perfection, it concludes the 2023 release calendar. This version comes with various exciting improvements, many of which will be helpful for users who run databases on Kubernetes.

Figure 1 - Mandala created in Excalidraw, not perfectly symmetrical.

Here, we highlight this latest release’s four key features and improvements. Let’s take a look at them together.

In-Place Update of Pod Resources

This alpha feature allows users to change requests and limits for containers without restarting. It simplifies scaling by a lot and opens new opportunities for auto scaling tools like HPA, VP, and Kubernetes Event-driven Autoscaling (KEDA). It removes the barriers of scaling the applications that were not easy to restart.

When resource resizing is not possible in-place, there are clear strategies for users and controllers (like StatefulSets, JobController, etc.) to handle the situation effectively.

It was first introduced in 1.27 but moved back to alpha as it requires additional architectural changes. It also has performance improvements and comes with Windows containers support. Read more about this in its Kubernetes Enhancement Proposals (KEP) and the issue #1287 created to add this feature.

Kubernetes VolumeAttributesClass ModifyVolume

The Kubernetes v1.29 release introduces an alpha feature enabling modification of volume attributes, like IOPS and throughput, by altering the volumeAttributesClassName in a PersistentVolumeClaim (PVC). This simplifies volume management by allowing direct updates within Kubernetes, avoiding the need for external provider API management. Previously, users had to create a new StorageClass resource and migrate to a new PVC; now, changes can be made directly in the existing PVC.

Discover further details in the KEP and issue #1287, which was established for the inclusion of this feature.

ReadWriteOncePod PersistentVolume Access Mode

Kubernetes offers access modes for Persistent Volumes (PVs) and Persistent Volume Claims (PVCs), including ReadWriteOnce, ReadOnlyMany, and ReadWriteMany. In particular, ReadWriteOnce restricts volume access to a single node, enabling multiple pods on that node to read from and write to the same volume concurrently. This setup ensures exclusive volume access on a per-node basis while allowing shared volume usage within the node. However, this introduces a potential issue, especially for applications that require exclusive access by a single pod. In this release, the ReadWriteOncePod access mode for PersistentVolumeClaims has become stable. Now that it is stable, a PVC can be configured to be mounted by a single Pod exclusively.

Here are the Kubernetes Enhancement Proposal (KEP) and issue #2485 that led to the development of this feature.

Make Kubernetes aware of the LoadBalancer behavior

kube-proxy’s handling of LoadBalancer Service External IPs is set to change. Traditional methods, such as IPVS and iptables, bind these IPs to nodes, optimizing traffic but causing issues with certain cloud providers and bypassing key Load Balancer features.

There are numerous problems with existing behavior:

Some cloud providers (Scaleway, Tencent Cloud, …) are using the LB’s external IP (or a private IP) as source IP when sending packets to the cluster. This is a problem in the ipvs mode of kube-proxy since the IP is bounded to an interface, and healthchecks from the LB is never coming back.
Some cloud providers (DigitalOcean, Scaleway, …) have features at the LB level (TLS termination, PROXY protocol, …). Bypassing the LB means missing these features when the packet arrives at the service (leading to protocol errors).

The solution would be to add a new field in the loadBalancer field of a Service’s status, like ipMode. This new field will be used by kube-proxy in order to not bind the Load Balancer’s External IP to the node (in both IPVS and iptables mode). The value VIP would be the default one (if not set, for instance), keeping the current behavior. The value Proxy would be used to disable the shortcut. This change allows more flexible handling of External IPs, maintaining current behavior as the default and offering an alternative to avoid these issues.

Read more about this in its Kubernetes Enhancement Proposals (KEP)

If you are interested in learning about databases on Kubernetes, you can start by running MySQL in Kubernetes. Explore the solutions, and weigh the pros and cons. Also, discover our predictions for Cloud Native technologies for this year.

Data on Kubernetes Community initiatives: Automated storage scaling

Sergey Pronin — Wed, 10 Jan 2024 00:00:00 UTC

In the world of Kubernetes, where everything evolves quickly. Automated storage scaling stands out as a critical challenge. Members of the Data on Kubernetes Community have proposed a solution to address this issue for Kubernetes operators.

If, like me, this is your first time hearing about Automated storage scaling, this will help you understand it better:

Storage scaling in Kubernetes Operators refers to the ability of an application running on Kubernetes to adjust its storage capacity automatically based on demand. In other words, it is about ensuring that an application has the right amount of storage available at any given time, optimizing for performance, cost, and operational efficiency, and doing this as automatically as possible.

As databases grow increasingly integral, the absence of unified solutions for storage scaling is becoming more evident. Let’s explore some existing solutions and their limitations:

pvc-autoresizer

This project detects and scales PersistentVolumeClaims (PVCs) when the free amount of storage is below the threshold. pvc-autoresizer It is and active open source project on GitHub.

There are certain downsides:

Works with PVCs only. It does not work with StatefulSet and does not have integration with Kubernetes Operator.
It requires Prometheus stack to be deployed.

Percona wrote a blog post about pvc-autoresizer to automate storage scaling for MongoDB clusters on Kubernetes.

EBS params controller

This controller provides a way to control IOPS and throughput parameters for EBS volumes provisioned by EBS CSI Driver with annotations on corresponding PersistentVolumeClaim objects in Kubernetes. It also sets some annotations on PVCs backed by EBS CSI Driver representing current parameters and last modification status and timestamps.

Find more about EBS params controller on GitHub.

Kubernetes Volume Autoscaler

This automatically increases the size of a Persistent Volume Claim (PVC) in Kubernetes when it is nearing full (either on space OR inode usage). It is a similar solution to pvc-autoresizer. Check out more about Kubernetes Volume Autoscaler

Kubernetes Event-driven Autoscaling(KEDA)

KEDA performs horizontal scaling for various resources in k8s, including custom resources. The metric tracking component is already figured out, but unfortunately, it does not work with vertical scaling or storage scaling yet. We opened an issue in GitHub to start the discussion.

As you can see, there are some limitations to performing Automated storage scaling, and to address this gap, the Data on Kubernetes community wants to develop a solution that solves practical problems and contributes to the open source community.

We’re tackling the significant challenge of unexpected disk usage alerts and potential system shutdowns due to insufficient volume space, a common issue in Kubernetes-based databases.

Possible Solutions

The following possible solutions were proposed:

Operators must be capable of changing the storage size when Custom Resource is changed.
Operators must create resources following certain standards, like applying annotations with indications of which fields should be changed
3rd party component (Scaler) will take care of monitoring the storage consumption and changing the field in the CR of the DB

Our goal as a community is to develop a fully automated solution to prevent these inconveniences and failures.

Final Thoughts

Once a new solution is validated and proven functional, it will benefit many communities, enabling them to integrate it with their operators. Additionally, it will present an excellent opportunity for Percona to incorporate it into our Operators, enhancing efficiency and facilitating automated storage scaling.

We invite those interested, especially in this particular project, to join us. This is an opportunity to be at the forefront of shaping the automated scaling solutions in Kubernetes. You can join the Data on Kubernetes community on Slack, specifically on the #SIG-Operator.

Are you interested in understanding Storage Autoscaling in databases? Explore our detailed example of Storage Autoscaling using the Percona Operator for MongoDB. For questions or discussions, feel free to join our experts on the Percona Community Forum

Volunteering as a Program Committee Member for Data on Kubernetes Day Europe 2024

Edith Puclla — Wed, 10 Jan 2024 00:00:00 UTC

The Data on Kubernetes Day Europe 2024 Program Committee is a group of professionals and experts responsible for organizing the Data on Kubernetes Day Europe 2024 content for the upcoming co-located events at Kubecon in Paris on 19 March.

As Data on Kubernetes community members, Sergey Pronin (Group Product Manager at @Percona) and I (Tech Evangelist) volunteered to evaluate proposal topics submitted for the event through the Sessionize platform. Not only us but also many other members of the Data on Kubernetes community participated as volunteers.

Community members who participate in this Program Committee evaluate proposals for talks, workshops, and other sessions submitted by potential speakers. This involves identifying each submission’s relevance, quality, and originality and being completely transparent and honest when reviewing a set of talks for the Data On Kubernetes Community co-located event.

Being a program committee means adhering to guidelines and following the Linux Foundation’s code of conduct.

Be professional and courteous.
Even express feedback constructively, not destructively.
Be considerate when choosing communication channels

After all program committee members completed their evaluations, the Data on Kubernetes Day Co-located Events Europe 2024 schedule was announced.

Look at this promising agenda:

We feel great to be a part of these efforts and to contribute to something significant by making it a reality at an in-person event. Thanks to the Linux Foundation for recognizing our efforts as Program Committee Members and for considering Percona, an active member of the DoK community.

In recognition of this support, we earned a badge from The Linux Foundation.

If you want to know more initiatives Percona have in the DoK community, read Data on Kubernetes Community initiatives: Automated storage scaling.

If you have any questions, remember to visit our Percona Community Forum.

Percona Bug Report: November 2023

Aaditya Dubey — Tue, 19 Dec 2023 00:00:00 UTC

In this edition of our bug report, we have the following list of bugs,

Percona Server/MySQL Bugs

PS-8086 : Increased memory usage in LRU manager with ROW_FORMAT=COMPRESSED, so it seems that after evicting uncompressed frames for a compressed table, Percona Server LRU manager uses more memory to track them than upstream MySQL.

Reported Affected Version/s: 5.7.x, 8.0.26, 8.0.32

PS-8737 / [Bug #110706] : Data will be lost when you perform table rebuild immediately after INSERT or DELETE commands. This means that all ALTER TABLE operations that require table rebuild, including a “null” alteration; that is, an ALTER TABLE statement that “changes” the table to use the storage engine that it already has Eg: “ALTER TABLE t1 ENGINE = InnoDB;”, So after INSERT and DELETE, please do not execute table rebuild statement immediately.You can find the full list of ALTER operations that require table rebuild at https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-operations.html Check for column “Rebuilds Table”. All operations for which this column contains “Yes” or “No*” are affected.

Reported Affected Version/s: 8.0.[28/29/30/31/32]

PS-8428 : ALTER TABLE t ADD FULLTEXT crashes the server when –innodb_encrypt_online_alter_logs=ON. The problem has nothing to do with either innodb_encrypt_online_alter_logs, or Parallel Threads for Online DDL Operations. The issues turned out that in binaries built with OpenSSL 3.0.x my_aes_crypt() function has a flaw and can no longer decrypt data encrypted with the same function previously.

Reported Affected Version/s: 8.0.30-22

Fixed version: 8.0.30-22

Please don’t get confused with the affected & fix version is the same for this bug since this bug was reported internally during testing of the release build & that’s the reason it gets the same affected & fix version.

PS-8987 / [Bug #112935] : This bug results in inconsistency seen between MYISAM and MEMORY for simple CREATE and SELECT operation.

Please check the below scenario where result inconsistency is seen:

Added sample data to MyISAM/InnoDB Tables.
Executed SELECT statement which is a bit complex so can’t be added here but it can be seen in the bug report.
Empty set return when SELECT executed against MyISAM/InnoDB Tables & 4 rows return when same query executed against Memory Engine.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27

PS-8990 / [Bug #112979] : MySQL server does not respect system variable binlog_transaction_compression_level_zstd so it sets the compression level for binary log transaction compression on this server, which is enabled by the binlog_transaction_compression system variable. The value is an integer that determines the compression effort, from 1 (the lowest effort) to 22 (the highest effort). If you do not specify this system variable, the compression level is set to 3. As the compression level increases, the data compression ratio increases, which reduces the storage space and network bandwidth required for the transaction payload.

Reported Affected Version/s: 8.0.34-26, 8.0.35-27

PS-9015 / [Bug #113256] : “DATA_FREE” shows a different value when comparing information_schema.TABLES vs information_schema.PARTITIONS. It is hard to say which result set is correct since we don’t have a source of truth.

Reported Affected Version/s: 5.7.43-47, 8.0.34-26

PS-9011 / [Bug #112946] : Prior to 8.0.29 INSTANT column exists on a non-system table with NULL columns in the MySQL SCHEMA which eventually leads to a corruption post 8.0.30+ upgrades. Although it’s probably not a best practice to create tables in mysql SCHEMA, it should not lead to corruption, especially when INSTANT is the default algorithm.

Percona Xtradb Cluster

PXC-4343 : Occasionally, during SST, InnoDB tablespace gets silently corrupted, resulting in the later Xtrabackup failure with the following error [MY-012224] [InnoDB] Header page contains inconsistent data in datafile. The triggering condition appears to be PXC 5.7 => 8.0 upgrade, where the corruption manifests in a 2nd node that joins later from the upgraded node. The corruption gets discovered once the 3rd node tries to join from the 2nd node as the donor or when a regular backup is taken from the 2nd node. To workaround this issue, always specify the first upgraded node as a donor.

Reported Affected Version/s: 8.0.34-26

PXC-4237 : When adding a new node, with PXC tarball installation error is being reported saying [WSREP] Failed to read 'ready ' from: wsrep_sst_xtrabackup-v2. However, this issue is expected to be fixed by the upcoming release of PXC 8.0.35, and fortunately, below workaround can fix the issue quickly:

Create /var/run/mysqld/ folder owned by mysql OS user.

Backup wsrep_sst_common file before editing:

shell> cp -nvp /usr/local/mysql/bin/wsrep_sst_common /usr/local/mysql/bin/wsrep_sst_common.orig.for-bug-PXC-4237

Implement the change:

shell> sed -i '297s/.*/set +e; & ; set -e/' /usr/local/mysql/bin/wsrep_sst_common

Verify the changes:

shell> sed -n '297p' /usr/local/mysql/bin/wsrep_sst_common.orig.for-bug-PXC-4237
 MYSQLD_PATH=$(readlink -f /proc/${WSREP_SST_OPT_PARENT}/exe)
shell> sed -n '297p' /usr/local/mysql/bin/wsrep_sst_common
set +e; MYSQLD_PATH=$(readlink -f /proc/${WSREP_SST_OPT_PARENT}/exe) ; set -e

Reported Affected Version/s: 8.0.32-24, 8.0.34-26

PXC-4318 : PXC cluster stalls and eventually crashes due to a long semaphore wait, which is happening because ha_commit_low does not commit a transaction that does not perform any changes such as an empty transaction and it can’t be controlled since it is an internal process. Fortunately, the upcoming release of PXC 8.0.35 will fix the issue.

Reported Affected Version/s: 8.0.33-25

PXC-4336 : PXC node eviction when a new CHECK CONSTRAINT is created which violates the condition, Eg. table is created with one entry say id 100 and after creation we added CHECK CONSTRAINT using ALTER TABLE t ADD CONSTRAINT CHK_id CHECK (id <=75); Here 100 is not less than 75 which violate the conditions and eventually node become inconsistent and get disconnected/evicted from the cluster. To avoid this scenario make sure to avoid violation of CHECK CONSTRAINT conditions.

Reported Affected Version/s: 8.0.34-26

PXC-4034 : When PXC cluster uses as a source and an async replication as a replica where “set @@session.sql_log_bin = off;” is used, this introduces a GTID gap in the “gtid_executed” set on the PXC source. As a workaround to the issue avoid using statement with “set @@session.sql_log_bin = off;” in the source/PXC

Reported Affected Version/s: 5.7.38-31.59, 8.0.28-19, 8.0.34-26

Percona Toolkit

PT-1724 : Percona toolkit unable to work if user using ‘caching_sha2_password’ Authentication plugin

Reported Affected Version/s: 3.0.13, 3.5.5

PT-2030 : pt-heartbeat is not compatible with PostgreSQL throwing Cannot get MySQL var character_set_server: DBD::Pg::db selectrow_array failed: ERROR: syntax error at or near “LIKE” LINE 1: SHOW VARIABLES LIKE ‘character_set_server’

Reported Affected Version/s: 3.3.1, 3.5.5

PT-2083 : when running pt-archiver with –charset option in MySQL 8.0 does not work.

Reported Affected Version/s: 3.3.1, 3.5.5

Fixed version: 3.5.6 [Pending Release]

PT-2106 : In pt-online-schema-change adding column to table (parent table) with having foreign key reference which triggers rebuilding constraints and can cause inconsistencies.

Reported Affected Version/s: 3.3.1, 3.5.5

PT-2207 : pt-archiver doesn’t work when ANSI_QUOTES is set in sql_mode

Reported Affected Version/s: 3.5.2, 3.5.5

Fixed version: 3.5.6 [Pending Release]

Percona Monitoring and Management (PMM)

PMM-4712 : PMM frequently crashes due to out of memory kills with postgres_exporter consuming 20-30GB of RAM and to debug it pprof endpoints to postgres_exporter was missing

Fixed version: 2.41.0 [Pending Release]

PMM-12013 : rds_exporter unreliable for large deployments which generate gaps in the gathered metrics and some improvement fix done here at PMM-11727

Reported Affected Version/s: 2.35.0

PMM-12349 : ReplicaSet Summary shows wrong data when a node is gone

Reported Affected Version/s: 2.40.1

PMM-12631 : Route of /logs.zip crashes with reflect: call of reflect.Value.NumField on string

Reported Affected Version/s: 2.40.1 Fixed version: 2.41.0 [Pending Release]

PMM-12738 : File certificate.conf is required, but not mentioned anywhere in helm charts but in fact, it should not be required.

Reported Affected Version/s: 2.40.1

Percona XtraBackup

PXB-2860 : Xtrabackup keeps locking table even using –tables-exclude and –lock-ddl-per-table.

Reported Affected Version/s: 8.0.33-28 Fixed version: 8.0.34-29

PXB-3168 : Under high write load, backup fails with “log block numbers mismatch” error

Reported Affected Version/s: 8.0.33-28, 8.0.34-29 Fixed version: 8.0.35-30, 8.2

PXB-3079 : Prepare skips rollback on encrypted tables and completes successfully if the keyring plugin is not loaded.

Reported Affected Version/s: 8.0.33-27 Fixed version: 8.0.34-29

PXB-3147 : Xtrabackup failed to execute query ‘DO innodb_redo_log_consumer_register(“PXB”); if sql_mode=’ANSI_QUOTES’ is used.’ This results in the Xtrabackup execution failure.

Reported Affected Version/s: 8.0.33-28 Fixed version: 8.0.35-30, 8.2

PXB-2954 : Xtrabackup failing with “[ERROR] [MY-011825] [Xtrabackup] innodb_init(): Error occurred” to prepare in case of orphan ibd

Reported Affected Version/s: 8.0.28-21 Fixed version: 8.0.32-26

Percona Kubernetes Operator

K8SPG-404 : Upgrade from percona PostgreSQL Operator 1.3 to 1.4 is ending up with a cluster without any replicas.

Reported Affected Version/s: 1.4.0 Fixed version: 1.5.0 [Pending Release]

K8SPG-420 : Ending up in multiple shared repo after cluster pause and unpause.

Reported Affected Version/s: 1.4.0 Fixed version: 1.5.0 [Pending Release]

K8SPG-435 : Pod is recreated when /tmp is filled

Reported Affected Version/s: 2.2.0 Fixed version: 2.3.0 [Pending Release]

K8SPG-443 : Only english locale is installed, missing other languages support in Postgres

Reported Affected Version/s: 2.2.0 Fixed version: 2.3.0 [Pending Release]

K8SPG-453 : pg_stat_monitor hangs primary instance and it’s impossible to disable it

Reported Affected Version/s: 2.2.0 Fixed version: 2.3.0 [Pending Release]

Summary

For the most up-to-date information, be sure to follow us on Twitter, LinkedIn, and Facebook.

Quick References:

Percona Monitoring and Management 2.41 preview release

Ondrej Patocka — Wed, 06 Dec 2023 00:00:00 UTC

Percona Monitoring and Management 2.41 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.41 is now available as a preview release.

To see the full list of changes, check out the PMM 2.41 Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.41.0-rc

PMM client package installation

Download the latest pmm2-client release candidate tarball for 2.41.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.41.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-0a04085f4c721e913

The Importance of Anti-Affinity in Kubernetes

Edith Puclla — Thu, 30 Nov 2023 00:00:00 UTC

Last week, I embarked on the task of deploying our Percona Operator for MongoDB in Kubernetes. After completing the deployment process, I noticed that the status of the Custom Resource Definition for Percona Server for MongoDB was still displaying as ‘initializing’ and two of our Pods remained in a Pending state.

bash
edithpuclla@Ediths-MBP % kubectl get perconaservermongodbs.psmdb.percona.com -n mongodb
NAME ENDPOINT STATUS AGE
my-db-psmdb-db my-db-psmdb-db-mongos.mongodb.svc.cluster.local initializing 4m58s

bash
edithpuclla@Ediths-MBP % kubectl get pods -n mongodb
NAME READY STATUS RESTARTS AGE
my-db-psmdb-db-cfg-0 2/2 Running 0 109m
my-db-psmdb-db-cfg-1 2/2 Running 0 108m
my-db-psmdb-db-cfg-2 0/2 Pending 0 107m
my-db-psmdb-db-mongos-0 1/1 Running 0 106m
my-db-psmdb-db-mongos-1 1/1 Running 0 106m
my-db-psmdb-db-rs0-0 2/2 Running 0 109m
my-db-psmdb-db-rs0-1 2/2 Running 0 108m
my-db-psmdb-db-rs0-2 0/2 Pending 0 107m
my-op-psmdb-operator-77b75bbc7c-qd9ls 1/1 Running 0 118m

Upon further inspection of the pod in pending status, I discovered a clear indicator of the error:

bash
kubectl describe pod my-db-psmdb-db-cfg-2 -n mongodb
Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal NotTriggerScaleUp 3m53s (x62 over 13m) cluster-autoscaler pod didn't trigger scale-up:
 Warning FailedScheduling 3m27s (x4 over 13m) default-scheduler 0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules. preemption: 0/2 nodes are available: 2 No preemption victims found for incoming pod..

I took a closer look at the YAML configuration of our CRD in the Replsets section, particularly drawn to the Affinity subsection.

kubectl describe perconaservermongodbs.psmdb.percona.com my-db-psmdb-db -n mongodb

Here’s what I discovered:

Affinity and Anti-Affinity are key parts of the scheduling process in Kubernetes, and both focus on ensuring that Pods are correctly assigned to Nodes in the cluster. You can configure a Pod to run on a specific node or group of nodes. There are several ways to achieve this, nodeSelector is the simplest way to constrain Pods to nodes with specific labels. Affinity and anti-affinity expand the types of constraints you can define and give you more flexibility.

Let’s explore what it means to specify anti-affinity rules for the ReplicaSets.

The key kubernetes.io/hostname is a well-known label in Kubernetes that is automatically assigned to each node in the cluster. It usually holds the value of the node’s hostname. When used as a topology key in anti-affinity rules, it implies that the rule should consider the hostname of the nodes. In simpler terms, it’s telling Kubernetes to not to schedule the pods of this ReplicaSet on the same physical or virtual host (node).

If we review our cluster, it has two nodes for installing the Percona Operator for MongoDB.

bash
edithpuclla@Ediths-MBP ~ % kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-mongo-operator-test-default-pool-7c118de9-b9vc Ready  68m v1.27.4-gke.900
gke-mongo-operator-test-default-pool-7c118de9-ts16 Ready  68m v1.27.4-gke.900

In the context of databases like MongoDB, High availability is often achieved through replication, ensuring that the database can continue to operate even if one or more nodes fail. Within a MongoDB Replica Set, there are multiple copies of the data, and these copies are hosted on different Replica Set members. The default HA MongoDB topology is a 3-member Replica Set. Percona Operator for MongoDB deploys MongoDB in the same topology by default. With anti-affinity set to kubernetes.io/hostname, that means at least 3 Kubernetes worker nodes are needed to deploy MongoDB.

We created the minimum three nodes that the Percona Operator for MongoDB needs. We see that we don’t have the error with Antiaffinity in the Pods because each Pod was located appropriately in different nodes. Now our operator and our database were deployed correctly.

bash
edithpuclla@Ediths-MBP ~ % kubectl get nodes
NAME STATUS ROLES AGE VERSION
gke-mongo-operator-test-default-pool-e4e024a8-1dj3 Ready  76s v1.27.4-gke.900
gke-mongo-operator-test-default-pool-e4e024a8-d6j2 Ready  74s v1.27.4-gke.900
gke-mongo-operator-test-default-pool-e4e024a8-jkkr Ready  76s v1.27.4-gke.900

If we list all the resources in our namespace, we can see that all pods are running properly and all the resources have been created.

bash
edithpuclla@Ediths-MBP ~ % kubectl get all -n mongodb
NAME READY STATUS RESTARTS AGE
pod/my-db-psmdb-db-cfg-0 2/2 Running 0 4m40s
pod/my-db-psmdb-db-cfg-1 2/2 Running 0 4m2s
pod/my-db-psmdb-db-cfg-2 2/2 Running 0 3m20s
pod/my-db-psmdb-db-mongos-0 1/1 Running 0 2m56s
pod/my-db-psmdb-db-mongos-1 1/1 Running 0 2m39s
pod/my-db-psmdb-db-rs0-0 2/2 Running 0 4m39s
pod/my-db-psmdb-db-rs0-1 2/2 Running 0 3m59s
pod/my-db-psmdb-db-rs0-2 2/2 Running 0 3m28s
pod/my-op-psmdb-operator-77b75bbc7c-q2rqh 1/1 Running 0 6m47s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-db-psmdb-db-cfg ClusterIP None  27017/TCP 4m40s
service/my-db-psmdb-db-mongos ClusterIP 10.72.17.115  27017/TCP 2m56s
service/my-db-psmdb-db-rs0 ClusterIP None  27017/TCP 4m39s

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-op-psmdb-operator 1/1 1 1 6m47s

NAME DESIRED CURRENT READY AGE
replicaset.apps/my-op-psmdb-operator-77b75bbc7c 1 1 1 6m47s

NAME READY AGE
statefulset.apps/my-db-psmdb-db-cfg 3/3 4m41s
statefulset.apps/my-db-psmdb-db-mongos 2/2 2m58s
statefulset.apps/my-db-psmdb-db-rs0 3/3 4m40s

In conclusion, affinity and anti-affinity in Kubernetes are tools for strategically placing pods in a cluster to optimize factors such as performance, availability, and compliance, which are critical for the smooth and efficient operation of containerized applications, also setting up anti-affinity rules like failure-domain.beta.kubernetes.io/zone in Kubernetes is a key strategy for keeping clusters running, especially in production environments. This approach spreads pods across different availability zones, which means if one zone has an issue, the others can keep the system running. It’s a smart way to ensure your cluster can handle unexpected outages, making it a popular choice for those who need their Kubernetes setups to be reliable and available at all times.

Learn more about our Percona Operator for MongoDB, and if you have questions or comments, you can write to us on our Percona Community Forum.

Day 02: The Kubernetes Application Lifecycle

Edith Puclla — Mon, 20 Nov 2023 00:00:00 UTC

If you are in the world of application development, you know that every application has a lifecycle. An application lifecycle refers to the stages that our application goes through from initial planning, building, deployment, monitoring, and maintenance in different environments where our application can be executed.

On the other hand, the Kubernetes Application Lifecycle refers exclusively to applications deployed and managed in Kubernetes clusters. This differs from the normal application lifecycle because Kubernetes introduces new principles, practices, and tools for managing applications on containers.

In this blog post, we will talk about these phases Day 0, Day 1 and Day 2 in the lifecycle of an application in Kubernetes and we will focus specifically on the phase of Day 2.

Image 1: Day 0, Day 1 and Day 2 in the Kubernetes Application Lifecycle

Day 0 refers to the preparation stage before deploying applications in Kubernetes. It’s the stage of identifying goals, planning the infrastructure. Ensuring that the development team has knowledge about Kubernetes and best practices. It’s a stage for investment in training. And the evaluation of the application components to determine which are suitable for use within containers and Kubernetes.

Day 1 is the stage that involves deploying the application in Kubernetes clusters and the creation of Kubernetes resources: deployments, pods, services. Additionally, it includes configuration management and the implementation of basic monitoring following the decisions made on Day 0.

Finally Day 2, our application is already running in Kubernetes clusters by reaching this stage. Day 2 refers to the management, monitoring, and optimization of our Kubernetes clusters over the long term.

Day 2 involves:

Gathering information from our Kubernetes clusters through monitoring and logging.
Scaling our application, either horizontally or vertically.
Application of security best practices and compliance with policies.
Establishing backups and recovery processes to protect our data and application from future disasters.

Day 2, activities focus on sustainability, efficiency, and long-term continuous improvement to ensure the stability of our application and meet customer expectations.

Let’s see how Percona takes charge of Day 2

For Percona, a company specializing in the management of open source databases like MySQL, PostgreSQL, MongoDB, Day 2 refers to the ongoing efforts to ensure that database systems are running efficiently securely, and in alignment with business objectives.

Here are some examples of how Percona handles this phase:

To achieve Performance Monitoring, if you use our Percona Kubernetes Operators, you can integrate it with Percona Monitoring and Management (PMM) to check the performance of your databases in real time. Monitor query execution times, resource utilization, and server health. PMM helps to identify bottlenecks and inefficiencies, allowing for timely optimization and tuning.

Image 2: this is what the PMM Dashboard interface looks like when monitoring your database resources.

If we discuss data protection and disaster recovery, using Percona XtraBackup, an open-source backup utility for MySQL-based servers. In that case, you can ensure that your database remains fully accessible during scheduled maintenance periods.

As for scaling strategy and high availability, adopting solutions such as Percona XtraDB Cluster or Percona Server for MySQL enables us to secure the database and efficiently manage increased workloads, all while keeping downtime to a minimum.

These were just some examples of what Percona does for Day 2 to maintain tasks crucial for the business and that relies on databases to keep critical applications and services running.

Are you interested in learning more about Kubernetes or need assistance with your cloud-native strategy? With Percona Kubernetes Operators, you can manage database workloads on any supported Kubernetes cluster running in private, public, hybrid, or multi-cloud environments. They are 100% open source, free from vendor lock-in, usage restrictions, and expensive contracts, and include enterprise-ready features by default. Learn more about Percona Kubernetes Operators

Data On Kubernetes

Edith Puclla — Fri, 10 Nov 2023 00:00:00 UTC

If you’ve attended one of the Kubecon talks or related events, you’ve likely encountered the phrase Data on Kubernetes. To understand what this means, let’s explore some fundamental concepts related to Kubernetes, workload, stateless, and stateful applications.

Kubernetes, workload, stateless and stateful applications

Kubernetes is a container orchestration tool that has already become an industry standard. When we talk about “container orchestration”, we are referring to the automated management and coordination of containers using Kubernetes.

Now, let’s explore what a workload is in the context of Kubernetes. A workload represents an application running on Kubernetes. An application may consist of a single component or multiple components working together. These components are packaged into containers operating within a group of Pods.

There are two types of workloads, depending on the nature of the application: Stateless and Stateful.

In a stateless application, the client session data is not stored on the server. This is because the application doesn’t need to retain past interactions to function. However, in a stateful application, storing client session data is essential as it is necessary for subsequent interactions within the application.

Now, we are already familiar with Kubernetes, workloads, stateless and stateful applications, and we also understand that Pods are responsible for managing these types of workloads.

Built-in Workload Resources in Kubernetes

In a Kubernetes cluster, we can have thousands of Pods, and we don’t need to directly manage them individually. Instead, we utilize workload resources to manage a group of Pods and choose what workload resource depends on the type of workload we are dealing with, Stateless or Stateful.

For example, if we have stateless applications, we can use the Deployment and ReplicaSet resources, which are well-suited for this type of workflow. On the other hand, the StatefulSet resource allows us to run Pods that need to maintain state.

Data on Kubernetes refers to the management and storage of data within the Kubernetes ecosystem. Kubernetes provides a robust framework for handling data, making it a versatile platform for both stateless and stateful applications while ensuring data durability, availability, and security.

The challenge

Kubernetes was initially designed to run stateless applications. However, the number of stateful applications running on Kubernetes has increased significantly. There are many challenges when it comes to running applications with state in Kubernetes, such as data management strategies, volume persistence, and others. According to the 2021 Data on Kubernetes report of more than 500 executives and technology leaders, 90% believe it is ready for stateful workloads, and a large majority (70%) are running them in production, with databases topping the list. This gives rise to initiatives aimed at standardizing the requirements for managing stateful applications on Kubernetes.

This is how Data on the Kubernetes Community emerges. The Data on Kubernetes (DoKc) community was established in spring 2020. It is an openly governed group of curious and experienced practitioners, drawing inspiration from the Cloud Native Computing Foundation (CNCF) and the Apache Software Foundation. They aim to help create and improve techniques for using Kubernetes with data.

There are several organizations that are part of the Data on Kubernetes community, and Percona is part of it as well. Percona is adding efforts in DoKC Operator SIG(Special Interest Groups), where we discuss gaps in information around K8s operators for the industry-at-large & co-creates projects to fill the gap. Watch the Kubernetes Database Operator Landscape panel discussion to learn more about the community efforts in Data on Kubernetes.

Conclusion

Data on Kubernetes is a crucial concept in the Kubernetes ecosystem. Kubernetes was initially designed for the stateless application, now faces the challenge of managing stateful workloads, and is more notable in databases. The Data on Kubernetes (DoKc) community has emerged to address these challenges and standardize the management of stateful applications, drawing inspiration from industry standards like CNCF and Apache Software Foundation.

If you want to be part of them, you are welcome to join DoKC. Also, check this outstanding Kubernetes Database Operators Landscape, where members of DoKC talk about operations for data workloads on Kubernetes.

Exploring Kubernetes Operators

Edith Puclla — Fri, 03 Nov 2023 00:00:00 UTC

The concept of Kubernetes Operators was introduced around 2016 by the CoreOS Linuxdevelopment team. They were in search of a solution to improve automated container management within Kubernetes, primarily with the goal of incorporating operational expertise directly into the software.

According to the Cloud Native Computing Foundation, “Operators are software extensions that use custom resources to manage applications and their components”.

Kubernetes is designed for automation, offering essential automation features. It can automatically deploy and manage workloads. The definition provided by CNCF regarding operators, as mentioned above, highlights the flexibility we have to customize the automation capabilities made possible by Kubernetes Operators using custom resources.

In Kubernetes, certain applications require manual attention as Kubernetes can not autonomously manage them. This is especially the case with databases. This is where Operators come into play. Databases are complex entities that also have complex database operations and features that Kubernetes itself may not inherently understand. While deploying a database in Kubernetes manually isn’t a problem, the true strength of operators shines during Day 2 operations, which include tasks such as backups, failover, and scaling. Operators automate these manual tasks for applications within Kubernetes.

The main challenge that arises when implementing containerized databases is the problem of data persistence. This is the challenge for containers in general, and it is more critical in the context of databases despite ongoing advances in container maturity. Kubernetes operators are designed to address this gap. While it is possible to use Kubernetes resources like Persistent Volume Claims (PVCs) without operators, operators simplify the process by providing a higher level of abstraction and automation.

It is possible to create new operators using the Kubernetes operator pattern concept. This allows you to extend cluster behavior without modifying the Kubernetes code by linking controllers to one or more custom resources. These Operators use and extend the Kubernetes API, a key component within the Kubernetes architecture, with the essential concepts for users to interact with the Kubernetes cluster. They create custom resources to add new functionality according to the needs of an application to be flexible and scalable. This is how we automate workloads using Kubernetes Operators.

One of the primary benefits of operators is the automation of repetitive tasks that are often managed by human operators, eliminating errors in application lifecycle management.

Conclusion

In this article, we explored an overview of what Kubernetes Operators are; we saw why they are necessary and the benefits of using them. I hope you have gained a general understanding of why Kubernetes Operators are valuable.

If you want to know more about Kubernetes operators designed specifically for databases, you can visit the Percona website, where you will find Kubernetes operators created by Percona for MongoDB, PostgreSQL, and MySQL.

Are you considering creating your own operator? Start by using the Operator-SDK. Additionally, you can watch Sergey Pronin’s (Group Product Manager At Percona) talk at the DoK Community about Migrating MongoDB to Kubernetes, where he discusses the reasons why Percona created an Operator for MongoDB.

Building and Running Percona Everest From Source Code

Daniil Bazhenov — Mon, 30 Oct 2023 00:00:00 UTC

Digging deeper into the architecture of an open source product

Recently, Percona team announced the public alpha version of a new open source product – Percona Everest. It allows you to create database clusters on Kubernetes cluster.

I have installed Percona Everest several times and tried its features. Standard installation is very simple and takes a few minutes.

But, to understand the product deeper, I came up with the idea to explore repositories, build and run Percona Everest from source.

In this post, I will explain what I did step by step and what components and frameworks are used in the development of Percona Everest.

Architecture, components, and tools

At the top level, we have two components or tools on the user side: Percona Everest App and everestctl CLI tool.

The Percona Everest App is a basic application that provides a web interface for database creation and management functions. Percona Everest App can be installed on your computer or remote server. The App consists of two major components:

Frontend is a browser-based application providing a web interface for managing clusters and interacting with backend APIs. It is developed with React and TypeScript.
Backend API that process requests from frontend, interact with Kubernetes clusters and databases. It is developed on Golang and PostgreSQL as a database.

everestctl is a CLI tool for provisioning of Percona Everest on Kubernetes clusters. It is used to install Percona Everest components such as database operators on the Kubernetes cluster. It is developed in Golang and is provided as a ready-made executable, but, in this post, we will also build it from source code.

Remember that normally, when you install Percona Everest following the instructions in the documentation, the frontend and backend are built and integrated into a single container and run as a single unit.

Let’s get started with our experiments.

Frontend installation and launch

Percona Everest Frontend is developed using the Bit framework.

Bit is an open source toolchain for the development of composable software using React library and TypeScript.

Bit is used by about 100K developers, 250+ community plugins and has 16K+ stars on GitHub.

Clone the Frontend repository

You need to clone a repository with the Percona Everest frontend: https://github.com/percona/percona-everest-frontend

git clone git@github.com:percona/percona-everest-frontend.git

Install Bit

Bit requires the npm package manager to install. npm is a popular registry of JavaScript packages and libraries. The npm registry contains over 800,000 code packages. Open source developers use npm to share software. Installing npm on your operating system is straightforward. I’m sure you can handle it. It installs along with Node.js.

Open the Percona Everest Frontend source directory and run the following commands.

npm i -g @teambit/bvm

bvm install 1.0.0

bit install --recurring-install

Launching the frontend application

Moving forward, we have two options:

Run the frontend application using Bit.
Build a ready application and copy it to the backend.

The Percona Everest frontend repository contains versioning branches release-[version] and the current version in development in the main branch. We will run the latest dev version from main.

Let’s run the command:

bit run everest --skip-watch

As a result, we can open localhost:3000 in the browser.

The Frontend is now built, and we can move on to the Backend.

Additional information

There is the other way to build a frontend to work with backend. You will need two commands:

bit snap --build

bit artifacts percona.apps/everest --out-dir build

In this case, the build will be done to the folder:

build/percona.apps_everest/artifacts/apps/react-common-js/everest/public/

You need to copy all the files to the public/dist folder of the backend repository. We will talk about backend in the next section.

The installation process may change over time, so I recommend to keep track of the up-to-date commands in the files:

README.md
Makefile
CI/CD of GitHub configuration, file .github/workflows/ci.yml in the repository.

Backend

So we’ve launched Frontend, and now it shows an error because it sends requests to the Backend API, and we don’t have it yet.

We will need to clone the repository with Percona Everest Backend

https://github.com/percona/percona-everest-backend

Percona Everest Backend is developed in Golang using the Echo framework. The Echo repository has over 26k stars on GitHub.

Generally, it is an API that interacts with the frontend, processing requests and sending them to the Kubernetes cluster.

Let’s get it up and running.

Run PostgreSQL locally

You need Docker to run it. I hope you have Docker installed.

Let’s run one of the two commands in the repository directory:

make local-env-up

docker-compose up --detach --remove-orphans

Using Docker for this process will be replaced by Kubernetes. You can see the YAML manifest in the file:

/deploy/quickstart-k8s.yaml

Launch the Go app

We have two options, I use:

go run cmd/main.go

But you can also use:

make run-debug

Starting with version 0.4.0, you will need to add the SECRETS_ROOT_KEY environment variable before starting the application. The secret key must be used on restarts if you do not start from the scratch. export SECRETS_ROOT_KEY=$(openssl rand -base64 32)

Now we can open the localhost:3000 in the browser again and check that the backend is running. But we see that we have no Kubernetes clusters connected and configured.

Everestctl and Kubernetes cluster

Another important component of Percona Everest is everestctl. everestctl is a CLI tool responsible for provisioning Percona Everest on Kubernetes clusters.

We will need:

Kubernetes cluster
Build and run everestctl

Preparation of the Kubernetes Cluster

You can use Kubernetes Cluster on AWS, Google Cloud, or minikube.

The Percona Everest documentation says:

You must have a publicly accessible Kubernetes cluster to use Percona Everest. EKS or GKE is recommended, as it may be difficult to make it work with local installations of Kubernetes such as minikube, kind, k3d, or similar products.

The documentation provides instructions on how to run the test cluster

In this post, we use minikube which is preliminarily installed.

The Makefile of the Percona Everest Backend repository contains the make k8s command to start the cluster in minikube.

Let’s open the directory of the backend repository and launch minikube:

make k8s-macos

make k8s

As a result, I see in the console this message:

🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

I also see that I have a minikube cluster with three nodes. It is time to install Percona Everest components using everestctl.

➜ percona-everest-backend git:(main) ✗ kubectl get nodes
NAME STATUS ROLES AGE VERSION
minikube Ready control-plane 3m44s v1.26.3
minikube-m02 Ready  3m22s v1.26.3
minikube-m03 Ready  3m3s v1.26.3

Build everestctl

Let’s clone the repository: https://github.com/percona/percona-everest-cli/

Open the repository folder and run build:

make build

everestctl is built in the binary file /bin/everest

Grant execution privileges

chmod +x ./bin/everest

Let’s run the Percona Everest installation:

./bin/everest install operators

That command will start the installation wizard.

I will leave all default values, just pressing Enter. Otherwise, you can experiment with the settings, it is your choice.

As a result, the following processes will run on the cluster:

Creating namespace percona-everest.

Installing Operator Lifecycle Manager (OLM).
Installing Percona OLM Catalog.
Installing Percona Operators for the databases selected in the wizard.
Installing everest-operator operator.
Creating services and roles.

That’s it! We’ve installed Percona Everest completely. You can open it in a browser and create a database.

What’s next?

Try creating databases with different configurations.

Repeat the installation with a different cluster or settings.

If you face any problems or have ideas on how to improve components, create Issues on GitHub in the appropriate repositories.

Stop and remove Percona Everest

Once you finished your experiments, you can:

Stop Frontend - stop Bit running by pressing CTRL+C in the console.
Stop Backend API - stop Golang script by pressing CTRL+C in the console.
Stop and remove PostgreSQL in Docker - run make local-env-down in the backend repository, or use Docker Desktop to stop.
Remove Kubernetes cluster.

Updates

Every day, developers make changes to the code and publish to repositories on GitHub.

You can stop the component, pull the changes with git pull, and start a new version. It’s just for experimentation and development. Some versions of components will not be compatible; uninstall all components and start over using the appropriate versions. Detailed instructions on how to upgrade will appear later.

You can see changes to the build process or parameters in the repositories.

Couple of useful commands

List of databases

kubectl -n percona-everest get db

List of pods

kubectl -n percona-everest get pods

Conclusion

I hope you were able to make it this far, and it was interesting for you.

I’d love if you leave your feedback in the comments.

Kubernetes Community Days UK: Keynote Cilium and eBPF

Edith Puclla — Tue, 24 Oct 2023 00:00:00 UTC

This week, at Kubernetes Community Days UK in London. Liz Rice, Chief Open Source Officer at Isovalent, delivered a keynote on Cilium, eBPF, and the new feature of Cilium: Mutual Authentication.

Figure 1. Liz Rice Keynote KCD UK, London. Tuesday 17, 2023

Cilium is an eBPF-powered open source, cloud native solution for delivering, securing, and observing network connectivity between workloads.

eBPF is a technology that allows us to create modules to modify the behavior of the Linux kernel, but why would we want to change the Linux kernel?

Some use cases for observability, security, and networking require tracking and monitoring our application, but we don’t want to constantly modify our application with these changes. It’s better to add a program that can observe the behavior of our application from the kernel.

But changing the Linux kernel can be, well, hard.

Figure 2. Adding features to the kernel (cartoon by Vadim Shchekoldin, Isovalent)

However, eBPF enables you to modify the kernel’s behavior without directly altering the kernel itself. It might sound unconventional, but eBPF makes this possible through the creation of programs for the Linux kernel. The Linux kernel accepts eBPF programs that can be loaded and unloaded as needed.

Figure 3. Adding kernel features with eBPF (cartoon by Vadim Shchekoldin, Isovalent)

To ensure that these eBPF programs written by us are secure, there is a mechanism in place that allows these programs to be verified safe for execution. This can be seen in eBPF verification and security. You don’t have to restart the kernel to deploy or remove eBPF applications, which makes eBPF one of the technology tools of the moment.

Liz also announced that Cilium recently graduated from CNCF. This means Cilium is considered stable and has been successfully used in production environments.

Figure 4. CNCF Project Maturity Levels

After understanding what eBPF is, let’s move on to the actual topic of Liz’s keynote. She spoke about Mutual Authentication with Cilium.

Mutual Authentication with Cilium was the last significant feature missing from Cilium Service Mesh. It’s a somewhat more complex topic related to mTLS (Mutual transport layer security).

mTLS is a mechanism that ensures the authenticity, integrity, and confidentiality of data exchanged between two entities in the network.

In Cilium 1.14, one of the most significant releases, Cilium introduces support for a feature that many developers have requested: mutual authentication. This feature simplifies the process of achieving mutual authentication between two workloads. It now only requires adding two lines of code to the YAML in the Cilium Network Policy to authenticate communication between two workloads.

Slightly more complex, isn’t it? Let’s explore Mutual Authentication with Cilium in a second blog post very soon. We’ll also examine how this is related to Kubernetes and why it matters when running databases on Kubernetes.

Check what the other Graduated and Incubating Projects are at CNCF, and don´t forget to subscribe to our Percona Community Blog to read more about Open Source, CNCF Projects, and Database.

Percona Monitoring and Management 2.40 preview release

Taras Kozub — Tue, 03 Oct 2023 00:00:00 UTC

Percona Monitoring and Management 2.40 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.40 is now available as a preview release.

To see the full list of changes, check out the PMM 2.40 Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.40.0-rc

PMM client package installation

Download the latest pmm2-client release candidate tarball for 2.40.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.40.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-09895e9b605f14cbc

Open Source And Recession – An Economic Outlook

Ninad Shah — Thu, 14 Sep 2023 00:00:00 UTC

Initially, I dropped an idea of writing this blog by reckoning the amount of time and energy required to perform research. However, my mentor Ramona Gerum motivated me and provided inputs to write this blog. This would not have been possible without her help. Thank you very much Ramona.

Introduction

In October 2022, the term recession gained popularity on Google trends. This happened as many economists already predicted a slowdown, and it started flashing on news channels world-wide. In the following year, many giant corporations, such as Google, Microsoft, Amazon and others, announced massive layoffs; this was followed by other chain of events from stock market crashes to high inflation rates across the globe. This development reinforced people’s belief about recession.

Majority of organisations who ran several rounds of layoffs claimed to have overstaffing issues as the hiring was done to anticipate the upcoming tide of work that never came, and they did not fire any technical/software experts. In such cases, open source products often rise as an alternative to proprietary ones. This blog will focus on the effects of a recession on open source products.

As this blog focuses on recession, it is important to understand some economic terms to understand this blog properly.

GDP(Gross Domestic Product):

This is the gross monetary value of goods and services produced within a country. Every country produces various products, such as food, heavy machinery, oil and others. Those commodities would give some value upon selling them in the market. The gross total of the values of all such goods is GDP.

Generally, it is calculated on an annual basis.

GDP Growth:

The change in GDP compared to the previous GDP figure is GDP growth. In general, economists calculate them on a quarterly and yearly basis as they represent the economic position of a country and predict its growth in the near and distant future.

Developed countries:

This one is a little complex as there are more than one key factors that are decision-makers. For a country to become a developed one, all the factors from GDP to standard of living and HDI(Human Development Index) should be taken into consideration. In terms of economics, countries with a GDP of 12,000 USD(some experts put it at 24,000 USD) per capita(person) are considered a part of the emerged world.

Boom:

A period of good economic growth: rising number jobs, increasing wages, increased profitability. This is characterized as a short stint of wealth creation and accumulation.

Recession:

A period when economic growth is very low or nil. In this period, higher unemployment, lower demand in the job market, higher inflation and layoffs are commonly observed in such a period.

In economics, negative GDP growth for 2 consecutive quarters is perceived as a recession, and if the condition of recession persists for 3 years, it is called an economic depression. However, economists analyze other indicators as well to decide if the recession prevails or not.

Inflation rate:

An average increase in percentage in the gross price of various commodities from the previously obtained number. For example, if the milk was being sold at 1.5 USD in 2018, and in 2019, it is being sold at 1.55 USD, the price rise is nearly 3.33% that is the inflation rate of milk for a year. Usually, the gross percentage is considered to determine the condition of a nation. It is one of the important indicators.

Ideally, 3-4% inflation rate is considered to be a healthy one.
Higher than 5% affects the livelihood of people and many people would be pushed below the poverty line.
While it is lower than 2%, it should be considered that the economy is in a reverse gear, which is not good for the growth of any country.

Third-world countries:

This term is often used in a pejorative manner; it was originally used for those countries who did not prefer to align themselves with either the USSR(socialist) or the USA(capitalist) in the era of the cold war that surfaced after the second world war. However, as they did not choose any side, they failed to grow like developed countries, hence the term “third-world countries” became synonymous with underdeveloped or developing countries.

Causes of a recession

A recession is caused by any disruption in an existing established economic flow. As a country grows, various channels are established through which money flows from sources to sources, and that is how money keeps flowing through the system. Whenever the rotation of money is affected, it gradually affects the economy of a country.

For instance, consider the below existing channel.

Farmers grow crop
During crop harvesting seasons, farmers employ additional people
Farmers sell goods to wholesalers
Wholesalers sell it further to retail stores and business houses
Retail stores sell it consumers, and business houses produce goods for consumer

Here, if a country is affected by a drought and there is no other method of farming available, people in the above hierarchy from farmers to employees of retail stores and business houses will become jobless for that year and the flow of money gets disrupted. This is to note that food is an essential commodity, and due to crop failure, the demand for food items exponentially rises and the prices of food items also skyrocket.

Here, just to understand, effects of recession will not be visible immediately after the drought; it takes time to surface. Although the above one is just taken as an example, this is not a completely hypothetical scenario.

As we saw a classical example of drought, there are various phenomena that can drive the economy to recession. It completely depends on the economical condition of a country.

In the third-world countries:

This group is already facing a number of internal problems and depends on prosperous countries. They may or may not have adequate resources, however if they have, they do not know how to utilise them efficiently. Also, due to internal clashes, poverty and economic instability, they witness recessions very frequently. Some evident factors for recession in such countries are

Drought
Heavy rainfall
Wars
Epidemic
International sanctions
Political instability
Uncontrollable inflation
Corruption and scams
Strikes

In the developed countries:

The developed world is often perceived as the utopia as processes are well-established there. The scope of recession appears limited, however this is just one side of the coin. There is a dialogue in the movie Wall Street: “Greed is good! Greed is progress!….”. Why I quoted this here is because greed is hardwired in the human gene, which propels humans to earn more money and exploit the system’s loopholes. Over the period, it causes an economic slowdown. Some of the driving factors for recession in such countries are

Policy loopholes
Stock market crash
Bad loans
Wars
Epidemic

Also, any economic slowdowns in influential countries resonate world-wide. Considering the classical case of the great recession of 2008, it started due to the housing bubble that resulted in the subprime mortgage crisis. In other words, due to sudden hikes in the prices of houses, banks started approving loan amounts with cheaper mortgage assets/commodities. For example, the loan amount is 100K USD, while the mortgaged item is worth 10K USD only. Also, people with poor credit rating got loans. This resulted into a burst of a bubble, which was an impetus to economic turmoil, millions of job losses and closure of businesses.

Repetitions And Durations Of Recessions

As described in the last paragraph, reasons for recessions may vary from country-to-country, also recessions in various countries may not necessarily coincide. Hence, the stint and repetition pattern for every recession are different.

Earlier, it was believed that recessions and booms are followed by each other. However, over the period, the political structure and law system in developed countries have evolved in such a way that their economies may stay resilient against recession.

Taking the case of the US, before the 1980s, the period of economic slowdowns were too frequent, nearly 3-5 years. However, since the 80s, the gap between the 2 recessions has significantly widened.

The snippet below taken from Investopedia shows the period of recession(“NBER Recessions” and “Length of Recession(Months)”) in the USA.

The condition of recession may not prevail across multiple countries at the same time, having said that, the great recession of 2008 was the first case of global recession, which cannot be considered as an exceptional one as it was the beginning of the era of globalisation. New waves of recessions will have global impact.

Performance of open source during and after recessions

During every recession, it has been observed that organisations lean towards less costly products, so that they can cut down the cost and maintain their profitability. Open source software emerges as a boon for them because they can save licensing cost for the product cost.

The amount required to acquire the license is very high. During a recession period, it is not at all an affordable option for companies, having said that, companies may run without new development work for a certain period and may eliminate some staff members, however platforms and databases are very crucial; hence, they cannot be eliminated as it is tantamount to shutting down a company.

Deploying open source products in an environment is the most viable option in such cases as they may continue running their businesses.

A survey was conducted by Linux foundations on OSS(Open Source Software) in which nearly 431 people, who were from “Fortune 500 group of companies”, working from middle-level to top-level management participated. They were asked various questions about open source and its benefits. When they were asked about adopting open source products, their answers are as mentioned in the chart below.

Source – Page 10

By analysing the chart, we can see surges in demand for open source in both the dot-com bubble burst(2001) and the great recession (2008). Besides, a sharp rise from 2016 can be noticed as well. However, it can be seen that the demand for open source has always been moderate.

In this section, it can be seen that open source software is resilient against recessions and keeps on flourishing during and after odd times as well.

Can Open Source Keeps Recessions At Bay?

This is a really interesting question that requires further research. Since the last few years, companies have become more optimistic about open source software products, such as MySQL, PostgreSQL, Kubernetes and so on. This is because these products provide all those features that meet industry standards and requirements. Over the period, open source databases, OS and web/app servers have evolved by working with various industries, and now, they meet all the requirements that companies give. Due to which, open source successfully carved a niche in the market.

The noticeable thing is that organisations do not need to pay a penny to use these products. Also, if one can afford to hire a small team of developers, the company comes up with its own version of the product. In case this is not the viable option, there are communities and a number of small-scale to medium-scale companies that provide support of open source.

If we can focus on the cost of products, it would give a more clear view on open source’s ability to stem recession.

The case of 2008’s great recession:

We all have heard tales of 2008 recessions: heavy pay cuts, job losses, suicides, loss of property and so on and on and on. As I mentioned in this blog earlier with an example, a disruption in the flow of money causes recession, and in the case of 2008, banks sanctioned loans that cannot be repaid. Due to which, banks were defaulted and it had badly hit the economy of the USA, which affected the whole world.

The most peculiar thing the Washington Post highlighted about its impact and losses.

Loss in the global economy – nearly 2 trillion USD
The US stock market loss – around 8 trillion USD
Gross individual losses – around 9.8 trillion USD

A hypothetical scenario with an open source in 2008:

Had it changed anything if organisations had placed open source products instead of proprietary software products back in 2008? As we can understand, the stock market loss and individual losses cannot be recovered due to this, because it has no relation with the product in the backend or frontend.

To understand this part, we have to check out some revenues. The revenue of Microsoft was nearly 52 billion USD in 2008, and if we start looking into revenues of other proprietary software producing organisations, their collective revenue may hardly reach half of the global economy loss(2 trillion USD). If we consider all the losses incurred due to proprietary products only, even in that case, open source could not have staved off the recession as the number itself is very huge.

As per Statitsa, the gross revenue of open source services was 32 billion USD in 2022. Considering this figure, it could have definitely helped mitigate the situation up to certain extent in 2008, and many of jobs could have been saved and many organisations might have avoided closures.

Open source’s role after 2008:

Due to loopholes in policy, the situation of economic turbulence arose, hence the US government took corrective actions and introduced a number of overhauls in economic policies, which made the economy more stable and immune to unwanted slowdowns.

Along with these things, open source gained popularity and percolated into different industries. Many companies have moved away from proprietary products and adopted open source products as they found them cost-effective with all the required features available. Of course, a penny saved is a penny earned. However, it seems no proper analysis was made to notice the economical impact of open source’s entry in the market. Also upon looking at the potential of open source a number of giant corporations, such as Google, Microsoft and others, have started contributing to the open source community.

But, if those people who contribute to open source are actually getting benefited? The answer lies in the below survey result.

Source – Page 13

Many economists predicted recession in the US between 2015-2018, which did not actually happen. Also, while I am writing this blog, experts have already announced a recession in 2023-24, which is yet to come true. Though I firmly believe that open source is pushing the recession back, it is difficult to prove it in absence of surveys and data. I hope some experts may pick this topic and make a detailed analysis on this part; I am sure it will give a great push to the open source community.

Conclusion

Booms and recessions were repeating on regular intervals earlier, however over the period, due to economic advancements, recessions became infrequent and started repeating after several years. It is not necessary that all countries may face economic slowdown at the same time, also reasons for slowdowns may differ as well. During such a period, organizations have to bear loss and fall in revenues, however open source becomes more adoptive during recessions. Also, it is possible that open source is keeping recessions away, but there is no data to back this claim.

Dolphie, your real-time MySQL monitoring assistant

Charles Thompson — Tue, 22 Aug 2023 00:00:00 UTC

For as long as I can remember, Innotop has been the go-to terminal tool for real-time MySQL monitoring. It is an invaluable addition to any DBA’s toolkit, but unfortunately, it’s not really actively maintained these days, except for addressing critical issues, and it hasn’t kept pace with the evolving capabilities of modern terminals. With no viable alternatives except for InnotopGo, which is also no longer actively maintained and limited to MySQL 8 (while many still use 5.7), I decided to build my own in Python.

I call it, Dolphie

Initially, I relied on Python’s Rich package for the user interface. However, I recently stumbled upon Textual a few months ago, and it piqued my interest. It’s a framework that extends the capabilities of Rich, opening up a world of possibilities in the terminal. After experimenting with it for a few days, it inspired me to redevelop Dolphie with it, and I’ve been thoroughly pleased with the results. It has allowed me to showcase many of the features that will be displayed in this blog post!

Getting started

When you first start Dolphie, you’ll be greeted with a dashboard displaying various important MySQL metrics, along with a sparkline below it to measure the QPS (Queries per second) + process list. There are multiple ways to manipulate the process list, such as changing how it sorts, filtering by user/host/query text/database/time, killing threads, and much more.

There are currently four panels that can be toggled interchangeably for display:

Dashboard
Process list
Replication/Replicas
Graph Metrics

A big perk of transitioning to Textual is the integration of graphs. It’s as if I’ve incorporated a mini-PMM (Percona Monitoring and Management) right into Dolphie! The switches you see can be toggled on and off to display or hide their corresponding metrics on the graph.

Buffer Pool Requests Graph + Replication Panel

Checkpoint Graph

Redo Logs Graph

How are your redo logs performing? Dolphie shows you how much data is being written per second, the active count of redo logs (MySQL 8 only), and how much data is being written to it per hour (inspired by this blog post)

DML Graph

Thread data

Dolphie lets you display a thread’s information with an explanation of its query along + transaction history

Kill threads

Dolphie lets you terminate threads using a selected option. Notice how it autocompletes the input for you. This is a feature across the board. It will autocomplete any input that it can

Quick switch host

After using Dolphie extensively myself, I realized the need to simplify host switching. I found myself restarting it frequently just to change the host. This feature saves all the hosts you’ve connected to, allowing for autocomplete when you want to switch

Error log

In MySQL 8, I was delighted to see that the error log was in performance_schema. Of course, I had to support it! It has switches to toggle on/off event types and search functionality

Errant transactions

The Replicas panel will let you know if your replicas have any errant transactions and what they are

These are just some of the features that Dolphie has. There are many more that I haven’t covered, which you can discover for yourself and try out!

If you’d like to try Dolphie, it’s just a pip away:

pip install dolphie

I’m open to feedback and suggestions so don’t be a stranger :) If you’d like to contribute to the project, I’d be delighted to have you!

You can find Dolphie on its GitHub

PMM Client on Raspberry Pi 4

Wayne Leutwyler — Tue, 15 Aug 2023 00:00:00 UTC

This will be the third in my series of Percona Products on a Raspberry Pi. My previous posts:

Before I get started I would like to thank guriandoro, for the work he did in compiling the PMM Client tools in his 2021 blog.

My regular readers know how much I love the Raspberry Pi and MySQL. I have several hobby projects that collect data, and MySQL on Pi is great
solution!

I recently decided that I needed to be able to monitor my two MySQL Server.

The steps below should work on the Raspberry Pi 3+ and 4.

What better tool to use than PMM

In this blog we will be using PMM Client 2.27 and PMM Server version 2.39.

Prerequisites

Important Note: Your Raspberry Pi must be on Raspbian Bullseye.

I will assume if you have made it this far into the BLOG post, then you already have a running PMM Server. If not, you can read about installing a PMM Server here.

Add the pmm user to your MySQL Server

CREATE USER 'pmm'@'127.0.0.1' IDENTIFIED BY 'pass' WITH MAX_USER_CONNECTIONS 10;
GRANT SELECT, PROCESS, REPLICATION CLIENT, RELOAD, BACKUP_ADMIN ON *.* TO 'pmm'@'localhost';

The following steps should be run on your Raspberry Pi

Download the pre-complied pmm-client tools

These were complied based on 2.27 source code. You can see the steps I followed to compile the tools here.

cd
wget https://github.com/cetanhota/pi4-arm-pmm-client/archive/refs/heads/main.zip
unzip main.zip

Once we have the client unzipped its a good idea to verify that the tools will work on your Raspberry Pi.

cd ~/pi4-arm-pmm-client-main/raspbian-bullseye
./pmm-agent --version

You should get output like below.

ProjectName: pmm-agent
ProjectName: pmm-agent
Version: 2.27.0-84-g78b8519-dirty
PMMVersion: 2.27.0-84-g78b8519-dirty
Timestamp: 2023-08-16 00:57:34 (UTC)
FullCommit: 78b85198ad1e2c319d4012ef5ae338f389e2000a
Branch: main

If the pmm-agent displayed info like above, then you good to keep moving down the document. If you ran into issues please post in the comments and we can see what can be done to get you working. The only problem I ran in to was that the root user could not execute the client tools.

Setup install directories

We need to create the install directory and move files to the correct locations.

sudo mkdir -p /usr/local/percona
sudo mkdir -p /usr/local/percona/pmm2
sudo mkdir -p /usr/local/percona/pmm2/tools
sudo mkdir -p /usr/local/percona/pmm2/config
sudo mkdir -p /usr/local/percona/pmm2/exporters
sudo mkdir -p /usr/local/percona/exporters
sudo mkdir -p /usr/local/percona/exporters/collectors/textfile-collector/high-resolution
sudo mkdir -p /usr/local/percona/exporters/collectors/textfile-collector/medium-resolution
sudo mkdir -p /usr/local/percona/exporters/collectors/textfile-collector/low-resolution

Copy files into directories

cd ~/pi4-arm-pmm-client-main/raspbian-bullseye
sudo cp pmm-admin pmm-agent /usr/local/bin
sudo cp node_exporter vmagent /usr/local/percona/pmm2/exporters

Configure the PMM Client

Its time to configure and test the PMM Client for the MySQL server where we are install PMM Client.

Please make sure you have the following three items:

PMM Server Hostname or IP Address
PMM Server admin ID
PMM Server admin Password

sudo pmm-agent setup --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml\
 --server-address= --server-insecure-tls\
 --server-username= --server-password=

Now that the PMM Client has been configured, lets move on to starting the PMM Client.

Start the PMM Client

 sudo pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml

Verify by reviewing the client is running correctly. If all is well you should see output like:

INFO[2023-08-15T17:00:34.294-04:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml. component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=main
INFO[2023-08-15T17:00:34.295-04:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=main
INFO[2023-08-15T17:00:34.296-04:00] Using /usr/local/percona/pmm2/exporters/vmagent component=main
INFO[2023-08-15T17:00:34.296-04:00] Starting... component=client
INFO[2023-08-15T17:00:34.296-04:00] Starting local API server on http://127.0.0.1:7777/ ... component=local-server/JSON
INFO[2023-08-15T17:00:34.296-04:00] Connecting to https://admin:***@192.168.1.127:443/ ... component=client
INFO[2023-08-15T17:00:34.298-04:00] Started. component=local-server/JSON
INFO[2023-08-15T17:00:34.320-04:00] Connected to 192.168.1.127:443. component=client
INFO[2023-08-15T17:00:34.320-04:00] Establishing two-way communication channel ... component=client
INFO[2023-08-15T17:00:34.341-04:00] Two-way communication channel established in 21.349589ms. Estimated clock drift: -84.399159ms. component=client

At this point press ctl+c to stop the pmm-agent.

If you want to start the pmm-agent again and running the background use this command:

sudo nohup pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml &

You may choose a different startup process for the PMM Client based on personal preference.

Add pt-summary to the PMM Client

cd /usr/local/percona/pmm2/tools
sudo wget percona.com/get/pt-summary
sudo chmod ugo+x pt-summary

Let’s see summary in action:

Add your Raspberry Pi MySQL Server to PMM Server

I am sure most of you know how to add a server to your existing PMM Server but if you have not done this before here is what you need to do.

Connect to your PMM Server.
Use pmm-admin command to add server for monitoring.

/usr/local/percona/pmm2/bin/pmm-admin add mysql --service-name= --server-insecure-tls --server-url=https://:@localhost --username= --password= --host=

PMM Server Screen Shoot

In Closing

I really loved working on this little project. I hope you find some use for the information.

DoKC Operator SIG Update

Edith Puclla — Mon, 14 Aug 2023 00:00:00 UTC

Before our meeting, we started with a question to begin the morning: What board game or tabletop game have you played that you would recommend to others?

Itamar Marom suggests that Catan as a good board game, which takes a lot of time, super annoying when you lose, but generally a lot of fun. So highly suggested!

Other members like Hugh Lashbrooke and Robert Hodges prefer Monopoly, where you must reach a higher level of monopoly awareness to enjoy it fully. The idea is that you can make pretty much any deal with anyone else within the rules of monopoly. That’s when it gets fun.

Terraforming Mars and Meadow are other favorite games for members of DoK.

Let’s get into our agenda for this meeting.

For our first agenda, Itamar Marom (From AppsFlyer and DoKC Ambassador) proposes joining more data operators’ maintainers and developers in the community.

Itamar was analyzing the data technology map and saw that some workloads are very common and act differently from what is found in SIG Operator (Special Interest Group) in the case of Spark and Kafka. It would be nice to see the maintainers and hear their opinions, especially in meetings like our one with Google, where they have very different and exciting use cases.

We’ve had folks from these communities present at our virtual meetups. Bringing them to the SIG to learn, share, and find ways to collaborate would benefit the broader DoK ecosystem.

Jim Halfpenny from Stackable mentions that they have several operators for open source projects, including Apache Kafka, Apache NiFi, Apache Superset, Trino, and more. And they will love input from the community on the direction the operators should take. His aim is that our operators should play nicely with others!

Melissa Logan was part of several discussions with the Argo community, and maybe there’s a chance they could join us as well.

Itamar will prepare a proposal for this project in which the group can collaborate and will share it soon.

The second item on our agenda was Carrier Hardening and Security Project Update by Robert Hodges (DoKC Ambassador).

This project is a guide to establishing a baseline for secure data management on Kubernetes by fortifying the database operators. The guide aims to identify the typical attack surfaces that exist for databases running on Kubernetes. It will establish a collection of best practices for enhancing their security using operators.

Robert is working on an August 12 talk at DataConLA on the topic: Tips for Sleeping Well with State-of-the-Art Data Management. It will contain the framing of the operator hardening guide.

Robert is playing around with a couple of approaches to divide up the problem space. One of them is to deal with security concerns as follows:

The database itself - E.g., setting passwords safely.
Kubernetes - Security it from outside attackers - E.g., encrypted client connections.
Data outside Kubernetes - Object storage used for backups, logs forwarded to log management systems.

Robert Will share a draft of the talk in the #sig-operator channel for discussion.

The last topic on our agenda is ArgoCD and general CI/CD compatibility for operators by Robert Hodges.

Robert created a public repository Argocd-examples-clickhouse, example of ArgoCD application definitions for ClickHouse analytic applications.

In this project, you’ll find ArgoCD applications and instructions to stand up a full analytic stack based on ClickHouse in a Kubernetes cluster.

Following suggestions from community members, a new Slack channel #topic-devops was created 🎉 . This is a channel to talk about CI/CD integration, specific solutions like ArgoCD & Flux, etc.

To learn more about our meetings, join the Data on Kubernetes. An Open Community for Data on Kubernetes. We host weekly live meetups where technologists share their stories, wisdom, and practical advice for running data on Kubernetes.

Percona Monitoring and Management 2.38 preview release

Taras Kozub — Fri, 30 Jun 2023 00:00:00 UTC

Percona Monitoring and Management 2.38 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.38 is now available as a preview release.

To see the full list of changes, check out the PMM 2.38 Release Notes

PMM server Docker installation

docker tag:

perconalab/pmm-server:2.38.0-rc

Important: To use the DBaaS functionality during the PMM preview release, add the following environment variable when starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.38.0-rc

PMM client package installation

Download the latest pmm2-client release candidate tarball for 2.38.
To install pmm2-client package, enable testing repository via Percona-release:

percona-release enable percona testing

Install pmm2-client package for your OS via Package Manager.

OVA

PMM2-Server-2.38.0.ova file

AMI

Run PMM Server hosted at AWS Marketplace instructions

ami-09895e9b605f14cbc

Percona University in Peru 2023

Edith Puclla — Tue, 20 Jun 2023 00:00:00 UTC

Peru is a country in South America home to a section of the Amazon rainforest and Machu Picchu ⛰️, an ancient Incan city high in the Andes mountains. Percona decided to hold the first event of Percona University 2023 in Lima, the capital of Peru, on June 10.

Percona University is a series of free technical events organized by Percona in various cities around the world since 2013. Percona uses these events to share its unbiased expertise on open source databases with the community, users, and organizations. The last Percona University was in 2022 in Istanbul, Turkey.

Something charming was designed for this first session of Percona University; there were all kinds of stickers and prizes at the end of the event.

The talks were related to open source databases and kubernetes. Let’s summarize them:

Let’s start with Peter Zaitsev, who talked about the Cloud of Serfdom vs. the Cloud of Freedom; why open source will win in the Cloud Age, spoke about the relationship between Cloud and Open Source, showing the historical changes in the impact of the Cloud on Open Source and examining the current state of affairs, and advocating for a specific approach to using Cloud and Open Source together.
Peter’s next talk was about 17 reasons to migrate to MySQL 8 MySQL 8 was different from previous major MySQL versions, as it underwent significant changes from its initial release in 2018 to the version in 2023. Many features were introduced through subsequent minor version releases. In the presentation, Peter focused on the most important Modern MySQL 8 features that had emerged since the initial release of MySQL 8.

The next on the agenda was Michael Villegas. He talked about Useful tools from the Percona Toolkit for DBAs. This talk was aimed at DBAs and developers of any level of experience responsible for MySQL database administration. They learned how to use some very useful tools within the Percona Toolkit to solve common problems within their databases. Michael showed these tools allowed them to save time and effort in resolving these issues.

After the following talk, we had a coffee break provided by Percona, it was an opportunity to chat with Peter and network with other attendees.

The next talk was by Peter Zaitsev about Advanced MySQL optimization and troubleshooting using PMM. Optimizing MySQL performance and troubleshooting MySQL problems are two of the most critical and challenging tasks for MySQL DBAs. The databases power their applications need to handle changing traffic workloads while remaining responsive and stable, ensuring an excellent user experience. Additionally, DBAs are expected to find cost-efficient solutions to these issues. In the presentation, Peter Zaitsev showed advanced options of PMM version 2, which allowed DBAs to address these challenges.

The next talk was for Edith Puclla (me). I did an Introduction to Kubernetes Operators. I provided a simplified overview of Kubernetes Operators, focusing on making the concept understandable for those new to the subject. I started explaining Kubernetes and why they are relevant in managing applications, then we go for Kubernetes Operators example, the reason behind that, and the benefits of using Operators.

Our last talk was about Deep Dive Into Query Performance by Peter Zaitsev. In this presentation, Peter explored this seemingly simple aspect of working with databases in detail. Peter answered questions like when you should focus on tuning specific queries or when it is better to focus on tuning the database (or just getting a bigger box). Peter also showed other ways to minimize user facing response time, such as parallel queries, asynchronous queries, queueing complex work, and as often misunderstood response time killers such as overloaded networks, stolen CPU, and even limits imposed by this pesky speed of light.

The event was well received. Many graduates, professionals, students, and open source enthusiasts attended the event.

And also nice to share moments with Peter and his fans.

We also thank ESAN University for providing us with the venue for the event.

Don’t miss our next Percona University event, in Istanbul!

See you at Percona University Peru in 2024!

Data on Kubernetes Meetup May 23

Edith Puclla — Thu, 01 Jun 2023 00:00:00 UTC

Percona has started to participate in Data on Kubernetes (DoK) meetings about Kubernetes Operators. These meetings are an initiative of DoK meetups that spotlight DoK case studies. In this blog post series, I will summarize the topics covered in each meeting.

On May 23, very interesting topics were discussed on the agenda. Let’s begin to summarize it.

We start with a new project proposal, which is called: Distributed Systems Operator Interface (DSOI). It is proposed by Adheip Singh from DataInfra, Nitish Tiwari from Parseable, and Itamar Marom from AppsFlyer.

This project is a set of best practices for building Kubernetes operators for distributed systems. The spec defines standard practices that can help define custom resources (CR). It consists of Kubernetes-native CRDs and specs and is not bound to any specific application. There are already two operators built using this set of practices:

If you want to contribute, send proposals, join datainfra-workspace or raise bugs in the GitHub repository of DSOI.

As the second item on the agenda, we have an update about the DoK Operator SIG Project Proposal - Security & Hardening Guide. This project is proposed by Robert Hodges, Altinity Inc.

This project is a guide to establishing a baseline for secure data management on Kubernetes by fortifying the database operators. The guide aims to identify the typical attack surfaces that exist for databases running on Kubernetes. It will establish a collection of best practices for enhancing their security through the utilization of operators.

Robert mentioned that he connected to TAG Security, which in turn led to a link to BadRobot, which is a scanner that checks operators for excessive privileges. Also, Robert presented to DoK Bay Area last week to introduce the problem of operator security.

For the Operator security & hardening guide, we can raise issues on sig-operator. They are currently seeking volunteers and contributors for their project; Find operator-security-hardening project on GitHub, or feel free to write Robert Hodges rhodges@altinity.com.

Finally, we have an update of the Operator Feature Matrix (OFM)

The Operator Feature Matrix (OFM) is a project from the Data on Kubernetes Community to create a standardized and vendor-neutral feature matrix for various Kubernetes operators that manage stateful workloads. This project is proposed by Alvaro Hernandez, and it is definitely a good project to contribute if you are looking to improve the end-user experience with the use of workloads in Kubernetes.

CloudNativePG project sent feedback to improve OFM. CloudNativePG is the Kubernetes operator that covers the full lifecycle of a highly available PostgreSQL database cluster. Planning to create a website for end-user adoption

There are other (non-Postgres) technologies, like Apache Druid, jumping on OFM. This is a work in progress.

The end of June is being considered for a 1.0 freeze, before which it is required to get as much feedback as possible. If you are interested, feedback can be as simple as opening an issue to discuss something; or sending a PR requesting improvements (or both). Feel free to do it on OFM GitHub Repo.

What experts said at Kubecon about Data on Kubernetes

Edith Puclla — Wed, 31 May 2023 00:00:00 UTC

Melissa Logan, managing director of Data on Kubernetes (DoK), led one of the best panels I’ve been to at a conference at Kubecon EU in Amsterdam about challenges with and the state of the art of running databases on Kubernetes.

This panel united the Data on Kubernetes Community Operator SIG and Kubernetes Storage SIG to discuss key features of Kubernetes database operators. Xing Yang from VMware, Sergey Pronin from Percona, and Álvaro Hernández from OnGres came together to discuss what works, what doesn’t, and where the industry is going. They also presented a feature matrix to help end users compare many database Operators.

If you are new to the topic of Kubernetes Operators, I wrote a blog post about Kubernetes Operators in a nutshell, you can read the first part of this article.

Let’s start by summarizing the challenges the panelists mentioned when running stateful applications on Kubernetes.

Some Operators have certain limitations, and there are security concerns. Database users always think about data encryption: is the data safe? What happens if the node goes down? What happens if we lose the storage?
The Operator model is very extensible and flexible, which is great, but on the other hand, there are so many Operators, and it becomes a challenge to choose the right one for our use cases.
People who are developing Operators also find challenges because every database has its native way of doing backups, but if you want to support more than one type of database, then it’s more challenging to find a generic way.

Does the framework capture what is needed for data workloads well?

There is a capability model for Operators that classify them into five levels. There is room for improvement in this model. It can be improved to build test compatibility and more objective measures of these capability levels if you look at level five, the top one for data workloads.

Security is the number one criterion that people use to evaluate Operators. How are they addressing security in Kubernetes Operators?
Users want to ensure that the Operator does not get a lot of privileges or does not interfere with other tenants in the Kubernetes cluster.
Now users are looking for more sophisticated ways with their existing security key-value storage. They can be sure they are safe.
The framework should provide ransomware protection, so when you back up your databases, you also want to have one immutable copy to provide your protection and recover from that.

Now Let’s talk about solutions

DoK started an Operator Special Interest Group (SIG), and community members have been meeting to discuss how as a group and as an industry, to collaborate to come up with solutions for some of the challenges end users face. The Operator SIG works with the Storage Technical Advisory Group (TAG), Storage SIG, and Security SIG.

According to data on Kubernetes’s (DoK) first report, 70% of responses are running workloads in production. More data workloads are running on Kubernetes, so it is essential to know what works well. Sharing knowledge and leveraging that expertise is vital at this stage.

There are things that SIG Operators detail in a document, like common patterns and features used when running databases on Kubernetes, best practices, the criteria for running a good operator, why observability is so important in the cloud-native environment and security.

The Operator Feature Matrix is a big initiative to help end users find operators based on different criteria to choose what they need. It is a project to compare different Operators. They are starting with database Operators, and one project is already defined: Postgres Operator Feature Matrix. Feel free to contribute to OFM.

New to Kubernetes Operators and databases? Check out Percona’s Operators for MySQL, PostgreSQL, and MongoDB.

Easy Way to Start Contributing to Open Source With PMM Documentation

Thu, 18 May 2023 00:00:00 UTC

If you are a user of Percona Monitoring and Management and noticed any typo or inaccurate information in its documentation, you can easily correct it yourself in the repository following detailed instructions in README.md. But if you are not experienced in open source contributions, you may still feel uneasy about following those steps. This post is for you! We will walk through the main steps with pictures and explanations.

Create a Fork

First, you need to create a fork from the main repository to your account. In the top-right corner of the page, click Fork - Create a new fork.

Build Documentation With Docker

The easiest way is to build documentation with Docker. If you don’t have it installed, download it from the Docker official website and follow the instructions. The process of installation is quick, and it is no more difficult than the installation of any other app.

Open your fork on GitHub and clone that repository to your local environment.

git clone git@github.com:{user-name}/pmm-doc.git

Change directory to pmm-doc.

cd pmm-doc

To check how our edits will look like, we need to build documentation for live previewing. Run:

docker run --rm -v $(pwd):/docs -p 8000:8000 perconalab/pmm-doc-md mkdocs serve --dev-addr=0.0.0.0:8000

Wait until you see INFO - Start detecting changes. When the documentation is ready to work with, it will be available at http://0.0.0.0:8000 in your browser, and it will reflect all changes that you make locally.

Make Changes

In a new Terminal tab, create a new branch and make your changes. Save them, create a commit, and push it to your fork.

Create a pull request to the main repository. You will also need to sign the CLA, so we could merge your changes.

You did it! Congratulations! Now wait for the feedback from the Percona team. If there is no problem with your PR, it will be merged into the main repository.

Next Steps

To make further changes, you need to keep your repository up-to-date with the upstream one. There are several ways to do it. You can find the information here. The simplest way is to do it using the GitHub interface. Just click on Sync fork and then Update branch.

After that, you will be able to update your local repository with git pull command.

If you face any problems with contributions to Percona repositories, don’t hesitate to contact us at community-team@percona.com or ask your question on the Percona Forum.

My Experience at Kubecon Europe in Amsterdam

Edith Puclla — Thu, 11 May 2023 00:00:00 UTC

Kubecon is the most significant event focused on the Kubernetes ecosystem. It takes place once a year in North America, Europe, and Asia. It is a perfect opportunity to learn from experts, meet friends, grow your network, and attend talks at a varied technical level and meetings focused on CNCF communities. This time I attended Kubecon in Amsterdam. The theme for this version of Kubecon was: community-in-bloom because we are still healing from COVID, and people are getting back to feeling comfortable participating in events.

It is not my first time attending Kubecon; this is my fourth time! What was different this time was that Percona, the company I work for, sponsored Kubecon. This means we also had a booth at the event to share what we do at Percona.

Yessss!! Percona had a booth in Kubecon, Amsterdam.

Percona is a 100% remote company, and Kubecon was the opportunity to meet part of the team I work with daily.

We started by meeting with friends in one of the most popular places in Amsterdam. I met people from different tech communities and CNCF ambassadors.

If you are in Amsterdam and don’t ride a bicycle, it is an incomplete experience. We rode a bike to the convention center and collected the event badge.

After that, there were three intense days, many activities, meetings, and sessions.

Community

I met with several members of the Docker and CNCF communities. This is the meeting of the Docker captains and various members of the Docker team. Such an amazing experience to know them in person.

And this is the group of photos that all the CNCF ambassadors had at Kubecon. There are 155 CNCF ambassadors around the world and contributors and advocates of the CNCF ecosystem.

Lightning Talks

I attended a large part of the lightning sessions, which were very inspiring. Each speaker has less than 5 minutes to explain a specific topic.

Kevin Patrick talked about the Armada as a Sandbox project in the CNCF, in which the principal goal is enabling batch processing across multiple Kubernetes clusters. Kevin made an introduction to Armada and showed the integration with Apache Airflow.

The next talk was about The CNCF Board Game Rules, where Peter O’Neill made an abstract about the world of the CNCF and imagined it as a role-playing game (RPG) board game. It was pretty nice for Peter to show us the adventure of CNCF with games.

This was one of my favorite lightning talks: [A Beginners Guide to Conference Speaking] (https://www.youtube.com/watch?v=jCz9QPrJ6Eo)with Paula Kennedy. She showed a friendly way to prepare proposals for CFPs and advice that will help us through the process.

Another good lightning session was about the Power-Aware Scheduling in Kubernetes with Yuan Chen from Apple. In this talk, Yan gave an overview of a new scheduler feature to support power-aware scheduling in Kubernetes and how it can help safely increase server hardware and data center infrastructure size and improve resource utilization and workload reliability for Kubernetes clusters.

Another of my favorites was Talking to Kubernetes with Rust with James Laverack, where he showed how to interact with Kubernetes in Rust.

Kubernetes Operators Panel Discussion

I also attended a Panel Discussion about Kubernetes Operators. In this panel discussion, Xing Yang, Melissa Logan, Sergey Pronin, and Alvaro Hernandez talked about the challenges that final users have when running data workloads in Kubernetes Operators. They also shared about the need to fulfill the process of data workloads. Check out this fantastic talk and learn more about Kubernetes Operators. A very curious and interesting fact is that most of the assistants use data workload with Kubernetes Operators

eBPF

The last talk I attended was an eBPF talk with Liz Rice (Chief Open Source Officer, Savant) In this talk, Liz shows a demo about how Cilium and its ClusterMesh feature can take care of many aspects of connectivity across multiple clusters in a cloud-agnostic way. Check this talk in CNCF YouTube Channel.

Clossign Kubecon Amsterdam

Finally, in Percona, we closed the event with a raffle to take home an Atari’; the expectations were relatively high and very fun, and many Percona lovers came to participate in the raffle.

You can check in which events Percona is going to be in the next months

PostgreSQL: Query Optimization With Python and PgBouncer

Mario García — Tue, 25 Apr 2023 00:00:00 UTC

Database application by Nick Youngson CC BY-SA 3.0 Pix4free

A few months ago I wrote a few blog posts on how to generate test data for your database project using Python, which you can find on the Percona blog and the Community blog:

The basic idea is to create a script that uses Faker, a Python library for generating fake data, and what the script does is

Divide the whole process into every CPU core available by implementing multiprocessing
The script will generate a total of 60 thousand records, divided by the number of CPU cores minus one
Each set of records is stored in a Pandas DataFrame, then concatenated into a single DataFrame
The DataFrame is inserted into the database using Pandas’ to_sql method, and pymongo’s insert_many method

How can the script be optimized? Instead of generating the data, storing it in a DataFrame, and then inserting it into the database, you can make every CPU core insert the data while generating it without storing it elsewhere before running the corresponding SQL statements. Multiprocessing is implemented to use every CPU core available but you also need to configure a connection pool for your PostgreSQL server.

Through this blog post, you will learn how to install and configure PgBouncer with Python to implement a connection pool for your application.

PgBouncer

PgBouncer is a PostgreSQL connection pooler. Any target application can be connected to PgBouncer as if it were a PostgreSQL server, and PgBouncer will create a connection to the actual server, or it will reuse one of its existing connections.

The aim of PgBouncer is to lower the performance impact of opening new connections to PostgreSQL.

Installation

Ir you’re an Ubuntu user, you can install PgBouncer from the repositories:

$ sudo apt install pgbouncer -y

If not available in the repositories, you can follow the instructions below for both Debian and Ubuntu as mentioned in the Scaleway documentation

Create the apt repository configuration file

$ sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'

Import the repository signing key

$ wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -

Update the apt package manager

$ sudo apt update

Install PgBouncer using apt

$ sudo apt install pgbouncer -y

Configuration

After installing PgBouncer, edit the configuration files, as stated in the Scaleway documentation.

Set up the PostgreSQL server details in /etc/pgbouncer/pgbouncer.ini

database_name = host=localhost port=5432 dbname=database_name

You may also want to set listen_addr to * if you want to to listen to TCP connections on all addresses or set a list of IP addresses.

Default listen_port is 6432

From this article by Abdullah Alger, the settings max_client_conn and default_pool_size, the former refers to the number of applications that will make connections and the latter is how many server connections per database. The defaults are set at 100 and 20, respectively.

Edit the /etc/pgbouncer/userlist.txt file and add your PostgreSQL credentials

“username” “password”

Add the IP address of the PgBouncer server to the PostgreSQL pg_hba.conf file

host all all PGBOUNCER_IP/NETMASK trust

By default, PgBouncer comes with trust authentication method. The trust method can be used in a development environment but is not recommended for production. For production, hba authentication is recommended.

After configuring PgBouncer, restart both the PostgreSQL and PgBouncer services

sudo systemctl reload postgresql
sudo systemctl reload pgbouncer

For more information about additional configuration options, check the PgBouncer documentation.

Python

Requirements

Dependencies

Make sure all the dependencies are installed before creating the Python script that will generate the data for your project.

You can create a requirements.txt file with the following content:

tqdm
faker
psycopg2

Or if you’re using Anaconda, create an environment.yml file:

name: percona
dependencies:
 - python=3.10
 - tqdm
 - faker
 - psycopg2

You can change the Python version as this script has been proven to work with these versions of Python: 3.7, 3.8, 3.9, 3.10, and 3.11.

Run the following command if you’re using pip:

pip install -r requirements.txt

Or run the following statement to configure the project environment when using Anaconda:

conda env create -f environment.yml

Database

Now that you have the dependencies installed, you must create a database named company.

Log into PostgreSQL:

$ sudo su postgres
$ psql

Create the company database:

create database company;

And create the employees table:

create table employees(
 id serial primary key,
 fist_name varchar(50) not null,
 last_name varchar(50) not null,
 job varchar(100) not null,
 address varchar(200) not null,
 city varchar(100) not null,
 email varchar(50) not null
);

Inserting Data

Now it’s time to create the Python script that will generate the data and insert it into the database.

from multiprocessing import Pool, cpu_count
import psycopg2
from tqdm import tqdm
from faker import Faker

fake = Faker()
num_cores = cpu_count() - 1

def insert_data(arg):
 x = int(60000/num_cores)
 print(x)
 with psycopg2.connect(database="database_name", user="user", password="password", host="localhost", port="6432") as conn:
 with conn.cursor() as cursor:
 for i in tqdm(range(x), desc="Inserting Data"):
 sql = "INSERT INTO employees (first_name, last_name, job, address, city, email) VALUES (%s, %s, %s, %s, %s, %s)"
 val = (fake.first_name(), fake.last_name(), fake.job(), fake.address(), fake.city(), fake.email())
 cursor.execute(sql, val)

if __name__=="__main__":
 with Pool() as pool:
 pool.map(insert_data, range(num_cores))

At first, the multiprocessing pool is created, and configured to use all available CPU cores minus one. Each core will call the insert_data() function.

On each call to the function, a connection to the database will be established through the default port (6432) of PgBouncer, meaning that the application will open a number of connections equal to num_cores, a variable that contains the number of CPU cores being used.

Then, the data will be generated with Faker and inserted into the database by executing the corresponding SQL statements.

In a CPU with 16 cores, the number of records inserted into the database on each call to the function will be equal to 60 thousand divided by 15, that is 4 thousand SQL statements executed.

This way you can modify the script and optimize it by configuring a connection pool with PgBouncer.

Percona Monitoring and Management 2.37 preview release

Taras Kozub — Thu, 20 Apr 2023 00:00:00 UTC

Percona Monitoring and Management 2.37 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.37 is now available as a preview release.

You can find the Release Notes here

Percona Monitoring and Management server docker installation

docker tag:

perconalab/pmm-server:2.37.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variable when starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.37.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.37 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.37.0.ova

AMI

ami-013c92f3d0c727b8f

FerretDB - A Quick Look

David Stokes — Wed, 12 Apr 2023 00:00:00 UTC

There is an old saying that what looks like a duck and quacks like a duck is probably a duck. But what looks like MongoDB and acts like MongoDB could be FerretDB! To greatly simplify the technology behind this project, FerretDB speaks, or quacks, MongoDB but stores the data in PostgreSQL. PostgreSQL has had a rich JSON data environment for years and FerrtDB takes advantage of this capability. This is a truly Open Source MongoDB alternative and was released under the Apache 2.0 license.

FerretDB has been in development for a while, but they announced the first Generally Available Release of their product recently.

In the announcement is a quick “How To Get Started” section which details how to get FerretDB running with the help of Docker. As can be seen below, this is a very simple process.

$ docker run -d --rm --name ferretdb -p 27017:27017 ghcr.io/ferretdb/all-in-one
Unable to find image 'ghcr.io/ferretdb/all-in-one:latest' locally
latest: Pulling from ferretdb/all-in-one
f1f26f570256: Pull complete
1c04f8741265: Pull complete
dffc353b86eb: Pull complete
18c4a9e6c414: Pull complete
81f47e7b3852: Pull complete
5e26c947960d: Pull complete
a2c3dc85e8c3: Pull complete
17df73636f01: Pull complete
713535cdf17c: Pull complete
52278a39eea2: Pull complete
4ded87da67f6: Pull complete
05fae4678312: Pull complete
56b4f4aeea2d: Pull complete
68c486387c4f: Pull complete
5eb3eee800a9: Pull complete
8e5dd809e820: Pull complete
d3e85fce5b45: Pull complete
e6810cdbd43b: Pull complete
Digest: sha256:072312577c1daf469ac77d09284a638dea98b63f4f4334fd54959324847b93aa
Status: Downloaded newer image for ghcr.io/ferretdb/all-in-one:latest
58f00a86bad172674479f3663563af274e0dd3d15249029a403d0c85039b7ab5

Now that FerretDB is ready, we can use the MondoDB shell to speak to it.

$ docker exec -it ferretdb mongosh

Current Mongosh Log ID: 6435963392d12db06bdb7ecc
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.8.0
Using MongoDB: 6.0.42
Using Mongosh: 1.8.0

(the remaining output was omitted for brevity)

Entering some very basic MongoDB commands work as expected. Well, for the most part.

test> db
test
test> show collections;

test> db.createCollection('test');
{ ok: 1 }
test> show collections;
test

test> db.test.insert({name: "Dave", state: "Texas"});
DeprecationWarning: Collection.insert() is deprecated. Use insertOne, insertMany, or bulkWrite.
{
 acknowledged: true,
 insertedIds: { '0': ObjectId("6435ac52c4a22ac27f30e2a2") }
}
test> db.test.insertOne({name: "Dave", state: "Texas"});
{
 acknowledged: true,
 insertedId: ObjectId("6435ac5dc4a22ac27f30e2a3")
}
test> db.test.find();
[
 {
 _id: ObjectId("6435ac52c4a22ac27f30e2a2"),
 name: 'Dave',
 state: 'Texas'
 },
 {
 _id: ObjectId("6435ac5dc4a22ac27f30e2a3"),
 name: 'Dave',
 state: 'Texas'
 }
]
test> 

I expected that the inset on the deprecated ’insert’ command would not create a document but I was wrong. It took me a moment to realize that the ‘insert’ and ‘insertOne’ commands both worked after looking at the different ObjectIds. But what do we know about the server itself? Issuing a serverStatus commands confirms we are talking to the FerretDB server.

test> db.runCommand({serverStatus: 1});
{
 host: '58f00a86bad1',
 version: '6.0.42',
 process: 'ferretdb',
 pid: Long("10"),
 uptime: 6277.435694035,
 uptimeMillis: Long("6277435"),
 uptimeEstimate: Long("6277"),
 localTime: ISODate("2023-04-11T18:59:46.488Z"),
 freeMonitoring: { state: 'undecided' },
 metrics: {
 commands: {
 ping: { total: Long("1"), failed: Long("0") },
 getFreeMonitoringStatus: { total: Long("1"), failed: Long("0") },
 create: { total: Long("1"), failed: Long("0") },
 insert: { total: Long("3"), failed: Long("0") },
 atlasVersion: { total: Long("1"), failed: Long("1") },
 getLog: { total: Long("1"), failed: Long("0") },
 buildInfo: { total: Long("1"), failed: Long("0") },
 getCmdLineOpts: { total: Long("1"), failed: Long("0") },
 listCollections: { total: Long("2"), failed: Long("0") },
 ismaster: { total: Long("611"), failed: Long("0") },
 find: { total: Long("4"), failed: Long("0") },
 getParameter: { total: Long("1"), failed: Long("1") },
 hello: { total: Long("1"), failed: Long("0") },
 unknown: { total: Long("5"), failed: Long("0") }
 }
 },
 ok: 1,
 catalogStats: {
 collections: 210,
 capped: 0,
 timeseries: 0,
 views: 0,
 internalCollections: 0,
 internalViews: 0
 }
}
test> 

Summary

FerretDB is a MongoDB protocol server built upon PostgreSQL. Those unhappy with the change in MongoDB’s license change away from open source now have another path they can follow. I will have a full session at Percona Live on FerretDB where I will delve into how complete of an option this is for those desiring an open solution.

Using the JSON data type with MySQL 8 - Part II

Edith Puclla — Tue, 11 Apr 2023 00:00:00 UTC

If you read - Using the JSON data type with MySQL 8 - Part I, you will see that inserting data into MySQL of JSON type is a very common and effective practice. Now we’ll see how to do it with a Python project, using SQLAlchemy and Docker Compose, which further automates this example. You can run this example using a single command: docker-compose up

Before getting down to work, we will review some important concepts:

Percona Server for MySQL is an open source, drop-in replacement for MySQL Community that provides better performance, more scalability, and enhanced security features.
SQLAlchemy is a library that allows us to communicate between Python programs and databases.
Docker Compose is a tool for defining and running multi-container Docker applications.

Let’s start with the structure of this project:

We have a folder called app which contains the db.py file, and this is where we create the library database and establish the connection with this database.

bash
db_user = os.environ['DB_USER']
db_password = os.environ['DB_PASSWORD']

engine = create_engine(f"mysql+pymysql://{db_user}:{db_password}@db:3306/library")

In this file, we also create the class transactions. This will create the fields for the library databases with SQLAlchemy; we define the attributes, and they will be the database fields. We have four attributes: book_id, tittle, publishes, and labels. The last one (labels) of JSON data type.

bash
class transactions(base):
 __tablename__ = 'book'

 book_id = Column(Integer, primary_key=True)
 title = Column(String(50))
 publisher = Column(String(50))
 labels = Column(JSON)

 def __init__(self, book_id, title, publisher, labels):
 self.book_id = book_id
 self.title = title
 self.publisher = publisher
 self.labels = labels

base.metadata.create_all(engine)

Now let’s review the Python script called insert.py, where we use the transactions class to insert data into the database.

bash
import db
from sqlalchemy.orm import sessionmaker

Session = sessionmaker(bind=db.engine)
session = Session()

tr1 = db.transactions(1,'Green House', 'Joe Monter', '{"about" : {"gender": "action", "cool": true, "notes": "labeled"}}')

session.add(tr1)
session.commit()

Now let’s explore the docker-compose.yaml file, we have two services, the db and the api

bash
version: "3.8"
services:
 api:
 build: .
 container_name: api
 depends_on:
 db:
 condition: service_healthy
 db:
 image: percona/percona-server:8.0
 container_name: db
 restart: always
 environment:
 MYSQL_USER: root
 MYSQL_ROOT_PASSWORD: root
 MYSQL_DATABASE: library
 healthcheck:
 test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]
 timeout: 20s
 retries: 10
 volumes:
 - my-db:/var/lib/mysql
 ports:
 - "3306:3306"
 expose:
 - "3306"

# Names for volume
volumes:
 my-db:

The db service uses the Percona Server for MySQL image (percona/percona-server:8.0) for the database and has a healthcheck that allows you to confirm when the database is started and ready to receive requests. The api service depends on the db service to start. The api service will build a Dockerfile, it does a build of the Python applications (of db.py and insert.py), so in this way, we can insert data into the database when it is ready.

It’s time to see the example in action; let’s locate it inside the json-mysql project and run docker-compose ps -d

Once this is done, we can connect to the database and query the table without needing to go inside the container with the following command:

bash
docker exec -i db mysql -uroot -proot <<< "use library;show tables;select \* from book;describe book;"

We can check the data types of our fields and the inserted data. You will also see the JSON data type “labels” data type.

bash
book_id title publisher labels
1 Green House Joe Monter "{\\"about\\" : {\\"gender\\": \\"action\\", \\"cool\\": true, \\"notes\\": \\"labeled\\"}}"
2 El camino Daniil Zotl "{\\"about\\" : {\\"gender\\": \\"documental\\", \\"cool\\": true, \\"notes\\": \\"labeled\\"}}"
3 London Bridge Mario Mesa "{\\"about\\" : {\\"gender\\": \\"drama\\", \\"cool\\": true, \\"notes\\": \\"labeled\\"}}"

bash
Field Type Null Key Default Extra
book_id int NO PRI NULL auto_increment
title varchar(50) YES NULL
publisher varchar(50) YES NULL
labels json YES NULL

Use “docker compose ps” to see your services running. In this case, we have the “db” service running, which is for the database, and we have “api” with the state “exited,” which means that the scripts to create the database and insert the data into the database “library” was created.

bash
NAME COMMAND SERVICE STATUS PORTS
api "/bin/sh -c 'bash -C…" api exited (0)
db "/docker-entrypoint.…" db running (healthy) 0.0.0.0:3306->3306/tcp, 33060/tcp

It was an example of inserting JSON data into MySQL using SQLAlchemy in Python and docker-compose for deployment.

You can find the project on GitHub. If there is any other way to make it better happy to hear it so I can improve this project.

You can explore more about Percona Server for MySQL, and if you want to see how this project start check Using the JSON data type with MySQL 8 - Part I

How a Database Monitoring Tool Can Help a Developer. The Story of One Mistake.

Daniil Bazhenov — Fri, 07 Apr 2023 00:00:00 UTC

I will tell you the real story of using database monitoring tools when developing an application. I will show you an example of how I managed to detect and fix a problem in the application.

A small clarification, the real story from my development practice happened a little more than a week ago, but for the article I took graphs of final debugging, so that the graphs show the correct sequence and fit into the available for explanation and demonstration. It’s just that in reality, I went out for coffee several times and thought for a long time about what was reflected in the graphs of monitoring:)

About the app and the process

I am developing a PHP application using MongoDB as a database. The application is lightweight, and most load falls on the database. I have implemented functions at the application level to adjust the number of queries, as the application can quickly load the database to 100%.

For development, I use several small dev instances in AWS, use Percona Server for MongoDB with three nodes as a database, and have Percona Monitoring and Management (PMM) installed for monitoring the databases.

My development process consists of the following steps:

I developed a new feature and ran it on the dev server for testing.
I check the prefiling on the PHP side, and there is no memory leak, and I am happy with the speed.
I check the database monitoring to ensure everything works fine.
I debug the feature, setting the number and types of queries in the function to balance the number of queries and the load on the database, if necessary.

Adding new functionality to the application

So I started the application and got ready to run the new feature. The feature was getting information from open sources, processing it, and saving it to the database. The second part of the functionality went through all the saved documents and did some additional processing.

At this point, the application already had a lot of features that loaded the CPU of the Primary Node by 25-40%, and everything was running stably. I decided to have a performance reserve, as I planned to add new features.

I checked several dashboards, and there were no anomalies or changes. PMM has many dashboards and charts, and I will only show a few, just some.

I saved the changes with the new feature and pushed it to the dev server to make it work. Then I checked that the function started without errors, and the result was visible in the database. I use MongoDB Compass to check the result of a database entry.

Something has gone differently than planned.

I waited a few minutes and rechecked the dashboard. At first glance, the main screen was fine. However, I was alarmed by the speed of processing. The number of operations has mostly stayed the same.

I scrolled down through the various charts on the dashboard and saw an anomaly.

The latency increased, and the app loaded the instance to 100% CPU.

I have made a test run on the application side and checked the profiler there, too. The app worked poorly, and queries were slow.

Finding the cause of the problem

I knew the reason was the new feature and immediately rolled back the last changes.

I had a rough idea of where the problem might be, made a few changes, and started again.

I did it several times, but the result was the same (the CPU was loaded at 100%).

I selected a period with a load and used the Query Analytics function built into the monitoring. Query Analytics shows a list of queries sorted by load or execution speed. Some of the queries to the Pages collection gave 90% load, and the Query Time was more than 3 minutes.

In Query Analytics, you can find slow queries, see their details, and then debug them in the application.

Fixing the problem

I made a few changes that fixed the problem.

The first problem was the indexes. I create indexes from within the application using the command.

$app['db']->CollectionName->createIndex(['index_key' => 1]);

Since the application uses many different collections and queries with conditions on various fields and with or without sorting, I have a lot of indexes.

I made a typo in this case, and the index was not created correctly.

After the indexes were created correctly, I needed quick runs to debug the number of queries to adjust the CPU load to around 50%.

You can see the final chart after debugging and fixing the problem.

Conclusion

Don’t forget to add indexes and make sure they work.

I am a simple developer who can make mistakes and do different experiments. Installing the monitoring was one of the experiments, and previously I just focused on the speed of the PHP script. From time to time, I have looked at the monitoring dashboard in the AWS control panel, but it gives less information, only about the instance itself, without being able to investigate in detail.

So, PMM have a great tools for debugging and searching “bottlenecks” in the databases. And I recommend installing and trying database monitoring with PMM if your application uses MySQL, PostgreSQL, or MongoDB.

How To Generate Test Data for Your Database With SQL

Maksim Gramin — Thu, 30 Mar 2023 00:00:00 UTC

Recently, I’ve noticed several posts on the Percona Community blog about test data generation. This is a great trend, as such data enables us to test applications more easily and efficiently and detect problems before they appear in production. One article was devoted to the Pagila standard DB schema and another to generating test data with Python. I’ve decided to continue this tradition and write an article about generating data using SQL. For our experimental schema, we’ll use Pagila, but we’ll generate much more data than it currently has.

And of course, in the current climate, we have no choice but to start with ChatGPT. We will politely ask the bot to generate data for the Pagila (publicly available sample database schema):

As a result, we will obtain several valid SQL scripts in the correct order. However, we may need to request additional details, such as ensuring that all tables are included. On the one hand, this is useful and may already suffice for some cases. On the other hand, there are numerous nuances during data generation, such as specific data distribution and proximity to the subject area. Explaining all of these nuances to the bot may be challenging and labor-intensive. Plus, for sure we will need to generate a large amount of data, and in a very limited time and for private corporate schemas… So let’s roll up our sleeves and go through all the basic steps of generating data from scratch using good old SQL.

Generate rows

As we know, SQL was designed for working with real data stored in tables. However, the SQL:1999 standard introduced recursive queries, which allow, among other things, to generate an arbitrary number of rows without referring to any particular table. At the same time, different DBMSs may have their own (often more convenient) constructs for generating rows:

sql
-- Standard SQL:1999 way
with recursive tmp (r) as (
 select 0 union all
 select r+1 from tmp
 where r < 365)
select r from tmp

-- PostgreSQL
select generate_series
 from generate_series(1, 365)

-- Oracle
select level
 from dual
connect by level <= 365

Generate values

So, we already have rows, and now we need data for these rows. To the best of my knowledge, SQL standard does not provide a way to generate random values. However, most DBMSs have their own methods for doing so. With a little tinkering, we can obtain some random data that is remotely similar to real names, emails, dates, and so on. Let’s create 1000 employees for the employee table without leaving your warm SQL-console:

sql
-- PostgreSQL
insert into employee(id, first_name, last_name,
 years_of_experience, email, order_date, is_student)
select generate_series
 , md5(random()::text)
 , md5(random()::text)
 , floor(random() * 99)::int
 , md5(random()::text) || '@gmail.com'
 , now() - (random() * (interval '90 days'))
 , case when random() > 0.5 then true else false end
 from generate_series(1, 1000)

Generate lifelike values

Thus, we can already generate tons of messy data, which in many cases will be quite enough. But we will not rest on our laurels and will try to generate something closer to real data. Let’s start with one of the most popular task: generating real people’s names. One simple and effective solution is to prepare two sets with common first and last names, respectively, and join them using a Cartesian join. I was impressed by this GitHub gist:

sql
-- PostgreSQL
select first_name
 , last_name
 from (select unnest(array['Adam',/*...*/'Susan']) as first_name) as f
 cross join
 (select unnest(array['Matthews',/*...*/'Hancock']) as last_name) as l
 order by random()

We were able to generate 1392 unique full names by joining 48 first names and 29 last names. This is a good start, and the same technique can be applied to generate other types of data, such as emails, addresses and so on, by wrapping it in a stored function or using a templating engine like Jinja for ease of use. We can also use numerous third-party advanced random data generation services and try to load the generated data into our database (e.g., in CSV format like Fake Name Generator). Some of them will even be able to generate a ready-made SQL script for you (like the Generatedata service):

sql
insert into persons (name, company, address, email)
values
 ('Berk Cotton','Tempus Eu Ligula Incorporated','Ap #633-4301 Tempus, St.','interdum.libero.dui@icloud.ca'),
 ('Ahmed Sandoval','Nullam Lobortis Foundation','P.O. Box 902, 9630 Convallis Rd.','magna.suspendisse@google.edu'),
 ('Hedy Mcbride','Risus Nulla Limited','5235 Lacinia Avenue','donec.felis@icloud.com'),
 ('Kermit Mcintosh','Erat Associates','278-141 Pellentesque St.','vel.faucibus@icloud.ca'),
 ('Susan Berg','Mauris Institute','Ap #876-781 Vehicula Street','ipsum.nunc@protonmail.ca');

The problem of generating data is not new, and there are many popular general-purpose libraries available for generating high-quality fake primitives such as names, addresses, and companies for various programming languages (e.g. Java, Python, JS, Ruby, etc.). Fortunately, some databases allow you to work with these libraries and generate more realistic data using SQL queries. For example, PostgreSQL Faker allows you to generate more realistic data using SQL queries like this:

sql
select faker.name()
 , faker.company()
 , faker.address()
 , faker.email()
 from generate_series(1, 5)

And that’s not all. Extensions such as faker_fdw provide a true relational way to generate data, using tables, joins, and other relational features:

sql
select p.name
 , c.company
 , a.address
 , i.ascii_email
 from (select row_number() over() as id, p.* from person p limit 5) p
 join (select row_number() over() as id, a.* from address a limit 5) a on a.id = p.id
 join (select row_number() over() as id, c.* from company c limit 5) c on c.id = p.id
 join (select row_number() over() as id, i.* from internet i limit 5) i on i.id = p.id

Unique values

Although our data is random, this does not mean that there are no requirements for it. For example, we may only need unique data for certain columns (e.g. id, isbn, code, etc.). There are several ways to achieve this. For instance, many DBMSs support the upsert concept based on the values of one or more columns (clauses like merge, on conflict, etc.):

sql
-- PostgreSQL
create table orders (code varchar(4) primary key, operation_date date);

insert into orders
select substr(md5(random()::text), 1, 4) as code
 , now() - (random() * (interval '90 days')) as operation_date
 from generate_series(1, 1000)
 on conflict (code) do nothing

We can also eliminate duplicate values during generation with the help of the distinct clause or analytical functions:

sql
-- PostgreSQL
select code
 , operation_date
 from (select code
 , operation_date
 , row_number() over (partition by code order by operation_date) as rn
 from (select substr(md5(random()::text), 1, 4) as code
 , now() - (random() * (interval '90 days')) as operation_date
 from generate_series(1, 1000)) s) s
 where rn=1

The disadvantage of the two previous solutions is that we may end up with fewer rows than specified. For greater accuracy, we can use the unique data generation tools built into the DBMS, such as sequences, generators, UUIDs, etc.:

sql
-- PostgreSQL
select gen_random_uuid()
select nextval('film_film_id_seq')

-- PostgreSQL Faker
select faker.unique_name()
 , faker.unique_address()
 from generate_series(1, 10)

Repeatable randomness

In specific cases, it may be necessary to generate the same random data on every run. This is particularly useful for running tests. To achieve this, many DBMSs and libraries allow you to set the initial value (seed) of the random generator:

sql
-- PostgreSQL
select setseed(0.5);

-- Oracle
exec dbms_random.seed(42);

-- PostgreSQL Faker
select faker.seed(4321);

Avatars

We may need to generate not only textual data, but also images, such as avatars. There are many special services with APIs for avatar generation, ranging from funny cartoons to real people photos (e.g. dicebear.com, api.multiavatar.com, randomusers). Let’s try to generate some people with random avatars using the captivating robohash.org service:

sql
select name
 , format('https://robohash.org/%s?set=set%s',
 replace(name, ' ', '_'),
 set_number) as avatar
 from (select trunc(random() * 4 + 1) as set_number
 , faker.name()
 from generate_series(1, 3)) s

Let’s generate something real

It’s time to do something useful! To conduct our experiments, we require a “guinea pig” database. Let’s use the wonderful and well-known Pagila sample database, which we already used with ChatGPT at the very beginning of the post. In fact, Pagila already has data, but it is quite small — a maximum of 16k in just a couple of tables. So we will generate a lot of data ourselves for the empty schema with the help of pure SQL:

To make our SQL generation scripts simpler and more readable, and the generated data more realistic, we will use the PostgreSQL Faker Postgres extension in our SQL queries. Fortunately, there is an easy way to obtain this extension by using a Docker image:

bash
docker run \
 -p 5432:5432 \
 --env POSTGRES_PASSWORD=postgres \
 registry.gitlab.com/dalibo/postgresql_faker

Also, don’t forget to connect to the database and register the extension:

bash
docker exec \
 -it pagila-faker sh -c \
 "psql -U postgres -d postgres \
 -c \"create schema faker;\" \
 -c \"create extension faker schema faker cascade;\" \
 "

And set up Pagila’s schema, using only the pagila-schema.sql script without any data:

bash
git clone https://github.com/devrimgunduz/pagila.git
cd pagila
docker cp ./pagila-schema.sql pagila-faker:/docker-entrypoint-initdb.d/pagila-schema.sql

docker exec -it pagila-faker sh -c \
 "psql -U postgres -d postgres -f /docker-entrypoint-initdb.d/pagila-schema.sql"

References

So let’s start with the easiest stuff — reference tables. In our case, these are things like tables of countries, languages, and so on. In such a situation, we can combine the generate_series and faker.unique_country functions (or unique_language_name, or any other suitable function from the PostgreSQL Faker extension):

sql
insert into country(country)
select faker.unique_country()
 from generate_series(1, 20) as id

A more complex case is the city reference table because the city table depends on the country table. It would be great if each country had a different number of cities, as is usually the case in reality. To achieve this, we can perform a cross join between the country table and a sequence of 1000 rows, and then randomly filter out some of the data, 90% in our case:

sql
insert into city(city, country_id)
select faker.unique_city()
 , country_id
 from country
 cross join generate_series(1, 1000)
 where random() > 0.9

As a result, we get around 2000 cities with different distributions by country:

country_id	country	cities_per_country	cities
27	Guinea	118	1943
32	Suriname	108	1943
40	Marshall Islands	107	1943
25	Romania	103	1943
36	Micronesia	103	1943

Okay, now comes the fun part — generating data for the staff table. This table has two parent tables, store and address, and we need to generate their random combinations somehow. The first thing that comes to mind is using a cross join (as we did already with small reference tables). However, in this case, we may end up with a very slow query, because the Cartesian product of two large tables will generate a huge number of rows that still have to be sorted in random order (and only after that can we cut off the extra rows). Fortunately, the SQL:2003 standard introduces the tablesample clause. This allows us to read not the entire table, but only a part of it as a percentage. The Bernoulli sampling method ensures that all blocks of the table are scanned and only some random records are read, which leads to a quite uniform distribution of randomly selected rows. This way, we can subtract only a portion of the random rows from the store and address tables, say 1%, and then confidently perform a cross join between them:

sql
insert into staff(first_name, ..., password)
select faker.first_name()
 , ...
 , faker.password()
 from (select a.address_id
 , s.store_id
 from store s tablesample bernoulli(1)
 cross join address a tablesample bernoulli(1)
 limit 50000) s

Generate time-series data

We often need to store time-series data in our tables. TimescaleDB has published three impressive articles (one, two, three) devoted to generating such data using SQL. However, in this post, we will only focus on generating rental_date values based on the current row number for the rental table:

sql
insert into rental(rental_date, inventory_id, customer_id, staff_id)
select current_date - (((row_number() over())::text) || ' minute')::interval
 , ...
 from (select ...
 from inventory i tablesample bernoulli(1)
 cross join customer c tablesample bernoulli(1)
 cross join staff s tablesample bernoulli(1)
 limit 1000000) s

Data multiplication

In some cases, we need to generate data from scratch, which is what we have done so far. But in other cases, we need to generate new data based on existing data while preserving current distributions and relationships. Let’s try to increase the number of cities in the city table by four times while maintaining the percentage of cities in each country. The idea is very simple: count the number of cities for each country, multiply it by 3, and generate this number of new cities for each country:

sql
insert into city(city, country_id)
select unnest(array(select faker.unique_city()
 from generate_series(1, c.cities_count*3) as id)) as city
 , country_id
 from (select country_id
 , count(1) as cities_count
 from city
 group by country_id
 order by country_id) c

As you can see, there are many more cities, but the top 5 countries have remained the same:

country_id	country	cities_per_country	cities
27	Guinea	472	7772
32	Suriname	432	7772
40	Marshall Islands	428	7772
25	Romania	412	7772
36	Micronesia	412	7772

This is a viable option for generating data based on existing data, but in some cases, we may need more advanced methods, such as interpolation, prediction/extrapolation, or even machine learning. Perhaps we will discuss this separately in one of the following posts.

Sewing it all together

The full set of scripts is available in this GitHub repository. Among other things, you can find a docker-compose.yaml file there, which will allow you to easily set up a test database and run generation scripts for it with just a few commands:

bash
git clone https://github.com/mgramin/pagila-data-generation
cd pagila-data-generation
docker-compose up

There’s still a long way to go…

On the one hand, we have described many useful tools and generated a lot of synthetic data for our schema. But on the other hand, we will face many other problems and challenges:

Different database schemas. Our current SQL scripts are rigidly tied to the Pagila database schema. We will have to rewrite the scripts every time for each new supported schema. Schema Migrations. We will also need to update the scripts every time we migrate the schema.
Complex Schemas. In the Pagila database, we have about two dozen tables, but in real life, we usually encounter schemas with hundreds and thousands of tables, and complex dependencies between them.
Data Quality. In this article, we did not imposed strict requirements on the quality and properties of the synthesized data. But in real life, it’s different. We may need a specific distribution of values for certain parameters, localization of values, and much more.
Specific Data Types. In addition to text, dates, and numeric values (which we have worked with in this post), various DBMSs support many other more complex data types. These include object types, domains, and JSON, among others, which may also need to be processed during generation.
Performance issues.

But not so bad

Fortunately, there are many ready-made solutions that can solve the aforementioned problems in various ways (an incomplete list can be found and supplemented here). We will attempt to generate data for our schema using the Synthesized TDK tool. TDK is a YAML-based tool, and for a quick start, we only need to prepare a small configuration file in YAML format and specify the mode and expected number of rows:

yaml
default_config:
 mode: GENERATION
 target_row_number: 100_000

This configuration is already sufficient to generate test data for both a small Pagila database and for real production schemas with hundreds of tables and relationships. TDK will determine and apply the necessary generation parameters independently depending on the mode, type of columns, and initial data. As a result, TDK will scan all the tables and their data in the input database and generate 100K rows for each table in the output database, taking into account the type of each column and the existing distribution.

However, we can change the default behavior of TDK by adding custom instructions to the configuration file. Suppose we want movies that are suitable for the whole family (MPAA-rating G — General Audiences) to be the majority. To achieve this, we will add additional configurations to our configuration file to generate the values of the film.rating column with a specific distribution:

yaml
tables:
 - table_name_with_schema: "public.film"
 target_row_number: 10_000
 transformations:
 - columns: ["rating"]
 params:
 type: "categorical_generator"
 categories:
 type: string
 values: ["G", "PG", "PG-13", "R", "NC-17"]
 probabilities: [0.5, 0.2, 0.1, 0.1, 0.1]

By applying this configuration, we can achieve the specified distribution among 10,000 films:

rating	count	ratio
G	5006	50
PG	1997	19
NC-17	1048	10
PG-13	976	9
R	973	9

We can also further configure the generation of addresses, names, strings, various sequences, and much more. You can read more about this in the documentation on transformation types. TDK also provide easy integration with CI/CD pipelines and Testcontainers. And for an even easier start, we have prepared a small demo — pagila-tdk-generation. It can be run with just a couple of commands using docker-compose:

bash
git clone https://github.com/synthesized-io/tdk-docker-demo
cd tdk-docker-demo
docker-compose run tdk

As a result, two PostgreSQL instances will be created, with port 6000 forwarded for the input database and port 6001 for the output database. The Pagila schema will be installed on each of them, and the TDK will be launched to generate data using a more advanced configuration. Once the TDK completes its work, control will return to the command line, and we can connect to the output database to examine the synthesized data (using 6001 port and postgres as the username, password, and database name).

Conclusion

In this post, we have demonstrated how easy it is to generate test data for a database using SQL and its extensions. With these tools, we were able to create a set of SQL scripts that generate test data for the Pagila schema. You can use these scripts as a basis for creating your own scripts for your own databases. However, as your database schema grows and your data quality requirements become more complex, maintaining and developing these scripts can become difficult and expensive. Fortunately, there are many ready-made solutions available, one of which is Synthesized TDK, which we examined in this post.

How to prevent unauthorized users from connecting to ProxySQL

Valentin TRAËN — Thu, 30 Mar 2023 00:00:00 UTC

ProxySQL is a great load balancer which however suffers from some shortcomings concerning the management of MySQL users. ProxySQL provides a firewall which, in my case, is not complete enough to properly manage users and secure their access. Indeed, this firewall does not accept subnets and keeps unauthorized connections in ProxySQL. We cannot then be sure of not suffering a DDOS attack on our ProxySQL instance. In this article, I will explain how I managed to overcome this problem.

Reminder of the principle of connection through ProxySQL

To understand what follows, you have to bear in mind how ProxySQL connects to MySQL. The user connects to Proxysql which then establishes the connection to MySQL. For this, ProxySQL maintains MySQL users in its internal database. The names of MySQL users, their passwords as well as the MySQL destination server are entered in the mysql_users table. At each connection request to a MySQL server, ProxySQL checks the presence of the user in the mysql_users table to connect itself to MySQL with this same user.

Something is missing, isn’t it?

Yes, the host associated with each MySQL user is missing!

In MySQL, users are configured to only be able to connect from ProxySQL. In ProxySQL we don’t have this information. By default, all users can therefore connect to ProxySQL from any IP address and ProxySQL will open connections to MySQL for them. As I specified in the introduction, ProxySQL provides a Firewall to overcome this problem, but this one is not really satisfactory.

Prevent connection to ProxySQL with an unauthorized user

In this part, our ProxySQL instance will allow user bob to connect to the MySQL (mysql_server) instance. Bob is allowed to connect from IP_1 but cannot from IP_2. The ProxySQL instance is running on IP_PROXYSQL.

In MySQL, user bob was created like this:

sql
CREATE USER 'bob'@'IP_PROXYSQL' IDENTIFIED BY 'PASSWORD';

In ProxySQL, let’s create bob like this:

sql
INSERT INTO mysql_users(username,password,default_hostgroup) VALUES ('bob','PASSWORD',0);
LOAD MYSQL USERS TO RUNTIME;
SAVE MYSQL USERS TO DISK;

and declare the MySQL server like this:

sql
INSERT INTO mysql_servers(hostgroup_id,hostname) VALUES (1,'mysql_server');
LOAD MYSQL SERVERS TO RUNTIME;
SAVE MYSQL SERVERS TO DISK;

As you may have noticed, I didn’t declare the same hostgroup when creating the user and the server. Hostgroup 0 does not correspond to any MySQL server. By default, our user bob will therefore be able to connect to ProxySQL but his queries will not be redirected to any MySQL server. Let’s move on to host management. I will declare each authorized host in the mysql_query_rules table. In ProxySQL, this table is used, among other things, to assign different parameters to a connection. You see what I mean? Let’s declare our rule!

sql
INSERT INTO mysql_query_rules (rule_id,active,username,client_addr,destination_hostgroup,apply) VALUES (1,1,'bob','IP_1',1,1);
LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

I have just declared a rule indicating that all requests coming from user bob connected from IP_1 must be played on host 1. And icing on the cake, IP_1 can be a subnet (IP_1%), which would not have could not be possible with the firewall. From now on, bob will be able to perform queries from IP_1 and get results from MySQL. If bob plays a request from IP_2, he will not be able to obtain a result since the hostgroup queried will be 0 which does not correspond to any MySQL server. However, this is not satisfactory. Nothing prevents bob from creating a very large number of connections from IP_2. It won’t reach any MySQL servers but may be able to crash my ProxySQL instance. It’s time to deal with those unauthorized connections!

ProxySQL provides a scheduler which will be very useful here. This scheduler will allow us to play a bash script every x ms. I created this script in the ProxySQL datadir:

kill_connections.sh

bash
#!/bin/bash

PROXYSQL_USERNAME="${1}"
PROXYSQL_PASSWORD="${2}"
PROXYSQL_HOSTNAME="127.0.0.1"
PROXYSQL_PORT="6032"

mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h$PROXYSQL_HOSTNAME -P$PROXYSQL_PORT -e "SELECT SessionID,user,cli_host FROM stats_mysql_processlist WHERE hostgroup = 0" | while read SessionID user cli_host; do
 if [ $SessionID != "SessionID" ]; then
 enabled_account=$(mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h$PROXYSQL_HOSTNAME -P$PROXYSQL_PORT -se"SELECT count(*) FROM mysql_query_rules WHERE username = '$user' and '$cli_host' LIKE client_addr;")
 if [[ "$enabled_account" -eq 0 ]]; then
 mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h$PROXYSQL_HOSTNAME -P$PROXYSQL_PORT -e "KILL CONNECTION $SessionID"
 fi

 fi
done

This script lists all the connections opened in ProxySQL on hostgroup 0. It then checks whether the connected user/host pair is authorized using the mysql_query_rules table. If not, the connection is killed. Let’s activate the scheduler in ProxySQL:

sql
INSERT INTO scheduler(filename, arg1, arg2, interval_ms) VALUES ('kill_connections.sh','proxysql_admin_user','proxysql_admin_password', 1000);
LOAD SCHEDULER TO RUNTIME;
SAVE SCHEDULER TO DISK;

Now, any connection opened in ProxySQL but not authorized will be automatically killed!

WARNING: unfortunately, the ProxySQL scheduler does not work like the MySQL scheduler. It is necessary to open the connection from a .sh file and therefore to indicate the ProxySQL administration credentials. These identifiers will then be visible by monitoring the list of server processes. To avoid this problem, I advise you to indicate the identifiers directly in the .sh file and to protect this file correctly on your server.

Additional Information

When I deploy ProxySQL, I always create a rule with a very high rule_id to block unauthorized connections; this is an additional barrier in case I forget something:

sql
INSERT INTO mysql_query_rules (rule_id,active,error_msg,destination_hostgroup) VALUES (999999999,1,'ProxySQL : Access denied',0);
LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

This rule redirects unauthorized connections to hostgroup 0 (if ever a user was declared in mysql_users with a hostgroup leading to a MySQL server) and displays an error message for each request. I create all my rules to manage hosts with a rule_id > or = 10000. This allows me to have 9999 empty slots if I ever want to create other priority rules in mysql_query_rules.

sql
INSERT INTO mysql_query_rules (rule_id,active,username,client_addr,apply) VALUES ((SELECT IFNULL(MAX(rule_id)+1,10000) FROM mysql_query_rules WHERE rule_id != (SELECT MAX(rule_id) FROM mysql_query_rules) AND rule_id > 9999),1,'USERNAME','HOST',1);
LOAD MYSQL QUERY RULES TO RUNTIME;
SAVE MYSQL QUERY RULES TO DISK;

Don’t hesitate to ask me questions, I’ll be happy to answer them.

How To Optimize the Structure of a Simple PHP Application as Your Project Grows

Daniil Bazhenov — Tue, 28 Mar 2023 00:00:00 UTC

Let’s discuss the structure of our simple PHP application, folders, files, and functions and transform the project to grow it.

You and I have developed a small PHP application that makes an HTTP request to the GitHub API and stores the result in the database.

Our code already performs different functions:

Reading environment variables.
Connecting to the database.
Running API queries.
Looping and writing to the database.

The code is already hard to fit on the screen, and it’s time to think about dividing the code into files, folders, and functions.

Frameworks are usually responsible for separating code into folders and files. But we don’t use frameworks, so we’ll do it ourselves. At this stage, we will not make the structure too complicated. Our task is to learn how to change it so that we can change it at any time in the future.

I prefer to change the structure of files and folders as the project grows, grouping files according to meaning and logic. In this article, we’ll make the first change, and we’ll do them more times in the future.

Let’s start optimizing our application.

Initializing and configuring the application

Usually, PHP scripts are executed sequentially starting with index.php, as in our case.

At the beginning of index.php, we connect the composer libraries, read environment variables, create a database connection, and create an object for HTTP requests. This will be required for each script. I propose to put this in a separate init.php file.

Create an app/init.php file and move the initialization to it. We simply move the code from the index.php file, leaving only the logic of the script.

app/init.php

// Enabling Composer Packages
require __DIR__ . '/vendor/autoload.php';

// Get environment variables
$local_conf = getenv();
define('DB_USERNAME', $local_conf['DB_USERNAME']);
define('DB_PASSWORD', $local_conf['DB_PASSWORD']);
define('DB_HOST', $local_conf['DB_HOST']);

// Connect to MongoDB
$db_client = new \MongoDB\Client('mongodb://'. DB_USERNAME .':' . DB_PASSWORD . '@'. DB_HOST . ':27017/');

$app['db'] = $db_client->selectDatabase('tutorial');

$app['http'] = new \GuzzleHttp\Client();

You may also notice that I created an array or $app object, and added HTTP and db as array elements. That way I can use $app anywhere to pass to functions and have HTTP queries and database handling there.

And in the index.php file itself, we simply include our init.php

_app/index.php

require __DIR__ . '/init.php';

dd($app);

And print $db and $http with dd() to make sure they work and are available in index.php.

Leave the rest of the code in index.php unchanged for now and run localhost in the browser.

So we split index.php into two files: init.php and index.php

When we start localhost, our nginx web server launches index.php, init.php connects there, does initialization and environment variables reading, and then the rest of the index.php code continues.

Let’s get to the functions

Functions are needed to group code that is executed multiple times. Functions can be initialized either in the executable PHP file itself, next to the code, or in a separate file that includes, such as composer libraries.

For example, we make a GET HTTP request to the GitHub API at a certain URL. We will probably need to make another type of request to another URL, but it will still be an HTTP request using guzzle.

I assume in advance that I will have many different functions: general, for GitHub, for the database. So let’s create a func folder in the app folder.

And in the app/func folder, create a github.php file. Create our first function there that will request the GitHub API.

app/func/github.php

function fn_github_get_repositories($app)
{

 $http = $app['http'];

 $url = 'https://api.github.com/search/repositories';

 $params = [
 'q' => 'topic:mongodb',
 'sort' => 'help-wanted-issues'
 ];

 try {
 $response = $http->request('GET', $url , [
 'query' => $params
 ]);

 $result = $response->getBody();

 $result = json_decode($result, true);

 } catch (GuzzleHttp\Exception\ClientException $e) {
 $response = $e->getResponse();
 $responseBodyAsString = $response->getBody()->getContents();
 echo $responseBodyAsString;
 }

 return $result;
}

A little clarification, we will modify this file in the future, the first function is not the best. You should understand that you will need to change frequently, changing parameters and variables within functions.

Now go back to the index.php, and connect our first function file and run the function.

app/index.php

require __DIR__ . '/init.php';
require __DIR__ . '/func/github.php';

$repositories = fn_github_get_repositories($app);

dd($repositories);

Run localhost and see the same result. Our request was executed, and we got the list of repositories.

So we cut our index.php file by more than half. All we have left in there is the database query. At the moment it doesn’t look too complicated to put it in a separate function, but. We need to change it because the $db object, we now have $app[‘db’].

In addition, the response from the API we have in the variable $repositories, which means that all loops and all operations must be done with it.

So far, my index.php file is like this.

// Enabling Composer Packages
require __DIR__ . '/init.php';
require __DIR__ . '/func/github.php';

$repositories = fn_github_get_repositories($app);

$app['db']->repositories->createIndex(['id' => 1]);

if (!empty($repositories['items'])) {
 foreach($repositories['items'] as $key => $repository) {

 $updateResult = $db->repositories->updateOne(
 [
 'id' => $repository['id'] // query
 ],
 ['$set' => $repository],
 ['upsert' => true]
 );

 }
}

dd($repositories['items']);

app/

.
├── func
│ └── github.php
├── vendor
├── composer.json
├── index.php
└── init.php

The result of running localhost in the browser will be the same, we will print the repositories that will be stored in the database.

I think we should stop for now and continue in the next article.

Conclusion

As a result of this modification, we learned how to split one script into several scripts and create functions.

You can divide it into files and folders and functions as you like, as it is convenient to you now or will be convenient in the future. It’s okay to change the structure as your project grows.

You can check and run the source code in the repository.

In the next post, we will continue to modify and improve our application. We will add new functions and features. I’m sure we’ll end up with a very useful app, and you’ll learn how to develop it.

Ask me questions, I’ll be happy to answer them.

How to Make HTTP Requests to API in PHP App Using GitHub API Example and Write to Percona Server for MongoDB

Daniil Bazhenov — Mon, 20 Mar 2023 00:00:00 UTC

We learn how to work with HTTP requests in PHP and make API requests using the GitHub API as an example. We’ll get the data from the API and save it to the database.

In the previous article, we developed a simple application that connects to the MongoDB database (Percona Server for MongoDB) and writes documents in a loop. We only used Composer packages to work with MongoDB. We have set up the Docker-compose environment and have the app/ directory where the application code is located. We can edit the code and check the result in the browser without restarting the Docker containers.

We plan to make HTTP requests to GitHub API. I prefer using the popular PHP HTTP client library guzzlehttp/guzzle.

Here we go!

1. Preparation

Open the project folder in the console or git clone the repository, and open the folder.

Start Docker-compose

docker-compose up -d

This will start the containers, and you can run localhost in the browser.

2. Connect PHP libraries using Composer

Composer is a package manager for PHP. You can find out-of-the-box libraries and functions for almost any task. I prefer to use something other than Frameworks for small projects, but just plug-in packages.

Open the app/composer.json file in the code editor.

Add two new packages, guzzle, and dd.

{
 "require": {
 "guzzlehttp/guzzle": "^7.0", // Guzzle for HTTP
 "larapack/dd": "1.*", // dd for debug
 "mongodb/mongodb": "^1.6",
 "ext-mongodb": "^1.6"
 }
}

Connect to the container with php-fpm

docker exec -it [php-fpm-container-name] bash

Update the packages with the command.

composer update

That’s it, we have the packages installed, more about Composer is here.

3. Let’s choose an API and read the rules of use.

I chose the GitHub API as an example because it is available with and without authorization and has good documentation.

First, we will get a list of repositories. We will use the repository search API for all repositories containing the topic mongodb.

I opened the REST API documentation and found the method, parameters, and examples I needed.

4. Let’s make the first request to the API.

Open the index.php file in the app folder and add the following code somewhere at the beginning, after the inclusion of vendor/autoload.php

$http = new \GuzzleHttp\Client();

$url = 'https://api.github.com/search/repositories';

$params = [
 'q' => 'topic:mongodb'
];

try {
 $response = $http->request('GET', $url , [
 'query' => $params
 ]);

 $result = $response->getBody();

 $result = json_decode($result, true);

} catch (GuzzleHttp\Exception\ClientException $e) {
 $response = $e->getResponse();
 $responseBodyAsString = $response->getBody()->getContents();
 echo $responseBodyAsString;
}

dd($result);

To summarize, here is what we did:

Initiated the HTTP client Guzzle.
Created a variable with a URL from GitHub API.
Created array with query parameters, in our case q search string.
Executed request in Try-Catch construct; if a request has errors, an exception will be thrown, and we can read the error.
Received response results from API into variable $result.
Converted the response from JSON to an array.
Printed out the array in a convenient form using dd($result).

Run localhost in the browser and see the result.

5. Exploring the result

dd() is a function that helps print any variable, array, or object. Just list them separated by commas.

You have printed the response from the API and see the array:

total_count - says that there are 72945 repositories for our query. This looks like an excellent dataset for our experiments with analytics and the database
items - contains an array of 30 other arrays with repository data.

I suggest modifying our API query to get a slightly different result. We’ll add sorting to the query.

Change the $params array in the index.php file.

Let’s add sorting by the count of help-wanted-issues.

app/index.php

$params = [
 'q' => 'topic:mongodb+language:php',
 'sort' => 'help-wanted-issues'
];

Run localhost in your browser, and you will see a different set of items in the API response.

6. Let’s save the data to the database

Instead of dd(), we add a foreach loop over items and print one repository data.

app/index.php

if (!empty($result['items'])) {
 foreach($result['items'] as $key => $repository) {

 dd($repository);

 }
}

Great, we see the data. This will be our document in the collection of repositories.

Replace dd($repository) with an insert query to the MongoDB database.

Remember to move the client db initialization, database connection, and read environment variables.

app/index.php

if (!empty($result['items'])) {
 foreach($result['items'] as $key => $repository) {

 $updateResult = $db->repositories->updateOne(
 [
 'id' => $repository['id'] // query
 ],
 ['$set' => $repository],
 ['upsert' => true]
 );

 }
}

dd($result);

Run localhost in your browser. I usually always type dd() at the end, since it stops the script and displays something like dd(‘finish’).

You may notice that making a updateOne request to the repositories collection, with the parameter upsert. This means that if such a document with an id (‘id’ => $repository[‘id’]) already exists in the database, it will be updated. If not, it will be inserted. This allows me to avoid duplicate data with multiple test queries.

7. Connecting to a database using MongoDB Compass.

We must see that we did everything correctly and the data appeared in the database.

Besides, we can play with them.

Open our new collection and check it out. We can also create an index by the id key in the Indexes tab. If you did not make the index with a PHP query.

Download and install MongoDB Compass here

Conclusion

The code of the application at the current stage can be obtained from the repository.

We did a great job and figured out how to make requests to the API, as you can make any HTTP requests.

In my next posts, we’ll figure out how to add authorization and work with API limits. The point is that GitHub lets unauthorized users make a few requests. We aim to get a lot of valuable data from the API and will do that soon.

We will also devote some time to the structure of the files and folders of our application. It’s already getting big, and we must make it more convenient.

Percona Monitoring and Management 2.36 preview release

Taras Kozub — Mon, 20 Mar 2023 00:00:00 UTC

Percona Monitoring and Management 2.36 preview release

Hello folks! Percona Monitoring and Management (PMM) 2.36 is now available as a preview release.

You can find the Release Notes here

Percona Monitoring and Management server docker installation

docker tag:

perconalab/pmm-server:2.36.0-rc

Important: In order to use the DBaaS functionality during the Percona Monitoring and Management preview release, you should add the following environment variablewhen starting PMM server:

PERCONA_TEST_DBAAS_PMM_CLIENT=perconalab/pmm-client:2.36.0-rc

Percona Monitoring and Management client package installation

Download the latest pmm2-client release candidate tarball for 2.36 by this link.

If you want to install pmm2-client package, please enable testing repository via Percona-release:

percona-release enable percona testing

install pmm2-client package for your OS via package manager.

OVA

PMM2-Server-2.36.0.ova

AMI