Digging deeper into the architecture of an open source product
Recently, Percona team announced the public alpha version of a new open source product – Percona Everest. It allows you to create database clusters on Kubernetes cluster.
I have installed Percona Everest several times and tried its features. Standard installation is very simple and takes a few minutes.
But, to understand the product deeper, I came up with the idea to explore repositories, build and run Percona Everest from source.
In this post, I will explain what I did step by step and what components and frameworks are used in the development of Percona Everest.
Architecture, components, and tools
At the top level, we have two components or tools on the user side: Percona Everest App and everestctl CLI tool.
The Percona Everest App is a basic application that provides a web interface for database creation and management functions. Percona Everest App can be installed on your computer or remote server. The App consists of two major components:
Frontend is a browser-based application providing a web interface for managing clusters and interacting with backend APIs. It is developed with React and TypeScript.
Backend API that process requests from frontend, interact with Kubernetes clusters and databases. It is developed on Golang and PostgreSQL as a database.
everestctl is a CLI tool for provisioning of Percona Everest on Kubernetes clusters. It is used to install Percona Everest components such as database operators on the Kubernetes cluster. It is developed in Golang and is provided as a ready-made executable, but, in this post, we will also build it from source code.
Remember that normally, when you install Percona Everest following the instructions in the documentation, the frontend and backend are built and integrated into a single container and run as a single unit.
Let’s get started with our experiments.
Frontend installation and launch
Percona Everest Frontend is developed using the Bit framework.
Bit is an open source toolchain for the development of composable software using React library and TypeScript.
Bit is used by about 100K developers, 250+ community plugins and has 16K+ stars on GitHub.
Clone the Frontend repository
You need to clone a repository with the Percona Everest frontend: https://github.com/percona/percona-everest-frontend
git clone email@example.com:percona/percona-everest-frontend.git
Open the Percona Everest Frontend source directory and run the following commands.
npm i -g @teambit/bvm
bvm install 1.0.0
bit install --recurring-install
Launching the frontend application
Moving forward, we have two options:
Run the frontend application using Bit.
Build a ready application and copy it to the backend.
The Percona Everest frontend repository contains versioning branches
release-[version] and the current version in development in the main branch. We will run the latest dev version from main.
Let’s run the command:
bit run everest --skip-watch
As a result, we can open
localhost:3000 in the browser.
The Frontend is now built, and we can move on to the Backend.
There is the other way to build a frontend to work with backend. You will need two commands:
bit snap --build
bit artifacts percona.apps/everest --out-dir build
In this case, the build will be done to the folder:
You need to copy all the files to the
public/dist folder of the backend repository. We will talk about backend in the next section.
The installation process may change over time, so I recommend to keep track of the up-to-date commands in the files:
CI/CD of GitHub configuration, file
.github/workflows/ci.ymlin the repository.
So we’ve launched Frontend, and now it shows an error because it sends requests to the Backend API, and we don’t have it yet.
We will need to clone the repository with Percona Everest Backend
Generally, it is an API that interacts with the frontend, processing requests and sending them to the Kubernetes cluster.
Let’s get it up and running.
Run PostgreSQL locally
You need Docker to run it. I hope you have Docker installed.
Let’s run one of the two commands in the repository directory:
docker-compose up --detach --remove-orphans
Using Docker for this process will be replaced by Kubernetes. You can see the YAML manifest in the file:
Launch the Go app
We have two options, I use:
go run cmd/main.go
But you can also use:
Starting with version 0.4.0, you will need to add the SECRETS_ROOT_KEY environment variable before starting the application. The secret key must be used on restarts if you do not start from the scratch.
export SECRETS_ROOT_KEY=$(openssl rand -base64 32)
Now we can open the localhost:3000 in the browser again and check that the backend is running. But we see that we have no Kubernetes clusters connected and configured.
Everestctl and Kuberentes cluster
Another important component of Percona Everest is everestctl. everestctl is a CLI tool responsible for provisioning Percona Everest on Kubernetes clusters.
We will need:
Build and run everestctl
Preparation of the Kubernetes Cluster
You can use Kubernetes Cluster on AWS, Google Cloud, or minikube.
The Percona Everest documentation says:
You must have a publicly accessible Kubernetes cluster to use Percona Everest. EKS or GKE is recommended, as it may be difficult to make it work with local installations of Kubernetes such as minikube, kind, k3d, or similar products.
The documentation provides instructions on how to run the test cluster
In this post, we use minikube which is preliminarily installed.
Let’s open the directory of the backend repository and launch minikube:
As a result, I see in the console this message:
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
I also see that I have a minikube cluster with three nodes. It is time to install Percona Everest components using everestctl.
➜ percona-everest-backend git:(main) ✗ kubectl get nodes NAME STATUS ROLES AGE VERSION minikube Ready control-plane 3m44s v1.26.3 minikube-m02 Ready <none> 3m22s v1.26.3 minikube-m03 Ready <none> 3m3s v1.26.3
Let’s clone the repository: https://github.com/percona/percona-everest-cli/
Open the repository folder and run build:
everestctl is built in the binary file
Grant execution privileges
chmod +x ./bin/everest
Let’s run the Percona Everest installation:
./bin/everest install operators
That command will start the installation wizard.
I will leave all default values, just pressing Enter. Otherwise, you can experiment with the settings, it is your choice.
As a result, the following processes will run on the cluster:
Creating namespace percona-everest.
Installing Operator Lifecycle Manager (OLM).
Installing Percona OLM Catalog.
Installing Percona Operators for the databases selected in the wizard.
Installing everest-operator operator.
Creating services and roles.
That’s it! We’ve installed Percona Everest completely. You can open it in a browser and create a database.
Try creating databases with different configurations.
Repeat the installation with a different cluster or settings.
If you face any problems or have ideas on how to improve components, create Issues on GitHub in the appropriate repositories.
Stop and remove Percona Everest
Once you finished your experiments, you can:
Stop Frontend - stop Bit running by pressing CTRL+C in the console.
Stop Backend API - stop Golang script by pressing CTRL+C in the console.
Stop and remove PostgreSQL in Docker - run
make local-env-downin the backend repository, or use Docker Desktop to stop.
Remove Kubernetes cluster.
Every day, developers make changes to the code and publish to repositories on GitHub.
You can stop the component, pull the changes with
git pull, and start a new version. It’s just for experimentation and development. Some versions of components will not be compatible; uninstall all components and start over using the appropriate versions. Detailed instructions on how to upgrade will appear later.
You can see changes to the build process or parameters in the repositories.
Couple of useful commands
List of databases
kubectl -n percona-everest get db
List of pods
kubectl -n percona-everest get pods
I hope you were able to make it this far, and it was interesting for you.
I’d love if you leave your feedback in the comments. ∎