Installation of MongoDB via a Kubernetes Operator - Community MeetUp - March 10th

Check this full recording of the Percona Community MeetUp of March 10th hosted and learn more about installation of MongoDB via a Kubernetes operator. Matt Yonkovit, the Head of Open Source Strategy at Percona, and Sergey Pronin, Product Owner at Percona, took us through the step-by-step setup to get us started with a new MongoDB cluster.

Video

Transcript

Matt Yonkovit:
Helloб everyone out there in the stream land. Welcome to another Percona Livestream. I’m your host, Matt Yonkovit, the HOSS at Percona, Head of Open Source Strategy. If you can hear us, feel free to give us a wave on chat. We’ll be following along here at the bottom of the screen. Hopefully, everyone can hear us audiovisual looks good. And today we’re joined by Sergey, Sergey Pronin, our group product manager, and he’s going to be talking to us about Kubernetes. Specifically the MongoDB. Operator for Kubernetes. How’re you doing today, Sergey?

Sergey Pronin:
I’m doing good. Good. Thank you. Thank you for having me today.

Matt Yonkovit:
Yes, and I’m glad to be on this. It actually gets me out of doing some housework. My wife has been home. She’s been on leave for a couple of weeks. And she’s been piling me up with chores. And so every time I can get on a stream and do something like this, it gets me out of actually having to like what, well, you’re not on a call, come down and help me with this clean this do the dishes do something else. So I’m glad that we can have this conversation to prevent me from having to go do those awful things and do something a little bit more fun. So if you can introduce yourself, you’ve been on this before. But for those who are joining us, maybe Sergey, give us a little bit of background. Before we jump in?

Sergey Pronin:
Yeah, yeah, sure. Sure. So it’s not my first time here. Definitely. I think it’s the third, right. I am a group Product Manager here, at Percona. My focus mostly is around our Kubernetes operators. But also help our product management teams to thrive and succeed. And today, I am mostly going to focus on our MongoDB operator, which we built here at Percona, which is fully open source. I’m going to even show you some demos, how to deploy it, how to manage it, what value it provides, and so on. But we can start with a short presentation and then jump to the demo face.

Matt Yonkovit:
Sounds good? Sounds good. All right, Sergey. So obviously, Kubernetes is a hot topic for many of us. It’s something that we’re seeing more and more people look into, to deploy not only their applications, which has been around for a while , but now databases. So why don’t you tell us? Yeah, you know all about it.

Sergey Pronin:
Yeah, Q&A has been a hot topic, I think. I don’t know, a couple of years ago, and now it’s just becoming the norm like Linux. Everyone knows how to use it, or at least understands how it works. And yeah, it’s getting more and more adoption. And it’s becoming known for lots of companies. So let’s start with a quick slideshow. I just want to give everyone some kind of an understanding about Kubernetes operators, what are? How Percona jumps in there, and so on, and how it all started. Okay, great. So let’s start with the whys. Right? Why databases on Kubernetes. So as you might know, it all started a long time ago, with bare metal servers, right? Then you were running some software, including databases on your bare metal. And it was kind of normal. You had a big box, he put your software there. And that’s it, right. But once we were progressing, and evolving, VMs appeared virtual machines. And it was a huge leap forward. Because now you can run multiple software pieces on a single box. And it was awesome, right? And it had isolation, it had everything you need. And at the same time, you can have your database, you can have your application running on one machine, which is great. But DBAs if you remember a long time ago, when VMs were introduced, what kind of hesitant, right they were saying, okay, VMs it’s something new. What about performance bottlenecks? What about performance degradation? I don’t want VMs. I’m used to my hardcore Kernel server. So why would I migrate? And nowadays if you see it was 20 or 30 years ago, even. And nowadays, you see that everything is a virtual machine. If you see RDS, Amazon RDS, Google Cloud. Everything is a virtual machine and databases run there and they run they’re pretty good, right? So you don’t have any performance thoughts, right? And later on, Docker containers appeared out of nowhere. Well c-groups, control groups were in Linux for a long, long time. But Docker, just helps users to adopt containers in an easy way. So you can tag to your application, you can run it anywhere you want. And it was a big leap forward. And now we’re in the phase where you need to manage all these hundreds of containers that you have with Kubernetes, and we’re also the stage where DBAs architects and SRE is still hesitant about moving the databases to containers. And it’s the same discussion like Okay, what about performance degradation? What about management? What about isolation? Right? And that is where we see the need to explain that containers are not evil for databases, it’s, it’s a good thing. And apart from the evolutional perspective here that took me some quiet time to get there to containers. But still, apart from the evolutional perspective, Kubernetes provides great tooling for automating stuff. So you don’t need to move your VM from one fast to another or your container from one place to another, in this case, Kubernetes does it for you. It’s a perfect tool to automate stuff and with operators is getting even simpler. And what we saw said Percona users and customers were coming to us asking me how can I run Percona databases on Kubernetes? How can I do that? And the obvious answer would be okay, take our Docker container and run it. Right. It’s the simplest way. But we understood that if we do that, the complexity of Kubernetes and its nature, that containers are volatile, it can heavily impact data consistency and data availability, database availability for all users and customers. And that is why we decided to think about operators how they can help in this sense, right.

Matt Yonkovit:
So, Sergey a couple of interesting points as you’re talking here, it’s interesting, because the database infrastructure and the infrastructure often lags behind the application development cycles, right? So when you talk about containers, and Kubernetes, for instance, it’s really become the de facto standard for a lot of development of applications now. And it’s just now really coming into its own on the database side. So it’s similar. You mentioned VMs, I remember these discussions like, Oh, should we put run Oracle, or SQL Server or whatever database on VMs? And it was like, no, because you’re gonna lose performance , and we need that 10% extra performance. I remember having those conversations back in the day. It seems like as the new things start to become established, the application paradigms kind of pulled the infrastructure and the databases along with it.

Sergey Pronin:
Yeah, yeah, I agree. This is what we see as well. And it’s a good observation that databases usually come as the last piece, right? Because first, you move your application, which is usually stateless, and the best-case scenario. And then you start thinking, okay, now I have 90% of my code or workloads running on Kubernetes. And I have 10%, which is databases are the staple stuff, running outside of Kubernetes. Hmm, do I need to move everything to Kubernetes? Now, and when you start thinking about it, definitely it starts to make sense because it’s easier to have one platform to manage versus two, or even three or four. Right? So this is what we also see nowadays.

Matt Yonkovit:
Now, the last thing I want to highlight on this is I think you’re calling me a caveman because I was part of that VM discussion and you have a caveman for the VM, so I’m not gonna take offense to it, but I think you’re calling me a caveman.

Sergey Pronin:
Oh, no. Well, I’m not that young. I look young. But I’ve been there as well. Right. So we’re both cavemen. Okay, right. Okay. Okay, moving forward. So let’s look into the use cases for databases on Kubernetes. At least how we see these use cases nowadays is the first one is the most common, I believe, for us is something as a service. When some company or usually its company decides to build a database as a service or platform as a service and they have hundreds or 1000s databases that they need to run this toy backup manage. It becomes a burden without any databases as service and operators and Kubernetes provided provide great tooling for that. So instead of doing lots of things manually, you can automate it with either your scripts or actual operators provided to you directly, right. Another good one is the dev and test environment. So usually, as I said, first you start running your applications on Kubernetes, and then slowly you start pulling your Stateful applications to the same platform. And dev test is a good place to start, as you have your application on Kubernetes already. And now you can spin up a dev-test database, connect to it, play with it, understand how the operator works, and then probably move to production environments. But this is where usually, new users start with like, Okay, I want to run it in depth-first. And they run it for a year, for example, and then they say, Okay, looks good. Let’s move production, the same paradigm. And the third, good use case that I was surprised to see a lot is Kubernetes-shop, right? It’s when some company makes a strategic decision. Okay, we are going all in to Kubernetes. We want everything on Kubernetes doesn’t matter whether it’s stateful, or stateless, everything goes there. And we’re going to use operators or something else to run our applications. And there are a few reasons behind it. One of them is vendor lock-in. We saw it with one of our customers, and they were running their applications on Amazon, on EC two on redshift on rds. So they were huge Amazon shops. And then one day, for some reason, they made a strategic decision that they want to go to GCP. I don’t know what was behind this decision. But they decided, Okay, let’s go to GCP. And the management understood that probably the next move would be to Asier or to on-prem or to some other cloud, as environments can change and decisions can change. And they decided, Okay, we are going to use Kubernetes for everything because it’s easier, you can move your Kubernetes cluster from one place or workloads from one Kubernetes to another without changing almost anything. And it’s an easy migration and that helps a lot to avoid this vendor lock-in. And I understand this Kubernetes is shelf mentality. Now,

Matt Yonkovit:
That’s more of a portability play, ensuring you can move and run wherever. I mean, honestly, cloud providers do have bad days, there are outages. So multi-cloud is a thing, even though some people don’t like to talk about it. Having that capability to flip back and forth is sometimes needed.

Sergey Pronin:
Yeah, yeah. And also is good for not only multi-cloud, but for hybrid cloud, when you have a piece of your workloads on-prem, but you want to touch cloud a bit. So Kubernetes is a good solution there as well. Okay, and bad use cases are huge mainframes. Obviously, if you have monstrosity, on monsters, I don’t know how to say that huge server with petabytes of data. Definitely, you can move it to Kubernetes. But it’s not a good use case if Kubernetes is still about containers. It’s still about microservices. And definitely, you can again, but we would not recommend it. It just doesn’t make any sense. And another one that we see a lot is what I call a startup virus, right?

Matt Yonkovit:
If you call it a virus, yeah.

Sergey Pronin:
Yeah. If you’re a small startup, and you say, oh, Kubernetes I heard, it’s great. Let’s use that. And then you start using Kubernetes and you have one small application written in Angular, and you have a small database. Obviously, why use Kubernetes for that. It is just over complicate things for you. Of course, you’re thinking about, Oh, we’re going to scale we’re going to have 1000s of servers or databases, probably. And I really hope you are but don’t do that. Start small with something simple. And then start using Kubernetes even I love Kubernetes, but I understand Kubernetes produces lots of complexity, you need to manage it, you need to run it. And it’s not an only database, Kubernetes as a whole, right? Good. So let’s move forward a bit quicker cuz I also have a demo. I don’t want to keep you guys waiting. So databases, On Kubernetes it’s an easy view, a or overview, if you look at any stateful application that Kubernetes consists of two parts, it is Kubernetes primitives, it is stateful, sets, pods, secrets, storage services, whatever it is. And it is configuration. So for MongoDB, imagine you have a replica set usually consisting of three nodes, and you need to configure all these nodes to talk to each other, you need to ensure that communication between these nodes is secure. So you need to put SSL certificates on all these nodes and so on. And if you’re on Kubernetes, you need to create files, you need to ensure that containers are there, storage is there, and so on. So there are lots of moving parts there on both ends. And doing it manually is a lot of complexity. And it also becomes pretty hard if you think about not only deployment but also about management, like okay, how do I scale up? Okay, I need to add more nodes, oh, I need to configure it, oh, I need to put a certificate there, I need to do that. Of course, you can use some kind of Ansible bash scripting for that. So but if I saw this, and it works, it usually requires a separate team to manage your bash scripts or infer the code to manage it all. And why bother if you have ready for this tool. And if we look at MongoDB on Kubernetes is this simplistic view of it on the left is when you have a single replica set, you have three pods. And you have free storage volumes for it. So they form a replica set. And then you have a service that just exposes your replica set to the outside world. So you can connect to it and do something with the data. And if we talk about the sharded cluster, it’s a bit more complex, as you add also Mongo into the picture, which is a relative for your queries for the sharded cluster, which talks to Config Server where metadata is stored and figures out okay, where should I route my query to which shard which replica set. And as you see in this picture on the right, there are nine parts already for a single sharded cluster. And imagine if you configure it all manually or with your scripts, it would be like, I don’t know how many hundreds of lines of code you’ll need to maintain. So it’s pretty complex. Right? And the operators, what operators do, they kind of simplify the way you interact with this Kubernetes primitives and database configuration files. So instead of going and configuring it all, you just go to the operator and send this YAML file to it. As you see on the right there is an example with something like okay, I want the replica set, I want three nodes, I want backups like this configured, I’ll show it in the demo. But the idea is you don’t need to know anything about Kubernetes primitives. You don’t need to know anything about how database internals is configured. You just say, Okay, I want this database cluster with this configuration period. And operator does the rightest rates the Kubernetes primitives it configures the containers, and so on. But on top of that operator also allows you to do day-to-day operations for maintenance operations. Like I want to scale, I want more nodes, I want more resources, I want to take it back up, I want to update my MongoDB cluster, I want to put it on the maintenance, I don’t know, pause it, or whatever. And you can also do that through the operators. You don’t need to write additional hundreds of scripts to manage it. Or remember all these commands operators can do it for you. Right. And for those of you who don’t know anything about operators, a simple operator is just a piece of code, which runs in Kubernetes. In a container in a pod along with the custom resource definition is CRD. Custom resource definitions allow you to extend Kubernetes API. So instead of creating pods and Kubernetes is you can create a custom resource so as a user, you just create this custom resource here and say okay, I want this MongoDB cluster and you talk to Kubernetes API about it. And then the operator does the rest. It monitors the creation of such custom resources. And like the rest as I said before, it creates Kubernetes primitives it configures the database, and all the magic is done and as a result, what you get you to get is service So you don’t get multiple nodes that are not configured or you need to figure out how to monitor them or whatever, you get the service, you get an endpoint to which you can connect, and play with the database right away. So this is the magic of the operator. If we talk about Percona operator, as I said before, it’s 100%. Open Source, we are open, shipped certified. So it means we own Red Hat marketplace, you can install it easily. And you would get support from Red Hat on that. We have an official Helm chart, I in my demo, I will be showing everything in YAMLl files, because it’s easier to demonstrate things and explain. But the same thing you can do with Helm chart, if you are more used to it, we have sharding support, which means you can deploy a huge sharded cluster on your huge Kubernetes environment, do some magic where the shard keys, and so on. Integrations Yeah, we do have an integration with Percona Monitoring management, I will not show it in the demo. But in a nutshell, we have this nice tool to monitor and manage databases. And our operator can just integrate with it. You don’t need to configure anything again, it just connects to it. And you have all the metrics and alerts in place out of the box. But if you have your monitoring stack, for example, you can use custom sidebars and configure them and hook it up with your own monitoring. And it will work as usual. And as I said before, they do operations, which is maintenance, like backups restores point in time recovery, scaling, and many more. It’s all possible with the operator and I’ll show some of them in the demo. So Backup and Restore, I will skip this slide I will show it better in the demo not to waste time now. Scaling, I also show it in the demo, but I hope you all understand how are the different types of scaling. So vertical scaling is when you have a node with seven CPU and RAM parameters. And then you say, Okay, I’m hitting my limits, and you add more CPU and RAM. Right. And obviously, vertical scaling is not internet, you will hit some time some kind of limit of I don’t know your server or virtual machine. And at some point, you will need to do horizontal scaling, which is adding more nodes into the picture. But with horizontal scaling, you can solve the problem with reads. So you can read from all these nodes but still write in MongoDB goes to a single node. And to solve the problem with scaling of writes, you need to do sharding. And sharding allows you to scale and break your data into chunks into the shard, right. And then you can scale them even more. So I’ll show some of this in the demo as well. But it’s important that you know that. Okay, and external nodes. This is I’m just praying that we have it in our home operator as well, is the functionality that I personally love is that you can have multiple clusters deployed in different Kubernetes clusters with or without an operator. And you can connect them so that you have a multi-regional deployment. It is especially useful for hybrid multi-cloud or edge nodes deployments. So I really love this functionality. And we have it in operator. I’ll show it, please, because I have it in the demo. Okay. Oh, yeah. Yeah. So demo agenda. And what I’m going to do, I’m going to deploy a fresh Kubernetes cluster so that you know that I don’t cheat. I’m going to use GKE for that.

Matt Yonkovit:
No, just as it’s no cheating.

Sergey Pronin:
Yeah, no cheating, no cheating. While the Kubernetes cluster is going to be deployed, I am going to explain the files, the YAML manifests. Then I will deploy the MongoDB cluster, a MongoDB operator, and then I will show some of our maintenance tasks through the operator that I was talking about. And I really hope the demo would go smooth, right?

Matt Yonkovit:
So we do have a question that came up. Well, while you’re pulling that up, John Moser asked can you address the problem of Kubernetes when members of replica sets are rescheduled to different nodes and cause a cluster to not be available suddenly?

Sergey Pronin:
All of the nodes. So, the fate so replica set. And I will show it to you, we have a replica set of three nodes, right? If one node dies, it’s going to be rescheduled automatically to another note, and a persistent volume claim is going to move along with it. It’s not a problem, right? If all of the nodes die, well, your cluster will be down, but then they will start off automatically. You don’t need to do anything. It just works. I’m just reading the question. So please let me know if I answered the question. If not, please clarify what exactly do you mean? But actually, it is all automatically recovering. If, if one node fails, for example, or two-node, it doesn’t matter. Yeah.

Matt Yonkovit:
John, if we need to get deeper into that specific one, we can also have Sergey in you connect offline as well. Yeah, but yeah, I don’t know that detailed. But if there’s any clarification, you can provide Feel free.

Sergey Pronin:
Okay, so ah, please type in the chat if you see my console. Good. Or do you need me to add the size of the font a bit.

Matt Yonkovit:
Yeah, I think, for now, it’s, it should be fine. Okay.

Sergey Pronin:
We can adjust going forward. Yeah. Okay. So we’ll keep it. Okay. So what I’m going to do now, I already have G Cloud installed G Cloud as a tool to connect to Google Cloud. And I have a command here, which is going to provision a cluster Kubernetes cluster called webinar demo. The version of Kubernetes is 121. I pick the one out of my head machine types, I’m using kind of standard, I’m going to use preemptable nodes, which are like spot nodes on Amazon because I don’t care. And I’m going to have a six-node cluster because it will be easier for me for the demo. Right? So I’m going to deploy it, it’s going to be created. And meanwhile, I’m going to jump to the files that I promise to, show you right, so we have this, this is the first file that I’m going to apply on Kubernetes what is it? It is a custom resource definition, right? This is what I was telling about extending the Kubernetes API. So instead of creating the pause, the services, and so on, I now will be able to tell Okay, create me Percona MongoDB cluster, instead, I will tell it to Kubernetes directly, right. And this is going to create it. Custom resource definition is pretty long. I don’t want to go into details, but it’s all the configuration of it, and so on. And what else it’s going to create, it’s going to create, again, a Custom resource definition for backups. I’ll show it to you a bit in the demo as well because we have a separate resource for backups and restores. It also creates some roles so that my operator can provision Kubernetes primitives on Kubernetes. So that I give access to it with this is going to create service accounts. It’s the same part of role-based access control of Kubernetes. And it’s going to deploy a deployment. And this is actually where the operator code is. So it’s going to create a container pod with the operator. That’s, that’s all it’s going to happen here. Right? Okay, Kubernetes is created. And another piece that I’m going to deploy today is this, at first look, it looks pretty big. But in reality, it’s not. Once you understand how it works, it is the Percona Server for MongoDB is actually our Kubernetes cluster, right it is what sorry, not given a cluster but MongoDB cluster. What I’m saying here are some more important pieces is that the version of MongoDB is going to be for then, I’m going to deploy a replica set called RS zero. And the size of a replica set will be three which means that I’m going to have three nodes in my replica set and here also So you see this configuration that I was bragging about its external nodes so that you can add any external node to this replica set, it can run on other Kubernetes cluster, or it can run on-prem, on VM, it doesn’t matter, but you can connect it and use it as a regular thing, right? This is the affinity configuration, which is also pretty important because we don’t want to have all replica set nodes on one Kubernetes node because if this node goes down, all my replica set goes down. So what I’m doing, I’m placing my replica set of nodes on different Kubernetes nodes, so I kinda ensure that they run separately. And this affinity topology key. You see, it’s based on hostname. But if you have Kubernetes cluster, which is deployed on multiple availability zones, you can have the different keys here, and you will ensure that your replica set nodes are running in different availability zones, which is awesome. So it’s pretty flexible. Sidecars is another important face. This is for customization. If you want, for example, to run, I don’t know MongoDB exporter to monitor your MongoDB cluster, you can use that. But here you can also configure PMM, which is Percona monitor and management and it will do it automatically without any configuration. I’m not using it for now, because it’s not part of the demo.

Matt Yonkovit:
So, sidecar is basically called out to other applications or other things that you want as these are getting built.

Sergey Pronin:
Yes, correct. So, if you have some custom monitoring, if you have some sysbench or you have some logging solution that you want to have here, you can just run it as a sidecar container pattern is invented to extend your application in Kubernetes, without changing the application, because containers in the simple pod, they have access to networking, they have access to storage. And to process this, I guess, right, so. And as long as you run it as a sidecar, you have full access to whatever you need to in MongoDB. So you can do lots of magic here with this customization. Right? So next, we are for disruption budget, which means that if we have a node fail, or at least one node is going to be alive, or if we do maintenance, it’s a standard Kubernetes thing. Exposed section, I’m not going to expose replica set nodes. I will explain a bit later why resources, as you see if you’re familiar with the concept of resources in Kubernetes requests and limits requests is what your what you think your application is going to consume and limits are the cap. So it cannot consume more than this. Right? Volumes back again, if you’re familiar with Kubernetes, it’s not going to scare you. I’m saying that I’m going to use persistent volume claim with regular lights only. I’m not going to put any data there. It’s just for the demo. So it’s enough. And this is also cool. I love this one, it’s nonvoting members. In the MongoDB world, it is very important specifically, edge computing. For example, if you want to run MongoDB in your main data center and have another node somewhere in your office so that you can access it faster, then you can use it. I really love it. And arbiter. Our operator also allows you to run arbiter nodes arbiters are, do not hold any data, but they help you in the primary selection process so that you don’t lose quorum. Sometimes it’s very important to have them and do not waste time on storage. Yep, moving forward, we also have sharding enabled. sharding can be done even for one single shard. So it’s fine for me. I’m just doing it for a demo. So I will show you how it works with everything. This is Config Server replica set is the replica set that holds all the metadata of your MongoDB cluster and this is Mongo’s. It is the router for Mongo. It kind of routes your queries and does magic. We’re also going to deploy three nodes of Mongos for high availability. And this one I’m going to expose with a public load balancer. Again, it’s only for the demos, so don’t do it at home. Don’t expose it with Luckily, I had to do at least with an internal load balancer, but I’m going to do that. And another piece, which is important here, I’m going to show you the backups how they work. And here, just configure the backups. So I’m going to store my backups on s3. Actually, it’s on GCS, but GCS is s3 compatible. And I’m going to use this bucket, I already created it on GCP. And that’s it. Yeah, this is the secret that I’m going to use. It has my GCP access keys to this bucket. So this is a huge file, right? It looks like it. But in reality, it’s going to deploy your MongoDB cluster and provide your service, you can connect and start using it. And I can remove almost all of this here and have just a few lines left. But for the sake of the demo, I decided to go through this huge file and show you the power of our custom resource. Okay. So let’s get to action Kubernetes cluster is ready, let’s see, how many nodes do I have? Okay, I have six nodes. Good. So everything is right. So at first, I’m going to deploy my bundle, it’s what I said it’s going to deploy our custom resource definitions. One for the cluster, one for backups, one for restores, all the roles and everything. And the last piece is deployment, which is the operator. So if you see, I have this deployment and already have this pod deployed here, which is the operator. So operator is our mastermind, it controls the class, the deployment, creation of Kubernetes, primitives, and configuration of the MongoDB cluster. So it is an important piece for sure. And the next piece, I’m going to deploy my backup s3 Yaml. It is my key for s3, so that I can demo the backup and restore process. Let’s deploy it. I will delete this case anyway.

Matt Yonkovit:
Yeah, yeah. Let me show you my secrets.

Sergey Pronin:
Yeah, no worries. They’re pretty restrictive. So. Okay. Yep. So the secret tax rate, and now it’s time to deploy the database itself. This is the CR custom resource Yaml, I’m applying it to this huge monstrosity. Right? And there we go. So, as you see, the podes are going to start creating a webinar demo. It starts with the CFGs config replica set. But we also have this PSMDB. Object now in Kubernetes, so I can directly go and say, Okay, give me this object. And it shows me the status, it is initializing. And it will take some time for initializing as we need to provision, nine pods, and the load balancer. So definitely, it will take some time. Let’s see. Yeah, there we have any questions while scanner?

Matt Yonkovit:
I don’t see any questions that came up? So I think it’s continued.

Sergey Pronin:
Okay, I was walking along, and I need some breaks.

Matt Yonkovit:
That’s okay. Because now, now’s the time of waiting, infinitely waiting. Right? So you always have to do that. Now, as things are deployed. You mentioned like backups are part of this. You can also do PMM, our backups are automatically scheduled, and will they be taken, like nightly? Or is there a schedule that’s already kind of implemented out of the box when it starts up?

Sergey Pronin:
This is a good question. I’m not using this functionality in this demo. But as you see here, we have the section tasks. And either allows you to configure your backups in crontab-like for you, right? So we can have a schedule, like think this backup is daily guessed right daily. You can choose a compression type, you can choose the storage where you want to upload the backup on this schedule and you don’t have multiple schedules. So yeah, definitely works like this.

Matt Yonkovit:
Okay, so right now you set it up. So you can do backups. You just won’t do backups unless you manually call it.

Sergey Pronin:
All right. Yes, yes. But in any production environment, obviously, you should have some kind of schedule to take the backups. And you don’t. You don’t want to do them manually. Yes.

Matt Yonkovit:
And also right now, while PMM will be connected, it won’t set up any alerts out of the box. Correct. So it’s not like it’s going to come out of the box already configured to monitor and alert you to certain activities yet.

Sergey Pronin:
Correct. Yeah, you’re right, you’re right. He will not arrive, you will have all the dashboards in place with all the metrics out of the box. And probably also going to have advisors. But it’s another, webinar, that goes through the PMM.

Matt Yonkovit:
I was just asking, because as people, like, if you could figure backups from the operators, you deploy out, I think that the next logical step for many would be they also want any alerts or anything that’s going to make them aware of an issue to automatically be configured as well. And knowing that it’s not quite there yet. There is some manual effort on that.

Sergey Pronin:
Yeah, yeah. And what you also need to remember, even if you set up your monitoring for the database, do not forget to set up something for Kubernetes. Because definitely need to know like, if your note goes down, yes. What’s happening with this environment? Because, again, it is all about Kubernetes complexity. Even it is easy to deploy a database with. Without an operator, you still need to manage the Kubernetes itself. Yeah. Okay. Let’s see what’s going on. With all this stuff in if for some reason.

Matt Yonkovit:
Why is it stuck? What did you do?

Sergey Pronin:
I guess it’s fine. Maybe PVCs provisioning or something. I’ll attach our volume is already attached. Oh, no. No. Yes. PVC already? Yeah, I think I already had PVC there. Let’s see. Let’s see if I can. I will delete this PVC. And see what happens. Because I think I had some leftovers. Ah. Oh, no, it’s good. Oh,

Matt Yonkovit:
the pod is still initializing demo Rs. 202. Right there.

Sergey Pronin:
There. Okay. No, it’s all good. Oh, it was some glitch, I guess. Okay, good. So let’s see. PSMdB. is ready. Okay. Magic happens. Right? Happens. Okay. So, now my cluster is ready, as you see. It says ready and also chose this endpoint for me, which is actually, if you see this Kubernetes service, which was created by the operators, so it’s the type of load balancer. And as long as I do publicly, no dude home. This is a public IP, which I can use to connect to my MongoDB server. And this is actually what I’m going to do. But before doing that, what I need to do is I need to get the password and user in a production environment, you are going to have something like what was it this one. This is actually a file with that with users and passwords. Right? It is some dummy users and passwords, which we have here as an example. Obviously, don’t use them for prod. But if you don’t specify this file, if you don’t create this secret, before deploying the cluster, it’s going to be created automatically for you with a randomized password. And this is what happened, actually here. So I have this webinar demo secret. And if I look at it, I’m going to see a bunch of users and passwords which were created for me automatically. And I’m going to use, for example, this cluster-admin user, it has a cluster-admin role so I can do some magic on the class and I will show you this There you

Matt Yonkovit:
So, which is an awesome username. I don’t I think you might have a unique username completely unique. I don’t know if anyone has ever used that as a username.

Sergey Pronin:
Now it’s base 64 encoded. So actually, it’s not that old, but it’s, you see, its cluster admin and password is actually much more unique. It’s, let’s see, what is it? Because it’s it was randomly generated. Yeah, it’s some Yeah, some rubbish. There you go.

Matt Yonkovit:
Which is, okay. Is it is a good point too, for everyone to remember that it’s base 64-encoded, right?

Sergey Pronin:
Yeah, it’s Kubernetes secret, it’s always base 64. Okay, let’s connect to my MongoDB database with this user and password on MongoDB cluster admin. And even if I want I cannot take this, let me copy-paste it.

Matt Yonkovit:
Yeah. So yes, you get a question here on this is running on GCP. How much effort would it involve to run on another platform like EKS? It should, it should be almost the same?

Sergey Pronin:
Yeah, yeah, I’m, I would be frank with you, I’m using GCP, as the creation of a new fresh cluster takes like, I don’t know, 10-minute stops or five minutes. When I want to play with EKS. It takes me 30 minutes to provision a cluster. And it’s not my fault. It’s just how EKS works. It’s pretty slow. But in a nutshell, we test our operators on GK on EKS. On regular upstream, Kubernetes on OpenShift were certified there. And this is where our creation ends, right? So we ensure that it is found in all these platforms on all the versions of these platforms. And yeah, I tested it on Asier. It works. I tested it on rancher it works. So I guess it should work on any Kubernetes flavor. But we should always remember that Kubernetes is like building blocks. So obviously, if you have some strange storage or networking like CNI, you might face some issues, which we cannot predict. But on most of the platforms think it’s 90% of the cases, it would work. Okay. Okay, good. So let me connect to my database. Okay, you see, it works. So I have Mongo here. Let’s do something. Let’s lose the shards, right? So as you saw in my custom resource YAML, I have only one shard called RS zero, which is one RS zero replica set. And that’s it. Right? So this says, My MongoDB cluster. Now, I can play with it, I can insert data, I can get the data wherever I want. But let’s do something else. Let’s, let’s scale it. As I promised to do, lads, instead of three nodes, I’m going to have five nodes, okay, in my replica set. So what I have to do is I have to save it. So change this YAML file, save it and apply. And that’s it.

Matt Yonkovit:
And same if you want it to adjust resources as well. So you have all of your resources as what was 300 Meg’s of, I don’t know 500 Or Meg’s of memory. So go to a gig per then you would change it and deploy the same way.

Sergey Pronin:
Yeah, yeah. So it is yeah, I will not apply it but if I go here, and I change like CPU limits or memory to one gig, I can just apply it and it’s going to affect all my replica set nodes at the same time. And operator does it all gracefully right so we are doing it’s in a smart way so we ensure that our primary node is not going down before your replica starts to update it so ensure that they all laugh so the reads and write flowing in your application does not experience any downtime when you perform such simple changes. Unfortunately, pods in Kubernetes are readable and we cannot change limits on the fly list yet. Maybe later it’s going to change. So you need to restart the pod to apply these changes and we do it nicely Simple. Okay, so Oh, it’s done already, as you see, we have three nodes 01 and two. And now we have three and four. So now instead of five nodes, or three nodes, I have five nodes in my replica set. This is how scaling is done. And again, the same would be with resources. So let me scale it back to three-node is an operator’s going to kill busy nodes in delay. Okay. Let’s watch. Yeah, so its terminating is going to kill this node. And let’s see what the SMTV tells us. It tells us that it is initializing because it’s killing this knows. But it doesn’t mean that your cluster is not accessible, it is still accessible. As you see this is my connection to it, I can still look into the shards or I can do anything on this database, it’s still

Matt Yonkovit:
If you change the number of shards, is it going to auto-rebalance? Or is that going to be something you’re going to have to do on your own?

Sergey Pronin:
We are not doing any data management with the operator. We are doing only information. So I am going to show it at the last piece because it probably going to take some time, I will add the shard. But the idea here is that okay, you can add the shard, you can add the replica set. But we don’t know what’s inside the data. So we don’t know, what kind of shard key do you want to use. And the problem becomes bigger when you want to delete the chart. Because deletion of the shard also requires us to understand what do you want to do with this data? How do you want to migrate it to other shards? Right? Yep. And we don’t know that. And this adds lots of complexity for us. So our main focus is to ensure that we can operate the infrastructure that we can provide it to you. And then if you want to play with the data, well do it as usual, right? Like create your shard key and balance the data.

Matt Yonkovit:
But if you are going to start to change the number of shards, delete add, that’s going to be still your own either scripts or your own processes outside the operator.

Sergey Pronin:
All right, yeah. Okay, let’s see. Okay. As is. Let’s see. All right word, let it Yeah, they were. Okay, good. So I’m going now to probably show you the backups. Backups are easy. So this is actually the YAML file the manifest to create a backup. What it actually does, it creates this object called Percona Server MongoDB backup. Well, it’s pretty long, but still, and it’s going to create a backup called backup one. It’s going to take the backup of this cluster webinar demo. My cluster is called webinar and demo. And I’m going to use Storage called webinar, this storage is defined as defined here in my backup section in my big CRM. So you see storage is I have this webinar, I can call it whatever I want. And here is the configuration. It’s the name of my bucket. My credentials, which I created the keys for and the endpoint URL, it’s the Google endpoints to connect to GCP storage, right. And another important piece here is the Finalizer. This one, what it means is when you delete this backup object in Kubernetes, you’re also going to delete this data on your in your market on GCP. You can comment this out if you don’t want this to happen. So if you delete this object, that data will still be there. But it is always usually done, I think. Okay, let’s create the backup. backup backup Yaml. So it’s going to create this PSMDB backup. Let’s see. Yeah. So you see this object was created and it’s going to get through some stages now and show us some status a bit later. Let’s wait a bit for it. But also I can show you the pocket. I think I have it here. Let me see. Refresh. Yeah, you see already some data appeared here. Underlying in our operator, what we’re using is PBM, which is Percona Backup for MongoDB is Percona tool, to backup MongoDB, it’s open-source, and you can backup sharded, cluster clusters and single replica set or whatever. And it just uploads your data to s3. It has a nice UI CLI tool, which you can use to monitor it. It also supports point-in-time recovery. Maybe I’ll show it to you as well in this demo. But anyhow, so the backup is going. Oh, it says error. Oh, real-time. Yeah. I don’t want to debug it. Yeah, no, I don’t want to debug it now is I I think I know what happened. I have a mismatch in the key. So but anyway, yeah, I think I used an old key. Ah, this is where security comes in. Right.

But anyway. Fail to run. must be true. First with oh, it’s still initializing. Come on. Looks like some port went down. Yeah, I guess one of my nodes went down. Oh, come on.

Yeah, RS zero went down. It happens on preemptable nodes. Hate this. So I ran on GKE on preemptable nodes. And one of the nodes went down at the moment when it was running back up. Again, happen. Yeah, I’m not sure if the node is off now, but Okay, let’s see. Let me find it. I’ll try to. We have time, right. You have as much time as you want, Sergey. Good. Okay. Let me try to keep you

Matt Yonkovit:
From doing what you think is necessary. That I’m a mean person.

Sergey Pronin:
Now, no, totally. I never thought. Yeah. Yeah, I’ll debug it. I’ll find it. And I’ll figure out what’s wrong with it. MongoDB, okay. It’s syncing. It’s syncing. It’s connecting. Okay, so it’s initializing. Okay, it’s up and running. Let’s see how PSMDB service analyzing , but it should go right as soon connection except there’s my cluster. It’s still live

Matt Yonkovit:
Oh, that’s weird that it’s still there. Right. Only initialized

Sergey Pronin:
Yeah, it’s taking some time to sync the data that you have the note went to another note was rescheduled, so it can take some time, unfortunately. Usually, it’s pretty fast. It can get stuck this was also terminated, but its fine was recreated.

Matt Yonkovit:
So just like always, every time I do a live stream, or we were talking to somebody on a podcast, the thing was wrong. The lawn people show up. So hopefully you can’t hear but like right now at my window. All I hear in the background is like mowing and stuff. So yeah.

Sergey Pronin:
Noise insulation and something.

Matt Yonkovit:
Well, it should I don’t think anybody will hear it. But it’s still like, making me conscious about the, that the sound of course because that’s how I am. I’m all about the extra value, I guess.

Sergey Pronin:
Okay, okay. I, I don’t want to spend lots of time on debugging it in a one-on-one node down and when I began to hate it, that’s why you should never use preemptable nodes in a demo because it can break things. But it’s fine. And I wanted to show you backups. I showed it to you. And interestingly, that piece of backup went here and not completely. So it was I guess in the middle The backup will last one node printed degraded state. So it’s an interesting use case, which I will need to battle later on. Okay, what else?

Matt Yonkovit:
Okay, yeah, something that, that you can enhance, right?

Sergey Pronin:
Yeah, I know how to get out of it, but it will take some time for me. So I don’t want to do it right now. But anyway, anyway, the operation of scaling, I showed you how it works, you can add more nodes or scale resources like CPU limits and memory. To take the backup, you just apply this PSM DB backup object , and it’s going to upload your backups to s3 or obviously, as you Matt mentioned, it can be scheduled, right. And the same time the restore itself is similar, right. So you just create Percona Server for MongoDB restore object with the configuration. And if you have a cluster or backup object in it, I’m in Kubernetes cluster and the backup object is there, you can just specify these things to just specify, okay, I want to restore from my backup, which is called backup one. And I want to apply it to this cluster with this cluster name. But if you don’t like your Kubernetes, the cluster is completely down, everything’s down and you start from scratch. You can say I want to recover from s3 bucket. So you just specify an s3 bucket the direction the key is again, and what’s going to happen, the backup is going to be fetched from s3 directly to your cluster, which you have Kubernetes. Right? So it’s the same approach.

Matt Yonkovit:
So could I use this, then, to initialize your clusters? So make this part of the rolling out? So if you had seat data, for instance, and you wanted as you spin up new clusters to add that seed data and from a backup, you’ve taken, or that you have out there, you could use this for that process as well?

Sergey Pronin:
Yeah, and it’s quite a common use case, specifically in dev and test environments alike. Okay, I have a production cluster. Now I want to play with this data that I have in prod on my test environment. So how do I do that I just spin up the new cluster and restore my production data to this test cluster? So they can I can play with it safely. It’s a common use case for sure. Okay, and another thing that I probably want to show you are, I’m not going to play it now because it’s going to take time. But here you see my replica set zero. And in a nutshell, it is a single shard. And the beauty here is that I can copy this and paste it somewhere below here. And call it an RS one, for example. And what’s going to happen if I apply this it’s going to create me another shard. This is the question of how to create and delete the shard is just the matter of altering this manifest YAML manifest, adding more shards into it, and applying and that’s it. But yeah, definitely, you should always remember that if you want to delete the shard, you need to balance your data. First, we have a check-in place, which is going to check if you are deleting the shard if there is any information on it, it will not allow you to delete the shard until you move all your data out. So it’s a kind of failsafe mechanism that we use.

Matt Yonkovit:
Okay, so you have to go through the manual process of clearing out the shard before you deleted, that it’s clear. Okay. Is there. Is there a way to override it?

Sergey Pronin:
Oh, I don’t think so. Okay, I get you

Matt Yonkovit:
A use case to override it. But yeah. I mean, make me Yeah, yeah, maybe somebody. I don’t have a use case. I was gonna say maybe somebody left the date and just wants it dropped. But they copied it over somewhere else or something. But that wouldn’t make sense. So I will just be clear,

Sergey Pronin:
No, no, I understand. Yeah. Okay. So I think this is all I wanted to show you today. Click this slideshow. Demo. Yeah, I guess. Thank you, everyone. All right. Listening.

Matt Yonkovit:
Yes. And for those who are not watching live, if you do have questions, feel free to reach out to us. We can follow up with a blog post or some forum activities later on. For those who are interested, Percona Live is coming up. So right now the call for papers is open. If you are someone who is interested in MongoDB you’ve been doing some different MongoDB things. We would love to have you talk. There’s a blog post right now in the Percona blog that gives you some ideas on topics we’re looking for. But we would love to have you come on along and talk. But until next time, we appreciate you and go out there and happy database everyone.

Sergey Pronin:
Exactly.

Matt Yonkovit:
Yes. All right. ∎