Open Source Database Testing, Tooling, Debugging Code, Using Microservices - Percona Podcast 35

Link to listen and subscribe: PodBean

Join Percona’s HOSS (Head of Open Source Strategy) Matt Yonkovit as he sits down with Rob Richardson, Developer Advocate at Cyral to talk about testing, tooling, debugging code, using microservices, and more. Rob is a Microsoft MVP, published author, frequent speaker at tech conferences, user groups, and community events. Recently he shared his knowledge with our community during a session about Databases in the Microservices World during the Percona Live 2021. Find more presentations on his blog https://robrich.org/presentations or follow him on twitter at @rob_rich to continue the conversation.

YouTube

Link: https://youtu.be/y_TWG4cXTKo

Rob Richardson

Developer Advocate at Cyral

Rob Richardson is a software craftsman building web properties in ASP.NET and Node, React and Vue. He’s a Microsoft MVP, published author, frequent speaker at conferences, user groups, and community events, and a diligent teacher and student of high quality software development. You can find recent talks and musings on https://robrich.org/presentations and follow him on twitter at @rob_rich.

• Internationally recognized technical speaker, speaking since 2008
• Microsoft MVP since 2014
• AZGiveCamp founders Award, 2015 and 2016
• Red Gate Database Delivery Panel 2013-2014
• Friend of Red Gate since 2009
• Microsoft Partner since 2005
• Leader, Southeast Valley .NET User Group since 2013
• Co-Author, SQL Source Control Basics
• Core contributor to Gulp v2 and v3
• Part of the AZGiveCamp core organizers
• .NET Rocks mug recipient
• Professional HTML and ASP.NET Developer since 2002
• Node and Angular developer since 2013
• Docker and Kubernetes developer since 2016
• Husband and Father of two since 1998

See all talks by Rob Richardson »

Matt Yonkovit

The HOSS, Percona

Matt is currently working as the Head of Open Source Strategy (HOSS) for Percona, a leader in open source database software and services. He has over 15 years of experience in the open source industry including over 10 years of executive-level experience leading open source teams. Matt’s experience merges the technical and business aspects of the open source database experience with both a passion for hands on development and management and the leadership of building strong teams. During his time he has created or managed business units responsible for service delivery ( consulting, support, and managed services ), customer success, product management, marketing, and operations. He currently leads efforts around Percona’s OSPO, community, and developer relations efforts. He hosts the HOSS talks FOSS podcast, writes regularly, and shares his MySQL and PostgreSQL knowledge as often as possible.

See all talks by Matt Yonkovit »

Transcript

Matt Yonkovit: Everybody, welcome to another HOSS Talks FOSS. I’m here with Rob Richardson, one of our speakers at Percona.Live. Hi, Rob. How are you doing today?

Rob Richardson: I’m doing well. I’m great to be on the show. It’s fun to be able to talk about all the things. The conference was really fun. And so it’s cool to be able to revisit it now.

Matt Yonkovit: Yeah, and I don’t know if you have attended Percona Live in the past? Or is this your first one?

Rob Richardson: This was my first Percona Live.

Matt Yonkovit: Okay. And you know, how did it go? Because typically, it’s in-person, but obviously, with COVID we’ve had to go online, the last couple ones.

Rob Richardson: It worked. Online and in-person conferences are definitely different. And each has their strengths and weaknesses. But I think we’re kind of getting the hang of doing online conferences. So I think it’s working out pretty well so far.

Matt Yonkovit: Yes, I think it is. Now, Rob, maybe for the listeners here. Give us a little bit about your background. Tell us a little bit about where you came from and what you do on a regular basis.

Rob Richardson: Yeah, definitely. I do a lot of programming, building websites, web properties for businesses small and large. And it’s fun to be able to build that kind of content and really dig in, and build solutions for users that are really interesting for them. I also like doing a whole lot of teaching. And so I’m a developer, evangelist, developer advocate for Cyral. And that gives me the chance to speak at a bunch of conferences, and to be able to teach really interesting things. So not only am I scaling my knowledge, just to myself, to one, but maybe I can scale it to two, or 20, or 200. And that’s really fun to be able to share and learn together and kind of ramp up our skills together. Raise the ocean for everybody as well.

Matt Yonkovit: Yeah. And what kind of things are you teaching people now? Like is there something interesting, the topic that you’re really hot on at the moment?

Rob Richardson: It’s really fun. When I got started, I loved drawing the short straw. Hey, we need to go figure out how to do this. And so as a web developer, my first task was, well, how do we deploy it? And so I intentionally drew the short straw, and I’m like, let me go figure out how to do IoS, how to configure Internet Information Services. Over time that led to Windows Server Administration, and in time Linux administration, I drew the short straw again on purpose, as we’re looking at automation, how can we get DevOps going? So I first started with CruiseControl.net. I’ve since done Jenkins and TeamCity, and Visual Studio Team Services, and now as your DevOps and now GitHub actions have similarly I drew the short straw when we came to unit testing, and I’m like, well, let me dig in. And so I have a really cool talk that talks about Cyral, and how Cyral unit testing fits into the scheme of all the things. But I also do server side testing, and I do unit testing, I have another talk. That’s all live demos of getting started with testing in JavaScript, synchronous, asynchronous promise, async, and await testing. It’s fun. And so what’s really fun is that because I’ve intentionally tried to stretch myself, I end up with skills in all kinds of places. So I teach .NET and node, React and view, dependency injection and version control, DevOps Server Administration, I’m having a lot of fun with Kubernetes and Docker, I did a full day Kubernetes workshop not too long ago. And it was fun to guide eager minds through this. One person, actually, in that workshop said, I’ve been using other people’s containers for a good long time. I wish I had this a year ago, so that I could have been helping and contributing to their Docker files. It was really fun.

Matt Yonkovit: Oh, that’s awesome. That’s awesome. I think it’s great when you can reach out and reach people who are eager to learn and teach them something new. That’s a passion of mine as well, I’d love to talk with people and to hear what stories they have, and to help them kind of move into the next story round. Now, for you, you mentioned testing and testing something that I’m kind of a hot topic for me lately, because I’ve seen, especially in the database space. And you know, a lot of people from a development perspective, aren’t testing their applications thoroughly enough, they’re specially kind of forgoing some of the more integration testing, or even some of the scalability testing or load testing. And that tends to lead to problems. In fact, I talked with several users in the database base who have to deal with the operation side, and they go, it’s those developers, they never test their code. That’s why it’s always broken. And I’m curious, having someone like yourself who teaches some of those frameworks and talks a lot about testing. What do you see in that space? Is there a lack of knowledge? Is there a lack of desire? Is it that developers just don’t like to test? I mean, what what, you know what, what leads to those

Rob Richardson: Yeah. And what’s interesting is that we see that kind of lack of interest in the industry where they’re like, well, I need to get this task done. Management says, Hey, we’re not shipping tests, we’re shipping production code. So why are you wasting your time on tests? Developers are like, well, I just need to get it done. Let me just do it. And then we end up with content that is well less tested. Now, in some cases, that’s totally fine. The purpose of testing is about risk mitigation. And so if the risk is low, then maybe not having tests was sufficient. And dealing with production outages when there’s a crisis is completely sufficient. And that was worth the investment. On the other hand, we may want to reduce that risk, we may want to have some smoke tests and integration tests, we may have a particularly hairy function. And we want to understand that this function works correctly in all of the scenarios that we’re working with it. Taking a step back, we may have a piece of functionality of microservice or a unit of content. And we wait, we may want to validate that this works correctly, we can look at it from a UI perspective and say, I want to validate that this website, or even this component works the way I expect, or from the other side, the API, we may want to say hey, I want to make sure that this API given these inputs, creates this output. We can even do unit tests around databases to say, if I have this data in the table, that this store procedure will validate, well run in an expected way, and that we validate the results in this way. So if it makes sense to reduce that risk, then unit tests can be a really elegant mechanism to be able to validate that. Now, a lot of people will say, hey, I’ve got these tests, and it’s really slowing me down. I would turn that on its head and say, a lot of the time that you’re spending is debugging. You’re ramping up your website, you’re pushing all of the buttons to get back to this page, and then you’re clicking the Next button that you want to validate. And then you go change some code, you refresh it, you go click all of those buttons again, wouldn’t it be cool if you could have a mechanism for debugging? By writing code, we really enjoy writing code. And so where the light bulb came on, for me was debugging was my way of validating or writing tests rather, is my way of debugging by writing code, I can write code that is much more simple in all of the scenarios that I want to validate. Once I have those tests in place, then I can just push the Go button. And for those that go green, I’m done debugging. For those that don’t go green. Now I can take a look and see what my function is doing. Maybe my test isn’t correct, maybe my function doesn’t validate. But then have you ever gotten to the end of that scenario where you finally got this use case to work, and you broke a different one? I just run all the tests. And once this one goes green, and all the rest of them go green, I know that my task is done, I’ve mitigated that risk. So in time, quote from Jeremy bytes, I am actually faster by writing unit tests. Because not only do I have that debugging by writing code, experience, but I also have confidence that all of the other scenarios that I tested yesterday or last week, still work the way I expect. It’s weird, it’s kind of counterintuitive to think that writing more code actually makes me faster. But the third time that I deploy, and I caught an exception in the middle of writing unit tests, that wouldn’t have escaped into production. It’s like, yeah, we mitigated that risk, testing was really valuable in this case.

Matt Yonkovit: Now, that’s really test driven development, right. So you start with building those unit tests, you start with building out what you want done in those tests, and then you code it to make it green, like you said. And that’s a philosophy that not everyone follows. You know, certainly it takes some getting used to start, and think tests first, or test second right up front. But I do think it does produce better code in the end, because you have everything kind of thought out logically. So you almost go a little bit slower up front to go faster later on.

Rob Richardson:
Right, it is about risk mitigation. And you don’t get this for free. But when you get good at it, it is pretty fast. And whether you’re writing the test first and then implementing the code, or you’re writing the code, and then implementing the test right away. That methodology allows you to think more like the people who will consume your content. And so now you can build it in a way that makes sense for them. I could grant it a test that isn’t necessarily a user. And so the closer you get to this, the more you may say, well, I want to really design for the user, not for the test, but it’s wonderful to get outside yourself. Get out your code and see that people will consume it. And to really facilitate them by thinking of them first. You’ll also notice that as you build code that is easier to test, that your code will be more loosely coupled, you’ll use more interfaces, you’ll use more abstractions, you’ll create this looser coupling between your components, which ultimately leads to a better design too.

Matt Yonkovit: Yeah, and I think one of the interesting things that we’ve seen and you know, your talk at percona live this year was about databases in the microservices world, as microservices have really taken off. And as we have developed more modern applications using more of a microservice architecture, the systems have gotten much more complex, which means that integration testing between the different pieces becomes infinitely more important. I mean, how do you test something at scale when you potentially have dozens of different microservices, dozens of different databases, all these different components that are sprawled all over the place?

Rob Richardson:
Exactly. What’s interesting about microservices is that we can no longer do unit or even integration tests, just solely around our microservice, to prove our system works. We need that integration test between components to really make it work. What I like to do is build out some end to end tests. And by end to end I mean fire up the browser, click some buttons, get it to call all the API’s get it to run through each of the microservices. Yeah, it’s gonna hit the database, and come back and validate that that works. And if we can build these end to end tests, for the hot pass the really critical paths, maybe we’re doing a login, add to a browse the shopping cart, add one product to the cart, get to the checkout screen, we’re not entering a credit card number, we’re not actually completing the purchase. But if I can get all the way from the homepage, to the checkout page successfully, then I know that my system will work and all of those crucial components will work. Now let’s rig that test to run in production once an hour. Now, as we’re deploying these updated microservices, we can’t necessarily test the entire system every time we deploy each test. But we can catch the integration pieces with this production service production test, before our users start to get outages in their systems.

Matt Yonkovit: Now, Rob, what tools do you typically use to capture that workload and to you know, set up those tests, maybe give us a little bit of a walkthrough on that stack. So our listeners can understand, what sort of tools you recommend, which aren’t necessarily all the tools, but the ones you find most useful?

Rob Richardson:
It really depends on what you’re trying to test. And each test will have different requirements. For example, if we’re trying to validate that this tax calculation works correctly, then we’re probably writing unit tests and mocking out the dependencies, we’ll need a mock library. If we’re in JavaScript, we’ll probably use oh, I just remember the C# one. Let’s back up into the C# case first. So if we’ve got a C# function, we’ll probably use Moq to be able to mock out those dependencies, we’ll probably use xUnit to be able to validate this function. If we’re in JavaScript land, we’ll probably use something like Mocha or Jest to be able to validate that function. In time when we’re starting to up level that maybe we’re looking at a component. And we want to validate that this component behaves as expected. I click this button and this action happens or it renders in this particular way. Now probably use test utils, together with my chosen framework. And so test utils is available for Angular, React and View. And it gives me the ability to mount my component within a unit test. And then to run either mocha with Chai, or Jest around that component to validate it works correctly. Now, the cool part about test utils is I can choose either to mock out all of the child components, or actually render them depending on my needs. So that’s how I would do a component test. levelling up again, if I want to validate my API, I can just fire requests at it, I don’t need a UI at all. So here, I might use super test, or I could even build tests in postman. And that allows me to just fire requests at the API and validate that it works. When I’m ready to do end-to-end tests all the way through all of the pieces. There’s a bunch of tools. Selenium is the one that we usually reach for, and then dismiss. I’ve chosen Cypress because I think Cypress is that great balance of deep interaction into the browser. So the tests are no longer brittle or slow, together with a cool API that allows me to reach into all of the things, one of the things that Cypress does really well, is waiting for the request to finish. In Selenium, I have to set the timeout was two seconds long enough, well, that test failed, let’s make it three seconds. Well, now it’s always gonna wait three seconds. So even if it finished in 30 milliseconds, it’s still gonna wait three seconds. By comparison, Cypress allows me to loop into the browser events. So when DOM content loaded, now I can do the next step of my test, or when this API call finished, now I can do the next step. Now I can definitely set the timeout and say, if you go longer than five seconds, call it a fail. But I can also proceed as fast as it’s ready to go. So a Cypress test that is set up to run all of the pieces can run really fast and efficiently, just in the midst of running in the browser or running as part of the DevOps pipeline.

Matt Yonkovit: Three, cool. Now, before we move off of testing, I’m curious, are there common issues? Or what are the most common problems that people make when they start building their tests out or trying to test their code?

Rob Richardson:
One of the first things that you’ll hit is how do I get started? How do I run one test? How do I get my tests into my DevOps pipeline? And the cool part is a lot of these tools now are really easy to get started with, I can just NPM instal the thing, and it’ll give me a command line, maybe I added to my package JSON, or I can maybe just call this straight from my DevOps pipeline, it’s just another command line to put in my pipeline. Once I’ve got one test in place. Now, that’s great. Maybe the test only just loads the homepage and validates that I get an HTTP 200 status. The cool part then is, the next thing that we’ll hit is, well, how do we write the tests. And that’s where you start to notice, probably you have too tight coupling between systems, it’s difficult to mock out all these dependencies. That’s really cool. You just learned something about your code. Now refactoring all of that code to have looser coupling, to have less dependencies to make it easier to test, that’s kind of a long process. But for the new functions that you write, you can build them in a testable way. And you can just increase your unit tests, to get from zero to one is pretty hard to get from one to two is, is much easier to get from two to 200. It’s just a matter of putting in the work to do that.

Matt Yonkovit: Okay. So, back to the microservices site the talk at Percona Live was specifically talking about databases in this space now. And as you had mentioned, a lot of different databases are used now across the organisation, an application could have a dozen different microservices, and it could have a dozen different databases behind it. The technology isn’t really tied to one single stack anymore. It’s not like everything must be a LAMP stack, like in the old days, or everything must be MySQL or Postgres. It’s - use the best tool for the job now. And that causes a lot of sprawl. So what are you seeing in that space?

Rob Richardson:
Yeah, does and what’s interesting is now we can optimise for the problem at hand in the past when we said, yeah, but I’d like to store a little bit of data here in Postgres, or this problem set best matches a document database, can I spin up Mongo? Then they’d say, well, you can use any database you want, as long as it’s here in SQL Server. Yeah, and I think, yeah, and then we will do all of these tricks to try and get our database to trace, maybe have a SQL Server, maybe it was Oracle, maybe it even was Postgres. But we would do all these tricks to try and get it to store the data that really made no sense. One of the principles of microservices is that each microservice owns its own data. And so one of the big things that I pull out in this talk is that, because we have now lots of little data stores, they don’t need to be the same vendor, we can go shopping for the vendor that best meets the needs of this problem. And the beautiful thing is now we can have a little Postgres database and a little SQL Server database and a little Mongo database. And, yeah, there is some administration involved in making all of those data stores work. But we can choose the correct one for the problem. Maybe the answer to this problem is an elk stack or an s3 bucket, rather than a whole nother relational normalised schema.

Matt Yonkovit: Yeah, and I think that that provides developers with an immense amount of flexibility to develop, you know their application quicker, to be able to get at what they actually need and ensure that they can utilise some of the built in inherent efficiencies of each of the different databases. But it does cause, especially at scale, a bit more administrative overhead for those who are handling the backend. So as a former DBA, I know, I call this the great inheritance problem like, Hey, I’m an Oracle DBA, let’s say, and now all of a sudden, I’ve got Oracle, Mongo, Postgres and Elastic that I now have to maintain. So it does push the skill set, especially of the folks in the backend, who are administering that a bit more. And I think this is where the benefit of some of the databases of service technology, some of the cloud providers provide most of these databases as a service now, as allowing a lot of companies to move faster, because they can choose what best fits them. And they can do it without having to worry about the overhead of the backend database administration, if you will. So it’s definitely an interesting space now.

Rob Richardson:
I completely agree. And that’s where cloud vendors are really ideal in this space, use Database as a Service, outsource that piece. So that you have one throat to choke, and it’s not yours, they’ll take care of point in time recovery, they’ll take care of the nearly 100% uptime, they’ll take care of securing physical access and proximity, ensuring that your database is sharded correctly, you can outsource all of those pieces to your database vendor. And if your cloud provider provides that one click provision your database. Even better now, you don’t need to worry about managing all the things and you can get back to doing the tasks that you need to do, the more automation you can put in place, the more you can make this problem easy. I’ve been on that team where we started sprawling into different types of databases. And then it’s like, oh, well, we had a problem in this database. Let’s just go to last night’s backup. We didn’t back it up. Right? Right. And it turns out, we didn’t. So as you start moving into lots of these databases, you do need that automation to make sure that your databases are backed up, and that you have the authentication piece in place. If you can get automation to subscribe your databases to backup and authentication, now you can start to move at scale and get into lots of different database vendors and lots of different database micro databases, to be able to produce your microservices and really elegant mechanisms to keep that microservice owns its own database methodology in place.

Matt Yonkovit: Now, as we’ve seen, like these applications, kind of evolve into these micro databases, microservices that this, this new architecture, it’s kind of made a new problem, or maybe, maybe maybe made an old problem a little bit more difficult, which is troubleshooting and optimising and trying to find kind of that needle in the haystack. You know, because previously when I worked on a single, monolithic system I would always get the call, the database is slow, because let’s be honest, everyone played the database first. And then until you can prove it with something else. But my caching tier is broken, but it’s the database. That’s right. Yeah, that’s Yes. So how do you find which of those applications or which of those microservices are causing the issue in which components are there, as you have this really diverse architecture? Now, it could be one of a dozen different components that has maybe different database technologies underneath, and maybe even like you mentioned has slightly different caching, as well. So, what sort of methods or processes can you find that problem that is occurring?

Rob Richardson:
That’s a good question. And trying to pick the correct cause is quite a dilemma, the first thing that I’ll do is I’ll open up the Chrome Developer Tools, or maybe I’ll use a tool like Fiddler or Postman to kind of proxy the requests between browser and server. And let me just watch the request going back and forth, that can usually identify whether the problem is on the server side or on the client side. And not to say that blaming is the correct thing. But we can now start to focus on the correct thing. Maybe the browser is caching the old results, and that’s leading to the problem. Maybe it’s not sending the authentication header, and that’s the problem. Or maybe the request is just taking a really long time. Now that we’ve kind of identified whether it is a client or server concern, we can start to dig in. If it is a server concern, we now have the URL. And the beauty here is that we can fire that URL at the service at the skillion times until we find the answer. Probably given that URL, you can bring that back to a development environment and fire it off too. Now you may need to adjust for a different authentication token. A different URL base URL, but you’ve kind of got a feel of where you’re going. That will probably also tell you which microservice received the request too. At that point, now we can start to traverse the microservices. Here’s where a tracing or logging platform can come in really handy. Open telemetry is a great piece that allows us to kind of visualise as requests move between microservices. Service meshes also do a really great job here, where it can kind of track the request as it moves through each of the services. Now, the cool thing about using either open telemetry or service mesh is that you kind of get a holistic view of the entire system. As you plug that into each system. With the service mesh, you just kind of plug it into Kubernetes. And it takes care of making all of the pieces in each microservice. With open telemetry, you do need to hand it to each microservice, you need to configure that service to expose the metrics. But once you’ve got that trace, now you can watch the request, move through each of the microservices, and maybe even see the SQL query that was created at the far end. Now you can time each piece and identify Oh, so here’s the hot path. Well, that hot path, the one that gets executed a whole lot of times or takes the most time, now we can start to dig in. Is it a stored procedure querying against something that needs another index? Is it that our WHERE clause is just humongous, and ugly? At that point, now we can start to zoom in on that piece? Do we add caching? Do we add bigger hardware? Do we rewrite our query? You know, very specifically, we’ve used a whole lot of tools to be able to scientifically experiment with the system. Let’s first identify client versus server, then let’s traverse that path and find where the majority of the time is taken. And then let’s dig into solve the problem that is actually broken. When we just started and say your database is slow. Okay, you want me to push the turbo button again?

Matt Yonkovit: Oh, the turbo button, I missed the turbo button. Back in the good old days.

Rob Richardson:
And for every few weeks, we just take out another thread dot sleep and the system goes faster. Right.

Matt Yonkovit: Well, Rob, let me ask you this final question here. You know, and I always like to ask something a little eclectic. There might be a little out there. I’m curious. If you know, as someone who goes out and tries to educate folks, is there something that if people could take away from this chat or take away from one of your presentations, just one thing, one thing they could learn and maybe change their habits, change what they’re doing try something new, what would that one thing be? Like, please learn this thing? Please do this thing differently, please. You know, try this new technology. What do you think?

Rob Richardson:
What is the one thing that I would teach you? We’re here on stage. And we’ve been doing it for a while. But there’s nothing magic about either you or I, or even the listener, there’s nothing magic about you. We have just been added a little longer. But the skills that we have are completely attainable. You the listener, the watcher, you can get to where we are, and you can exceed us. The beauty of this path is it’s just about learning and experimenting and doing. The next step is yours. You can go do wonderful things. You can go experiment with this technology, you can go find that passionate project that you want to do. You can have a lot of fun with technology. There is nothing that is unattainable about anything that we’ve talked about today, or anything that you saw at the conference. You can take this step and you can be awesome. Go beyond.

Matt Yonkovit: Thank you, Rob. Thank you for that. And you know, you can check out Rob’s presentations and you know, his blogs at https://robrich.org. Awesome. And Rob, thank you.

Rob Richardson:
Follow me on Twitter at @rob_rich. And let’s continue the conversation. I’d love to see what I got wrong and how I can learn.

Matt Yonkovit: All right. And thank you very much Rob for hanging out with us for about a half an hour today and also for presenting at Percona Live. We look forward to hearing from you in the future.

Rob Richardson:
Most definitely. Thanks for having me at the conference. It was fun learning and teaching with you.

Matt Yonkovit: Wow, what a great episode that was! We really appreciate you coming and checking it out. We hope that you love open source as much as we do. If you liked this video, go ahead and subscribe to us on the YouTube channel. Follow us on Facebook, Twitter, Instagram and LinkedIn. And of course tune in to next week’s episode. We really appreciate you coming and talking open source with us. ∎