KVM Hypervisor Experience, Developing a New OS, and All Things ScyllaDB - Percona Podcast 18

Link to listen and subscribe: PodBean

The HOSS talks with Dor about his career covering working on the KVM hypervisor, developing a new OS, and also building an Open Source Database. We then go Deep diving into all things ScyllaDB including design decisions, the benefits, and what users are looking for.

YouTube

Link: https://youtu.be/2KLDTiFQP30

Dor Laor

CEO, ScyllaDB

Dor Laor is the CEO of ScyllaDB. Previously, Dor was part of the founding team of the KVM hypervisor under Qumranet that was acquired by Red Hat. At Red Hat Dor was managing the KVM and Xen development for several years. Dor holds an MSc from the Technion and a PhD in snowboarding.

See all talks by Dor Laor »

Matt Yonkovit

The HOSS, Percona

Matt is currently working as the Head of Open Source Strategy (HOSS) for Percona, a leader in open source database software and services. He has over 15 years of experience in the open source industry including over 10 years of executive-level experience leading open source teams. Matt’s experience merges the technical and business aspects of the open source database experience with both a passion for hands on development and management and the leadership of building strong teams. During his time he has created or managed business units responsible for service delivery ( consulting, support, and managed services ), customer success, product management, marketing, and operations. He currently leads efforts around Percona’s OSPO, community, and developer relations efforts. He hosts the HOSS talks FOSS podcast, writes regularly, and shares his MySQL and PostgreSQL knowledge as often as possible.

See all talks by Matt Yonkovit »

Transcript

Matt Yonkovit: So I’m here with Dor Laor, the CEO of ScyllaDB. And we’re here to talk about all the fun stuff that they’re doing over there. And also talk a little bit about his journey and why he started the company. So, Dor, how are you today?

Dor Laor: I’m great. Thanks for having me here. And looking forward for our chat.

Matt Yonkovit: Okay, great. And so I know that your background is very development centric. So you did you start as a developer, I see you were an engineering manager. And you were deep into the Linux kernel for a while. So how did you get your start?

Dor Laor: And so I’m an engineer, I went to kind of Israeli Technion Institute, which is like MIT. And actually, it’s nice because I went there in order to it’s an institute in Israel, but I went there in order to network with people. And it really worked out well, because I met a bunch of people that I work with in my first startup. And it didn’t go that well, it was a terabit router company. Okay, what’s really went well, is that I had friends, it goes back 20 years back, so have lots of friends from that endeavour, and currently happened to be that I have three people who went with me to university for my first degree and master’s degree there who are at Scylla: the VP of engineering, the VP of product, and another really good developer, all we learn together, so.

Matt Yonkovit: And that’s a really good way, right? So once you start to know and trust someone, we see that a lot where you want to work with people who you can trust who you know, and who have good knowledge and can work together. So once you build that rapport, it’s a very good thing to be able to bring those people along with you. And going from that startup, you ended up at Red Hat, and you actually worked in the kernel virtual machine. And worked as a platform virtualization director. I mean, how did you kind of jump into that open source space with Red Hat? I mean, that seems like an interesting move from an engineer having your own startup, you kind of jump into Red Hat for a little bit. What was that like?

Dor Laor: And so initially, it wasn’t my own startup, I will join as an employee and I caught it and the bunch of stuff. And I did try to have my own style back then in 2003 or 2004 but I didn’t have the credibility to get money. I had to wait until the KVM project was successful in order to be able to easily get funded. So KVM was helpful. But the way that we got into KVM was kind of more of eventual, I joined a company called Kumanet. And with its name, it was supposed to be a network type of device. And we pivoted, we had to pivot three times. It was my school four pivots. The second pivot was an idea that Avi, the current CTO brought, it was really super cool, nice idea about lockstep virtual machine execution for AJ with with recent hypervisor, and we hit a wall with that project, it worked but wasn’t performing enough. And we had to pivot again. And then this is where Avi came up with a really novel idea on how to re-implement a hypervisor from scratch using the CPU technology. And that was the KVM Hypervisor, we started to boom pretty quickly and was part of the Linux kernel. And two years after we scratch the idea, read out, acquire the company and we stand there for years.

Matt Yonkovit: Okay, and so that’s how you ended up at Red Hat. And so you were at Red Hat for four years. And then you had an itch to do something else. So going from working on a hypervisor working on the KVM, to databases, what, what inspired you to jump into the database space, it seems like, I mean, it’s kind of in the infrastructure space, but it is a jump.

Dor Laor: It’s a jump in I can elaborate how it happened. And basically, being able to accomplish something like the KVM gives you a lot of sometimes too much confidence and sometimes in order to jump into the database, to be honest, you need to be more full than an expert to because yeah, you may not know how big is of endeavour it is and so you’re feeling too confident and you jump on it. But just to be honest, that has been the case. And now, especially when it’s similar to we were new to hypervisors. At some point it came through a pivot, then we learn the domain. This is how we were new to databases. And originally, we founded this company not around databases. Because we weren’t experts in database, which is what we wanted, the company was founded with the promise of developing a new OS from scratch including our own kernel. So, okay, back in late 2012, we identified that everybody run just one workload inside their virtual machine. So like, you get a full fledged, and you just run a single app, it doesn’t make sense. So we developed a new OS called OSD, it’s a unicorn… unikernel I’m sorry, not the you’re definitely not the unicorn, not a unicorn, he didn’t succeed. And the target was to optimise OS for just one task, and to load it into the kernel space. So we developed that, it was an open source. So that’s how the company got it started. But at the time, a Docker started to emerge, Docker wasn’t there when we did it. And Docker was more of a mainline standard to package and run a virtualized applications in a different way. So Docker succeeded the lots. We haven’t, and we needed to pivot. And one of the workloads that we wanted to show improvement was the Cassandra NoSQL database. Because Cassandra was successful back then it was around the year of 2014. Cassandra was quite successful back then. So we wanted to show the value that we give to Cassandra in terms of speed. And while our iOS when it used to run up in application like Redis, we managed to boost Redis by 70%, because we had our own specialised kernel. And the, the application was living in in the kernel space, and we had our own network stack, our own TCP IP. So it was really beneficial for us super simple and trivial application like Redis. In the case of Cassandra, it was the opposite. We didn’t really move the needle. And something really looked wrong for us. first, it’s like, Java is a great language for management applications, but pretty wrong choice for infrastructure. It’s just a big layer between you and the hugger, that does exactly the opposite of what we’re trying to d, and also in the past. So that’s one thing. And also, at the time, Google published a Cassandra benchmark, and they used 330 virtual machine to get 1 million operations per second database operations. And we left Red Hat when the KVM broke the world record of 1.6 million iOps operation with a single VM. Now, it’s pretty apples to oranges, because the iOps is file system, and database iOps with replication. But still, it’s 123-130. And since we needed to pivot, we did additional market research and technology research. And we decided, this is a big opportunity. The API and all of the promise of Cassandra looks really solid, and it has great things in it like, it’s the best platform with high availability and disaster recovery. But it needs like in terms of scale, all of the nodes are homogenous, we really like the model. But we thought if we bring all of our experience not in database that will take the good database design from Cassandra, which is built on DynamoDB and Bid Tables, and we just implement it in the way that we used to implement things in KVM. Then we’ll bring lots of value. So that was late 2014. And that’s exactly what we do since.

Matt Yonkovit: Yeah. And what’s really interesting is I just had a one another podcast with Karthik Ranganathan, who was one of the key developers at Facebook of the original Apache Cassandra, right. So I asked him what if he could go back and do anything again, he said he wouldn’t have used Java. So obviously there are some limitations with Java in the Cassandra. But it’s really interesting that you brought a lot of the OS optimizations and techniques that you use to optimise the KVM and the Kernel, directly to the database. That’s a very unique pathway because many of the engineers who I’ve talked to, especially from companies that have designed and built databases, come from the database space. They’re not necessarily OS engineers. And so what are some of you you mentioned, you had to pivot a couple times. But was there? Was there anything that you found that you felt like, oh, if I only knew that, when we started, it would have been way easier, this would have been a way easier project for us to get involved in.

Dor Laor: And it’s not just one, it’s plenty. Yeah. And sometimes, it’s not just about us, I think it’s also learnings that other vendors and even entire communities undergo, like things that are not purely in the database space like MapReduce initially, you think, oh, what a great idea. But then it’s, it’s more challenging. So in our case, now, for example, we’re doing a huge change right now. So we mimic the Cassandra functionality. And we recently added the DynamoDB API, I think maybe the DynamoDB API triggered a lot of things to it. So a first many times, it’s good to be a late comer, because it’s a latecomer you, you can come late, and review everybody’s made mistakes and take the best and and try to improve, you’re not you don’t necessarily need to be the smartest, you just get the experience of others. And so we’re trying to do exactly that. And, for example, with the Change Data Capture. So Cassandra basically doesn’t have a changing data capture, they just kind of piggyback on the commit log, but it’s a horrible solution, it doesn’t take in mind replication. And it’s super hard to consume it. And on the other hand, DynamoDB has a pretty good Stream Processing Dynamo streams, which is a Change Data Capture, and they did it the right way. And you can, you can consume it in a fully replicated way. And you also get to decide when, whether you will want the key, the data, the previous data. So we took the dynamite idea both had on our implementation with improvements. So our chain data capture is a table. So we since we came last we got advantages because of it. But, for example, something else that we do now, which is a completely big revolution, is reimplementing basically everything using the raft consensus protocol. So traditionally, we mimic the Cassandra eventual consistency which, of course, is a trade off. And I always tell my engineers that when they go to implement the best solution, I tell them, go back to, if you take such a weird concept like eventual consistency, we wouldn’t have invented eventual consistency, because it looks somehow wrong. But it has a lot of benefits too it in terms of high availability and partition, tolerance, all that. So we get to benefit from it. But now with by implementing raft, we’ll be able to allow everything to be consistent without performance bounty. So that’s a huge change. For us. It’s also super complicated in the way we do sharding. And we kind of do it very late in the game after six years of development. And this thing goes across the board. It’s not just the consistency that we provide to the end user. But also basic things like topology agreement, and schema based transactions that are not in Cassandra or not in Ceylon. Now, after we have lots of experience. And now finally, we have also more time, after we managed to be on par with the API now we can take it to the next level. But if we would have started with this knowledge, like going back to 2014, we would have started with this to begin with.

Matt Yonkovit: And I think that that’s one of the interesting things that you see. There’s a rush often to develop the new cool thing, the new feature, and there’s definitely benefits when you are the first to have something. But there’s a lot of drawbacks as well. Because a lot of times you’re blazing the trail and that can open yourself up to bugs or reputation problems, there’s a lot more risk. And it seems like being a little bit behind and focusing on the best solution enables you to ensure that the features that do come out are more polished and ready to go. I mean, is that what you’re seeing as well, like, you’re getting things that oh, this is how this other place did it? You know, they use draft. So let’s do that. Because they have proven that it works. They’ve solved that problem is kind of how you work from an engineering perspective, you look at those,

Dor Laor: It was less kind of a, it was less a decision that we’ve made. Normally, it was more for lack of choice, because we came not just a little bit behind, but we came after everyone had their own NoSQL solutions and they were relatively mature. So we have to just rush and go and implement everything. Initially, it’s very helpful because you don’t necessarily need to make a lot of decisions, you need to make a decision in terms of the underlying implementation, but not the API. So it’s relatively easy, and it allows you to move fast, and also to do things better. But now, it’s now that we’re kind of accomplished the API of Cassandra and API of DynamoDB. Now, this is where it gets complicated. And we create new API’s. And that’s a harder job to be the first person that ever toys with new features. But, it’s also interesting and cool.

Matt Yonkovit: Oh, yeah. Oh, yeah, definitely. So you guys have gotten a lot of press around some of the performance that you have done, I’ve seen some numbers that you’ve thrown out there on the scalability and how you’ve helped certain customers achieve that. It seems like scalability and performance is often desired. But it’s often becoming a lost art in a lot of databases. Because people are more comfortable just paying more, and trying to use the easy button on the cloud, or wherever, like, oh, let’s just add more hardware. Let’s add more, add more, add more. So maybe tell me a little bit about, like, from a performance perspective, what are you seeing in the market, like the customers that you’re showing on your website, and that you’ve done press releases on, they’ve seen some pretty staggering improvements in performance. You know, has that focus on performance done well for you, is that something that you’re seeing the market starts to, like, look at, okay, performance does matter, again?

Dor Laor:
So our approach was all started with performance. So we’re very focused on performance, but it’s not just performance that we focus on. So as with my answer to 2. Performance matters, but it’s not everywhere. If performance doesn’t matter, like everywhere, where users have low amounts of volume, I tell them go ahead, use relational database, it’s not a big data problem, go ahead and solve it with a better tool that offers better flexibility, a Scylla as much as we try to do tonnes of good things, relational services are better for for smaller databases, if they can be a good match for big data sets even better, but they cannot. So they break at some point. So if you do needs, many companies today, since data explodes all the time, then they do need to have a a to deal with large data sets with with sometimes performance can be throughput can be latency, m can be very large data volume, in some databases cannot scale at all, or some databases can scale but it becomes super hairy to scale them. And you need to maintain them a lot. And this is where they break. And sometimes it becomes too expensive. And sometimes the whole project won’t have any meaning if it’s so expensive. So performance overview, this performance and scalability solve you this whole range, but we’re really not right. We did start with a performance goal but because we wanted to improve a pretty good standard as Cassandra or as DynamoDB. But it’s definitely not just performance. So we take a lot of pride, a about simplicity. So a Scylla is performance oriented. But in order to get the whole idea to have great out of the box experience, we have a shell script, which is the installation, you just run it and it’s automatically configures. All of the right kernel settings of the rights or settings with one command that creates a raid device with the right striping. It configures NTP, it does everything. And we even measure that to run some mini benchmark in order to evaluate the disk performance because one of the benefits of Scylla is not to queue on the file system or on the disk, do the queuing within Scylla. So we can control and prioritise what’s low latency IO, and what’s not latency sensitive, sensitive. So our installation tool measures the disk performance. So we try to, it’s just one command that you run once, and that’s it. So simplicity matters a lot, I relate to it a lot. And also functionality, like the CDC now is super beneficial. And you can hook a Scylla to Kafka super easily, etc, etc. And there’s other features that are really important. It’s not just about performance.

Matt Yonkovit: Yeah, and we’ve seen, like, with the growth of the cloud, a lot of the cloud growth is because it’s easy, right? People want to click the button and just have it work, and have it set up and not have to worry about it. And the more you have to adjust by hand, it satisfies the engineers to have to do that. But we’re seeing more developers not want to have to get down to that low level, which is a far cry from 5-10 years ago, and even when you startedб when you worked at Red Hat, and when you were doing KVM, it was probably a lot more hands on in the weeds in the settings. So I definitely agree. And I’ve seen that growth in the easy side of desire to have things just kind of be the point. And click. Now I’m curious as you’re talking with users and customers, what are you starting to see as the trends that they’re experiencing, that you’re helping them with? Like, where is the market going? How is it evolving?

Dor Laor:
And so it’s pretty much related to what you said about cloud and cloud and databases, service is super beneficial. And, people need that in. And one aspect is the ease of use of it. But also the core infrastructure needs to support that. So if you a cloud user would expect elastic deployment. And this is a big trend. Yeah, you want the disability database of yours, React dynamically to load, whether it’s Black Friday, or whether it’s every day. The night has less amount of traffic, and over the day, there’s more. And ideally, the database will react to that. And there’s lots to be done both in terms of ease of use, but also under the hood. Because if you need a flexible deployment, you’ll always be moving terabytes of data around. And that’s again goes back to having a perfect scheduler that can schedule low priority queries on top of just streaming of terabytes of data that happen all the time. So it’s important that one trend is Database as a Service. Another very good trend is Kubernetes. Like two years ago, people wouldn’t do stateful applications with Kubernetes. Now, it’s possible in many go to Kubernetes route and use in a Scylla with Kubernetes to do that.

Matt Yonkovit: Are you seeing growth in that space? So you’re seeing growth in that? Okay, okay. We’re seeing about a third of the users out there starting to do that to use it for a database production workload. So it’s not the majority yet, but it is moving in that direction.

Dor Laor: I definitely agree. No, not sure it will be a majority, especially with the Database as a Service will be the majority. Maybe not. Also, maybe not today, and we see a high growth of it. But it will become the majority for everyone and and there will be users who want to use it manage their own, and a large portion of it will be Kubernetes.

Matt Yonkovit: Yeah, and I know you have your own cloud offering your own databases and serve as the Scylla.Cloud. Is that where you’re seeing? Like, have you seen the movement more towards that, then either the on prem? Is that where you’re seeing your trends?

Dor Laor: Yeah, but by far, some of our kind of the non managed solutions, Scylla enterprise, or still open source, many times people manage it on their own, but run on the cloud. And sometimes they insist on running it themselves. And it’s, it’s fine, we provide all of the options. But the fastest growing service of ours is silica, silica, which is a Database as a Service. So that’s indeed, it makes sense. Because no matter how we will simplify this bit of database management, it’s still complicated. And you need to be aware, to wait for node failures, data centre failures, recently, over the age burn, has the fire in one of their data centres. And we have the customers and nothing happened big because because Scylla is prepared to win. But it’s hard to get it right. And to test it. And some people don’t do backup at all, some do backup, but don’t test the Restore. So just buying something, especially for low scale. If you have a high scale, you may be able to have your own engineers who are going to manage it. But for low scale, it’s impossible for a small company or a small project within a big company to invest all of the right time to make sure that their deployment is flawless. And it’s just easier to buy from a service.

Matt Yonkovit: Absolutely. Well, Dor, I want to thank you for joining me today. And we’re just about out of time. I do appreciate it. This was a great conversation. You know, you’ve been doing great work over there. I’ve seen you know, a lot of good press I’ve seen a lot of users who use our products have also tested yours. So really excited to see your growth and congratulations on pivoting and getting to the business where it is today.

Dor Laor: Thanks a lot. Thanks for having me. And also keep on doing a great job with Percona.

Matt Yonkovit: All right, we’ll do it.

Wow, what a great episode that was. We really appreciate you coming and checking it out. We hope that you love open source as much as we do. If you like this video, go ahead and subscribe to us on the YouTube channel. Follow us on Facebook, Twitter, Instagram and LinkedIn. And of course tune into next week’s episode. We really appreciate you coming and talking in open source with us. ∎