Selecting a Load And Performance Testing Framework - Gatling vs. Grafana k6
I am an enthusiastic developer since my childhood. I have started with Plus 4 and C 16 machines programming in Basic. I still remember when we were jumping on the bed with my brother, and accidentally saw the Commodore 64 computer on the top of the wardrobe, which was planned to be a Christmas gift for us. I don't have to tell you that we couldn't wait till then. :) I will never forget this memory. Of course in the beginning we only played on the computer, but later I became interested in how can I make my own programs. I have to tell when I first faced the binary number system I did not understand it, but I was still able to draw some sprites on the screen. :)
I continued learning software development during grammar school. I got my Masters degree at the Budapest University of Economics and Technology where I learnt many languages and also learnt a lot about general software development theory and practice.
I have started my professional career as a C++/C# developer, but I switched to Java after a few years. I've visited many meetups and Code Retreats during this time and I've learnt and introduced Agile methodologies in our company. During the years I lead the development team in close cooperation with the product management and testing department.
In 2020 I felt I need something new and I turned towards test automation both API and UI. I did not regret my decision. Since then I tried many frameworks like RestArrured, Selenium, Protractor, Appium, Selenide and Karate.
At the moment I am working as a Senior Software Developer in Test using Selenium and Selenide for UI testing and Karate for API testing.
I believe knowledge sharing is very important, so apart from helping my colleagues work I started my technical blog recently: https://mszeles.com Sharing your knowledge provides many ways for your own development. Whenever I write an article many questions come into my mind which might be interesting for the readers, so I try to answer all of them in my posts. Doing this I learn a lot about the subject. In additionally sharing your knowledge gives a good opportunity for feedback based on which you can further improve.
I just got the task, to advise on which performance testing framework to use for our product. As we already use Grafana, the suggestion coming from our Dev Lead, to examine whether it would be a good fit or not.
As we are using Karate DSL for API testing, a straightforward choice would be to go with Gatling, as it nicely integrates with Karate DSL, and we can reuse our existing tests for performance testing.
I compiled a list of important aspects to examine when choosing a performance and load testing framework. I reused the list from my Selenide vs. Selenium article and adapted it a little bit. I will use a simple scoring system but of course, again, you cannot simply decide based on the final score.
- Skillset available in the company
- Learning curve and ease of use
- License cost
- Integration with cloud testing services
- Test case reusability
- CI/CD integration
- Detailed reports
Skillset available in the company
Our team has experience with Karate DSL. The test cases written with Karate DSL using the Gherkin language can be reused with Gatling so Gatling seems to be a reasonable choice.
Grafana k6 requires JavaScript knowledge. Of course, our front-end developers are familiar with JavaScript, but our test automation engineers are not.
So the standing is Gatling:Grafana 1:0.
Learning curve and ease of use
In order to use any load testing framework, you have to be familiar with the theory of performance testing, which is needed independently from the tool. I am planning to take the ISTQB Performance Testing training and exam later this year.
Having the theoretical background, using Gatling is very simple, thanks to the reusability of the existing test cases.
As Grafana k6 requires JavaScript and learning a new syntax for testing code, it is one point again to Gatling. The standing is Gatling: Grafana 2:0.
License cost
Grafana k6
Grafana k6 has both a free and a paid version. Grafana k6 itself is an open-source framework that you can use free of charge.
The pricing of the cloud solution's Developer version is $99/month which allows 50 test runs monthly with a max duration of 15 mins, and it can only run on one thread.
The pricing of the cloud solution's Team subscription is $499/month and it allows 75 runs per month, which I still consider very little. The max run duration is 30 mins and you can run your tests on 2 parallel threads.
There is also a pro version for $1499/month which allows 100 test runs with a 60-minute time limit and 4 parallel threads.
There is also an Enterprise version which has a custom price depending on your needs.
Gatling
Similarly, Gatling has both a free and a paid version. Gatling is also an open-source framework. For the paid version you have the option to use the cloud service, or a self-hosted version.
The Scout cloud subscription costs 99€/month with a limit of 60 minutes test running duration and it has a 1-month data retention period.
The Scale version costs 396€/month with a 24hours maximum test execution duration and 3-months data retention period. This version also comes with a CI/CD plugin.
The Corporate version costs 1380€/month with more than 24 maximum test execution time and 6 months of data retention
The self-hosted version starts from 4800€/year.
However we won't use the cloud service in the short run, but we might use it in the long run. With the pricing of Gatling, it is a clear winner. Gatling: Grafana k6 3:0.
Integration with cloud testing services
This topic is the same as the one above, so the standing is still Gatling:Grafana k6 3:0.
Test case reusability
With Grafana k6 we have to start building our performance tests from scratch, and with Gatling, we can reuse our existing Karate DSL tests, so it is a clear win again for Gatling. Standing is Gatling:Grafana k6 4:0.
CI/CD integration
Grafana k6 can be integrated into your CI/D pipeline, you can find some instructions about it here.
Gatling free version can be also easily integrated into your CI/CD pipeline and from the cloud version, it is available from The Scale subscription.
This is a draw, so the standing is 5:1 for Gatling.
Detailed reports
Both Gatling and Grafana k6 have a detailed reporting with lots of useful charts.
Conclusion
It is a clear win for Gatling. So for our team, I suggest using Gatling framework for performance testing.
What is your number one performance testing framework? Please share your experiences in the comment section.
In case you do not want to miss my posts, just follow me here on Hashnode, on LinkedIn, on Twitter on Medium.com, on dev.to and even on Instagram. 😊
📚Join the Selenide community on LinkedIn! ✌
