I just got the task, to advise on which performance testing framework to use for our product. As we already use Grafana, the suggestion coming from our Dev Lead, to examine whether it would be a good fit or not.
I compiled a list of important aspects to examine when choosing a performance and load testing framework. I reused the list from my Selenide vs. Selenium article and adapted it a little bit. I will use a simple scoring system but of course, again, you cannot simply decide based on the final score.
- Skillset available in the company
- Learning curve and ease of use
- License cost
- Integration with cloud testing services
- Test case reusability
- CI/CD integration
- Detailed reports
Skillset available in the company
Our team has experience with Karate DSL. The test cases written with Karate DSL using the Gherkin language can be reused with Gatling so Gatling seems to be a reasonable choice.
So the standing is Gatling:Grafana 1:0.
Learning curve and ease of use
In order to use any load testing framework, you have to be familiar with the theory of performance testing, which is needed independently from the tool. I am planning to take the ISTQB Performance Testing training and exam later this year.
Having the theoretical background, using Gatling is very simple, thanks to the reusability of the existing test cases.
The pricing of the cloud solution's Developer version is $99/month which allows 50 test runs monthly with a max duration of 15 mins, and it can only run on one thread.
The pricing of the cloud solution's Team subscription is $499/month and it allows 75 runs per month, which I still consider very little. The max run duration is 30 mins and you can run your tests on 2 parallel threads.
There is also a pro version for $1499/month which allows 100 test runs with a 60-minute time limit and 4 parallel threads.
There is also an Enterprise version which has a custom price depending on your needs.
The Scout cloud subscription costs 99€/month with a limit of 60 minutes test running duration and it has a 1-month data retention period.
The Scale version costs 396€/month with a 24hours maximum test execution duration and 3-months data retention period. This version also comes with a CI/CD plugin.
The Corporate version costs 1380€/month with more than 24 maximum test execution time and 6 months of data retention
The self-hosted version starts from 4800€/year.
However we won't use the cloud service in the short run, but we might use it in the long run. With the pricing of Gatling, it is a clear winner. Gatling: Grafana k6 3:0.
Integration with cloud testing services
This topic is the same as the one above, so the standing is still Gatling:Grafana k6 3:0.
Test case reusability
With Grafana k6 we have to start building our performance tests from scratch, and with Gatling, we can reuse our existing Karate DSL tests, so it is a clear win again for Gatling. Standing is Gatling:Grafana k6 4:0.
Grafana k6 can be integrated into your CI/D pipeline, you can find some instructions about it here.
Gatling free version can be also easily integrated into your CI/CD pipeline and from the cloud version, it is available from The Scale subscription.
This is a draw, so the standing is 5:1 for Gatling.
Both Gatling and Grafana k6 have a detailed reporting with lots of useful charts.
It is a clear win for Gatling. So for our team, I suggest using Gatling framework for performance testing.
What is your number one performance testing framework? Please share your experiences in the comment section.
📚Join the Selenide community on LinkedIn! ✌
Did you find this article valuable?
Support Miki Szeles by becoming a sponsor. Any amount is appreciated!