We’re excited to announce a new rating system for our testers!

 

Why?

Many testers were being unfairly rated. For example, there could be a tester who submits numerous high value bugs with a great bug report, but maybe they didn’t follow instructions for a few issues. Instead of ending up with a test cycle rating of 4.8 which they would’ve probably deserved, they end up with a 3.7.

 

We felt this was incredibly unfair. What matters is the value you provided to the project and customers. Now we have introduced a weighted system, meaning factors that we find valuable will have different weights based on their importance.

What’s New

As a tester, your testing process won’t be any different. You’ll continue to report high quality issues that deliver value. However what you’re probably curious about is what happens behind the scenes. In other words, what makes your score.

Bug Depth Redefined

Previously you were rated on how many high priority issues you reported. We listened and agree that this was the wrong approach on how to rate our testers. Sometimes there are projects where no high priority issues exist. Testers ended up avoiding these projects because they knew no high priority issues would be in there and therefore not get a great rating.

Now your rating is compared to the relative value you bring to the test cycle. So for example, if you are in a project and you submit 3 medium severity bugs (assuming they’re all high quality) in a project where no high severity bugs exist, then you will receive maximum score in the “issue depth” portion of your rating formula.

Bug Report Quality

Something else we decided to try to make more concrete is the Bug Report Quality. Each of your issues will be individually rated on how clear the bug reports are. And then your Bug Report Quality score will be the average of all of the approved issues you submitted in the test cycle.

If you’re looking to keep your bug quality score high, please follow these tips which we rate on:

  • Clear title (information: as little as possible as much as needed)
  • Clear and valid steps (one should be able to get the point reading it the first time)
  • Valid proof (logs, screenshot, video) that match the issue report
  • Valid estimation of severity
  • Label, Feature, Environment and other requirements are met

Cover all of these and your Bug Report Quality will be max points.

New Feedback Screen

Previously you were shown your feedback given to you per test cycle.

You’ll be shown each comment written on how you can improve your approved issues. Don’t worry, you can still see the feedback you receive per test cycle as well!

Please note as well that we decided to remove the ability for a tester to see their rating per test cycle. At Testlio we’re strong supporters of improving our tester community through constructive feedback, not by arbitrary ratings and grades. You’ll still be able to see your overall rating, but the focus should be on the feedback you receive from a test lead on how to improve.

What’s Next?

If you’re a good tester, you should be consistently seeing higher and fairer ratings. In which case, congratulations and good job!

On our side, we’re going to be continuing to monitor how the rating system is working and making sure all is going well. We know this is a very delicate part of your user experience, and we’re fully attentive on making sure that it rolls out smoothly.

 

We would love to hear your thoughts in the comments below!