2pm is the golden hour for reporting bugs
Turns out that 2pm is the golden hour for reporting bugs. It is interesting to see that activity is growing in the morning up to 9am, and then again towards the afternoon, peaking at 2pm. There is also a local spike at 5. It’s hard to know for sure why that is. A possible explanation is that some people do a round of testing before the end of their work day. That brings us to the next point – working hours.
Is there even such a thing as a standard working day in software development circles? Turns out – not really. We compared how many recordings are uploaded within the regular 9-to-5 as opposed to other times.
The split is almost equal, which was at first a bit surprising to us. When you look at the previous graph (bugs reported by hour), there seems to be more activity during the normal working day. You therefore don’t expect an almost perfect 50/50 distribution. The explanation is quite straightforward though – 9-to-5 is only 1/3 of the day. So the remaining period is twice as long in terms of hours, but the rate of reporting bugs is two times lower. That calls for a question: are the individuals in product teams more flexible these days? Or are we just working longer and should reconsider our habits?
A quarter of bug reports are filed on Tuesdays
We expected people to file fewer bug reports on Fridays and weekends, than the rest of the week. This assumption proved to be correct. What we didn’t expect is that the first 2 days would see higher activity. Our assumption was that there would be little to no difference between Monday, Tuesday, Wednesday and Thursday.
2/3 of bugs are reported on pre-production environments
In our particular sample about 2/3 of bugs are reported on pre-production environments (local and staging). That number is lower than the one in the original research by Capers Jones.
One of the plausible reasons can be a different user group. While Capers Jones seems to have focused primarily on software development teams, our product is used not only by Engineers, QA, PMs and Designers. We see an increasing amount of activity from customer support teams, for example. That means 2 things. First of all, business and operations roles in general don’t use staging, so they can only report bugs on production. Secondly, certain things they might report as bugs, aren’t actually bugs. Even a person, who knows the product inside-out, can’t immediately tell if something’s indeed a bug every single time. Some behaviours can be caused by hard-to-find settings, ad blockers or connection issues.
We can’t say for sure if this assumption correctly explains the 20% difference in the number of bugs reported on pre-production environments. Many factors are at play: company size, development methodology (agile/waterfall), quality assurance process, complexity of business logic and even how a particular company defines as a bug.
“Bug” is the most common word in bug reports
The result turned out to be rather predictable. Words like Bug, Error, Page, Test were the most common. To be frank, we did not expect “Collection” and “Date” to be that frequent – could be due to the sample, though.
What we found the most interesting, and in a very positive way, is what words were NOT on top of the list. From personal experience we know that sometimes people describe bugs in very general terms, i.e. “Nothing works”, “X is broken”. To help companies avoid that, we have even created a bug report template to help especially non-technical people communicate bugs better. Yet, in our sample that behaviour was gladly uncommon.
Most bug reports are made with macOS
The data we have on Windows is a lot less granular. We can only say that almost all recordings were uploaded using Windows 10.
Google keeps Chrome users on the latest software
The earliest bug reports in our dataset are 2 months old. At the same time, very few of them came from Chrome versions released over 4 months ago. It seems that Google is fairly good in making sure that users upgrade to newer versions.
We can only recommend this strategy as a best practice – it makes things much easier to debug. It also ensures a more homogeneous user base, which has positive implications not only on software development teams, but also technical support.
Bug error. Test collection date issue.
And here it is – the most “average” bug report of this fall.
Title: Bug error. Test collection date issue.
- Time: Tuesday, 2pm
- Environment: Staging
- Operating system: macOS 10.14.6
- Device vendor: Apple
- Browser: Chrome 77
- Screen size: 1920×1080
- Window size: 1920×798
- Locale: en-US