The Cost of Fixing Bugs in Production
Ok, so let’s address why software bugs that reach the production environment are so hard to fix. If you don’t have a structured process of identifying and repairing bugs prior to release, you will inevitably wind up with many bugs in production. And finding and fixing bugs in production is resource heavy and very time consuming.
The 11 Steps to Fix a Bug in Production
Here are my recommended steps for fixing bugs in production:
- Find out where in the code the bug is occurring
- Determine the damage caused (e.g. database corruption)
- Write scripts to correct the damage
- Write code to fix the bug
- Write code to test the bug fix
- Copy the production database to a staging environment
- Run correction scripts on staging
- Run scripts to check the results on the staging database
- Push code fixes to production
- Run correction scripts on production
- Run scripts to check the result on the production database
There… That’s not so bad, right? Wrong – this approach is really expensive!
The Cost of the 11 Steps of Fixing Bugs in Production
So, what’s the average cost of fixing bugs that reach your production environment? We can use the following equation:
number of people * number of days * cost per person-day * number of bugs = Cost of Fixing Bugs
What would be the cost for a simple bug fix needing only 1 person for 10 hours?
1 person * 10 hrs * $50 per/hr * 1 bug = $500
What’s the cost for a medium to average bug fix needing 3 persons each working 36 hours?
3 persons * 36 hrs * $50 per/hr * 1 bug = $5400
And what is the cost for a hug bug fix needing 5 persons each working 40 hours?
5 persons * 40 hrs * $50 per/hr * 10 bugs = $100,000
As you can see above, based on the number of resources, number of hours, cost of each resource and number of bug fixes, repairing bugs once they have reached your production environment can be very high in time, resource, and cost.
Conclusion
So look, if you haven’t read my blog article on Proactive vs Reactive White Box Testing, I encourage you to give it a read. I provide you with strong arguments that the cost and benefits of proactive, automated white box testing your software applications far outweigh the time, resources, and costs of being reactive and waiting to fix the bugs when they are found in production.
