After reading “Technology Radar” from Thoughtworks, I decided to buy a book they recommend in one of the chapters. Accelerate by Nicole Forsgren presents a scientific approach to productivity in modern software development. After reviewing 2000 organizations, Forsgren outlines key takeaways on how to improve your company culture and performance.
In this article I am going to focus in how to effectively measure software development performance and giving examples of how we implemented them at FXStreet.
Over the years there have been many attempts to difficulty measure the performance of software teams. The problem is that most models suffer from 2 major deficiencies:
During the gloried past, some companies used number of lines of code to measure performance and productivity. The flaws of this approach are obvious, less lines are usually more maintainable but, on the other hand, a few lines of code can be difficult to understand. So, this KPI is not ideal in most cases.
With the introduction of Agile methodologies, a popular metric was velocity. Despite being way better than the lines of code, it is still tricky and only works with isolated teams or companies where the team members do not change a lot. In large organizations, this measurement can be use to compare productivity between teams leading to inflate their estimates.
In Accelerate, Forsgren, Humble and Kim identify 4 metrics that resolve the 2 common pitfalls explained before.
Lead time tells you how much time it takes from a customer making a request to the request being satisfied. We have changed a little bit the definition for the sake of simplicity and for us the definition is how much time it takes from a task since we start it until we give value to our customers (being in production).
We use Jira as our Task Management Tool and, luckily, Jira has the Control Chart report. It shows you how much time a task (story, bug, etc..) is in certain state. So after each sprint we use this chart to derive lead time. JIRA Service Desk
Number of deploys tells you how many times a software development has been deployed to production in a period of time, usually a sprint.
Since we use Azure DevOps to manage the pipelines of CI and CD we have created a step (powershell) in the release process that stores what and when anything is deployed. Azure DevOps
Typical deployment pipeline at FXStreet
And this is stored in Azure Table like this:
Table where we store the name of the service and the environment deployed
Change fail percentage tells you the percentage of changes to production that fail, including hotfixes, rollbacks, fix-forwards, etc..
This is the trickiest one to measure, there could be various strategies in order to record this KPI:
In this case our approach is manual and we just review all the deployments (picture 2) in a sprint looking for the same service being deployed twice in a small period of time and taking notes if someone has spotted a bug after a feature being released.
Mean time to restore tells you how much time it takes to solve a critical situation like a service outage.
Unless you have this type of failures in a daily basis this is a measure easy to track manually, just time since the beginning of the outage until is solved. If you monitor all of your services with an APM like NewRelic, DataDog or Application Insights you can easily get this number. At FXStreet we also create a postmortem document every time something like this happens helping us to track this KPI. Datadog HQ
After reading Accelerate, I decided to put these 4 metrics into practice and see if could improve our software delivery process as a team. We have been using this approach for the last 3 months and I can see 2 advantages:
Javier Hertfelde
Originally published:
April 8, 2019
