How often do we change our sbt builds?

Back

How often do we change our sbt builds

A non-scientific analysis across many Scala projects

I spend a lot of time thinking about build tools and popular setups, and what changes are more likely to impact developers positively. In one of these ramblings, one question popped up: “How often do we change our builds?”

It’s not easier to answer this question, data on the topic is scarce. So I challenged myself to come up with some data to satisfy my curiosity.

Next I show some of the data I collected from Scala projects using sbt, the most popular Scala build tool. I would love to see how these numbers change across build tools — I’m sure they do.

Methodology

  1. Clone the repo and follow installation instructions if required.
  2. Run a script that detects changes to all *.sbt files across all commits.
  3. Get three data points: total commits, commits that modified the build and percentage of commits that changed the build.

The source code of the script is available in the following Gist. Feel free to run it in your project.

Disclaimer

There are two limitations with this approach:

Results

I collected the data in Airtable. It’s sorted in descending order. The results vary between ~37% and 0.43%.


Conclusions

To my suprise, the projects seemed to be clustered in two groups in a homonogenous way:

  1. A group of 16 projects covering the range of 15-37% of modifications to the build.
  2. A second group of 16 projects covering the range 0.43-8.7% modifications to the build.

It’s surprising that the division is so perfect… After all, I picked the projects randomly1. It seems that there’s no clear-cut percentage of build modifications in a project.

The data points analyzed are not enough to draw meaningful conclusions. However, we can safely assert that the developers (on average) of 50% of popular open-source Scala projects change their build at least 15%.

If your project belongs to the second group, changes in the build files are scarce and therefore unlikely to affect your developer workflow. If you are in the first group, especifically if you are working on a codebase like circe, your developer workflow is slown down by reloading the build after every change.

The builds of these projects are all complex and, therefore, 15% of changes in the build are bottle-necked by sbt which does not provide fast reload times (let alone fast compilation times aside). Modifications to these builds slow down developers significantly if every change to the build takes at least 15s 2.

I don’t have any particular advice for maintainers of such projects except for identifying why so many build changes are required and trying to outsource them to external tools. Most of the build-related problems can be solved outside of sbt.


  1. The analyzed projects have medium-to-large size and they have been sampled from the local Scala community build in my computer (they contain both libraries and applications). It’s likely there’s still a bias.

  2. Note that the data only shows committed changes to sbt build files. I don’t know how many changes were required before the commit got merged. Maybe a good approximation would be two or three local changes per one committed change, on average?