In a very detailed article, Slack engineers Ahmed Eid and Arpita Patel provided an interesting look at the processes they adopted over the years to improve developer experience in a number of different areas and the tools they used for this purpose.
The developer experience in Slack is catered by a dedicated team of eight people, which was established to provide the answer to growing costs as the organization and its development team grew. Among the areas of the development process that caused the most costs, Slack engineers focused on merge conflicts, long-running CI jobs, flaky tests and CI infrastructure failures.
Although developers can learn to solve some of these issues, the time spent and the cost incurred is not justified when the team grows. Having a dedicated team that can focus on the problem areas and identify ways to make the developer teams more efficient will ensure that developers can maintain a deep product focus.
Estimated at $2,400,000 annual cost per 100 developers, merge conflicts are the single most expensive pain point. This results from Xcode project merge conflicts, concurrent merges to the main, and more drawn out request review process.
Xcode project is notoriously difficult to merge without running into multiple conflicts. To address this problem, Slack engineers used Xcodegen to generate .xcodeproj files from a YAML file, a much more forgiving format.
Multiple concurrent merges to main bring an increased risk of merge conflicts, stop the merging of additional PRs until the conflict is resolved. For this, Slack adopted Aviator to queue all PRs and process them one by one. Instead of merging a PR into main directly, Aviator tries to merge main into a developer branch. If the step breaks major, the PR is rejected and the author notified.
Finally, to accelerate the lifecycle of lift requests, Slack engineers found it useful to introduce timed alerts for PR assignments, comments, approvals and direct messages for successful builds, including the option to merge the PR without leaving Slack. To achieve all this, they created their own GitHub Bot, called MergeBot.
Mergebot has helped shorten the pull request review process and keep developers in the loop. This is yet another example of how saving just five minutes of developer time saved ~$240,000 for a 100-developer team in a year.
Fortunately, GitHub supports a similar feature called Scheduled Reminders, although it doesn’t provide one-click merging of the message itself.
Improving the PR/merge process wasn’t the only action taken at Slack to improve the dev experience. Another area that incurs high costs is testing and failures in their CI infrastructure. In the first count, the solution was parallel test execution along with a strategy to run only the tests that are strictly required for a given PR based on the PR diff. On the second count, BuildKite proved effective to increase C-infrastructure reliability.
According to Slack, improving developer experience both made developers happier and reduced overall development costs. If you’re interested in the full details of how Slack achieved this, don’t miss the original article.