Software development organizations define success by providing the right products to their customers that meet quality, schedule and budgetary constraints. That means the organization delivers a product that works the way it is supposed to work, when it is planned to be available, and within the project budget. From a pure project management perspective, other constraints deserve consideration as well, such as scope, benefits and risks, but we will keep things simple for now.
The activities that transform an idea into a functioning software product are often collectively referred to as the software development pipeline. You can think of an organization’s development pipeline as the series of related tasks that incrementally build a functioning software product. These tasks include specification, design, development, testing, quality assurance, building and deployment. Inefficiencies related to any of these tasks degrade the organization’s ability to get a software product into the hands of its users. Increasing the efficiency of the development pipeline makes happier customers and generates higher profits.
One of the best ways to meet or exceed quality, schedule and budgetary constraints is to implement and rely on repeatable processes along the software development path. Let’s look at a few ways to make your pipeline more efficient.
Automating mundane development and deployment tasks
The main key to pipeline efficiency is to lean into automation. Many manual tasks are efficiency killers, so any task that you can initiate, control, monitor and manage using software tools and well-defined rules is a good candidate for automation. Any task that does not really require human interaction or assessment could be a good place to start.
Automated testing is a good candidate for deploying automation. Developers who create test scripts for their code make it possible to iteratively test that code with minimal human interaction. That can be a huge efficiency gain when ongoing changes require multiple rounds of testing to assess software stability.
But testing is not the only candidate for automation. Start by defining each step in your development pipeline. What does each step require as input, what happens during that step, and what output or deliverable does each step produce? Answering those questions will help identify the most likely automation candidates.
Start small. Avoid automating everything at once. Start with the tasks that seem to be the best automation candidates, and expand your scope from there.
You also do not have to invest in an enterprise-wide automation software suite. There are lots of good tools out there (PhoenixNAP has a good starting list of tools in this article), but you can start with a few timed scripts using crontab in Unix and Linux or scheduled tasks in Windows.
The idea is to start with your low-hanging fruit and get your hands dirty with automation. Use automation to reduce windows of risk by merging early and often and implementing shorter development cycles. Reducing the amount of time any code is being changed by one or more developers reduces the exposure to integration problems. The point is to centralize the code base view on which developers base their modifications.
Building best practices into your development pipeline
If your organization is just starting to implement automation in your development pipeline, invest the time and effort to learn from best practices first. If you already have automation implemented, compare how well your solution aligns with best practices.
These suggestions are not hard rules, but general guidelines that other organizations found to be helpful.
Limit the scope of any change: Focusing on small changes to software requires less time, effort and assessment to evaluate stability.
Find the low-hanging fruit first: Automate the simplest and most easily automatable tasks first before attacking the more complex tasks.
Remove sequential task barriers when possible: Many tests, and some other tasks, can occur in parallel. Let automation help you execute parallel tasks to reduce time spent at any step in the development pipeline.
Decrease human touchpoints: Make as many decisions as possible without human interaction to reduce artificial delays.
Automate testing at multiple levels: Avoid limiting test automation to any type or phase of testing. Most tests, from unit tests to full regressions tests, can be automated.
Identify and eliminate unnecessary duplication: Never do anything twice that you only need to do once. As you implement automation, be on the lookout for duplicated tasks.
Making your development pipeline better
Once you have a development pipeline that works well, it makes sense to evaluate it periodically and see if you can make it more efficient. Even a pipeline that aligns with best practices can deliver suboptimal efficiency. How your organization uses your development pipeline is as important as how you set it up in the first place.
Here are a few tips for evaluating your pipeline and making it more efficient.
Avoid skipping pipeline steps to address urgency: Too many organizations directly apply hotfixes to problems ASAP. Doing so risks those fixes being overwritten during standard pipeline tasks. It also skips your testing and QA steps, which could allow fixes to carry bugs into production. The best approach to hotfixes is to use your own pipeline for all production code. If your pipeline does not handle hotfixes well, change it.
Harden your pipeline to resist security threats: The SolarWinds Orion supply chain attack showed how vulnerable the development pipeline can be. Treat all code and configuration data as vulnerable intellectual property, and protect it with aggressive access controls, VPNs and other restraints to limit exposure to security threats.
Use risk management for enhancing your pipeline: When assessing steps to automate or optimize, use common risk management techniques to determine the most frequently used steps and those with the greatest impact on the final software product. Those steps should be the ones that get the most attention. As time and budget allow, move to the steps that are less common and have a smaller impact on the final product.
Cultivate a culture of continuous improvement: Never stop improving. Frequently assess how well your pipeline is performing, and review any reported issues in the development process. Expand what works and fix what does not work so well.Your development pipeline governs how well your organization’s software development process meets its goals. Efficiency is not the lone goal, but an efficient pipeline makes it a lot easier to deliver the right products to your customers that meet quality, schedule and budgetary constraints. And that is the whole point.
