Our team at OnGraph has always inquired about that “How would we guarantee a quality of a project? Or How would we release quality software to production every day?” In the following blog posts, we dig a bit deep into the process we approach and focus on how we do it.
Essentially, in the software development process when we step forward, we work on a series of strategies and protocols. Here is what we do:
1. Produce Small, Shippable Increments And Improvements
It’s easier to work, focus and make most out of the smaller things. Therefore, we separate the work into small additions as dimensions. This is a unique thinking about software development and delivery and has helped us maintain quality in our software projects. Those days are gone when a huge release of software is made at a time that brought a bunch of bugs and errors together.
Instead, we work and send small chunks of code every day as small changes lead to lower risk. We ship most parts of the feature in small increments prior to launch day for a large feature. However, shipping codes in installments and keeping features open to customers using features switches is more likely a business decision rather than a decision based on deployment success.
2. Define The Development Process Steps
Defining development process step-by-step is the beginning makes it easier to keep track of things and make corrections. At OnGraph, we work together with an issue tracking system. Hence every single code change is associated with a work item and goes through a very well-defined set of steps. The means – including peer reviews, UX reviews, and sometimes customer success reviews – are designed to make sure the code changes meet a high-quality bar before they are deployed to production.
3. Use Work Item Templates
Once the team project is first created, we develop Work Item Templates that help meet team’s specific needs and process. It enables team and managers follow a consistent standard. while enabling seniors to track the work that remains to be done on a project, as well as the work that has been already been completed. Like, each user story and code change contain at least the following items:
User story: Defines the core and high-level requirement from the user’s point of view
Acceptance criteria: Defines the details of the requirement, including the edge cases
Quality checklist: Verifies that all the steps in the development process (such as automated tests) were executed as a reminder for the team members.
4. Use Branches And Shippable Head-Of-Code Stream
An application development process involves a bunch of different features or ideas in progress at any given time. Among those features or ideas, some are ready to implement and others are not. Branching there help significantly if used properly. It helps manage the work-flow while reducing the risk and improve quality. The strategy is to protect that master branch (the trunk) at all costs and GitHub have been helping us work on our strategy and achieve our project requirements.
GitHub is our branch strategy tool that relies on branches for every single change. While not everyone in the industry believes on the productivity GitHub offers like continuous integration and deployment, but as we have used the tool and come up with positive results.
Our trick is to not have long-lasting branches and instead have small shippable increments as we have mentioned in section 1. There is only one rule: anything in the master branch is always deployable. Changes in branch don’t affect the master branch, hence it enables us to do a free experiment and commit changes.
The other significant point to focus on is that your head-of-code stream must always be shippable. This means your CI build on the head-of-stream must always be green. Keeping it healthy should be the team’s top priority. Focusing on it crucial as if the head-of-stream is in a red state (broken) for too long it will block the releases and potential hot fixes (quick fixes reacting to regression issues introduced by the releases that need to be deployed immediately.
5. Incorporate Continuous Integration Into Development Process
Continuous integration works as a defense mechanism against bugs. We can’t emphasize enough the significance of incorporating CI in software development practices. Continuous integration is at the core of any deployment pipeline and a must-have for a team that produces high-quality software.
We employ CI works for every single branch as well as for the master branch (the head-of-code stream, also referred to as trunk). When a developer builds a new branch, our CI system automatically detects the branch and starts running CI on it, such as merging the master into their branch and discover any integration issues.
Moreover, once a branch is merged into the master branch, we run a CI build on the master branch. Keeping that CI build green is everyone’s priority. At OnGraph, CI builds comprises a few steps such as diverse automated tests and static analysis.
6. Use Test Automation
A quality-driven software development process is incomplete without test automation. Yet it is something too many companies/teams avoid. It’s always cringing to hear people when they say:
1. There’s not enough time to write automated tests as we need to move faster. Or
2. When people ask if they can write the tests after they ship
Those two mentalities, which we now and then come around, usually indicate that there is not enough understanding about why automated tests are written. However, their reasons are simple. We write automated tests:
1. Because we want to save time and deliver high-quality software
2. Because we want to save money by avoiding running the same repetitive tests manually
3. Because it’s much more expensive to fix the bugs after the feature is shipped than it is earlier in the development cycle; and,
4. Because introducing regression issues can be very costly for your brand
Simply put, automated tests are a valuable weapon as it helps save a bunch of time and money. Automated tests are often categorized into the following three groups.
1. Unit tests (developers)
2. Integration tests
3. Functional tests
It’s cheaper to write and maintain unit tests, so we try to include it more of those and it helps in fewer integration and functional tests.
7. Use Static Analysis/Automated Code Reviews
Automated tools perform static analysis on the code. Static analysis enhances the quality of the code base and also educate the team on the best practices. Several tools like FindBugs and Lint, can be used to find issues in code. These tools also enable developers to increase understanding, learn best practices, avoid bugs and security issues, and make the code more consistent.
If you are planning to use one of these tools, we would recommend you to be sure that developers either have tools right in their IDE (Integrated Development Environment) to get in-context feedback or can run them locally.
Moreover, the same analysis should also be run on every code commit to a repository. You don’t want your developers to be frustrated because there is no way for them to get the same results in their local environment that they see on the CI system.
8. Make Code Reviews Mandatory
Conducting reviews of the code is the safest way to enhance the quality of the code and overall software development craftsmanship of the team. It grants several amazing benefits such as improve the overall design, enable you to search bugs, perform mentoring as well as allow knowledge-sharing.
We also approach of pairing developers even before any code is written. We discuss things in the design phase with them which bring convenience later on. Additionally, we make use of tools in order to review code easily. Here again, we rely on GitHub that enable us to reward great reviews as much as we emphasize new features that help us come up with the desired outcome.
9. Include teams from User Experience (UX), Customer Success (CS), and Product Managers or Owners (PM/PO) in the review process
Perhaps developers get engaged in building features that being worked on, other teams such as UX (user experience), CS (customer service) and PM (Project Manager) also work in coordination and monitor every progress. Depending on the feature, we interact with teams, collect feedback in order to perform several rounds of a verification process. With the help of UX team, we develop better user experience before the feature goes out.
Even though developers implemented the feature to the exact specifications of the UX team, we still look for opportunities to make things better when we use the software as a customer world. After that, customer success come into the picture and provide critical input about how customer uses the product and remind us that how a customers don’t always use the software as we expect.
10. Use Canary Releases
Before releasing the changes to the public, we deploy new releases internally two hours prior to identifying any issues within them. This process enables us to test them and uncover any issues. When corrections are ready, we merged them into the master branch. At first, the made changes are deployed to a set of servers internally that have access only by the people in the office as they do their daily activities. Such activities enable us to develop a dashboard for over selves and for our customers which eases investigation of customer issues. During such personal checkups, our team actively look out for any issues and report them if found any over our internal messaging tool. We appreciate this process since we don’t like issues to be found during canary release. The process saves us from shipping an embarrassing bug to the entire customer base.
11. Use A Crowd Sourcing Quality Assurance (QA) Platform
Being technology-driven company, we make use of crowd-sourcing QA platform that simplifies the process of running a large number of regression tests in a short amount of time. These platforms help our testers automate the testing process. Additionally, it produces aggregating results and also sends them back as notification to your linked messaging app or CI tools. Such platforms help us in the following ways:
1. Complement your automated tests with manual tests that are better run by humans
2. It helps us quickly run manual tests as they are distributed among a pool of QA resources and are in run in parallel
3. Our QA team focus on exploratory and in-depth testing of releases and features while automation enables them to stay away from repetitive regression testing.
12. Use Feature Switches Often
Features switches turn features on and off for a targeted set of customers. With this tool, it is possible to increase the quality of the features and reduces the risk before exposing them to the entire customer base. We found this way effective and efficient as it helps us observe the impact without exposing it to the customers. We have been using this approach both for risky features as well as for features which have been in development for months but are not ready to be exposed to customers yet.
13. Accept the risk
No procedure is free from risk and mistakes are unavoidable. Indeed, we believe if you are not making mistakes, you are not moving fast enough. However, we practiced all of the above steps, still there associated some risk in deploying to production every day. We go for broke and grasp it. Each time an issue emerges, we step back and take a gander at how we can enhance things.
14. Perform Post-Release Monitoring
Performing post-release monitoring, and the last thing we’ll cover, proceed with the string of tolerating hazard. While we expose ourselves to risk, we are additionally prepared to manage the outcomes. We utilize different tools to monitor the production.
1. ELK stack for monitoring logs;
2. New Relic for monitoring performance and client-side errors;
3. Our internal fleet of monitoring and self-healing bots.
15. Have Hotfix And Rollback Processes In Place
Being an active player in the industry, we keep ourselves prepared for a process that enables us easily fix mistakes or roll back to previous versions. Truth be told, getting the hotfix and rollback process right, and making it as simple as possible, is as important as making the release process easy. We prepare over-selves with the capability to rollback a release with the click of a button. We keep the process simple, reliable and automated in order to ensure a flawless experience.
It’s additionally a smart thought to ensure that more than one person, ideally many people on the team, know how to use it.
Last Words On Ensuring Quality In Continuous Delivery
We release codes each day but that doesn’t mean rushing the code changes out of production. However, we keep an eye and watch cycle time every week to find opportunities to improve our process and velocity, we consider quality one of our highest priorities. We and our team work hard to use the right tooling and processes as described above to deliver high-quality software.