How We Improved Release Process with a Hackathon Project
In this article, I’d like to share the story of how we have improved the company’s release process at TrueMoney over the last year. This is not to propose a definitive solution for a problem of release management or anything close to that. All I want to do here is to write down about the experience and the thought process of taking one’s idea to production.
The situation
First, to understand what is the problem we want to solve here. In Ascend-TrueMoney, our hundreds of microservices can be divided into 2 big categories: apps on various CI/CD with manual release process, and apps on the new CI/CD with automated release process.
And what do I mean when I said manual release process? Actually it is the classic way of handling change management. We have 2 release cycles each week. For the release planning, the coordination of release between each team is done by filling out document as a release note and Excel sheet as a deployment plan for each release date, the details consist of ‘what are the changes, what are to be deployed, how to do sanity test, how to do rollback if testing failed’. This document is looked after by one team who is assigned the role as a release manager for that date. They have manual tasks to create the document sheets, make the data filled consistent and then review it with the team who is responsible for taking care of the deployment.
Don’t get me wrong. This process helps everyone keep in mind of the importance when you apply changes on the production environment. One major bugs can cause the lost of money that need to be fixed immediately. That’s why you need to keep track of these information, and can do rollback of the change properly. But overall, the process can be described as cumbersome for developers. When you have 10+ teams working on one Excel sheet, it can cause many errors and conflicts in the file.
Even though the new standard is the CI/CD with automated release process, we still have many apps on the first category to consider. The time taken in migrating all of them to the new CI/CD will take years. Even for the second type, sometimes there will arise a dependency or plan to be taken care of. Rather than waiting until the complete transition to automated release process for the whole company, we felt the urge to build a better tool right now.
How we build it
Our proposal is to create our own release management tool to replace this manual document filling process. It is a collaborative web app that will take care of gathering data from each team and presented them in a format that is easy to adjust without creating conflicts. Many details can be saved as a template for autofill in later release. The data submitted will also be changed to locked state to prevent editing when we reach the release date. It will also automatically generate a deployment hash for CI/CD that needs one.
We are a team of 5 people. During the hackathon competition, we spent 2 days on building a working prototype. We chose Angular for frontend, and Java-Spring Boot for backend. The reason for choosing Angular was because I felt it had what we needed to build prototype quickly for Hackathon demo. It is a full-feature frontend framework that should provide for all the requirements without the time wasted to consider many different options for each task. I worked on the frontend, where I found it a good place to channel my experience when I worked as a frontend dev. Angular material is easy to use, and help me in building a nice UI with drag-and-drop feature in the short period of time.
The presentation went well apart from a small bug in the demo. After the idea got approval from the judge, we started building the thing to make it to production. We treated it as a proper software project. A Kanban board is created to track the works to be done. When the app is quite ready, we do a demo to everyone and launch a beta testing. In the course of 6 months, we have completed more than 100 improvements on our prototype. We added the support for all the CI/CD available for release.
But due to covid-19 situation, after just one month, we decided to stop the beta testing period, since everyone has enough problems with trying to adjust to the WFH situation, we don’t want to burden everyone with having to use the existing process and testing our new tool at the same time.
After long persisting, more requirements gathering, lots of messing with the environment and some struggling with the terraform configurations, our production environment is ready to use. We have also created a 94-pages presentation slides as completed guide on how to use our app. Then we do a final presentation to prepare everyone with the new process. Finally, we release the project to production in July 2020. There were some minor issues during the first week, but we do a hot fix on it in time, so the production deployments were not affected.
The result
Six months since then, our app has already taken care of 88 release cycles, with 807 versions of application release done without the need of Excel sheets. Many manual tasks are removed from the process. The roles of release manager are reduced to only review the release plan on the day before the release. We did a survey of the user experiences. The overall user feedback is good. They think it is easier and faster than the old process, and help reduce human errors and conflicts.
What’s next?
The app is still a work in progress. We plan to add more feature like a report for tracking the release result afterwards, more template data and more automations for the release manager. There are still many possibilities for improvement.
When looking back, the problem of release management is something that is still not taken care properly in many company. Apart from documents and excel sheets, there are some full-fledged solutions in the market that come with the cost, or there is the way of giving developer the full responsibility of releasing their own feature by themselves. It all depends on many things, like how mature is the engineering practice or how good is the regression test coverage, and find what is matched for your company’s requirement. Our way comes as a middle ground between these options: retain the process that help control quality of the release, but reduce the burden on the users, and try to do that with less cost.
