From April 2019 to July 2019.​
​
The system we had built so far to do the testing, beta and general availability worked. But it was starting to show stress. We were stretching its capabilities too much. One of the difficulties we had was to synchronize all the elements and configuration to accomplish a deployment.

If you would categorize the type of system it was object storage, it would be in a category called micro services based. That is programs that talk to each other and work on small and well defined tasks. These programs were layered in a specific way to accomplish our goals.

Each of these programs had different source code and each of these code base was in a different location. The code to deploy had the daunting tasks to assemble the components and versions. Synchronizing everything was starting to be a mess.

Because we used a specific tool called stalk stack to deploy it was even a bigger mess. The most common way to use it was to create a branch for each environment. The code base starts to drift into multiple directions. This created so many issues to keep the environments synchronized. We started to really dislike it. At least a few production issues were created because of it.

We had reached the limits of the software, tools and organization we had so far in the storage team. We needed to inject some energy to change to a more stable orbit.

The process started with changing the deployment tool to ansible. This was the tool Scaleway had chosen to use for every team and product internally. Louis (a) created a custom inventory program that simplified greatly the size. From that we started to import software, bits by bits, inside this new location. Starting with the configuration, then programs, then more..

In a single repository, we had the source code of the services. Encompassing at least five different programming languages, deployment code, build code, quality test code, inventory code. Everything was in there. We even built our PXE boot images in that repository. (1)

Every single component was there; it made it very easy to synchronize everything. The limits of this system did not reveal themselves to me. Later I even included the legacy storage code and its associated idiosyncrasies in it. The purpose was to unify as much as we could.

It had two outcomes. First removing roadblocks to push to production. Essentially the new system reduced the cognitive load to enact changes on the live systems. The second was on the velocity side. Because we customized, dare I say, twisted, ansible to do our bidding we had a very tightly coupled system that did exactly what we wanted it to do. Adding a rack? Easy. Adding a region? Easy. Adding a service? Easy. The advantage of doing this compounded over time.

Many things had simply not been possible because they were too expensive to do. Now, the future was wide open for us. We could now do these impossible things. I only understood the power of what we had done much later. I only understood it when I got exposed several years later to other companies and their own deployment systems.

By that time, I was also changing orbits. I was moving into my own place.

(a) Louis Solofrizzo, Storage DevOps at the time, still at Scaleway

If you have missed it, you can read the previous episode here​

To pair with :

• Dog shelter - Burial
• EAFNOSUPPORT