Autumn 2019​
​

We had our first tech loan recalled in November 2019. One of the limitations of OpenIO was the number of objects in a bucket. They had recommended not putting more than 500 000 objects in a single bucket. We had documented this publicly. It was a recommendation but, of course, we didn’t limit the number of objects for customers. The limitation was there because after the 500 000 object threshold, the underlying metadata systems started to hit edge cases.

What happened was twofold. First behind the scenes, we used two buckets. One of the objects and one for the object parts. S3 enables you the option to split your objects into multiple parts. A 50 gigabyte object is, in reality, 1,001 objects. One contains the metadata and the other 1000 are the data parts. So if you store 500 of these objects, the parts bucket reaches 500 000. As a user there is no way you can know. This is where we started to see issues. Customers that got hit by that limitation were creating more objects than the recommended number.

The second aspect was that we couldn’t tell them to alter their systems. If a customer’s flow stored millions of objects in a single bucket because it worked at AWS, who were we to tell them to split the flow into multiple buckets and do accounting on the number of objects per bucket.

We started to have more and more customers having this problem. The builtin self healing mechanism didn’t work at this scale so we had to fix the metadata ourselves. We first developed a hand procedure that we followed by hand to fix the buckets. A mix of shell commands. Then, over time, Louis (a) added automation, statistics and monitoring. We started from connecting to the machines, manually copying the database to having an integrated command line interface (CLI) that would synchronize metadata on demand. That CLI ended up being a tool we used for everything; rolling deployments, migrations and more.

I remember the team wanting to add automation for the customer support so that we wouldn’t have to handle the re-synchronization escalation ourselves. I pushed back. Initially we didn’t have enough experience on this very delicate procedure to fully understand what could go wrong. For a few months, the customer support team escalated the tickets to us and then we would apply, manually or later, automatically the fix. From memory, we only added automation for them at least six months in. If we botched anything, that would mean customer data loss. It was not worth taking any risk by trying to go too fast.

It was the start of a difficult period. Keeping those issues was not sustainable. Object storage worked for almost every customer. Except for a few, some of them were very vocal about their issues, for good reasons. We needed a long term plan. We started to research what we would do next.

​
Yet, I was getting tired, but one thing kept me going. The team.

(a) Louis Solofrizzo, Storage DevOps at the time, still at Scaleway

If you have missed it, you can read the previous episode here​

To pair with :

• Blood rain - Ital tek
• EBADFD