March 2019

After we had sent our first bills to customers, less than a year and a half after the pivot, we were preparing for a new Object Storage region.

DC5 had finally come online, our racks of hardware had been installed there. We were doing final adjustments on the installation scripts and deployment steps. However we had one issue.

At AWS, a bucket name is a world wide unique identifier. This has interesting properties such as being able to have a unique cross region endpoint that redirects you to the right region. This was a new challenge for us. Because we chose to be as much compliant as we could, we needed to ensure the unicity of the bucket names across regions.

OpenIO did not have a builtin mechanism for this. So we added one. Precisely, we added a new metadata layer that used CockroachDB as a worldwide distributed and synchronized database. Then we added a two phase step to ensure the uniqueness of the bucket name.

Quentin (a) did the service to make the bridge between OpenIO and CockroachDB. I did the deployment steps and the hooks into the OpenIO code.

While this was being completed and finalized to go to production, we enabled the fr-par region for customers. A few days later, after the final deployment, we announced via unofficial channels that the region was in general availability.

We still had a few kinks. One of the network links on a load balancer was off. There was a weird compatibility issues with the router and we didn’t fix it before launch.

We had not gotten approval to advertise the region. We had prioritized the launch. After the block storage storm, we didn’t want to wait and be blamed for not pushing to production.

The customers started to push requests to the fr-par region. The amount of data stored started growing. We continued to check that there was no issue.

Putting Paris into production without proper approval did rack some nerves. We got a lighting strike back. I pushed back with force and arguments. We were moving forward. Later, I understood that this was a mistake. I should have handled the situation with less conflict. But I did not know better at the time.

We were mavericks. Acting like mavericks. But we were live with a new product, Object Storage, with two regions. The next one, block storage, was near private beta with multiple AZ. This was in addition to maintaining five existing storage products in production.

Those were the Iena days.

(1) https://lafibre.info/opcore/datacenter-dc3-diliad/

(2) https://x.com/Scaleway/status/1108024039493255174​

(a) Quentin Selle: DevOps engineer then, now System Development Engineer at AWS

If you have missed it, you can read the previous episode here​

To pair with :

• The mountain will fall - DJ Shadow
• Mir Hardware Heritage by David S.F. Portree