By November 2018, we had released object storage in public beta. It was now time for block storage. It was ready to be in private beta. The hardware was racked, configuration was done. API was done. From the storage team side it was ready.

You are probably surprised that I haven’t talked about block storage a lot. True. Object storage was a lot more work than block storage. Why? It is mainly because of how the product is exposed to customers.

Object storage is an internet facing service. Block storage is hidden inside the infrastructure. You have to take different paths. The careful planning was on the network, the hardware choices and topology. We didn’t need fancy credentials or quotas. Early on, it was decided that block storage would reside in the Compute API. We just had to expose a simple internal API to the compute team. They would integrate it.

It came as a surprise for us to learn that we were late in delivering block storage. The rumors said that the reason Scaleway did not have a block storage product was because the storage team was late. It was so absurd to us that we just ignored those rumors.

After a few weeks being blamed for no reason, we started to react. We pushed back. On our side we were ready. Only the API integration was missing. It was missing even though we had given the specs a few months before.

We eventually learned that the work for the integration had not been started. They had waited until the product was available to start working on it. It infuriated me. In the hardware lab we prepared work in advance. Sometimes months in advance. It was infuriating because especially In the pure software world, you could mock or simulate the calls. It was so easy to do so.

Nothing moved for weeks. We kept being blamed. Internally we discussed doing many things. One option came up. We talked about doing the compute integration ourselves. Part of the team pushed back on that so we waited.

After a month and a half, we got fed up with waiting. We triggered lightning. We decided we were going to implement it ourselves. So we started diving into the code. The thunder echoed and the rest of the company learned about our initiative.

Very quickly we got an order from higher ups to stop. We had sparked a fire. We didn’t do the integration but we had triggered enough energy to no longer bear the blame for something that was out of our hands. The conclusion of this was, for me, not who to blame. Neither us nor the compute or the kubernetes team were to blame strictly per say.

The root issue was scheduling. You could almost say it was the fault of KPIs and OKRs. The integration was not prioritized on compute side because it would not increase their KPI. It was a side effect of setting clear objectives. You did what you were measured for. Pretty normal situation.

The problem was more in the sequencing of things. That required an external person to the teams to schedule dependencies. Something Scaleway would eventually do two years later.

The lightning that we triggered had the effect of reshuffling the compute team priorities. Block storage entered beta a few months later.

However it had consequences inside and outside the team…

​
​

If you have missed it, you can read the previous episode here​

To pair with :

• Half light - Wilkinson, Tom cane
• Tender: A Cook and His Vegetable Patch by Nigel Slater