Splitting light

Season 2 Episode 37

Hardware made redundant

If you are no longer interested in the newsletter, please unsubscribe

Around May 2019

In May 2019, the hardware lab’s C3 was in production. It was the third generation compute hardware. The last one I had worked on before switching to storage. There was an issue with the compute node. How it had been fitted with RAM and storage created a voltage drop in certain conditions and that would shut the node down. A capacitor had to be soldered by the hardware team on-site to fix the issue. But how did that issue come in the first place?

The nodes had been procured with a configuration that had not been tested in the lab. Even though since the C2 in 2015 we, the lab, had requested that every deployed configuration be tested in the lab before procurement and deployment. This step, for some reason, had been skipped this time.

Around May 2019, I had initiated the discussion to start the process to create custom storage hardware. Théo (a) and Greg (b) were defining the requirements. It was now appropriate since we had gained a lot of experience and could focus on what would really make a difference. What would bring leverage.

C1 servers without radiator: codename Pimouss (1)

It was a surprise when one day a message was posted on slack. It shocked everyone. For me, it shocked me to the core. It nearly broke me. It was suddenly announced that Scalway would no longer be doing custom hardware. The hardware team and the lab was declared redundant. We would immediately stop the R&D and no longer have custom hardware.

I correlated the shutdown of the lab to the capacitor issue. But who was at fault? Who had created the bug? Was it the hardware team that had not installed that capacitor? Or the person who made the procurement order on components that had not been tested?

Since we had done custom hardware, the instruction had always been given to test before procurement. The lab had to approve the configuration. Yet for some reason, that step had been skipped.

First generation network router: codename Choucou (3)

After confusion came anger, I could not find any logical reason why such a decision had been taken. But then, none of us were in the meeting where that decision was taken. Making the hardware team redundant would create a very long lasting impact. We, in the storage team, would feel it deeply. Some things were only possible at the hardware level.

When Scaleway would start having to do hardware again they would have to start from scratch again. That time would come, I was 100% certain of that. Almost every single hyperscaler does custom hardware. The vendor relations would have to be rebuilt. The experience re-acquired. The processes streamlined again. Everything would have to be redone. It was such a waste. Scaleway did start doing hardware 3 years later and expanded later on.

For me, personally the impact was very hard. All that metaphorical blood and sweat. All those years spent, all that effort invested, it was for nothing. There was no legacy. I thought the lab was this single differentiator. The one thing that made us different from other small cloud providers. Our pride. Just like Free, the telecom operator, builds its own hardware, we did it ourselves and it brought us business value. Or so I thought. I would later understand that it did have a legacy, just not the one I thought.

Racks of cold storage: codename MrFreeze (5)

After the hardware team was suddenly disbanded, they were told to find another product to work on. After a few weeks they found a product to work on, they built the IoT product line. Gregoire continued to work on heavy R&D within Scaleway and built many other products.

For me, the anger eventually led to sadness. I can now see in me the legacy of the hardware team as well as the legacy of this distressing event. From that event, I learned to always prove the value of what I built with business numbers and focused a lot more on potential issues before they happened. The legacy of my work, I find it everywhere with me. The knowledge, the work process, the resourcefulness, the resilience… This is the legacy. The legacy was not bundles of metal, silicon and plastic. The legacy is what was etched in my mental circuits.