Splitting Light: Season 1 - Episode 4


Splitting light

Season 1 Episode 4

The fall of the issue

If you are no longer interested in the newsletter, please unsubscribe

One month turned into two months. After writing an especially weird piece of code that had turned my brain into mush. I looked at the blade, an assembly of metal, components and PCB layers. I was getting tired. I pushed it in hoping this was going to be the end but the LEDs stayed red. My hope sank. There was a sense of despair.

This was blocking the manufacturing order. The support engineer had a call with me. We zoomed out of the issue. I explained to him the big picture. How we had connected the chips together. He asked for some more details. He eventually told me that what we wanted to do was not possible.

We had interpreted the high level chip documentation wrongly when the board had been designed. We both brainstormed to find a solution. Every suggestion was not doable without changing the PCB and the months of qualification afterwards. This, in turn, would delay the manufacturing and the launch of the cloud platform.

We ended our call, I put down the phone and turned to the side to debrief with Greg. I explained the assumptions and went through all the suggestions, one by one, to make sure I had not forgotten something.

He interrupted me and told me that one of the solutions might actually be doable. We had a secondary network for telemetry and we would hijack part of it to accomplish our goal. I started making a list of all the things that were required to be changed.

I fired up my code editor and started doing the edits. For each edit I would quickly test that it was working as expected. One part of the solution was intimidating because it required writing a device driver. By looking at the operating system code, I was able to write the code.

After a few days, I had all the elements in hand. Every software that needed to be modified was ready. I upgraded the software everywhere, pushed in blade by blade.


All the LEDs were green. The last was one in my hands. I slid it in. I watched the LEDs. I anxiously watched the diagnostics scrolling on my laptop. The LED turned orange. I stopped breathing and my heart stopped. A few seconds later it turned green. It worked. It worked!

I was able to fix the problem. I told Greg, he sent the manufacturing order. We were finally going to be able to move on. Over the next week, I tested many combinations and edge cases to make sure we handled every possible situation.

The dopamin hit was very good. I was proud that I had been able to navigate this issue. It was my trial by fire. It forced me to go through every part of the project. It would set the tone of what the next few years were going to be.

A large portion of the work we did was custom. No documentation existed on the internet. If someone else on the planet had the same issue, they would certainly not ask a question on a forum. We were by ourselves. We had to figure out almost everything by ourselves. This was the way.

If you have missed it, you can read the previous episode here

To pair with :

  • Louder - Jose Amnesia, Jennifer Rene
  • A fire upon the deep by Vernor Vinge

Vincent Auclair

Connect with me on your favorite network!

Oud metha, Dubai, Dubai 00000
Unsubscribe · Preferences

Symbol Sled

Business, tech, and life by a nerd. New every Tuesday: Splitting Light: The Prism of Growth and Discovery.

Read more from Symbol Sled

Splitting light Season 2 Episode 23 Beat the cluster to a pulp If you are no longer interested in the newsletter, please unsubscribe With proper observability we could now push the cluster even further. This was the final set of tests that we would perform before wiping everything and going to beta after a new setup. We huddled and concocted a strategy. Picked up our tools and went on the field to beat the cluster to a pulp one last time. Our goal was explicitly to overwhelm the cluster as...

Splitting light Season 2 Episode 22 Too many logs If you are no longer interested in the newsletter, please unsubscribe I’ve rarely seen people talk about this effect. The effect being the amplification of requests. This effect can overwhelm your system. We had to deal with it. The object storage, at least OpenIO, was a collection of distributed services. You might call them micro services if you want. That had implications. When a request comes in, from the user perspective, it’s a single...

Splitting light Season 2 Episode 21 All nighter If you are no longer interested in the newsletter, please unsubscribe As we were moving forward, in mid June 2018, we hit a point where we needed to be able to check the logs of the cluster as a whole. The way we had done it until then was manually connecting to the machines and opening the right files to look inside. This was no longer viable. One of the main office rooms (1) Scaleway’s monitoring team had done a metric stack which we already...