Splitting Light: Season 2 - Episode 16


Splitting light

Season 2 Episode 16

Iterative process

If you are no longer interested in the newsletter, please unsubscribe

All throughout these first few months, from December 2017 to May 2018, we did iterations on object storage. This was both necessary to pick the skills and also reduce risks. The work to bring up an object storage is a subset to making a public facing object storage product. In the first case you control everything, in the second, you control much less.

The work was multi dimensional. We had to learn the S3 protocols as well as the client tools. We had to learn how to manage multiple servers in a distributed computing environment. The procurement process, testing components together. The network topology and addressing.

We had to learn what was OpenIO. Configure it, install it and test it. How to authentication and check quotas with the user database. How to bill. Had we gone straight for the final goal without these intermediate steps, it would have been impossible. For each dimension, for each component or step, we quickly tried something then improved. Sometimes, the improvement came much later. It was almost like a skill tree in a game. We unlocked skills as we gained experience points and levels.

The way I see it, each of these components is a dimension. To make the product work, you combine them together. Once combined, mistakes and bugs will cascade and amplify. The larger the number of dimensions, the larger the bug volume is.

Each step required us to do checks. We splitted the work and isolated components to make it possible. During the design phase and after, I made sure that we had no circular dependencies. To make sure we could do a cold start. That we did not depend on a service that depended on S3. If a component broke, I tried to make sure that the blast radius would be as small as possible. More details later on.

A side effect of this methodology was that we had achievements all the time. Every week we accomplished something. It fueled an enormous amount of energy. We stayed late, we worked a lot. It was like a flying wheel and it was going faster and faster. Some of these dimensions we mapped but they were not necessary to do for the beta launch, so we scheduled for later in the roadmap.

Sometimes, however, with an iteration, we had to increase our scope.

(1) Photo by Quentin: https://www.linkedin.com/in/quentinselle/

(2) Taken from a talent calculator on wowhead, https://www.wowhead.com

(3) Photo by Quentin, Ramzy: https://www.linkedin.com/in/ramzib/

If you have missed it, you can read the previous episode here

To pair with :

  • Mbira Spirit - Juj
  • The Hidden Family by Charles Stross

Vincent Auclair

Connect with me on your favorite network!

Oud metha, Dubai, Dubai 00000
Unsubscribe · Preferences

Symbol Sled

Business, tech, and life by a nerd. New every Tuesday: Splitting Light: The Prism of Growth and Discovery.

Read more from Symbol Sled

Splitting light Season 2 Episode 23 Beat the cluster to a pulp If you are no longer interested in the newsletter, please unsubscribe With proper observability we could now push the cluster even further. This was the final set of tests that we would perform before wiping everything and going to beta after a new setup. We huddled and concocted a strategy. Picked up our tools and went on the field to beat the cluster to a pulp one last time. Our goal was explicitly to overwhelm the cluster as...

Splitting light Season 2 Episode 22 Too many logs If you are no longer interested in the newsletter, please unsubscribe I’ve rarely seen people talk about this effect. The effect being the amplification of requests. This effect can overwhelm your system. We had to deal with it. The object storage, at least OpenIO, was a collection of distributed services. You might call them micro services if you want. That had implications. When a request comes in, from the user perspective, it’s a single...

Splitting light Season 2 Episode 21 All nighter If you are no longer interested in the newsletter, please unsubscribe As we were moving forward, in mid June 2018, we hit a point where we needed to be able to check the logs of the cluster as a whole. The way we had done it until then was manually connecting to the machines and opening the right files to look inside. This was no longer viable. One of the main office rooms (1) Scaleway’s monitoring team had done a metric stack which we already...