Splitting light

Season 2 Episode 28

Side quests

If you are no longer interested in the newsletter, please unsubscribe

After shipping the hardware to Amsterdam, we quickly launched private and public beta. We were the first product to launch to public beta in November 2018. Database as a service was not far behind.

Image of the Object Storage in public beta! (1)

Théo (a) had instructed the customer success team to forward almost all support tickets to us. We did level 1 (L1) support. Every single issue a customer faced we took care of. We helped the onboarding of customers. We would handback L1 as we grew confident. Until we were comfortable doing level 2 (L2). It was a tradition in the team. We would do many things by hand, for some time, until we were comfortable then we would handover or automate the details.

We took L1, then L2 and lastly L3. That meant at the launch the support was faster because we could answer directly. Afterwards, customer success had more experience and documentation.

We helped customers and internal teams get plugged into object storage. Helping them adjust their code logic and close support if necessary. The team was doing a great job. There were multiple tracks in parallel. The bases of the billing components, the maintenance, the bug fixing and more.

Florian (b) was making good progress on block storage. Because he had prior knowledge of Ceph, he was going fast. By September, he had selected the hardware, done the network and was now testing the performance of the first cluster. We were near private beta on block storage.

I was doing side quests. Some chosen and some imposed.

One of them was trying to build request traces from logs. We had everything in the logs. I thought it would be possible to parse the logs and assemble back traces. I was almost able to do that. But using the API I could not go further. Why not integrate the SDK in the code you ask? Because it was too risky to add code that we didn’t master to a huge codebase we didn’t master either. The next best thing was to parse the logs.

Without being able to add the SDK into the software, the next best thing was to emulate it from logs.

An imposed one, was to inventory every single machine, its rack, and room in the datacenter. I remember that one clearly. I was told you are going to spend a few weeks on that. But I didn’t have time. I bolted together a few hundred lines of python and used several existing discovery mechanisms to populate our machines in three or four days. Then went back to storage team tasks. It was dirty code, almost throwaway but it did the job. I made sure it was accurate before ending the side quest for me.

My job was also to assess if side quests led to benefits or not. We had discussion inside the team whether to use kubernetes to run the cluster. I was pushing back. For me it was a useless burden. Maxim (c) did an interesting move. He forced me to explain in detail why I didn’t want it. I asked first, what was kubernetes used for? Scaling and running services. What did we run the cluster on? A fixed size of servers. We could not scale further than we had servers. What did kube require? An extra set of services just to make it run. The net value for us would be extra work for us. For our context and use case it was not appropriate to use kubernetes.