As we learned more and more about the existing products and how to maintain them online, we learned about different components and how many spare parts we had in stock.

A question you could ask is why did we keep these products online? We could’ve pulled a Google and just killed them all to only have the products we were expected to build. Except, we cared about the customers and, most importantly, these products brought in a significant amount of money. For quite a few, the hardware had already been paid off or was near paid off. It was almost “free” cash. So we kept them alive. Except for two of them.

One was the RPN-Rsync. It was a platform where customers would upload their data using rsync. It’s the last one we picked up and it was one of the most obscure ones for us. It used a specific hardware raid card and by the time we had learned how to replace hard drives in it, it was in a feverish state. By replacing the dead disks, we set in motion the rebuild that killed the RAID. We tried everything we could but there was nothing we could do. We lost the data.

After communicating with the customer support team to inform them and the customers, we looked at the other ones that were at risk. Another product was using similar old hardware, we decided it was best to sunset it on our terms rather than lose the data because of a catastrophic failure. In addition, replacement parts were out of stock from the vendor. We notified customer support, disabled new clients and set a deadline a few months in the future to end the service.

We followed up with clients that needed help to migrate, we monitored the service to see if there was still data being accessed and notified the related customers. Then progressively, we shut the network for one day, and caught a few more customers. Then it was cut for one week, one or two more customers and then one month with no network and finally we powered off the machine.

Had we sunsetted everything right away to only focus on the new products we would not have cashed in a lot of experience that would be critical later on. We had learned that hardware RAID was not an option. We learned that components had a statistical dying rate over time. We learned how these products had been set up, most importantly the diskless boot system that we would use everywhere and enhance a lot.

Even though we learned how to manage these and prepared the new products, everything felt hazy personally. Somehow my brain was fogged up.

​

(1) Photos available here by Emmanuel Caillé​

​

If you have missed it, you can read the previous episode here​

​

To pair with :

• Doom box - DJ Qbert
• A Chinese Life by Li Kunwu