Object storage was our first priority. Many existing and future products depended on having an object storage product. We looked at the state of the art. There were a few competing protocols. Amazon S3 was the oldest but there was also Blackblaze B2, Openstack SWIFT, Google GCS and lastly Azure Blob storage. When we looked into how object storage was used, where it was used and why, we discovered that many tools had S3 and SWIFT support and after that, it dropped sharply.

Object storage is something that is ubiquitous now. In essence it is a very “simple” system where you can send, fetch and delete blobs of data in a bucket. That bucket is made to look infinite in size which reduces the cognitive weight of using it. You just send the data. The operator handles the rest.

The details of how you interact is the pickle. How you connect, how you authentify, how you push and pull the data, this is the implementation logic. The protocol, the way in which you interact with the product.

My experience in building Carbon14, and working on hardware in general, was extremely handy. I knew the incredible power of using standards. I knew how important plug and play was. We could have done an engineer’s dream, building a shiny new protocol, but that would have been just an ego trip.

Few people realize how much work it is to implement a new protocol or API. Not only is there work to design and implement the protocol itself, but there is a lot more effort in doing the SDK, the tooling, the documentation, the marketing, the customer support and all that has to be done before even convincing engineers at customers that they have to add code and refactor code instead of just changing a setting.

I did not want to go that way. I didn’t want to spend my time doing what was already done by other companies much better staffed and financed. S3 had a much higher presence in the market. It was the default, the standard. If we would use S3, we could leverage the ecosystem, we could leverage all the stack overflow questions and answers, we could leverage AWS against itself. By choosing to be fully compatible with S3, we reduced the number of things for us to accomplish to only building a compatible system. A drop-in replacement.

​
​

This is the direction I took and that the team agreed to. The great thing about the way things were organized back then was that each team could do what it thought was best. We didn’t have to do things because someone had decided so. We chose. We defended. We assumed. We were fiercely independent.

But, could we implement a S3 compatible product ?

​

(1) Photos available here by Emmanuel Caillé​

​

If you have missed it, you can read the previous episode here​

​

To pair with :

• Ashes in the snow - MONO
• Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems by Martin Kleppmann