Business, tech, and life by a nerd. New every Tuesday: Splitting Light: The Prism of Growth and Discovery.
Share
Splitting Light: Season 1 - Episode 17
Published 11 months ago • 3 min read
Splitting light
Season 1 Episode 17
Mission Statement
If you are no longer interested in the newsletter, please unsubscribe
Our job in the lab was to create new hardware and make it work. Then both dedicated team and cloud team would send an order depending on what they expected to sell, and we would support both throughout the lifetime of the hardware. This was our mission. Keeping this in mind was important. It was important because we could often get distracted from it.
The important components of the mission
Some of our tools were written in old languages. We used old software tracking systems, and we used old version tracking software. There was this itch to update them. To redo them in a new shiny language or use a shiny new tool. Sure maybe it would be fun; however, that was not our mission. Our mission was not to redo the tools that we used. If the tool became a constraint, we would change it but it was only a tool. The shiny was the chips or the shiny was how they were interconnected. It was not using a newer language or a language with a lot of hype to redo the flashing tool.
We did not have shiny macs or powerful Linux stations. Most of our tools worked only on Windows. Sure we could change tools, sure we could maybe use open source or contribute to these tools, but, in the end, what was the added value for the team and the company to do that? The work was very time and cost intensive with multiple colliding timeframes. Yes, we used Windows, yes we coded in a putty terminal, yes the laptop was bulky and ugly but that did not change the work we did.
Scaleway tried to get traction on hacker news as well, but it didn't work that much. I don't know if it was the way the communication was done or if hackers werenotinterested. It's also hard to tell the interest in technology. I could try to spend time arguing online and hyping up interest. What would it bring in the end? It would not make the hardware work by itself.
At first, it was hard to resist those urges. But as time went by, it became easier. The sheer volume of work to make the hardware work crushed any time we had to do these kinds of things.
Filter out webcomics, HN, and ShinyNewLanguage
This did not make the hardware we produced less useful. We knew what we were doing, and we had internal validation because we directly talked to our customers. The whole idea was to reduce the cost and simplify the workflow of other teams. The cost we could calculate. On the other hand, the workflow we could not know. We had to talk to people, ask questions and fit in their model.
The most user friendly features did not come from me or the hardware team. They came by observing how the hardware was used, how they handled other hardware and trying to fit our model to make life easier for them. The netlink feature and the ipmitool feature saved many hours for them because it fitted directly in their information system. It was plug and play.
To do support, we had to write a lot of documentation. How everything was wired from an outsider perspective, workflows to put into production, binary artifacts for multiple distributions and procedures to diagnose issues. We also collected different bug reports and fixed issues with new releases.
It was a very important part of the job. We might have ideas on how hardware should be but didn't handle hundreds of servers or routers on a day to day basis. We could improve density, but it would only be useful if the people installing these could manage the increased density and make a profit out of it. If the system was too complicated to use, it did not make sense to add so much effort into producing it.
To pair with :
Don't Know - Sd Laika
Hackers: Heroes of the Computer Revolution by Steven Levy
If you have missed it, you can read the previous episode here
Splitting light Season 2 Episode 23 Beat the cluster to a pulp If you are no longer interested in the newsletter, please unsubscribe With proper observability we could now push the cluster even further. This was the final set of tests that we would perform before wiping everything and going to beta after a new setup. We huddled and concocted a strategy. Picked up our tools and went on the field to beat the cluster to a pulp one last time. Our goal was explicitly to overwhelm the cluster as...
Splitting light Season 2 Episode 22 Too many logs If you are no longer interested in the newsletter, please unsubscribe I’ve rarely seen people talk about this effect. The effect being the amplification of requests. This effect can overwhelm your system. We had to deal with it. The object storage, at least OpenIO, was a collection of distributed services. You might call them micro services if you want. That had implications. When a request comes in, from the user perspective, it’s a single...
Splitting light Season 2 Episode 21 All nighter If you are no longer interested in the newsletter, please unsubscribe As we were moving forward, in mid June 2018, we hit a point where we needed to be able to check the logs of the cluster as a whole. The way we had done it until then was manually connecting to the machines and opening the right files to look inside. This was no longer viable. One of the main office rooms (1) Scaleway’s monitoring team had done a metric stack which we already...