I point out in the intro to Technical Debt that development has changed over the years. A core theme of the change is that computers used to be expensive and humans were cheap, and then this relationship reversed where now humans are expensive, and computers are cheap. (I’ll lovingly ignore the easy jokes about sky-high cloud vendor bills)
There are many reasons for change, but I’m going to zoom in on one specific aspect: abstractions. Abstractions (should) hide the details of complex interfaces, and take advantage of cheaper hardware to help engineers produce more business value.
The effects of this highlight a general theorem. Being able to do more with less code is good, because less code = less time and less code = less bugs. These equivalences aren’t universally true, but they’re mostly true.
Abstractions also help manage complexity, because instead of having to think about a million different machine instructions, you can think of a high-level capability like “get a user’s name from a database”.
You can visualize these abstractions as a hierarchy with machine code as the lowest level, followed by assembly language, followed by languages like C, and then higher-level languages like Java, Ruby, Python, and then followed by frameworks, libraries, and other DSL (domain-specific languages) equivalents.
As a side note, I think the existence and benefits of abstraction are universal. You could consider “specialization of labor” as a form of abstraction. Companies use specialization of labor when they hire employees, or hire vendors. There’s an equivalent “specialization” in biology, where high-level organisms like humans rely on microbes for life. The concept of composing complex things out of simple things is universal, but we’ve seen especially rapid change in software because it is easy to abstract ideas, and programming is (essentially) about managing ideas/information, and ideas can change faster than the physical world.
Abstractable Infrastructure
We’ve seen and demonstrated the practicality and value of abstraction with software, because creating abstractions for ideas is easy to do. As software eats the world, we see the same patterns when it comes to cloud infrastructure — infrastructure as code (IaC). It used to be mandatory that you dealt with low-level details for managing infrastructure like racking servers and installing software. Now we have interfaces and abstractions that help us accomplish the same goals with a few minutes of typing commands.
This is why I’m massively excited about Crossplane. I’ll ignore the smart way that it’s implemented on top of Kubernetes (so we get the benefits of gitops, standard resource definitions, reconciliation loops, and scalability), and just focus on it as a component of the developer platform.
As long as we’ve been able to write code to configure infrastructure, we’ve been writing abstractions on top of infrastructure, usually in the form of semi-structured bash scripts. So, the developer platform layer of abstraction on top of IaC has existed for a while.
The classic example of a capability of a developer platform is installing a database. Normally you’d have to think about the low-level steps: provision a server, install the software, and apply the configuration for your use case. It is better to have a “create database” capability that sets things up the way your team needs it done.
The developer platform layer is important. Even if we ignore the time savings, there is a complexity aspect: it’s not practical to expect a product engineer to know the details of infrastructure management. It’s far better to give reduce complexity by giving a product engineer an abstraction for a database instead of making them understand all the underlying details. By removing that burden from product teams, they can deliver business value faster, with higher quality.
But Crossplane isn’t just an abstraction. Crossplane is a tool to create new abstractions. This gives us a standard to easily create, understand, and maintain the developer platform abstractions, in contrast to the “semi-structured” nature of bash or similar automation where a new engineer needs to understand a unique implementation every time.
Crossplane brings ease of use and manageability to the developer platform layer.
Final thoughts
Nothing that I write here should imply that I don’t believe in the idea of full stack engineering, because I do. It’s important for engineers to understand the full stack, much as any domain expert should understand related domains. But there’s a difference between being able to understand a component and being able to produce that component. Producing something requires a deep understanding of the nuances of that domain, and it is impractical to expect people to stay current on the details of more than a certain number of domains.
This does highlight one of the subtler value propositions of Crossplane. Because it is a standard way to implement abstractions, it becomes easier for engineers at other levels of the stack to read the code and understand how their developer platform works.
More broadly, as I write this in 2023, everyone is speculating on how generative AI is going to change software development. Although some core dynamics are going to change in our industry, even as AI gets better at producing code, AI will need human oversight for the foreseeable future. Good interfaces and abstractions will continue to be required so that humans can partner with the AI systems and review their code.