Top-down, bottom up

I haven’t written in a while! Let’s hope this casual, fun entry spurs my love of writing again.

It’s folly to think you have everything figured out without experience, isn’t it? As a kid, I remember rolling my eyes many times when an adult advised me from their “experience”, only for me to disregard them and plunge head-first into whatever bad decision had me running like an Irish Setter chasing a tennis ball.

Now I am that adult.

It’s well known that having experience of the world is valuable when understanding it. A famous debate exists in philosophy: on one side the empiricists, who believe you gain true understanding of the world through experience. On the other side, the rationalists, who believe understanding originates from first principles and pure logical deduction. I’m simplifying - but the ending is that both are important.

In this article I want to go out to bat for empiricism.

The empiricists won through science and engineering. Experimentation is crucial to the latter, a field to which I belong. Working as a computer programmer, I am miffed at how often people forget the importance of experimentation in a project. The leaders and managers will produce a grand plan that will see two or three teams coordinate to deliver a new product. They set dates, line up the inputs and outputs, brief they staff, and fire the people-machine up.

But halfway through, teams start realising that things don’t quite fit together as expected. What looked so sure on paper needs tweaking - and even the smallest misunderstanding can cause the system to break down. Fingers get pointed, passive-aggressive messages abound, and deadlines get missed. But eventually we scrape out something that works. Benefiting from hindsight - we agree we won’t make the same mistake.

And then the process repeats itself.

Most projects are too complicated to set out precisely on a plan. Planning prevents piss-poor performance, one cliché goes. And even the best laid plans go awry, counters another. The reason is well-known - the real world is too unpredictable to fit in a plan. But if you’re clever, your plan will change with the conditions. You won’t assume everything up front. This is where experimentation comes into play.

A good project allows for experimentation and adaptation. At work, I love when I can get my hands on an early version of the other team’s software which my team depends on it. After trying it out, I can give feedback to them, and get a newer version with bugs fixed. And vice-versa. Rather than starting debates and litigating the requirements, we stop, collaborate and listen. This is basically Agile, if you know it.

In systems thinking, you can take a bottom-up, or top-down approach. The top-down approach usually involves the “grand plan” I hinted at before. You think about the goal you have, break it down into the expected components, and set your system in motion. The smaller components emerge out of the larger components. But you might find that when the final components are done, they don’t work nicely together - or you missed a whole area of the plan.

In a more bottom up approach, you think about your goal, and piece it together by first producing smaller components. Piece by piece, you assemble the components, until it becomes a whole. Experimentation is part of a bottom-up approach. You can make a piece, connect it to another, and see if it works. If it doesn’t work, you’ll try again until your piece works with the others. Eventually you’ll get usable whole out out of the parts.

You never use one approach over the other. If you’re top-down only, you’ll find that the real world puts a stop to your plan. If you’re bottom-up only, you might find the final product is a formless mess. The best approach combines the two. You have structure and ideals from the top down, and work with reality from the bottom up.

Though businesses benefit from bottom-up approaches, they are scared of uncertainty. So leaders push a top-down approach. Bottom-up involves experimentation and change - an anathema to business owners and investors. Businesses are, by nature, hierarchical. It is very hard to reconcile the top-down nature of a large business with the effectiveness of a bottom-up approach. I’m not in a state to look it up now, but I know a wealth of literature exists on the subject.

Outside of business, experimentation is crucial for both children and adults to learn. The science student doesn’t just read textbooks. The teacher usually gets them to perform experiments. The musician doesn’t just play pieces from a sheet, but jams out until he or she has created something that sounds good. Very few people become competent at a skill through rote memorisation. When a learner can ask, what happens when I …?, and then get the answer, then they quickly grow in their abilities.

Some things cannot be an experiment. Raising children will always be an experience from which parent, as well as child, learn from. Even with millennia of ancestors behind us, there exists no solid plan for raising kids. Parents figure it out from experience. But no child wants to be the experiment. But if you have four, like my parents did, you are bound to learn something from the first pair of children which you can apply to the last. I am experiment number three.

Other kinds of projects are very top-down and not really suitable for an experimental, bottom-up approach If you are building a house, you are not going to take a bundle of materials straight to the site, and see what you can put together. Construction is very top-down - with plans being drawn at one end, and built at another. Nonetheless, anyone who works in construction will tell you how they had to change an overzealous architect’s grand idea, once they saw how sandy the soil was, or something like that.

When we allow experimentation in life, we are admitting we don’t know everything. You might not expect the result you get out of an experiment. You might not not like it. Experimentation can be humbling. We all want our projects to be top-down, structured, houses of logic that stand firmly on the ground of reality. But we need to think bottom up as well, get down to the ground, and test how solid that ground is. I could go on with the metaphor - but I won’t. I just want you all to experiment more.