Here’s a nice puzzle for you: Is there any lesson that we could learn from the Malthusian Trap and apply that to Moore’s Law? Or is that a bridge too far? Well, let’s look at a couple of arguments.
Thomas Malthus
At the very beginning of the 19th century, the political economist Thomas Malthus wrote “An Essay on the Principle of Population“. Based on historic analysis, Malthus concluded that productivity gains in agriculture have not led to higher living standards for all people. Rather, these gains only resulted in a higher number of people living at the same subsistence level. Essentially, the productivity improvements were eaten up by an increasing number of hungry souls, all of them ultimately living in the same misery as before. As a result, for most of human history since the agricultural revolution, the living standard stagnated, as if it would bounce off an invisible barrier – that is what Malthus considered an unescapable trap.
However, in the luxury of today’s hindsight, we can see very clearly that we left that trap behind. The event that helped us break through the barrier was the Industrial Revolution, which developed is full impact only after Malthus had published his ideas. And the driving force behind that revolution was cheap and accessible energy, generated from coal and moderated by steam. As a consequence, the income per capita took off exponentially as the industrialising societies successfully escaped from the Malthusian Trap.
Gordon Moore
In 1965, the chemist and businessman Gordon Moore expressed the expectation that for the next 10 years, the number of components in an integrated circuit (i.e., the hardware computing power) would double every year. Ten years later, he adjusted this postulation to doubling every two years. Today, Moore’s Law is often expressed as the doubling of computer power every 18 months. And even though it is not a law of nature, but “only” a prediction, Moore’s Law has fairly accurately forecast the trajectory of the computer industry over almost half a century.
But if you then look at your personal day-to-day experience over the past years and decades, there seems to be a disconnect. The tremendous advance in hardware power didn’t translate into equally impressive improvements in “user-experienced computing speed”. Examples? I’m still waiting for websites to build up on my screen; okay, websites have fancy layouts, highres images, embedded videos, and don’t get me started on social media integration; but all of that is eating power, and it’s steeling my time. Or take word processing, it still takes ages to send a file to print (yes, I am that old); the software offers millions of templates and helpful gizmos, but the “simple” act of getting the electrons into print-ready form is as time-consuming as ever before. It seems as if the progress in hardware power is exploited to drive ever more complex software that then absorbs the growth in hardware power entirely. It seems that we didn’t speed up our lives, we only enriched our toolbox. Rather than making things faster, we get more things done in the same time, or get them done on smaller devices, or get them done in hyper-connectivity.
A correlation?
Now, is there a correlation between Moore and Malthus? At first glance, the dividing lines are clear: Malthus and Moore are separated by almost two centuries, and they relate to very different fields of the human endeavour.
But looking a bit deeper, both have an unexpected lot in common. Both started out from real progress: Malthus’ observations focused on agricultural productivity, Moore’s on hardware computing power. For both, we can formulate a desired outcome: the case of Malthus that’s improved living standard, for Moore it’s increased computing speed. And in both cases, the real outcome is somewhat surprising, failing to deliver the desired effect: Malthus observed that more people had to endure the same low living standard, while in the case of Moore’s Law, we only get more things done at the same overall speed. In both cases, technological change seems to be going forward, and society of course responds to that. But the response is not really as we might have expected it. Something essential appears to be missing.
For the Malthusian Trap, we know how the story continued: the advent of cheap energy shattered the invisible ceiling of living standard. As a result, since the Industrial Revolution we observe the steady increase of living standards and of population. And that’s where the really interesting question comes up. In general colloquial terms: What can we learn from Malthus that could help us with Moore? Or more specifically: What could be the trigger event that would allow us to accelerate day-to-day computing exponentially, while continuously adding more functions? Could we possibly identify such a trigger in advance and accelerate it? Or do we need to sit and wait for it to reveal itself? Could that be energy again? Or something entirely unexpected?
Lots of questions, no easily available answers. Happy to learn your views …
understanding innovation 
I will get things started…
Let me first summarize my take on the Malthusian “curse” and the solution.
MALTHUSIAN CURSE — Increases in productivity lead to higher living standards which lead to lower mortality, which leads to larger population, which leads to lower marginal productivity, which leads to lower average living standards. Thus the net result is more people but approximately the same living standards.
MALTHUSIAN CURE — We shifted from humans as a resource drain to a resource benefit as per Julian Simon and Unified Growth Theory. Humans can solve problems for each other faster than they create problems due to increased competition for scarce resources. After 1776 or so, we learned how to solve problems faster than we created problems (We also slowed the pace of pop growth due to demographic transition). Increased capitalization of energy was one such meta-solution which powered billions of other solutions. Institutions (freer markets, open access government, the scientific method, etc) and other technologies were other metasolutions. Thus the “problem” more people was converted to the “solution” — more problem solving entities working together more constructively.
MOORE’S CURSE — Additional processing speed leads to larger software, which eats up the increased speed, thus the user experience is not on net faster.
MOORE’S CURE — The take-away from the Malthusian cure is that the increased software can be used or turned against the problem. The software can be aimed back at solving the problem of faster processing speed.
All that said, I don’t actually accept the problem as given. Processing IS FASTER. Incomprehensible faster in the domains which need it. The so called problem is that web pages — which DO load faster than the past — are not loading as fast as you like because the persons paying for that page (the advertisers) have different priorities than you. There is an arms race of advertisers for attention, leading to them capturing more and more of the processing speed of pages you are reading for free. Most other programs are faster, but not just faster — they also gain in capabilities. These are reasonable tradeoffs. Speed isn’t the only thing we need or want.
Feedback appreciated…..
Thanks for sharing your thoughts.
You make a good point regarding website speed and the role of advertising. The ads (and their battles with ad blockers) do eat processing power that I would LIKE to invest in faster website build-up. But on the other hand, as long as I can access that information for free, I must ACCEPT the objectives of those paying for it.
I fully acknowledge the capability gains we have achieved in software, they are without a doubt enormous. My point is that we used up the additional processing power to drive those additional capabilities, with little to no speed gain for the user. Here I meant PERCEIVED speed gain, which is a rather soft concept, I’d agree. For those applications I use more or less daily, I have a hard time observing better speed, while I do certainly see (and appreciate many of) the additional functionalities I get with every single update.
On Unified Growth Theory, I must admit that I have a few challenges. You suggest that the root cause of the Malthusian cure was population growth, while energy, institutions, and technologies were “only” enablers (essential no doubt, but somewhat secondary). Does that then imply we’d need to keep population growing to keep our problems in check? I hope not. But if population growth does not need to be maintained into the future, how could it be the main driving force for humanity’s problem solving capacity?
Taking it from a different angle: in your view, we have outpaced our problems. Does that mean we are just slightly ahead of our problems (which is a frantic race to stay ahead); or does that imply that we are actually pulling away, solving more problems than we create (theoretically leading to the state of “all problems solved)? In the first case, the progress is only in the speed of the race against the problems, while the second case is –I’m afraid– too good to be true.
I fully agree that the competition for scarce resources forces us to find solutions. But that is only looking at the demand side of the equation, it doesn’t say anything about the supply side. And more pressure on the demand does not automatically generate more supply, not in the area of intellectual skills. I’ve addressed that a while ago in Shaping the innovation landscape. Building on Thomas Homer-Dixon’s “Ingenuity Gap”, I am not convinced that the human capacity for problem solving is entirely unconstrained, or that it is bound to solve any kind of problem. Based on that cautious scepticism, I’m rather trying to identify those potential limitations and understand their functioning and effects.
Probably it’ll be difficult to arrive at complete agreement, but there’s a lot to learn from the discussion. So please keep going.
I will gladly keep the discussion going, hoping others jump in. Let me say in advance that I am aware much we are saying is speculative at best. But fun…
“On Unified Growth Theory, I must admit that I have a few challenges. You suggest that the root cause of the Malthusian cure was population growth, while energy, institutions, and technologies were “only” enablers (essential no doubt, but somewhat secondary). Does that then imply we’d need to keep population growing to keep our problems in check? I hope not. But if population growth does not need to be maintained into the future, how could it be the main driving force for humanity’s problem solving capacity?”
A brief blog comment is probably not the perfect medium for explaining something as complex as the Malthusian Cure/ Great Escape/ Great Enrichment / Great Divergence/ Modern Breakthrough or whatever we want to call it. I too have issues with UGT. But to clarify, I certainly did not mean to imply that population increase explains it. Improved network problem solving explains it. But human beings are essential parts of the problem solving networks (nodes?). As the network became more effective at solving problems, the net impact of nodes was slightly more solutions created on net than problems created. The number of nodes in the network increased due to improved connection of nodes (joining larger geographic areas) and greater population, but I completely agree that a population could grow too fast, and that it is possible to improve — even radically improve — a network without growing the number of nodes. You can increase the efficiency of interaction, the “positive sumness” (the tendency of parts of the network to benefit as opposed to harm other parts), the speed or rate of interaction, and so forth.)
“Taking it from a different angle: in your view, we have outpaced our problems. Does that mean we are just slightly ahead of our problems (which is a frantic race to stay ahead); or does that imply that we are actually pulling away, solving more problems than we create (theoretically leading to the state of “all problems solved)? In the first case, the progress is only in the speed of the race against the problems, while the second case is –I’m afraid– too good to be true.”
Yes I believe we have seen on net more problems solved than created over the past 250 years. This shows up in the 30-100 times higher living standards, double lifespan, better health, 7-8 times larger population, more freedom, more equality of opportunity, widespread literacy, better understanding and control of nature and so on.
No I do not believe we are approaching a state of all problems solved. I would suggest that solutions virtually always lead to or create problems. As such, progress is possible, but never inevitable, and the process never ends. The Malthusian curse is just one of many negative feedback loops that show up as “wakes” or externalities of problem solving.
“I fully agree that the competition for scarce resources forces us to find solutions. But that is only looking at the demand side of the equation, it doesn’t say anything about the supply side. And more pressure on the demand does not automatically generate more supply, not in the area of intellectual skills. I’ve addressed that a while ago in Shaping the innovation landscape. Building on Thomas Homer-Dixon’s “Ingenuity Gap”, I am not convinced that the human capacity for problem solving is entirely unconstrained, or that it is bound to solve any kind of problem. Based on that cautious scepticism, I’m rather trying to identify those potential limitations and understand their functioning and effects.”
I do not think I disagree. I certainly agree there are problems which may be unsolvable, at least within any reasonable time frame. I also think it is possible that we create problems or negative externalities faster than we solve them. As above I certainly don’t think this represents the last few centuries, but there is no guarantee it won’t be true going forward — the best we can do is learn from what tends to work and tends to fail and experiment from there. In summary I think progress is rare, special and unlikely. But it is possible.
I think it is critical that we identify “potential limitations and understand their functioning and effects.” But also that we understand in broad conceptual generalities how we overcame these limits in the past (which I read as the nature of this post).
But to be more concrete, this is my best stab at explaining whatever it is we are calling the shift over the past 250 years.
1). Intermediate levels of fragmentation/integration between hundreds of states for a thousand plus years led to a competitive arms race for network effectiveness within states. Some states ended up with the culture/mindset, institutions and technology of extremely effective problem solving. Science, liberal ideology, open access government and markets, coal, steam engines, safe navigation, etc. There was never anything guaranteed about this process. In many other eras or places continued competition led to mutual destruction, or state unification. In Europe it led to competition which in the end was slightly more constructive than destructive.
2). These above ways of thinking, institutions and technologies then self amplified. Markets funded technology and energy capture which fed back into science, which broadened the epistemic base, which then fed back into markets, which fed governments….
3). This enabled the improvements in health, prosperity, freedom, opportunity and knowledge (along with lots of new problems never envisioned before the process began)
4). And none of this is certain going forward, but it is essential to humanity and our future that we not forget what got us this far.
Once again, thanks a lot for sharing. This is very rich and inspiring. Blog comments are really not the ideal channel for this exchange, but I think we make them work quite well.
I see now more clearly where you are coming from, and I fully subscribe to your closing sentence that “none of this is certain going forward, but it is essential to humanity and our future that we not forget what got us this far”. That wraps it up very well: we must understand our roots, and history teaches us a lot. At the same time, the trajectories of the past will simply extrapolate into the future. The concept of path-dependence in complex adaptive systems applies: Yes, we can describe and explain the road we took in the past, because we have the benefit of hindsight. No, we cannot prescribe or predict our road into the future. Of course that does not imply that planning is utterly useless; but it means, as you suggest, there is no guaranteed outcome.
The idea of network problem solving capacity is striking. I didn’t yet come across that expression, and it clearly differs from my usual thought patterns; that makes it even better, as I’m forced to reframe my thinking. I see a lot of relations to my earlier writing on flow patterns (information, energy, material) and their increasing decentralisation. There’s the overarching concept of emerging behaviours in complex adaptive systems. And there’s Stephen Johnson’s work on emergence (mainly focused on ants and humans). So your idea resonates, and it’s something I need to invest some more thought into. I’ve been playing with the idea of emergence for a while, but didn’t yet find the right approach to bring it in. This discussion pushes me to revisit some older ideas and to develop those further. Thank you for the additional content and context. Grateful for this trigger.
On a more personal note: You offer very stimulating ideas, and I imagine those should be of interest to a wider audience. I looked around but couldn’t find you in the blogosphere. Maybe I searched in the wrong places, but if not, then I’d really encourage you to express, expose, explore your thoughts through a blog of your own. Of course I’ll be very happy to continue our discussion on my posts, but that’s driven by my thinking and my way of structuring ideas. Hence I’d really like to see your own thoughts developed in your own train of thought and expressed in your own frame of mind.
I have chosen not to blog, but rather spend my time gathering and refining my thoughts for a future packaged release. I am a bit afraid of solidifying my thinking too soon. Plus I am not willing to put in the time necessary to keep up a regular blog.
To get feedback from others, I do expose my thinking in comments such as this.
I have been attracted to your writing for the obvious reasons that much of it resonates with what I am interested in on network effectiveness, progress, economic prosperity, problem solving and such.
Keep up the great work.
Well understood. I’ve been struggling with a similar challenge.
In the end, it’s the exchange of ideas that matters. And what we have seems to work both ways.
Looking forward to seeing more from you.