Software engineers suffer from Dunning-Kruger - do you too? — dateo. software
Ever heard about the Dunning-Kruger effect? Let me share my thoughts on it and how I think that all software engineers suffer directly or indirectly from its symptoms and consequences.
There are patterns in our day-to-day life, in our relationships, in our jobs, that are worthwhile to be aware of. For example, if I were a person to constantly overreact to other people's opinions, it would be good to be aware of this, so that I could try to own it, maybe even change my behaviour around those kinds of triggers. One of the patterns that are very obvious in the community of software engineers and developers is the so called Dunning-Kruger effect, which you may have heard of already. If not, let me introduce you. The Dunning-Kruger effectLet's first have a look at the definitionWikipediais giving us:The Dunning–Kruger effect is a cognitive bias whereby people with low ability, expertise, or experience regarding a certain type of task or area of knowledge tend to overestimate their ability or knowledge. Some researchers also include in their definition the opposite effect for high performers: their tendency to underestimate their skills. So, we are dealing with a cognitive bias or, in other words, amisconceptionabout our own capabilities. The symptoms of this effect are said to be found in various fields such as politics, driving, medicine - it is not something that directly relates to the software industry but rather to all areas, whereskillsare required to do something, and where peopleestimatingtheir own skill is somehow involved. The different stagesThe Dunning-Kruger effect typically unfolds through different stages for which the independent variable is the "wisdom" orknowledgearound a certain topic. To illustrate this, I am borrowing a figure byUlf Ehlert, 2015As you can see, as someone, a person, gains knowledge or wisdom around a certain topic or field of expertise, theirconfidenceundergoes different stages with highs and lows. Know-nothingThe earliest stage is the start of any learning process, where you actuallyknowthat you don't know anything. I am sure you felt like this at some point in your career, whether it being when starting a completely new programming language, a new framework, in a new team, or anything similar. You don't know anything and you are also not very confident in accomplishing anything productive yet. So you set about learning and practising as much as possible. Mt. StupidSome (short) amount of time has passed, you read many tutorials and debugged all hello-world samples, found answers to the most trivial questions on stack overflow, you even built a button into that sample thatchanges its colour when hovering over it on your own, or maybe you outer-joined the person and the address tables using raw SQL. So that's it, right? Conquering the world has become possible. What problem could possibly be out there, that you werenotable to solve with your newly acquired skill set?(See yourself in there somewhere already? I certainly do. )Valley of DespairThe descent from Mt. Stupid can be initiated by various triggers. Could be you are assigned a real-world problem in that domain, or maybe someone asks you a question for which in-depth knowledge is required. Or maybe you simple continue to learn and about the thing you started with. In any case, a sudden (or sometimes not so sudden) realization kicks in: Actually, you still don't knowanythingprofound yet. There is no way you could satisfy a customer with what you know so far, no way to actually develop something useful and make money from it, no way you would survive an in-depth code review with one of your senior developers. But the good news is: You are nowawareof this. And you can make your situation even better by publicly admitting that you still have much to learn here. Personally, I find that the best people to work with are those that are open to admit that theydon't know everything. FutureEvery despair grows hope - and if you dare to pursue, the only way forward is upward again. So you continue to learn, experiment, discuss with peers, ask questions and slowly regain the confidence you had when you actually didn't know much - except this time, you really do know something already. Keep going, and you are on a path to become an expert in something. Ill-placed Complexity in SoftwareI am pretty sure that if you've read this far, you could in part either relate to the things I spoke about, or at least think about some situation or even a colleague that would fit into this schema. I want to extrapolate the effects of Dunning-Kruger one step further to the domain of software engineering. To me, it is not a coincidence that many memes and comics around software development feature a similar overall shape to the diagram above. Take, for example, this figure:Again, you might have seen this before and you'll probably have smiled at the inherent truth around it, but, actually, it issadlytrue. This graph depicts the more general progress of us software developers during our careers and it is highly correlated to the implications of the Dunning-Kruger effect. Know-nothingYou start out as a newbie in your field, barely managing to get even the simplest things done. People assign you only with the tasks they think you're capable of accomplishing - your code is simple, because you don't know how to do otherwise. Mt. StupidYou are learning from your peers, you took courses, read books and memorized all 273. 5 design patterns by heart. You are ready to conquer the world with your skills. Also, your code is starting to get more and more complex. "That's how the experts do it!", one might here you say. Or,"But we might need that later. "Maybe even"But wouldn't it be cool if. . . ", which is one of the gravest mistakes we can make according toBilly Hollis. In fact, your code a gained a level of abstraction and complexity that, yes, it might be totally generic, but everyone else needs a few days until he understands it, your colleagues are afraid to make changes because it causes compilation errors in dozens of places and it is abstract to such an extent that no one knows the actual implementation until the start the application in debugging mode. Damn it. You aresuch an expert. RecoveryWe need to make a small detour from the phases of Dunning-Kruger. Usually, in case of the pattern that I just described, there is no single thing to trigger the descent into the valley of despair. Instead, people slowly start to realize, that their code is too complex, too hard to understand. How? Well, experience, for one. But hopefully there are more senior engineers around and people willing to speak up and demanding code that everyone could maintain. If this criticism is well-meant and positive, it will spark a slow progress of realization. One cannot simply start writing simpler code. That is a craft in itself to be practised and honed and it'll take time. In the end, the good programmer will write simple code again. This code is not simple because of lack of knowledge though. It is feature-rich, based on design patterns, but it iseasy to understand and work with. And is something that we all should aim for in the long term. RoundupTo be honest, I think that the majority of us go through phases of the Dunning-Kruger effect regularly. And if you are, like me, a software engineer, I claim that you are going through the stages of ill-placed complexity as well. And also:That's totally fine. Sometimes, there is just no way to avoid it, we are human after all (not true for ChatGPT which is a KitchenAid taking human nonsense as input, stuck on Mt. Stupid). I am only advocating for more awareness around the things we don't know. If you are willing to question your own decisions, guidelines, your own code, then you are on the right path to becoming agooddeveloper.