Summary
This is a set of chapters for young engineers. We give practical advice and discuss goals, challenges and approaches used in modern software engineering.
Along with classical foundations this article contains original ideas of conceptualizing engineer's work with emphasis on bringing order to the situation and finding an insight. Engineering is approached as work in uncertainty with other people which requires special skills. Non-obvious complications regarding modern production in big companies are discussed.
This article is based on 15 years of experience in engineering and management in high-tech industries.
Contents
What is Engineering?
Questioning
Messaging
Communication
Iterations
Modelling
XY-problem and Experience
Traits of Good Engineer
Traits of Good Manager
Promotion of Changes and Ideas
What is Engineering?
Engineers create or support systems which help people in their life.
Engineering tasks are ill-posed problems from mathematical standpoint. In general case you may lack knowledge, if there is a solution; you may have multiple possible solutions; and small changes in problem statement may cause drastic consequences. Important addition is that in most cases you may not evaluate your solution in terms of proximity to the ideal one.
In these obstacles your task is not to find an ideal solution, but to find a good compromise taking limitations into account.
In many cases you may not even be sure what exactly you need to achieve, what problem to solve.
It makes engineering a discipline where you need to make many decisions subjectively.
This is very different from classical educational courses when you are given with a well-posed problem, for which you already know what formula to use. This complicates work of students in engineering. Previously they had a map and they had a vehicle to use and a road. Now they are in a desert/a jungle with some random items here and there and a task to “make it work” or to “fix it”.
Of course, usually managers try to soften the adaptation for interns by making the problem definition, a map, and a path a responsibility of senior teammates. However, this does not change the need of acquiring a new skill – move sanely in uncertainty.
Proper methodology of adding a value into an unclear situation is the foundation of a good engineering.
Engineering historically is about application of some ideas or (scientific) knowledge to real-life situations. Engineers identify a real-life situation that requires attention, create a model of it and then try to work with the model to help people in the situation.
We will consider engineering methodology aspects in the next chapters.
Questioning
It seems that a wheel was developed from rolling logs. Logs, rolling under a cargo (e.g. a boat), required additional effort to move them back in front of the cargo. At some point stoppers were used, so the log does not move out from below the cargo. Later the middle part of the log was made thinner to reduce weight. Later the thinner part became a separate part – an axis. Later the axis was made independent in rotation from the wheel, so the vehicle can rotate, as it requires different rotation speed of the opposite wheels.
As you see from the story above the most problematic part was to admit that something can and should be changed. One can ask proper questions to himself and to others and try to understand the situation better (see the Socratic method).
So, engineering starts from asking questions.
Don’t know what question to start from? You might need to be present in the situation for some time, wait. Mentally touch one thing, another, and somehow your brain will adapt and will find a thread to pull, will distinguish a theme and a rheme (discussed in the next chapter). This simple approach of being present presenting in the topic works in most situations very well.
You may also start from the basic questions. What we have here? How is it different from what we saw previously? What can we do with that? What we would like to have? After some time, you ask another type of questions. Are we going into the right direction? Is our solution sufficient? What can be wrong with it?
Number of such questions is numerous, and the more you have them the higher your quality bar is.
The quality bar IS what questions you ask yourself and how frank and precise you are while answering them. Can my solution be used improperly? How can I ensure proper usage? Was I wrong from the start?
You may also say that engineering is an art of asking proper questions.
Usually your questions are based on your values and your internal driver. What you want to see in result of your work? Something nontrivial? Something beautiful, precise, reliable? Or you just want the working day to end faster?
Imagine you have a subtle feeling that everything may fail in a rare case because of this unclear part in a legacy code you look at. Always conduct sufficient investigation to figure out if it is the case. Wishful thinking is the worst possible thing you want to happen with you in engineering. Reliability is based not on wishes, but on throughout analysis and objective principles. The Murphy’s Law is very practical – “If something may go wrong, it will”, such attitude should be a standard. Think against your profit and your will to defend a good result.
Messaging
So how the questions can be made more efficient? And – how to help others in questioning about result of your work?
Cognition may be partially described with help of linguistic terms theme and rheme.
“There is a book on the table.”
Here “the table” is a theme – a topic/a setting of a deliverable thought, “a book” is a rheme – what we talk about in particular in the setting.
Since engineering tries to operate with complex objects and topics, it is necessary to produce messages which are easier to understand. Every time we listen to/read a message, we subconsciously try to identify a theme, and after that a rheme. So, the message should help to do it faster and more precise.
“Hi colleagues! А book on gasdynamics from my table has suddenly started levitating. What should I do?”
The message has the following structure:
1. A recipient is named – good, I may filter out what is not for me immediately.
2. The book is a theme, it goes next, so I can understand the setting right away.
3. The sudden levitation is a rheme, good, I may understand the problem in the setting.
4. A question about the rheme, so the message to be interpreted as a call to action, not just a FYI notification.
Can we improve the message?
“Hi colleagues! I need your immediate help in an extraordinary situation. А book on gasdynamics from my table has suddenly started levitating. What should I do?”
Here we have added a small summary of the story we are going to tell with the message. The summary also includes the urgency explicitly.
Can we improve the message?
“Hi colleagues! I think I have a hallucination. I see a book levitating in front of me. This is not a joke and please come to my desk now, if possible, I am very worried.”
We have omitted a story summary because the story is short. We removed all unnecessary details and included what exactly we want from people. We also made the story more trustable by including some sane interpretation.
Also, it is better to say this in person to any of the surrounding peers, and not to send an email or a message to a chat.
As you see, we added a new level of consideration on top of the theme and rheme level. Instead of transmitting of our own internal language and thought flow we crafted a story which is more likely to lead to a positive outcome.
It required a special internal effort of thought on interpretation and adaptation. Message is good when it contains such work.
It is a good habit to test your texts and questions based on these concepts. You can check this text to see if it conforms to what it calls for.
Communication
So far, most has been said about the factual containment of the message. We touched on delivery a bit, but only in terms of being adequate to recipients’ context.
But communication has its own value to us, irrespectively of a local occasion. We communicate not only to share interpretations, but to build a relationship, to cooperate, to support and to feel support, and to express and understand ourselves among the rest.
This means that constructive communication is based on peace. So, first delivery and containment have to support it.
These principles based on the New York’s police negotiator speech can help:
Seek first to understand.
Make sure you understand correctly.
Know when to deliver your message.
Deliver using suitable language.
How you say it makes the difference.
Respect unconditionally and expect good intentions by default.
For this you need to decode others’ situation to correctly interpret complications and impressions they might have.
The iceberg model originated from anthropologist Edward T. Hall’s “Beyond Culture” (1976) shows what complicates communication between people from different cultures.
Per Hall, two interacting cultures are like two icebergs, with only a part being visible to both sides. We may see behavior, partially can see beliefs, but values and thought patterns stay hidden, while impacting the behavior the most.
Hall say that we may overcome the gap by active participation in other’s culture. Fast interpretations in this situation are usually wrong.
This might be applied to any communication, especially in cases when communication is stuck.
When we see objections from a person, we might start thinking about the underwater part. The goal is to make the part visible – convert it to known.
Simple example – if a person is tired, he can averse any active plans for now by referring to obviously unlikely risks. This might raise frustration or anger in others. However, if this will be understood and pronounced – that if a person is tired it is a legit reason to postpone the discussion, then there will be no any basement for frustration from both sides. We will discuss this more in the “Promotion of changes or ideas” chapter.
Iterations
Basic engineering process is iterative. You may not know how to reach the goal right away, but you may check surroundings with a cane to find another piece of soil to stand on in a fog.
On each new step you open hidden parts of a map. This is called experimentation. Then you decide on the next step based on analysis of the new information. This is called synthesis. At some point you may find, that on the actual map you are, the better practical solution would be different from the originally expected.
You may also say that analysis is extraction of important properties based on the experimentation results, while synthesis is creation of a model with needed properties. After synthesis you test its results.
Usually physical experiments are most expensive and time consuming. Computational experiments are in general cheaper than physical. In software engineering, however, there is not much distinction. When you run your program on a computer using some digital data you may call it a physical experiment for your program. Instead, there is a distinction between smaller artificial tests and real heavy load.
Experimentation is a strategy of running some system in an environment to get knowledge about the interaction of the environment and your system.
Speculative (thought) experimentation is different. You ask questions and find answers by modelling the future and corner cases in your head. Depending on engineer’s skill this approach may be very powerful.
You may also think of speculative experiment as a fast rolling in the above scheme in mind.
Usually a mixture of the approaches is used. Good balance between speculative modelling and real runs come with experience. At first steps in engineering the speculative approach skill is not developed enough and physical experiments are overused.
For example, imagine a failed test in a software production process. Trying to change the code here and there and see what happens leads to loads of failed tries, and usually makes the situation worse, since fixing in such manner may hide the underlying problem and make it much harder to detect subsequently, since formally the test is not failing anymore.
The iterative approach is the basic one because it gives the result faster. It is such efficient because it reflects the core properties of the working situation: we may not take everything into account at once, and, at the same time, experiments require post- and pre-thinking.
Modelling
The iterative process includes creation of model of reality which incorporates sufficient properties of it. Such model is used for working on an abstract algorithm of your prototype behavior.
Our brain is unable to work effectively with more than about 5-7 entities at the same time. So, an engineer tries to limit number of parameters considered at once. There are multiple ways of doing this.
Hierarchy is what can make the job. While traversing the graph of cases you can limit cognitive load by considering only a node or few. You may go deeper and deeper in a branch if needed, because previous nodes (more general cases or classes) in the branch are formulated and documented, so you may easily get back to them.
The picture above is an example of documentation, which is another method of reduction of cognitive load in a research moment. Pictures are great for that since they can be visually memorized easier and may be used later as an entry point for details in recalling process. So colorful catchy pictures fit serious engineers’ work very well.
The same idea is used in dialogic modelling sessions. You sit together in a room with one or two colleagues and share statements or questions on the topic. A statement or question leads to a thought work and after some time another statement or question is pronounced. This is like moving up on a ladder. Each new statement in a common space of investigation allows everyone to see a new part of the map.
All significant statements are written in a simple linear list as they come. These minutes are the result of the session. After the session one may analyze them with a fresh look and create a graph of branches of modelling cases.
Modelling implies procedures of abstraction, specification, differentiation, classification, systematization. All this can be described as work with formalism. Engineer build formal systems, so he needs to operate in formal spaces fluently. What is the relation between the given entities? IsA? HasA? What is the difference between the two synonyms? Are these classes overlap? Is there a strict precedence between the two events? Do we have a correlation or a causality between the two samples?
Metaphors are another tool. Metaphors help you to grasp complex phenomena such that you may reduce number of entities you work with, while preserving their core properties. “A map”, “a foggy swamp” are examples of metaphors I use in this article.
Finding meanings of entities and naming them accordingly is another way of revealing an order, thus making chaotic picture more approachable for analysis. One needs to create a consistent vocabulary for a model based on it to be consistent. You may read more about naming and its connection to design in this article.
XY-problem and Experience
Most difficulties in the process come from two things – problem statement and previous cases you worked on.
Problem statement you usually receive is in most cases a hidden XY-problem. Someone experiences X, either consciously or not, and feels that Y is needed in the situation. Your task is to get to the X and understand what you may do with it. You may call it exploration of context. The more you know about the context the more adequate your solution can potentially be.
Research shows that more experienced professionals spend more time on initial analysis. Why? At first glance they should use experience to understand what to do fast. Contrary to this, experience shows that the more details and specifics you find out in the beginning, the more time you save in future.
When you approach a task, it is better to avoid starting from application of a set of picklocks you have in your pocket. Try to think of the problem in its own space, in its own terms, not in terms of your set of picklocks. Only after identification of uniqueness of the problem you may try a combination of previously used solutions if they fit.
Inexperienced people frequently try to apply some known algorithm to a new problem straight away. We call it hammering nails with a microscope. And they do it fast. Sometimes they successfully report achievement and move to a similar case. By doing this they shape their professional future in a way that don’t imply creative, challenging, and complex work.
A task is absolutely regular? It might happen. Can you afford yourself a subtle touch of unexpected brilliance in some aspect of your work on it? Such attitude change things.
Nowadays a solution you create need to be scalable, so it can be used (possibly indirectly) by many, many people. Otherwise your work may not be payed sufficiently. Such solutions need to be very qualitative and polished. This requires deep analysis and creativeness, not fast nails hammering with a microscope.
Traits of Good Engineer
When we know what process engineers are involved in, we may think of what traits they are expected to show to be productive. And to refine vision of what they do.
As you remember, at first you need to understand what to do. This requires soft skills for understanding people (requestors), delivering your questions, setting up relations with all stakeholders. No, this is not another guy’s task. Even if you have a separate team to compile a functional specification for your task, there is a communication with the team.
So, a misanthropic geek who knows what to do by himself is not an ideal engineer.
But also, this initial step requires good abstract thinking to set up a formal model of reality you work with. Abstract math, applied math, applied physics, logics, philosophy, poetry are what may help.
Nowadays most important and complex systems are built by many people. An engineer is a team player. Team player means you can help others, you are able to describe something vividly to preserve focus of people, be adequate in conflicts, and so on.
Ask yourself a question – what do I engineer and what I don’t, but I should? Find things around you and in your work, which are not addressed seriously with analysis, quality assurance, planning, retrospectives, etc. This distinguishes a good engineer from a mediocre one – he cares about all aspects which influence his work.
In a big company or in a big industry, where usually complex systems are developed, there is an inherent problem. Many levels of management, high specialization of labor and distribution of production and consumption chains blurs the overall picture. One man in his position sees only local context. That reduces possibilities of global optimization extremely. And this is what you may use as a natural hidden resource for finding innovative ideas.
A good engineer thinks of a broader context of his task – in team workflow, in product, in adjacent products, in customer company and so on to optimize his work, his plans and his product.
In this sense a good engineer behaves like an owner of his product/project/module, because there is a high chance that nobody else will do that for his product/project/module – everybody have their own local context to care about.
An owner thinks about future of his system, about possible caveats, about public relations regarding his system. Because he cares about it.
Ownership is more about mindset and base focus, less about skills and knowledge, so you may exercise it from the start.
Do not limit your questions by technical ones, because success of your work depends on technical aspects only partially.
An average worker is assumed to do the same thing he did previously, and no big professional growth is actually expected by management in big companies. It is faster to hire a more skillful guy for a more complex project than to convert a newbie into an expert and lose him to a competitor. Most tasks are aimed at supporting stable operational flow.
A good engineer understands that quality is something which may make his work stand out. And he sets high quality bar for what he does. This is a constant resource of ideas and of progress in skills, because higher expectations require better skills.
You need to do more and better than expected to evolve.
This will give you more challenging assignments from your management, because otherwise you will leave. This may be done by extra work hours on early stages. But each such hour will be basement of your future.
Overall, a good engineer brings value in every context he is in, because the culture behind his approach influence technology, processes and people in a good way. And this does not require knowledge or expertise for exercising.
The value a good engineer brings in is two-fold: order and insight. Order is what you create in the technology, in a fuzzy problem statement, in a dead end, in process. Insight is a finding of something missing before, which might change the direction of development or just resolve current issue. Nontrivial information is more important nowadays than just preserving the process since the process won’t last long without it due to rapid change of environment.
Thirdly, personal experience and flair add something unique to the work. You do something which others (in your environment) won’t do. You need something others may don’t care about. You want something others avoid. This is valuable. Creativity is personal. You may not avoid making your work a personal, intimate field of your life if you want to be creative.
All of this can be summed up as taking care of an undertaking which is worth it.
Taking care is different from following your own star and following your wishes. If the deal is not worth sacrificing your wishes than maybe you are not doing the right thing.
It is also different from winning a competition because success in competition is a value outside of what your product brings to people’s lives.
“Different” does not mean you should forget about your wishes or lose competition. However, this sets the hierarchy of values.
When we talk of “becoming the best” we imply competition. But instead you should do good and proper things where you are, with people you are with, with past you have, and with resources you have – for the benefit of everyone. You need not to stress on that you are better than someone else, but stress on that you are currently doing the best you can do. Others might not have the same background, help, environment you have, or you had, so there is no basement to compare and to be proud of something you achieved. You achieved something because of place you were in, because of your parents, because of your teachers, because of books you read which was written by other people, because of good attitude to you from others. This is a perspective in which you see more and in proper light.
The basement of a good motivation is not being proud of yourself but that you do a proper thing (the thing which is worth doing).
Traits of Good Manager
So how is manager different from his subordinate engineer?
Mainly by the object of his work. Per the Georgy Shchedrovitsky’s methodological study management is activity over activity.
One cannot manage a standing chair; he may only transform it or move it. However, one may manage a thrown chair by changing its movement because he predicted the trajectory. It is easier to turn a horse currently running in the opposite direction, than to push a dead horse in the right one.
Real situation is a bit more complex: you have plenty moving entities which have their own internal drivers, may change direction, and interact. Manager grasps situations of work using thinking and predicts the movement to shape it.
If you are a manager than your subordinates’ wellbeing, destiny is partially your duty, your responsibility. Their talents realization is your duty. You are not just another guy which is responsible for another technical part of the production. As manager you need to have a will to take care of people and of the common.
This is also connected with your managerial abilities – if you do not know self-movement of people in your organization, or it is absent, then you may not manage.
Everything which helps to predict helps to manage. Criticism from subordinates? Great! New information, including info about their self-drivers. This is very similar to the market research, but it is coming to you by itself.
Manager works on vision and integrity.
Manager create a common space of meanings to preserve integrity in actions and in understanding of the team. It includes vocabulary, values, mindset, methodology, process, treatment of others, culture; including alignment between personal, team and company goals.
What we distinguish in the context? What we treat important? Who we are? Where we go, why? You may assess concrete decisions based on that.
Manager sets role model. Culture starts from his actions.
Manager focuses on culture, not on ad-hoc solutions. Because culture is scalable. When a manager works on ad-hoc solution with his subordinate he needs to focus on how efficiently he establishes a culture to deal with such problems, so next time it may happen that his involvement is not needed.
If a manager does not engineer perspective of integral movement, culture his team shares; if he does not invest in finding meanings of phenomena his team faces, then he is not doing his primary job. Usually such manager stays in borders of administration of external requests and distribution of them among his team members. Operational flow is his main focus, which he handles by a sequence of ad-hoc solutions.
Promotion of Changes and Ideas
Usually we talk about promotion when we touch context, which extends to other people’s situations. Since we do not touch or change it every day, usually our first couple of tries to describe it ends up with either an overly simplified picture, or a picture which is further from the reality than the simplistic one.
Ideas and actions which are not fitting the time and the context may lead to the opposite and impact future possibility to change something.
So, before promotion of an idea you need to do preparatory work.
When you try to change something first try to answer why it is not there now. Most certainly there is a good systematic reason for this, which is not just a will of some person. Most probably the system you are currently in is in a state of potential well, it means it is rigid to changes. Inertia and averse to the risk are the other side of stability.
What benefits the situation you want to change gives to people? Will the change give them more? Do they have energy to change or a clear path how?
Another practical method of approaching inertia is to create anonymous polls with straight questions, which will reveal unspoken or unpleasant facts.
What if someone does not agree with your proposal even if you think you provided a very sane analysis? Let us remember the two icebergs model from the Communication chapter. Logics does not work if you do not figure out the hidden part. Very commonly what you think is good is not the same the other guy thinks is good.
Frequent hidden reasons of objections are:
I am not yet convinced in your arguments.
I still don’t have enough information, which I need.
I still cannot rely on you to sufficient extent.
I am still not feeling that you helped me.
Try to figure our what is the case and address that. You will see, that in most cases it is hard. For example, you will not be able to help the person just because you are not in the formal position to do it.
Such iterative and careful investigation of the people, the system and the processes will in most cases give you completely new picture of where you are, what are real roles of people, what are real goals of the system.
This new picture will give you understanding of practical steps toward your original goal or it will change the goal. Anyway, such undertaking is worth doing, since the original proposed change might occur much less interesting than the reality.