The Zen of Parallel Programming: The Big I and the Global Sum - smolnero

Skip to content

As I continue to trek my way through An Introduction to Parallel Programming, I find myself seeing how parallelism is not only a priority in our communication with hardware, but also in our communication with one another and, perhaps more importantly, with ourselves. I am still learning the technical extent of the subject, but I cannot help noticing how often its problems resemble our own: we may possess enormous amounts of power, intelligence, memory, and information, yet remain limited by our inability to coordinate what is already available to us.

The textbook explains that most programs written for conventional single-core systems cannot automatically make use of multiple cores. We may have more processors available, but the original program was not designed to coordinate them. If a game is running slowly, opening eight copies of it does not give us one faster game with more realistic graphics. We have multiplied the number of programs, but we have not transformed the structure of the program itself.

Before I send myself in circles trying to understand the full extent of this, I want to focus on the distinction that feels most important. More people do not automatically create better cooperation, just as more information does not automatically create understanding. More effort does not always create progress, and more power does not automatically produce a system capable of using that power. In both hardware and human beings, additional capacity means very little when the structure was never designed to coordinate it.

This leads me toward the idea that we cannot always translate the “old self.” We often try to transform ourselves one behavior at a time while leaving the larger arrangement untouched. We try to sleep better, communicate better, become more productive, control our anxiety, or respond differently to the people around us, but we may never stop to question the structure producing those behaviors. We preserve the same identity, assumptions, expectations, and attachment to how things have always been done, while hoping a few improved habits will somehow produce an entirely different life.

Parallel computing encounters a similar limitation. Researchers have attempted to create translation programs capable of converting serial programs into parallel ones, but with limited success. A translation program may recognize certain operations and divide them among several processors, yet the result may still be inefficient. Each individual step may have been parallelized successfully while the program as a whole remains poorly coordinated. The original structure survives inside the new program, carrying its old limitations into a system that now possesses far more power.

The textbook’s deeper point is that the best parallel implementation may not come from translating every serial step into a parallel equivalent. Sometimes the programmer must step away from the original sequence and devise an entirely new algorithm.<br>This is where the human connection becomes difficult for me to ignore. How often do we try to transform ourselves by translating an old life one behavior at a time? We add discipline where honesty may be required, productivity where rest may be required, and control where communication may be required. We attempt to make ourselves more efficient inside structures that are already exhausting us. The problem may not be that we lack the power to change, but that the different parts of us are not allowed to communicate truthfully enough to participate in that change.

Honesty, then, may be one of the keys to parallelism between human beings and within the individual self. Without honest communication, each part operates from incomplete information. The mind may produce one conclusion while the body communicates another. Our emotions may recognize something that our speech refuses to acknowledge, while memory continues influencing the system beneath our immediate awareness. Every part is performing its own calculation, but the results are not being shared.

Perhaps some parts of ourselves cannot simply be converted from serial to parallel because the original structure depends upon one part remaining in control of all the others. Sometimes the structure itself has to change.

my_sum

The textbook demonstrates this through the act of adding a collection of values. In a serial program, one processor computes each value and adds it to a running total, one after another:

sum = sum + next_value

With multiple cores, the work can be divided. Each core receives a portion of the values and calculates its own partial sum. The textbook calls this private variable my_sum, and I find the name stupidly philosophical.

my_sum is my result, my work, my contribution, and my experience of the problem. Each core possesses a real result, but only a partial one. No individual core can see the entire computation from its...