What Makes Parallel Programming Hard?
Anwar Ghuloum has posted his opinion about What Makes Parallel Programming Hard at the Intel Research Blogs (which are buzzing with activity with regards to parallel programming at the moment). The funny thing is, I have asked myself the same question last week, because I have just written a section in my PhD.-thesis about it. And since at least some of my answers are different from his, I have decided to post them here.
But I don’t want to get ahead of myself, as a good host of course I need to let the guest speak first. Here are Anwar’s reasons in a nutshell:
- Finding The Parallelism
- Avoiding The Bugs
- Tuning Performance
- Future Proofing
- Using Modern Programming Methods
These are all good reasons. Here are the reasons I have found, some of them overlap with Anwar’s, some of them are new. They are in no particular order, some are more important than others, but I am sure not the one to judge which one is the most important one in your particular situation or for most programmers.
Added Complexity
There are a number of steps involved in creating a parallel algorithm (e.g. task decomposition, mapping and taking care of communication / synchronization issues). None of these steps are necessary for a sequential program. For most of them, there is no support from tools available. The only thing that really helps is experience. Naturally, these additional and difficult steps are the number one reason why parallel programming is considered difficult.
Parallel Programming is Error-Prone
Not only are there additional steps involved when creating parallel programs, but these steps are very prone to errors. If the wrong task-decomposition is chosen, you might not see any performance increases from parallel programming (I will have a post about task-decomposition next week). If the wrong mapping is chosen, synchronization overhead may kill performance. And if mistakes are made during the communication / synchronization phase, not only performance may suffer, but the program might not run or produce incorrect results altogether.
Too Little Knowledge of New, Innovative Parallel Programming Systems
There are some parallel programming systems out there that promise to be easier to use. OpenMP is the primary example, but also e.g. functional ones like Erlang or Haskell. The problem here is that most programmers don’t even know about these systems or their strengths and weaknesses.
Parallel Programming is not yet Mainstream
A lot of information can be found about mainstream languages in books or on the Internet, where a lot of tutorials or solutions to common problems can be found. Programming is easier, when there is a lot of help available. The situation is different for most parallel programming systems. There are some books available, but most of them concentrate on numerical algorithms and problems found in High Performance Computing. The number of resources on the Internet talking about parallel programming is also quite thin, when compared to the more general topic of programming and software development.
Compilers Are Not as Well-Tested as their Sequential Counterparts
Working with a compiler that is not well-tested can be a huge problem. A programmer can spend large amounts of time searching for mistakes in his code, when really the compiler is to blame. In my experience, many compilers for parallel code are less well-tested than their sequential counterparts.
No Good Platform for Parallel Programming Available
There are a lot of tools available today to help programmers of sequential programs. IDEs, Debuggers, Profilers, Correctness Tools are all important to make programming easier and together with the abstractions provided by the language and the libraries available they form a so-called platform. For the parallel case, there are tools available for some systems, but they are usually not as good as their sequential counterparts.
Not Enough Powerful Abstractions Available
Programming in assembler is more difficult than programming in a higher-level language. The languages used for parallel programming are often on a very low level, as each communication or synchronization operation needs to be managed in every detail by the programmer. OpenMP is a good step in raising the level of abstraction, as it hides many details from the programmer.
A second aspect here is the lack of good libraries. Programming becomes way easier, when the programmer can rely on powerful libraries to encapsulate complex behavior. There are some numerical libraries available for many parallel programming systems, yet others are largely missing.
Lack of Standards / Portability
I have described earlier that parallel programming systems are not yet mainstream. Related to this point, there are very little standards available, that programmers could rely on, especially when changing platforms. OpenMP and Java Threads are a big win in this regard, since they enable portable parallel programming, even including the Microsoft Windows platform.
It Is difficult, to Move from a Sequential Program to a Parallel One
In most cases, parallelizing a sequential program is not an incremental process. For many systems, the programmer has to start refactoring his code for parallelism and until that is completely done, there is no way to check any intermediate steps. This is of course difficult. OpenMP has a good approach to this problem, as it is quite easy to add pragmas one at a time without having to refactor the whole program at once.
It is Hard to Test Parallel Programs
Related to the last paragraph, testing parallel programs is a problem. Most errors can be classified into one of the two categories: there are problems related to parallelism and there are problems with the actual algorithm. Unfortunately, finding a mistake is difficult when the culprit could be in any of the two sources. OpenMP once again gets it right, because it enables programmers to turn of parallelism for many programs and still get a correct, sequential one, that can be debugged for errors using the advance sequential debugging tools available.
Disclaimer
Some of them you may already know from my post on the Top 5 Reasons why I Hate Parallel Programming (Sometimes). You can see from a lot of these points, that I am using them to justify basing my research on OpenMP. This shows that of course not all points are valid for each and every system, there is e.g. a lot of learning material available on Java Threads now, both on the net and in good books. And there has never been a lack of good IDEs for that particular language. But they are all I could think of and I am convinced most of them are true, at least for a large subset of the available parallel programming systems.