Thinking Parallel

A Blog on Parallel Programming and Concurrency by Michael Suess

Ten Questions with William Gropp about Parallel Programming and MPI

William GroppThis is the second post in my Interviewing the Parallel Programming Idols-Series. Today I am talking with William Gropp (who I am allowed to call Bill :P). Bill had a major role in the creation of the MPI standard and is co-author of one of the most influential MPI-implementations to date (MPICH). He presently works at the Argonne National Laboratory as a Senior Computer Scientist. Bill has also been named an ACM Fellow in 2006. And let’s not forget he also writes books, among them the definite reference for MPI: MPI: The Complete Reference. Never program an MPI-program without that book on your desk :P. All this makes him the perfect candidate for a Parallel Programming Idol!

But enough with the preface, let’s start with the interview. The first five questions are about parallel programming in general:

Michael: As we are entering the many-core era, do you think parallel computing is finally going to be embraced by the mainstream? Or is this just another phase and soon the only people interested in parallel programming will be the members of the high performance community (once again) ?

Bill: Clearly, everyone will be using small-scale parallelism. But only people interested in performance will *ever* program parallel machines – after all, performance (which may include efficient use of memory) is what the parallelism is for. (I’m using performance in the most general sense – using separate threads to avoid blocking is on a GUI is an example that isn’t HPC). Many people will end up using parallel computing, but only because their tools and building blocks are parallel.

Michael: From time to time a heated discussion evolves on the net regarding the issue of whether shared-memory programming or message passing is the superior way to do parallel programming. What is your opinion on this?

Bill: Both have their strengths and weaknesses. Message-passing puts a burden on the programmer to manage their distributed data structures. Shared-memory requires great care to avoid race conditions and performance bugs (such as false sharing). Neither is a great and complete programming model, and until we get past the heated discussions and address the deficiencies of both, we’ll keep having those heated discussions.

Michael: From your point of view, what are the most exciting developments /innovations regarding parallel programming going on presently or during the last few years?

Bill:

  1. Global view languages are addressing some of the real issues, but they aren’t enough. They are encouraging interest in parallel languages, which is great.
  2. Over 128,000 processor machines and demonstrated sustained performance over 200 TF. Scalability and performance!
  3. Transactional memory, even if it isn’t a success, may help push thinking away from locks and toward better tools for the appropriate abstractions.
  4. The DARPA HPCS program, for tackling the issue of productivity, not just performance.

Michael: Where do you see the future of parallel programming? Are there any “silver bullets” on the horizon?

Bill: All you have to do is look at the relatively poor performance of single-processor dense-matrix-matrix multiply when compiled with most Fortran compilers to see how far we are from achieving portable performance for code that is cache-friendly on single processors. (And if your compiler does well on matrix-matrix multiply, what about the other BLAS3 operation?). We still don’t have the silver bullet for uniprocessors.

Having said that, there have been great successes using libraries that encapsulate the operations needed in that problem domain.

Michael: One of the most pressing issues presently appears to be that parallel programming is still harder and less productive than its sequential counterpart. Is there any way to change this in your opinion?

Bill: Use libraries. If you don’t have a good library, find (and support!) someone to build it. If you can’t use a library, build a tool to help you with the programming. And good design can often reduce the extra burden of parallelism – there are good, prize-winning parallel applications that use libraries for almost all of their parallelism, and rely on MPI to provide the infrastructure for those libraries. That’s a good model to follow; a code that has thousands of MPI calls is probably not well-designed.

That was an interesting perspective. Now let’s move on to the second part of the interview, where we will be talking about MPI:

Michael: What are the specific strengths and weaknesses of MPI as compared to other parallel programming systems? Where would you like it to improve?

Bill: The biggest weakness in MPI is what it doesn’t address at all – support for distributed data structures. The second biggest weakness is the library overhead, which makes it hard to achieve low overhead on small messages. The first of these can be address by developing suitable tools; the second is more fundamental.

There are many strengths; see my paper Learning from the Success of MPI. One under-appreciated strength is that you can write a deterministic algorithm in MPI in a way that you can prove is deterministic – that is, you can avoid all of those hard-to-debug race conditions that plague other (equally low-level) parallel programming models. The other is parallel I/O – MPI has a well- designed, powerful, and precise model for parallel I/O. It is a good building block creating libraries for community file formats such as HDF5 and netCDF.

Michael: If you could start again from scratch in designing MPI, what would you do differently?

Bill: The RMA operations (single-sided put/get) in MPI-2 were designed to meet the needs of the fastest machines of the time, but the times have changed and a different design is possible now. I’d also look at finding a way of meeting compiler writers half-way to address the library overhead issue I raised above.

Michael: Are there any specific tools you would like to recommend to people who want to program in MPI? IDEs? Editors? Debuggers? Profilers? Correctness Tools? Any others?

Bill: There are a number of good performance debugging tools. We include some with our implementation; there are others such as Tau and Vampir that are well-known. The MPI Profiling interface has made it easy to develop tools for MPI. There is also some encouraging work going on in adding MPI-smarts to the Eclipse environment.

Michael: Please share a little advice on how to get started for programmers new to MPI! How would you start? Any books? Tutorials? Resources on the internet? And where is the best place to ask questions about it?

Bill: Two books that I’ve coauthored, Using MPI, and Using MPI 2, were written to help programmers get started; they use an example-oriented approach. There are other good books as well. One of the things to remember is that MPI doesn’t provide any tools for defining distributed data structures, so the first step is to understand how to develop parallel applications that do not share memory.

Many meetings include MPI tutorials. Supercomputing has several each year. That’s often the best way to get started, as you can ask questions about anything that’s unclear, and tutorials are designed to get you started quickly. There are many online resources; many supercomputing centers have developed MPI programming courses and often make them available.

Michael: What is the worst mistake you have ever encountered in an MPI program?

Bill: I could cheat and say that it was:

< View plain text >
c

  1. n = n + 1

within an OpenMP parallel do; because it was inside of an MPI program, the user filed a bug report against our MPI implementation.

There are a lot of errors; we even include a page of common errors in our tutorials. But my vote for worst mistake is one like this:

< View plain text >
c

  1. if (rank + 1 < size) MPI_Send( ..., rank+1, 0, MPI_COMM_WORLD );
  2. if (rank > 1) MPI_Recv( ..., rank-1, 0, MPI_COMM_WORLD );

This gets my vote for worst mistake because the code works but (for large enough messages) serializes the communication. The worst bugs are always the ones that you don’t see but hurt you none the less.

Michael: Thank you very much for the interview!

By: Michael Suess | 2007/03/29 | Interviews - Parallel Programming | Trackback | Comments [RSS 2.0]

2 Responses to Ten Questions with William Gropp about Parallel Programming and MPI


Comments