This is the second post in my Interviewing the Parallel Programming Idols-Series. Today I am talking with William Gropp (who I am allowed to call Bill :P). Bill had a major role in the creation of the MPI standard and is co-author of one of the most influential MPI-implementations to date (MPICH). He presently works at the Argonne National Laboratory as a Senior Computer Scientist. Bill has also been named an ACM Fellow in 2006. And let’s not forget he also writes books, among them the definite reference for MPI: MPI: The Complete Reference. Never program an MPI-program without that book on your desk :P. All this makes him the perfect candidate for a Parallel Programming Idol!
But enough with the preface, let’s start with the interview. The first five questions are about parallel programming in general:
Michael: As we are entering the many-core era, do you think parallel computing is finally going to be embraced by the mainstream? Or is this just another phase and soon the only people interested in parallel programming will be the members of the high performance community (once again) ?
Michael: From time to time a heated discussion evolves on the net regarding the issue of whether shared-memory programming or message passing is the superior way to do parallel programming. What is your opinion on this?
Michael: From your point of view, what are the most exciting developments /innovations regarding parallel programming going on presently or during the last few years?
- Global view languages are addressing some of the real issues, but they aren’t enough. They are encouraging interest in parallel languages, which is great.
- Over 128,000 processor machines and demonstrated sustained performance over 200 TF. Scalability and performance!
- Transactional memory, even if it isn’t a success, may help push thinking away from locks and toward better tools for the appropriate abstractions.
- The DARPA HPCS program, for tackling the issue of productivity, not just performance.
Michael: Where do you see the future of parallel programming? Are there any “silver bullets” on the horizon?
Having said that, there have been great successes using libraries that encapsulate the operations needed in that problem domain.
Michael: One of the most pressing issues presently appears to be that parallel programming is still harder and less productive than its sequential counterpart. Is there any way to change this in your opinion?
That was an interesting perspective. Now let’s move on to the second part of the interview, where we will be talking about MPI:
Michael: What are the specific strengths and weaknesses of MPI as compared to other parallel programming systems? Where would you like it to improve?
There are many strengths; see my paper Learning from the Success of MPI. One under-appreciated strength is that you can write a deterministic algorithm in MPI in a way that you can prove is deterministic – that is, you can avoid all of those hard-to-debug race conditions that plague other (equally low-level) parallel programming models. The other is parallel I/O – MPI has a well- designed, powerful, and precise model for parallel I/O. It is a good building block creating libraries for community file formats such as HDF5 and netCDF.
Michael: If you could start again from scratch in designing MPI, what would you do differently?
Michael: Are there any specific tools you would like to recommend to people who want to program in MPI? IDEs? Editors? Debuggers? Profilers? Correctness Tools? Any others?
Michael: Please share a little advice on how to get started for programmers new to MPI! How would you start? Any books? Tutorials? Resources on the internet? And where is the best place to ask questions about it?
Many meetings include MPI tutorials. Supercomputing has several each year. That’s often the best way to get started, as you can ask questions about anything that’s unclear, and tutorials are designed to get you started quickly. There are many online resources; many supercomputing centers have developed MPI programming courses and often make them available.
Michael: What is the worst mistake you have ever encountered in an MPI program?
- n = n + 1
within an OpenMP parallel do; because it was inside of an MPI program, the user filed a bug report against our MPI implementation.
There are a lot of errors; we even include a page of common errors in our tutorials. But my vote for worst mistake is one like this:
- if (rank + 1 < size) MPI_Send( ..., rank+1, 0, MPI_COMM_WORLD );
- if (rank > 1) MPI_Recv( ..., rank-1, 0, MPI_COMM_WORLD );
This gets my vote for worst mistake because the code works but (for large enough messages) serializes the communication. The worst bugs are always the ones that you don’t see but hurt you none the less.
Michael: Thank you very much for the interview!