Comments on: Thoughts on Larry O’Briens article at devx.com http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/ A Blog on Parallel Programming and Concurrency by Michael Suess Wed, 09 Jul 2008 02:35:59 +0000 http://wordpress.org/?v=2.5 By: Joe http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-30 Joe Mon, 28 Aug 2006 18:30:57 +0000 http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-30 Hi Christopher: I like the idea of measuring the multi core systems as compared to postulating the overall impact. We did some experiments and wrote some white papers on what we found for "normal" threaded apps using OpenMP, pthreads, and other models on a single node of dual Opteron 275 more than a year ago. While there was contention, as you could see if you looked at the papers, this was not a major issue. It does not mean that it isn't a major issue for some codes. Any code which fills the memory pipeline (or any other shared resource) is likely to have serious contention issues. What I was measuring was the performance of these codes built for parallel execution on a parallel machine using realistic data sets. You can pull the PDF's off my company home page (<a href="http://www.scalableinformatics.com" rel="nofollow">http://www.scalableinformatics.com</a>) towards the bottom. We are working on expanding the testing. Quad cores will, I believe, have a more severe contention issue, and we are trying to develop tests which allow us to explore this with real codes. Joe Hi Christopher:

I like the idea of measuring the multi core systems as compared to postulating the overall impact. We did some experiments and wrote some white papers on what we found for “normal” threaded apps using OpenMP, pthreads, and other models on a single node of dual Opteron 275 more than a year ago. While there was contention, as you could see if you looked at the papers, this was not a major issue.

It does not mean that it isn’t a major issue for some codes. Any code which fills the memory pipeline (or any other shared resource) is likely to have serious contention issues. What I was measuring was the performance of these codes built for parallel execution on a parallel machine using realistic data sets.

You can pull the PDF’s off my company home page (http://www.scalableinformatics.com) towards the bottom. We are working on expanding the testing. Quad cores will, I believe, have a more severe contention issue, and we are trying to develop tests which allow us to explore this with real codes.

Joe

]]> By: Michael Suess http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-25 Michael Suess Fri, 25 Aug 2006 08:13:30 +0000 http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-25 Christopher: I am not entirely sure I understand your comment fully. That said, I think Larry knows where superlinear speedups come from ;-). And I do not see a reason why they should be any less common with multicore-chips, because they usually have more cache than their single-core counterparts. But as I said, I am not even sure you were suggesting that :-). What I am sure about, though, is that you are probably right with your assumption that 32-core CPUs are not going to be very efficient without changes to the memory subsystems. I am also sure, the smart people designing these processors will come up with a solution. Maybe we will see NUMAs on a chip by that time :-). I also agree in part to your comments regarding locality - this is very important for performance. Nevertheless, Larry stated in his article that it is not about "pedal-to-the-metal" optimizations (and unfortunately, I still regard locality optimization as one of those as long as there are not better tools to help). Therefore, I do not see locality optimizations as a prequisite for parallelization, but rather as an orthogonal technique. Both can be done independently to gain performance and for <em>best</em> results on multi-core architectures you need to employ both - but on the other hand, sometimes you do not need <em>best</em> results and in that case I do not see anything wrong with doing either parallelization or locality optimization. Cobo: This blog is about parallelization in general and I will be blogging about other parallel programming systems - or else I would have probably called it <em>Thinking OpenMP</em> :-). This blog has an emphasis on OpenMP because it is the system I know about the most. But I am always interested in playing with other systems, and in fact I have done the first half of the Erlang tutorial again just yesterday to refresh my memory on it. You will definitly see posts about other systems. You may even see them very soon, since I am going to <a href="http://www.europar2006.de/" rel="nofollow">Europar</a> next week and there are talks about a lot of exiting systems happening there... Christopher:
I am not entirely sure I understand your comment fully. That said, I think Larry knows where superlinear speedups come from ;-). And I do not see a reason why they should be any less common with multicore-chips, because they usually have more cache than their single-core counterparts. But as I said, I am not even sure you were suggesting that :-).

What I am sure about, though, is that you are probably right with your assumption that 32-core CPUs are not going to be very efficient without changes to the memory subsystems. I am also sure, the smart people designing these processors will come up with a solution. Maybe we will see NUMAs on a chip by that time :-).

I also agree in part to your comments regarding locality - this is very important for performance. Nevertheless, Larry stated in his article that it is not about “pedal-to-the-metal” optimizations (and unfortunately, I still regard locality optimization as one of those as long as there are not better tools to help). Therefore, I do not see locality optimizations as a prequisite for parallelization, but rather as an orthogonal technique. Both can be done independently to gain performance and for best results on multi-core architectures you need to employ both - but on the other hand, sometimes you do not need best results and in that case I do not see anything wrong with doing either parallelization or locality optimization.

Cobo:
This blog is about parallelization in general and I will be blogging about other parallel programming systems - or else I would have probably called it Thinking OpenMP :-). This blog has an emphasis on OpenMP because it is the system I know about the most. But I am always interested in playing with other systems, and in fact I have done the first half of the Erlang tutorial again just yesterday to refresh my memory on it. You will definitly see posts about other systems. You may even see them very soon, since I am going to Europar next week and there are talks about a lot of exiting systems happening there…

]]> By: Cobo http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-24 Cobo Fri, 25 Aug 2006 00:20:06 +0000 http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-24 To Michael: I have been willing to ask you this for a long time, but thought it wasn't the time. Now, as you mentioned it... Do you have any background or plan on blogging about other parallelization techniques and languages such as the aforementioned Erlang? I find quite interesting things like their processes implementation which seems to be much more efficient than mapping to OS threads, and all that kind of stuff. Also (and I suppose as consequence of my inexperience), I see OpenMP like a great short-term solution for existing sequential languages, but I think the future should stand for languages where you don't have to consider which blocks of code should be concurrent or not, instead of having concurrency by default. Just the thoughts of a newbie... Very interesting both Larry's article and Christopher's comment on locality of reference and future data fetching from memory (which is a very interesting topic too). Cheers! To Michael:
I have been willing to ask you this for a long time, but thought it wasn’t the time. Now, as you mentioned it… Do you have any background or plan on blogging about other parallelization techniques and languages such as the aforementioned Erlang?
I find quite interesting things like their processes implementation which seems to be much more efficient than mapping to OS threads, and all that kind of stuff.
Also (and I suppose as consequence of my inexperience), I see OpenMP like a great short-term solution for existing sequential languages, but I think the future should stand for languages where you don’t have to consider which blocks of code should be concurrent or not, instead of having concurrency by default.

Just the thoughts of a newbie…

Very interesting both Larry’s article and Christopher’s comment on locality of reference and future data fetching from memory (which is a very interesting topic too).

Cheers!

]]> By: Christopher Aycock http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-23 Christopher Aycock Thu, 24 Aug 2006 23:31:51 +0000 http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-23 Larry, superlinear speed comes from better cache exploitation; memory-intensive applications work better on multiple processors simply because there's more cache. As for multicore chips, it's more likely the case that improved locality (from sharing memory regions among the threads) is the primary benefactor. There is the contention issue to worry about on an SMP, which is why I doubt that 32-core CPUs will ever make it without a serious change to the way data is fetched from memory. Therefore, if you really want to speed-up an application, improve the locality of reference. There are a number of tricks that unfortunately aren't exploited by most compilers, such as a simple reordering of iterations in a nested loop. Only once the cache misses have been significantly reduced should a programmer investigate multicore execution. Larry, superlinear speed comes from better cache exploitation; memory-intensive applications work better on multiple processors simply because there’s more cache. As for multicore chips, it’s more likely the case that improved locality (from sharing memory regions among the threads) is the primary benefactor.

There is the contention issue to worry about on an SMP, which is why I doubt that 32-core CPUs will ever make it without a serious change to the way data is fetched from memory.

Therefore, if you really want to speed-up an application, improve the locality of reference. There are a number of tricks that unfortunately aren’t exploited by most compilers, such as a simple reordering of iterations in a nested loop. Only once the cache misses have been significantly reduced should a programmer investigate multicore execution.

]]> By: Larry O'Brien http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-22 Larry O'Brien Wed, 23 Aug 2006 23:27:04 +0000 http://www.thinkingparallel.com/2006/08/24/thoughts-on-larry-obriens-article-at-devxcom/#comment-22 Thanks for the comments. Re "the only way..." well, yeah, it _is_ a hook and the way I justify it is the "mainstream" qualifier. There are _lots_ of programming languages that have models that allow for automatic parallelization, but I think it's fair to say that they are not in the mainstream. Whether they will _enter_ the mainstream is another question... As to the 70% number, I guess I did use it a couple times, because it happened to be the number I saw on this example and a couple others I've recently measured. Familiarity breeds... I don't know ... expectation, I guess. Even since I wrote the article, I was looking at some results showing superlinear speedup. It seems more common today than it was "back in the day." I don't know if that's a reflection of more complexity in computer architectures or a reflection of more sophisticated algorithms. A little of both, I suppose. Anyway, thanks for the comments! Thanks for the comments. Re “the only way…” well, yeah, it _is_ a hook and the way I justify it is the “mainstream” qualifier. There are _lots_ of programming languages that have models that allow for automatic parallelization, but I think it’s fair to say that they are not in the mainstream. Whether they will _enter_ the mainstream is another question…

As to the 70% number, I guess I did use it a couple times, because it happened to be the number I saw on this example and a couple others I’ve recently measured. Familiarity breeds… I don’t know … expectation, I guess.

Even since I wrote the article, I was looking at some results showing superlinear speedup. It seems more common today than it was “back in the day.” I don’t know if that’s a reflection of more complexity in computer architectures or a reflection of more sophisticated algorithms. A little of both, I suppose.

Anyway, thanks for the comments!

]]>