Thinking Parallel

David: Multi cores are pointless and useless without “parallel programming”. In fact, the unused communications hardware and supporting firmware costs chip space and cycles. A multi-core chip used strictly for “nonparallel” programming could never compete with a single core chip of the same technology family, on cost, speed, or anything else. But you have to remember that “parallel programming” extends way beyond the use of fine-grain application-level threading APIs.

ALL modern operating systems, including Mac OS X and Windows, have been “inherently multiprocessing” long before multi-core chips trickled down from the enterprise to the desktop. And even a user who only launches sequential programs benefits from having multiple cores. OS daemons and drivers, for example, can be scheduled on the other core(s). And many modern support libraries (including especially the OS bundled graphics frameworks) are inherently parallel even if the application doesn’t think it is. Pop up the Mac OS X activity viewer, or the Windows task manager, and see how many processes have a single thread. Sure, there ARE some; but a tiny minority of the processes running on any typical system.

I don’t think any of this is going to go away. After all, it started long before multi-core chips, for completely different reasons, and only ACCIDENTALLY allows trivial exploitation of them.

What will change is the methodologies and techniques by which the capability is exploited at the user/application level. We’ve seen widespread adoption of basic threading API technologies; and even though there are indeed still a lot of programmers hanging back in fear of the unknown, the front wave is pressing on beyond that simplicity into generalized lock-free techniques and alternate language metaphors.

“Always in flux, the future is.” I don’t think the genie is going back into the bottle anytime soon; but the thing about genies is that almost nothing is impossible… but a lot is “inconceivable”.

David: That people like to argue. This is good, because the tradeoffs depend greatly on the specific application AND on the current shifts in technology. Both are tools any programmer should have in the toolbox. You’ll use both, in various combinations and alone, for various jobs. Which jobs go best with which tool will change over time. If I had to choose whether to have oxygen or water, I’d stay the choice is moot: the mere fact that I have to make it means I’m in a great deal of trouble. It may be great fun to argue over which is “superior”, but in practice we do, and will, depend on both to a degree that any distinction becomes academic. Academic arguments are great fun, because they can never be factually or permanently resolved; but unless you have a vested interest in selling one or the other, (or just love to argue), my advice is to simply stand back and observe, and learn what you can from both. Use what you can from each, and laugh at all those who claim “the one true path”.

David: Generalization of lock-free technology has come a long way, to the point where it’s feasible to consider incorporating some basic enabling techniques into language standards. (Though whether this will actually happen is a different matter.) Calmly rational synthesis of lock and lock-free mechanisms has a lot of promise for a wide range of parallel workloads.

Transactional memory has a lot of promise to simplify synchronization and visibility issues across all models of shared memory multiprocessing; but (and?) it’s still young.

David: Sure; but the silver bullets are coming AT you, from all directions, and your job is to dodge or absorb. No, there is no one answer, and the future lies in all possible directions. We’re likely to diverge on API and language paths, and argue for years or decades on which is better and why, before eventually realizing the common principles and formalizing them into something that at least comes close to a unification. But by then the physical technology will have changed (quantum computing?) and it’ll all become academic and open ended again…

David: I don’t even think it’s true. Large serial programs are hard to build, hard to debug and test, and hard to maintain. The “state of the practice” in parallel programming is a little behind only because it’s more recently come into widespread use, and true “state of the art” practices and technologies take a long time to work their way into actual practice. But the gap isn’t nearly as wide as a lot of people would like to believe. You really just need to cast off your illogical and unnatural preconceptions that only one thing happens at a time (no child would last long in the real world with such ideas!) and embrace asynchrony.

David: The principal strength is that it’s there. We have a nearly universal fine-grain parallelism API; with the near-exception of always standards-shy Microsoft. (And the semantics and even syntax of Win32 threads are heavily based on the same ancestry as POSIX threads, with strong influences from UNIX and POSIX. Except for its dependency on archaic “events”, moving back and forth isn’t that difficult.)

David: Probably not much, really. Some things might be easier now, but that’s only because POSIX threads exists already and it’s raised the degree of knowledge and experience across the industry.

But if you go deeper, the biggest flaws in POSIX threads really aren’t in POSIX threads at all, but in the fact that we relied on existing language syntax and semantics to support the API. We did it deliberately, and had no practical choice at the time because language designers wouldn’t adapt until thread technology had been proved in the real world. THAT needs to be addressed, and still can be. A real parallel-safe memory model for C++ and C, for example, is long overdue. ANSI C needs to borrow some simplistic form of exception technology from C++ — even if it’s only based on Microsoft’s minimalist try/except syntax. For consistency and reliability, for example, POSIX cancellation needed to be an exception, as it was in the original CMA. And truly portable POSIX thread code would be a lot easier with a real parallel-safe C memory model.

If such changes happen, they’ll enable (in fact, require) the POSIX thread standard to become more like what it should have been in the first place.

David: Tru64’s ladebug and Visual Threads were awesome tools, and ATOM allowed constructing simple analyzers. Nobody else really has anything that comprehensive, despite various gdb add-ons. (Then again, Intel has ladebug… but hasn’t really done anything with it.) Totalview is a great portable thread debugging environment, although the GUI is a bit “opaque”.

There are a wide range of others, but I’ve never really kept up on all of the development; and the landscape changes awfully fast.

The biggest problem people see is asynchronous memory corruption. It’s really hard (and expensive) to monitor and detect. (Though Visual Threads had hooks to help, at a big cost.) Hardware monitoring technologies to express and detect “rule violations” may be essential here. There might even be work on it, though I haven’t seen any. Transactional memory is, in a sense, an attempt to get around all of this; but if it ever gains widespread deployment, that day is a long way off.

David: Beware of the net. A lot of thread references are ancient and poorly (or un-) maintained. Many are simply wrong, others wildly misleading. Read critically! There are a lot of good books. Mine, of course. Bil Lewis has a thrice-recycled book on Solaris UI threads, POSIX threads, and Java threads. I’ve seen a couple that look OK even focusing on Windows threading; though that’s a whole area that doesn’t interest me much. 😉

The comp.programming.threads newsgroup is a great resource for anyone involved in threads. The discussions can be fascinating and educational. (I’ve usually followed it fairly regularly and frequently contributed… between an extremely hectic schedule and unreliability of the HP news server I’d been using, I haven’t “been there” much in the past few months.)

Grab a system and experiment, once you know anything at all. See what happens, think about why, and figure out experiments to increase your knowledge. “The Scientific Method” applies to engineering, too; and hard-won personal experience is often far more valuable than something you’ve read.

David: No, no; too many contenders. I couldn’t choose. Nor, I’m afraid, can I really justify the time it’d take to write up even a few anecdotes. It’s easy to do stupid things when programming. It’s easier when you don’t fully understand what you’re doing.

Then again, as the old saying goes, “wisdom comes from experience, and experience comes from lack of wisdom”.