You gotta love Sun (well, I don't, but I thought it was a good opening line).  In June, they publish a benchmark that shows their web services performance is 300% better than Microsoft's.  Not 3%, not 30%, but 300 whopping percent.  Wow.  Time for me to write my resignation letter, cuz Microsoft is doomed.  We clearly don't have a clue how to write good software.

But wait.  Maybe we just don't know how to do benchmarks.  A little bit of inspection shows the Secret of Sun's Success:  Don't show the benchmark code.  Just write a bunch of gobbledygook about configurations and what your code supposedly did and everybody will believe you.

Wrong.

Microsoft, once again, took the high road.  We submitted a rebuttal to The Middleware Company, which was subsequently published on TheServerSide.Net.  We were very very thorough.  We followed Sun's publication as closely as we could, and implemented both sides openly and fairly, without taking special “shortcuts” on our side to make us look better.  What happened?  By golly, we did better.  Not 300% better, but definitely better overall.  More importantly, as we scaled up the web service messages to “real world payload” size, our performance became much more pronounced.  Then we did something really brave...

We published all the source for everybody to see.

Yesterday, Dennis MacNeil of Sun write up a thorough rebuttal to the work Microsoft did.

But they neglected to back it up with source.  Again.

And then to add fuel to the fire, Dennis challenged Microsoft to participate in the SPEC group's new web service benchmark standard.   Which we had been fully participating in until IBM and Sun demanded that the benchmark leave out price/performance ratios.  Talk about insane.  That's almost like somebody saying that the way to fix a deficit is to give away money (*cough*).  That's not how benchmarks work, just look at the TPC benchmarks.  They're respected, reliable, and give business decision maker's a foundation on which to make a decision when it comes to price as well as performance.  I mean, I don't know about you, but I've NEVER done any consulting for a CxO who says “Who gives a damn about money, just give me something big and powerful!” (which probably explains why I'm not a filthy rich consultant .

Fortunately, sanity has prevailed, and Greg Leake (who manages a lot of benchmark work at Microsoft) politely responded to the rebuttal.  I'll summarize for you here:  “Dude, you are so full of it.”

Dennis: We await your reply.  But even more, we would ALL love to see the source code that proves Sun's point.  We've stepped up to the plate.  Now it's your turn.