Data: I obtained the data from Berkshire-Hathaway's 2010 annual report. On page 4, one can find Berkshire vs. S&P500 annual returns from 1965 to 2010.
Design & Statistics: It's a 1 variable 'within subject' (or matched-pairs) design with 2 levels (Berkshire and S&P 500). The statistical analysis I performed is a within-subjects t-test (or t-test for matched-pairs). However, to test the statistical significance of the result, I used a randomization approach, rather than a conventional statistical significance test, as the data have clearly not been obtained using random sampling. You can learn more about randomization testing here.
Software: I used a comprehensive stats packaged called NCSS 2007, which very impressively provides the option to use randomization testing as a matter of course (i.e., not a separate add-on or separate GUI menu) for basically all statistical analyses (this is unique, in my experience). I selected 10,000 randomized samples for the randomization analysis. You can watch me demonstrate the analysis here. (nb: I was very tired when I recorded this video)
Results & Discussion: Berkshire average performance = 21.56; S&P 500 average performance 10.96; difference = 10.60 (Wow!)
Randomization statistical significance testing: p = .0011.
In general terms, this means that Buffett's investment performance is very unlikely due to chance. In fact, on average, in only 1.1 out of a 1000 randomized samples does the randomization procedure identify a larger mean difference than the observed mean difference (i.e., 10.60). Therefore, the people who are lucky in this scenario are those who invested with Buffett early on in his investment career : )
However, I should note that I'm not totally convinced that using a paired t-test with randomization statistical significance testing is the best way to test the hypothesis that Warren Buffett is simply lucky. At least on the surface, it appears to be a relatively straightforward and valid approach to doing so. I can think of alternative approaches, but they would require more data collection. It should be noted that no matter what statistical method is used, there will always be some level of chance associated with the results. Therefore, I can't see how it could ever be demonstrated definitively (i.e., p = .00000∞) that Warren Buffett's performance is above chance.
≈ Gilles
Disclaimer: Despite the results within this blog, I do not own any Berkshire-Hathaway shares. Go figure.
Check out this youtube video where I discuss the analysis and results:
http://www.youtube.com/watch?v=7gfXKJPJsFA