How large a sample size ?

I responded to an invite to get some info on helping understand if my webstats were an adequate basis to make a decision. Its from MarketingExperiments.com, a MECC Labs company.


Their site and blog are an absolutely fantatic resource of much higher quality than most of the marketing drek on the web. Nevertheless, I find that I want to nitpick in the area of statistical validity.

Since it has a copyright notice on it, I'll only quote this part:

"There are 2 ways to evaluate your data…
􀂄 Understand the complex math (which they then provide alot of complex formulas on




􀂄 Utilize the MEC test protocol - which makes it somewhat simpler but still, nothing that you could (or I could) remember.

I did, in my youth, take some statistics. I remember an old rule.

DIVIDE YORU DATA SET IN HALF RANDOMLY. COMPARE THE DATA FROM THE TWO HALFS. IF IT MATCHES. BELIEVE IT. IF NOT, GET A BIGGER SAMPLE.

This means that if you have conversion rate data on 2000 people. You could sort alphabetically and take every other data point. Add them up. If both data sets show a 1.6 conversion rate, your sample set is adequate.

If one set say 1.3% and the other says 2.0%, your margin of error is pretty large.

Also, if you division of the data is not arbitrary, this is meaningless. For instance, you cannot divide by the first half of the month or the second since this not a random division.

Question to you smart guys. Does this rule hold true?
Secondly, how about my idea for a pool tool to use when the normal pools of data are too small to use.