Any statisticians out there?

Doug Hemken wrote:Mark, given your description of the design, "effect size" is likely based on paired differences? So the pre- and post- summary statistics don't really tell the story ... that's the whole reason paired differences is useful. I take it effect size is mean(paired differences)/sd(paired differences) ?

It's not clear to me what effect size actually means, but that seems like a reasonable definition. Under the assumption of positive correlation between the individuals (it does seem like a paired analysis, my original computations only saw the numbers given and I assumed it wasn't), then one can use the given variances to compute an upper bound on the actual (paired) variance.
This is because Var(X - Y) = Var(X) + Var(Y) - 2Cov(X,Y), so Var(X-Y) is less than Var(X)+Var(Y).
(I can't think of a paired experiment where there would be negative correlation, so we'll run with this).
So this would give us an upper bound of around 20 for the standard deviation of the differences (of course this is a point estimate, 20 is not an upper bound that is accounting for sampling variability).
So a bad point estimate for effect size is 9/20 = 0.45. 0.7 seems plausible I suppose.
On the other hand, one way to get nearly identical marginal standard deviations for both your pre and post groups is to have every pair change by a constant, i.e. effect size of infinity (as you'd be dividing by a standard deviation of 0 in our definition of effect size).

It has been my experience as a statistical consultant that most of our clients mean some version of Cohen's D when they talk about effect size.

So Juan's 0.45 is a lower bound on effect size.

Doug Hemken wrote:Mark, given your description of the design, "effect size" is likely based on paired differences? So the pre- and post- summary statistics don't really tell the story ... that's the whole reason paired differences is useful. I take it effect size is mean(paired differences)/sd(paired differences) ?

It's described as being calculated using Hedges g if that helps. Mean post- mean pre all divided by SD pooled.

Does that clarify anything?

I get Hedges' g of -0.54 and Cohen's d of -0.62 using the formulas here: polyu.edu.hk/mm/effectsizef…

My work in R:
n1 <- 4
n2 <- 4
sd1 <- 15
sd2 <- 14
mean1 <- 47
mean2 <- 56

sd_pooled <- sqrt(((n1 - 1) * (sd1 ^ 2) + (n2 - 1) * (sd2 ^ 2)) / (n1 + n2 - 2))
print(sd_pooled)
[1] 14.50862

cohens_d <- (mean1 - mean2) / sd_pooled
print(cohens_d)
[1] -0.620321

hedges_g <- cohens_d * (1 - (3 / ((4 * (n1 + n2)) - 9)))
print(hedges_g)
[1] -0.5394095

These results haven't been checked and I worked very quickly. Use at your own risk.

The website where I got the formulas has a calculator that confirms my results: polyu.edu.hk/mm/effectsizef…

Can you share what the study is about? Interpreting effect size depends on context. A large effect size in one field is a small one in another.

Also, your sample seems very small. I'm not sure if what we are doing here is appropriate.

Hedges being a corrected Cohens.

I don't think I can come up with an effect size of 0.7 calculated in the way you specify. I can, however, imagine getting to 0.7 if you instead base the calculation on paired differences.

Edit: I'm on the same page as Justin.

If the language they use is actually "post intervention" it certainly seems like they have a CRB design. Unless they are clueless, I have to take pooled variance to mean MSTR.BL (or whatever notation you prefer for the interaction mean square term between the treatment and the blocks). Interestingly, they don't provide summary statistics that allow you to reproduce this value. But it's completely conceivable that .7 could be your value, though I'm not thrilled with how they are disseminating results.

The phrase "an experimental group" is confusing.

"Experimental" suggests a "block design", as Juan supposes - treatment versus control- but "an" and "group" suggest just one group, a pre-test/post-test design (so the only random element would be selecting the study subjects).

We are clearly not going to get 0.7 out of the summary statistics Mark passed on, so something is missing in the description of the problem.

Not a statistician here. Was merely forced to teach myself stats to complete my research, haaa. Luckily I also had a team of professional statisticians paid on my grant to advise me from time to time, but honestly asking a statistician to give you input on your problems is like asking a firefighter to clean chocolate off your face by turning on the firehose.

What I took away about effect size: not sure it means anything concrete. I was advised to use Cohen's D cut-offs as well and to allow the reader to interpret the result "as they wish", but there are all kind of cut-offs you can use. It's like some nebulous dimension that looks different from every angle you view it.

But to conclude, I can't even spell statistician without spellcheck correction. Everyone thinks stats is easy when they learn it in its basic format, but once you actually have to understand and apply it to your own invented story, that shit is fuckin' voodoo.

Apologies to statisticians out there. Some of my best friends are statisticians!

Aerili wrote:... that shit is fuckin' voodoo.

After 20+ years I'm often inclined to agree!

This is why I stay away from data.

Aleks Zebastian wrote:climbing friend, May I suggest you get a girlfriend?

I'm pretty sure statistics are less confusing than women, though not as fun to onsite.

Not sure what any of this means ^^

YER GONNA DIE!