Is the randomization of blocks truly random?

[jarvis24]

There is a lot here. Lets start with the fact that 61% of your subjects received a "set1-set2" order and 39% (was 41% a typo or am I missing something?) received a "set2-set1" order. You mention that the p level of this is p<.01. Can you tell me a bit more here with regard to how that p level was calculated? esp., with respect to the type of analysis and n-sizes. Any specifics you can provide will help me to understand where the problem could be coming from.

[benwood23]

Thanks for the reply. Yes, the 41 percent was a typo. It is actually 39 percent. As for your other questions, the study had 222 subjects, and I calculated the p-value using a simple t test that tested whether the percentage of people in the set1-set2 order was 50 percent. The p-value testing whether I can reject the null hypothesis that the percentage equals 50 percent is .0007.

[jarvis24]

This is unusual. I checked the pilot combined que and it looks fine in terms of its structure. Although, technically, I think you need to do a categorical/binomial analysis (chi-square, logistical regression? a quick java applet for such is here: http://stattrek.com/Tables/Binomial.aspx) rather than a t-test, the probability is still going to make it very surprising to get 88/222 assigned to one of two combinations. Hmm.

We have been using the same BGR randomization routines in MediaLab for almost 14 years now and I've never heard of a report like this so I don't know what to say. If it were me, I would run a simulation to see if I could replicate the problem--e.g., delete all but the essential items from the pilot questionnaire and then run the shortened (maybe 10 item revision) twenty or thirty times and see if the distribution is off balance in any predictable way. If it DOES recur then please please let me know.