Empirisoft Support

    Welcome to Empirisoft Support
Results 1 to 4 of 4

Thread: Paper and Pencil vs. Computerized Experiments (case 5457)

  1. #1
    Join Date
    Nov 2005

    Paper and Pencil vs. Computerized Experiments (case 5457)

    [edited from support email]
    I have a methodological question for you. I am using MediaLab to replicate a study originally conducted with pencil and paper. Is it known whether computer administration of an instrument affects the results of the research?
    Last edited by jason_reed; 12-15-2015 at 09:55 AM.

  2. #2
    Join Date
    Nov 2005
    As far as I know the differences are negligible, at least in the lab. In reviewing any research on this issue, be sure to take note if it's a comparison between web-based or lab-based computerized experiments. The tendency for reduced accountability in web based responding makes it difficult to generalize to lab-based experiments. If any one knows of additional references, please let me know and I'll list them here.

    Birnbaum, M. H. (1999). Testing critical properties of decision making on the Internet. Psychological Science, 10, 399-407.

    Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125-144.

    Coomber, R. (1997). Using the Internet for survey research. Sociological Research Online, 2 (2). Available: http://www.socresonline.org.uk/2/2/2.html.

    Frame, J. H., & Beaty, J. C. (2000, April). An Empirical Investigation of High-Technology Survey Methods: Paper-and-Pencil, Email, and Web-Based—Which is better? Poster presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.

    Helgeson, J. G., & Ursic, M. L. (1989). The decision process equivalency of electronic versus pencil-and-paper data collection methods. Social Science Computer Review, 7, 296-310.

    Krantz, J. H., Ballard, J., & Scher, J. (1997). Comparing the results of laboratory and Word-Wide Web samples on determinants of female attractiveness. Behavior Research Methods, Instruments, and Computers, 29, 264-269.

    Pasveer, K. A., & Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW. Behavior Research Methods, Instruments, and Computers, 30, 309-313.

    Pettit, F. A. (1999). Exploring the use of the World Wide Web as a psychology data collection tool. Computers in Human Behavior, 15, 67-71.

    Schaefer, D. R., & Dillman, D. A. (1998). Development of a standard email methodology: Results of an experiment. Public Opinion Quarterly, 62, 378-397.

    Shaw, D., & Davis, C. H. (1996). The Modern Language Association: Electronic and paper surveys of computer-based tool use. Journal of the American Society for Information Science, 47, 932-940.

    Smith, M. A., & Leigh, B. (1997). Virtual subjects: Using the Internet as an alternative source of subjects and research environment. Behavior Research Methods, Instruments, & Computers, 29, 496-505.

    Stanton, J. M. (1998a). An empirical assessment of data collection using the Internet. Personnel Psychology, 51, 709-725.

    Swoboda, W. J., Muhlberger, N., Weitkunat, R., Schneeweiss, S. (1997). Internet surveys by direct mailing: An innovative way of collecting data. Social Science Computer Review, 15, 242-255.

    Tse, A.C. B. (1998). Comparing the response rate, response speed and response quality of two methods of sending questionnaires: E-mail vs. mail. Journal of the Market Research Society, 40, 353-361.

    Yost, P.R. & Homer, L.E. (1998, April). Electronic versus Paper Surveys: Does the Medium Affect the Response? Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology. Dallas, TX.
    Last edited by jarvis24; 11-13-2006 at 03:13 PM.

  3. #3
    Join Date
    Feb 2013

    more resources on web surveys

    Here are some more readings that have proven helpful in my own work that uses web-based surveys:

    Couper, M. P., Traugott, M. W., & Lamias, M. J. (2001). Web survey design and administration. Public Opinion Quarterly, 65, 230-253.

    Dillman, D. A., & J. D. Smyth. (2007). Design effects in the transition to Web-based surveys. American Journal of Prevention Medicine, 32, S90-S95.

    Dillman, Don A., Reips, U. & Matzat, U. (2010). Advice in Surveying the General Public Over the Internet. International Journal of Internet Science, 5, 1-4.

    Ganassali, S. (2008). The influence of the design of Web survey questionnaires on the quality of responses. Survey Research Methods, 2, 21-32.

    Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust Web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59, 93-104.

    Sax, L. J., Gilmartin, S. K., & Bryant, A. N. (2003). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 4, 409-432.

  4. #4
    Join Date
    Jan 2013

    non web-based computer adaptive test and paper-pencil test equivalence

    Below, I append several references to studies of the similarities and differences between (non web-based) computer adaptive test and paper-pencil test equivalence. Most of these concern tests with correct and incorrect responses, rather than subjective self-reports. The conclusion seems to be that several individual characteristics better predict mode-of-assessment context effects than item characteristics (i.e., most tests are not any easier or more difficult via adaptive and non-adaptive computer assessment).

    Clansing, C., & Schmitt, D. (1990). Paper versus CRT: Are reading rate and comprehension affected?Paper available from ERIC. (ERIC Document Reproduction Service No. ED323924)

    Gould, J. D., Alfaro, L., Finn, R., Haupt, B.,&Minuto, A. (1987). Reading from CRT displays can be as fast as reading from paper. Human Factors, 29, 497517.

    Lee, J., Moreno, K. E., & Sympson, J. B. (1986). The effects of mode of test administration on test performance. Educational and Psychological Measurement, 46, 467473.

    Lunz, M. E., & Bergstrom, B. A. (1994). An empirical study of computerized adaptive test administration conditions. Journal of Educational Measurement, 31, 251263.

    Mason, B. J., Patry, M., & Bernstein, D. J. (2001). An examination of the equivalence between non-adaptive computer-based and traditional testing. Journal of Educational Computing Research, 24, 2939.

    Mazzeo, J., & Harvey, A. (1988). The equivalence of scores from automated and conventional educational and psychological tests (College Board Rep. No. 88-8). New York: College Entrance Examination Board.

    Olsen, J. B., Maynes, D. D., Slawson, D., & Ho, K. (1989). Comparison of paper-administered, computer-administered and computerized adaptive achievement tests. Journal of Educational Computing Research, 5, 311326.

    Pomplun, M., & Custer, M. (2005). The score comparability of paper and pencil and computer K-3 reading tests. Journal of Educational Computing Research, 32, 153166.

    Pomplun, M., Custer, M., & Ritchie, T. D. (2006). Factors in paper-and-pencil and computer reading score differences at the primary grades. Educational Assessment, 11, 127-143.

    Zandvliet, D., & Farragher, P. (1997). A comparison of computer administered and written tests. Journal of Research on Computers in Education, 29, 423438.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts