Measuring Production and Comprehension of Written Arguments in Upper-Elementary Grades

Alina Reznitskaya, Ian A. G. Wilkinson

Abstrakt


We describe a systematic process of developing measures of argument production and comprehension. These measures, designed for students in upper-elementary language arts classrooms, are called Writing Argument and Reading Argument. We discuss the rationale and theoretical framework for the measures, describe pilot and validation studies, and present initial findings to support the reliability, validity, and usability of these measures. Our results showed that both measures had acceptable inter-rater reliability. The correlations among Writing Argument, Reading Argument and an established reading comprehension test were moderate, which highlights the importance of task-specific competencies. The performance on both measures was not associated with ethnicity of the students. Gender was a significant predictor, with girls performing better than boys. Teachers found both measures to be pedagogically useful. Although some teachers initially struggled with learning how to use the scoring rubrics, they generally found the scoring for both tasks to be informative for their practice.

Klíčová slova


writing argument; reading argument; measures; validation; elementary grades

https://doi.org/10.5817/SP2019-4-3

Celý článek:

PDF (English)

Reference

Zobrazit literaturu Skrýt literaturu

Anderson, R. C., Nguyen-Jahiel, K., McNurlen, B., Archodidou, A., Kim, S.-y., Reznitskaya, A., Tillmanns, M., & Gilbert, L. (2001). The snowball phenomenon: Spread of ways of talking and ways of thinking across groups of children. Cognition and Instruction, 19(1), 1–46. | DOI 10.1207/S1532690XCI1901_1

Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading comprehension. In P. D. Pearson, R. Barr, M. L. Kamil & P. Mosenthal (Eds.), Handbook of reading research (pp. 255–291). New York: Longman.

Asher, S. R., Hymel, S., & Wigfield, A. (1978). Influence of topic interest on children's reading comprehension. Journal of Reading Behavior, 10(1), 35–47. | DOI 10.1080/10862967809547253

Chambliss, M. J., & Murphy, P. K. (2002). Fourth and fifth graders representing the argument structure in written texts. Discourse Processes, 34(1), 91–115.

Chinn, A. C., Duncan, R. G., Hung, L. C.-C., & Rinehart, R. W. (2016). Epistemic criteria and reliable processes as indicators of argument quality in science students' argumentation. Paper presented at the Annual Meeting of the American Educational Research Association, Washington, DC.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont: Wadsworth.

Driver, R., Newton, P., & Osborne, J. (2000). Establishing the norms of scientific argumentation in classrooms. Science Education, 84(3), 287–312. | DOI 10.1002/(SICI)1098-237X(200005)84:3<287::AID-SCE1>3.0.CO;2-A

Ennis, R. (1996). Critical thinking. Upper Saddle River: Prentice Hall.

Ennis, R. (2003). Critical thinking assessment. In D. Fasko (Ed.), Critical thinking and reasoning (pp. 293–313). Cresskill: Hampton Press, Inc.

Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of toulmin's argument pattern for studying science discourse. Science Education, 88(6), 915–933. | DOI 10.1002/sce.20012

Ferretti, R. P., & Fan, Y. (2016). Argumentative writing. In C. A. MacArthur, S. Graham & J. Fitzgerald (Eds.), Handbook of writing research, Second edition. New York: Guilford Press.

Fischer, F., Chinn, C. A., Engelmann, K., & Osborne, J. (Eds.). (2018). Scientific reasoning and argumentation: The roles of domain-specific and domain-general knowledge. New York: Routledge.

Gick, M. L., & Holyoak, K. J. (1983). Schema induction and analogical transfer. Cognitive psychology, 15(1), 1–38. | DOI 10.1016/0010-0285(83)90002-6

Goldman, S. R., Britt, M. A., Brown, W., Cribb, G., George, M., Greenleaf, C., Lee, C.D., Shanahan, C., & Project, R. (2016). Disciplinary literacies and learning to read for understanding: A conceptual framework for disciplinary literacy. Educational Psychologist, 51(2), 219–246. | DOI 10.1080/00461520.2016.1168741

Govier, T. (1987). Problems in argument analysis and evaluation. Providence: Foris.

Govier, T. (2010). A practical study of argument. Belmont: Wadsworth Publishing Company.

Graff, G. (2003). Clueless in academe. New Haven: Yale University Press.

Halpern, D. F. (2003). The "how" and "why" of critical thinking assessment. In D. Fasko (Ed.), Critical thinking and reasoning (pp. 331–354). Cresskill: Hampton Press, Inc.

Healy, P. (1987). Critical reasoning and dialectical argument: An extension of Toulmin's approach. Informal Logic, 9(1), 1–12. | DOI 10.22329/il.v9i1.2654

Hidi, S. (2001). Interest, reading, and learning: Theoretical and practical considerations. Educational Psychology Review, 13(3), 191–209.

Hollihan, T. A., & Baaske, K. T. (1973). Arguments and arguing: The products of human decision making. Prospect Hights: Waveland.

Hughes, J. N. (1992). Review of the Cornell critical thinking tests. In J. J. Kramer, J. C. Conoley & L. L. Murphy (Eds.), The eleventh mental measurements yearbook. Lincoln: The Buros Institute of Mental Measurements.

Johnson, K. M., (2005). Gates-MacGinitie reading tests fourth edition forms S and T. In B. S. Plake & R. A. Spies (Eds.), The sixteenth mental measurements yearbook (Vol. 16). Lincoln: Buros Institute of Mental Measurements.

Klahr, D., Zimmerman, C., & Jirout, J. (2011). Educational interventions to advance children's scientific thinking. Science, 333(6045), 971–975. | DOI 10.1126/science.1204528

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. | DOI 10.1016/j.jcm.2016.02.012

Kuhn, D. (1991). The skills of argument. Cambridge: Cambridge University Press.

Kuhn, D. (2005). Education for thinking. Boston: Harvard Education Press.

Kuhn, D., & Crowell, A. (2011). Dialogic argumentation as a vehicle for developing young adolescents' thinking. Psychological Science, 22(4), 545–552. | DOI 10.1177/0956797611402512

Kuhn, D., Hemberger, L., & Khait, V. (2016). Argue with me: Argument as a path to developing students' thinking and writing. New York: Routledge.

Lipman, M. (2003). Thinking in education. New York: Cambridge University Press.

MacGinitie, W. H., MacGinitie, R. K., Maria, K. & Dreyer, L. G. (2002). Gates-MacGinitie reading tests. Rolling Meadows: Riverside Publishing.

Mason, L., & Scirica, F. (2006). Prediction of students' argumentation skills about controversial topics by epistemological understanding. Learning and Instruction, 16(5), 492–509. | DOI 10.1016/j.learninstruc.2006.09.007

McCabe, P. P. (2005). Review of the Gates-MacGinitie reading tests. In R. A. Spies & B. S. Plake (Eds.), The Sixteenth Mental Measurements Yearbook. Lincoln: Buros Institute of Mental Measurements.

Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal reasoning among children of different grade, ability, and knowledge levels. Cognition and Instruction, 14(2), 139–178. | DOI 10.1207/s1532690xci1402_1

Mercer, N. (2011). Reasoning serves argumentation in children. Cognitive Development, 26(3), 177–191. | DOI 10.1016/j.cogdev.2010.12.001

National Governors Association. (2010). Common core state standards: Appendix A. research supporting key elements of the standards. Washington DC: National Governors Association Center for Best Practices, Council of Chief State School Officers.

Newell, G. E., Beach, R., Smith, J., VanDerHeide, J., Kuhn, D., & Andriessen, J. (2011). Teaching and learning argumentative reading and writing: A review of research. Reading Research Quarterly, 46(3), 273–304.

Nielsen, J. A. (2013). Dialectical features of students' argumentation: A critical review of argumentation studies in science education. Research in Science Education, 43(1), 371–393. | DOI 10.1007/s11165-011-9266-x

Norris, S. P. (1995). Format effects on critical thinking test performance. The Alberta Journal of Educational Research, 41(4), 378–406.

Norris, S. P., Leighton, J. P., & Phillips, L. M. (2004). What is at stake in knowing the content and capabilities of children's minds? A case for basing high stakes tests on cognitive models. Theory & Research in Social Education, 2(3), 283–308. | DOI 10.1177/1477878504046524

Nussbaum, E. M. (2011). Argumentation, dialogue theory, and probability modeling: Alternative frameworks for argumentation research in education. Educational Psychologist, 46(2), 84–106. | DOI 10.1080/00461520.2011.558816

Nussbaum, E. M., & Bendixen, L. D. (2003). Approaching and avoiding arguments: The role of epistemological beliefs, need for cognition, and extraverted personality traits. Contemporary Educational Psychology, 28(4), 573–595. | DOI 10.1016/S0361-476X(02)00062-0

Nussbaum, E. M., & Kardash, C. M. (2005). The effects of goal instruction and text on the generation of counterarguments during writing. Journal of Educational Psychology, 9(2), 157–169. | DOI 10.1037/0022-0663.97.2.157

Nussbaum, E. M., & Ordene, V. (2011). Critical questions and argument stratagems: A framework for enhancing and analyzing students' reasoning practices The Journal of the Learning Sciences, 20(3), 443–488. | DOI 10.1080/10508406.2011.564567

Oka, E. R., & Paris, S. G. (1987). Patterns of motivation and reading skill in underachieving children. In S. J. Ceci (Ed.), Handbook of cognitive, social, and neuropsychological aspects of learning disabilities (pp. 115–145). New Jersey: Lawrence Erlbaum Associates.

Page-Voth, V., & Graham, S. (1999). Effects of goal setting and strategy use on the writing performance and self-efficacy of students with writing and learning problems. Journal of Educational Psychology, 91(2), 230–240. | DOI 10.1037/0022-0663.91.2.230

Partnership for 21st Century Skills. (2012). A Framework for 21st Century Learning. Retrieved from http://www.p21.org/index.php

Phillips, L. M., & Patterson, C. C. (1987). Test of inference ability in reading comprehension. Newfoundland: Institute for Educational Research and Development.

Poteet, J. (1989). Review of the Ennis-Weir critical thinking essay test. J. C. Conoley & J. J. Kramer (Eds.), The tenth mental measurements yearbook (pp. 289–290). Lincoln: The Buros Institute of Mental Measurements.

Rapanta, C., Garcia-Mila, M., & Gilabert, S. (2013). What is meant by argumentative competence? An integrative review of methods of analysis and assessment in education. Review of Educational Research, 83(4), 483–520. | DOI 10.3102/0034654313487606

Renninger, K. A., Suzanne Hidi, & Krapp, A. (Eds.). (1992). The role of interest in learning and development. Mahwah: Lawrence Erlbaum Associates.

Reznitskaya, A., Anderson, R. C., Dong, T., Li, Y., Kim, I., & Kim, S. (2008). Learning to think well: Application of argument schema theory. In C. C. Block & S. Parris (Eds.), Comprehension instruction: Research-based best practices (pp. 196–213). New York: Guilford Press.

Reznitskaya, A., Anderson, R. C., McNurlen, B., Nguyen-Jahiel, K., Archodidou, A., & Kim, S. (2001). Influence of oral discussion on written argument. Discourse Processes, 32(2 & 3), 155–175.

Reznitskaya, A., & Gregory, M. (2013). Student thought and classroom language: Examining the mechanisms of change in dialogic teaching. Educational Psychologist, 48(2), 114–133. | DOI 10.1080/00461520.2013.775898

Reznitskaya, A., Kuo, L., Clark, A., Miller, B., Jadallah, M., Anderson, R. C., & Nguyen-Jahiel, K. (2009). Collaborative reasoning: A dialogic approach to group discussions. Cambridge Journal of Education, 39(1), 29–48. | DOI 10.1080/03057640802701952

Reznitskaya, A., Kuo, L., Glina, M., & Anderson, R. C. (2009). Measuring argumentative reasoning: What's behind the numbers? Learning and individual differences, 19(2), 219–224. | DOI 10.1016/j.lindif.2008.11.001

Reznitskaya, A., & Wilkinson, I. A. G. (2015). Professional development in dialogic teaching: Helping teachers promote argument literacy in their classrooms. In D. Scott & E. Hargreaves (Eds.), Sage handbook of learning (pp. 219–232). London: Sage Publications.

Reznitskaya, A., & Wilkinson, I. A. G. (2017). The most reasonable answer: Helping students build better arguments together. Boston: Harvard Education Press.

Rumelhart, D. E., & Ortony, A. (1977). The representation of knowledge in memory. In R. C. Anderson, R. J. Spiro, & W. E. Montague (Eds.), Schooling and the acquisition of knowledge (pp. 99–136). Hillsdale: Erlbaum.

Shepard, L. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 1–14. | DOI 10.3102/0013189X029007004

Stein, N. L., & Trabasso, T. (1982). Children's understanding of stories: A basis for moral judgment and dilemma resolution. In C. J. Brainerd & M. Pressley (Eds.), Verbal processes in children: Progress in cognitive development research (pp. 161–188). New York: Springer-Verlag.

Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4), 1–19.

Sutton, R. E. (1992). Review of the New Jersey test of reasoning skills. In J. J. Kramer, J. C. Conoley & L. L. Murphy (Eds.), The eleventh mental measurements yearbook (pp. 606–608). Lincoln: The Buros Institute of Mental Measurements.

Toulmin, S. E. (1958). The uses of argument. Cambridge: Cambridge University Press.

Walton, D. (1998). The new dialectic: Conversational contexts of argument. Toronto: University of Toronto Press.

Weinstock, M. P., Neuman, Y., & Glassner, A. (2006). Identification of informal reasoning fallacies as a function of epistemological level, grade level, and cognitive ability. Journal of Educational Psychology, 98(2), 327–341. | DOI 10.1037/0022-0663.89.2.327

Wilkinson, I. A. G., Reznitskaya, A., Bourdage, K., Oyler, J., Nelson, K., Glina, M., Drewry, R., Kim, M.-Y. (2017). Toward a more dialogic pedagogy: Changing teachers' beliefs and practices through professional development in language arts classrooms. Language & Education, 31(1), 65–82. | DOI 10.1080/09500782.2016.1230129

Wolfe, C. R., Britt, M. A., & Butler, J. A. (2009). Argumentation schema and the my side bias in written argumentation. Written Communication, 26(2), 183–209. | DOI 10.1177/0741088309333019




 


Časopis Ústavu pedagogických věd FF MU.

Výkonná redakce: Klára Šeďová, Roman Švaříček, Zuzana Šalamounová, Martin Sedláček, Karla Brücknerová, Petr Hlaďo.

Redakční rada: Milan Pol (předseda redakční rady), Gunnar Berg, Michael Bottery, Hana Cervinkova, Theo van Dellen, Eve Eisenschmidt, Peter Gavora, Yin Cheong Cheng, Miloš Kučera, Adam Lefstein, Sami Lehesvuori, Jan Mareš, Jiří Mareš, Jiří Němec, Angelika Paseka, Jana Poláchová Vašťatková, Milada Rabušicová, Alina Reznitskaya, Michael Schratz, Martin Strouhal, Petr Svojanovský, António Teodoro, Tony Townsend, Anita Trnavčevič, Jan Vanhoof, Arnošt Veselý, Kateřina Vlčková, Eliška Walterová.

Časopis vydává čtyři čísla ročně.

ISSN 1803-7437 (print), ISSN 2336-4521 (online)