Abstract
Asynchronous discussions are a popular feature in online higher education as they enable instructor-student and student–student interactions at the users’ own time and pace. AI-driven discussion platforms are designed to relieve instructors of automatable tasks, e.g., low-stakes grading and post moderation. Our study investigated the validity of an AI-generated score compared to human-driven methods of evaluating student effort and the impact of instructor interaction on students’ discussion post quality. A series of within-subjects MANOVAs was conducted on 14,599 discussion posts among over 800 students across four classes to measure post ‘curiosity score’ (i.e., an AI-generated metric of post quality) and word count. After checking assumptions, one MANOVA was run for each type of instructor interaction: private coaching, public praising, and public featuring. Instructor coaching appears to impact curiosity scores and word count, with later posts being an average of 40 words longer and scoring an average of 15 points higher than the original post that received instructor coaching. AI-driven tools appear to free up time for more creative human interventions, particularly among instructors teaching high-enrollment classes, where a traditional discussion forum is less scalable.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The data that support the findings of this study are available from the corresponding author, A.A., upon reasonable request.
References
Abe, J. A. A. (2020). Big five, linguistic styles, and successful online learning. The Internet and Higher Education, 45. https://doi.org/10.1016/j.iheduc.2019.100724
Aloni, M., & Harrington, C. (2018). Research based practices for improving the effectiveness of asynchronous online discussion boards. Scholarship of Teaching and Learning in Psychology, 4(4), 271–289. https://doi.org/10.1037/stl0000121
Arbaugh, J. B. (2010). Sage, guide, both, or even more? An examination of instructor activity in online MBA courses. Computers & Education, 55(3), 1234–1244. https://doi.org/10.1016/j.compedu.2010.05.020
Bryant, J., Heitz, C., Sanghvi, S., & Waple, D. (2020). Artificial intelligence in education: How will it impact K-12 teachers | McKinsey. Retrieved April 10, 2020 from https://www.mckinsey.com/industries/education/our-insights/how-artificial-intelligence-will-impact-k-12-teachers
Bartolic, S., Matzat, U., Tai, J., Burgess, J. L., Boud, D., Craig, H., Archibald, A., De Jaeger, A., Kaplan-Rakowski, R., Lutze-Mann, L, Polly, P., Roth, M., Heap, T., Agapito, J., & Guppy, N. (2022). Student vulnerabilities and confidence in learning in the context of the COVID-19 pandemic. Studies in Higher Education, 47(12), 2460–2472. https://doi.org/10.1080/03075079.2022.2081679
Cheng, C. K., Paré, D. E., Collimore, L.-M., & Joordens, S. (2011). Assessing the effectiveness of a voluntary online discussion forum on improving students’ course performance. Computers & Education, 56(1), 253–261. https://doi.org/10.1016/j.compedu.2010.07.024
Dahlstrom-Hakki, I., Alstad, Z., & Banerjee, M. (2020). Comparing synchronous and asynchronous online discussions for students with disabilities: The impact of social presence. Computers & Education, 150. https://doi.org/10.1016/j.compedu.2020.103842
Dennen, V. P. (2008). Looking for evidence of learning: Assessment and analysis methods for online discourse. Computers in Human Behavior, 24(2), 205–219. https://doi.org/10.1016/j.chb.2007.01.010
Ding, L., Er, E., & Orey, M. (2018). An exploratory study of student engagement in gamified online discussions. Computers & Education, 120, 213–226. https://doi.org/10.1016/j.compedu.2018.02.007
Dixson, M. D. (2010). Creating effective student engagement in online courses: What do students find engaging? M. D., 10(2), 13.
Domingue, B. W., Hough, H. J., Lang, D., & Yeatman, J. (2021). Changing Patterns of Growth in Oral Reading Fluency during the COVID-19 Pandemic. Policy Analysis for California Education, PACE.
E. Gilbert, J. (2021). Equitable AI. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–2). Association for Computing Machinery. https://doi.org/10.1145/3411763.3457780
Elder, L., & Paul, R. (1998). The role of socratic questioning in thinking, teaching, and learning. The Clearing House: A Journal of Educational Strategies, Issues and Ideas, 71(5), 297–301. https://doi.org/10.1080/00098659809602729
Garrison, D. R., & Anderson, T. (2003). E-Learning in the 21st Century: A Framework for Research and Practice (0 ed.). Routledge. https://doi.org/10.4324/9780203166093
Heap, T., Hudson, C., & Archibald, A. (2020). Investigating the impact of an AI-driven discussion platform on educator perceptions and feedback. EDEN Conference Proceedings, 1, 127–136. https://doi.org/10.38069/edenconf-2020-ac0010
Hudson, C., Archibald, A., & Heap, T. (2020). Integrating an ai-driven discussion platform: The impact of platform on engagement and quality. EDEN Conference Proceedings, 1, 117–126. https://doi.org/10.38069/edenconf-2020-ac0009
Hurt, N. E., Moss, G., Bradley, C., Larson, L., Lovelace, M., Prevost, L., Riley, N., Domizi, D., & Camus, M. (2012). The ‘facebook’ effect: College students’ perceptions of online discussions in the age of social networking. International Journal for the Scholarship of Teaching and Learning, 6(2). https://doi.org/10.20429/ijsotl.2012.060210
Indrajit, R. E., & Wibawa, B. (2020). Portrait of higher education in the covid-19 period in a digital literacy perspective: a reflection on the online lecture process experience. In 2020 Fifth International Conference on Informatics and Computing (ICIC) (pp. 1–5). IEEE.
Knowlton, D. S. (2005). A taxonomy of learning through asynchronous discussion. Journal of Interactive Learning Research, 16(2), 155–177.
Lantz, J. L., Liu, J. C., & Basnyat, I. (2022). Piloting Artificial Intelligence (AI) to Facilitate Online Discussion in Large Online Classes: A Case Study [Chapter]. Cases on Innovative and Successful Uses of Digital Resources for Online Learning; IGI Global. https://doi.org/10.4018/978-1-7998-9004-1.ch009
Lee, J.-E., & Recker, M. (2021). The effects of instructors’ use of online discussions strategies on student participation and performance in university online introductory mathematics courses. Computers & Education, 162, 104084. https://doi.org/10.1016/j.compedu.2020.104084
Nedelkoska, L., & Quintini, G. (2018). Automation, skills use and training. OECD. https://doi.org/10.1787/2e2f4eea-en
O’Brien, S., & Baugh, V. P. H. (2013). Effects of minimum word counts on writing tasks. Modern Psychological Studies 7, 11–15.
Park, J. H., Schallert, D. L., Sanders, A. J. Z., Williams, K. M., Seo, E., Yu, L.-T., Vogler, J. S., Song, K., Williamson, Z. H., & Knox, M. C. (2015). Does it matter if the teacher is there?: A teacher’s contribution to emerging patterns of interactions in online classroom discussions. Computers & Education, 82, 315–328. https://doi.org/10.1016/j.compedu.2014.11.019
Peters, V. L., & Hewitt, J. (2010). An investigation of student practices in asynchronous computer conferencing courses. Computers & Education, 54(4), 951–961.
Popenici, S. A. D., & Kerr, S. (2017). Exploring the impact of artificial intelligence on teaching and learning in higher education. Research and Practice in Technology Enhanced Learning, 12(1). https://doi.org/10.1186/s41039-017-0062-8
Rochera, M. J., Engel, A., & Coll, C. (2021). The effects of teacher feedback: a case study of an online discussion forum in Higher Education. Revista de Educación a Distancia (RED), 21(67). https://doi.org/10.6018/red.476901
Salter, N. P., & Conneely, M. R. (2015). Structured and unstructured discussion forums as tools for student engagement. Computers in Human Behavior, 46, 18–25. https://doi.org/10.1016/j.chb.2014.12.037
Schartel, S. A. (2012). Giving feedback–An integral part of education. Best Practice & Research Clinical Anaesthesiology, 26(1), 77–87.
Shum, S. B., Sándor, Á., Goldsmith, R., Bass, R., & McWilliams, M. (2017). Towards reflective writing analytics: rationale, methodology and preliminary results. Journal of Learning Analytics, 4(1), 58–84. https://doi.org/10.18608/jla.2017.41.5
Skowronek, M., Gilberti, R. M., Petro, M., Sancomb, C., Maddern, S., & Jankovic, J. (2022). Inclusive STEAM education in diverse disciplines of sustainable energy and AI. Energy and AI, 7, 100124. https://doi.org/10.1016/j.egyai.2021.100124
Smith, T.W. (2019). Making the Most of Online Discussion: A Retrospective Analysis. The International Journal of Teaching and Learning in Higher Education, 31, 21–31.
Stein, D. S., & Wanstreet, C. E. (2020). E-coaching success strategies for synchronous discussions. Distance Learning, 17(4), 113–118.
Stein, D. S., Wanstreet, C. E., Slagle, P., Trinko, L. A., & Lutz, M. (2013). From ‘hello’ to higher-order thinking: The effect of coaching and feedback on online chats. The Internet and Higher Education, 16, 78–84. https://doi.org/10.1016/j.iheduc.2012.03.001
Wang, Q., & Woo, H. L. (2007). Comparing asynchronous online discussions and face-to-face discussions in a classroom setting. British Journal of Educational Technology, 38(2), 272–286. https://doi.org/10.1111/j.1467-8535.2006.00621.x
Wuttikietpaiboon, K. (2013). Engaging graduate students in rich asynchronous online discussions (2013–99090–524; Issues 11-A(E)) [ProQuest Information & Learning]. Retrieved January, 2021 from https://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=2013-99090-524&site=ehost-live
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? International Journal of Educational Technology in Higher Education, 16(1), 39. https://doi.org/10.1186/s41239-019-0171-0
Zydney, J. M., deNoyelles, A., & Kyeong-Ju Seo, K. (2012). Creating a community of inquiry in online environments: An exploratory study on the effect of a protocol on interactions within asynchronous discussions. Computers & Education, 58(1), 77–87. https://doi.org/10.1016/j.compedu.2011.07.009
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of Interests
The authors declare no affiliation or involvement with any organization or party of financial interest that presents a conflict of interest with the present research.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Archibald, A., Hudson, C., Heap, T. et al. A Validation of AI-Enabled Discussion Platform Metrics and Relationships to Student Efforts. TechTrends 67, 285–293 (2023). https://doi.org/10.1007/s11528-022-00825-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11528-022-00825-7