Identifying Signals of Continuance and Discontinuance in Peer-Supported JSL Learning: A Human-LLM Qualitative Approach

Abstract

Improving the continuity of Japanese language learning among international graduate students remains a critical challenge in Japan, particularly for those enrolled in English-taught programs. Peer-based collaborative learning has been widely implemented to support these learners, yet little is known about the early-stage signals related to participation being maintained or discontinued. This study attempts to identify "process-level signals" (i.e., qualitative indicators or markers) in participants’ reflection texts that relate to later partnership sustainability through an exploratory qualitative approach. Using 36 reflection entries collected from 10 pairs at a Japanese university, human qualitative analysis identified four major categories and ten sub-categories of interactional signals. Furthermore, we evaluated the feasibility of automating signal detection using a Large Language Model (LLM) under two prompting conditions: naïve zero-shot (Prompt A) using raw definitions, and devised zero-shot (Prompt B) with refined rules. The results reveal clear differences between the two prompting strategies. While Prompt B achieved perfect recall (1.00) and substantial agreement (κ=0.78) in identifying logistical signals such as time constraints, the stricter instructions induced an over-conservative bias. Consequently, Prompt B overlooked nuanced relational and value-oriented signals captured in Prompt A. These findings suggest that different prompting strategies are required depending on signal types, with stricter rules being effective for logistical signals and more flexible settings being necessary for relational nuances.