تعیین مدل بهینۀ تشخیصی شناختی (cdm) بخش دستور زبان مجموعۀ زبان انگلیسی کنکور سراسری کارشناسی ارشد

یکی از نوآورانه ترین ابزارهای محققان برای بهبود آموزش و ارزشیابی به خصوص درمورد آموزش های سرنوشت ساز که حاصل اتحاد روان شناسی شناختی و سنجش آموزشی است استفاده از رویکردهای تشخیصی شناختی است، اما متاسفانه این رویکردها مورد توجه و استفادۀ اندک پژوهشگران حیطۀ آموزش و ارزشیابی زبان، به خصوص دستور زبان، در ایران قرار گرفته است. از مسائل اساسی در جریان سنجش تشخیصی شناختی، تعیین مهارت‌ها و ریزمهارت‌های شناختی لازم برای پاسخ به پرسش ها و همچنین، انتخاب مدلی بهینه است. اهداف اصلی پژوهش کیفی و کمی حاضر تعیین این مهارت‌های شناختی از طریق تدوین ماتریس کیو، تعیین مدل بهینۀ تشخیصی شناختی (cdm) برای آزمون دستور زبان کنکور کارشناسی ارشد مجموعۀ زبان انگلیسی و درنهایت، شناسایی نقاط قوت و ضعف آزمودنی هاست که 5000 نفر از شرکت کنندگان سال 1396 بودند. یافته های حاصل از بررسی ادبیات موجود، ارزیابی متخصصان و پروتکل های تفکر با صدای بلند در فاز تدوین ماتریس کیو نشان می دهد، هر کدام از پرسش های آزمون دستور زبان به بررسی دو تا چهار مهارت از شش خرده مهارت تصریف زمان فعل، کاربرد فعل صحیح، کاربرد عبارات اصطلاحی، کاربرد تعدیل کننده ها، توافق و کاربرد حروف اضافه می پردازند. بررسی مدل های dina, dino , gdina از میان مدل های تشخیصی شناختی موجود نشان داد که مدل gdina با داده ها برازش مطلوب تری دارد و درنتیجه، تحلیل های تکمیلی بر روی عملکردهای این مدل انجام شد. درنهایت، با توجه به تحلیل عملکرد آزمون دهندگان مسلط و غیرمسلط، کاربردها و پیشنهاداتی برای آموزش و سنجش زبان برای آزمون‌های سرنوشت ساز همچون کنکور کارشناسی ارشد و همچنین، پژوهش‌های مشابه ارائه شده است.

The Optimal Cognitive Diagnostic Model (CDM) for the Grammar Section of MA Entrance Examination of State Universities for EFL Candidates

One of the most innovative tools for researchers in order to improve the quality of education and assessment is the application of Cognitive Diagnostic Approaches (CDA) which is the result of the unification of cognitive psychology and educational measurement; unfortunately, they are scarcely utilized by (grammar) language education and assessment researchers (in Iran). Among the most important decisions to be made in the process of cognitive diagnostic assessment are determining the (sub)skills required to respond correctly to each question as well as choosing an optimal cognitive diagnostic model. The present qualitative and quantitative study aims to develop a Qmatrix in order to identify such cognitive (sub)skills, to determine the optimal cognitivediagnostic model (CDM) for the grammar section of MA entrance examination for English majors, and to pinpoint mastery and non mastery states of the examinees who were 5000 MA entrance participants. The findings from the literature review, informants rsquo; and experts rsquo; evaluations, and thinkaloud protocols in the Qmatrix development phase revealed that each MA examination grammar item taps into between two to four of the six attributes of verb tense, correct verb usage, idiomatic expressions, modifiers, agreement, and preposition. Evaluation of three alternative models [i.e. the DeterministicInput, NoisyAnd (DINA), Deterministic Input, NoisyOrgate (DINO), and Generalized DINA (GDINA)] from among the existing cognitive diagnostic models indicated that GDINA was the best fit for the Grammar data. Considering the performance of master and nonmaster participants, the study concludes with suggestions, implications, and applications of the findings for highstakes language education and testing 1. IntroductionCognitive diagnostic assessment (CDA) is designed to measure specific cognitive skills of students, so as to provide information about their cognitive strengths and weaknesses (Leighton Gierl, 2007). Previous research on CDA in language testing mostly focused on reading comprehension and listening sections, whereas less attention has been paid to grammar. Moreover, in most of the previous studies (Baghaie Ravand, 2015; Clarck, 2013; Jang, 2009; Lee Sawaki, 2009; Ravand, 2015; Ravand, Barati Widhiarso, 2013) just a specified cognitive diagnostic model (CDM) was fitted to the language test data while searching for an optimum CDM was generally overlooked.Given the importance of highstakes tests such as university entrance examinations, the current research aims to apply CDA to an Iranian highstakes English grammar test to specify the underlying skills required to answer the test items correctly; furthermore, it intends to detect strengths and weaknesses of the students based on the identified skills. In doing so, searching for an optimum CDM was adopted to find the best fitting model to the second language grammar test data. 2. Literature ReviewCognitive diagnostic models are confirmatory multidimensional latent variable models with complex structures. These models let researchers propose exact hypotheses about the nature of cognitive processes that students use in response to test items (Rupp et al., 2010). Three classes of saturated, compensatory, and noncompensatory models of CDA are available to researchers. That is, a the saturated model titled generalized deterministic inputs, noisy ldquo;and rdquo; gate (GDINA), bcompensatory models (e.g., the Deterministic Input Noisy Output ldquo;OR rdquo; gate (DINO) and the additive CDM (ACDM)) which allow for other skills to contribute to the chance of responding to an item correctly, and c the noncompensatory models (e.g., The Deterministic Input Noisy Output ldquo;AND rdquo; gate (DINA) and the reduced reparametrized unified model (RRUM)), where possessing all cognitive skills is necessary to answer a test item correctly (de la Torre, 2011).In recent years, many CDM studies were conducted on data from different fields of studies including language assessment (Alavai Ranjbaran, 2018; Alderson et al., 2015; Baghaie Ravand, 2015; Clarck, 2013; Jang, 2009; Li, 2011; Lee Sawaki, 2009; Minaei et al., 2014; Moghadam et al., 2015; Park Cho, 2011; Ranjbaran Alavi, 2016; Ravand, 2015; Ravand et al., 2013; Yie, 2016, 2017). Although Park and Cho (2011) applied CDA on the English grammar for Korean EFL learners, only Yie (2017) searched for an Optimal Cognitive diagnostic model in a second language grammar test data. The latter study is very similar to the present study in many respects but the eventual cognitive diagnostic model. 3. MethodologyA retrofitting approach (Jang, 2009) to CDA was adopted to reach the cognitive diagnostic model. In doing so, at the first stage of the CDM, a Q matrix (de la Torre, 2011) was qualitatively developed based on the findings from the literature review, the viewpoints of an English language expert panel, and the cognitive processes extracted from college studentschr('39') thinkaloud protocols. The resulting Qmatrix provided all of the required skills needed to answer all of the grammar test items of the Iranian MA entrance examination for English majors. Then, DINA, DINO, and the GDINA models were empirically fitted to the grammar test data of 5000 participants through the CDM package of R (George et al., 2016). 4. ConclusionThe findings of this study showed that the saturated GDINA model was the best fitting model for the grammar data. The compensatory DINO model also fitted the data, yet the noncompensatory DINA model did not fit the grammar item response data based on the absolute model fit indices.In line with the results of Park and Cho (2011), this study also confirms that the six underlying skills including 1verb tense, 2 correct verb usage, 3 idiomatic expressions, 4 modifiers, 5 agreement, and 6 preposition encompass almost all of the required grammar skills. Moreover, verb tense skill was identified as the weakness of the students, while idiomatic expressions skill was a strength point. Altogether, in a second language context, it seems that even the students majoring in the English language do not master all of the required skills of grammar. The study concludes with suggestions, implications, and applications of the findings for highstakes language education and testing