Projects

1. Language Assessment

Projects in language assessment explore issues in the development use and validation of assessments for a variety of purposes.

• Carol Chapelle is working on increasing the use of argument-based validity in language testing as well as in educational and psychological testing. Her forthcoming book, Argument-based validity in testing and assessment, presents the academic and professional concepts for developing and interpreting validity arguments. It is intended as a reference that describes the basics of argument-based validity to a broad audience. A second volume, Validity argument in language testing: Case studies of argument-based validation research (edited with Erik Voss, forthcoming) presents validity argument as it has been used in language testing and illustrates the use of validity argument in chapters in multiple different language assessment projects.

• Elena Cotos‘ and Yooree Chung’s project on validating the interpretation of TOEFL iBT® Speaking Scores investigates functional language use in the lecture, recitation, and lab curriculum genres of the academic ITA contexts to determine whether the language functions elicited by TOEFL iBT® Speaking tasks are representative of language functions used in the target domain. Implications of relevance to theory and research are also expected, as the results may inform task design, test use, and further investigations of the validity of TOEFL iBT® for instructional settings. This work is funded by Educational Testing Service.

• Elena Cotos is working with Hyejin Yang to validate the diagnostic uses of R-PLAT, a web-based rating platform used in the Oral English Certification Test (OECT) at ISU. R-PLAT captures multiple aspects of language performance that can serve as diagnostic evidence. Empirical insights are sought to support the judgmental assumption that the diagnostic descriptors recorded via R-PLAT are indicative of target speaking ability levels used for placement into proficiency-based levels of the course.

• Bethany Gray’s project on the longitudinal development of grammatical complexity in speech and writing analyzes data from the TOEFL iBT. The project documents changes in the frequency of use and accuracy of phrasal complexity features (which are more common in written language) and clausal complexity features (which are more common in spoken language). The study contributes to the validity argument for the TOEFL iBT, demonstrating how language learners develop over time and how that development relates to language assessments.

• Bethany Gray and Volker Hegelheimer are building and analyzing a corpus of test responses to study test takers’ performance on the writing section of the English Placement Test (EPT). The goal is to examine the use of grammatical features associated with language complexity and language development in the EPT data, to investigate how these features are used differentially across placement levels, student level (graduate vs. undergraduate), and different prompts. The data analyzed in this project will be used to study language development, and inform the interpretation, development, and validation of the EPT.

• Gary Ockey is leading a team of graduate students in refining a performance-based oral communication assessment, which recently became part of the Iowa State University English Proficiency Test. This project involves the creation of test prompts, refinement of the rating scales and rater training procedures, setting cut scores, and aligning an assessment to a course curriculum. Embedded in this project are a number of graduate student and faculty research studies, including investigating the construct of academic oral communication, investigating the use of generalizability theory and Many-facet Rasch measurement to aid in refining rating scales and rater training procedures, examining new task types for better assessing oral communication, and setting cut scores.

• Gary Ockey and Jim Ranalli are working with a group of graduate students to set cut scores for the English Proficiency Writing Test, which is used for placement at Iowa State University. Recently, the test moved in the direction of the use of integrated read-write tasks, in an effort to better assess the academic writing construct. These task types have necessitated the need for new cut scores. Graduate student research embedded within this project includes investigating different techniques for developing cut-scores, examining the effectiveness of rater training and rating scales, and exploring the effectiveness of integrated task types.

2. Language Learning and Teaching

Applied linguistics faculty are working on projects to investigate new approaches to language learning pedagogy as well as the construction and evaluation of learning materials.

• Gulbahar Beckett is working on several research projects including investigation of English as a medium of instruction policies and practices in EFL contexts through document analyses, classroom observation, and interviews with students and faculty. Findings of these studies are expected to contribute to language policy work.

• Gulbahar Beckett is working on meta-analysis of project-based language learning research with Tammy Slater. Findings of this study are expected to contribute to experiential learning and technology integrated second language acquisition research.

• Gulbahar Beckett is co-editing, with Tammy Slater, a book currently titled “Theory, Research, and Models of Technology-Infused Project-Based Language Learning and Teaching: Focusing on Form and Function,” a sequel to Beckett & Miller (2006).“Project-based second and foreign language education: Past, present, and future.”

• Carol Chapelle is investigating implications of the connection between content and language for technology-mediated language learning materials and assessments. At the advanced level, she is working with discipline-specific language in the teaching of academic writing in English. At the beginning level, she is working with presentation of cultural narrative in French learning materials.

• Evgeny Chukharev-Hudilainen and John Levis are developing, in conjunction with Texas A & M University and funding from the National Science Foundation, a pronunciation training system using “accent conversion” (the language learner’s own voice combined with the segmentals and suprasegmental features of a matched native speaker voice). The resulting voice is called a “Golden Speaker” voice.

• Elena Cotos is leading the continuous development and evaluation of the Research Writing Tutor (RWT) [https://vimeo.com/90669213], a unique automated writing evaluation tool for teaching and learning research writing in the disciplines. This project is interdisciplinary, stimulating a fruitful collaboration among faculty and graduate students in Applied Linguistics, Computational Linguistics, Human-Computer Interaction, Computer Science, and Education. This work was funded by the Computation Advisory Committee, Graduate College, College of Engineering, and College of Liberal Arts and Sciences.

• Elena Cotos’ projects with Sarah Huffman and Stephanie Link investigate the effects of RWT modules integrated in learning environments. RWT’s affordances are examined from the perspective of data-driven learning and revision processes with a focus on their potential to foster the transfer of genre knowledge to genre writing. Currently, we are examining fundamental processes specified in cognitive writing theories by focusing on how novice writers interact with multi-level rhetorical feedback and scaffolding during revision and in what ways this interaction may create conditions necessary for enhanced metacognition and writing improvement.

• Elena Cotos and Sarah Huffman are evaluating a new model for academic writing support to graduate students and post-doctoral associates implemented by the Center for Communication Excellence. The objectives are to analyze the discipline-specific research communication needs of graduate students and post-doctoral associates; to investigate the impact of the writing support to meet these needs; and to assess the effectiveness and efficiency of the approach to training Graduate Peer Mentors.

• John Levis and ISU-PRL have developed a nonnative version of the Arctic Corpus (from Carnegie Mellon University) including speakers from varied first languages (Chinese, Arabic, Vietnamese, languages from India, Spanish, and Korean). We are now exploring how the corpus can inform speech synthesis, mispronunciation detection, and language teaching priorities. This project is being carried out in conjunction with Texas A & M University and with funding from the National Science Foundation.

• John Levis and members of ISU-PRL will begin a longitudinal study of L2 learner pronunciation (advanced L2 speakers of English). The project will explore how speakers’ pronunciation and spoken language develops and changes over two years in response to being in the L2 environment and in relation to factors that promote greater or lesser attention to using English. PhD students and others interested in pronunciation research are welcome to be involved.

• Jim Ranalli is studying how students engage with Automated Written Corrective Feedback (AWCF); specifically, the feedback provided by the popular writing-enhancement tool Grammarly. This project investigates whether students’ engagement with AWCF conforms with conditions that support writing-skills development, L2 development, or both, as well as the individual, task, and contextual factors that influence students’ engagement with AWCF. The methodology includes the use of eye-tracking to enhance the veridicality of retrospective verbal reports.

• Jim Ranalli is investigating how use of tools for Automated Written Corrective Feedback (AWCF) affect students’ revision behaviors and features of their written output (grammatical accuracy, complexity, and writing quality) on timed writing assignments. The study compares student performances across two AWCF tools that provide feedback in real-time (i.e., while students write): the popular web-service Grammarly and the built-in proofing tools in Microsoft Word, with the latter representing a “business-as-usual” condition. The revision analyses are based on keystroke logging and screen-capture data.

• Jim Ranalli is experimenting with a new approach to second-language writing instruction called Self-regulated Learning of L2 Writing. The approach is based on a highly successful L1 writing intervention used with young learners and adults with learning disabilities—groups who, like L2 students, are challenged in their attempts to monitor and manage their engagement in complex writing tasks because of shortfalls in attentional capacity. The approach emphasizes the role of formative feedback on students’ cognitive and metacognitive activities, with the former achieved through process-tracing via the revision-history feature of GoogleDocs, and the latter by means of reflection tasks timed to coincide with key junctures in the self-regulated learning cycle.

• Tammy Slater and Barbara Schwarte are conducting research into how ESL (English as a second language) teachers make sense of, value, and apply linguistic concepts addressed in their preparatory programs. While having a strong foundation in linguistics remains a core requirement in many ESL teacher programs, there has been a lack of research on how teaching linguistic concepts to pre-service ESL teachers impacts their teaching experience as full-time ESL teachers. The investigation of this topic will provide insight into (1) the beliefs that ESL teachers have about how to use and apply linguistic concepts in their instruction and (2) the variation in these beliefs among pre-service (both early and late program) and experienced ESL teachers. In other words, the research sets out to understand what beliefs and linguistic concepts have/have not transferred from the teacher education program to the actual teaching experience, and which of these appear to be the most important to those learning and using the concepts.

• Tammy Slater is working on an edited book (Equinox Publishers) that looks at research and practice into language-and-content integration using Mohan’s (1986) knowledge framework, a theoretical perspective based on Systemic Functional Linguistics. The collection focuses on Mohan’s theory as it has influenced work in higher education.

3. Linguistic Analysis

Projects in linguistic analysis produce knowledge about language use that contribute to the scientific goal of understanding human language in a manner that can contribute to language teaching and assessment.

• Bethany Gray and collaborators Jesse Egbert and Doug Biber (both Northern Arizona University) are working on a volume empirically exploring corpus representativeness. In the project, they investigate how corpus design decisions (such as size and sampling method) influence corpus representativeness. The goal of the project is to raise awareness of the importance of corpus representativeness, and propose methods for empirically evaluating the representativeness of language corpora.

• Bethany Gray is working on how to identify ‘text types’ within the research article register by looking at the interaction of discipline and research method. She will use dimension/factor scores from the a previous multi-dimensional analysis as the dependent variables for the cluster analysis, which groups together texts which behave similarly (i.e., which use these linguistic features in similar ways). The groups are then analyzed to identify why they use these features in similar ways (e.g., one possibility is that history writing, and qualitative political science writing, are similar in some ways, and that theoretical philosophy and theoretical physics articles are similar in other ways). The goal of the analysis is to uncover underlying factors which result in linguistically similar and dissimilar texts, without arbitrarily saying that texts are similar just because they belong to the same discipline or research type.

• Bethany Gray is building and analyzing additional subcorpora for the Academic Journal Register Corpus, a collection of 270 quantitative, qualitative, and theoretical research articles in philosophy, history, political science, applied linguistics, biology, and physics. The new subcorpora will add 180 texts representing articles with an evaluative purpose (rather than reporting primary research), such as forums, state-of-the-art synthesis, and article commentaries and author responses. Situational and linguistic analyses are carried out on the resulting corpus, adding to our knowledge of disciplinary variation beyond primary research reporting, and exploring how these articles contribute to knowledge-building and debate within their respective disciplines.

• Bethany Gray and her collaborators are investigating quantitative methods for characterizing discontinuous lexical frames, a type of formulaic language where words form a ‘frame’ around a variable slot (e.g., the * of the, in the * of). The project explores a range of measures of predictability and variability such as type-token ratios, proportions of most frequent fillers, mutual information, entropy, and ∆p (delta p) for c. 500 4-word frames) occurring at least 40 times per million words in large corpora of conversation and academic writing (c. 4.5 and 5.3 million words respectively from the Longman Corpus of Spoken and Written English).

• Elena Cotos is collaborating with Sowmya Vajjala and Carol Chapelle to identify discipline-specific patterns in scientific writing by applying approaches from machine learning and natural language processing. Their current study explores different algorithms for both general move/step models and for discipline-specific ones, as well as patterns and variation within and across the Methods sections of thirty disciplines in the RWT corpus.

• Elena Cotos and Ágnes Sandor from the Xerox Research Centre Europe (XRCE) are using the RWT corpus to test an integrated method for the automated analysis of rhetorical intent. Motivated by the need to develop theoretically-grounded methods for automated analysis of academic discourse, this study integrates genre analysis and concept-matching frameworks to explore the realization of rhetorical intent through functional language from two perspectives: interpreted as moves/steps and represented as linguistically-instantiated patterns of concepts.

• Elena Cotos is collaborating with David Kaufer, Suguru Ishizaki, and Xizhen Cai from Carnegie Mellon University on a study that integrates EAP genre theory and the rhetorical theory of representational language effects. The purpose is to identify and describe patterns of language that prime interactive experiences in scientific writing. This work merges methodological paradigms by analyzing the RWT corpus annotated for moves and steps in the DocuScope text analysis environment, which allows for examining the semantic variation that distinguishes the functional meanings of rhetorical moves in disciplinary discourse.

• Elena Cotos and Sarah Huffman are analyzing functional language indicative of study design and mapping study design to rhetorical moves and steps to inform the development of customized rules for automated detection of study design elements. This work is part of the AFLEX project, a large-scale cross-disciplinary project. The AFLEX team aims to develop research synthesis technologies with the long-term goal of improving the translation of research findings from scientists to society. The project is funded through ISU’s Presidential Initiative for Interdisciplinary in Data Driven Science.

• Elena Cotos is leading several corpus-based move analysis studies aimed to comprehensively describe the conventions of different academic and professional genres. Her research teams are engaged in top-down analysis of corpora representative of the following genres: research article, grant proposal, cover letter, teaching philosophy, and research statement. The study of Broader Impacts sections in grant proposals was supported by the National Science Foundation under grant number HRD-0963584. Corpora of cover letters, teaching philosophies, and research statements are compiled in collaboration with ISU’s Center for Excellence in Learning and Teaching, which is also funding the research.

4. Applied Linguistics

Projects in applied linguistics cut across multiple areas of the field to address language-related issues and produce and disseminate knowledge about the field itself.

• Carol Chapelle is working on a volume entitled the Concise Encyclopedia of Applied Linguistics (forthcoming), which includes updated versions of the most frequently accessed entries from the electronic version of the Encyclopedia of Applied Linguistics (Wiley, 2013). She continues to edit updates for the Encyclopedia, which is being used by over 750 libraries and professional organizations throughout 45 countries worldwide.