Junyan Guo is a PhD candidate at the University of Auckland, conducting research on English as a Foreign Language (EFL) students. She’s specifically focusing on Chinese EFL, and is looking at the role vocabulary plays in speaking EFL. The pandemic had disrupted her data collection plans, and she had to pivot to using online data collection tools.
Could you tell me more about what you were doing in terms of research when the pandemic happened, and how things changed for you?
I was collecting data on the specific role of vocabulary in speech in the context of English as a foreign language, specifically aiming to investigate Chinese EFL students. In the original plan for my study, participants had to take part in three vocabulary tests and a face-to-face speaking task. But when COVID-19 broke out, it was actually impossible to follow the original plan. I had to transform the original face-to-face data collection mode into an online one. I was initially hoping to be able to do it on a self-made website, so that participants can log in, and take all the tests by themselves. However, it wasn’t possible for me to do it on my own as a PhD student, and it was also difficult to hire a person who could design such a website due to limited resources. I was hoping to create a website for my data collection similar to the University of Auckland’s DELNA system, which is the university’s free English language diagnosis tool, and it is mandatory for all doctoral candidates; this turned out to be too big of a project for the context of a PhD study.
After consulting some friends specialized in computer science and web design, I received recommendations about using some existing websites for the same purpose. For example, there is a website named wenjuanwang, which is similar to SurveryMonkey, but it has a specific testing function that I can adjust to create my data collection task and fulfill my specific goals. In addition, some of its basic functions are free. I invited six participants and did a pilot on wenjuanwang to see if it worked or not. Generally speaking, the website was fine with multiple-choice and gap-filling questions, but for the speaking task participants needed to use their own recording devices and upload their recordings by themselves. In the process of this trial, I came to know an online teaching and testing system (Super Star Learning), used by many teachers in schools and universities in mainland China during the COVID-19 pandemic. It has both a web-based and an app-based version. Participants can take the tests on their computer or their phone. This system has a few advantages. First, I can set a fixed period of time for all the tests. Let’s say, in the course of three days, participant can choose to take these tests during their free time in these three days. Second, unlike surveys, I can set time limits for the tests. Once participants click on the test link, the countdown begins and the clock keeps ticking till the test ends no matter you close the website/app, finish taking the test or not. Third, if participants switch their testing page during the test, the computer will record whether and how many times they switch their page to other websites. In this case I can know if participants are trying to find the answer to a question or not. Fourth, the system has an embedded recording function. Participants can click the recording button and start recording for the speaking task. Another thing is that if I want to see the whole process of participants’ taking their tests, I need to contact the company who designed this system, so that they can set the webcam function on. This is the advanced function of this system. The premise is that participants have cameras on their devices. If I want the cameras on, I do have to obtain the participants’ consent first. I didn’t use webcam in this study because, on the one hand, the company only gives webcam consent to schools and universities for online final examinations purposes; on the other hand, there is the concern for personal privacy. But I do emphasize and restate the importance of participants’ integrity and honesty in participating in this study.
Did you notice if there was a very big difference between the way that you planned to collect the data and the way that you ended up collecting the data. Do you think that there would be a very big difference in the structure of the data?
I don’t think there is a very big difference, but I’m sure there are some differences. I haven’t analyzed all the data yet, and I just did a very general summary of all the data. Actually, one advantage of collecting data online is that I can collect data without geographical restrictions; for instance, in this study I could reach participants from different provinces of China. But when I collect data face-to-face, I can only find participants within one or two universities, due to the limitation of time and money. Another advantage is that when I build all the tests in the system, participants don’t have to finish all the tests in one go. If participants have only half an hour, they can complete the first test, and when they have another fifteen minutes, they can complete the second one.
The disadvantage, or a big difference, in the structure of the data is probably in the speaking part. As I had to rely on participant’s devices, either their phone or their computer. the sound quality of their speech samples varied. Some of the participants just recorded what they say, but when I replayed the recordings, I could barely hear what they were talking about. They were either too far from the microphone, or the volume was too low, or there was too much background noise. I can’t control such factors. The only thing I can do is to discard this kind of data. In delivering face-to-face speaking tasks, however, I am in charge of the recording devices.
Would something like a data-collection app, for instance the one developed by Adrian Leemann and his colleagues, be a welcome alternative or an improvement, especially if you’re doing spoken data collection?
In the initial plan, video conferencing software like Zoom was indeed an option. The problem is that my participants are in different time zones, and I have quite a few participants. I need to make a detailed timetable to guide them or talk to them one by one, and the whole process may take much longer than I expected. Another reason is that this is just the first stage of my study. In the following stages, I will invite raters to assess all the speech samples, and I will interview both raters and a few participants one by one about the participants’ performances in the speaking task. If the first stage lasts lasts too long, which would happen if I guide them one by one, it is possible that in the interview they have totally forgotten their experiences in taking these tests at that moment. That’s a problem.
What is your personal experience or comment, or where do you think things are going to go in the future with this kind of situation? What are your thoughts or feelings or experience about how this affects researchers?
Every coin has two sides. We can’t say online teaching or online data collection or online testing are better or worse than the traditional method. It’s another direction we should work on. One thing is that future collaboration or cooperation with researchers in different disciplines and faculties, computer science experts, or web designers in my case, is possible and should be encouraged. Because my focus is on speaking, I think e-speech data collection in the context of linguistic data, as well as e-assessment, is one of the directions for the future. For example, in my study, a great majority of participants find it very difficult to practise speaking or test their speaking proficiency by themselves, because they are in an EFL environment. Nowadays with all the apps, it’s possible. If we don’t have the opportunities to practise with native speakers, at least we can practise on our computer and phones. When we speak to a computer or a phone and they can record what we say and give some comments on how to improve.