Implementing the Flipped Approach in an English-Medium-Instruction Educational Psychology Course : Student Learning Outcome and Perception

Academic courses taught in English are already difficult for many EFL students. This is especially the case if the class is conducted primarily through lectures. The aim of this study is to show that the flipped approach (FA) in an English-medium-instruction (EMI) course may be an alternative to the traditional teacher-lecture course for EFL undergraduates, if certain weaknesses in the approach are addressed. The students overall had positive perceptions of the FA, but perhaps more important and less frequently noted is that, by using the FA approach, EFL instructors can more clearly identify student-learning challenges. It is not clear, however, if the FA approach can subsequently address these challenges. Therefore, some appropriate pedagogical suggestions and shortcomings are given for those who plan to implement the FA in an EMI course.


INTRODUCTION
As English has become the dominant lingua franca for international conferences, global business, and academic research, English-medium instruction (EMI) has become a necessary aspect of education where English is not spoken as the native language. EMI is regarded by many non-English speaking countries as a privileged form of linguistic capital for developing advanced English skills, enhancing academic English literacy (reading and writing) and communication skills (speaking and listening) (Sah & Li, 2018). However, this has created added challenges for the EFL students who then must understand a second language just to comprehend the content being taught. These students are especially at a disadvantage when the content is given through a series of academic lectures. Although such a conclusion would seem obvious, what is not so widely recognized is that EFL student difficulties are exacerbated because academic English is a different genre from the common English used in general by a population at large. Many students may perform well in courses where the focus is not on English for academic purposes (EAP) but may be overwhelmed in classes where English is taught in the more-formal style used by instructors and that students are expected to use for oral presentations and discussions and in the writing of their essays, as well as for much more involved research reports, theses, and dissertations. Even though researchers have noticed the value of using English as a medium in higher education, many studies argued that EMI may not be effective because it imposes a high cognitive load (Sweller, Ayres, & Kalyuga, 2011 the poor English language proficiency of many EFL students becomes a critical problem that sets the students up for failure before they can even comprehend the content being conveyed by the language (Chang, 2010). Traditional lectured-based instruction further worsens this problem. In this situation, students need to process content knowledge and language simultaneously. Faced with this double challenge, which is not being experienced by their native-speaking classmates, EFL students may likely fall behind in EMI courses. In order to maximize learning opportunities for these students, traditional lecture-based models of teaching need to be replaced. One suggestion has been to flip the classroom (Berrett, 2012).
Moravec, Williams, Aguilar-Roca, and O'Dowd (2010) and later Alvarez (2011) have given a simple explanation of a flipped classroom by stating that the FA replaces after-lecture homework with the expectation that students study course material prior to class. Class time is dedicated to practice assignments, targeted remedial help, or activities designed to promote higher-order thinking skills (Khan, 2012). In a flipped classroom, the instructor's lecture is recorded and uploaded to an online site where the students watch the lecture before coming to class. Because the lectures are recorded, students also have the opportunity to review the lecture content whenever they need. Students then gain a better understanding of the content before coming to class. Class time is used by the instructor for troubleshooting difficult concepts, answering student questions, engaging students in active learning, and creating connections to every-day life (Stone, 2012).

IJELS 8(4):1-12
This approach has been implemented in higher education across different disciplines, such as nursing education (e.g., Betihavas et al., 2016;Choi, Kim, Bang, Park, Lee, & Kim, 2015), biology (Stone, 2012), physics (e.g., Bates & Galloway, 2012) and foreign-language learning (e.g., Hsieh, Wu, & Marek, 2017;Hung, 2015). The advantages of the flipped classroom, according to Fulton (2012), are: 1. Students learn at their own pace, 2. They do "homework" in class, which gives teachers better insight into student difficulties, and 3. Classroom time can be used more effectively and creatively. Flipped-classroom materials offer sufficient prior knowledge and promoted knowledge internalization (Chang, 2016). Research results show that the FA has positive effects on student learning, critical thinking, and attitudes toward learning.
Despite the recent pervasiveness of FA studies, there is still an opportunity to expand the implementation of this approach to a broader educational sphere. For the most part, FA research has been narrow, focused predominantly on university-based STEM (science, technology, engineering, and mathematics) disciplines. Limited studies have investigated the implementation of this approach on non-STEM courses (e.g. Hsieh et al., 2017;Hung, 2015). Among these, many were conducted in second-or foreign-language classrooms but only a few studies (e.g., Choi et al., 2015) analyzed the implementation of a FA on EMI courses, such as those in education, at the tertiary level. As mentioned, the EMI class is a unique learning and teaching environment due to its combination of content knowledge and use of a second/foreign language. The emphasis, though, is not on teaching language. The instructors simply use English to deliver other course content. The implementation of the FA in this context and the outcomes when students do not speak English as their first language has not been widely researched.
This study, therefore, aims to investigate the effect of the FA instruction on EFL undergraduate student-learning outcome in an EMI course. In addition, student responses after they have learned through FA instruction are analyzed to gain insight into these students' perceptions of this relatively new teaching approach. The purpose of this study is not only to measure the learning outcomes but also to work toward a conclusion regarding the usefulness of the FA for EFL learners in EMI courses. The questions being addressed will be: 1. Does the FA instruction have an effect on EFL student learning outcome, focus mainly on academic literacy performance, in an EMI course? 2. Does the FA instruction have an effect on EFL student perception of the course?

LITERATURE REVIEW
In this section, I will first discuss the definition of EMI and some challenges EFL students may encounter when learning academic subjects in English. The second part of this section will present previous studies related to the FA.

EMI Studies
The use of EMI at universities has been expanding around the world. EMI is essentially the teaching of academic subjects in English to second-or foreign language students at schools and universities whose traditional instructional language is not English (Brown, 2014, pp. 50). In other words, EMI classes are conducted in English for the purpose of teaching school subjects, excluding language-instruction courses. But why have universities in countries where English is not spoken natively implemented EMI, thus placing a further burden on student learning? Costa and Coleman (2010) state that it helps "the growth of student mobility and the evolving epistemology of university disciplines in a globalizing academy" (p.4). For another reason, it enables EFL universities to attract international students (Hu & Lei, 2014).
With the growth of EMI, particularly in Asian countries, the effectiveness of courses taught in English has lately raised researchers' concerns. The study of student attitudes has been extensively researched (Dafouz, Camacho, & Urquia, 2015). Overwhelming findings have proved that most of the EFL students considered EMI an opportunity to improve their English proficiency (Aguilar & Muñoz, 2013;Chang, 2010;Hsieh & Kang, 2007;Lasagabaster, 2008;Tamtam, Gallagher, Olabi, & Naher, 2012;Wang, 2010;Yao, 2004). EMI advocates believe that EMI courses provide students with more exposure to the language and more chances to use English because it is used to perform academic tasks like gaining information (listening & reading) and conveying information (speaking & writing) (Ibrahim, 2001). For example, Aguilar and Muñoz (2013) compared 205 graduate students' listening and grammar skills before and after the implementation of an EMI course for a semester. They found that gains in students' listening skills reached statistical significance. Students' self-perception of improvement in listening comprehension was also observed. The findings of this study, however, were questionable because the sample size at the conclusion of the analysis was only 63. The reason for dwindling subject participation was not discussed. Park (2007) compared pre-and post-test scores of 51 Korean university students who took a linguistics class taught in English. In her study, the students showed improvements in knowledge of the content area as a result of taking an EMI course. Further, many students expressed positive opinions about the English-medium lectures with regard to their improvements in reading and listening proficiency and vocabulary. EMI certainly allowed students to read in English (textbooks and related tasks) more extensively can contribute to the success of acquiring the language. Textbook-reading, according to Ibrahim (2001), is certainly a good source for rich language input for students, which is a potential source for intake or language acquisition. Finally, Hsieh and Kang (2007) looked at the effectiveness and influences of English instruction for subject courses in Taiwanese universities. They compared two groups of students: One group received instruction in Mandarin, and the other group received instruction in English. The results showed that there was no difference in grades between the two groups. However, students receiving EMI tended to show a more Implementing the Flipped Approach in an English-Medium-Instruction Educational Psychology Course: Student Learning Outcome and Perception 3 positive learning attitude. They also felt that their English proficiency in the four skills had improved. However, studies also reported negative attitudes of EFL students towards the implementation of EMI. A large-scale survey done by a Taiwanese university asked 1,326 students from seven schools about their perceptions of EMI classes. Quantitative results showed that students overall would not recommend academic courses taught in English. Moreover, about 86% of the students would have liked summaries in Mandarin Chinese. Challenges, as reported from written feedback, included: difficulty in acquiring academic knowledge is difficult even in Chinese, not to mention in English, and that lectures in English made learning less effective. Even those with a higher proficiency level in general English courses reported that their language proficiency was still not advanced enough to learn academic content. In the same vein, Hengsadeekul, Hengsadeekul, Koul, and Kaewkuekool (2012) reviewed EMI-related literature done in a Thai context. The most common difficulty mentioned pertained to the language-proficiency issue. Students reported that instruction in English created too great a burden resulting in learning difficulties for them. Wu (2006) investigated students' attitudes towards EMI at a Taiwanese university. A survey was administered to 28 graduate students. Some advantages reported among this group of students were that it helped improve English, gave more exposure to global views and international culture, provided opportunities for expression in English, and led them to better understand English textbooks. Disadvantages mentioned were that EMI made it difficult to understand course content, prevented students from expressing themselves smoothly in class, and discouraged discussion and interaction between professors and students.
Learning difficulties in EMI courses are not exclusively limited to students with low English proficiency. Even students with high language proficiency can experience the same difficulties. As Bradfore (2012) pointed out, even students in countries with strong traditions of English-language instruction, such as Norway and the Netherlands, have reported concerns with unfamiliar vocabulary and trouble taking notes while listening in English-medium classes. This is because academic English is a different genre from daily-used English. For example, when learning academic subjects, students need to acquire conceptual knowledge, which involves specific terminology.
Therefore, it seems apparent that to aid successful implementation of EMI programs, institutions should direct attention to addressing these challenges listed above, with a specific focus on the main concern: understanding content knowledge. Even though students are generally asked to review textbook contents before coming to class, students may encounter content the first time when listening to lectures. As mentioned earlier, academic English is more challenging than general English. University students need more time to understand academic English lectures than ordinary ESL/ EFL lessons (Akbari, 2015). Moreover, university students need to develop higher-order thinking skills, such as applying, analyzing, evaluating, and creating. If students cannot understand what they are supposed to learn, there will be no chance for them to develop and apply high-order thinking skills later on when they are asked to produce their own work through oral presentations or writing. To promote EFL undergraduate students' learning in EMI courses, the use of a different teaching mode appears to be necessary.

Flipped Approach
A "flipped classroom" approach (Baker, 2000), also known as an "inverted classroom" (Lage, Platt, & Treglia, 2000), is one approach that focuses teaching activity on what the student actively does. The approach calls for active student engagement with the material (e.g., problem-solving, case studies, etc., usually in collaboration with other students) directly in the classroom while more passive activities (such as reading course notes and textbooks, as well as viewing and listening to lectures) outside of the classroom. To students, the syllabus and teaching material in a flipped class may not look particularly different to more traditional approaches, but the form of accessing the syllabus and teaching material is different. As such, a flipped class can be seen as a stepping stone to less structured and inquiry-based learning environments such as problem-based learning (Hmelo-Silver, 2004).
Benefits of the FA have been cited in previous studies. Some common advantages shown are: (a) increased student engagement, increased student-teacher feedback, and self-paced learning (Goodwin & Miller, 2013;Roehl, Reddy, & Shannon, 2013;Sadaghiani, 2012); and (b) improved student learning outcome (Ferreri & O'Connor, 2013;Missildine, Fountain, Summers, & Gosselin, 2013). For example, Ferreri and O'Connor (2013) investigated 150 student exam scores from a second-year undergraduate pharmacy course and found better scores involved with design problems. Zappe, Lieicht, Messner, Litzinger, and Lee (2009) flipped a large undergraduate architectural engineering course. Student evaluations of the course indicated that the flip had a positive impact on student learning: Students perceived the method of teaching as more effective than lecturing and reported that they enjoyed the class and benefited from watching the lecture videos outside of class. The FA in English language teaching has gained popularity among educators and researchers after 2014 (Turan & Akdag-Cimen, 2019). Hung (2015), for example, compared three intact classes of foreign-language learners on a communicative English course. By using mixed methods, Hung investigated student learning outcomes and perception. The results showed that flip lessons helped the students attain better outcomes. In addition, students with varying language-proficiency levels appreciated this approach and considered it helpful in their language learning. Similar situation can be found in some literacy studies (e.g., Abaeian & Samadi, 2016;Abedi et al., 2019;Karimi & Hamzavi, 2017). Abedi et al. (2019) compared the effect of the FA and traditional instruction on EFL learners' English composition writing. The results showed that flipped learning had a positive effect on student writing abilities. For example, students through flipped learning improved their English writing proficiency by consciously following taught strategies (p.52). Karimi and Hamzavi (2017) investigated the IJELS 8(4):1-12 effect of the FA on EFL learners' reading comprehension. The findings were in consistent with previous studies stating that the FA had a significant positive effect on EFL students' reading comprehension.
Despite these positive results, the literature reveals negatives about the approach as well. Strayer (2012), for example, compared the learning environments of a FA with the traditional classroom for an introductory statistics course. The findings demonstrated that although students in the FA were more open to cooperative learning, they were less satisfied with the teaching format than those in the traditional classroom. One of the reasons offered was that students felt they were getting lost in class activities and were not clear about what was expected of them. In Ferreri and O'Connor's (2013) study, students expressed significantly more negative comments about the student-centered classroom than with the traditional lecture despite the improved grades. They complained of inconsistent grading, unclear examination questions, and general course policies regarding examination reviews and release of instructor slide sets and examinations (p. 5). Wilson (2013) reported a lot of negative comments from the students participating in the FA. For example, they did not like the increased personal expectation for their own learning. Some students felt bored and disengaged, and some considered this method was only a benefit to those who were proactive. The inconsistencies in student perception on the FA may be due to learner motivations. Simpson and Richards (2015) and Missildine et al. (2013) reported that students initially struggled to adjust to the FA. In addition, more out-of-class preparation time for students was also linked to student dissatisfaction (Missildine et al., 2013).
The FA has been in existence within the broader educational sphere for a while now. Due to the scope of the current study, application of this approach for primary and secondary education is not included. At the tertiary level, there are many studies regarding the use of the FA and its results in many fields, such as science (Bates & Galloway, 2012;Gross et al., 2015;Kettle, 2013), math (Butt, 2014;Heuett, 2017;Love, Hodge, Grandgenett, & Swift, 2013), and healthcare training (Critz & Knight, 2013;Geist, Larimore, Rawiszer, Al Sager, 2015;Harrington, Vanden Bosch, Schoofs, Beel-Bates, & Anderson, 2015;Missildine et al., 2013). Results from applying the FA to non-STEM disciplines are limited, however. Most of the studies focused on foreign-language classrooms (Engin, 2014;Hsieh et al., 2017;Hung, 2015). Engin (2014), for instance, evaluated the impact of the FA on language learning in an academic writing course. The findings showed that student-created videos could promote second-language writing regarding English form and accuracy. Furthermore, Hsieh et al. (2017) explored the benefits of the flipped-class model for EFL learners in English oral-training classes. The results revealed that the flippedclass model not only enhanced the participants' motivation but also significantly improved their idiomatic knowledge. Nevertheless, student feedback hints that the FA approach may have some pitfalls that have not received major attention in the literature.

Research Context and Design
This comparison study was carried out with 110 freshmen at a Taiwanese university from two intact classes (49 from Class A and 61 from Class B). The students in this pre-service teaching development course were hoping to become English teachers. Thus, all courses offered in this education department were taught in English. The students were all Taiwanese with at least ten years of English-learning experience. This required course, called "Educational Psychology," was offered in the second semester of the freshman year. Students in general had taken at least two EMI courses in their first semester. The students were randomly assigned into an FA group (N = 49) and a lecture-based (LB) group (N = 61). The age range of the students was from 18 to 20 years. Most of them were female students; there were only 20 males (10 from each class).
Both groups were taught by the same instructor to minimize the influence of teaching characteristics or personality on students' learning. The teaching content covered throughout the semester was the same in both classes. The difference was in the instruction design. In the FA group, the course was structured according to the flipped-classroom format as described by Kim, Kim, Khera, and Getman (2014). Outside class, students were introduced to new theoretical content by working with an E-learning system. When students came to class, they completed activities that were designed to help them apply the content they had learned before class. Students were put in a group of four so that they could interact with each other. While students were discussing in groups, the professor walked around the classroom and listened to the students' comments to check their understanding of the contents. Often, the in-class activities required students to analyze, compare, or create based on the learned theories.
On the other hand, in the LB group, students were exposed to direct lectures. The instructor also used the same videos as presented in the FA materials to help students understand course content. After the lectures, students participated in group discussions as those in the FA class. However, since lectures occupied some class time, the students in the LB group had less time for discussion than those in the FA class.

FA instruction
Since the FA takes many forms, this study followed the strategy Abeysekera and Dawson (2015) proposed in which the instructor directs students to a PowerPoint video lecture (recorded using Screencastify) to teach them key concepts of a particular theory of educational psychology as part of their homework. Along with the PowerPoint video, students were provided some other videos and outside readings along with the learned concept. In the class, the instructor monitored student engagement in a range of problem-solving activities, which required them to apply the knowledge they Implementing the Flipped Approach in an English-Medium-Instruction Educational Psychology Course: Student Learning Outcome and Perception 5 acquired through the completion of their homework. These problem-solving or critical-thinking activities were generally done in small groups, ideally resulting in the creation of small communities of peer learners (Sweet & Michaelsen, 2012). Flipped-classroom teachers could also provide some direct instruction in response to student need (Berrett, 2012). The details of the course design were as follows: • Before class: PowerPoint videos (with audio lectures) were uploaded to the learning platform three days prior to class. Students needed to learn the content outside of the class. While watching the lectures, some quizzes were inserted in the audio clips to ensure student engagement and understanding of the lecture. Students were encouraged to take notes or write down questions while learning by themselves. The length of each PowerPoint video did not exceed 15 minutes. For weekly content, the instructor uploaded two to three videos depending on lesson content. In addition, some online videos or outside readings that could facilitate students' understanding of the content were also uploaded. Students were encouraged but not required to watch and read. • In class, the instructor: 1. Checked if students had any questions regarding the learned content, 2. Gave pop-quizzes to make sure students had learned outside class, 3. Went through the quiz answers and explained some key points of the content, 4. Designed in-class discussion for students to do problem solve, to discuss group projects, and to analyze cases in small groups. While students were discussing, student learning was monitored and checked if there were common problems among the students. 5. The instructor asked students to present their discussion results and gave feedback, 6. Summarized or pinpointed some difficult concepts after students' discussion, and 7. Cycled through the procedure once or twice until the class ended.

Lecture-based instruction
In the traditional lecture-based instruction, the instructor lets students know what content they would be learning and encouraged students to read the textbook. During class, the instructor lectured about the course content first. Along with the lectures, the instructor occasionally showed videos to help students understand the content. After the lectures, students were put into groups of four to participate in group discussion. As mentioned earlier, students in the LB class were given less time for discussion because the lectures took up some of the class period. After the group discussion, the instructor went over answers.

Data Collection and Analysis
Data were collected through multiple means. First, students' academic learning outcomes were examined. They were collected through students' two chapter-review quizzes (T1 and T3), as well as their midterm (T2) and final (T4) scores. Students from the two classes took the same quizzes and exams. The question types in the exams focused mainly on students' academic literacy performance, included multiple choice, matching, short-answer questions (case analysis), and essay questions. The multiple choice section consisted of questions testing basic concepts of theories in educational psychology and some application questions. The matching section tested the meaning of terminology and the short answer questions required students to provide examples, analyze cases, and evaluate theories. Students' test scores were analyzed using the independent t-test to depict any differences between the two groups.
To understand student perception of the EMI course, students' semester-end course evaluations were used. This course evaluation was designed by the school and distributed to all students to collect their views on courses. It was a 5-Likert evaluation containing 21 questions, with 18 questions related to students' opinions about the course and the instructor. The other three questions related to student self-evaluations. Eighty-eight evaluation results were collected (47 from the LB group and 41 from the FA group). Cronbach's alpha for the 18 course evaluation items was .99, which is highly reliable.
Furthermore, to go beyond the descriptive data and understand students' experiences with the FA, two focus-group interviews were conducted. As Shorten and Smith (2017) have pointed out, a mixed-method approach allows researchers to explore diverse perspectives "that exist between the intricate layers of our multifaceted research questions" (p.74). Thus, the interviews were conducted for two purposes: (a) to ensure that data could be collected thoroughly, and (b) to triangulate quantitative results. The invitation letter was sent out to all students in the FA group, but only six students chose to participate. The interviews were conducted one week after the course ended. The interview questions focused on students' FA learning experiences and their comments on the implementation of this approach in an EMI course. The first interview lasted about one hour, and the second lasted about 45 minutes. All interviews were conducted in the students' native language and were audio recorded. Interview data were later transcribed and checked by the interviewees for member checking. After the member checking, the data were translated into English for the results. The interview data were coded in relation to three themes: the helpfulness of the FA, challenges, and suggestions. The interview data were first categorized into the three themes and later examined in conjunction with the research question.

Research question one: Does the FA instruction have an effect on EFL student learning outcome, focus mainly on academic literacy performance, when learning in an EMI course?
To see whether the FA instruction had an effect on students' learning outcome, four-time test scores were compared. The descriptive results (Table 1) show that students from the FA IJELS 8(4):1-12 group did much more poorly than their counterparts from the LB group in the first review-chapter quiz (T1). However, their scores improved gradually. By the midterm exam, they got almost the same scores as their LB counterparts. After midterm, FA group students' scores increased and even topped their LB counterparts. The plot clearly showed the slopes of the two groups. In short, the FA group started low but eventually exceeded the LB group.
To see if the differences between the two groups were significant, the independent sample t-test was computed. Results showed that the difference between the two groups was statically significant only in T1 (Table 2). Other scores did not show any statistically significant difference. This indicates that even though students in the FA group improved their academic performance gradually and even outperformed their LB counterparts on the T2 and final exam, the differences were not significant overall.

Research question 2: Does the FA instruction have an effect on EFL student perception of the EMI course?
To determine students' perceptions of the course between the two groups, descriptive statistics were initially carried out. As shown in Table 3, students in the FA group seemed to have a more positive perception of the course than those in the LB group, except on item 6. The statement of item 6 was "the teaching approach inspires my learning interest in this course." Frankly speaking, it was surprising to find that students in the FA group were less positive than those in the LB group. To see if there was any difference between the two groups, Mann-Whitney U test was conducted. There was no statistically significant difference found in any of the items ( Table 4). The results indicated that even though students in the FA group showed slightly positive perceptions of the course, the differences were not significant.

Interview Result
To depict students' opinions about the FA instruction in depth, interviews were scheduled. Interview data can be categorized according to three themes (helpfulness of the FA, challenges, suggestions), and key findings for each theme are shown in Table 5.

Reasons for the FA being helpful
All six interviewees believed the FA approach helped them learn in an EMI course. Let's face it, for students who have a tentative grasp of English because it is not their native language, using English to learn complex academic subjects can be potentially overwhelming. The FA allowed the cohort of students described here to learn at their own pace and review the online lectures if they could not understand the content initially. One student contrasted this course with other EMI courses structured according to the traditional LB style, saying that The FA allowed us to think by ourselves. In other courses, I did not have time to process the information in one lesson but was forced to move on to the next. The students also indicated that the FA was especially helpful if the academic subject was especially difficult. As one student commented: If the course required students to think and apply, the FA is necessary. However, if the course content was easy, and students could just listen to the teacher and understand, the FA may not be effective. All students reported that the FA increased their engagement when learning in an EMI course. Two students commented: This approach forced me to preview the content. Before I was passively sitting in the classroom and listening to the teacher. After I engaged in this approach, I developed preview habits. I tried to comprehend the content first and confirmed my understanding in class. That helped me study and understand this subject. I like flipping approach. In an EMI course, I could easily lose my concentration. Participating in this approach, I previewed the content before class and checked my own understanding in class. It helped me stay focused. I also used the pre-recorded lectures to review before the exam. Students also mentioned the benefits of group discussion in class. Students generally agreed that discussing content with other students led to better understanding about the course content. In addition, they needed to present their results after discussion. They could see results from other groups and compare their answers with other students. Finally, the instructor provided the acceptable answers. In so doing, students gained better understand and could remember the concepts better. As one student described: Even though we argued a lot in our discussion, it was a helpful process to clarify my understanding. In order to let others agree with me, I needed to know the content and be able to explain it clearly to others. When other group members had different answers, I needed to check if my understanding was incorrect.

Reasons for the FA not being helpful
As mentioned, students agreed that the FA approach helped improve their learning in an EMI course. However, when being asked to evaluate their own learning outcomes, three out of six students hesitated to claim that the FA instruction help improve their grades on quizzes or examinations. One student stated: I do think that the FA instruction helps me understand the content. But when it comes to exams, what I can understand is not equal to what I can express in English. Another student added: I did not do well on quizzes or exams because my English was not proficient enough. Sometimes I even could not understand the questions. As for short-answer questions, I knew what the answers were but just could not answer them well in English. Students also observed that a few students simply held back and relied on the more motivated students to clarify content. They did not preview nor contribute to group discussion. One student complained that The FA is only beneficial if you are motivated and complete the online lecture before class. Some group members did not even preview. They just relied on those who did their homework. In other words, students without motivation did not benefit from the FA approach. Even though the instructor gave pop quizzes to check whether students previewed the content, they still did not care much about their own performance.

Challenges when learning through the FA instruction
Despite positive opinions, students also expressed their frustration and challenges during the previewing process. Five out of six students said that they encountered difficulties at the beginning of the course as they got used to this new approach. Some students stated that they expected the instructor would lecture and go over the content again in class, but the instructor did not. As one student explained: I was totally in shock in the first class. I realized that if I did not preview the course content prior to class, I could not participate at all….I was very upset at the beginning because the content itself was difficult, but I had to understand it by myself [because the instructor would not teach the content again in class].
The students also shared their lack of experience with learning in a flipped classroom. One student explained: At first I could not understand the lecture content much. I tried to find the content from the textbook. It was very difficult because I did not read the textbook content before I watched the lecture video. I did not know in which page it was. It was very frustrating. This leads to another challenge of the FA. Students reported that extra time and effort needed to be spent on previewing the content. One student stated: I usually previewed only once but would repeat the parts that I did not understand. It usually took me about one hour to complete a video. I dictated the video lecture or copied information on slides if I could not understand.
Other students echoed this comment by saying that they usually spent more than one hour previewing the content: The content was difficult since this was an academic course [as opposed to a general English course]. Many academic words and content were difficult for us to understand. I tended to dictate the difficult parts while I was studying. It took me a long time.

Comments and suggestions
Five out of six students mentioned that the FA would have been more effective if the instructor would lecture or review some key concepts in the beginning of the class. One student explained that Academic content was very difficult. Even I previewed, I could not 100% understand. If the instructor could explain before group discussion, it would be very helpful. It would also make our group discussion more effective [because everyone at least had basic understanding about what to discuss]. The only student who disagreed argued that Learning was students' responsibility. If the instructor summarized the key points again in class, it may not be fair to those who previewed. The students also suggested that the instructor could have provided some chapter guiding questions to help them pay attention to key concepts of a chapter. One student elaborated on this: I like guiding questions because they focus on some important points we need to learn in each chapter. Another student commented: I also like guiding questions. They can be a part of our homework [in the previewing process]. In class, the instructor can go through the guiding questions and check the answers. It will help us understand the content. In summary, we know that the students in general agreed that the FA helps them learn an academic subject in English. They overall enjoyed the FA even though they encountered some learning challenges. Because of those challenges, they suggested that the instructor provided more supports or scaffolding tools to facilitate their self-studying process.

DISCUSSION
Research question one was designed to focus on the effect of the FA on EFL student learning outcomes. Quantitative results showed no statistically significant differences between the two groups except on T1 where students in the FA group received significant lower scores than those in the LB group. However, students in the FA group gradually improved their test performance. On T3 and T4, students in the FA group exceeded their counterparts in the LB group although the differences were not significant. The reason why students in the FA did not perform well at the beginning of the study may be due to their lack of familiarity with this different learning environment. As reported in the interviews, students were not used to this instructional approach. Initially, they did not know the importance of previewing and lacked strategies to learn from this learning environment. This inexperience reflected on their learning outcome on T1. However, once they found a way to learn in the FA environment, their learning outcomes improved.
It is important to note that the results here are in contradiction to some previous studies stating that the FA improved student learning outcome (Ferreri & O'Connor, 2013;Hung, 2015). This may be due to different examination types.
Interview results showed that one of the challenges the students encountered was when they needed to express their understanding in English. In other words, even though the FA facilitates students' understanding of English lectures, it does not improve the results of students' paper-and-pencil tests. As found in the interview, students even though understood video lectures had difficulties locating textbook content to go along with the lectures. Some of them took notes while watching the lecture videos. However, this type of note-taking skills may not be sufficient to answer short-answer questions appearing in the pencil-and-paper exam. English-language proficiency level, and not the flipped strategy for content learning, appears to be the major influence on test results for EFL students in an EMI course. In other research, the suggestion that FA is nearly completely successful is suspect. For example, Ferreri and O'Connor (2013) had students "role-play" their problems to demonstrate their understanding or lack thereof the pharmacist's approach to self-care patients. The evaluation did not require much reading and writing skills. Other positive results when using the FA may actually have been facilitated due to course type and less to the changed learning environment. Hung (2015) implemented the FA in a communicative English course, Karimi and Hamzavi (2017) and Abedi et al. (2019) implemented the FA in general English reading and writing courses. As mentioned in the literature review section above, general English courses are different from EMI academic courses. I believe the typical mistake of comparing apples to oranges has occurred here. The positive results of the FA in a general English course focusing on the language itself are not analogous to the positive results of the FA in academic disciplines where English is only used as a medium to convey content in other fields of study.

Implementing the Flipped Approach in an English-Medium-Instruction Educational Psychology Course: Student Learning Outcome and Perception 9
Research question two seeks to investigate the effect of the FA on students' perception of the EMI course. The descriptive results showed that although students in the FA group showed slightly more positive perceptions than their counterparts in the LB group, the differences were not statistically significant. Students in the FA group were even less positive than those in the LB groups when discussing whether this approach inspired their learning in this course. Again, the result here is different from previous studies reporting that students have expressed positive learning experiences in FA classes (Chang, 2016;Hsieh et al., 2017;Hung, 2015). Interview data here provide some possible explanations for this contradiction. As mentioned, students with no or low motivation tended to rely on the motivated students. Frequent "pop" quizzes or peer pressure did not seem to inspire those with low motivation. Since the instructor did not lecture again in class, low-motivated students would likely be underprepared and lose their interests in class discussions. Second, the content was difficult. Students may preview some parts of the content but give up in the middle because the process was so time consuming. The incentive of getting good grades was not strong enough to inspire low-motivated students to learn. This blunts the perceived effectiveness of the FA.
Nevertheless, the students interviewed still reported some positive learning experiences, which have been reported in previous studies (Lee & Wallace, 2018). The students found some elements particularly effective in helping them learn in an EMI course. First, through online videos, the students could preview and review the lecture content whenever necessary. This provides the students more opportunities and time to comprehend English-language lectures than with a traditional-lectured lesson. Second, the thought-provoking discussion questions give the motivated students a reason to participate and receive feedback. Through group interactions and the instructor's feedback, the students reinforce or clarify their understanding. They are no longer passively seated and only taking notes; instead, they actively construct and reconstruct their own learning. This is reflected by the FA students' slightly positive attitudes toward this course.
This study supports previous research that has identified excessive time consumption as one of the main weaknesses of the FA (Betihavas et al., 2016;Choi, et al., 2015). Additionally, this study identifies some reasons for this. First, leaning strategies are insufficient. This was more obvious at the beginning of the course since most of the students had not been exposed to a flipped-classroom environment before. Because of this, students tended to dictate and copy content from the PowerPoint videos. In addition, they had difficulties finding related sections in the textbook. Thus, previewing took them a long time. Eventually, the students became familiar with the learning environment as the course progressed. Furthermore, it is obviously difficult to learn an academic subject by using a language with which one is not totally familiar. Thus, EFL students, when learning an academic subject in English, need to process both language and content. In contrast, learning a subject in the native language only requires comprehension of the subject content. This is not new information, but, when the FA is involved as the learning process, these students needed to review content sections they could not understand and look up words or search information for better understanding. This can take inordinate amounts of time that may frustrate especially low-motivated students.
Finally, the students in this study made some comments regarding improvements of the FA in EMI courses. Most of them suggested that the instructor should still lecture or at least provide guiding questions along with the online lectures. In other words, students asked for more supports and scaffolding tools to help them understand the course content during the preview process and in class. This would require a modification of the FA that, essentially, would be a step back toward the traditional learning model. Even though the instructor pointed out some important concepts during the feedback section, most of the students thought those concepts should have been given at the beginning of the class. This finding is in contradiction to FA advocates who have suggested that teachers should not repeat what has been recorded in class because it is a waste of time (Alvarez, 2011;Khan, 2012;Moravec et al., 2010). However, considering the unique EMI learning contexts and the data presented here, EMI instructors may be wise to consider a quick review or a brief summary before they move on to group discussions.

CONCLUSION
Academic courses taught in English are obviously difficult for EFL students. The aim of this study was to investigate the effectiveness of implementing the FA in an EMI course. The teaching method presented here has been advocated as an alternative to a traditional teacher-lecture course because it can ostensibly be more effective for increasing EFL student learning autonomy and in-class participation. This study contributes to the literature by expanding the application of the FA to non-STEM and non-language classroom disciplines. Concerning the specific impacts of this pedagogical approach on an EMI course, the present study has yielded generally positive but not greatly significant results based on students' perceptions. Additionally, this study elucidates one of the biggest criticisms of the FA as mentioned in some previous studies that it is too time consuming for learners. Perhaps most important, this study suggests that academic literacy still need to be addressed with more emphasis in EMI courses. Even though FA can help students better comprehend English lectures, it does not automatically improve students' academic literacy abilities. Use of the FA may not be entirely successful unless EMI instructors are willing to provide academic literacy scaffolding when implementing the FA in an EMI course.
Although this study is a further step in the understanding of the FA in an EMI course, it is subject to several limitations. First, this study did not compare students' learning outcomes with different question types. Students' test scores were recorded as a sum. Since students may perform differently as they indicated in the interviews, their performances with different question types may vary. Second, no further investigation was done regarding whether or not the students maintained the self-learning habits in subsequent courses that IJELS 8(4):1-12 they acquired in the flipped course. One of the purposes of using the FA is to establish self-learning habits. If this is not occurring or if extraneous factors are discouraging students from embracing the flipped strategy going forward, then the merits of the FA should be reassessed. Further follow-up research could determine whether students maintain some FArelated study habits in subsequent courses or simply revert to the traditional learning style they are used to and reject the FA approach as a time sink with minimal upside.