When the Research Design Matters in EFL Context : A Case of Different Responses to Open-response and Closed-response Questions on Motivational Orientation

This study aimed at investigating and exploring the degree of motivational orientation (i.e. instrumental and integrative motivation) among sixty Iranian TEFL students. A survey research design was used to collect both quantitative and qualitative data, using a Likert-scale questionnaire containing six open-response items. Inferential analysis of quantitative data identified the subjects of the study with integrative orientation. However, descriptive analysis of open responses in terms of coded recurrent themes indicated contradictory results. In sum, the findings suggest a gap between Iranian TEFL students’ “desirable motivation” and “desired motivation”. The researcher attributes the contradictory results not only to the social context, but also to the type of questionnaire and the psychological status of respondents. Also, the findings provide support for other components of motivation other than just instrumentality and integrativeness, mainly “personal” aspects as proposed by Benson’s model of motivation.


Iranian EFL context 1. Introducation
According to the recent trend in the field of language learning, a great deal of attention has been paid to learners' individual differences.From one perspective, the premises of social learning theory suggest that adult learners become better learners when they are given the opportunity to reflect on their learning.From nother perspective, as stated by Knowles (1980), since adult learners have different motivation and goals from children's, they take different strategies to become successful learners.Thus, it can be inferred from these premises that adults perceive learning as a process to help them meet their affective needs.Furthermore, as suggested by a number of studies (e.g.Brown, 1994;Carter and Nunan, 2001;Huitt, 2004;Powers & Sanchez, 1982), language learning and psychological factors have been found to positively affect one another.More specifically, Nunan and Lamb (1996) state that "there is a high correlation between motivation and success" (p.208).Also, investigating individual differences, Dörnyei (2005) tends to link motivation as a learner variable to the process of second language acquisition.
Viewing motivation as having strong influence on language acquisition, Gardner and Lambert (1972) came up with two categories of motivation: "Integrative" motivation is defined as having a desire to integrate with the culture of the target language while "instrumental" motivation refers to having a desire to pursue and achieve personal, social, educational and career goals.These two types of motivation are operationally known as motivational orientation.
Generally speaking, studies have shown us that language learning and motivation are positively correlated in a way that successful language learners are highly motivated, especially if they are oriented by integrative approach."Integrative" motivation has been said to have stronger influence, as opposed to "instrumental" motivation because it contributes more to long-lasting personal relationship (Crookes & Schmidt, 1991).

Review of Literature
A number of studies (e.g.Masgoret & Gardner, 2003), using Gardner and Lambert (1972 ) model of motivation, have explored learners' integrative and instrumental motivation.However, certin issues need to be taken into consideration in respect to this distinction.For instance, Dörnyei (2005) makes the point that actually instrumental motivation/orientation is not a part of Gardner's theory of motivation, but it has been introduced as only an alternative to integrative motivation/orientation.Also, the importance of the factor of social context has not been overlooked by Gardner and his associates who hold that the superiority of each orientation depends on the social situation in which they concur.For example, even in a foreign language context where the learners are dominantly motivated to learn for academic success, their "motivation remains socially grounded" (Clément, Dörnyei & Noels, 1994, p. 421).
Although the issues of motivation and orientation have been distinguished in literature, one finds very close relationship between these to the extent that one cannot be determined without drawing on the other.For instance, motivation has been referred to as the second / foreign language learners' power for attaining an overall goal while the orientation to the process of language learning reflects their persistence in making an effort to attain that goal.In other words, learners' type of motivational orientation does not necessarily indicate they have high motivation unless they make sufficient effort to succeed (Gardner, 1985;Gardner & Lambert, 1972).It is based on such assumption that MacIntyre, Baker, Clément and Conrod (2001) assert that although some learners may show high instrumental motivation (they have realized the need), they do not make much effort to attain their goal.In other words, their low effort reflects their low integrative motivation (they do not want to appreciate the target culture), indicating contradictory behaviors which will result in failure.Additionally, Brown (1994) asserts that usually a combination of both types of motivational orientation is selected by second/foreign language learners.
In addition to Gardner's model which considers motivation from the dimensions of integrativeness and instrumentality, more recent models tend to incorporate other factors.For instance, the Constructivist Model, introduced by Williams and Burden (1997), proposes that choice and decision, influenced by internal (i.e. the learner's attitudes to the subject of study) and external factors (i.e. the attitudes of the family, peers and society), play important roles in motivation.A second type of motivation, "personal model", was introduced by Benson (1991).He claims that learners' motivation simply reflects their intrinsic motivation and personal satisfaction and cannot be viewed as integrativeness or instrumentality.Altogether, due to the dynamic nature of motivation, research studies have not yet come up to a full understanding of motivation in language learning (Dörnyei, 2005).The problem can be attributed to the fact that learning a language as a foreign language cannot fit into the model prescribed for learning a language as a second language.It is in this respect that the present study seeks to examine whether such motivational orientation can be applied in an EFL context, as well.

The Significance of the Study
As literature informs us, Iranian EFL learners do not appear as proficient as they should in spite of the high rate of motivation found among them (e.g.Eslami Rasekh & Valizadeh, 2004;Hayati & Ostadan, 2008;Sadighi & Zarafshan, 2006;Vaezi, 2008).The problem can be attributed to the lack of success in meeting learners' affective variables.One way to deal with this problem is to conduct research studies to first determine the real type of motivational orientation Iranian TEFL students are identified with; then, decide for the most appropriate teaching strategies based on their needs and expectations.This study is significant in that it suggests the approh of surveing larners' motivational orientation via a combination of techniques (i,e, open-and closed-response questionnaires) to cross validate the results and have better understanding of learners' true motivation.It is very important to identify larners' motivational orientation in different settings; i.e. whether they are influenced by external factors or by internal factors.It is hypothesized that closed-response questions account for external factors because they resemble exam type of questions whereas open-response questions provide freer setting where the respondent feels more comfortable to reflect on their attitudes toward learning a foreign language.

Research Objectives and Questions
In respect to the issues discussed above, the researcher has been motivated to do the study because of the gap between the focus of the recent trend in EFL learning/teaching and what is being practiced in Iranian context.As mentioned in the introduction section, recent studies have shed light on the importance of learner individual variables so much so that by reflecting on their idiosyncratic characteristics they become more successful learners.However, most of the Iranian studies on motivation have practically failed to identify learners' true motivation.Based on the hypothesis that the aspects of language proficiency might be connected to learner variables, the present study aims at determining Iranian TEFL students' motivational orientation (i.e.instrumental and integrative).More importantly, the learners perception needs to be explored in depth to cross validate the findings.Thus, the research questions are as follow: 1. What are the motivational orientations of Iranian TEFL students?2. Is there any consistency in Iranian TEFL students' answers to closed-and open-response questions regarding their motivational orientations?

Participants
The subjects of this study comprised a total of 60 (50 females and 10 males with the age range of 19 to 26 years old) sophomore university students majoring in TEFL at Islamic Azad University, Mashhad Branch, Iran.The researcher had to attach to the protocol administered by the university and practically had no control over the sampling.So, the subjects of the study were an intact group of students who were assumed to be homogeneous since they were all in the same grade at the same learning situation.All the subjects had studied English as a compulsory course for an average of 4 to 6 hours per week over a period of seven years at high school level.

Instruments
A modified version of Gardner's (1985) Likert-scale questionnaire was employed to identify the respondents' motivational orientation (hereafter MO).The 5-point scale questionnaire was adopted from Vaezi (2008) which was reported to have acceptable rates of reliability and validity (as cited in Gardner, 1985).However, for the sample of the present study, the reliability rate was estimated and established as Cronbach Alpha = 0.801.Items 1-12 on the questionnaire are intended to investigate integrative motivation (INTG) while the last 13 questions (i.e.items 13-25) would measure the participants' utilitarian reasons for learning English (INST).As for qualitative data, 6 open-response questions were devised to elicit more in-depth responses regarding the participants' motivational perception about learning English.(See Appendix)

Procedure
The participants' different types of motivation (i.e.INST or INTG) for learning English were examined by having them complete the related questionnaire of the study.All 25 questions were rated through Likert scales ranging from 'Strongly Disagree' to 'Strongly Agree' which were coded from 1 to 5, respectively.As for the other phase of the study; i.e. qualitative aspects, the participants were instructed to carefully read the 6 open-response questions and answer them based on what they really thought and believed.
As for data analysis, the collected quantitative data were first analyzed descriptively using measures of central tendency.Then, the mean differences between INST and INTG among the participants were calculated by running a Paired-Sample t-test to determine the MO among them.
In order to organize the collected qualitative data, it was necessary to reduce the large amount of responses to arrive at smaller number of patterns.To this end, "thematic coding" was adopted and the process of data collection proceeded recurrently.The responses to each of the six questions were documented separately.Then, codes or categories were created for each of the emerging concepts.There was no pre-determined category, but they emerged from the data.This was done by highlighting the words and phrases which were connected to the concepts of MO.
Categories were extracted from the items in Likert-scale questionnaire (Appendix) in terms of INST and INTG.These categories were determined based on the definitions given by Gardner & Lambert (1972).The participants' stated reasons for learning English were determined as integrative motivation since they deal with integrating into the target community.On the other hand, the categories derived for instrumental motivation suggest the desires for utilitarian purposes such as getting a job, being successful in academic life, being regarded as and feeling important in life.Then, the data were organized accordingly.The categories are: Integrative motivation: Importance of studying English for: It should be noted here that Question #8 (Studying English is important to me) was identified as a key item which was excluded from analysis.
Each category was then divided into sub-categories representing participants' perceptions in different aspects.
For example, some of the participants' responses to open-response questions that were similar were clustered as: active participation, grouping, problem solving, interaction, etc.More specifically, having collected and then sorted out the responses, the next step was to develop a thematic coding.To do so, the answers to each question were first separated in a numerical order to be coded as either "INST" or "INTG".Then, by going through the coded responses (categories), they were examined to see what themes reoccurred.For example, those responses which were thematically similar (e.g."I like working in groups" or "I can better express myself in groups") were clustered into one coded category; i.e. "grouping", which was interpreted as an integrative issue because such a concept deals with "interaction".They were tabulated to determine the most frequent ideas the participants were sharing.As for analyzing the data, the frequency of recurrent thematic categories was calculated for both "INST" and "INTG" orientations.
Finally, the results of both quantitative and qualitative data were compared to cross-validate the findings obtained from the entire study.In this respect, as suggested by Ivankova and Creswell (2009), the strategy was to "qualify" the qualitative findings by categorizing the items into clusters or themes as well as transforming the qualitative data into quantitative (quantifying) through counting and descriptive analysis of those clusters or themes.
The use of uppercase and lowercase letters was adopted from Creswell (2002).He proposed the convention that, in a mixed-methods research, uppercase letters are used when the method (either quantitative or qualitative) is weighted more heavily (e.g.QUAN or QUAL).For the less-weighted method, lowercase letters are used.In cases when both methods are equally in balance, both are presented in uppercase.The model which best fits this study is a QUAN-QUAL.Both types of data were concurrently collected throughout the study (Gay, Mills, & Airasian, 2009).Figure 1 illustrates the model adopted for data analysis:

Results and Discussion
To provide answers to the research questions, the results of quantitative data are discussed followed by a discussion of qualitative results.First, as presented in Table 1, the mean score of motivation was calculated by dividing the total score by the number of participants (N=60).The data were obtained out of a total of 125 points.So, the participants were found to be highly motivated.The scores were calculated on the base of scales, ranging from 1 to 5. The scores for each scale were calculated by multiplying the number of responses (out of 60) for each question item by the equivalent quantitative amount of that scale; i.e. 1 to 5.
Then, there was also a need to distinguish between integrative motivation and instrumental motivation to determine the motivational orientation among participants.A paired sample t-test was used to compare the mean scores obtained by the participants on the items testing integrative motivation (first 12 items) and those which test instrumental orientation (items 13 to 25).The descriptive results are shown in Table 2. Also, Table 3 shows the results of Paired-Sample t-test.Based on the results of paired-samples t-test presented above, the probability value (Sig.(2-tailed)) of .000 is less than the specified alpha value of .05indicating a significant difference between the scores obtained for integrative and instrumental motivations.The results also show the t value of -8.50 and the degree of freedom of 59.Also, the mean increase was -5.12, with a 95 per cent confidence interval stretching from a lower bound of -6.95 to an upper bound of -4.30.Altogether, there was a statistically significant difference between instrumental motivation (mean = 47.48,SD = 5.22) and integrative motivation (mean = 53.11,SD = 4.85), t (26) = 4.98, p < .0005(two-tailed).Thus, on a quantitative basis, the results of statistical analysis suggest that the participants of the study showed more tendencies towards integrative motivation.
As for analyzing the qualitative data, "thematic coding" was incorporated to the study by using the categories derived from the Likert-scale questionnaire.Continuing the procedure, the researcher read through the participants' open responses over and over to find commonalities related to those categories.Since the respondents were still learners of English and their production of English was not error-free, the researcher corrected and restructured their responses.Some of the typical responses, followed by interpretations follow: Learning English for Real use and Interaction • "Before I entered the university, I thought I knew everything I should know about English, but now I've realized that learning English is very different.There are a lot of things people should know for communication…" This response reflects the participant's change of attitude towards learning English after entering the university.She believes in learning English for the purpose of communication.
• "Conversation classes are good for using English in real-made situations.They are very good opportunity for the students to pretend they are using English for real purposes." The respondent expresses her attitude about conversation classes which, to her, serve to the purpose of real use of English.Here, the participant reflects her positive attitude towards English for the purpose of integration.
• "I am more motivated now because I am planning to continue my studies in Canada and . . .I'll need it to interact with the people there." This particular respondent has developed more motivation about learning English for using it in her real life.Given the common trait among learners with integrative orientation, it makes sense to have such a motive.

Learning English for Understanding English Speaking People's Culture
• "Because my fiancé is an American citizen, we are planning to move to America after he gets his engineering degree in Iran.So, I thought I could be familiar with English and the native speakers' culture if I studied English."For this participant, integrating into English and the target culture is the main reason for choosing English as a major of study.

•
"…We need some ways to practically use that knowledge in more real situation like watching movies and discussing the happenings in those movies or having role plays about American or Western culture." As a suggestion for improving conversation courses, this respondent expresses her reason for learning English; i.e. real use of the language and integrating into the target community.

Learning English due to having Positive attitudes towards English and the speakers of English
• "I like conversation classes because the students are very active and creative.It is interesting to talk about a variety of topics and we can add to our information about English and the native speakers."A part of this statement reflects the respondent's positive attitude suggesting integrative motivational orientation.
• "I like them very much.They are not as boring and difficult as other courses and we have a good chance to practice our knowledge of English." The respondent expresses her concern about both personal satisfaction and utilitarian purposes like improving knowledge.

Learning English for career / job
• "I think there is always vacancy for teaching English."This statement shows the respondent's instrumental motive.Such a motive for a typical TEFL student well reflects their expected common trait; i.e. being concerned with finding a job after graduation.
• "I think teaching is an appropriate job for women …" This makes sense since the majority of TEFL students are female who are most concerned with respecting the social conventions.
Learning English for Academic success • "I need it to improve my oral skills.Besides, I had to pass it to be able to get other classes next semester."This statement shows how a typical TEFL student is concerned with utilitarian purposes.In this case, they seem to have developed a realistic perception toward learning English as a subject of study.
• "If I didn't take it this term, I would be behind the program of study."Again, this statement shows that such students reflect their most developed desire for learning English: getting a grade and passing the course.
• "I didn't want to be left behind from my friends.It is a pre-requisite course."This is another example of instrumental motive for these students.
• "I was aware of the importance of English for success at school since I was in high school.I still think all the students, especially at the university level, should know English in order to improve their knowledge about the recent developments." This kind of perception reflects best what a typical English major is like.Such students perceive learning English for academic success because the social including the educational system expects them so.

Learning English for Expected prestige
• "It's a good feeling to know an international language.Somehow, you feel superior."Such a statement cannot be interpreted as any attribute associated with integrativeness.However, maybe for these students English is a means of achieving a higher status in society.
• "Because it is considered a degree of high class in our society." If it is true that their learning is influenced by their attitude, then prestige may have a strong influence on their learning English.

Learning English for gaining personal satisfaction
• "The best thing I liked about the activities was practicing pronunciation, especially in the lab."This is expected from an English major who is very much concerned with linguistic elements of language.They gain an inner satisfaction once they realize they are making progress in developing accuracy.
• "I think conversation classes help me to improve my self-confidence." This well represents a concern about personal feelings.
• "The activities in this course book are more attractive for me than those in the other books offered by other teachers." Knowing that the course book includes a good variety of activities, the individuals' preferences of almost any kind are rightfully met.
• "As the name suggests, the idea is to practice conversation and improve both listening and speaking.They are good for that purpose."This statement represents a desire for making progress in different aspects of language, bringing about a feeling of satisfaction.
Next, the results of qualitative data were needed to be compared with those of quantitative for the purpose of cross-validation.Table 4 compares the frequency results of the participants' responses to Likert-Scale questionnaire in terms of extracted themes with those to the open-response questions by the same participants.The number of occurrence for quantitative data (closed responses) was calculated by multiplying the number of corresponding questions for each theme by the total number of participants who all answered all the items of the close-response questionnaire (i.e.N=60).It should be noted here that question 8 (learning English is important for me) was excluded from the analysis since it does not imply any particular reason or desire and there was no one-to-one correspondence in the qualitative data.Moreover, the number of responses to open questions did not necessarily correspond to the number of participants since there were some cases of overlapping for some of the categories whereas for some others the participants did not make any particular comment.Also, some "miscellaneous" responses were excluded from the analysis.Thus, the numbers obtained for quantitative data are presented out of a total of 1440 (i.e.24 items multiplied by 60 participants).As for qualitative data (open responses), the frequency of relevant answers to each thematic category was calculated out of 386; i.e. the total responses related to all determined categories.4 shows, this comparison, in contrary to the results obtained from quantitative data, suggests that the participants are mainly oriented in instrumental motivation rather than integrative motivation.In other words, the participants' in-depth open responses indicate that they have a strong desire to learn English for being successful in their academic life.Both types of responses suggest that "academic success" was the most frequently responded motive for learning English.Such a result is suggestive as the participants were studying English for a degree in teaching English.
The second most important reason for learning English can be attributed to "real use and interaction" as the results of the quantitative data suggest.Almost the same conclusion could be drawn for the qualitative data, noting that there is almost no difference between the amount of favorable responses to "personal satisfaction" (22.79%) and "real use and interaction" (22.53%).Thus, these findings provide support for other components of motivation other than just instrumentality and integrativeness, mainly "personal" aspects as proposed by Benson's model.
The findings of this mixed methods design reveal other contradictory results, as well.The participants' open responses do not support what they scored for "culture" and "prestige" as their reasons for learning English.For example, regarding "culture", there was a significant difference in responses between quantitative (16%) and qualitative data (0.51%).Also, the findings of the quantitative and qualitative data do not converge in that the quantity of closed responses which favored "personal satisfaction" (8%) is significantly different from that of qualitative phase (22.79%).Such findings serve as an indication that the participants are truly oriented in instrumental motivation, not integrative motivation.

Conclusion and Suggestions
As the findings indicate, the participants of this study were highly motivated in both instrumental and integrative orientations.The findings testify to Brown (1994) who asserts that usually ESL/EFL learners select a combination of both.However, the results of the quantitative data do not confirm that EFL learners, as opposed to ESL learners, are more likely to develop instrumental motivation (e.g.Brown, 1994;Oller & Perkins, 1978).On the other hand, according to the qualitative results, the respondents showed more tendency towards instrumental orientation.
A comparison between the qualitative findings and quantitative findings reveals that respondents' open and closed responses diverge in respect to the concepts of "culture", "prestige" and "personal satisfaction".Such results are somehow congruent with what MacIntyre et al. (2001) assert in that learner's motivational orientation is subject to change and it is usually the social context (in this study, academic life) which functions as an underlying factor for such a change (Clément, Dörnyei & Noels, 1994).
Further, the results of this study may add to the formerly established proposals by proposing that the type of surveying instrument (e.g.open-response or closed-response questionnaires) as well as the respondents' psychological status may be at work, too.When the respondents are given closed-response questions, they may associate the situation to that of sitting for an exam.Thus, they are somehow bound to choose the items which seem more appropriate according to their learned information as TEFL students whereas open-response questions may give them the impression that they are free to state what they personally believe in.In other words, as proposed by the Constructivist Model, the former situation reinforces a kind of "external factor" while the latter provides for the presentation of "internal factor".
Thus, it can be concluded that, the learners' high integrativeness is defined as their surface beliefs which are influenced by what they have learned from their studies as TEFL students.However, their underlying beliefs, as derived from their in-depth open responses, were somehow different.Borrowing Hoftstede's (1980) terms, the surface beliefs can be categorized as "desirable motivation" and the deep beliefs as "desired motivation"; i.e. what people ought to desire and what they actually desire.
Nevertheless, whether instrumentally or integratively oriented, learners' career/academic-related needs or socially/culturally-related ones should be met in the context of Foreign Language Teaching (Gardner & MacIntyre,1993).Teachers need to address both instrumental and integrative motivations as useful factors for getting students engaged in the process of learning.
There is a need to fill the gap left by the differences between desirability and reality.Iranian EFL instructors and educators are encouraged to further investigate whether any correlation can be found in reality, where the learner is not influenced by any external factor, between instrumental and integrative orientations.If so, further studies are needed to first explore whether the prevalent motivational models like Gardner's (1985) could be applied in Iranian cultural context as well.Second, it is necessary to pay more attention to taking into account students' individual differences, such as motivation, from other perspectives.For instance, EFL educators can study raising students' "desired" motivation by implementing learner-centered methodologies to EFL settings.Also, there is a need to investigate how to determine which of learner variables, at the level of affective or cognitive, would engage more Iranian EFL learners to become motivated to learn and become successful.

Table 1 .
Descriptive statistics of total motivational orientation (MO)

Table 2 .
Paired samples statistics

Table 3 .
Statistical results of paired-sample t-test for integrative and instrumental

Table 4 .
Comparison between frequency of responses to closed-response and open-response questionnaires