A Survey on the Levels of Questioning of ELT : A Case Study in an Indonesian Tertiary Education

This present study focused on examining the levels of questions in Indonesia tertiary education. A survey research was conducted in one of the private universities in North Sumatra. The English summative assessment in an undergraduate education was used as target of survey. There were a collection of questions that had been administered by four English lecturers from Faculty of English Education in Universitas Muslim Nusantara (UMN) Al-Washliyah Medan, Indonesia. The sample of research included the sixty five questions of summative tests. The qualitative content analysis which is based on Gallagher/Aschner Bloom Classification System used to categorize the questions. It was found that the lower order thinking level still dominated the question types (69%). There was only 31% higher order questions used on the summative test.


Introduction
Questioning students is regularly used by educators as a pedagogical approach and an instrument of assessment.It is an indispensable expertise which the teacher or instructors must possess; regardless their style of teaching or the lesson content (McNamara, 1981, p.106).Unluckily, Hill and Flynn (2008) reveal that remembering factual records is what many teachers mostly employ when they test the students.This practice focusing in lower order level questions might not contribute to the development of students' cognitive skill in reasoning and analyzing (pp. 48 -49).Bloom taxonomy (as cited in Forehand, 2010) established a scheme for sorting questions into different thinking domains.The six classified stages consist of knowledge, comprehension, application, analysis, synthesis, and evaluation.It is graded from the least to the most difficult logical processes.Wilen (1991) also develops the Gallagher-Aschner/Bloom hybrid system which will be employed in this study to categorize the questions level.This scheme is a simplified and has been appropriately used to examine teachers' questioning level.Unlike, the six hierarchical questions in the original Bloom taxonomy; there are only four levels of questions in this classification.The questions are mainly assigned into convergent and divergent questions.The convergent questions' purpose is to assess students' basic knowledge and understandings (lower order).On the other hand, divergent questions oblige students to employ in higher-order thinking process as seen in Table 1.Soleimani and Khairi (2016) examined Iranian testing classes quality by using recorded corpus acquired from MA testing classes and found that ten university professor in MA class used lower order thinking questions (69, 445%), medium (30, 55%) and no higher order questions thinking at all (pp.867 -868).Inamullah (2011) conducted a research in Pakistan secondary school and found that most of the questions asked are low order thinking from the total of 267 questions, 67 percent were dominated by knowledge based questions (p.151).A case study in a Chinese university carried out by Shen and Yodkhumlue (2012) also showed that during their classroom observation, teacher asked 79, 2% questions and only 20, 8% higher order questions (p.201).Shuhidan, Hamilton and D'Souza (2009) observed questions of summative assessment in the programming subject by applying Bloom Taxonomy and found out that the majority questions are in the comprehension level (p.150).In contrast, by using qualitative approach and testdriven content analysis, Tikkanen and Aksela (2012) examined cognitive skills questions of Bloom Taxonomy in Finnish Chemistry examination and found that the questions are immensely dominated by higher order cognitive skills (p.263).Okanlawon and Adeoti (2014) also used qualitative approach and a content analysis to investigate cognitive complexity of West African Senior School Certificate Chemistry Examination questions and recommended that lower order and higher order questions must be distributed evenly in the test (p.19).
However, most of the studies above had been conducted in the field of non -English lessons.The studies pertaining to the level of English summative questions made by teachers or lecturers that rarely documented.It might be related to the belief that English learners still have inability in the deep thinking (Harklau, 2000).On the other hand, communicating in English requires speakers to respond and express ideas in a natural way instead of remembering written expressions or structures.Yet, many English teachers still assume that their ESL students cannot answer the questions if it is made in higher order (Anil, 2015;pp. 47-48).Thus, this belief will affect the choice of question employed by teachers in their tests.There are some reasons why it is so pivotal for language teachers to develop the higher order thinking of questions that enforce their students do higher their analysis.It is because literate student is not only those who can read language but also those who can think critically (Nagappan, 2001).Thus, teaching skills in language classroom should also about developing students' thinking skills.Redfield and Rousseau (1981) argue that questioning students can be used to enhance critical thinking.Language achievement in skill lesson will also be progressively increased as thinking competencies are strengthened (Collin, 1991).DeWaelsche ( 2015) reveals that some factors like English proficiency limitation, socio -cultural and institutional reasons can affect Korean university students' inability to participate in higher order thinking skills.These can also be some fundamental concerns for English language instructor in Korea to formulate their questions (pp. 144 -145).Those reasons might be similarly found in some other Asian countries.Vogler (2005) also states that when it comes to verbal questioning, teachers still face problem due to having lack of knowledge and understanding of the importance of higher cognitive questions (p.98).It is not only faced by the novice teachers; because, apparently, experienced teachers also need the skill of formulating questions more than just testing factual knowledge (Pagliaro, 2011).The lack of teacher training programs which is intended to support teachers with the knowledge of questioning skills might be one of the problems (Wilen, 1991) Only a few related researches on the infusion of higher order thinking questions, particularly in Indonesian tertiary education, analyze the language tests.Some studies are commonly conducted in lower level education.For example, a survey research conducted by Iskandar & Senam (2015) found that 15 chemistry teachers in senior high schools infused only 13, 9% higher order thinking questions in their summative assessment.Sunggingwati & Nguyen (2013) also report that the Indonesian secondary school teachers' who tend to employ more low order thinking questions in reading lessons and need support to face challenges in creating higher order questions (p.89).The World Bank published a book called "Skills for the Labor Market in Indonesia: Trends in Demand, Gaps, and Supply" by Di Gropello, Kruse and Tandon (2011) and report that Indonesian secondary school graduates have gaps in English, thinking and other practical skills.It also describes that Indonesian youth lack of skill preparedness which turns into poor outcome in employment.
Thus, one of their recommendations is to develop quality and relevance of higher education.One interesting statement is "although of generally better quality, the skill of higher-education graduates also need improvement, with particular emphasis on higher order thinking skills, English, and job specific skills (p-148)".
Therefore, this present study is aimed to explore the gap by investigating the level of questions used by English Education lecturers in their summative assessment.The decision to choose summative assessment is because it has a pivotal role for it is executed to measure and record students' attainment towards the end of an instructional component (McMillan, 2008;Nitko& Brookhart, 2011).Therefore, it is imperative to see what kinds of outcome lecturers plan to have on their English skills subject.The choices of questions in their summative tests surely reflect the teaching goals that the lecturers have at the end of the term, particularly on what the type of cognitive skills they target to assess.This could also be an avenue to look at the lecturers questioning preferences which eventually lead to the evaluation because summative assessment is known to benefit educators particularly in determining how their instructional and learning objectives have been successfully met (Johnson & Jenkins, 2009).Thus, the research question of this research paper is: "Which level of question that mostly found in Indonesian's English Language Teaching (ELT) ?"

Research Methodology
This research was applied a survey technique in Universitas Muslim Nusantara (UMN) Al Washliyah Medan, Indonesia.A qualitative content analysis was utilized to categorize the levels of questions of ELT tests.Mayring (2002) argues that the qualitative content analysis objects can be any type of recorded communication (audio and video tapes, interviews transcripts, discourses, and documents).The data of this research were the documents from the collection of questionsheets of summative tests of 2015 -2016 and 2016 -2017 academic year of four English lecturers from the Faculty of Education UMN Al-Washliyah Medan.There were four lecturers chosen through three criteria, namely: 1) they have been teaching in the university at least for two years and half (it is important as this duration of teaching has supposedly given them adequate time for doing at least four successive times of summative test) 2) The subjects they teach are English but not in the area of grammar, pronunciation or vocabulary 3) Instead, four of them are teaching skill lessons such as speaking, writing and reading.The reason for excluding grammar, pronunciation or vocabulary lesson is dues to the higher order thinking questions.The skill lessons might be assumed to generate more output in higher order thinking questions.
In order to collect the data above, the researchers made the process of collecting the data in two semesters.Then, as long as that time, there were 65 questions from four English lecturers were gathered, and all the data transcribed into teacher's questioning levels.Furthermore, the data found would be analyzed by using Gallagher/Aschner Bloom Classification System.Each question would be coded into four different level of thinking skills and analyzed to find out level of question is mostly employed by English lecturers in their summative assessment.Thus, the involvement of another analysis was employed to do a countercheck of the content data analysis to increase the reliability of the study.

Finding
An analysis of the question sheets data pointed out that most of the questions asked by the lecturers were lower-order questions.There are total 65 questions observed which were taken from English reading, writing and speaking related subjects summative test sheets.Data analysis showed that only 31% of the whole questions were directed to Level 3 (Low Order Convergent) and Level 4 (High Order Convergent) questions as shown in the Table 2.The table above obviously shows that only 31% of higher order questions included.There were 5% Level 3 lower order divergent and 26% level 4 higher order divergent.Theoritically, level 3 question refers to analysis in which the lecturers require the students to analyze, to present reasoning, and to support an argument.While, level 4 is the highest level points out synthesis and evaluation.In this highest level, the lecturers demand the students to be able to produce the innovative ideas and to offer solution and practical actions towards a substantive problem.In brief, the finding slightly shows that some lecturers tend to require the students to think critically.In contrast, lower order question is visually shown in Table 3.The result of data analysis above shows that most lecturers dominantly used lower-order questions.There was 55% of level 1 lower order question.As Gallagher/Aschner Bloom Classification System stated, level 1 indicates that the questions demanded students' basic knowledge and understanding in which the lecturers tested the students solely based on the ability to memorize a particular idea.In addition, there was 14% of level 2 question found in this research which means that the lecturers have requested the students to apply some formula or pattern of structures and to exhibit understanding.In short, the comparison between high-order questions and low-order questions is described in details in Table 4.The results indicated that there was a vast difference between the number of lower-order questions and higher-order questions enquired.The outcomes in the present study are consistent with those of existing literatures (Hill and Flynn, 2008;Inamullah 2011;Soleimani and Khairi 2016).As lower-order questions ware dominantly found in this research, there are two possible underlying factors.Firstly, the lecturers of the Faculty of English Education UMN Al-Washliyah Medan Indonesia might have lack of understanding on the importance of infusing higher order questions into their summative assessment.Secondly, the lecturers probably have low perception of their own students.They possibly doubt whether students can answer the higher order questions or not.Surely, these factors will significantly affect students' critical thinking in the long term.Students need to be introduced with higher order thinking questions in order to improve their thinking ability particularly in tertiary level of education.If this is a bit overwhelming to create, balancing the lower and higher order questions in a test can be an option.Students must not only be given questions which only test them to recall information but also to evaluate and to innovate.Critical thingking skill does not only allow higher education students to think critically, but it also allows them to solve the problem, to make critical arguments and to come up with some cutting -edge ideas.These skills are surely valuable for them to compete in the future employment opportunity.

Discussion And Conclusion
After conducting data analysis, the research finding showed that four levels of higher order thinking question appeared in analyzing data, namely; level I-low order covergent, level II-high order covergent, level III-low order divergent, and level IV-high order divergent.These four levels of question are formed from six classified stages, namely; knowledge, comprehension, application, analysis, synthesis and evaluation.In fact, the most dominant questions asked by the lecturers were lower-order question from the total of 65 questions analyzed, 55% were dominated by level-I question which refers to the knowledge level as shown in data 1.

Data 1:
-What is the definition of prediction in reading comprehension?-Every body has .. (know) that she's a bad girl.(past participle form) -Give an example of fantasy and realistic fiction!Furthermore, 14% belongs to level-II which demands the students to comprehend and to apply a particular concept as seen in data 2.

Data 2 :
-Choose a subject as noun phrase and a verb, then match them with an adverb phrase on the right to make a good sentence -Find the cause and effect in this sentence.Underline the signal word!-Arrange paragraphs about your family at least 150 words!Then, there was 5 % of level-III higher order question which focuses on analysis skill as seen in data 3.This level is the least type of question chosen by the lecturers.

Data 3:
-Why do you need to study Writing Skill?Please elaborate clearly!-How do you explain the relationship between public speaking and ethical responsibility?Lastly, there is 26 % of level-IV question which demands the ability of students to synthesis, to evaluate and to create innovative ideas as seen as follows, Data 4 : -Please make AREL (argument, reasoning, examples, link-back) from these following motions!-Your lecturer asks all students to convey some suggestions relating to the topic discussed, yet all students seem to be silent including you.This situation makes the lecturer keeps encouraging you all to speak up without considering the grammar mistake while you are speaking.What do you think of this situation ?Do you have an intention to speak up ?why or why not ?
This finding of the researh is similar to research conducted by Soleimani and Khairi (2016) which found that ten university professors in MA class mostly used lower order thinking question (69,445%).Although, this result does not echo, Tikkanen and Aksela ( 2012) study reveals the questions in Finnish Chemistry examination are dominated by higher order cognitive skills.Similar research was conducted by Iskandar & Senam (2015) and it states that there was only 13,9% higher order questions made by chemistry teachers of senior high schools in their summative assesment.
English is particularly a subject which does not only ask the students to remember structures but most importantly, to express ideas.Therefore, treating students with more challenging questions which trigger their thinking skills is more encouraged.The lower order questions may be still be given but this has to be less allocated as the semester ascends.English major skill (reading, writing, speaking, and listening) tests must not be given in form of remembering information because it does not last permanently.When the students recall the answers, they basically don't create new thoughts instead they try to solely tell some exact information which have been stored in the memory.This information might have already been provided on the textbooks.This recalling factual information level does not train them to produce an idea which is conflicted to the notion of learning a language.Particularly for adult learners who are not supposed to be given a model of rote memorization on their language learning.Thus, the choice of choosing the right type of question still plays an important role in improving students' critical thinking especially in English subjects.
Critical thinking skill will absolutely give a great benefit for those who are studying in undergraduate school since the students will have the opportunity to prepare themselves for their future working life.As one of the issues is the lack of teacher's knowledge in questioning skill, therefore, they are suggested to attend workshop and training program in order to have an understanding and ability to produce different level of questions which can be instilled through either their formative or summative test.However, there must be more emphasis given on summative test as it is a final product of a particular teaching period which sums up the learning objectives of a subject.This is a pivotal benchmark for lecturers or teachers as it also highlights what elements of thinking ability they have been developing during the semester.Thus, if the tests are still dominated with lower order questions then it is hard to yield a decent outcome in terms of graduates' higher order thinking ability.Particularly, English proficiency requires students to expand their ideas critically rather than only memorizing scripted structures.This tertiary level of education is a stage for students to prepare themselves before getting into the real world of work search or further academic programs.Employing the appropriate questions will trigger them to have strong analysis and reasoning ability which have become the key capitals in finding better future opportunities in both educational or employment.

Table 2 .
Levels of Questions asked by the participant in the four observed lessons

Table 3 .
Levels of Lower-order Questions asked by the participant in the three observed lessons

Table 4 .
The Comparison of Levels of Questions