Developing a Questionnaire

Conducting Educational Research
Developing the Questionnaire Format

When developing the questionnaire format, keep in mind that questionnaires typically have three parts: general instructions, personal information, and the body.

Part 1: General Instructions

The purpose of the general instructions is to help the person completing the questionnaire have a general understanding of the purpose of the research study, provide a general orientation of the topic of the questionnaire, and describe informed consent. As described on the Research Ethics page, participants have the right to informed consent when participating in a research study, so the researcher must explain the purpose of the research, the requirements of participation, and who is sponsoring the research. Therefore, the general instructions need to include the following:

Introduction of researcher. What is your name? What institution are you affiliated with? My name is Dr Katrina Korb and I am completing a research study from the University of Jos.
Purpose of the questionnaire. What is the purpose of the questionnaire? Take caution with this statement, though. Sometimes the introduction will state The purpose of this research is to identify academic malpractices that students engage in. This statement will likely put students on their guard and they will not be honest in how they respond. Instead, give only a vague, general description of the purpose of the questionnaire. A less biasing statement might read: The purpose of this research is to identify students' study habits. You only have to describe the purpose of the questionnaire, not necessarily the purpose of the entire study. Do not lie about the purpose of the questionnaire, but also be sure that the statement of purpose does not lead respondents to a specific response. In general, keep this purpose statement as general as possible.
Confidentiality Statement. One ethical consideration of research is that participants' responses are confidential, and participants need to know this. Your responses will be confidential. If it is possible for the participant to complete the questionnaire anonymously (meaning that they do not write their name on the questionnaire), add: Do NOT write your name on this questionnaire, so your responses will never be linked to you personally. However, as described on the Research Ethics page, this is not always possible. If participants need to write their name on the questionnaire, add: Only the researchers involved in this study will see your responses. Be specific about who those researchers are, if possible.
Voluntary participation. Another ethical consideration of research is that participants are not forced to engage in the research study. Participants need to be aware that they do not have to complete the questionnaire, and also that they do not have to answer any items they feel uncomfortable with. Your participation in this study is voluntary. If you do not want to participate, please return the questionnaire to the researcher. You also do not have to answer any question that makes you uncomfortable.
How to submit the questionnaire. If the researcher will not be present the entire time that the participant will be completing the questionnaire, tell them how to return the questionnaire to the researcher. Please return the completed questionnaire to my office, K07, by 5 September.

Part of the general instructions also includes the title of the questionnaire. Just as the directions should not lead participants to a desired response, the title also should not bias participants. Again, it is better to keep the questionnaire title as general as possible. A sample of general instructions is:

Part 2: Personal Information

The specific personal information items were developed in the last step, Writing Questionnaire Items. However, there are a few formatting points that need to be made regarding the personal information.

Label response options for coding. If you want to analyze your data on the computer, then you will need to code the data. This will be described more fully in Data Coding. Briefly, the general idea of coding is to attach symbols to responses. For example, you won't want to type Male, Male, Male, Male for every male in the sample. Instead, it is easier to attach a symbol to the Male response (maybe a, M, or 1). Therefore, it is oftentimes easiest to add a label to each response, like a multiple choice test. For example: What is you age? a) Under 10, b) 10-11, c) 12-13, d) 14-15, e) 16-17, f) 18 or older. Then, when entering data in the computer, you can simply type a, b, or c.
Instruct participants to circle, not tick. Asking participants to tick their responses can lead to confusion on what was really the intended response, as can be seen below.

In Item 3, does the participant mean that they teach at a Government School, or at an NGO Private school? The tick looks like it could go with either response. To avoid this uncertainty that could lead to error, it is better to ask participants to circle their response. Oftentimes, participants will still tick even when directed to circle. To draw attention to the instruction to circle, I circle the word "circle" in the instructions.

Below is the entire Personal Information section for Korb (2010).

Note how items 1, 3, and 4 have labels so they can be coded. I also chose to have participants write their age so I could know exactly how old they are. Since the questionnaire was designed for teachers, I needed to know what type of school they taught at. Two of the research hypotheses for the study asked about the effect of level of education on teachers' beliefs, and the relationship between years of teaching experience and teachers' beliefs. Items 4 and 5 measure the variables of level of education and teaching experience. The data analysis will statistically determine whether level of education affects teachers' beliefs and whether there is a relationship between teaching experience and teachers' beliefs. Since I was examining teachers' beliefs of early literacy development, I thought that it was also important to determine their experience teaching nursery or primary school and their experiences teaching English.

Part 3: Body

Specific Instructions
Directions are required every time that participants switch how they respond. For example, the first 20 questions may use Strongly Agree, Agree, etc. The next 10 questions may use Very important, somewhat important, etc. These two sets of items require separate directions. These directions should include the following:

General directions for responding. Participants need a general idea of what to expect for the items that they will soon be responding to. Write a statement or two to prime them for what to expect in the subsequent items. For example, I included the following statement in the questionnaire about teachers' beliefs of literacy development: The following items describe statements about children learning how to read.
How to respond. Should participants circle their responses, tick their agreement, pick two or three out of a list that represents them, etc.?
Explain the response categories. Oftentimes, I will see a questionnaire that does not explain SA, A, D, or SD until the very end of the questionnaire, or not at all! Not many people who will be completing your questionnaire will have taken a research methods course. Therefore, your participants will not understand what SA stands for. What would the ordinary person think when they see these letters? These response categories must be clearly labeled before the participant has to respond.

To keep the questionnaire short and clear to participants, do not give them too much information that they do not need. For example, I oftentimes see questionnaires that read "SA=4, A=3, D=2, SD=1, for reverse coded items, SA=1, A=2, D=3, SD=4." Participants do not need to know this information; they only need to know how to respond. This information is only for the researcher when it comes time to code the questionnaire responses, as will be described in Coding Data. Indeed, providing this information will likely confuse the participant. It will also increase the time it takes to complete the questionnaire because the participant will have to read over these confusing directions and think through it, which will cause them to be more fatigued when it comes to information they actually need to know.

Organizing the Questionnaire Items
Once the items have been developed, it is time to organize and format the questionnaire items. When developing the questionnaire items, you probably wrote the items on each variable together: five intrinsic motivation items, six extrinsic motivation items, seven self esteem items, etc. However, on the final questionnaire, it is typically best to mix up the items as long as the items require the same response categories, such as Strongly Agree, Agree, etc. Mixing up the items helps the participants keep alert when reading the questionnaire, and it also masks the fact that many items are similar. Therefore, mix up items that have the same directions and response format. For example, some items may have Strongly Agree responses, and others may have Very Important responses. These two sets of items should obviously be separated. However, if items have the same directions and response formats, then they should be mixed. For example, the order of questionnaire items may be listed as follows. (Only the name of the variable that the item is measuring is listed, not the item itself.)

Intrinsic Motivation
Self Esteem
Extrinsic Motivation
Intrinsic Motivation
Self Esteem
Self Esteem
Extrinsic Motivation
Self Esteem
Intrinsic Motivation
Self Esteem
Extrinsic Motivation
Extrinsic Motivation
Self Esteem
Extrinsic Motivation
Self Esteem
Extrinsic Motivation
Intrinsic Motivation
Self Esteem

Formatting the Questionnaire Items
Participants also need to have a space to clearly indicate where they are to respond. I recently evaluated a student who had already collected data from a questionnaire using the following format. What is wrong with it?

For item 2, did the respondent tick SA or A? What about item 4? Did they intend D or SD? When it comes time for data analysis, it is impossible to know what the participant intended to tick. Guessing will result in serious error, in your data which will invalidate the study's conclusion. The best option is to throw away every questionnaire that is not clear. In this particular case, the student had to discard over 100 questionnaires because it was impossible to determine what the participant had intended to tick.

Here is another questionnaire that I recently reviewed. This questionnaire is formatted slightly better, but there is still a serious problem. There are actually three problems with the questionnaire. What is wrong with it?

First, the response categories (SA, A, D, SD) should be described before the items. Second, the serial numbers are wrong - there are two number 3's. . But most importantly, notice item 1. This "item" is really a statement of the types of items that will come. However, the participant ticked a response to this nonsensical statement. Because this statement was numbered, the participant apparently thought that it was an item that needed to be responded to. Can you imagine the confusion of the participant when they saw this questionnaire? That confusion likely confused the participant, and they probably did not respond as they would have if the questionnaire was formatted better and they did not mistake a direction for an item. Likewise, the space in between the first part and the second part (serial number 7) has a box like the participant should respond there. This, too, is quite confusing. There are two ways to avoid this problem. The first is to use a similar tabular format, but somehow indicate that there should not be a response there, by blacking it out as indicated below.

The second strategy to fix this problem is to completely reformat the questionnaire. Another problem with using this tabular format for a questionnaire will present itself when you will be coding your data. If you are going to analyze your data in the computer, you will need to code each participant's response to each item. This is typically done by assigning SA=4, A=3, D=2, SD=1. When the questionnaire is formatted as above, the tick is located in an empty box so it is more difficult to identify whether a tick should be assigned 4 or 3. This can lead to error by the researcher when doing data entry. For these reasons, I prefer to have participants circle the number that corresponds to their response as illustrated below.

Notice a couple of aspects about this questionnaire. First, the response categories are clearly described at the top. Second, SA and SD are given at the top of the numbers to remind participants of whether Agree is high (6) or low (1). Third, I left a space after item 6 to help make the place to respond more clear. When many items get lumped together, participants can sometimes get confused about which response options correspond to which items and might circle two numbers to one item, and/or skip another item. After every five items or so, I leave a blank space to make the formatting more clear. By circling the numbers, it is clear where participants are to respond to each question. Likewise, data coding is very easy when completed questionnaires are returned.

Conclusion
To summarize, when formatting a questionnaire, keep the following points in mind:

Every item should have an obvious place for each response.
Response categories (i.e., SA, A, D, SD) should be clearly described before participants respond.
Circling responses is typically better than ticking in a box.
Every set of items with a different style of responding requires a new set of directions.

Here is an example of an entire questionnaire. (Note that I cut out a number of items so you can see the entire lay-out.)

Revising and Piloting the Draft

When developing questionnaire items, it was mentioned that it is good to read over the questionnaire as if you were a participant. Likewise, when you get a final draft of the questionnaire printed, make one copy of it. Complete the copy of the questionnaire like you are the participant. Are the directions clear? Is it clear how to respond? Is it clear where to respond? Is the formatting appealing? Likewise, read over each question looking for spelling or grammatical mistakes. Make the typist correct any formatting, spelling, or grammatical errors that you find.

Once the questionnaire has been drafted and revised, revised, revised by the researcher and colleagues, then it is time to pilot the questionnaire. The purpose of the pilot study is simply to determine whether the questionnaire is effective for the purposes of the study. Therefore, you are not interested in answering the research questions or testing the hypotheses, but merely interested in whether participants understand the questionnaire, give adequate responses, and the responses can answer the research questions. The pilot study does not need to be large, and a smaller pilot study can actually be more effective because the researcher can then talk to the participants and ask them what they were confused about on the questionnaire. After getting results from the pilot study, make corrections to the questionnaire. If significant changes are necessary, then the researcher needs to conduct a second pilot study and make more changes. The pilot studies are finished when the researcher is finished making substantial changes to the questionnaire.

Return to Educational Research Steps