Surveys and Survey Design

Types of Survey

There are two main types of survey, self-completed and interviewer-administered surveys:

Self-Completed Surveys

A self-completed questionnaire is, as the name suggests, completed by the survey subject. This may be either a postal questionnaire, or a web-based survey. Both of these are much cheaper than face-to-face interviews.

Postal questionnaires are the old-fashioned way, but they do ensure that you do not eliminate a large section of the population because they don’t have computers or internet access. On the other hand, they often have a low response rate because if people don’t complete them immediately they tend to forget about them. There is also no guarantee whether the addressee has completed the survey themselves and so it’s very important to check the accuracy and completeness of the answers before using the data for analysis.

There are now many tools available for web-based surveys, such as SurveyMonkey and Qualtrics. These make administering a survey quicker and easier, as well as cheaper, because there are no postal costs involved. However, there may be other costs involved: for example, SurveyMonkey offers a membership level allowing free surveys but they can have a maximum of 10 questions.

Web-based surveys can also improve response rates and completeness by being able to use pop-up boxes to explain difficult sections, or to prompt for missing data, and they can be customised for individuals much more easily than a postal survey. It’s also helpful to have the data in electronic form without the need for any data input before analysis.

SkillsYouNeed uses LimeSurvey an open source survey tool - take either our Interpersonal Skills Self-Assessment or What Sort of Leader are You? quiz to see this in action.

Interviewer-Administered Surveys

The alternative to self-completion is interviewer-administered surveys. These can be either face-to-face or telephone interviews. Both have their drawbacks.

Face-to-face surveys are expensive and time-consuming. However, they do allow the interviewer to check the respondent’s understanding and answer before completion, which improves the accuracy of the data, and can also mean that hard-to-reach groups can be included in the survey population. For example, those without access to technology are often older, which would introduce bias into the sample if you used a web-based survey.

Telephone surveys combine the best of both worlds: an ability to interact with the respondent without the expense of travelling. This means that the data can be accurate and complete, but is obtained more cheaply than with face-to-face interviews. However, some people object to telephone interviews, especially without prior arrangement. It is therefore probably best used for management research, where you can set up the interview in advance. There are, and have been, large scale studies that have used telephone interviews. However, it is important to consider your sample selection to avoid any bias, such as exclusion of the mobile-only population, or those who need help to have a telephone conversation.

For more about sample selection, see our page on Sampling and Sample Design.

Principles of Survey and Questionnaire Design

It is very easy to write poor survey questions. Good ones take a bit more thought and effort, but the good news is that there are five fairly straightforward principles of good survey design.

1. Each question or item should only express one idea.

It’s very easy to overcomplicate. Pare your questions back to make sure that they only cover one idea. If necessary, turn one question into two.

Example of a bad question: Do you agree that food prices are increasing to the point of unaffordability?

Someone might agree that food prices are increasing, but not to the point of unaffordability. You won’t know. Instead, make it two questions:

Do you agree that food prices are increasing?

How much do you agree that they will soon be unaffordable?

2. Avoid jargon, abbreviations and colloquialisms

3. Use simple language and expressions

Points two and three are similar, but slightly different. Whenever you write anything, you should make your meaning as plain as possible. But this is even more important in surveys.

You want to be confident that your respondents have understood the question correctly, and all in the same way. You therefore need to use simple language and avoid jargon that may either not be understood, or be understood differently, by some groups or individuals.

4. Word your questions positively

This doesn’t mean that you have to be relentlessly enthusiastic. It just means that you should ensure that your questions are worded so that if people feel good about something, they ‘agree’ with the sentence or question.

This is for two reasons:

People may miss the negative, especially if most other questions are positively worded; and
If you’re using a Likert-type scale (which answers on a scale from ‘Strongly agree’ to ‘Strongly disagree’), people may get confused about which way to reply.

Some surveys deliberately shift backwards and forwards between negative and positive statements, often about the same idea or subject. This is one way to test whether your respondents are paying attention , or just automatically ticking or clicking on ‘Agree’, but you do need to use it carefully.

For example, you might use phrases like ‘I always…’ and ‘I never…’ rather than ‘I do…’ and ‘I do not…’ because the difference is more obvious.

5. Avoid leading questions

A leading question is one which points the respondent towards a ‘right’ answer. Beware of imposing your views on others, however inadvertently.

The Golden Rule

In designing your questions, you need to keep all these five principles in mind, and then think:

“What do I want to do with the answer?”

You need to be confident that the answers that you get will enable you to do the analysis that you want, and then take the action that you need as a result.

A Worked Example

You are designing a survey for the Parent-School Association to find out how parents feel about their programme of social events.

They want to know how they can improve their events and whether people will support particular events, or would prefer new ones, so that they can plan their social timetable for the next year.

The original question was:

In the last year, have you attended any PSA events?

Yes, 5 or more

Yes, 3-4

Yes, 1-2

No, they are at inconvenient times or did not appeal

No, I’m a new parent

No, other reason (please specify)

Which events that you attended would you attend again? (Tick from the list of possible events).

This question is positively phrased, but does not give sufficient information for planning events.

The answers will tell you which events people liked, but not which they disliked, or what aspects they disliked, to support improvement. It is particularly difficult if someone ticks ‘Yes, 5 or more’ in answer to the first question, and then only one event in answer to the second.

On balance, despite the desire to avoid negative questions, it might be better to add a question about which events, if any, would not be supported in future, with an optional box for reasons. That way, you will get information that will enable the group to change their events to meet parents’ needs.

Types of Answer Scale

The answers to your questions may be two types: category scales and continuous scales.

Category scales use only a few distinctions, and may be either ordered or unordered.

Ordered scales, as the name suggests, have a numerical relationship between the possible answers, so that shuffling the scale makes a difference. An example is the UK government’s ABC1 categorisation of socio-economic status, and the Likert scales mentioned on our page on Quantitative and Qualitative Data.
Unordered, or nominal, scales include things like ethnic origin.

Continuous scales have a large number of possible responses, for example, a temperature scale.

Piloting and Testing

Finally, once you have developed your survey or questionnaire, it is worth taking a bit of time to test it.

Suitable tests come in several forms. You might, for example, share your survey with a focus group (see our page on Focus Groups for more) of experts and obtain their views on it.

Alternatively, you might carry out a small-scale pilot study. Here, you would distribute your questionnaire to perhaps 20 or 25 people with known characteristics, usually part of your study population, and test whether their replies show good reliability and validity.

You may, for example, get a large number of non-responses to one or more questions, which suggests that those questions were hard to understand and might need to be revised. Equally, if you are using a Likert scale, and you get a lot of ‘neither agree nor disagree’ responses, you may be asking about issues that are of no importance to your target population. Again, you may want to revise your questions to address other issues instead.

An Acquired Skill

Designing surveys is a skill that takes a bit of time, and plenty of practice, to develop.

You probably won’t get it right first time, but following the five principles, and asking for advice and support from those with more experience, will ensure that your skills quickly develop.

Continue to:
Simple Statistical Analysis
Analysing Qualitative Data