Doing research surveys

Information on research, statistics and publications - tips including how to recruit participants, gain funding, understand your results and get them published.
Post Reply
Posts: 66
Joined: Fri Aug 12, 2022 10:34 am

Doing research surveys

Post by Wiki »

If you are undertaking a survey you normally want to get as many people to respond as possible, and for participants to be representative of the group you sample. One way to do this is with a postal survey, sent out to everyone known to a service, or on a mailing list, or in a particular catchment area. However, this uses a lot of copies of questionnaires, and has costs of printing, postage, collating papers, return postage, data entry etc.

An alternative is to use electronic surveys online (eg qualtrics, survey monkey, limesurvey), and to send out the link via email or text, or to recruit via social media. This can often be cheaper and quicker, but it is easier to ignore, and there can be bias in whether users of technology or social media sites really represent your service users or clinical population, as certain demographics may be more likely to respond. Whilst there is now internet in 96% of households - some population groups may not have access to the internet or use it as readily (eg older people, those in lower socioeconomic groups, homeless people, those in certain religious or ethnic groups, and those in certain geographic areas).

There are some general rules when thinking about survey methodology, as well as more general good practice about data protection, confidentiality and good research practice. The starting point is always what you are trying to find out, who can give you that information, and how you can motivate them to engage in your research - this can be by showing the value of the research, as well as by offering incentives like a payment, or entry into a prize draw.

Recommendations for postal surveys:
- Using an identification number rather than personal details
- Design: required to write identifier only once; demographic details at the end; care with initial items (easier first); initial items addressed respondents' concerns
- Postcard reminders
- Shorter questionnaires
- Distributed by hand
- Cover letter
- (Significant enough) monetary incentive
- Stamped return envelope
- Green paper(!)
- Pre-notification letter

The best review of postal survey methodology is probably this one: Edwards et al 2002, Increasing response rates to postal questionnaires: systematic review, BMJ 2002;324:1183

Response rates:

Response rates from people with whom you have had no previous contact can be expected to be about 15-25%. Response rates from various patient groups are around 30-60%.

Response rates to advertising (maildrops, leaflets) are generally 1-2%, but can be executed in bulk, particularly by a commercial company either with newspapers (cheaper) or by themselves (a "solus" drop). There is also the advantage that most firms can target by postcode.

Response rates are lower in surveys addressing sensitive issues.

Here are some tips to maximise response rates in order to hopefully minimise bias:

Send out using first class post, and mark it on the envelope if using a franking machine.

Getting a stamp that says "private and confidential" will make it clear that the envelope is not junk mail.

The use of incentives (money, lollipop, pens) does increase response rates and may reduce bias. Financially, only a small amount is needed (~£5); more than this is not exponentially better.

Longer questionnaires have lower response rates. Questionnaires that have a stem-and-leaf format (i.e. if you answered yes to question 5, please answer questions 6,7,and 8....) are more likely to have missing data that questionnaires with entirely "self contained" items.

Enclose an SSAE, with a first class stamp or freepost business reply service. A study found response rate 6% higher using stamps. (Streiff 2001, A mail survey of United States hematologists and oncologists: A comparison of business reply versus stamped return envelopes. Journal of Clinical Epidemiology, Volume 54, Issue 4, Pages 430-432)

Paper quality has no effect on return rates. However, printing in a different colour may do

It helps to make the letter feel personal, rather than mass produced. Try to address the recipient by name, and to sign the letter using pen, rather than printing your signature, preferably in non-black ink to make it more obvious you have signed it.

When sending to patients, using the letterhead of a trusted institution can be helpful, as can mentioning that the work is in collaboration with their doctor.

Check with your ethics committee whether you are allowed to telephone a.) non-responders, b.) people with missing data

Get a pack of blank postcards, print a little label reminding people about your questionnaire or asking them to contact you if you haven't received it, send it out 2 weeks after the initial mailing.

To maximise response rates from surveys sent to GPs, it has been suggested that you should send at least three reminders to maximise your response rate (Barclay et al 2002).

Using email surveyscan be good when you need answers quickly and cheaply. You may also be more likely to get more comments to open-ended questions when there is more space provided. Obviously the internet-using population are systematically biased (eg only 54% of people over 75 have used the internet recently) so email is only suitable for some uses. It can be brilliant if you are trying to contact a lot of university undergrads, or certain kinds of professionals. However, it is very easy to accidentally recruit a biased group with snowball recruitment or online recruitment via social media or web forums, or for a group of people to forward a survey encouraging people to respond according to a particular agenda (eg the survey of parents who believe peer influences made their children trans that was represented as a survey of parents of trans young people - this critique higlights some of the methodological flaws).

Other helpful tips:

If you are posting surveys, print directly onto your return envelopes, or get some printable labels to stop you having to write out your return address 300 times.

Make sure you weigh everything down the post office. Paul Wicks, who wrote part of this wiki says "The pack I sent out to people was more than a 1st class stamp would pay for, but they were supposed to keep 90% of it and just send back one sheet so the SSAE just had a first class stamp. Sometimes people would send back the whole thing and the stamp wasn't enough, and it'd get diverted to the post office with a £1 fine on it. VERY annoying". Also remember the new postal rates charge for size as well as weight - so anything making the envelope larger or thicker than a letter can end up much more expensive.

Paul also says "A LOT of stuff seems to go missing in the post, compared to what people say. Check with your ethics guys if you're allowed to phone to check if people have received it. I'd say about 10% goes missing on the way to them, and about 5% goes missing on its way back from them to you".

For more info:

Nakash, R.A., Hutton, J.L., Jørstad-Stein, E.C. et al. Maximising response to postal questionnaires – A systematic review of randomised trials in health research. BMC Med Res Methodol 6, 5 (2006).

Viljoen, D. & Wolpert, M. (2002). Increasing return rates from postal outcome questionnaires: 10 pointers from the literature. Clinical Psychology, 19, 18-21

Note: If you have a suggestion about how to improve or add to this wiki please post it here. If you want to discuss this post please post a new thread in the forum. If you want to cite this wiki, please use the wiki title and URL. There is information about the structure, rules and copyright of the wiki here.

Content checked by a clinical psychologist on 14/09/2022
Last modified on 14/09/2022 by Miriam
Post Reply