Sampling and Recruitment 101 - EvalCentral Blog

This is an Eval Central archive copy, find the original at evalacademy.com.

This article is rated as:

You’ve got your evaluation plan; you’ve developed your data collection tools and you’re ready to go live with collecting the data you need to answer your evaluation questions. Step 1: Identify your sample. Step 2. Recruitment. But how do you get participants to take part in the data collection process?

Recruitment is often difficult. In my experience, it’s been a rare but welcome occurrence that my target sample size is reached easily. This means I’m often hustling and employing a number of strategies to boost my recruitment efforts.

Sufficient recruitment for your data collection methods is important. Poor recruitment can lead to:

Bias
insufficient sample size (poor power)
under or over-representation of specific groups
costly delays
drawing inappropriate conclusions
limited generalizability

I’ll share with you some of the strategies and methods I’ve used that help me to identify and reach my target recruitment levels for the data collection process.

Identify your Sample

Identifying your sample size and characteristics of the sample ahead of your data collection will likely help you to select your recruitment methods and will definitely help you to identify when you can stop recruiting.

Let’s start with inclusion and exclusion criteria. For example, if you’re evaluating a program, you may only want to include people who actually participated in that program (this is the inclusion criteria). If you want to evaluate the long-term impact of a service, you may want to exclude anyone who didn’t receive the full service or dropped out (this is the exclusion criteria).

Identifying these characteristics about your sample is important, but it’s also wise to reflect on what bias this may be introducing. You likely don’t want to create such strict inclusion/exclusion criteria as to end up with a sample that doesn’t actually reflect the population you are exploring. Evaluation is not research; evaluation often benefits from a more heterogenous, representative sample.

For example, you may choose not to include attendance rates of a program as an inclusion criterion to allow you to explore barriers faced in participation.

Similarly, you can’t control for everything and there may be unknown characteristics introduced in your sample.

For example, you likely can’t create an inclusion criterion for participant motivation to take part in a program, but participant motivation may impact your evaluation findings.

When identifying your sample, you’ll also want to consider if there are any inherent ethical considerations for your participant groups. For example, are they youth, or could they have any cognitive disabilities? This may require close attention to your consent process.

Access is another key consideration. Does your identified sample have access to the medium you’re using, like a phone or computer; are they likely to be available at the times you may be offering; and, are they comfortable participating in the primary language of the evaluator? If not, you may want to consider using an interpreter in your data collection process. Check out our article on Top Tips for Using a Real-Time Interpreter for Interviews and Focus Groups.

Recruiting participants who dropped out, quit, or never started a program can be very difficult. If this is part of your identified sample, you will likely need to work with program staff to implement a process where there is an exit survey, or a few questions asked by intake or administrative staff at the time of contact with (potential) participants.

Sampling Methods

There are several ways to sample! Here are some of the more common ones used in evaluation:

Random – include all individuals who fit your inclusion criteria. Random sampling means everyone in the population has an equal chance of participation. Random sampling is simple and mitigates risks of bias; however, the simplicity often means it’s difficult to get people to volunteer.
Convenience – you recruit those who are most accessible to you. For example, you may attend a program session and use the participants from that session as your sample, or you may sit in the waiting room of an office and use people who have appointments that day as your sample. Convenience sampling is also simple and likely quick, but it may introduce bias or generate a sample that doesn’t represent the population.
Snowball – using word of mouth you build your sample starting from the first participant. If you can identify one or only a small number of participants, you can use the assumption that your first participants likely know others that fit your inclusion criteria as they did. In snowball sampling, you ask your participants to help you recruit by spreading the word, or at least to help you identify other means to recruit your sample. You likely want to have recruitment cards or flyers available to give out (see an example below).
Maximum Variation – you intentionally recruit for variation. Not only do you have an identified overall sample size, but you have it broken down – for example, perhaps you want 10 participants from each of the three program sites or spanning certain ages, genders, or backgrounds. This is a favourite of mine in evaluation because I often try to capture varied experiences. The disadvantage is that sometimes recruitment can take longer, and you may need very targeted recruitment strategies.
Deviant Case – if you’re limited in your ability to recruit a sufficient sample size, you may target individuals for whom things went particularly well (the gold star cases) or those whose experiences were off track. These “deviant” cases offer experiential insight into what worked well and contributed to success, or what not to do to avoid failure in achieving intended outcomes.
Typical Case – you may choose the alternative to the deviant case, which is the “typical case” which is targeting your recruitment to describe the average experience. This can create a profile that describes what a “normal” experience is like.

Remember when I said recruiting sometimes involves hustling and using a number of strategies? You can definitely use a combination of these sampling methods!

Recruitment Strategies

Recruitment isn’t as easy as picking your sampling methods, from there, you still need to actually identify and make contact with your intended sample. Some concrete methods for doing so include:

Posters – Particularly if you are recruiting from individuals who attend a certain office or program, hanging posters in a waiting room can be effective. If you’re looking for a broader sample, like the general public, you could consider coffee shops, libraries, or community centres.
Virtual posters – most people are online, so using social media is probably one of the strongest recruitment tools. You could post to a program’s Facebook or Instagram account, or work with a communications team to Tweet about opportunities to participate.
Website pop-ups – like virtual posters, if your program has a website you can work with the web team to include a pop-up for visitors to the site inviting them to participate.
Newsletters – If you’re planning is established ahead of time, blurbs in newsletters can alert participants to upcoming opportunities.
Collaborators – I often tell my clients at our kickoff meeting that I expect them to be champions of evaluation, which includes making connections or introductions, and advocating the importance of participation in evaluation. Your evaluation advisory committee or working group, if one exists, can probably do a lot in terms of identifying staff to talk to, or customers to recruit. Just keep in mind that they can introduce their own bias and direct you to the more favourable participants.
Attendance – suitable to convenience sampling, sometimes asking permission to wait in a waiting room on a certain day or attending a program session will help in your recruitment efforts. I do find that putting a face to a name can make potential participants feel more comfortable signing up. I’ve also attended program graduation or celebration events where many participants come together. I come armed with recruitment flyers to leave on tables or hand out directly to people I meet. Here’s an example of one I’ve used:

7. Existing databases – many programs have existing databases of clients’ names and contact information. Be careful. You want to ensure you have the right to access contact information and can contact them for evaluation purposes. Often, I get program staff to do the first cold call, introducing me.

8. Recruiting for Mixed Methods – sometimes you need to recruit for multiple methods. I’ll often send out a survey and include a question at the end about a participant’s willingness to take part in a follow-up interview or focus group, or even gauge their willingness for future data collection. That way I have permission to contact them directly, with the contact information they provide.

Again, using many of these strategies will make your recruitment faster and hopefully get you the sample you need.

In most of these recruitment methods and strategies, having a link where participants can access more information or even sign up directly for an interview or focus group using tools such as Calendly will boost your chances of reaching your sample, rather than asking participants to email or phone you.

Adherence to ethical practice is important throughout recruitment. Make sure you reflect on your strategies for accessibility and inclusion, but also look for potential coercion, including reviewing your use of an incentive.

Do you have any go-to recruitment methods? Or have you tried any of these before? Comment on this article or connect with us on LinkedIn or Twitter! 