Guide to Doing Large Scale Online Qualitative Research using Mobile Ethnography
Context
Here at Indeemo, we’ve noticed a recent trend where the number of respondents participating in mobile ethnography [What is Mobile Ethnography?] projects have been increasing significantly.
Where once numbers averaged 20-40 respondents, we are now supporting clients recruiting 100-200 respondents and have seen some qual research briefs requesting as many as 800 respondents.
As a result, we’re spending a lot of time helping researchers recalibrate client expectations. This short article looks to explore what is driving this change and shares some lessons you can embrace to evolve your research offering to this recurring shift.
What is Driving the Increase in Scale?
We are finding that clients looking for “hundreds” of respondents typically fall into three main buckets:
Qualitative Researchers familiar with mobile ethnography but who are dealing with a client with a significant budget and who may not be au fait with the methodology.
Those new to Mobile Ethnography where a client / or boss asks for a study with 100’s of respondents without them fully understanding the implications of this requirement.
Qualitative research projects covering multiple markets with multiple personas / segments or target markets included in the brief.
The Drivers of Scale
This ‘scale-change’ that we have observed is mostly evident with large blue chips and technology clients who typically have deeper pockets.
Firstly, due to the COVID pandemic, the amount of travel has collapsed and many early adopters have switched to online qualitative research methodologies like Mobile Ethnography and mobile diary studies.
The budget previously ring-fenced for hotels and travel (particularly for multi-market research) has essentially been reallocated to invest in larger samples.
A second driver of the increased demand for larger respondent pools aligns with many end clients embracing mobile ethnography for the first time.
With the perceived travel savings described above and the fact that research is being done remotely, some end clients want to use a quantitative data set in a qualitative way.
However, as I go on to describe below the consequences of the increased numbers is something that needs to be more widely understood (and messaged to clients) to ensure that larger scale qualitative research can deliver on its promise.
The final trend we’re noticing is a sharp increase in technology first / digitally native brands shifting rapidly into qualitative research.
For years these companies have relied solely on quantitative data and analytics and their expectations around numbers are much higher than traditional consumer brands. Conversely, there is a concern their stakeholders might mistrust smaller sample sizes.
Either way, whether you’re a consultancy or an internal researcher, it’s imperative for researchers to manage the expectations of project sponsors when it comes to handling requests for larger-scale qual projects with “hundreds” of respondents.
The Implications
There are three main implications to be aware of with large scale online qualitative research projects and larger respondent sample sizes:
The budget needed for recruitment and incentives
The more significant cost in terms of moderation, analysis, reporting etc
The marginal benefit of recruiting additional respondents beyond a certain thematic saturation point
Budget Implications for Recruitment and Incentives.
Increased numbers mean increased cost, as well as additional management costs associated with the administration of a larger pool. Respondents need to be recruited, added to the platform, monitored to see who has engaged and assessed as to who needs to be probed or followed up with, as well as managing the admin associated with processing the incentives.
More respondents equate to higher budgets for incentives, and the maths for a 100 respondents study is likely to take our incentive costs alone towards $10,000+. This increases sharply with hard to recruit respondent personas.
“For a one week study, you're looking at anything from $75 to $100 as an incentive per respondent, depending on the complexity of the respondent set that you're looking for. Once you multiply that by 100, your costs start to get quite high relative to other methodologies” argues, Maeve Dunne, Strategic Account Manager at Indeemo.
Are The Extra Numbers Worth It?
Like many decisions, there is a trade-off when scaling your online qualitative research projects from tens to hundreds.
Notwithstanding the increased cost of recruitment and incentives, the largest cost to the client (and the biggest pricing risk for consultants) will be the time spent by the researchers managing the project, moderating respondents, analysing the data and preparing their reports.
There is an increased amount of rich data that will need to be analysed. For example, 100 respondents each submitting 10 mins of video footage, means almost 17hrs of video to work through. So this approach is best suited to larger agencies or clients with the capability and processes to analyse all the extra data (not to mention pay for it.)
In short, due to thematic saturation, it is argued that data should be collected until there are fewer surprises in the data and no more patterns or themes are emerging from it.
However, there is also the issue of the law of diminishing marginal returns: after a certain point, you need to ask yourself will you learn a lot more from all the additional data?
Indeemo’s Formula for Large scale Online Qualitative Research Projects
More not always more when it comes to large scale qual research. It’s critical to get the sample size right to ensure the best return on investment.
Therefore, when sizing projects, we recommend you first figure out how many unique segments or target groups of respondents you need. For example, if you are a banking client looking to understand “high net worth individuals”, “families” and “single urban professionals” then the following maths apply:
- minimum of 8 respondents / target group = 24 respondents.
- Max of 30 respondents / target group = 90 respondents.
Or to put it another way, it’s always worth asking yourself (and your client) what will you learn from the 31st respondent in a particular segment that you have not already learned from the first 30?
Narrow the Lens
We’ve supported a lot of large scale projects recently and one thing is critical when you’re managing qualitative research projects at pseudo quantitative research scale: the larger the pool of respondents, the narrower your lens needs to be according to Dunne:
“If you have a very narrow lens and you're doing UX research, where you just want to understand three key interactions across a larger data set, that's easier to manage. However, if you're undertaking an exploratory type study, for example, a consumer insights project, it's very difficult for the researcher to mine insights, out of hundreds of participants, and to develop themes from across all of the video, photo and written content.”
Whereas narrow-focused UX studies will scale, doing exploratory / foundational research with hundreds can be a boil the ocean type scenario.
In short, what Dunne is arguing is that a key issue is the ‘lens’ of the project. If the research is more “wide lens” exploratory, the increased numbers make it much more difficult to manage, than if it is a more focused narrow-lens project e.g. perhaps a UX assessment.
Finally, If you do plan to undertake a large scale qualitative or user research project it is also important to ensure the tasks you set respondents are clear and well structured.
The last thing you want is to have to moderate significant numbers of respondents who are getting confused because of a poorly designed task list.
Working closely with your platform provider will help de-risk this potential scenario.
The Benefit of Mobile Ethnography for Large Scale Qualitative Research Projects
Mobile ethnography applications like Indeemo are powerful at helping you handle large scale qual projects especially because of their easy to use social networking style UX and powerful dashboards featuring automated video transcription, automated keyword analysis and tagging / coding capabilities.
Furthermore, the ability to design and execute fieldwork in an agile way using platforms that support phased, flexible tasking strategies is uniquely valuable to large scale qual projects.
The strong trend we see emerging is the use of Mobile Ethnography in a phased approach where clients start with a small sample in the initial phase of fieldwork.
This enables researchers to iron out any kinks in the process and ensure that everything is designed to support a scale up. Once the initial phase of fieldwork is complete, researchers can confidently scale up the sample to hundreds of respondents.
Using the power of this agile, phased research approach, clients can engage a wider respondent pool than was traditionally the case with in-person research.
As an example, undertaking nationwide US research whereby respondents are recruited across all markets is now feasible using remote tools where it was previously impossible using in-person approaches.
This is even more powerful when it comes to multi-country research projects.
The fact that there is no need for travel on the part of the researcher or the respondent creates more “budget headroom” to both scale up respondent numbers to get a deeper sample and achieve broader coverage of key markets without increasing the budget of the end client.
Because clients no longer have to pay to have researchers go and meet respondents in person, these significant cost savings can be redeployed without compromising on the quality of the research.
Simply put, the result is less time spent in airports and more depth for the same cost, assuming you have correctly sized the sample, designed your tasks suitably and accurately estimated your own time.
Summary and Conclusion
We are seeing a strong increase in the number of respondents being recruited for online qualitative research projects using mobile ethnography and mobile video diary studies.
Because of the capabilities of these remote research methodologies, Researchers are now able to research multiple markets with a larger pool of respondents. Consequently, they can observe more people and capture more context-rich behavioural data. As a result, the demand for larger sample sizes is increasing in the briefs we are seeing.
Clients, who are reallocating freed up travel budgets toward larger samples are demanding more bang for their buck.
Given the need to devote more time for moderation, analysis and reporting, it is critical however that Researchers ensure there are enough people on their teams to manage this increase in scale. It is also critical that researchers have budgeted appropriately for same.
Given the implications cited above, we believe that in many instances clients are best advised to right-size their sample size and should always prioritise quality over quantity.
Learn More: Undertake Multimarket Qualitative Research Effectively
Contact us now
To learn more about Indeemo and how we can support your next research project, call us or submit your contact details.
US Toll Free: +1 888 917 7480 | UK: +44 (0) 845 528 0870