It was a great pleasure to present at last week’s 79th Annual Conference of the World Association of Public Opinion Research in Mexico City. Surrounded by the Dia de Muertos or Day of the Dead celebrations, it felt fitting to muse about the fate of humans in our rapidly approaching AI-dominated future.
I compared human analysis of qualitative data (cognitive interviews to validate survey questions, in this case) to that of two AI models, Gemini 4 Argon and Claude Sonnet 5.5. The dataset I used were cognitive interviews conducted in 2018-2019 to validate the Workforce Outcomes Reporting Questionnaire (WORQ), which aimed to measure youth employment and earnings in low- and middle-income countries. Cognitive interviews were conducted in five countries (Philippines, Kenya, El Salvador, Rwanda and Kyrgyzstan).
The two AI models were asked to analyze the cognitive data for three of the survey questions and to provide recommendations. They were also asked to evaluate the final questions humans arrived at post-validity and reliability testing and assess their own recommendations against these final questions.
Key among the findings was humans ignored minor discrepancies, such as two out of more than 100 cases misinterpreting “earn money” and answering the question incorrectly in the interests of brevity and simplicity, both Claude and Gemini were insistent on fixing these “issues.” Thus their recommendations and preferences were for more complex questions designed for absolute precision–the very versions we were trying to move away from due to their complexity and respondent burden.
My main takeaway was that AI does a thorough job in analyzing these data and finding the issues that need to be addressed in question design. Both AI models found the issues that humans had identified and sought to fix those issues. However, when it comes to solutions, AI models fall back on “solutions” that already exist, often “solutions” that we were actually trying to move away from because they didn’t work that great in the first place.
When I asked ChatGPT to tell me what it saw as the difference, it created this fascinating chart:
SURVEY RE-ENGINEERING
| HUMAN DESIGN FOCUS | AI DESIGN FOCUS |
|---|---|
| Concise Question Length | Total Semantic Security |
| Streamlined Flow | Deep Structural Auditing |
| Minimal Operational Bloat | Exhaustive Boundary Logic |
So AI, as of today, is a wonderful tool to aid in the analysis of these data and can create huge efficiencies in processing and analyzing large quantities of qualitative data. But when it comes to distilling all that analysis down to making recommendations, especially in making innovative, creative leaps, human agency is still required. It can, however, be argued that AIs in the not-too-distant-future might take that role away from us too. But that day is not yet here. For now, the clear path forward is to leverage the analytical efficiencies AI tools provide while being aware that humans are still necessary to refine those analyses into clear insights and recommendations.
The full presentation is available here.
Due to time constraints, the analysis presented at the conference focused on one question (our initial branching question). The following table shows recommendations for all three questions that were tested for this exercise:
| 📋 Survey Concept | ❌ Initial Draft | 🔹 Human Benchmark | ♊ Gemini Framework | ☿️ Claude Framework |
|---|---|---|---|---|
| Q1: Employment Screener | “In the last 3 months, did you do anything to earn money?” | “In the previous month, meaning in [insert month year], did you do anything to earn money?” | Question: “In the previous calendar month—meaning from the first to the last day of [insert Month, Year]—did you do any kind of work or activity to earn money or bring income to your household?”Field Enhancements: • Focuses on preventing false-negative screenings. • Includes a mandatory read-aloud list of cash/in-kind work and informal side-gigs. • Adds strict interviewer screening instructions to block non-work money (lottery, gifts, savings groups). | Question: “In the previous month, meaning in [insert month year], from [first date] to [last date], did you do any work or activity for which you were paid money or goods?”Field Enhancements: • Completely strips out the vulnerable term “earn money”. • Mandates a short follow-up check (Q1B: “In [insert month year], did you do any other work… even a small one?”) to catch secondary or family activities that youth routinely omit. |
| Q2: Status in Employment | “Think about the main way you earned money… CHECK ONE” | “Now, think about what you did to earn most of your income… Was that: A helping in a business run by a relative who lives with you; or B working for yourself; or C working for someone else?” | Question: “Think about the ONE actual work activity that brought in the MOST money… In that main work, which of the following best describes your role? CHECK ONE:” Options: 1. Helping in a business/farm owned by a co-resident family member. 2. Running your own business/farm alone or with partners (explicitly including custom to-order client work). 3. Working as an employee, casual worker, or paid intern for an outside entity. | Question: “Now, think about what you did to earn most of your income… that is, the work that brought you the most money. Was that:” Options: A. Helping in a business run by someone related to you and lives with you. B. Running a business you own alone or with someone else, that is, working for yourself and deciding how the work is done. C. Working for an employer or boss who pays you. Interviewer Note: Automatically skips this entire question if the respondent only reported one single activity in Q1. |
| Q3: Job Regularity & Type | “Is this job Permanent; Fixed-term… Temporary; Seasonal; Freelance; or an internship? CHECK ONE” | “Is the work you did for someone else in the previous month…: Regular, that is, something you do for a set number of hours every week, or Irregular, that is, something that you did when work was available?” | Question: “Thinking ONLY about this main work you did for someone else… was this work:” Options: 1. Ongoing, regular work (steady arrangement to work every week on a regular schedule for regular pay or salary). 2. Casual, on-call, or ‘when work is available’ work (not guaranteed work every week; working shifts/tasks only when a boss or client calls). | Question: “Is the work you did for someone else in the previous month, meaning in [insert month year]:”Options: 1. Work with a set number of hours every week. 2. Work that you did when work was available. Field Enhancements: • Permanently bans abstract labels (Regular, Irregular, Permanent). • Mandates a post-answer interviewer factual cross-check to record total hours, days, and shifts to map the real working pattern. |
What do you think of the recommendations Claude and Gemini came up with compared to the human team’s recommendations? Did we do better or did AI make the right call? Going forward, what do you see as the role of AI and humans as we jostle about in this increasingly “efficient” world?
I am planning to revisit this analysis in another 6 months or so to see where the ever evolving AI models will have taken us by then. Stay tuned.