What if your search engine's AI stopped giving you answers - and started asking you questions to help you better evaluate search results? ?
Our paper
"Can You Tell Me?": Designing Copilots to Support Human Judgment in Online Information Seeking
by Markus Bink, David Elsweiler, and Udo Kruschwitz in collaboration with Marten Risius (Neu-Ulm University of Applied Sciences)
explores exactly that idea, rethinking GenAI as a copilot, not an answer machine.
AI-generated responses are often fluent and seem reliable. This can discourage independent verification and reasoning.
Instead of replacing users' cognitive effort, we asked:
Can AI be designed to support human evaluation skills rather than shortcut them?
To explore this, we developed a chat-based AI copilot that:
- Encourages reflection instead of giving direct answers
- Promotes independent verification
Aims to foster digital literacy skills
In a randomised controlled trial (N = 261) comparing three interface conditions (copilot vs. AI overview vs. 10-blue-links baseline), we found:
? Users engaged deeply with the copilot and demonstrated metacognitive reflection.
⚖️ However, answer correctness and search engagement did not significantly improve as users invested more time in chatting with the copilot than exploring actual search results.
? Qualitative findings revealed tension between the copilot's pedagogical support and users' desire for efficiency.
? Key takeaway
These results highlight the promise and pitfalls of pedagogical copilots. Designing systems that balance literacy goals with efficiency expectations seems the way to go.
?? Find our design pathways and the pre-print of the paper here: https://arxiv.org/abs/2601.11284 (external link, opens in a new window)
Looking forward to discussing this work in Seattle! ?
#Copilot #GenerativeAI #AI
#DigitalLiteracy
#InformationInteraction #InformationSeeking
#InformationRetrieval #IR
#InteractiveSearch
#HumanAIInteraction
#EmpiricalStudy #MixedMethods
#PhDResearch
#SearchResearch #UXResearch #AIResearch
#ACM #SIGIR #ConferenceOnHumanInformationInteractionAndRetrieval #CHIIR #CHIIR2026
#ResearchSuccess #ResearchPaperAccepted
#InformationScienceRegensburg #StayInformed
