What if your search engine's AI stopped giving you answers – and started asking you questions to help you better evaluate search results? ?
Our paper
"Can You Tell Me?": Designing Copilots to Support Human Judgment in Online Information Seeking
by Markus Bink, David Elsweiler, and Udo Kruschwitz in collaboration with Marten Risius (Hochschule Neu-Ulm)
explores exactly that idea, rethinking GenAI as a copilot, not an answer machine.
AI-generated responses are often fluent and seem reliable. This can discourage independent verification and reasoning.
Instead of replacing users’ cognitive effort, we asked:
Can AI be designed to support human evaluation skills rather than shortcut them?
To explore this, we developed a chat-based AI copilot that:
- Encourages reflection instead of giving direct answers
- Promotes independent verification
- Aims to foster digital literacy skills
In a randomized controlled trial (N = 261) comparing three interface conditions (copilot vs. AI overview vs. 10-blue-links baseline), we found:
? Users engaged deeply with the copilot and demonstrated metacognitive reflection.
⚖️ However, answer correctness and search engagement did not significantly improve as users invested more time in chatting with the copilot than exploring actual search results.
? Qualitative findings revealed tension between the copilot’s pedagogical support and users’ desire for efficiency.
? Key takeaway
These results highlight the promise and pitfalls of pedagogical copilots. Designing systems that balance literacy goals with efficiency expectations seems the way to go.
?? Find our design pathways and the pre-print of the paper here: https://arxiv.org/abs/2601.11284 (externer Link, öffnet neues Fenster)
Looking forward to discussing this work in Seattle! ?
#Copilot #GenerativeAI #AI
#DigitalLiteracy
#InformationInteraction #InformationSeeking
#InformationRetrieval #IR
#InteractiveSearch
#HumanAIInteraction
#EmpiricalStudy #MixedMethods
#PhDResearch
#SearchResearch #UXResearch #AIResearch
#ACM #SIGIR #ConferenceOnHumanInformationInteractionAndRetrieval #CHIIR #CHIIR2026
#ResearchSuccess #ResearchPaperAccepted
#InformationScienceRegensburg #StayInformed
