What if your search engine's AI stopped giving you answers - and started asking you questions to help you better evaluate search results? 🤔
Our paper
"Can You Tell Me?": Designing Copilots to Support Human Judgment in Online Information Seeking
by Markus Bink, David Elsweiler, and Udo Kruschwitz in collaboration with Marten Risius (Neu-Ulm University of Applied Sciences)
explores exactly that idea, rethinking GenAI as a copilot, not an answer machine.
AI-generated responses are often fluent and seem reliable. This can discourage independent verification and reasoning.
Instead of replacing users' cognitive effort, we asked:
Can AI be designed to support human evaluation skills rather than shortcut them?
To explore this, we developed a chat-based AI copilot that:
- Encourages reflection instead of giving direct answers
- Promotes independent verification
Aims to foster digital literacy skills
In a randomised controlled trial (N = 261) comparing three interface conditions (copilot vs. AI overview vs. 10-blue-links baseline), we found:
🔎 Users engaged deeply with the copilot and demonstrated metacognitive reflection.
⚖️ However, answer correctness and search engagement did not significantly improve as users invested more time in chatting with the copilot than exploring actual search results.
💬 Qualitative findings revealed tension between the copilot's pedagogical support and users' desire for efficiency.
💡 Key takeaway
These results highlight the promise and pitfalls of pedagogical copilots. Designing systems that balance literacy goals with efficiency expectations seems the way to go.
👉📖 Find our design pathways and the pre-print of the paper here: https://arxiv.org/abs/2601.11284 (external link, opens in a new window)
Looking forward to discussing this work in Seattle! 😊
#Copilot #GenerativeAI #AI
#DigitalLiteracy
#InformationInteraction #InformationSeeking
#InformationRetrieval #IR
#InteractiveSearch
#HumanAIInteraction
#EmpiricalStudy #MixedMethods
#PhDResearch
#SearchResearch #UXResearch #AIResearch
#ACM #SIGIR #ConferenceOnHumanInformationInteractionAndRetrieval #CHIIR #CHIIR2026
#ResearchSuccess #ResearchPaperAccepted
#InformationScienceRegensburg #StayInformed
