Measuring and Improving Generative AI Literacy

Overview of Problem

As generative (GenAI) tools like ChatGPT become more embedded into our daily lives, an emerging issue today is how appropriate people especially students are using GenAI. We called this concept AI literacy, which is defined to be the ability to not only understand how AI works and AI's capabilities and limitations, but also know how to evaluate its output and when to use it appropriately.

Role

As the lead researcher, I was involved in all parts of the project from literature review, survey design, design of the activities to data analysis and coding the LLM as a judge as part of the analysis.

Our Intervention

Our intervention was a quiz contain interactive metacognitive activities, to improve GenAI literacy in students. The activities had the following goals in mind:

With the activities it aims to improve generative AI literacy by:

• Bring awareness of one's goals, context, and available resources
• Calibrate one's expectation of GenAI's abilities and limitations
• Learn when to use GenAI appropriately based on context and situations

Research Methodology

We explored the following research questions:

1. How does training metacognition for Generative AI impact user’s prompting and verification skills?
2. How do the interventions affect people’s perceptions and confidence with using AI chatbots?

What was done:

• Created Qualtrics survey → Ran between-subjects study (Metacognitive intervention vs The University of Toronto's GenAI Literacy Course Modules (OER))
• 50+ participants but 22 participants (12 in intervention & 10 in OER) from various disciplines after filtering based on attention check questions. We hope to obtain ~400 participants (200 per condition) in order to achieve a statistical power of 0.8 with a small effect size of 0.25 with a t-test.
• Used GPT-4o with LLM as a judge framework for prompt quality and for chickpea challenge
• Semantic analysis on prompts using TF-IDF with n-grams

Results

Effects on Prompting and Verification Skills: How does training metacognition for Generative AI impact user's prompting and verification skills?

Open-Ended Prompting Task (Chickpea challenge) Results

In this task, the participants will submit prompts to later have the LLM try to guess the correct number of chickpeas in a jar in an image correctly.

Prompt Quality Rating using LLM as a judge (GPT-4o)

Mean absolute error of the number of chickpeas obtained from GPT-4o

We saw the intervention condition received poorer ratings across all prompt dimensions by GPT-4o (p = 0.287 (relevance), p = 0.010 (quality), p = 0.1920 (coherence)). However, the prompts written by the intervention condition was closer to the ground truth/true value in the open-ended prompting task (ex: Get GPT to estimate the number of chickpeas in a jar from 2 photos) (p = 0.008). Why could that be the case?

Preliminary qualitative analysis

• Saw some outlier prompts for intervention → Lower rating across dimensions by GPT-4o
• Intervention condition didn't use prompting techniques as often (ex: chain-of-thought) compared to the OER condition → Lower rating across dimensions by GPT-4o.
• Intervention condition tend to assist with calculations, express their thought process, and specify the view of the chickpea jar the images are showing → Hypothesis of why they were closer to ground truth

Linguistic analysis

Linguistic analysis shows that both conditions used similar vocabulary, however the intervention conditions specified the views of the images.

Sample of prompts written by intervention group for open-ended prompting task

Verification skills

Using Bo et al (2024)'s appropriate reliance metrics, we obtained the following:

Outcome	OER Modules	Interventions
Appropriate Reliance on LLM	2	2
Appropriate Self-Reliance	3	5
Under-Reliance	1	2
Over-Reliance	1	1

Intervention condition tend to correctly rely on themselves when chatbot gave the wrong answer compared to OER module condition.