Shared Confidence and Uncertainty Using Google: The Effects of Mental Models on Older and Younger Users’ Search Experiences
Abstract
Prior research found disparities in information-searching processes between older and younger adults. This study shows that generational differences between older and younger users’ mental models and experiences of online search engines are lessening over time as users gain experience and Google’s pervasiveness in daily life expands. 17 younger adults and 18 older adults, all experienced search engine users, participated in a series of methods that facilitated comparative analysis, including a questionnaire, realistic information search session using Google, and interview. The results indicated that previous experience strongly influenced mental models, instilled self-efficacy, and heightened expectations of Google for both groups. In the face of uncertainty, however, both older and younger users abandoned their confidence in Google and employed manual strategies to overcome Google’s limitations. While mental models developed over time generated satisfactory results for both older and younger users, identifying where their confidence faltered revealed opportunities for improvements to search engine design. This study concludes with recommendations that address areas where users’ mental models mismatched the system and promote a positive user experience for all generations.
Research Questions and Objectives
- Research Question: Do similarities and differences in how older and younger users use search engines in their everyday lives help to explain users’ mental models of search engines?
- A: How do older and younger users’ previous experiences using search engines inform users’ search experiences?
- B: How do older and younger users’ perceptions of search engines inform users’ search experiences?
To achieve the research aims, this study carried out the following objectives:
- Investigate and compare how the role of information search in older and younger users’ everyday lives informs users’ mental models.
- Conduct interactive information search sessions to identify patterns within and compare between older and younger users’ mental models, search strategies, and resulting search experiences.
- Measure and compare older and younger users’ number of search modifications made, perception of task difficulty, level of satisfaction with results, and level of satisfaction with own performance to quantify users’ search experiences in relation to mental models.
Methods
This study was conducted between 1 July 2021 and 1 October 2021. A naturalistic user-centered evaluation of Google was conducted to understand how well the user, retrieval mechanism, and the database interact to retrieve information under real-life conditions. The study was conducted with 35 participants, where the younger group was made up of 17 participants ages 21 to 36 years and the older group was made up of 18 participants ages 60 to 75 years. Participants completed an online questionnaire to record demographic data, interactive information search session with think-alouds, post-task metrics to collect complementary quantitative data, and a structured post-search interview. Realism of search tasks was central to the observation of genuine search behavior and assessments of user experience. Using multiple methods with a focus on qualitative data enabled the collection of varied data through which to build a deeper, richer understanding of participants’ mental models and search experiences. A pilot study was also conducted in three rounds. All data was collected remotely.
Qualitative data was analyzed using the thematic analysis framework presented by Braun and Clarke (2012), which facilitated identification of important themes and comparison between older and younger users’ search experiences. The distributions of the quantitative data collected were analyzed to pinpoint patterns or disparities in user experience, satisfaction, or search tactics between age groups. Differences were tested for statistical significance and analyzed for implications in light of the corresponding qualitative data.
Pre-Search Questionnaire
The questionnaire asked participants about demographic information, the role of information search in their everyday lives, and their levels of experience and confidence using online search engines.
Both younger and older participants reported using search engines for many years; on average, 21 years for the older group and 16 years for the younger group. Both groups also reported high levels of confidence using their preferred search engine, with an average of 8 out of 10 for the older group and 9 out of 10 for the younger group. Both groups have had time to build mental models, strategies, confidence, and perceptions of search engines.
The younger group reported spending significantly more time on the internet per day (p value=0.002) and using their preferred search engine significantly more frequently (p value=0.005) than the older group. While these differences suggest that the younger group would be more confident searching, they did not impact the main outcomes of the study because the older group had been using search engines for longer and reported similar levels of confidence as the younger group.
Interactive Search Session
Participants were given 4 problem-solving tasks to complete using Google. Task design and facilitation followed the user-centered information retrieval evaluation technique detailed by Borlund (2009) offering both realism and control. The level of difficulty of the tasks was determined based on previous research, balancing value to the research questions with ethical implications of task complexity.
TROUBLESHOOTING TASKS:
Imagine you have a MacBook Air. When you press the “8” key on your keyboard, nothing appears on the screen. The key does not feel stuck and you haven’t spilled anything. Use Google to find a few possible solutions. [If only one solution is found, ask participants to show what they would do next if the solution did not fix the problem.]
Your shower is making a high-pitched noise when you turn it on. You bought a new shower head 6 months ago and never had an issue with your old shower head. Use Google to find a few possible solutions. [If only one solution is found, ask participants to show what they would do next if the solution did not fix the problem.]
COMPARATIVE SHOPPING TASKS:
Use Google to help decide which mobile phone to purchase next for yourself or a family member.
Imagine you’re buying a gift for a friend or family member who likes music, traveling, and eating out. Use Google to help.
All 35 participants were familiar with the majority of or all tasks having done the same or similar tasks before. Familiarity with tasks was influential as participants often repeated strategies used previously; “I knew what words to put in to find what I wanted because I’ve searched for comparing products to each other before” (PY4), or anticipated the type of results they would get; “Because I actually did this search once, I know how it works” (PO1). Participants frequently drew on their familiarity and previous experience, key factors in the development of mental models (Norman 1983) to formulate search queries, filter results, and make decisions about which results explore further.
Participants were also familiar with Google, using Google to find information instinctively for daily questions and problems. Participants spoke about a sense of spontaneity, where Google was used to search for “whatever’s going on in your life at a given time” (PY5). Both groups also addressed how their use of Google has changed as their life has changed, for example, “I’m starting to look at stuff that has to do with being a senior or being an older person” (PO17), further evidencing a high level of familiarity with Google formed over time.
Post-Task Questions
After each task, participants were asked 3 questions to record their perceived level of difficulty of the search task, satisfaction with the results, and satisfaction with their performance, measuring participants’ immediate reactions to their search experience. The quantitative data collected supplemented observational and interview data, allowing for comprehensive comparison between age groups.
- On a scale of 1-5, how easy or difficult did you find this search task?
- Very difficult
- Somewhat difficult
- Neither difficult or easy
- Somewhat easy
- Very easy
- On a scale of 1-5, how satisfied are you with the results?
- Highly dissatisfied
- Dissatisfied
- Neutral
- Satisfied
- Highly satisfied
- On a scale of 1-5, how satisfied are you with your performance?
- Highly dissatisfied
- Dissatisfied
- Neutral
- Satisfied
- Highly satisfied
Summary of Findings
Participants were highly confident finding information using Google based on previous experience, and their familiarity and fluency with the tasks given in the interactive search sessions as well as the format and content of Google results created a natural environment for the observation of participants’ genuine search strategies.
Both the younger and older groups showed confidence in Google’s capacity to understand their query and return suitable results, as well as in their own abilities to search effectively, manipulate results, and find helpful information using Google. While these two realms of confidence may seem incongruous, they together made tasks feel easier, amplified participants’ satisfaction with the search results and their performance, and improved the search experience. Self-assurance and confidence in Google were built on an understanding, or mental model, that led to success often enough to have built confidence, regardless of the mental model’s technical accuracy. Participants developed search strategies over time using Google and discovered what worked well for them and what made sense with their mental models, for example ways of filtering results or influencing Google’s top results. These mental models informed search strategies, which, although differing, worked for participants as shown by their answers to post-task questions. Participants continued to build these mental models and modify their search strategies as they identified frustrating or undesirable parts of their experience in their daily lives based on Google’s shortcomings.
The major exception for both the older and younger groups to the confidences shown was moments of uncertainty when people found Google’s limitations. Contrary to expectations, identification of these limitations was not largely skewed towards one group or the other. The majority of participants in both the older and younger groups faced uncertainty, recognized Google’s limitations, and used strategies they had formed to overcome those limitations. The distribution of older and younger participants across those strategies varied, however. Younger users were more likely to save options for later reference and verify results or options across different sites. Older users were more likely to go directly to a known site. Both groups compared options by using tabs or comparative language in search queries. The limitations participants faced and strategies they used to overcome these limitations are the result of mental models formed over years of experience.
Key Similarities
- Built mental models, search strategies, and confidence over many years of using search engines, demonstrated by satisfaction with results of tasks and performance during tasks
- Showed high levels of both self-confidence finding information using Google and confidence in Google’s ability to return relevant results
- Used Google to find information spontaneously and instinctively for daily questions and problems
- Developed strategies for formulating effective search queries and choosing known results based on previous experience
- Expected efficiency, as evident by engagement with top results and featured snippet
- Conducted search queries to return results to be used as inspiration or a starting point
- Recognized Google’s limitations in comparison, decision making, and specific services
- Felt that despite their attempts, superior results may exist
- Used strategies to compare between results, solutions, or options and overcome Google’s limitations
Key Differences
| Older group | Younger group |
|---|---|
| More often mentioned benefits of a large number or results or options | Used more strategies to target the top results: “top”/”best” in query, lifting keywords from results, including a known website in query, and personalizing tasks |
| More often carefully considered search strategy or query given anticipated outcome of a search based on previous experience | More often filtered results in their mind, as opposed to adding constraints to their search query |
| More often commented on appreciation, admiration, and enjoyment using Google | More often discussed using Google to access “other people’s” experiences |
| Interacted more often with ‘People also ask’ | Interacted more often with YouTube videos |
| Demonstrated patience and perseverance more often | Demonstrated impatience and haste more often |
| More often overwhelmed by large numbers of options | More often compared options to verify validity or quality |
| More often went directly to a known site rather than using Google | More often used strategies to save options for later reference or comparison |
| Found troubleshooting tasks easier | Found comparative shopping tasks easier |
| More satisfied with their performance than with the results of the search within a given task | Less satisfied with performance when tasks were perceived as easier, implying high expectations of themselves |
Recommendations for Improving Search Engine Design
While participants generally had highly satisfactory experiences, there are opportunities for improvements to search engine design that promote a positive user experience for users of all generations. Along with improving the experience for today’s groups of older and younger users, the recommendations made in this section will remain relevant even as more tech-savvy generations retire; “We predict that today’s younger adults will experience loss of confidence in technological ability as they age; that they will favour familiar ways of accomplishing their goals, which are intertwined with the technology available at the time they established these practices” (Knowles and Hanson 2018). Search engine design and functionality will no doubt continue to advance, and it is crucial that designers are attentive to the following factors when evaluating and promoting the user’s experience.
ENCOURAGE EXPERIENCED USERS TO SHARE THEIR PERCEPTIONS TO IMPROVE USER EXPERIENCE.
Be clear how the design of online search engines corresponds to the role of search in users’ everyday lives.
Seek both positive and negative perceptions from experienced users when evaluating the user experience of online search engines.
Recognize and include a broad range of generations in definition of ‘experienced user’.
LEARN FROM EXPERIENCED USERS’ SEARCH STRATEGIES TO IMPROVE SYSTEM DESIGN.
Embrace common search strategies experienced users implement to overcome system limitations.
Help users identify similarities and differences between the content of top results.
Recognize search queries that seek comparison and generate a succinct comparison table.
Allow users to indicate where they have had success and include that website in the top results of related future searches.
Conclusions
Far more similarities than differences were found between older and younger users’ search strategies in everyday use of Google. Both groups showed self-efficacy and confidence in Google based on an ability to predict and an understanding, or mental model, built out of previous experience using Google in their daily lives and familiarity with search tasks. The exception to this confidence for both groups was where users displayed uncertainty and identified Google’s limitations. The similarities in search strategies formed to overcome limitations identified in this study suggested that older and younger users formed mental models based on experience and faced similar uncertainties and limitations regardless of their generation.
The data substantiated earlier research, providing evidence that previous experience was a very important factor in the development of mental models of search engines. Previous successes, failures, and trial and error influenced users’ search query wording, decisions on which results to interact with, and choices of if or when to modify their search. Positive previous experience built self-efficacy over time. With substantial previous experience, users instinctively repeated strategies that resulted in success in the past and formed strategies over time to overcome Google’s limitations, enriching individual satisfaction with their search experience.
The results indicated that experienced users’ perceptions of search engines influenced their expectations of ability to access information in an efficient manner, implying a high level of confidence in Google. These expectations were accurate the majority of the time, with the exception being when participants reached Google’s limits. In the face of uncertainty, frustration, or impatience, users abandoned their confidence in Google and turned to their own abilities, employing strategies to overcome Google’s limitations.
Users of both generations were mindful of Google’s limitations and formed strategies established through previous experience to overcome these limitations, including comparing between options to determine the best choice at hand, corroborate results, or seek out differences. Participants’ natural use of these strategies suggested that their mental models informed approaches that surmounted the negative impact to the user experience from Google’s shortcomings. These conclusions are indicative of improvements that can be made to search engine design to promote a positive user experience for users of all generations.
A full report is available upon request.
CONDUCTED AS PART OF INM363 INDIVIDUAL PROJECT (PRD3 S 2020/21) AS PART OF MSC HCID AT CITY, UNIVERSITY OF LONDON.
More projects:
Becca Epstein 