An interview study examining Browser Privacy and Security settings

About the project

Our team conducted a funded usability study to evaluate how good the current browser privacy/security dashboards are with respect to their ability to provide notice and control to different types of users.

Team

This study was conducted under the guidance of Prof. Norman Sadeh (Carnegie Mellon University) and Prof Yaxing Yao (UMBC). Our team consisted of Phd scholars (2) and graduate students (2).

My Role

As a UX Designer, my tasks included:

  • Conduct thematic analysis on the corrected transcripts in conjunction with teammates in order to answer research questions.

  • Analyze transcripts, find errors, and make corrections. This task helped in familiarizing with the research approach, conducting interviews, and demonstrating ways to interact with research participants.

Problem

Today’s web browsers offer privacy and security user interfaces, which are intended to address users’ needs for awareness and control over common online data practices which have alarming privacy and security concerns.

However, it is unclear whether these designs are effective in providing users with the notice and control they expect to have. It is vague that dashboards present information in a way that users can understand.
There has been an ambiguity whether the controls provided on dashboards (and elsewhere) actually enable users to set their preferences.

Research Questions that we answered

RQ1 How good are these browsers in ensuring that their users are aware of relevant privacy and security risks?

RQ2 Do users have the control necessary to mitigate these risks, and are they able to effectively take advantage of the controls to do so?

RQ3 What improvements can/should be made to make browsers capable of giving their users more awareness and control?

The Process

In order to answer the research questions, we have incorporated qualitative and quantitative research methods in the user research process. This methodology was useful in eliciting detailed accounts of participant behaviour.

Phase 1 - Literature Review

Our team has explored the parameters and considerations inherent in designing privacy and security interfaces which are intended to provide awareness and control. In particular, a number of previous studies have sought to formalize and evaluate principles which make designs more effective, as we have also done in this work. This rigorous search helped us to understand the development from other researchers' perspectives.

Phase 2 - Identifying the User scenarios

Our contextual interview scenarios were centered around participants working with the interviewer to accomplish the following tasks:
Scenario 1 - To identify the presence of a list of practices
Scenario 2 - To identify ways to control them

Phase 3 - Pre-Screening Survey

The Pre- Screening Survey collected demographic information and asked questions which measure the participants’ self-professed familiarity with the browsers in our sample groups. This survey was used to evaluate candidacy for the next step of the process(Contextual interviews).

Phase 4 - Contextual Interviews

The goal of conducting contextual interviews was to collect qualitative information about users’ experiences and to relate these experiences to common themes. During our contextual interviews, the interviewer and interviewee worked together to complete a variety of tasks through a remote screen-sharing session. Participants were asked to guide the interviewer through the steps required to detect the presence of a variety of data practices, thinking aloud and explaining their reasoning as they explored.

Number of participants interviewed: 49

Time Limit: Approximately 50 minutes

Demographic: Participants who were users of web browsers like Brave, Chrome, Firefox, Edge and Safari.

Gender: Any

Ethnicity: Any

Phase 5 - Correcting the transcription errors

Prior to analysis, each interview transcript was verified for accuracy with respect to the video recording by at least two annotators, resolving and correcting any transcription errors.

Phase 6 - Thematic Analysis

We performed grounded analysis on the corrected transcripts in conjunction with teammates. We looked for trends, patterns and interesting quotations. The goal was to identify segments of transcript text which are associated with different categories of responses (codes) that we are looking for in order to answer our research questions.

Challenges

1. It was difficult for the team to recruit users of browsers with small user bases (Brave, possibly others).

2. To maintain the attention span for over 50 minute long interviews while correcting the transcription errors.

Key Findings

  1. Users had problems in recognizing and controlling data practices as they were unable to distinguish between practices because of imprecise terminology and generalized or unexpected categorizations of practices used by the browser.

  2. Participants who used Chrome were not able to take control of most of the practices we mentioned.

  3. Most of the participants were not aware of the described data practices. Moreover, these practices were not mentioned by their browser, or used terminology that would be unlikely to help them to find information about the data practices they were concerned with.

  4. Users of browsers with limited/simplified controls (such as Chrome, Safari, and Edge) seemed to show a pattern of experiencing resignation and lack of confidence.

  5. More granular controls,such as those seen in Brave and Firefox, seemed to improve confidence somewhat, but this confidence was easily lost due to confusing terminology or a lack of affirmative feedback about participants’ actions.

Suggestions for improvement

  1. Browsers need to be redesigned in such a way that user should be able to better see what practices are allowed by their browser, and what is blocked or restricted.

  2. Edge and Brave browers need more granular and accurate clasification for different practices.

  3. Browsers should provide clear and precise descriptions of data practices which are technically consistent and granular standardizing descriptions of data practices based on standard taxonomies,such as the practices we studied in this work, could improve consistency.

Impact

  1. We gauged the user’s abilities to make use of their preferred browser settings toward identifying and mitigating several common privacy and security risks.

  2. We determined and characterized the disadvantages, advantages, and trade-offs users experience when interacting with privacy and security settings in their primary browsers.

  3. We remarked on the fact that, less than ideal choices that users encounter in certain browsers could be potentially improved, perhaps by incorporating features that work well in other browsers.

My learnings

  1. Listening to 50 minute interviews recordings and correcting the transcripts was mentally exhausting as it demanded continuous focus and concentration. I have understood that paying attention to detail and patience is very important while conducting an interview study.

  2. This project was conducted under strict time constraints. Working under pressure and deadlines helped me in pushing my boundaries and performing better in limited time.