AAPOR 2019

McClelland, S.I., Carillo, D., Dutcher, H., Jozkowski, K.N. (2019, May 16 - May 19). What's in the Item Counts Too: Implicit Attitudes in Abortion Attitude Survey Items. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

When studying controversial or sensitive topics, such as attitudes towards abortion, item wording and content is especially important. In the current study, we examined items used to study abortion attitudes over the last ten years to assess potential biases that explicitly or implicitly affect the quality of data. We used systematic review procedures to create an "item bank," representing all of the items used to measure abortion attitudes since 2008 (456 items, drawn from 81 studies). Studies represented both large, national studies with probability-based samples, as well as smaller studies with non-probability samples in order to represent the fullest extent of items. We used content coding procedures; three trained coders independently coded the item and response options. Inter-rater reliability (the extent to which coders applied the same codes) was very high. We highlight this coding strategy as useful for survey researchers looking to assess the quality of the items they are using, especially when measuring complex public opinion about topics such as abortion. Our focus was how survey items queried issues such as morality, termination language, and issues of funding for abortion. In our sample of found important, but subtle, word choices. For example, survey items that tapped opinions about funding for abortion consistently use the word "should" to query the respondent (i.e., "Tax dollars should cover abortion services for poor women"). The respondent in each of these cases is assumed to have knowledge about these issues and an attitude informed by this knowledge. Given the fact that abortion knowledge tends to be very low, items that assume high knowledge and ask the respondent about what "should" happen likely miss important aspects of this attitude such as lack of certainty or lack of information. Our study offers survey developers insights and methods for creating measures that attend to implict meanings. 

Valdez, D., Turner, R., Crawford, B., & Jozkowski, K.N. (2019, May 16-19). Comparisons of response patterns and distributions of attitudes about abortion for English and Spanish samples. American Association for Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Developing surveys to be administered in multiple languages can be a challenge for multi-cultural research teams. Many factors have to be considered including the type of translation process to be used (e.g., forward-backward, dual-focused), the type of language format (e.g., formal, informal, dialect), and the type of language alignment (e.g., literal vs. conceptual translations). Once the survey forms have been created, items can be evaluated across language groups for response distributions after controlling for response differences on other item sets and sample demographics. This helps provide information about the potential alignment or congruence of the language-translation. After items with similar responses for those with matched attitude and demographic characteristics have been identified, they can be compared across language samples to investigate similarities and differences in response patterns and distributions for the samples as a whole, or specific sample subgroups across the two language groups (e.g., gender, age category). These analyses can provide important information about cultural group/subgroup differences. A brief overview of the language translation process will be provided in the presentation along with results from the item comparisons (focusing on translation assessment), with a more detailed presentation of item comparisons across the two language groups.

Turner, R.C., Reimers, J., Jozkowski, K.N., Lo, W.J., & Crawford, B.L. (2019, May 16-19). Investigating differential response patterns for sample subgroups: Using DIF analyses to select scaling vs. polling items. American Association for Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

The number of items that can be included for measuring a specific topic is limited in many national polls, thus item selection is important for determining which combination of items can provide the most information about general trends and subgroup differences. Differential item functioning (DIF) and differential bundle functioning (DBF) are procedures commonly used for developing scales (groups of items) that provide correlated and consistent responses to ensure measurement invariance across subgroups and uni-dimensionality within a measure being used to collect attitudes from a population. DIF and DBF procedures can be helpful for polling researchers developing scales for their instruments. However, selecting individual polling items that have high internal consistency and strong measurement invariance can be contrary to the purpose of these items which are typically used to understand differences in the nuances of attitudes of people from different subgroups. We will present how DIF and DBF procedures can also be used to help polling researchers identify items that provide unique pieces of information about abortion attitudes. Items exhibiting DIF will be examined for differences that may be due to extraneous factors not desired by the researcher as compared to differences that may provide valuable insight into the underlying interpretation and perceptions of population subgroups. Data from national surveys that have been collected by the research team will be presented in addition to publicly available datasets such as the General Social Survey.

Lippman, J., Pasek, J., Turner, R.C., Lo, W.J., Jozkowski, K.N. (2019, May 16-19). Extracting common information across diverse measures: Identifying the latent attitudes of underlying abortion responses. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Since the 1960s, when researchers began asking about Americans’ attitudes toward abortion, tens of thousands of items have been employed across thousands of surveys. These questions vary in the wordings used as well as the more general matters they interrogated. Some items examined feelings toward abortion on a 101-point scale, others inquired about preferences for legality, and yet others implored respondents to identify as pro-choice or pro-life. Respondents were also often asked to evaluate abortion-related court decisions or candidates’ positions on abortion. To further complicate matters, questions about acceptability, legality, and availability were frequently conditioned on particular contexts (e.g. in cases of rape or incest) or particular stages of pregnancy (e.g. second trimester). And all this ignores the multiple question wordings used to tap each construct. Collectively, heterogeneity in measurement has served to complicate attempts to describe the state of Americans’ attitudes toward this issue and stymie attempts at understanding shifts in attitudes over time. In this study, we examine the role that various measurement strategies have on our understandings of both the state of abortion attitudes and changes in attitudes over time. By gleaning survey data repositories and reaching out to select firms, we have assembled a micro-level dataset of abortion attitudes for nearly 1000 nationally representative surveys on the topic across more than 50 years. Using textual analysis tools, we examine how the presence of various terms in questions and response options alter distributions of individual attitudes both within and across surveys. The results indicate that a few core clusters of types of measures constitute the key latent constructs underlying abortion attitudes and differentiating the trends of various sets of attitudes over time. These results highlight the need for novel measures that tap these distinct constructs.

McClelland, S., Carillo, D., Dutcher, H., Jozkowski, K.N. (2019, May 16-19). Implicit attitudes in survey measures: Using item banks to assess the state of knowledge in public opinion research. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

When studying controversial or sensitive topics, such as attitudes towards abortion, item wording and content is especially important. In the current study, we examined items used to study abortion attitudes over the last ten years to assess potential biases that explicitly or implicitly affect the quality of data. We used systematic review procedures to create an “item bank,” representing all of the items used to measure abortion attitudes since 2008 (456 items, drawn from 81 studies). Studies represented both large, national studies with probability-based samples, as well as smaller studies with non-probability samples in order to represent the fullest extent of items. We used content-coding procedures; three trained coders independently coded the item and response options. Inter-rater reliability (the extent to which coders applied the same codes) was very high. We highlight this coding strategy as useful for survey researchers looking to assess the quality of the items they are using, especially when measuring complex public opinion about topics such as abortion. Our focus was how survey items queried issues such as morality, termination language, and issues of funding for abortion. In our sample we found important, but subtle, word choices. For example, survey items that tapped opinions about funding for abortion consistently use the word “should” to query the respondent (i.e., “Tax dollars should cover abortion services for poor women”). The respondent in each of these cases is assumed to have knowledge about these issues and an attitude informed by this knowledge. Given the fact that abortion knowledge tends to be very low, items that assume high knowledge and ask the respondent about what "should" happen likely miss important aspects of this attitude such as lack of certainty or lack of information. Our study offers survey developers insights and methods for creating measures that attend to implicit meanings.

Crawford, B.L., Jozkowski, K.N., Turner, R.C. (2019, May 16-19). Roe v. Wade should go, but women shouldn’t lose their constitutional right to have an abortion: The effects of word choice when assessing abortion. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Wording used in surveys is vital to accurately communicate meaning and reducing bias. Item wording can influence how people respond and certain subgroups may be more sensitive to these wording effects, further biasing results (Knobe & Person, 2010). This is particularly relevant when assessing people’s attitudes around sensitive issues such as abortion (Singer & Cooper, 2014), perhaps due to the complex nature of abortion attitudes (e.g., Hans & Kimberly, 2013; Jozkowski, Crawford, & Hunt, 2018). Given the current salience of Roe v. Wade following the confirmation of Justice Kavanaugh, the current study assesses potential wording effects in items measuring attitudes regarding Roe v. Wade. The current study assesses participants’ knowledge of Roe v. Wade and then provides participants the following description of Roe v. Wade: “Roe v. Wade said that a woman has a constitutional right to choose to have an abortion until she is about 24 weeks pregnant.” We then compare differences in responses when asking people’s attitudes towards Roe v. Wade versus the constitutional right of a woman to choose to have an abortion. The current study uses a panel purchased from Qualtrics comprised of English (n = 1,783) speaking and Spanish (n = 918) speaking adults in the United States. Findings indicate that over half the sample accurately knew that Roe v. Wade provided constitutional rights regarding abortion (67.92%; n = 1,831), prior to us providing that information. Findings also show that 39.32% (n = 1,008) of participants responded differently when asked about overturning Roe v. Wade compared to taking away a woman’s constitutional rights to choose to have an abortion. We outline which demographic sub-groups were most likely to vary in their responses and outline recommendations for dealing with such wording effects when surveying about abortion and other socially contentious issues.

Jozkowski, K.N., Simmons, M., Crawford, B., & Turner, R. (2019, May 16-19). Attitudes regarding Roe v. Wade in the wake of Brett Kavanaugh’s confirmation to the Supreme Court of the United States. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Although national abortion attitudes and opinions seem stable (Bowman & Sims, 2017), state legislation restricting access has significantly increased (Guttmacher, 2017). With Justice Kavanaugh joining the Supreme Court of the United States, there is speculation that access to abortion will continue to be restricted, including the potential overturning of Roe v. Wade--the Supreme Court decision protecting a woman’s constitutional right to choose to have an abortion up to 24 weeks. These recent events have brought abortion to the forefront of public discourse. To assess the current climate regarding abortion—and Roe v. Wade specifically— data were collected via Qualtrics’ panels from a sample of English (N = 1,783) and Spanish speakers (N = 918) in the United States. The sample was stratified by gender, race/ethnicity, age, and political affiliation. Using a combination of closed-ended and open-ended items, respondents’ knowledge and attitudes regarding Roe v. Wade were assessed, including participants’ understanding of current abortion legality in their state and the United States and their perception of what may happen in their home state and the United States at large if Roe v. Wade is overturned. We also assessed whether participants thought Roe v. Wade should be overturned and their feelings if it were overturned. Findings indicate that participants were relatively split with regard to how they would feel if Roe v. Wade was overturned and whether they think it should be overturned. Despite abortion currently being legal in the United States, 11% of participants thought abortion was currently illegal in their state, and 8% said abortion was currently illegal in the United States. Qualitative findings indicate that feelings about Roe v. Wade and knowledge about abortion legality are associated with abortion-related identities and experiences, as well as moral assessments about abortion and the people who seek them.

Pasek, J., Lippman, J., Jozkowski, K.N.(2019, May 16-19). It's not that I oppose abortion, it's just that I am a Republican: Explaining the correspondence between partisan identity and abortion attitudes. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Abortion is one of the most politically polarized issues in the United States, with Democrats generally supportive of abortion rights and Republicans largely opposed to them. Yet, there is some evidence that attitudes toward abortion have not always been so divisive. In early studies of the topic, attitudes were not only less extreme than they are today, but they were also less aligned with party identity. Although others have noted that partisan polarization in attitudes toward abortion has increased over time (DiMaggio, Evans, & Bryson, 1996; Jelen & Wilcox, 2003), the mechanisms underlying this process are unclear. We contend that this increase in partisanship may, in part, be a product of affective polarization, whereby partisanship serves as an orienting social identity. If this is true, many respondents may be basing their reported opinions on abortion on their identities as Democrats or Republicans (and their opposition to the political outgroup). To test this possibility, we examine whether the increasing polarization of abortion attitudes over time occurred differently for measures of attitudes that were identity-based (e.g., identifying as pro-choice vs. anti-choice) as opposed to those that reflected underlying beliefs about the topic (e.g. the circumstances under which abortion should be legal). Using data from a corpus of hundreds of survey datasets assembled from public opinion archives that contained measures on both partisanship and at least one of these types of questions, we examine trends in the correspondence between partisan identity and abortion attitudes between 1969 and the present. The results suggest that polarization for these different sorts of attitudes are indeed inconsistent over time, but that the relations between the both types of measures and party identification strengthened over time.

Simmons, M., Maier, J., Jozkowski, K.N., McClelland, S., & Crawford, B. (2019, May 16-19). Examining differences in “Pro-Life” and “Pro-Choice” definitions among adults in the US: How critical conceptual analysis can improve abortion-related polling. American Association for Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Public opinion polls frequently assess abortion-related attitudes by asking participants whether they identify as pro-choice and/or pro-life.  Pro-choice and pro-life ideologies and identities are typically thought of as being at polar ends of a pro-abortion/anti-abortion continuum. However, recent survey data indicating that adults in the US simultaneously endorse both these labels calls into question the extent that they are truly ideological polar opposites. As public opinion data drive not only the rhetorical strategies of pro- and anti-abortion movements but also political action related to abortion access, it is important to understand how people in various social locations conceptualize the terms “pro-choice” and “pro-life.” As feminist scholars have noted, assuming a shared definition of terms can lead to inaccuracies in the generation and application of data (Unger, 1993, McClelland, 2017).  In order to understand how US adults are defining “pro-life” and “pro-choice”, we surveyed both English speakers (N = 1, 783) and Spanish speakers (N = 918) through Qualtrics about their thoughts on a variety of abortion-related items. In this presentation, we report on variations in responses to the open-ended questions: 1) How do you define the term pro-choice; and 2) How do you define the term pro-life.  We use critical conceptual analysis and statistical comparisons to examine how definitions relate to the sociodemographic characteristics of participants, including race, gender, political affiliation, and endorsement of pro-life/pro-choice identity. Although many definitions mirrored the traditional understandings of “pro-life” and “pro-choice” ideologies, we found nuanced differences in definitions across and within demographic subgroups. This may influence how people respond to polling items. Our findings support the need for continual concept clarification when developing and fielding polling items.

Valdez, D., Turner, R., Crawford, B., Jozkowski, K.N., & Kaplan, A. (2019, May 16-19). Roe v. Wade and Brett Kavanaugh: Differences in attitudes among English and Spanish speakers in a national, bilingual survey. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Historically, public opinion on major social issues has differed between Spanish-speaking Latinx populations and English-only speaking populations (Sanchez, 2006). Many of these variations stem from cultural differences unique to the Latinx population (e.g. familyism, religiosity, among others) which may influence the content of responses as well as how people interpret response options (Kerevel, 2011). While sources that predict divergent public opinions among English and Spanish speakers are noted in the literature, few studies have compared the extent that attitudinal differences may (or may not) exist on controversial issues— such as abortion— within the same translated survey. The purpose of this study is to assess attitudinal differences among a sample of English speakers (N = 1,783) and Spanish speakers (N = 918) on current divisive issues related to abortion in the United States, such as (1) the confirmation of Brett Kavanagh, (2) the future of Roe V. Wade, and (3) other beliefs regarding abortion ideologies of political leaders. Although previous research suggests that English and Spanish speakers will vary, our findings suggest that these group differences are actually quite small or non-existent. We will highlight similarities among these historically divergent groups and suggest implications for future polling strategies for the Latinx sample.

Maier, J., Simmons, M., Turner, R., & Jozkowski, K.N. (2019, May 16-19). Abortion-related identity and sentiment among LGBTQ individuals. American Association for Public Opinion Research (AAPOR) Annual Meeting.

In recent decades, there has been an effort among scholars, activists, and laypeople to “queer” reproductive justice, or bring greater attention to the way people with non-normative gender identities, sexualities, and family configurations engage in sexual, romantic, and familial practices (Mamo, 2007; Nixon, 2013). Included in this movement is the push to make the abortion rights movement less heteronormative and more inclusive of LGBTQ individuals (Sutton & Borland, 2018). In line with this goal of queering reproductive justice, there is a need to better understand LGBTQ individuals’ views on abortion. While there is some evidence to suggest that LGBTQ people are more supportive of abortion rights compared to their heterosexual, cis-gendered peers (e.g., Grollman, 2017), there has been a relative dearth of research exploring the complexity of this population’s views on the topic. A failure to survey LGBTQ individuals about their opinions and attitudes on abortion perpetuates the misconception that abortion is not of interest or concern to this community. It also creates a knowledge gap related to LGBTQ individuals’ opinions on reproductive health issues. In order to understand the attitudes, opinions, and abortion-related political activities of LGBTQ adults in the United States, we surveyed both English speakers (N = 1, 783) and Spanish speakers (N = 918) through Qualtrics. Preliminary analysis of both open-ended and closed-ended measures suggest that, compared to their heterosexual peers, LBGTQ participants are more likely to identify as strongly pro-choice and engage in pro-choice activism. Our presentation will expand on these preliminary results, providing a more in-depth depiction of LGBTQ individuals’ abortion-related attitudes and activism.

Lo, W.J., Turner, R., Crawford, B., & Jozkowski, K.N. (2019, May 16-19). To slide or not to slide, that is the psychometric question. American Association for Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Entering the 21st century, web-based surveys have become a common mode of data collection in many research fields, in which respondents are asked about their attitudes, perceptions, or evaluations. With online surveys, Likert scales and slider scales are the most widely used, and the psychometric properties of both scales have been argued over for decades. While some researchers support Likert scales for (e.g., Toepoel & Funke, 2018), others either reached inconclusive results (e.g., Roster, Lucianetti, & Albaum, 2015) or argued that slider scales greatly improved reliability and response style (e.g., Sung & Wu, 2018). Therefore, the purpose of this study was to investigate both scales’ psychometric properties by applying an existing psychological measure: the Parental Bonding Instrument (PBI; Parker, Tupling, & Brown, 1979). The 25-item (i.e., 13 positively worded items and 12 negatively worded items) PBI is a self-report questionnaire in which respondents reflect on parental attitudes and behaviors observed in their first 16 years of childhood. Parental care and protection were originally defined as the constructs. The original response format was on a 4-point Likert scale, but we changed it to a 6-point scale based on several studies suggestions about the optimal number of response categories (e.g., Krosnick & Fabrigar, 1997). Two separate samples (one with Likert and the other with slider) were collected through Qualtrics. Confirmatory factor analysis was used to validate the model with different scales. To test the internal consistency, test-retest reliability was assessed by both samples, which returned approximately 2 weeks after the original assessment. This study intends to investigate phenomena in threefold: (1) Does one scale outperform the other? (2) Do different scales impact the psychometric properties of a measure? and (3) Does wording influence the response patterns?

Crawford, B.L., Jozkowski, K.N., Turner, R. C., Lo, W.J. (2019, May 16-19). Attitudes regarding Roe v. Wade before and after the confirmation of Justice Kavanaugh. American Association of Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Although research consistently shows that attitudes regarding abortion are relatively stable over time (Bowman & Sims, 2017), less research has focused on short-term modulations in abortion attitudes that may coincide with current events. The retirement of Justice Kennedy and subsequent confirmation of Justice Kavanaugh brought discussions of abortion and Roe v. Wade to the forefront. Specifically, the potential weakening or overturning of Roe v. Wade and the implications for abortion access in the United States dominated much of the early discussion around the nomination and confirmation of Justice Kavanaugh. This increased media attention presents a unique opportunity to assess if and how these current events might be related to short-term changes in attitudes regarding abortion and Roe v. Wade. The current study uses two cross-sectional samples purchased from Qualtrics that are comprised of English (n=~2,000) and Spanish (n=~1,000) speaking respondents. The samples were obtained before the confirmation process began and after Justice Kavanaugh’s confirmation. The samples were quota sampled based on gender, age, race/ethnicity, and political affiliation. In wave 1, after being provided a brief summary of what Roe v. Wade allowed, 55% of the sample did not want to see Roe v. Wade overturned, 14% were undecided, and 31% of the sample desired to see Roe v. Wade overturned. This paper will outline changes in sentiment regarding the overturning of Roe v. Wade. The paper will also discuss which groups saw the biggest changes after the confirmation of Judge Kavanaugh. We will also discuss the lessons learned from this paper that can be applied to assessing short-term changes in attitudes regarding other sensitive issues.

Kaplan, A.M., Valdez, D., Jozkowski, K.N., Crawford, B.L. (2019, May 16-19). Increasing the effectiveness of Latinx outreach in survey-designed research: Important procedural considerations to improve the functionality of Likert-items among Spanish-speaking samples. American Association for Public Opinion Research (AAPOR) Annual Meeting. Toronto, Canada.

Though a staple in survey-design research, Likert-items do not function well among Latinx samples due to cultural and linguistic challenges unique to that population (Bernal et al., 1997). Consequently, when researchers use Likert-style questionnaires among Spanish speakers, the data are not often reflective of the behavior, attitudes, or opinions measured by those survey items (Boone & Boone, 2012). Newer strategies in Latinx polling, however, attempt to make Likert items more accessible to Spanish-speaking audiences. This presentation will highlight some of those strategies by discussing the procedural steps taken when translating a survey on controversial US issues— such as abortion— into a culturally competent Spanish language version. Specifically, we highlight the various contemporary considerations undertaken in the survey design process to improve the quality of our Latinx sample— which include: the importance of concurrent English and Spanish survey construction and implementation, translating concepts over language, and using a conceptual translations framework versus the more commonly used back-and-forth practices. The goal of this presentation is to provide scholars with practical recommendations when implementing surveys among Spanish speakers— especially if Likert items are involved. Overall, we discuss why these steps are important and how, precisely, they may potentially influence the reliability of Spanish surveys.