In this annotated statistical appendix, I describe in greater detail the statistical analyses and analytical choices I make in my post “The Myth of the Evangelical Voter” on the Emerging Scholars Network.
- Why the General Social Science Survey?
In my analysis, I use the General Social Survey to examine the attitudes of evangelicals. While there are many robust surveys on the social and political views of evangelicals (for example, here, here, here, and here), the 2014 General Social Survey is the most recent study that uses random sampling techniques, contains comprehensive questions on the religious and social, cultural, economic, and political attitudes of Americans, and has made individual respondent data available for analysis. Without access to the raw respondent data, it is impossible to make inferences about evangelicals’ attitudes and preferences beyond the topline results provided by the study authors.
- Who is an evangelical?
At first glance, identifying who is an evangelical should be straightforward. However, there remains significant debates on how to best conceptualize and measure or identify evangelicals in survey research (for more, see here, here, here, and here).
In the General Social Survey, I begin by identifying evangelicals as those who respond yes to the question “Would you say you have been “born again” or have had a “born again” experience — that is, a turning point in your life when you committed yourself to Christ?” 40% of respondents in the survey are identified as evangelicals using this broadest criteria.
However, I also exclude respondents who do not belong to a Christian denomination, never attend church, do not believe God exists, or do not believe in life after death. Though my more restrictive criteria reduces the number of respondents who qualify as evangelicals (30% instead of the original 40%), this coding better corresponds to research that religious self-identification should be consistent with some degree of religious beliefs and religious practice.
- Demographic and Religious profile of evangelicals and non-evangelicals
I argue that evangelicals and non-evangelicals are not directly comparable because of significant baseline differences in social, cultural, and religious profiles:
However, these unadjusted differences do not account for baseline demographic and social differences between evangelicals and non-evangelicals. Evangelicals are older, more likely to be women, less white, and more likely to be from the South than non-evangelicals.
Below, I summarize the differences in demographic and religious profiles of evangelicals and non-evangelicals.
- Adjusting for baseline differences in demographic and religious profiles
Differences in baseline demographic and religious profiles mean that simply comparing differences in attitudes of evangelicals and non-evangelicals may be confounding two possible sources of differences. As I write:
Comparing attitudes without accounting for demographic differences cannot reveal if political attitudes are attributed to respondents’ born-again experience or simply predicted by social and cultural predispositions.
My argument is that analyses that compare political attitudes at the group level fails to sufficiently account for individual-level similarities and predispositions that could also affect respondents’ political attitudes and preferences. In other words, while there may be differences in political attitudes when aggregating all evangelicals and non-evangelicals, there are unlikely to be differences when comparing the average aggregate difference between similar evangelicals and non-evangelicals at the respondent level.
To calculate the adjusted differences reported in the post, I use propensity score matching. Propensity score matching is based in a causal inference framework that claims the average of individual differences (average treatment effects) at the lowest level of aggregation (pairs of respondents) are a more accurate effect of a causal effect than the difference of aggregate differences at a higher level of aggregation. With propensity score matching, respondents are matched up with a most similar respondent based on a defined set of demographic characteristics in order to isolate the treatment effect, in this case, parameterized as whether a respondent identifies as an evangelical. This methodology thus allows us to identify the independent causal effect of identifying as an evangelical on political attitudes and preferences of interest.
Respondents’ propensity score are based on their demographic and religious profile. The demographic factors I include in the algorithm are: age, gender, marital status, ethnicity, Census region, education, income, and size of residence. I also include in the algorithm respondents’ religious beliefs and practices: frequency of church attendance, frequency of prayer, belief in the Bible as the inspired word of God, belief in God, whether they consider themselves religious, and how active they are in a local congregation.
While on average evangelicals have greater religious practice and stronger religious beliefs, there are non-evangelicals who are as devout in their religious beliefs and practices as evangelicals and there are evangelicals whose faith and practices are comparable to non-evangelicals. Moreover, not all Christians (those whose religious preference is Protestant, Catholic, or other Christian) identify as evangelicals. Among self-identified Protestants, 50% identify as evangelicals and 50% identify as non-evangelicals; among Catholics, 16% identify as evangelicals and among those who identify as Christian, 53% identify as evangelical.
After all respondents have been assigned a propensity score, the algorithm pairs up an evangelical with the nearest neighbor non-evangelical, or the non-evangelical who is most similar based on the composite of social, cultural, and religious characteristics. The reported adjusted differences is the average of each pair’s difference in political attitudes.
As a robustness check, I also estimate the treatment effect of identifying as an evangelical with regression adjustment. The results reveal the same inference as the propensity score matching, that identify as an evangelical has no statistical significant effect on political attitudes and preferences when factoring for respondent-level similarities in demographic and religious profiles.