Engaging the Election: Evangelicals’ (dis)approval of Clinton statistical results

These are the statistical results discussed in my ESN post here:

Predicting Approval of Clinton

Statistically significant results at the p=.05 level in red. “Difference of effects” column compares the regression coefficient between the evangelical and non-evangelical model; statistically significant differences are indicated with a “yes”

Engaging the Election: Evangelicals’ (dis)approval of Trump statistical results

These are the regression results discussed in my ESN post: http://blog.emergingscholars.org/2016/07/engaging-the-election-what-predicts-evangelical-approvaldisapproval-of-trump/

Predicting Approval of Trump

Statistically significant results at the p=.05 level in red. “Difference of effects” column compares the regression coefficient between the evangelical and non-evangelical model; statistically significant differences are indicated with a “yes”

The Myth of Evangelicals’ Political Apathy

This is the online appendix for my article “The Myth of Evangelicals’ Political Apathy” on the Emerging Scholars Network blog.

1. The 2016 National Election Pilot Study

The data I use was sourced from the 2016 National Election Pilot Study that polled a nationally representative sample of 1200 respondents. The survey was conducted online between January 22 and January 28, 2016. For more details, please go to the study page here.

2. Who is an evangelical?

Who to count as an evangelical remains a difficult question (for my previous discussions about it, see here and here). When using public datasets, researchers need to use the best available inferences from the questions that are included. In this survey, I code evangelicals as anyone who self-reports as born again and attends church at least monthly.

While some studies use the most expansive definition of evangelicals (anyone who self-reports as evangelicals), I include the qualifier that they must also attend church at least monthly to differentiate between “just” those who self-identify as evangelicals and those for whom their born-again experience can be seen borne out in some form of religious behavior, here measured as at least some (monthly) church attendance.

3. Self-report of political action

In this article, I analyze respondents’ self-report of their political action. Self-reporting should be taken with a grain of salt as respondents tend to over-report their political behavior. For example, 77% of respondents in the survey report having voted in the 2012 election when the turnout was closer to 55%.

However, while respondents over-report how politically active they actually are, it is unlikely  there are systematic bias in what type of respondents over-report. Thus, because everyone likely over-reports, we can still make sound comparisons of differences between groups of respondents (evangelicals and non-evangelicals, weekly churchgoers versus infrequent churchgoers).


A Less Imperfect Measure of Evangelicals

Who is an evangelical? Is it someone who attends a church traditionally considered an evangelical denomination? Is it someone who checks the “evangelical” box on a survey?

In his thoughtful reply to my argument that evangelicals’ political attitudes are indistinguishable from non-evangelicals, Ryan Burge uses the RELTRAD definition of evangelical and finds different empirical results. I do not contest his findings, but argue that the REBORN+ measure I use (detailed here) is a better measure of who is an evangelical because it better captures the racial diversity, denominational preferences, and minimal religious criteria of evangelicals.

At first glance, it appears there are not significant differences between the RELTRAD measure and REBORN+ measure of evangelicals.  The RELTRAD and REBORN+ measures identify 25% and 30% of Americans as evangelicals respectively. As the table below shows, with the exception of race and gender, the two groups of evangelicals also have similar demographic profiles.


However, the two measures identify two different groupings of evangelicals.  Only 66% of RELTRAD evangelicals are also REBORN+ evangelicals while only 55% of REBORN+ evangelicals are RELTRAD evangelicals.

So which measure is a less imperfect measure of evangelicals?

I argue that the REBORN+ measure is better than the RELTRAD measure in three key ways.

First, the REBORN+ measure does not exclude minority evangelicals.  The RELTRAD measure undercounts black evangelicals who share similar beliefs and practices of white evangelicals since those who attend Black Protestant churches, not considered a traditional evangelical denomination, are not counted as evangelicals.

Using the RELTRAD definition, 19% of blacks are evangelicals and only 12% of evangelicals are black. By contrast, using the REBORN+ measure where black evangelicals can be counted regardless of what church they attend, I find 51% of blacks are evangelicals and 26% of evangelicals are black.

Second, the RELTRAD measure also does not capture the changing denominational preferences of evangelicals. Historically, evangelicals may have only attended evangelical churches.  However, contemporary evangelicals blur the distinctions between evangelical and non-evangelical denominations.

For example, the RELTRAD measure does not include the United Methodist Church as an evangelical denomination.  However, not only does the president of a prominent evangelical seminary identify as both evangelical and United Methodist, but the official statement of the denomination emphatically defines itself as an evangelical church.  Using the REBORN+ measure, I find that in fact, 31% of United Methodists can be identified as evangelicals.

Similarly, the RELTRAD measure does not allow Catholics to also be counted as evangelicals.  However, the growing prominence and number of evangelical Catholics suggests that an evangelical Catholic is no longer an oxymoron. Indeed, using the REBORN+ measure, I find 16% of Catholics identify as evangelicals.

Finally, the REBORN+ measure is better because it excludes those who have not had a born again experience. While not meeting the comprehensive criteria established by the National Association of Evangelicals, a commitment to Jesus is essential to being an evangelical, someone who has accepted the Gospel (good news or euangelion) of Jesus. While all REBORN+ evangelicals meet this essential criteria, nearly 1 in 4 (23%) of RELTRAD evangelicals have not had a born again experience.

Any measure of personal religiosity and evangelical convictions will be imperfect and incomplete. However, the REBORN+ measure is a less imperfect measure of evangelicals than the RELTRAD measure because it does not undercount minority evangelicals, counts evangelicals who do not attend traditionally evangelical denominations, and excludes anyone who have not experienced personal conversion and commitment to Jesus.

“Myth of the Evangelical Voter” annotated statistical appendix

In this annotated statistical appendix, I describe in greater detail the statistical analyses and analytical choices I make in my post “The Myth of the Evangelical Voter” on the Emerging Scholars Network.

  1. Why the General Social Science Survey?

In my analysis, I use the General Social Survey to examine the attitudes of evangelicals. While there are many robust surveys on the social and political views of evangelicals (for example, here, here, here, and here), the 2014 General Social Survey is the most recent study that uses random sampling techniques, contains comprehensive questions on the religious and social, cultural, economic, and political attitudes of Americans, and has made individual respondent data available for analysis.  Without access to the raw respondent data, it is impossible to make inferences about evangelicals’ attitudes and preferences beyond the topline results provided by the study authors.

  1. Who is an evangelical?

At first glance, identifying who is an evangelical should be straightforward. However, there remains significant debates on how to best conceptualize and measure or identify evangelicals in survey research (for more, see here, here, here, and here).

In the General Social Survey, I begin by identifying evangelicals as those who respond yes to the question “Would you say you have been “born again” or have had a “born again” experience — that is, a turning point in your life when you committed yourself to Christ?” 40% of respondents in the survey are identified as evangelicals using this broadest criteria.

However, I also exclude respondents who do not belong to a Christian denomination, never attend church, do not believe God exists, or do not believe in life after death. Though my more restrictive criteria reduces the number of respondents who qualify as evangelicals (30% instead of the original 40%), this coding better corresponds to research that religious self-identification should be consistent with some degree of religious beliefs and religious practice.

  1. Demographic and Religious profile of evangelicals and non-evangelicals

I argue that evangelicals and non-evangelicals are not directly comparable because of significant baseline differences in social, cultural, and religious profiles:

However, these unadjusted differences do not account for baseline demographic and social differences between evangelicals and non-evangelicals.  Evangelicals are older, more likely to be women, less white, and more likely to be from the South than non-evangelicals. 

Below, I summarize the differences in demographic and religious profiles of evangelicals and non-evangelicals.



  1. Adjusting for baseline differences in demographic and religious profiles

Differences in baseline demographic and religious profiles mean that simply comparing differences in attitudes of evangelicals and non-evangelicals may be confounding two possible sources of differences. As I write:

Comparing attitudes without accounting for demographic differences cannot reveal if political attitudes are attributed to respondents’ born-again experience or simply predicted by social and cultural predispositions.

My argument is that analyses that compare political attitudes at the group level fails to sufficiently account for individual-level similarities and predispositions that could also affect respondents’ political attitudes and preferences.  In other words, while there may be differences in political attitudes when aggregating all evangelicals and non-evangelicals, there are unlikely to be differences when comparing the average aggregate difference between similar evangelicals and non-evangelicals at the respondent level.

To calculate the adjusted differences reported in the post, I use propensity score matching.  Propensity score matching is based in a causal inference framework that claims the average of individual differences (average treatment effects) at the lowest level of aggregation (pairs of respondents) are a more accurate effect of a causal effect than the difference of aggregate differences at a higher level of aggregation.  With propensity score matching, respondents are matched up with a most similar respondent based on a defined set of demographic characteristics in order to isolate the treatment effect, in this case, parameterized as whether a respondent identifies as an evangelical.  This methodology thus allows us to identify the independent causal effect of identifying as an evangelical on political attitudes and preferences of interest.

Respondents’ propensity score are based on their demographic and religious profile.  The demographic factors I include in the algorithm are: age, gender, marital status, ethnicity, Census region, education, income, and size of residence.  I also include in the algorithm respondents’ religious beliefs and practices: frequency of church attendance, frequency of prayer, belief in the Bible as the inspired word of God, belief in God, whether they consider themselves religious, and how active they are in a local congregation.

While on average evangelicals have greater religious practice and stronger religious beliefs, there are non-evangelicals who are as devout in their religious beliefs and practices as evangelicals and there are evangelicals whose faith and practices are comparable to non-evangelicals.  Moreover, not all Christians (those whose religious preference is Protestant, Catholic, or other Christian) identify as evangelicals.  Among self-identified Protestants, 50% identify as evangelicals and 50% identify as non-evangelicals; among Catholics, 16% identify as evangelicals and among those who identify as Christian, 53% identify as evangelical.

After all respondents have been assigned a propensity score, the algorithm pairs up an evangelical with the nearest neighbor non-evangelical, or the non-evangelical who is most similar based on the composite of social, cultural, and religious characteristics.  The reported adjusted differences is the average of each pair’s difference in political attitudes.

As a robustness check, I also estimate the treatment effect of identifying as an evangelical with regression adjustment. The results reveal the same inference as the propensity score matching, that identify as an evangelical has no statistical significant effect on political attitudes and preferences when factoring for respondent-level similarities in demographic and religious profiles.