Polling: Methodology Disclosure
By: Dave Fako
President and Senior Strategist
Fako Research & Strategies, Inc.
NOTE:
Please see updated main phone number below.
We plan to retire the old office number in the near future.
Our current primary phone number is:
708-705-6791
As we noted in our previous newsletter, we highlighted important information on the changes occurring in public opinion research and shared
some information and announcements from Pew about changes in their methods that we felt were important.
Below are links to those articles from / about Pew and the status of polling.
With all of the changes in survey research, we also believe its time to remind consumers of public opinion research about the importance of
full and proper methodology disclosure for internal strategic data or publicly released surveys.
Numerous polls are released with minimal, if any substantive methodology disclosure. In our home state of Illinois, with a very active legislative session on numerous issues, we observed many publicly released and internal surveys with what can best be described as sub-standard methodological disclosure. In some cases, the client released polls making claims about the data that simply were not applicable, such as claiming a survey of select regions of the State represented statewide opinions. This was in part due to sub-standard methodological disclosure. With the 2020 Presidential elections gaining momentum and other major issues to be debated on the horizon, we can expect to see numerous publicly released polls.
Frequently we see methodology statements that state something to the effect of the sample size and margin of error and that cell phones were included, with minimal if any other disclosure.
So, what should be included in all methodology statements?
1.
Who paid for the Survey. This is important because it informs the consumers of the data who is sponsoring it.
2.
The name of the firm, or firms in cases where it was a joint project, that conducted the survey.
3.
The dates the survey was conducted. This is important so consumers of the data know how recent the survey was conducted.
4.
Disclosure of the data collection mode. Was it conducted via professional interviewers, Interactive Voice Recording (IVR / Robo), online, or some other technique or a combination of modes (mixed mode)?
5.
Defining the sampling technique, sample frame and sample source. This is disclosure of how the survey was conducted, the type of list(s) utilized and what was the frame, such as adults age 18+ in X, registered voters in State Legislative District X, Female Independent voters in X State, etc.
6.
Declaring the total sample size and the
exact number of interviews and their share of all interviews by mode. This means disclosing exactly how many interviews were conducted and by each mode, such as the number completed via cell phone and their share of the entire sample population.
7. Disclosure of any stratification of the sample. Many pollsters conduct surveys in proportion to certain known demographics, such as region, gender, or other definable parameters to ensure a representative sample.
8. Declaring what groups have been weighted. Virtually all polls apply weighting to various demographic groups to adjusts for sampling error to ensure the survey is a close proportional match to the known demographics of the sample population.
9. Declaring the Calculated Margin of Error and Confidence Interval. Each sample size has a margin or error calculated, assuming it was a probability sample. The confidence interval (Usually 95%) is simply a calculation inferring the level of confidence that the survey is accurate within a certain parameter and unknown variable. For example, 95% confidence interval means that the approximately 1 in every 20 surveys may be off. POLLSTERS NOTE: There are current debates in the survey research profession about calculations of margin of error due to the challenges and limits on collecting data via phone. Also, a margin of error cannot be calculated for a survey, or portion of a mixed mode survey, that is not conducted via a probability sample.
10. Identify the length of the survey (usually in minutes). This is important so the consumer knows how comprehensive the poll was and to assess what, if anything, is not included in the internal or publicly released data.
There are other, more technical levels of data that can be disclosed, such as incidence rates, cooperation rates, but for the most part this type of data is best for internal peer review.
Below is a an example of a bad methodology statement and what FR&S typically uses.
Bad Methodology Statement = The survey is a sample of 600 likely voters in Illinois with a margin or error of + / - 3.94%.
Good / FR&S Standard Methodology Statement:
M
ethodology:
The ABC Advocacy Group commissioned the survey. Fako Research and Strategies, Inc. (FR&S) of Lemont, Illinois conducted the survey by telephone on January 20 - January 24, 2019 using professional interviewers. FR&S interviewed a random sample of n=600 registered voters within the State of Illinois. The interviews lasted an average of 10 minutes.
Scientific sampling techniques using a listing of registered voters within the State of Illinois was used to give all registered voters within this group living in a telephone-equipped household, or with a listed cell phone number, an equal chance of being interviewed. Cellular phones were included within the sample and accounted for 46% (276 Completed Interviews) of all weighted interviews. The interviews were conducted in proportion to gender and regional shares of the population based on known demographics. Weighting was applied to Age, Gender, Race, and Region to bring these groups into closer proportion to known demographics.
The survey has an overall margin of error of +/- 3.94% at the 95% level of confidence. This means that if the survey were replicated the results would be consistent for 95 out of 100 cases. The margin of error is higher among the various sub-groups.
The bottom line is that as public opinion survey research continues to adapt to the challenges facing the profession in collecting reliable data, it is vital that full disclosure of how the poll was conducted. This is needed so consumers of polling have quality information to assess the reliability of the data.
For more than two decades FR&S has always adapted to the changing public opinion research profession to ensure reliability and quality in our work and we will continue to adapt to maintain our quality standards.
Dave Fako founded Fako Research & Strategies in 1998 with the goal of providing clients with quality, actionable research and strategies. Dave Fako is recognized for his attention to detail, hard work ethic, competitive drive and refusal to accept the "I can't" attitude. Dave Fako approaches all challenges and finds solutions to problems based on the winning philosophy espoused by the ancient Carthaginian General Hannibal and quoted by Saul Alinsky in his famous book, Rules for Radicals: "we will either find a way or make one". For thirty years, Dave Fako has developed strategy for campaigns at all levels throughout the country, has advised numerous public policy advocacy organizations, as well as national and regional private sector companies. Dave's expertise is state, local and judicial elections, as well as marriage equality, health care, and smoke-free issues, gaming, clean energy, education, tax and bond initiatives, telecommunications and pension and budget issues. More information on Fako Research and Strategies is found at www.fakoresearch.com.
