A prominent statistician, Carroll D. Wright once said when addressing the Convention of Commissioners of the Bureau of Statistics of Labor back in 1889: “The old saying is that ‘figures will not lie,’ but a new saying is ‘liars will figure.’ It is our duty, as practical statisticians, to prevent the liar from figuring; in other words, to prevent him from perverting the truth, in the interest of some theory he wishes to establish.”
All too often people attempt to prove a point using statistics or research figuring that numbers are factual evidence of the truth. The problem often is that many people use numerical information incorrectly, either innocently or with a motive to mislead. Numbers are manipulated all the time when trying to prove a point or win a debate, whether by deliberate misuse, negligence, or plain incompetence.
So how can factual data so often be wrong? Well, it occurs in one or both of the following areas.
First, numbers must be gathered. If they are collected incorrectly, or by someone with an agenda or bias, then the data is flawed and the conclusion can be suspect.
Second, numbers must be analyzed or interpreted. Again, this process can be done incorrectly, or misused by an individual or group. Once you learn what to look for in these two areas, you can evaluate the numerical data you encounter and rely on it only when it is objective and correct.
Here are some examples of how data collection or assumptions can impact the accuracy or objectivity of research:
Marketers, politicians and the media all use statistics regularly to support points of view or to attempt to validate or disprove activities or public opinion. Problem is, if numbers are not gathered accurately or objectively, they can result in misleading conclusions.
A few things that should be considered when determining if surveys were conducted accurately are:
- Sample Design
- Sample Size – if the sample number is too low, it won’t be representative of a larger population; asking just two people if they like a new product and finding that one person does doesn’t mean that 50% of all potential product users, a number that could be in the millions, arrive at the same opinion.
- Target Population Profile – if the target population includes all adults using financial services, then selecting participants from consumers that just use financial service centers to cash a check or take a payday loan will not be representative of how the general population manages their finances
- Question Structure. Questions must be structured in an objective, non-threatening, non-influencing manner. Compare, “Do you think people should be allowed to borrow money at 429% annual interest rates?” to “Do you think people with limited access to short term credit should be able to decide whether or not they want to borrow money if they understand all of the fees, terms and conditions?” The second question is obviously much less biased in its approach.
Margin of Error
Remember, surveys can’t prove anything with 100% certainty unless they ask the questions to 100% of the population. Most survey results end with a statement such as “there is a margin of error of three percentage points.” What does this mean? It tells how confident the surveyors are that their results are correct; the lower the percentage or margin of error, the greater their confidence in projecting the results to the whole population being studied.
Once numbers are gathered, they must be interpreted or evaluated, and this step affords many opportunities to distort the truth. For example, researchers often do correlation studies to find out if a link exists between two sets of data. One such example is a study published by The Center For Responsible Lending (“CRL”) back in 2005 entitled “Race Matters: The Concentration of Payday Lenders in African-American Neighborhoods in North Carolina” that intended to impugn the motives of the industry to target the African-American community by correlating the number of stores located in census tracts that were heavily populated by this ethnic group. A subsequent paper critiquing this study by Thomas Lehman, Ph.D. an Associate Professor of Economics from Indiana Wesleyan University clearly demonstrated that the CRL study “… fails to consider a host of potential determinants of storefront location that go well beyond the demographic composition of the census tract in which the store is located. Thus, there are likely a number of omitted variables that may explain storefront location decisions but which were excluded from the model developed by the authors of the study.” The refutation goes on to explain many other variables that could determine store location strategies beyond racial profiles that disprove this correlation.
Statistics is simply a mathematical science that gathers information about a population so that group may be described usefully. Statistics are often used to draw conclusions and make decisions based on that information. So, what’s the problem?
In general problems with statistics are similar to those of other types of numerical data; namely, they can be gathered, analyzed, and/or interpreted incorrectly, or mishandled by someone with a bias.
For example, how many times have you read an article that implies that only poor people that are closed out of the banking system use check cashing stores? The fact is that most people use check cashing stores by choice; and as many as half of the consumers that use check cashing stores have bank accounts. While the statistics may indicate that the median incomes of those using check cashing services are below those using banks, it is not an absolute conclusion.
In summary, it is just as easy to deceive with numbers as it is with words. Surveys, studies, and statistics are conducted and interpreted by researchers who might have a bias, or simply lack the skills necessary to do their jobs properly. Therefore, it is important to evaluate numbers before accepting them as truth. Ask questions about how the information was gathered, what its margin of error is, and how meaningful it is. Does the conclusion make sense, or does it seem to distort the findings? Thinking critically about the many numbers you encounter will help you to rely only on information that is objective and accurate.