Here's a statement of the obvious: The opinions expressed here are those of the participants, not those of the Mutual Fund Observer. We cannot vouch for the accuracy or appropriateness of any of it, though we do encourage civility and good humor.
Thank you for the excellent reference to the Morningstar ETF article that challenges the validity of various investment back-tested strategies. Simply put, these strategies are mostly nothing more than statistical correlations.
Correlations are just collections of data that, at times, can be tortured into any desired admission. Most statisticians make the case that not all such correlations are created equally. Surely one takeaway from the referenced article is that trust should not be automatically bestowed on any research. I agree. Even if the research is finally trusted, independent verification is required.
Many correlations are nothing more than numerical trash. The piece’s author not only debunks the tactics used by unscrupulous and fraudulent researchers, he also identifies necessary criteria to sort the good from the bad analyses. We all are familiar with the nonsense correlation that coupled the butter production in Bangladesh to US equity market rewards.
I deploy many similar sorting criteria when judging the robustness of an investment correlation since data snooping is a persistent compromising issue in this field. I have long practiced a cautionary and skeptical watch-list.
Here are several criteria that I use to test (challenge) the validity of a proposed investment return correlation:
1. The size of the database and its inclusiveness. Large and contiguous data sets are better. 2. Realistic models. The independent parameters must logically support the dependent outcome in an economic sense and a business perspective. 3. Reasonable Alpha (excess profits) expectations. Outlandishly optimistic projections are immediately suspicious. Even super investors like Warren Buffett and Peter Lynch struggled to achieve Alphas in the plus 10 - 15 % category. 4. The authority, respect, and trustworthiness that the researcher has accumulated. His background and earlier work are critically assessed. 5. The incentive structure for the work. Who sponsored it and who benefits most by its findings? Payoffs influence the quality and integrity of the work; industry research is often suspect for this reason. A decoupled alignment of incentives with personal interests is mandatory. 6. Out of sample tests. The data base can be expanded by incorporating either earlier and/or later data sets. If these do not exist, the data set can be parsed or bifurcated to generate within sample test cases. The original data must contain enough entries to permit this operation. 7. Independent verification. Can other researchers duplicate the reported work? It is amazing that even in the academic scientific community, a measurable fraction of the supposed research is totally fabricated or the data is falsified.
Statistics can be invented or the data can be manipulated to produce predetermined goals. We must all guard against that possibility. Your referenced article provides useful guidelines that contribute to that task.
Once again, thank you for the reference. I’ll certainly add some of the insights presented in the Morningstar paper to my incomplete watch-list. Its author did a splendid job. Indeed, some statistically-based analysis produces Fool’s Gold.
5. The incentive structure for the work. Who sponsored it and who benefits most by its findings? Payoffs influence the quality and integrity of the work; industry research is often suspect for this reason. A decoupled alignment of incentives with personal interests is mandatory. UUGGHH ,would that include the Congressional Budget Office?
Thanks again your insight, MJG. From the link below; On June 25th of 1889 the statistician Carroll D. Wright gave the opening remarks at a Convention of Commissioners of Bureaus of Statistics of Labor. Two different versions of his remarks were published, and excerpts from both are presented below. Wright is recorded using the proverb in both versions, but QI believes that the first version given below displays the greatest fidelity to Wright’s speech. The second version is described as a “condensed report”.
In the first version Wright labels part of the expression “a new saying”, and this is evidence that he is not the creator of the aphorism. This is consistent with the citations presented above which show the saying was already being disseminated. Here are Wright’s words [CDW1]:
The old saying is that “figures will not lie,” but a new saying is “liars will figure.” It is our duty, as practical statisticians, to prevent the liar from figuring; in other words, to prevent him from perverting the truth, in the interest of some theory he wishes to establish. We can only do this by being absolutely fair ourselves. The second version It has been said that figures will not lie. It is equally true that liars will figure. It is our duty to prevent liars from figuring in the interest of any theory, by presenting original data fairly. http://quoteinvestigator.com/2010/11/15/liars-figure/
Thank you for the thoughtful reply. A special thank you for the superior reference; I was not aware of the existence of Quote Investigator. I’ll surely use its service in the future. It is a fascinating subject.
I completely agree that it is everyone’s challenge to recognize, respond, and report the many statistical untruths and distortions that are commonly promulgated. Politicians of all stripes are especially guilty of statistical misrepresentation and fact cherry picking. Certainly the financial community is not immune to this manmade disease.
Since much of our population is statistically innumerate, we all fall victim to these devilish tactics because we are a democracy dedicated to majority rule. Again, it’s a buyer beware scenario.
Please keep fighting the good fight and have a Happy Thanksgiving.
Comments
Thank you for the excellent reference to the Morningstar ETF article that challenges the validity of various investment back-tested strategies. Simply put, these strategies are mostly nothing more than statistical correlations.
Correlations are just collections of data that, at times, can be tortured into any desired admission. Most statisticians make the case that not all such correlations are created equally. Surely one takeaway from the referenced article is that trust should not be automatically bestowed on any research. I agree. Even if the research is finally trusted, independent verification is required.
Many correlations are nothing more than numerical trash. The piece’s author not only debunks the tactics used by unscrupulous and fraudulent researchers, he also identifies necessary criteria to sort the good from the bad analyses. We all are familiar with the nonsense correlation that coupled the butter production in Bangladesh to US equity market rewards.
I deploy many similar sorting criteria when judging the robustness of an investment correlation since data snooping is a persistent compromising issue in this field. I have long practiced a cautionary and skeptical watch-list.
Here are several criteria that I use to test (challenge) the validity of a proposed investment return correlation:
1. The size of the database and its inclusiveness. Large and contiguous data sets are better.
2. Realistic models. The independent parameters must logically support the dependent outcome in an economic sense and a business perspective.
3. Reasonable Alpha (excess profits) expectations. Outlandishly optimistic projections are immediately suspicious. Even super investors like Warren Buffett and Peter Lynch struggled to achieve Alphas in the plus 10 - 15 % category.
4. The authority, respect, and trustworthiness that the researcher has accumulated. His background and earlier work are critically assessed.
5. The incentive structure for the work. Who sponsored it and who benefits most by its findings? Payoffs influence the quality and integrity of the work; industry research is often suspect for this reason. A decoupled alignment of incentives with personal interests is mandatory.
6. Out of sample tests. The data base can be expanded by incorporating either earlier and/or later data sets. If these do not exist, the data set can be parsed or bifurcated to generate within sample test cases. The original data must contain enough entries to permit this operation.
7. Independent verification. Can other researchers duplicate the reported work? It is amazing that even in the academic scientific community, a measurable fraction of the supposed research is totally fabricated or the data is falsified.
Statistics can be invented or the data can be manipulated to produce predetermined goals. We must all guard against that possibility. Your referenced article provides useful guidelines that contribute to that task.
Once again, thank you for the reference. I’ll certainly add some of the insights presented in the Morningstar paper to my incomplete watch-list. Its author did a splendid job. Indeed, some statistically-based analysis produces Fool’s Gold.
Happy Thanksgiving.
Best Regards.
UUGGHH ,would that include the Congressional Budget Office?
Thanks again your insight, MJG.
From the link below;
On June 25th of 1889 the statistician Carroll D. Wright gave the opening remarks at a Convention of Commissioners of Bureaus of Statistics of Labor. Two different versions of his remarks were published, and excerpts from both are presented below. Wright is recorded using the proverb in both versions, but QI believes that the first version given below displays the greatest fidelity to Wright’s speech. The second version is described as a “condensed report”.
In the first version Wright labels part of the expression “a new saying”, and this is evidence that he is not the creator of the aphorism. This is consistent with the citations presented above which show the saying was already being disseminated. Here are Wright’s words [CDW1]:
The old saying is that “figures will not lie,” but a new saying is “liars will figure.” It is our duty, as practical statisticians, to prevent the liar from figuring; in other words, to prevent him from perverting the truth, in the interest of some theory he wishes to establish. We can only do this by being absolutely fair ourselves.
The second version
It has been said that figures will not lie. It is equally true that liars will figure. It is our duty to prevent liars from figuring in the interest of any theory, by presenting original data fairly.
http://quoteinvestigator.com/2010/11/15/liars-figure/
Hi TSP Transfer,
Thank you for the thoughtful reply. A special thank you for the superior reference; I was not aware of the existence of Quote Investigator. I’ll surely use its service in the future. It is a fascinating subject.
I completely agree that it is everyone’s challenge to recognize, respond, and report the many statistical untruths and distortions that are commonly promulgated. Politicians of all stripes are especially guilty of statistical misrepresentation and fact cherry picking. Certainly the financial community is not immune to this manmade disease.
Since much of our population is statistically innumerate, we all fall victim to these devilish tactics because we are a democracy dedicated to majority rule. Again, it’s a buyer beware scenario.
Please keep fighting the good fight and have a Happy Thanksgiving.
Very Best Wishes.