A study of NYC Carriage horses by Western University is being promoted as the definitive study of carriage horse stress. Dr. Joseph Bertone, DVM, MS, DACVIM, WesternU College of Veterinary Medicine (CVM) Professor of Equine Medicine, apparently approached the carriage trade to sample salivary and fecal cortisol levels in a selection of horses:
“The WesternU CVM group measured saliva cortisol from the animals at multiple time points in the workday. They also measured medial canthus temperature using a FLIR thermal imaging camera (IRT), and collected feces samples for fecal cortisol.”
Four time points were used during testing:
• Time point 1: feces, saliva and IRT tests were collected an hour before horses went to work.
• Time point 2: saliva and IRT were collected after the horses were harnessed and as the horses walked were hitched to their carriages.
• Time point 3: saliva and IRT were collected as the horses returned from working.
• Time point 4: saliva and IRT were collected an hour after they worked, as they rested in their stalls.
As you might expect due to the contentious issues surrounding the horses, it has drawn both supporters and detractors. I realize that what was posted on the WesternU website is a press release and not an abstract, but I was surprised that Dr. Bertone immediately set about politicizing the study by mentioning NYC Mayor Bill deBlasio.
Normally, scientists don’t concern themselves with political incumbents, because politics does not and should not drive scientific outcomes. He went on to state that “I’m also concerned over claims that could dismantle, or likely end, the lives of these grand horses,” and “the loss of an iconic New York City institution, the loss of the important human-animal bond the drivers have with these spectacular animals, [would] have a profound negative economic impact on the people whose lives this would touch.” As a veterinarian, he probably can’t be equally concerned with both those issues. And why would the lives of the horses come to an end? Would the owners send them to auction where they might then be shipped to slaughter? If so, only the owners could be blamed for that. Dr. Bertone then proceeded to draw animal-rights advocates into the conversation, apparently not understanding that animal-rights advocates also own animals including horses. So his politicized preamble is surprising as are the statements that followed. He needs to let his study stand on its own and either succeed or fail on its own merits.
What’s potentially wrong with this study?
It’s probably not possible for anyone to obtain a copy of the manuscript or even the abstract until publication. Authors submit their manuscripts to the Journal of the American Veterinary Medical Association with the understanding that the manuscripts and their contents will be kept confidential until they are published. So until the study is published, no one can really know how well it was conducted. Framing the press release with comments about Bill deBlasio is probably not very suggestive of an impartial study, which ideally should conform with frameworks for research ethics that demand that the researcher be able to show that their research is independent and impartial by:
- Avoiding assumptions
- Writing in a style that is formal and depersonalized, avoiding the use of personal expressions
- Starting with a premise and impartially proceeding to the conclusion, which may or may not support your premise
- Disclosing a professional or financial of interest by a study’s authors. Although such conflicts do not necessarily disqualify a reviewer, they should be considered when an editor or peer reviewer makes a decision about a manuscript’s disposition.
- Including numerical results that make mention of standard deviations or 95% confidence limits and the degree of statistical significance.
Other characteristics of the study stand out:
- Why were only 13 horses out of approximately 220 selected?
- Why were all the horses chosen from the Clinton stable?
- Were the horses tested on pasture these same 13 horses or different ones? If different, why? How long were they there before they were tested?
- Why was the study period so short (August 3 – 5, 2014) By comparison, here is a salivary cortisol study that was truly longitudinal in nature. This study took place over six months, not three days, in an effort to analyze the subtle variations that take place over time.
- Salivary cortisol follows a diurnal rhythm with the highest concentrations in the morning and a decrease throughout the day. How did Dr. Bertone control for these factors?
- Why were none of the horses tested at the hackline or after arriving at the hackline after their trip from the stables?
- Bertone tells us that “behavior associated with equine gastric ulcers was observed.” What behaviours did they observe for? But why not test for gastric ulcers while they had the feces samples? Fecal tests for pathological conditions of the GI tract is a simple ELISA test, which can detect most (but not all) gastric ulcers. Why not just take this extra step and know with reasonable certainty whether ulcers were a factor?
I’ve seen several comments about this study from pro-carriage people operating with the view that once it’s peer-reviewed it will be unassailable as to its accuracy. There’s actually a lot of confusion about the “peer review” process and what a “peer-reviewed journal” is. There are plenty of impartial, scientific journals out there, whose peer review process is to guard scientific integrity. However, there are also as many journals founded and funded by an industry or a professional group, whose peer-review is intended to protect the interests of that industry or group.
The American Veterinary Medical Association is a professional group. Sometimes its journal publishes articles with scientific intent. Other times it publishes articles that are intended to protect financial interests of its members — and in those cases, its “peer review” is done by hand-picked “peers” who will reject anything that doesn’t support a purely political position the AVMA or other self-interested groups are protecting. Furthermore, many journals have been systematically pulling studies that were peer-reviewed once corruption or falsification was determined. So peer review works great in theory and in practice most of the time, but it’s hardly infallible. It won’t make any study immune to critique.
John Ioannidis holds the C.F. Rehnborg Chair in Disease Prevention at Stanford University, and he is Professor of Medicine, Professor of Health Research and Policy, and Director of the Stanford Prevention Research Center at Stanford University School of Medicine. In 2005 he wrote an essay, “Why Most Published Research Findings Are False.” In this essay he describes the false positive findings, biases, conflicts, and design flaws prevalent in modern research. In his essay he identified several factors associated with research fields susceptible to false positive (or false negative) findings:
- Samples are typically small (13 of 220+ carriage horses tested)
- Effect sizes are typically small (we’ll know more about this once the study is published)
- Large number of potential relations that could be explored
- Flexibility in design, definitions, and outcomes (at what cortisol levels are horses considered “stressed?”)
- Popular subject matter (carriage horses in NYC are a hotly discussed topic and the Dr. Bertone admits that he contacted the carriage trade in order to prove wrong the accusations that the horses were unhappy)
- Topics have potential financial or political interest (tremendous political interest, financial interest if the carriage trade is discontinued, and apparent conflicts of interest with regard to the source of the funding for the study)
With the exception of Przewalski’s horses (who have a different number of chromosomes) domestic horses and wild horses are genetically exactly the same animal. That means that the horse living in your back yard or at a stable somewhere is genetically the same as the horse who evolved in the wild and those still living in the wild. It makes no difference that most horses we have were all born in captivity. Our horse’s genetics are still the same as those horses who roamed across North America thousands of years ago.
Some people believe that a few hundred years of selective breeding can change all that, but we know that it takes a few thousand years to even begin to change the genetics of any species. Which means the horses in our back yard have been programmed for hundreds of thousands of years to live in wide open spaces where they can see predators coming, eat grass and other forage for 18 hours a day, move 10 miles a day on unshod feet and spend the day with multiple other horses for safety and security. That’s why the suggestion by Dr. Bertone that pastured horses *may* be under greater supposed stress than the carriage horses are in the city is ridiculous – horses were born to live on grasslands – they did not evolve to work in cities. Unless you stressed a horse by putting it in a pasture the day before with a bunch of unfamiliar horses, most horse owners would find that premise laughable.
It’s important that studies and their designers work through an impartial evaluative lens. Studies funded by a group that can benefit from the outcomes are suspect. They can never truly be impartial.