Author(s): GORMAN, B., WARNE, J., Institution: TU DUBLIN, Country: IRELAND, Abstract-ID: 2418

INTRODUCTION:
Sport and exercise science (SES) researchers often formulate hypotheses which contain few and vague dependent variables (DVs), but then test multiple, specific DVs with multiple statistical tests. This creates interpretive ambiguity for clinicians and coaches, and affords researchers excessive analytic flexibility. Therefore, the primary aim of this study was to investigate the ratio of the number of statistical tests conducted to the number of DVs stated within the hypothesis (stat : hypo) within applied SES studies. Secondly, we aimed to investigate if studies with a low stat : hypo ratio would have more “clearer” primary hypothesis outcomes as decided by meta-scientists and SES researchers, compared to studies with a high stat : hypo ratio.
METHODS:
269 applied SES studies from 13 quartile one SES journals met inclusion criteria. The following was coded per study, the number of:
- DVs within the hypothesis
- Statistical tests believed to be conducted
- DVs tested within the results section
Expert meta-scientists (n = 4) and SES researchers (n = 4) examined blinded samples of high-ratio and low-ratio studies, and coded the outcome of the primary hypothesis on the basis of only the results presented. The threshold for “high” and “low” ratio studies was set at the 25th and 75th percentiles respectively (using the stat: hypo). All values below are median [min, max].

RESULTS:
The median number of DVs stated within the hypothesis was 3 [1,11]. The median number of statistical tests conducted was 64 [3, 1299]. The median stat : hypo was 26 [1, 488] : 1. The median number of DVs tested was 13 [1, 107]. The median var : hypo was 4.7 [0.5, 53.5] : 1.
Clear hypothesis outcomes (“Accept” / “Reject”) as determined by meta-scientists and researchers were more frequent in low ratio studies (79.7%) compared to high ratio studies (61.6%).

CONCLUSION:
For every DV stated within the hypothesis statement SES researchers test 5 DVs producing 26 p-values. When studies had a lower ratio, there was a higher percentage of clearer outcomes as determined by meta-scientists and SES researchers. Vague hypotheses tested using multiple statistical tests doesn’t allow the reader to clearly map the hypothesis DVs to the statistical tests conducted and subsequently to the hypothesis claim. If we cannot understand the claims being made from research what is the purpose of our research? Therefore, SES researchers should instead aim to more commonly test simple hypotheses with clearly defined DVs and clearly mapped statistical tests. Until these practices become more common, practitioners should interpret applied SES findings with caution, as the conclusions may be difficult to pin to a specific outcome or decision-relevant metric. In practice, this means avoiding overconfidence in “positive” results that may reflect selective emphasis among many tested outcomes, and prioritising evidence where primary outcomes are clearly specified and adequately tested relative to their stated claim.