This is part 1 of a series of three webinars

• Part 1; Introducing Reproducible (Open) Science (June 11th, 2020)
• Part 2; Managing your project files and data with ‘Guerilla Analytics’ (June 23rd, 2020)
• Part 3; Reproducible (Open) Science - Tools (July 6th, 2020)

The complete source code for the webinars and all dependent data, and files can be found on Github.com/uashogeschoolutrecht.

1. When things go wrong
2. Why Reproducible (Open) Science?
3. The need for learning programming
4. An example of Reproducible (Open) Science

\(Reproducible\ (Open)\ Science =\) \(Reproducible\ Research + Open\ Science\)

As you probably know, hydroxychloroquine was repeatedly touted as a promising cure for COVID-19 by US President Donald Trump

https://www.washingtonpost.com/politics/2020/04/07/trumps-promotion-hydroxychloroquine-is-almost-certainly-about-politics-not-profits/

https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)31180-6/fulltext

“Our independent peer reviewers informed us that Surgisphere would not transfer the full dataset, client contracts, and the full ISO audit report to their servers for analysis as such transfer would violate client agreements and confidentiality requirements”

• Company Surgisphere (‘data owner’) did not share raw data
• At time of publication (raw) data and analysis (code) was not included in the manuscript
• The authors initiated the retract

https://www.sciencemag.org/news/2020/06/two-elite-medical-journals-retract-coronavirus-papers-over-data-integrity-questions

• Scientific conclusions get picked up by the media, retracting statements is difficult
• The credibility of the Journal, the researchers and the affiliated institutions are at stake (people got sacked over this!)
• Clinical studies to hydroxy(choloroquine) were halted because of this paper
• The credibility of the company Surgisphere is at stake (they should have prevented this…)
• The credibility of Science as a whole is at stake (‘in the eye of the beholder’)

• To assess validity of science and methods we need access to data, methods and conclusions
• To learn from choices other researchers made
• To learn from omissions, mistakes or errors
• To prevent publication bias (also negative results will be available in reproducible research)
• To be able to re-use and/or synthesize data (from many and diverse sources)
• To have access to it all!

Nature Collection on this topic

How would you ‘describe’ the steps of an analysis or creation of a graph when you use GUI* based software?

“You can only do this using code, so it is (basically) impossible in a GUI”**

*Graphical User Interface (GUI)…is a form of user interface that allows users to interact with electronic devices through graphical icons and audio indicator such as primary notation, instead of text-based user interfaces, typed command labels or text navigation…

**The file “./Rmd/steps_to_graph_from_excel_file.html” shows you how to do this using the programming language R. In webinar part 3, we will revisit this example.

• Scientific context, research questions and state of the art [P]
• (Experimental) model or characteristics of population or matter studied [P]
• Data that was generated and corresponding meta data [D, C]
• Exact (experimental) design of the study [P, D, C]
• Exploratory data analysis of the data [P, C]
• Exact methods that were used to conduct any formal inference [P, C]
• Model diagnostics [C]
• Interpretations of the (statistical) model results/model fitting process [P, C]
• Conclusions and academic scoping of the results [P, C]
• Access to all of the above [OAcc, OSrc]

\(P = Publication\), \(D = Data\), \(C = Code\), \(OAcc = Open\ Access\), \(OSrc = Open\ Source\)

Assume we have the following question: “Which of 4 types of chairs takes the least effort to arise from when seated in?” We have the following setup:

• 4 different types of chairs
• 9 different subjects (probably somewhat aged)
• Each subject is required to provide a score (from 6 to 20, 6 being very lightly strenuous, 20 being extremely strenuous) when arising from each of the 4 chairs. There is some ‘wash-out’ time in between the trials. The chair order is randomised.

To analyze this experiment statistically, the model would need to include: the rating score as the measured (or dependent) variable, the type of chair as the experimental factor and the subject as the blocking factor

A typical analysis method for this type of randomized block design is a so-called ‘multi-level’ or also called ‘mixed-effects’ or ‘hierarchical’ models. An analysis method much used in clinical or biological scientific practice.

You could also use one-way ANOVA but I will illustrate why this is not a good idea

Wretenberg, Arborelius & Lindberg, 1993

library(nlme)
ergoStool %>% as_tibble()

## # A tibble: 36 x 3
##    effort Type  Subject
##     <dbl> <fct> <ord>  
##  1     12 T1    1      
##  2     15 T2    1      
##  3     12 T3    1      
##  4     10 T4    1      
##  5     10 T1    2      
##  6     14 T2    2      
##  7     13 T3    2      
##  8     12 T4    2      
##  9      7 T1    3      
## 10     14 T2    3      
## # ... with 26 more rows

• Can you say something about within-subject variability (note ‘Minster Blue’)?
• Can you say something about between-subject variability (note ‘Mister Green’, vs ‘Mister Black’)?
• Which chair type takes, on average the biggest effort to arise from?

1. Which chair type takes, on average the biggest effort to arise from? (ANOVA / MEM, fixed effects)

• Do individual (within subject) differences play a role in appointing a average score to a chair type? (MEM, random effects)
• Does variability between subjects play a role in determining the ‘best’ chair type (ANOVA / MEM, confidence intervals)

Statistical models (in R) can be specified by a model formula. The left side of the formula is the dependent variable, the right side are the ‘predictors’. Here we include a fixed and a random term to the model (as is common for mixed-effects models)

library(nlme)

ergo_model <- lme(
  data = ergoStool, # the data to be used for the model
  fixed = effort ~ Type, # the dependent and fixed effects variables
  random = ~1 | Subject # random intercepts for Subject variable
)

The lme() function is part of the {nlme} package for mixed effects modelling in R

Example reproduced from: Pinheiro and Bates, 2000, Mixed-Effects Models in S and S-PLUS, Springer, New York.

plot(ergo_model) ## type = 'pearson' (standardized residuals)

odz: Practice what you preach

If you want to reproduce, add-on, falsify or apply your own ideas to this example, you can find the code (and data) in Github.com

In webinar 3, I will show you how to actually run, use and organize code like this!

UPCOMING WEBINARS:

• Part 2; Managing your project files and data with ‘Guerilla Analytics’ (~June 23rd, 2020)
• Part 3; Reproducible (Open) Science @HU - Tools (~July 6th, 2020)

Peer Support Group Data Science support voor onderzoek