+ - 0:00:00
Notes for current slide
  • code of conduct
    • respectful of others and their opinions
    • harassment will not be tolerated
  • Please feel free to interrupt with questions, preferably through chat or Zoom's raise hand feature
  • for those not familiar with Xaringan slides, "h" brings up a help menu for navigating the slides
  • photo credits are also listed in the presenter notes

Photo by Severin Höin on Unsplash

Notes for next slide
1/23
  • code of conduct
    • respectful of others and their opinions
    • harassment will not be tolerated
  • Please feel free to interrupt with questions, preferably through chat or Zoom's raise hand feature
  • for those not familiar with Xaringan slides, "h" brings up a help menu for navigating the slides
  • photo credits are also listed in the presenter notes

Photo by Severin Höin on Unsplash

A little about me

I'm a fourth year doctoral candidate in the Quantitative Methods program within the Psychology Department at York University (Toronto, Canada).

2/23

A little about me

I'm a fourth year doctoral candidate in the Quantitative Methods program within the Psychology Department at York University (Toronto, Canada).

You find out more about my projects and other workshops I have conducted at www.standard-deviator.com

2/23
  • I attended the Oxford | Berlin Summer School on Open Research in the fall of 2019
  • Started a local branch of the Reproducibilitea journal clubs here at York University
  • I run workshops which emphasize reproducible research
  • My own research focuses on current practices for estimating and interpreting effect sizes, improving statistical pedagogy, and developing tools which instructors can implement within their own classrooms to help improve statistical literacy, thinking, and reasoning.
  • I need you help. Please politely interrupt me or message in chat/raise your hand if you want further clarification or have a question.
    • This workshop will be more beneficial to everyone by it being interactive. After all, you can just watch a video on YouTube or read one of the many excellent blog posts explaining the process

A little about you

3/23

I was told to fill out a prereg
I heard it might be a good thing to do
I have done one before, but I want to get better at them
The workshop was free, so I figured "why not?"

Center for Open Science (COS)

Open Science Framework (OSF)

4/23

Demonsrate COS link

Open Researcher and Contributor ID (ORCID)

Some Benefits:

  • You will be distinguished from every other researcher, even researchers who share your same name,
  • Your research outputs and activities will be correctly attributed to you,
  • Your contributions and affiliations will be reliably and easily connected to you,
  • You will enjoy improved discoverability and recognition,
  • Your ORCID record is yours, for free, forever.

https://orcid.org/

5/23

Some of the journals which I submit my manuscripts to requires an Orcid ID, I mainly use it to signin to OSF and link my projects. Another great benefit is that this ID stays tied to you, even if you change institutions. You might want to use a personal/stable email address in case you no longer have access to your institutional email.

What is preregistration?

"...you're simply specifying your research plan in advance of your study and submitting it to a registry."

www.cos.io/initiatives/prereg

6/23

What is preregistration?

"...you're simply specifying your research plan in advance of your study and submitting it to a registry."

www.cos.io/initiatives/prereg

"Preregistration of an analysis plan is committing to analytic steps without advance knowledge of the research outcomes"

Nosek, Ebersole, DeHaven, & Mellor, 2015

6/23

What is preregistration?

"...you're simply specifying your research plan in advance of your study and submitting it to a registry."

www.cos.io/initiatives/prereg

"Preregistration of an analysis plan is committing to analytic steps without advance knowledge of the research outcomes"

Nosek, Ebersole, DeHaven, & Mellor, 2015

There are some key things which should be present for any preregistration:

  • it should relate to a specific study, not a class of studies
  • ideally, the preregistration should be public and findable/accessible by anyone
  • authors should state data collection and analysis plans prior

Navarro, 2019

6/23

So before you sit down to start a preregistration, I encourage you to think carefully about what you aim to achieve via this process

What a preregistration is not?

Preprint

  • A manuscript posted (usually after data collection) on a public repository which has not been peer-reviewed
  • https://psyarxiv.com/
7/23

What a preregistration is not?

Preprint

  • A manuscript posted (usually after data collection) on a public repository which has not been peer-reviewed
  • https://psyarxiv.com/

Registered Report

7/23

Badges

  1. A public date-time stamped registration is in an institutional registration system (e.g., ClinicalTrials.gov, Open Science Framework, AEA Registry, EGAP).
  2. Registration pre-dates the intervention.
  3. Registered design corresponds directly to reported design.
  4. Full disclosure of results in accordance with registered plan.
8/23

Badges

  1. A public date-time stamped registration is in an institutional registration system (e.g., ClinicalTrials.gov, Open Science Framework, AEA Registry, EGAP).
  2. Registration pre-dates the intervention.
  3. Registered design corresponds directly to reported design.
  4. Full disclosure of results in accordance with registered plan.

3.Registered design and analysis plan corresponds directly to reported design and analysis.

Check out the badges

8/23

A general warning, I know that no one hear is doing a prereg just for a badge, but there is concern that trying to achieve these badges will just become the new ritual which researchers "mindlessly" follow. However, I think that much good would come from the typical increase in transparency.

Transparency Openness Promotion (TOP) Guidelines

"The TOP Guidelines were created by journals, funders, and societies to align scientific ideals with practices."

https://www.cos.io/initiatives/top-guidelines

The Open Science Framework (OSF) maintains a database which ranks journal policies in terms of their compliance with these guidelines. This database can be used to see if your target journal requires preregistration ro any of the other eight modules contained in the TOP guidelines.

9/23

Demonstrate the database link

What preregistrations can do

10/23

What preregistrations can do

It depends on your purpose!

10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research
10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research

  • reduce researcher degrees of freedom

10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research

  • reduce researcher degrees of freedom

  • reduce file drawer effect

    • though publishing a preprint would be more effective
10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research

  • reduce researcher degrees of freedom

  • reduce file drawer effect

    • though publishing a preprint would be more effective

  • reduce publication bias

10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research

  • reduce researcher degrees of freedom

  • reduce file drawer effect

    • though publishing a preprint would be more effective

  • reduce publication bias

  • help reduce unintentional false positive inflation of results (Forstmeier, Wagenmakers, & Parker, 2017)

10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research

  • reduce researcher degrees of freedom

  • reduce file drawer effect

    • though publishing a preprint would be more effective

  • reduce publication bias

  • help reduce unintentional false positive inflation of results (Forstmeier, Wagenmakers, & Parker, 2017)

  • be a tool to help clarify your thinking

10/23

What preregistrations can do

It depends on your purpose!

A preregistration can ...

  • improve the transparency of your research

  • reduce researcher degrees of freedom

  • reduce file drawer effect

    • though publishing a preprint would be more effective

  • reduce publication bias

  • help reduce unintentional false positive inflation of results (Forstmeier, Wagenmakers, & Parker, 2017)

  • be a tool to help clarify your thinking

  • delineate prediction from postdiction (i.e., exploratory versus confirmatory analyses)

10/23

What preregistrations can do

Mistaking postdiction as prediction underestimates the uncertainty of outcomes and can produce psychological overconfidence in the resulting findings.

Nosek, Ebersole, DeHaven, & Mellor, 2015

11/23

I like to think of it like a game of pool. What's more believable: a called shot who someone who just hits the balls really hard and one happens to go in?

Photo by Alexandre Lion on Unsplash

What preregistrations can NOT do for you?

Preregistration cannot ...

  • improve your research simply through the act of preregistering it
12/23

What preregistrations can NOT do for you?

Preregistration cannot ...

  • improve your research simply through the act of preregistering it

  • increase the transparency of your research (however a "good" preregistration can)

12/23

What preregistrations can NOT do for you?

Preregistration cannot ...

  • improve your research simply through the act of preregistering it

  • increase the transparency of your research (however a "good" preregistration can)

  • prevent p-hacking and other questionable research practices (QRPs)

12/23

A preregistration plan is NOT a prison sentence

  • It is perfectly fine to deviate from your preregistration plan, so long as you clearly state where and WHY you deviated from the plan
  • The closer you stick to the plan, the more credibility your predictions will have

https://www.cos.io/blog/preregistration-plan-not-prison

13/23

This is an opinion I have encountered both online and in person. The crux is that by having to preregister everything leaves little room for the creative process of making inquiries about nature.

You can still do those things. A prereg plan merely delineates one from the other.

I think the claim that it limits scientific curiosity is more of a statement about a researcher than it is of this tool.

Photo by Denny Müller on Unsplash

What should be included in a preregistration?

This is a tricky question with no clear answer.

As a methodologist, the answer is:

14/23

What should be included in a preregistration?

This is a tricky question with no clear answer.

As a methodologist, the answer is:

It depends!

14/23

What should be included in a preregistration?

This is a tricky question with no clear answer.

As a methodologist, the answer is:

It depends!

  • On the type of research you are conducting (confirmatory or exploratory)
  • On the level of transparency you are striving for
  • the time you have to front-load on your project
14/23

What should be included in a preregistration?

This is a tricky question with no clear answer.

As a methodologist, the answer is:

It depends!

  • On the type of research you are conducting (confirmatory or exploratory)
  • On the level of transparency you are striving for
  • the time you have to front-load on your project

Whatever you choose to include should be in service of your reason for completing a preregistration plan.

14/23

What should be included in a preregistration?

If the goal is to limit researcher degrees of freedom, then care must be taken at each stage:

  • planning
  • running
  • analyzing
  • reporting
15/23

What should be included in a preregistration?

If the goal is to limit researcher degrees of freedom, then care must be taken at each stage:

  • planning
  • running
  • analyzing
  • reporting

I suggest you check out this article by Wicherts, Veldkamp, Augusteijn, Bakker, van Aert, van Assen; 2016 which lists researcher degrees of freedom across the many stages in their Table 1.

15/23

What should be included in a preregistration?

If the goal is to limit researcher degrees of freedom, then care must be taken at each stage:

  • planning
  • running
  • analyzing
  • reporting

I suggest you check out this article by Wicherts, Veldkamp, Augusteijn, Bakker, van Aert, van Assen; 2016 which lists researcher degrees of freedom across the many stages in their Table 1.

You can also follow this guide which has a checklist for what to include

Download Guide to Pre-Analysis Plans from OSF.io

15/23

What should be included in a preregistration?

An often overlooked part of the preregistration process is how to avoid "Questionable Measurement Practices" (QMRs).

Flake and Fried (2020) listed a series of 6 questions which were developed to promote transparent reporting of measurement practices.

  1. What is your construct?
  2. Why and how did you select your measure?
  3. What measure dd you use to operationalize the construct?
  4. How did you quantify your measure?
  5. Did you modify the scale? And if so, how and why?
  6. Did you create the measure on the fly?
16/23

Let's try to complete a Preregistration

Notes:

There is no single way to do this. I'm just showing my preferred method.

I will be using my ORCID to access OSF and create a repository.

17/23

Create A Preregistration

  1. Log into https://www.osf.io/ using my ORCID
  2. From the OSF dashboard, click "Create new project" button
    • Enter Title, Storage Location, and a description
  3. Within the new project page, select the "Registrations" tab from the top navbar, then press the "New registration button"
    • select the template most appropriate for your project (you can preview them at https://osf.io/zab38/wiki/home/)
    • I will select the standard, comprehensive, and general purpose form
      • Metadata
      • Study Information
      • Design Plan
      • Sampling Plan
      • Variables
      • Analysis Plan
      • Other

https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration

18/23

Tips

It can be helpful to visualize parts of your design or analysis using a flowchart, http://ecomodeler.org/ is a great/free tool for generating decision trees and flowcharts.

  • You can export the image and the model file to upload as part of your OSF project or a preregistration plan
19/23

Tips

It can be helpful to visualize parts of your design or analysis using a flowchart, http://ecomodeler.org/ is a great/free tool for generating decision trees and flowcharts.

  • You can export the image and the model file to upload as part of your OSF project or a preregistration plan

Sometimes you might want to include a decision tree which outlines how you might proceed WHEN particular analytics conditions arise (e.g., violated model assumptions)

19/23

Tips

It can be helpful to visualize parts of your design or analysis using a flowchart, http://ecomodeler.org/ is a great/free tool for generating decision trees and flowcharts.

  • You can export the image and the model file to upload as part of your OSF project or a preregistration plan

Sometimes you might want to include a decision tree which outlines how you might proceed WHEN particular analytics conditions arise (e.g., violated model assumptions)

If you are familiar with the variables and analysis plan, you can upload a script or syntax file which you will use to conduct your analysis.

19/23

Demonstrate ecomodeler.org Show my prereg_script.R

Reporting Preregistered Research

  • Include a link to the original preregistration plan
  • Report the results of all prespecified tests (e.g., tests)
  • Clearly label any unplanned analyses
  • Include a transparent changes document

https://help.osf.io/hc/en-us/articles/360021390833-Preregistration

20/23

Closing Remarks

"Every research practice comes with costs and benefits, and it is useful to evaluate whether and when it is worth preregistering your study"

"If the use of a tool is detached from a philosophy of science it risks becoming a heuristic. Researchers should not choose to preregister because it has become a new norm, but they should preregister because they can justify based on their philosophy of science how preregistration supports their goals."

Lakens, 2019

21/23

Questions

22/23

Advanced Tips and Tricks

23/23

A little about me

I'm a fourth year doctoral candidate in the Quantitative Methods program within the Psychology Department at York University (Toronto, Canada).

2/23
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow