layout: true --- background-position: center background-size: contain background-image: url(images/Preregistration-a-practical-guide-splash.jpg) ??? * code of conduct * respectful of others and their opinions * harassment will not be tolerated * Please feel free to interrupt with questions, preferably through chat or Zoom's raise hand feature * for those not familiar with Xaringan slides, "h" brings up a help menu for navigating the slides * photo credits are also listed in the presenter notes Photo by <a href="https://unsplash.com/@sevhoein?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Severin Höin</a> on <a href="https://unsplash.com/s/photos/guide?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a> --- # A little about me I'm a fourth year doctoral candidate in the Quantitative Methods program within the Psychology Department at York University (Toronto, Canada). -- You find out more about my projects and other workshops I have conducted at [www.standard-deviator.com](http://www.standard-deviator.com) ??? * I attended the Oxford | Berlin Summer School on Open Research in the fall of 2019 * Started a local branch of the Reproducibilitea journal clubs here at York University * I run workshops which emphasize reproducible research * My own research focuses on current practices for estimating and interpreting effect sizes, improving statistical pedagogy, and developing tools which instructors can implement within their own classrooms to help improve statistical literacy, thinking, and reasoning. * I need you help. Please *politely* interrupt me or message in chat/raise your hand if you want further clarification or have a question. * This workshop will be more beneficial to everyone by it being interactive. After all, you can just watch a video on YouTube or read one of the many excellent blog posts explaining the process --- # A little about you .center[ <div id="htmlwidget-58f51b977edcee1ed312" style="width:100%;height:500px;" class="leaflet html-widget"></div> <script type="application/json" data-for="htmlwidget-58f51b977edcee1ed312">{"x":{"options":{"crs":{"crsClass":"L.CRS.EPSG3857","code":null,"proj4def":null,"projectedBounds":null,"options":{}}},"calls":[{"method":"addTiles","args":["//{s}.tile.openstreetmap.org/{z}/{x}/{y}.png",null,null,{"minZoom":0,"maxZoom":18,"tileSize":256,"subdomains":"abc","errorTileUrl":"","tms":false,"noWrap":false,"zoomOffset":0,"zoomReverse":false,"opacity":1,"zIndex":1,"detectRetina":false,"attribution":"© <a href=\"http://openstreetmap.org\">OpenStreetMap<\/a> contributors, <a href=\"http://creativecommons.org/licenses/by-sa/2.0/\">CC-BY-SA<\/a>"}]},{"method":"addMarkers","args":[[43.7742901795184,43.7742901795184,42.452547977893,40.4239014553604,33.2141593233524,43.7742901795184,43.7742901795184,53.9460773079359,-22.7864576382996,21.4936314945287,40.1019686922611,40.0076549399465,51.5280386270635,38.9404642283407,35.9048687287113,42.9328521444856,40.0067873306839,35.1539780880117,40.0076549399465,38.4350919795447,41.8733529895323,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184,43.7742901795184],[-79.502018603698,-79.502018603698,-76.4811084051002,-86.9217739280261,-87.5385635591834,-79.502018603698,-79.502018603698,-1.05193273397094,-46.5136100497049,39.2392545332618,-88.2272473325232,-105.265802226979,-0.102473130349411,-92.3276409423479,-79.047192351666,-85.581676147791,-83.0306155343746,136.968079265508,-105.265802226979,-78.8696475135248,-87.6697309245307,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698,-79.502018603698],null,null,null,{"interactive":true,"draggable":false,"keyboard":true,"title":"","alt":"","zIndexOffset":0,"opacity":1,"riseOnHover":false,"riseOffset":250},null,null,null,null,null,{"interactive":false,"permanent":false,"direction":"auto","opacity":1,"offset":[0,0],"textsize":"10px","textOnly":false,"className":"","sticky":true},null]}],"limits":{"lat":[-22.7864576382996,53.9460773079359],"lng":[-105.265802226979,136.968079265508]}},"evals":[],"jsHooks":[]}</script> ] ??? I was told to fill out a prereg I heard it might be a good thing to do I have done one before, but I want to get better at them The workshop was free, so I figured "why not?" --- .pull-left[ # Center for Open Science (COS) .background-white.center[  ] * [https://www.cos.io/](https://www.cos.io/) ] .pull-right[ # Open Science Framework (OSF) .background-white.center[  ] * [https://osf.io/](https://osf.io/) * [Helpful guides](https://help.osf.io/hc/en-us/) ] ??? **Demonsrate COS link** --- # Open Researcher and Contributor ID (ORCID) .pull-left[ ### Some Benefits: * You will be distinguished from every other researcher, even researchers who share your same name, * Your research outputs and activities will be correctly attributed to you, * Your contributions and affiliations will be reliably and easily connected to you, * You will enjoy improved discoverability and recognition, * Your ORCID record is yours, for free, forever. ] .pull-right[  [https://orcid.org/](https://orcid.org/) ] ??? Some of the journals which I submit my manuscripts to requires an Orcid ID, I mainly use it to signin to OSF and link my projects. Another great benefit is that this ID stays tied to you, even if you change institutions. You might want to use a personal/stable email address in case you no longer have access to your institutional email. --- # What is preregistration? >"...you're simply specifying your research plan in advance of your study and submitting it to a registry." .right[ [www.cos.io/initiatives/prereg](https://www.cos.io/initiatives/prereg) ] -- >"Preregistration of an analysis plan is committing to analytic steps without advance knowledge of the research outcomes" .right[ [Nosek, Ebersole, DeHaven, & Mellor, 2015](https://www.pnas.org/content/115/11/2600.full/) ] -- There are some key things which should be present for any preregistration: * it should relate to a specific study, not a class of studies * ideally, the preregistration should be public and findable/accessible by anyone * authors should state data collection and analysis plans prior .right[ [Navarro, 2019](https://psyarxiv.com/wxn58/) ] ??? So before you sit down to start a preregistration, I encourage you to think carefully about what you aim to achieve via this process --- # What a preregistration is not? .center[ ## Preprint ] .pull-left[  ] .pull-right[ * A manuscript posted (usually after data collection) on a public repository which has not been peer-reviewed * [https://psyarxiv.com/](https://psyarxiv.com/) ] -- .center[ ## Registered Report ] .pull-left[  ] .pull-right[ * A preregistration is the starting block for a registered report. * [https://www.cos.io/initiatives/registered-reports](https::/https://www.cos.io/initiatives/registered-reports/) ] --- # Badges .pull-left[ .background-white[  ] 1. A public date-time stamped registration is in an institutional registration system (e.g., ClinicalTrials.gov, Open Science Framework, AEA Registry, EGAP). 2. Registration pre-dates the intervention. 3. Registered design corresponds directly to reported design. 4. Full disclosure of results in accordance with registered plan. ] -- .pull-right[ .background-white[  ] 3.Registered design and analysis plan corresponds directly to reported design and analysis. [Check out the badges](https://osf.io/tvyxz/wiki/1.%20View%20the%20Badges/) ] ??? A general warning, I know that no one hear is doing a prereg just for a badge, but there is concern that trying to achieve these badges will just become the new ritual which researchers "mindlessly" follow. However, I think that much good would come from the typical increase in transparency. --- # Transparency Openness Promotion (TOP) Guidelines >"The TOP Guidelines were created by journals, funders, and societies to align scientific ideals with practices." .right[ [https://www.cos.io/initiatives/top-guidelines](https://www.cos.io/initiatives/top-guidelines/) ] The Open Science Framework (OSF) maintains a [database](https://topfactor.org/) which ranks journal policies in terms of their compliance with these guidelines. This database can be used to see if your target journal requires preregistration ro any of the other eight modules contained in the TOP guidelines. ??? **Demonstrate the database link** --- # What preregistrations can do -- It depends on your purpose! -- A preregistration can ... * improve the transparency of your research -- * reduce researcher degrees of freedom -- * reduce file drawer effect * though publishing a preprint would be more effective -- * reduce publication bias -- * help reduce unintentional false positive inflation of results ([Forstmeier, Wagenmakers, & Parker, 2017](https://onlinelibrary.wiley.com/doi/full/10.1111/brv.12315/)) -- * be a tool to help clarify your thinking -- * delineate prediction from postdiction (i.e., exploratory versus confirmatory analyses) --- # What preregistrations can do > Mistaking postdiction as prediction underestimates the uncertainty of outcomes and can produce psychological overconfidence in the resulting findings. .right[ [Nosek, Ebersole, DeHaven, & Mellor, 2015](https://www.pnas.org/content/115/11/2600.full/) ] .pull-left[  ] ??? I like to think of it like a game of pool. What's more believable: a called shot who someone who just hits the balls really hard and one happens to go in? Photo by <a href="https://unsplash.com/@alxlion?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Alexandre Lion</a> on <a href="https://unsplash.com/s/photos/billiards?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a> --- # What preregistrations can **NOT** do for you? Preregistration cannot ... * improve your research simply through the act of preregistering it -- * increase the transparency of your research (however a "good" preregistration can) -- * prevent p-hacking and other questionable research practices (QRPs) --- # A preregistration plan is **NOT** a prison sentence .pull-left[  ] .pull-right[ * It is perfectly fine to deviate from your preregistration plan, so long as you clearly state where and WHY you deviated from the plan * The closer you stick to the plan, the more credibility your predictions will have [https://www.cos.io/blog/preregistration-plan-not-prison](https://www.cos.io/blog/preregistration-plan-not-prison) ] ??? This is an opinion I have encountered both online and in person. The crux is that by having to preregister everything leaves little room for the creative process of making inquiries about nature. You can still do those things. A prereg plan merely delineates one from the other. I think the claim that it limits scientific curiosity is more of a statement about a researcher than it is of this tool. Photo by <a href="https://unsplash.com/@redaquamedia?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Denny Müller</a> on <a href="https://unsplash.com/s/photos/prison?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a> --- # What **should** be included in a preregistration? This is a tricky question with no clear answer. As a methodologist, the answer is: -- It depends! -- * On the type of research you are conducting (confirmatory or exploratory) * On the level of transparency you are striving for * the time you have to front-load on your project -- Whatever you choose to include should be in service of your reason for completing a preregistration plan. --- # What **should** be included in a preregistration? If the goal is to limit researcher degrees of freedom, then care must be taken at each stage: * planning * running * analyzing * reporting -- I suggest you check out this article by [Wicherts, Veldkamp, Augusteijn, Bakker, van Aert, van Assen; 2016](https://www.frontiersin.org/articles/10.3389/fpsyg.2016.01832/full) which lists researcher degrees of freedom across the many stages in their [Table 1.](https://www.frontiersin.org/files/Articles/222767/fpsyg-07-01832-HTML/image_m/fpsyg-07-01832-t001.jpg/) -- You can also follow this guide which has a checklist for what to include [Download Guide to Pre-Analysis Plans from OSF.io](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjfvfiEm83wAhVGZM0KHe-qCIIQFjAAegQIAhAD&url=https%3A%2F%2Fosf.io%2Fsh4v8%2Fdownload&usg=AOvVaw255Pud_0CqwZzgIhl4KFmf) --- # What **should** be included in a preregistration? An often overlooked part of the preregistration process is how to avoid "Questionable Measurement Practices" (QMRs). [Flake and Fried (2020)](https://journals.sagepub.com/doi/full/10.1177/2515245920952393/) listed a series of 6 questions which were developed to promote transparent reporting of measurement practices. 1. What is your construct? 2. Why and how did you select your measure? 3. What measure dd you use to operationalize the construct? 4. How did you quantify your measure? 5. Did you modify the scale? And if so, how and why? 6. Did you create the measure on the fly? --- # Let's try to complete a Preregistration Notes: There is no single way to do this. I'm just showing my preferred method. I will be using my ORCID to access OSF and create a repository. --- # Create A Preregistration 1. Log into [https://www.osf.io/](https://www.osf.io/) using my ORCID 2. From the OSF dashboard, click "Create new project" button * Enter Title, Storage Location, and a description 2. Within the new project page, select the "Registrations" tab from the top navbar, then press the "New registration button" * select the template most appropriate for your project (you can preview them at [https://osf.io/zab38/wiki/home/](https://osf.io/zab38/wiki/home/)) * I will select the standard, comprehensive, and general purpose form * Metadata * Study Information * Design Plan * Sampling Plan * Variables * Analysis Plan * Other [https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration](https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration/) --- # Tips It can be helpful to visualize parts of your design or analysis using a flowchart, [http://ecomodeler.org/](http://ecomodeler.org/) is a great/free tool for generating decision trees and flowcharts. * You can export the image and the model file to upload as part of your OSF project or a preregistration plan -- Sometimes you might want to include a decision tree which outlines how you might proceed WHEN particular analytics conditions arise (e.g., violated model assumptions) -- If you are familiar with the variables and analysis plan, you can upload a script or syntax file which you will use to conduct your analysis. * Here is an example I used for a data analysis challenge with a few colleagues [https://osf.io/qc9e6/](https://osf.io/qc9e6/) ??? **Demonstrate ecomodeler.org** **Show my prereg_script.R** --- # Reporting Preregistered Research * Include a link to the original preregistration plan * Report the results of all prespecified tests (e.g., tests) * Clearly label any unplanned analyses * Include a transparent changes document [https://help.osf.io/hc/en-us/articles/360021390833-Preregistration](https://help.osf.io/hc/en-us/articles/360021390833-Preregistration) --- # Closing Remarks >"Every research practice comes with costs and benefits, and it is useful to evaluate whether and when it is worth preregistering your study" >"If the use of a tool is detached from a philosophy of science it risks becoming a heuristic. Researchers should not choose to preregister because it has become a new norm, but they should preregister because they can justify based on their philosophy of science how preregistration supports their goals." .right[ [Lakens, 2019](https://psyarxiv.com/jbh4w/) ] --- # Questions --- # Advanced Tips and Tricks * Check out the R package {DeclareDesign} at [https://declaredesign.org/](https://declaredesign.org/) * Check out [Improving Transparency, Falsifiability, and Rigor by Making Hypothesis Tests Machine-Readable](https://journals.sagepub.com/doi/full/10.1177/2515245920970949/) by Daniël Lakens and Lisa Debruine