Bibliography – Reproducibility in Data Science and Digital Humanities

Selected DH research and resources bearing on, or utilized by, the WE1S project.
(all) Distant Reading | Cultural Analytics | | Sociocultural Approaches | Topic Modeling in DH | Non-consumptive Use


2133649 Reproducibility chicago-fullnote-bibliography 50 date desc year 1 1 1 5445 https://we1s.ucsb.edu/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%224IVIKSUK%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Heaven%22%2C%22parsedDate%22%3A%222020%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHeaven%2C%20Will%20Douglass.%20%26%23x201C%3BAI%20Is%20Wrestling%20with%20a%20Replication%20Crisis.%26%23x201D%3B%20%26lt%3Bi%26gt%3BMIT%20Technology%20Review%26lt%3B%5C%2Fi%26gt%3B%2C%202020.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fwww.technologyreview.com%5C%2F2020%5C%2F11%5C%2F12%5C%2F1011944%5C%2Fartificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai%5C%2F%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fwww.technologyreview.com%5C%2F2020%5C%2F11%5C%2F12%5C%2F1011944%5C%2Fartificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai%5C%2F%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3D4IVIKSUK%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22AI%20is%20wrestling%20with%20a%20replication%20crisis%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Will%20Douglass%22%2C%22lastName%22%3A%22Heaven%22%7D%5D%2C%22abstractNote%22%3A%22%5BBeginning%20of%20article%3A%5D%20Last%20month%20Nature%20published%20a%20damning%20response%20written%20by%2031%20scientists%20to%20a%20study%20from%20Google%20Health%20that%20had%20appeared%20in%20the%20journal%20earlier%20this%20year.%20Google%20was%20describing%20successful%20trials%20of%20an%20AI%20that%20looked%20for%20signs%20of%20breast%20cancer%20in%20medical%20images.%20But%20according%20to%20its%20critics%2C%20the%20Google%20team%20provided%20so%20little%20information%20about%20its%20code%20and%20how%20it%20was%20tested%20that%20the%20study%20amounted%20to%20nothing%20more%20than%20a%20promotion%20of%20proprietary%20tech.%5Cn%5Cn%5Cu201cWe%20couldn%5Cu2019t%20take%20it%20anymore%2C%5Cu201d%20says%20Benjamin%20Haibe-Kains%2C%20the%20lead%20author%20of%20the%20response%2C%20who%20studies%20computational%20genomics%20at%20the%20University%20of%20Toronto.%20%5Cu201cIt%5Cu2019s%20not%20about%20this%20study%20in%20particular%5Cu2014it%5Cu2019s%20a%20trend%20we%5Cu2019ve%20been%20witnessing%20for%20multiple%20years%20now%20that%20has%20started%20to%20really%20bother%20us.%5Cu201d%22%2C%22date%22%3A%222020%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.technologyreview.com%5C%2F2020%5C%2F11%5C%2F12%5C%2F1011944%5C%2Fartificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai%5C%2F%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-11-14T06%3A21%3A30Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Artificial%20intelligence%22%7D%2C%7B%22tag%22%3A%22Machine%20learning%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22Y7Q3CUCI%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Koenzen%20et%20al.%22%2C%22parsedDate%22%3A%222020%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BKoenzen%2C%20Andreas%2C%20Neil%20Ernst%2C%20and%20Margaret-Anne%20Storey.%20%26%23x201C%3BCode%20Duplication%20and%20Reuse%20in%20Jupyter%20Notebooks.%26%23x201D%3B%20%26lt%3Bi%26gt%3BArXiv%3A2005.13709%20%5BCs%5D%26lt%3B%5C%2Fi%26gt%3B%2C%202020.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.13709%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.13709%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DY7Q3CUCI%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Code%20Duplication%20and%20Reuse%20in%20Jupyter%20Notebooks%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andreas%22%2C%22lastName%22%3A%22Koenzen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Neil%22%2C%22lastName%22%3A%22Ernst%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Margaret-Anne%22%2C%22lastName%22%3A%22Storey%22%7D%5D%2C%22abstractNote%22%3A%22Duplicating%20one%26%23039%3Bs%20own%20code%20makes%20it%20faster%20to%20write%20software.%20This%20expediency%20is%20particularly%20valuable%20for%20users%20of%20computational%20notebooks.%20Duplication%20allows%20notebook%20users%20to%20quickly%20test%20hypotheses%20and%20iterate%20over%20data.%20In%20this%20paper%2C%20we%20explore%20how%20much%2C%20how%20and%20from%20where%20code%20duplication%20occurs%20in%20computational%20notebooks%2C%20and%20identify%20potential%20barriers%20to%20code%20reuse.%20Previous%20work%20in%20the%20area%20of%20computational%20notebooks%20describes%20developers%26%23039%3B%20motivations%20for%20reuse%20and%20duplication%20but%20does%20not%20show%20how%20much%20reuse%20occurs%20or%20which%20barriers%20they%20face%20when%20reusing%20code.%20To%20address%20this%20gap%2C%20we%20first%20analyzed%20GitHub%20repositories%20for%20code%20duplicates%20contained%20in%20a%20repository%26%23039%3Bs%20Jupyter%20notebooks%2C%20and%20then%20conducted%20an%20observational%20user%20study%20of%20code%20reuse%2C%20where%20participants%20solved%20specific%20tasks%20using%20notebooks.%20Our%20findings%20reveal%20that%20repositories%20in%20our%20sample%20have%20a%20mean%20self-duplication%20rate%20of%207.6%25.%20However%2C%20in%20our%20user%20study%2C%20few%20participants%20duplicated%20their%20own%20code%2C%20preferring%20to%20reuse%20code%20from%20online%20sources.%22%2C%22date%22%3A%222020%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F2005.13709%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T06%3A06%3A35Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Data%20notebooks%22%7D%2C%7B%22tag%22%3A%22Data%20science%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22XY9MX23K%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Colavizza%22%2C%22parsedDate%22%3A%222020%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BColavizza%2C%20Giovanni.%20%26%23x201C%3BAre%20We%20Breaking%20the%20Social%20Contract%3F%26%23x201D%3B%20%26lt%3Bi%26gt%3BJournal%20of%20Cultural%20Analytics%26lt%3B%5C%2Fi%26gt%3B%2C%202020.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.22148%5C%2F001c.11828%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.22148%5C%2F001c.11828%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DXY9MX23K%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Are%20We%20Breaking%20the%20Social%20Contract%3F%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Giovanni%22%2C%22lastName%22%3A%22Colavizza%22%7D%5D%2C%22abstractNote%22%3A%22The%20ambition%20of%20scholarship%20in%20the%20humanities%20is%20to%20systematically%20understand%20the%20human%20condition%20in%20all%20its%20aspects%20and%20times.%20To%20this%20end%2C%20humanists%20are%20more%20apt%20to%20interpret%20specific%20phenomena%20than%20generalize%20to%20previously%20unseen%20observations.%20When%20we%20consider%20scholarship%20as%20a%20collective%20effort%2C%20this%20has%20consequences.%20I%20argue%20that%20most%20of%20the%20humanities%20rely%20on%20a%20distinct%20social%20contract.%20This%20contract%20states%20that%20interpretive%20arguments%20are%20expected%20to%20be%20plausible%20and%20the%20grounds%20on%20which%20they%20are%20made%2C%20verifiable.%20This%20is%20the%20scholarly%20purpose%20%28albeit%20not%20the%20rhetorical%20one%29%20of%20most%20of%20what%20goes%20in%20our%20footnotes%2C%20especially%20references.%20Reference%20verification%20is%20mostly%20a%20virtual%20act%2C%20i.e.%2C%20it%20all%20too%20rarely%20happens%20in%20practice%2C%20yet%20it%20is%20in%20principle%20always%20possible.%20Any%20individual%20scholar%20in%20any%20domain%20in%20the%20humanities%20can%2C%20by%20virtue%20of%20this%20contract%2C%20verify%20the%20evidence%20supporting%20any%20argument%20in%20a%20non-mediated%20way.%20This%20is%20essential%20to%2C%20at%20the%20very%20least%2C%20distinguish%20between%20solid%20and%20haphazard%20arguments.%22%2C%22date%22%3A%222020%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.22148%5C%2F001c.11828%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fculturalanalytics.org%5C%2Farticle%5C%2F11828-are-we-breaking-the-social-contract%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-04-09T06%3A50%3A40Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22DH%20Digital%20humanities%22%7D%2C%7B%22tag%22%3A%22Humanities%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22EQD7ZF2Z%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22National%20Academies%20of%20Sciences%2C%20Engineering%2C%20and%20Medicine%22%2C%22parsedDate%22%3A%222019%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BNational%20Academies%20of%20Sciences%2C%20Engineering%2C%20and%20Medicine.%20%26lt%3Bi%26gt%3BReproducibility%20and%20Replicability%20in%20Science%26lt%3B%5C%2Fi%26gt%3B.%20One%20of%20the%20Pathways%20by%20Which%20the%20Scientific%20Community%20Confirms%20the%20Validity%20of%20a%20New%20Scientific%20Discovery%20Is%20by%20Repeating%20the%20Research%20That%20Produced%20It.%20When%20a%20Scientific%20Effort%20Fails%20to%20Independently%20Confirm%20the%20Computations%20or%20Results%20of%20a%20Previous%20Study%2C%20Some%20Fear%20That%20It%20May%20Be%20a%20Symptom%20of%20a%20Lack%20of%20Rigor%20in%20Science%2C%20While%20Others%20Argue%20That%20Such%20an%20Observed%20Inconsistency%20Can%20Be%20an%20Important%20Precursor%20to%20New%20Discovery.%26%23xA0%3B%20Concerns%20about%20Reproducibility%20and%20Replicability%20Have%20Been%20Expressed%20in%20Both%20Scientific%20and%20Popular%20Media.%20As%20These%20Concerns%20Came%20to%20Light%2C%20Congress%20Requested%20That%20the%20National%20Academies%20of%20Sciences%2C%20Engineering%2C%20and%20Medicine%20Conduct%20a%20Study%20to%20Assess%20the%20Extent%20of%20Issues%20Related%20to%20Reproducibility%20and%20Replicability%20and%20to%20Offer%20Recommendations%20for%20Improving%20Rigor%20and%20Transparency%20in%20Scientific%20Research.%26%23xA0%3B%20Reproducibility%20and%20Replicability%20in%20Science%20Defines%20Reproducibility%20and%20Replicability%20and%20Examines%20the%20Factors%20That%20May%20Lead%20to%20Non-Reproducibility%20and%20Non-Replicability%20in%20Research.%20Unlike%20the%20Typical%20Expectation%20of%20Reproducibility%20between%20Two%20Computations%2C%20Expectations%20about%20Replicability%20Are%20More%20Nuanced%2C%20and%20in%20Some%20Cases%20a%20Lack%20of%20Replicability%20Can%20Aid%20the%20Process%20of%20Scientific%20Discovery.%20This%20Report%20Provides%20Recommendations%20to%20Researchers%2C%20Academic%20Institutions%2C%20Journals%2C%20and%20Funders%20on%20Steps%20They%20Can%20Take%20to%20Improve%20Reproducibility%20and%20Replicability%20in%20Science.%20Washington%2C%20DC%3A%20National%20Academies%20Press%2C%202019.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fnap.nationalacademies.org%5C%2Fcatalog%5C%2F25303%5C%2Freproducibility-and-replicability-in-science%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fnap.nationalacademies.org%5C%2Fcatalog%5C%2F25303%5C%2Freproducibility-and-replicability-in-science%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DEQD7ZF2Z%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22book%22%2C%22title%22%3A%22Reproducibility%20and%20replicability%20in%20science%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22name%22%3A%22National%20Academies%20of%20Sciences%2C%20Engineering%2C%20and%20Medicine%22%7D%5D%2C%22abstractNote%22%3A%22%26quot%3BOne%20of%20the%20pathways%20by%20which%20the%20scientific%20community%20confirms%20the%20validity%20of%20a%20new%20scientific%20discovery%20is%20by%20repeating%20the%20research%20that%20produced%20it.%20When%20a%20scientific%20effort%20fails%20to%20independently%20confirm%20the%20computations%20or%20results%20of%20a%20previous%20study%2C%20some%20fear%20that%20it%20may%20be%20a%20symptom%20of%20a%20lack%20of%20rigor%20in%20science%2C%20while%20others%20argue%20that%20such%20an%20observed%20inconsistency%20can%20be%20an%20important%20precursor%20to%20new%20discovery.%20Concerns%20about%20reproducibility%20and%20replicability%20have%20been%20expressed%20in%20both%20scientific%20and%20popular%20media.%20As%20these%20concerns%20came%20to%20light%2C%20Congress%20requested%20that%20the%20National%20Academies%20of%20Sciences%2C%20Engineering%2C%20and%20Medicine%20conduct%20a%20study%20to%20assess%20the%20extent%20of%20issues%20related%20to%20reproducibility%20and%20replicability%20and%20to%20offer%20recommendations%20for%20improving%20rigor%20and%20transparency%20in%20scientific%20research.%20Reproducibility%20and%20Replicability%20in%20Science%20defines%20reproducibility%20and%20replicability%20and%20examines%20the%20factors%20that%20may%20lead%20to%20non-reproducibility%20and%20non-replicability%20in%20research.%20Unlike%20the%20typical%20expectation%20of%20reproducibility%20between%20two%20computations%2C%20expectations%20about%20replicability%20are%20more%20nuanced%2C%20and%20in%20some%20cases%20a%20lack%20of%20replicability%20can%20aid%20the%20process%20of%20scientific%20discovery.%20This%20report%20provides%20recommendations%20to%20researchers%2C%20academic%20institutions%2C%20journals%2C%20and%20funders%20on%20steps%20they%20can%20take%20to%20improve%20reproducibility%20and%20replicability%20in%20science%26quot%3B--%22%2C%22date%22%3A%222019%22%2C%22language%22%3A%22en%22%2C%22ISBN%22%3A%22978-0-309-48616-3%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fnap.nationalacademies.org%5C%2Fcatalog%5C%2F25303%5C%2Freproducibility-and-replicability-in-science%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222022-04-01T21%3A32%3A01Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22EA6Y7Z2Z%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Pimentel%20et%20al.%22%2C%22parsedDate%22%3A%222019%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BPimentel%2C%20Jo%26%23xE3%3Bo%20Felipe%2C%20Leonardo%20Murta%2C%20Vanessa%20Braganholo%2C%20and%20Juliana%20Freire.%20%26%23x201C%3BA%20Large-Scale%20Study%20about%20Quality%20and%20Reproducibility%20of%20Jupyter%20Notebooks.%26%23x201D%3B%20In%20%26lt%3Bi%26gt%3BProceedings%20of%20the%2016th%20International%20Conference%20on%20Mining%20Software%20Repositories%26lt%3B%5C%2Fi%26gt%3B%2C%20507%26%23x2013%3B17.%20MSR%20%26%23x2019%3B19.%20Montreal%2C%20Quebec%2C%20Canada%3A%20IEEE%20Press%2C%202019.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FMSR.2019.00077%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FMSR.2019.00077%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DEA6Y7Z2Z%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22A%20large-scale%20study%20about%20quality%20and%20reproducibility%20of%20jupyter%20notebooks%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jo%5Cu00e3o%20Felipe%22%2C%22lastName%22%3A%22Pimentel%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Leonardo%22%2C%22lastName%22%3A%22Murta%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Vanessa%22%2C%22lastName%22%3A%22Braganholo%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Juliana%22%2C%22lastName%22%3A%22Freire%22%7D%5D%2C%22abstractNote%22%3A%22Jupyter%20Notebooks%20have%20been%20widely%20adopted%20by%20many%20different%20communities%2C%20both%20in%20science%20and%20industry.%20They%20support%20the%20creation%20of%20literate%20programming%20documents%20that%20combine%20code%2C%20text%2C%20and%20execution%20results%20with%20visualizations%20and%20all%20sorts%20of%20rich%20media.%20The%20self-documenting%20aspects%20and%20the%20ability%20to%20reproduce%20results%20have%20been%20touted%20as%20significant%20benefits%20of%20notebooks.%20At%20the%20same%20time%2C%20there%20has%20been%20growing%20criticism%20that%20the%20way%20notebooks%20are%20being%20used%20leads%20to%20unexpected%20behavior%2C%20encourage%20poor%20coding%20practices%2C%20and%20that%20their%20results%20can%20be%20hard%20to%20reproduce.%20To%20understand%20good%20and%20bad%20practices%20used%20in%20the%20development%20of%20real%20notebooks%2C%20we%20studied%201.4%20million%20notebooks%20from%20GitHub.%20We%20present%20a%20detailed%20analysis%20of%20their%20characteristics%20that%20impact%20reproducibility.%20We%20also%20propose%20a%20set%20of%20best%20practices%20that%20can%20improve%20the%20rate%20of%20reproducibility%20and%20discuss%20open%20challenges%20that%20require%20further%20research%20and%20development.%22%2C%22date%22%3A%222019%22%2C%22proceedingsTitle%22%3A%22Proceedings%20of%20the%2016th%20International%20Conference%20on%20Mining%20Software%20Repositories%22%2C%22conferenceName%22%3A%22%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1109%5C%2FMSR.2019.00077%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1109%5C%2FMSR.2019.00077%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T05%3A38%3A24Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Data%20notebooks%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22DIYQUIAZ%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Rule%20et%20al.%22%2C%22parsedDate%22%3A%222019%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BRule%2C%20Adam%2C%20Amanda%20Birmingham%2C%20Cristal%20Zuniga%2C%20Ilkay%20Altintas%2C%20Shih-Cheng%20Huang%2C%20Rob%20Knight%2C%20Niema%20Moshiri%2C%20et%20al.%20%26%23x201C%3BTen%20Simple%20Rules%20for%20Writing%20and%20Sharing%20Computational%20Analyses%20in%20Jupyter%20Notebooks.%26%23x201D%3B%20%26lt%3Bi%26gt%3BPLOS%20Computational%20Biology%26lt%3B%5C%2Fi%26gt%3B%2015%2C%20no.%207%20%282019%29%3A%20e1007007.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1371%5C%2Fjournal.pcbi.1007007%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1371%5C%2Fjournal.pcbi.1007007%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DDIYQUIAZ%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Ten%20simple%20rules%20for%20writing%20and%20sharing%20computational%20analyses%20in%20Jupyter%20Notebooks%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Adam%22%2C%22lastName%22%3A%22Rule%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Amanda%22%2C%22lastName%22%3A%22Birmingham%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Cristal%22%2C%22lastName%22%3A%22Zuniga%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ilkay%22%2C%22lastName%22%3A%22Altintas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Shih-Cheng%22%2C%22lastName%22%3A%22Huang%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Rob%22%2C%22lastName%22%3A%22Knight%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Niema%22%2C%22lastName%22%3A%22Moshiri%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Mai%20H.%22%2C%22lastName%22%3A%22Nguyen%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sara%20Brin%22%2C%22lastName%22%3A%22Rosenthal%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fernando%22%2C%22lastName%22%3A%22P%5Cu00e9rez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Peter%20W.%22%2C%22lastName%22%3A%22Rose%22%7D%5D%2C%22abstractNote%22%3A%22As%20studies%20grow%20in%20scale%20and%20complexity%2C%20it%20has%20become%20increasingly%20difficult%20to%20provide%20clear%20descriptions%20and%20open%20access%20to%20the%20methods%20and%20data%20needed%20to%20understand%20and%20reproduce%20computational%20research.%20Numerous%20papers%2C%20including%20several%20in%20the%20Ten%20Simple%20Rules%20collection%2C%20have%20highlighted%20the%20need%20for%20robust%20and%20reproducible%20analyses%20in%20computational%20research%2C%20described%20the%20difficulty%20of%20achieving%20these%20standards%2C%20and%20enumerated%20best%20practices.%20We%20aim%20to%20augment%20this%20existing%20wellspring%20of%20advice%20by%20addressing%20the%20unique%20challenges%20and%20opportunities%20that%20arise%20when%20using%20computational%20notebooks%2C%20especially%20Jupyter%20Notebooks%2C%20for%20research.%22%2C%22date%22%3A%222019%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1371%5C%2Fjournal.pcbi.1007007%22%2C%22ISSN%22%3A%221553-7358%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fjournals.plos.org%5C%2Fploscompbiol%5C%2Farticle%3Fid%3D10.1371%5C%2Fjournal.pcbi.1007007%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T05%3A31%3A36Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Data%20notebooks%22%7D%2C%7B%22tag%22%3A%22Data%20science%22%7D%2C%7B%22tag%22%3A%22Open%20science%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22QXN4AUVS%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Mendez%20et%20al.%22%2C%22parsedDate%22%3A%222019%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMendez%2C%20Kevin%20M.%2C%20Leighton%20Pritchard%2C%20Stacey%20N.%20Reinke%2C%20and%20David%20I.%20Broadhurst.%20%26%23x201C%3BToward%20Collaborative%20Open%20Data%20Science%20in%20Metabolomics%20Using%20Jupyter%20Notebooks%20and%20Cloud%20Computing.%26%23x201D%3B%20%26lt%3Bi%26gt%3BMetabolomics%26lt%3B%5C%2Fi%26gt%3B%2015%2C%20no.%2010%20%282019%29%3A%20125.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs11306-019-1588-0%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs11306-019-1588-0%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DQXN4AUVS%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Toward%20collaborative%20open%20data%20science%20in%20metabolomics%20using%20Jupyter%20Notebooks%20and%20cloud%20computing%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kevin%20M.%22%2C%22lastName%22%3A%22Mendez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Leighton%22%2C%22lastName%22%3A%22Pritchard%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Stacey%20N.%22%2C%22lastName%22%3A%22Reinke%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22David%20I.%22%2C%22lastName%22%3A%22Broadhurst%22%7D%5D%2C%22abstractNote%22%3A%22A%20lack%20of%20transparency%20and%20reporting%20standards%20in%20the%20scientific%20community%20has%20led%20to%20increasing%20and%20widespread%20concerns%20relating%20to%20reproduction%20and%20integrity%20of%20results.%20As%20an%20omics%20science%2C%20which%20generates%20vast%20amounts%20of%20data%20and%20relies%20heavily%20on%20data%20science%20for%20deriving%20biological%20meaning%2C%20metabolomics%20is%20highly%20vulnerable%20to%20irreproducibility.%20The%20metabolomics%20community%20has%20made%20substantial%20efforts%20to%20align%20with%20FAIR%20data%20standards%20by%20promoting%20open%20data%20formats%2C%20data%20repositories%2C%20online%20spectral%20libraries%2C%20and%20metabolite%20databases.%20Open%20data%20analysis%20platforms%20also%20exist%3B%20however%2C%20they%20tend%20to%20be%20inflexible%20and%20rely%20on%20the%20user%20to%20adequately%20report%20their%20methods%20and%20results.%20To%20enable%20FAIR%20data%20science%20in%20metabolomics%2C%20methods%20and%20results%20need%20to%20be%20transparently%20disseminated%20in%20a%20manner%20that%20is%20rapid%2C%20reusable%2C%20and%20fully%20integrated%20with%20the%20published%20work.%20To%20ensure%20broad%20use%20within%20the%20community%20such%20a%20framework%20also%20needs%20to%20be%20inclusive%20and%20intuitive%20for%20both%20computational%20novices%20and%20experts%20alike.%22%2C%22date%22%3A%222019%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1007%5C%2Fs11306-019-1588-0%22%2C%22ISSN%22%3A%221573-3890%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1007%5C%2Fs11306-019-1588-0%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T05%3A02%3A08Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Data%20notebooks%22%7D%2C%7B%22tag%22%3A%22Open%20science%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22349TC9XC%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Liu%20et%20al.%22%2C%22parsedDate%22%3A%222017%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BLiu%2C%20Alan%2C%20Scott%20Kleinman%2C%20Jeremy%20Douglass%2C%20Lindsay%20Thomas%2C%20Ashley%20Champagne%2C%20and%20Jamal%20Russell.%20%26%23x201C%3BOpen%2C%20Shareable%2C%20Reproducible%20Workflows%20for%20the%20Digital%20Humanities%3A%20The%20Case%20of%20the%204Humanities.Org%20%26%23x2018%3BWhatEvery1Says%26%23x2019%3B%20Project.%26%23x201D%3B%20In%20%26lt%3Bi%26gt%3BDigital%20Humanities%202017%20Conference%20Abstracts%26lt%3B%5C%2Fi%26gt%3B.%20Montreal%3A%20Alliance%20of%20Digital%20Humanities%20Organizations%20%28ADHO%29%2C%202017.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3D349TC9XC%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22conferencePaper%22%2C%22title%22%3A%22Open%2C%20Shareable%2C%20Reproducible%20Workflows%20for%20the%20Digital%20Humanities%3A%20The%20Case%20of%20the%204Humanities.org%20%27WhatEvery1Says%27%20Project%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Alan%22%2C%22lastName%22%3A%22Liu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Scott%22%2C%22lastName%22%3A%22Kleinman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jeremy%22%2C%22lastName%22%3A%22Douglass%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lindsay%22%2C%22lastName%22%3A%22Thomas%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ashley%22%2C%22lastName%22%3A%22Champagne%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jamal%22%2C%22lastName%22%3A%22Russell%22%7D%5D%2C%22abstractNote%22%3A%22%5BFirst%20paragraph%3A%5D%20This%20panel%20reports%20on%20the%20open%2C%20shareable%2C%20and%20reproducible%20workflow%20methodology%20for%20digital%20humanities%20research%20developed%20by%20the%204Humanities.org%20%26quot%3BWhatEvery1Says%26quot%3B%20%28WE1S%29%20project.%20WE1S%20is%20topic%20modeling%20a%20large%20corpus%20of%20articles%20related%20to%20the%20humanities%20in%20newspapers%2C%20magazines%2C%20and%20other%20media%20sources%20in%20the%20U.S.%2C%20U.K.%2C%20and%20Canada%20from%201981%20on.%20While%20the%20panel%20presents%20WE1S%26%23039%3Bs%20conceptual%20goals%20and%20prototype%20experiments%20in%20using%20outcomes%20in%20humanities%20advocacy%2C%20its%20focus%20is%20on%20the%20technical%20and%20interpretive%20workflow%20developed%20by%20the%20project%20for%20humanities-%20oriented%20data%20work.%20WE1S%26%23039%3Bs%20manifest%20system%20for%20data%20provenance%20and%20workflow%20management%2C%20its%20virtual%20workspace%20manager%20for%20integrated%2C%20containerized%20data%20manipulation%20and%20processing%2C%20and%20its%20interpretation%20protocol%20for%20how%20humans%20read%20topic%20models%20suggest%20a%20generalizable%20open%20approach%20based%20not%20on%20particular%20technologies%20and%20methods%20but%20on%20annotated%20methods.%20Moreover%2C%20there%20is%20a%20philosophical%20fit%20between%20such%20an%20approach%20and%20the%20public-facing%20goals%20of%20the%20WE1S%20project.%20WE1S%20is%20about%20opening%20public%20culture%20to%20view%20through%20analytics%2C%20while%20its%20DH%20methodology%20is%20about%20opening%20up%20scholarly%20expertise%20itself%20through%20shareable%2C%20transparent%20processes%20not%20locked%20into%20technically%20complex%2C%20pre-established%2C%20or%20largescale%20research%20frameworks.%22%2C%22date%22%3A%222017%22%2C%22proceedingsTitle%22%3A%22Digital%20Humanities%202017%20Conference%20Abstracts%22%2C%22conferenceName%22%3A%22Digital%20Humanities%202017%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T06%3A28%3A33Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22DH%20Digital%20humanities%22%7D%2C%7B%22tag%22%3A%22DH%20Distant%20reading%22%7D%2C%7B%22tag%22%3A%22Data%20notebooks%22%7D%2C%7B%22tag%22%3A%22Humanities%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%2C%7B%22tag%22%3A%22Topic%20modeling%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22B9J2Z543%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Kluyver%20et%20al.%22%2C%22parsedDate%22%3A%222016%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BKluyver%2C%20Thomas%2C%20Benjamin%20Ragan-Kelley%2C%20Fernando%20P%26%23xE9%3Brez%2C%20Brian%20Granger%2C%20Matthias%20Bussonnier%2C%20Jonathan%20Frederic%2C%20Kyle%20Kelley%2C%20et%20al.%20%26%23x201C%3BJupyter%20Notebooks%20%26%23x2013%3B%20a%20Publishing%20Format%20for%20Reproducible%20Computational%20Workflows.%26%23x201D%3B%20%26lt%3Bi%26gt%3BPositioning%20and%20Power%20in%20Academic%20Publishing%3A%20Players%2C%20Agents%20and%20Agendas%26lt%3B%5C%2Fi%26gt%3B%2C%202016%2C%2087%26%23x2013%3B90.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.3233%5C%2F978-1-61499-649-1-87%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.3233%5C%2F978-1-61499-649-1-87%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DB9J2Z543%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Jupyter%20Notebooks%20%5Cu2013%20a%20publishing%20format%20for%20reproducible%20computational%20workflows%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Thomas%22%2C%22lastName%22%3A%22Kluyver%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Benjamin%22%2C%22lastName%22%3A%22Ragan-Kelley%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Fernando%22%2C%22lastName%22%3A%22P%5Cu00e9rez%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Brian%22%2C%22lastName%22%3A%22Granger%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Matthias%22%2C%22lastName%22%3A%22Bussonnier%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Frederic%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kyle%22%2C%22lastName%22%3A%22Kelley%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jessica%22%2C%22lastName%22%3A%22Hanrick%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jason%22%2C%22lastName%22%3A%22Grout%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sylvain%22%2C%22lastName%22%3A%22Corlay%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Paul%22%2C%22lastName%22%3A%22Ivanov%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Dami%5Cu00e1n%22%2C%22lastName%22%3A%22Avila%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Safia%22%2C%22lastName%22%3A%22Abdalla%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Carol%22%2C%22lastName%22%3A%22Willing%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22%22%2C%22lastName%22%3A%22Jupyter%20Development%20Team%22%7D%5D%2C%22abstractNote%22%3A%22It%20is%20increasingly%20necessary%20for%20researchers%20in%20all%20fields%20to%20write%20computer%20code%2C%20and%20in%20order%20to%20reproduce%20research%20results%2C%20it%20is%20important%20that%20this%20code%20is%20published.%20We%20present%20Jupyter%20notebooks%2C%20a%20document%20format%20for%20publishing%20code%2C%20results%20and%20explanations%20in%20a%20form%20that%20is%20both%20readable%20and%20executable.%20We%20discuss%20various%20tools%20and%20use%20cases%20for%20notebook%20documents.%22%2C%22date%22%3A%222016%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.3233%5C%2F978-1-61499-649-1-87%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T05%3A22%3A26Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Data%20notebooks%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22J8CFVKRP%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Peng%22%2C%22parsedDate%22%3A%222011%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BPeng%2C%20Roger%20D.%20%26%23x201C%3BReproducible%20Research%20in%20Computational%20Science.%26%23x201D%3B%20%26lt%3Bi%26gt%3BScience%26lt%3B%5C%2Fi%26gt%3B%20334%2C%20no.%206060%20%282011%29%3A%201226%26%23x2013%3B27.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1126%5C%2Fscience.1213847%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1126%5C%2Fscience.1213847%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DJ8CFVKRP%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Reproducible%20Research%20in%20Computational%20Science%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Roger%20D.%22%2C%22lastName%22%3A%22Peng%22%7D%5D%2C%22abstractNote%22%3A%22Computational%20science%20has%20led%20to%20exciting%20new%20developments%2C%20but%20the%20nature%20of%20the%20work%20has%20exposed%20limitations%20in%20our%20ability%20to%20evaluate%20published%20findings.%20Reproducibility%20has%20the%20potential%20to%20serve%20as%20a%20minimum%20standard%20for%20judging%20scientific%20claims%20when%20full%20independent%20replication%20of%20a%20study%20is%20not%20possible.%22%2C%22date%22%3A%222011%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1126%5C%2Fscience.1213847%22%2C%22ISSN%22%3A%220036-8075%2C%201095-9203%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fscience.sciencemag.org%5C%2Fcontent%5C%2F334%5C%2F6060%5C%2F1226%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-09-03T05%3A26%3A30Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Data%20science%22%7D%2C%7B%22tag%22%3A%22Reproducibility%22%7D%5D%7D%7D%5D%7D
Heaven, Will Douglass. “AI Is Wrestling with a Replication Crisis.” MIT Technology Review, 2020. https://www.technologyreview.com/2020/11/12/1011944/artificial-intelligence-replication-crisis-science-big-tech-google-deepmind-facebook-openai/. Cite
Koenzen, Andreas, Neil Ernst, and Margaret-Anne Storey. “Code Duplication and Reuse in Jupyter Notebooks.” ArXiv:2005.13709 [Cs], 2020. http://arxiv.org/abs/2005.13709. Cite
Colavizza, Giovanni. “Are We Breaking the Social Contract?” Journal of Cultural Analytics, 2020. https://doi.org/10.22148/001c.11828. Cite
National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science. One of the Pathways by Which the Scientific Community Confirms the Validity of a New Scientific Discovery Is by Repeating the Research That Produced It. When a Scientific Effort Fails to Independently Confirm the Computations or Results of a Previous Study, Some Fear That It May Be a Symptom of a Lack of Rigor in Science, While Others Argue That Such an Observed Inconsistency Can Be an Important Precursor to New Discovery.  Concerns about Reproducibility and Replicability Have Been Expressed in Both Scientific and Popular Media. As These Concerns Came to Light, Congress Requested That the National Academies of Sciences, Engineering, and Medicine Conduct a Study to Assess the Extent of Issues Related to Reproducibility and Replicability and to Offer Recommendations for Improving Rigor and Transparency in Scientific Research.  Reproducibility and Replicability in Science Defines Reproducibility and Replicability and Examines the Factors That May Lead to Non-Reproducibility and Non-Replicability in Research. Unlike the Typical Expectation of Reproducibility between Two Computations, Expectations about Replicability Are More Nuanced, and in Some Cases a Lack of Replicability Can Aid the Process of Scientific Discovery. This Report Provides Recommendations to Researchers, Academic Institutions, Journals, and Funders on Steps They Can Take to Improve Reproducibility and Replicability in Science. Washington, DC: National Academies Press, 2019. https://nap.nationalacademies.org/catalog/25303/reproducibility-and-replicability-in-science. Cite
Pimentel, João Felipe, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. “A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks.” In Proceedings of the 16th International Conference on Mining Software Repositories, 507–17. MSR ’19. Montreal, Quebec, Canada: IEEE Press, 2019. https://doi.org/10.1109/MSR.2019.00077. Cite
Rule, Adam, Amanda Birmingham, Cristal Zuniga, Ilkay Altintas, Shih-Cheng Huang, Rob Knight, Niema Moshiri, et al. “Ten Simple Rules for Writing and Sharing Computational Analyses in Jupyter Notebooks.” PLOS Computational Biology 15, no. 7 (2019): e1007007. https://doi.org/10.1371/journal.pcbi.1007007. Cite
Mendez, Kevin M., Leighton Pritchard, Stacey N. Reinke, and David I. Broadhurst. “Toward Collaborative Open Data Science in Metabolomics Using Jupyter Notebooks and Cloud Computing.” Metabolomics 15, no. 10 (2019): 125. https://doi.org/10.1007/s11306-019-1588-0. Cite
Liu, Alan, Scott Kleinman, Jeremy Douglass, Lindsay Thomas, Ashley Champagne, and Jamal Russell. “Open, Shareable, Reproducible Workflows for the Digital Humanities: The Case of the 4Humanities.Org ‘WhatEvery1Says’ Project.” In Digital Humanities 2017 Conference Abstracts. Montreal: Alliance of Digital Humanities Organizations (ADHO), 2017. Cite
Kluyver, Thomas, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, et al. “Jupyter Notebooks – a Publishing Format for Reproducible Computational Workflows.” Positioning and Power in Academic Publishing: Players, Agents and Agendas, 2016, 87–90. https://doi.org/10.3233/978-1-61499-649-1-87. Cite
Peng, Roger D. “Reproducible Research in Computational Science.” Science 334, no. 6060 (2011): 1226–27. https://doi.org/10.1126/science.1213847. Cite