Bibliography – Reporting and Documentation Methods

Selected DH research and resources bearing on, or utilized by, the WE1S project.
(all) Distant Reading | Cultural Analytics | | Sociocultural Approaches | Topic Modeling in DH | Non-consumptive Use


2133649 Reporting and documentation methods 1 chicago-fullnote-bibliography 50 date desc year 1 1 1 4188 https://we1s.ucsb.edu/wp-content/plugins/zotpress/
%7B%22status%22%3A%22success%22%2C%22updateneeded%22%3Afalse%2C%22instance%22%3Afalse%2C%22meta%22%3A%7B%22request_last%22%3A0%2C%22request_next%22%3A0%2C%22used_cache%22%3Atrue%7D%2C%22data%22%3A%5B%7B%22key%22%3A%222TJL9B2H%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Droge%22%2C%22parsedDate%22%3A%222020%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BDroge%2C%20Abigail.%20%26%23x201C%3BWE1S%20to%20Develop%20%26%23x2018%3BResearch-to-Action%20Toolkits.%26%23x2019%3B%26%23x201D%3B%20%26lt%3Bi%26gt%3BWE1S%26lt%3B%5C%2Fi%26gt%3B%20%28blog%29%2C%202020.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fwe1s.ucsb.edu%5C%2Fresearch_post%5C%2Fwe1s-to-develop-research-to-action-toolkits%5C%2F%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fwe1s.ucsb.edu%5C%2Fresearch_post%5C%2Fwe1s-to-develop-research-to-action-toolkits%5C%2F%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3D2TJL9B2H%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22blogPost%22%2C%22title%22%3A%22WE1S%20to%20Develop%20%5Cu201cResearch-to-Action%20Toolkits%5Cu201d%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Abigail%22%2C%22lastName%22%3A%22Droge%22%7D%5D%2C%22abstractNote%22%3A%22WhatEvery1Says%20is%20excited%20to%20announce%20that%20the%20project%20will%20be%20developing%20and%20launching%20a%20series%20of%20Research-to-Action%20Toolkits%20in%202020%20to%20present%20our%20research%20findings.%20Each%20Toolkit%20will%20be%20struct%5Cu2026%22%2C%22blogTitle%22%3A%22WE1S%22%2C%22date%22%3A%222020%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwe1s.ucsb.edu%5C%2Fresearch_post%5C%2Fwe1s-to-develop-research-to-action-toolkits%5C%2F%22%2C%22language%22%3A%22en%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222021-05-25T22%3A47%3A08Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%2C%7B%22tag%22%3A%22WE1S%20blog%20post%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22YT7FHHAY%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Gebru%20et%20al.%22%2C%22parsedDate%22%3A%222019%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BGebru%2C%20Timnit%2C%20Jamie%20Morgenstern%2C%20Briana%20Vecchione%2C%20Jennifer%20Wortman%20Vaughan%2C%20Hanna%20Wallach%2C%20Hal%20Daume%26%23xE9%3B%20III%2C%20and%20Kate%20Crawford.%20%26%23x201C%3BDatasheets%20for%20Datasets.%26%23x201D%3B%20%26lt%3Bi%26gt%3BArXiv%3A1803.09010%20%5BCs%5D%26lt%3B%5C%2Fi%26gt%3B%2C%202019.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1803.09010%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1803.09010%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DYT7FHHAY%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Datasheets%20for%20Datasets%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Timnit%22%2C%22lastName%22%3A%22Gebru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jamie%22%2C%22lastName%22%3A%22Morgenstern%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Briana%22%2C%22lastName%22%3A%22Vecchione%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jennifer%20Wortman%22%2C%22lastName%22%3A%22Vaughan%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hanna%22%2C%22lastName%22%3A%22Wallach%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Hal%22%2C%22lastName%22%3A%22Daume%5Cu00e9%20III%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kate%22%2C%22lastName%22%3A%22Crawford%22%7D%5D%2C%22abstractNote%22%3A%22Currently%20there%20is%20no%20standard%20way%20to%20identify%20how%20a%20dataset%20was%20created%2C%20and%20what%20characteristics%2C%20motivations%2C%20and%20potential%20skews%20it%20represents.%20To%20begin%20to%20address%20this%20issue%2C%20we%20propose%20the%20concept%20of%20a%20datasheet%20for%20datasets%2C%20a%20short%20document%20to%20accompany%20public%20datasets%2C%20commercial%20APIs%2C%20and%20pretrained%20models.%20The%20goal%20of%20this%20proposal%20is%20to%20enable%20better%20communication%20between%20dataset%20creators%20and%20users%2C%20and%20help%20the%20AI%20community%20move%20toward%20greater%20transparency%20and%20accountability.%20By%20analogy%2C%20in%20computer%20hardware%2C%20it%20has%20become%20industry%20standard%20to%20accompany%20everything%20from%20the%20simplest%20components%20%28e.g.%2C%20resistors%29%2C%20to%20the%20most%20complex%20microprocessor%20chips%2C%20with%20datasheets%20detailing%20standard%20operating%20characteristics%2C%20test%20results%2C%20recommended%20usage%2C%20and%20other%20information.%20We%20outline%20some%20of%20the%20questions%20a%20datasheet%20for%20datasets%20should%20answer.%20These%20questions%20focus%20on%20when%2C%20where%2C%20and%20how%20the%20training%20data%20was%20gathered%2C%20its%20recommended%20use%20cases%2C%20and%2C%20in%20the%20case%20of%20human-centric%20datasets%2C%20information%20regarding%20the%20subjects%26%23039%3B%20demographics%20and%20consent%20as%20applicable.%20We%20develop%20prototypes%20of%20datasheets%20for%20two%20well-known%20datasets%3A%20Labeled%20Faces%20in%20The%20Wild%20and%20the%20Pang%20%5C%5C%26amp%3B%20Lee%20Polarity%20Dataset.%22%2C%22date%22%3A%222019%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1803.09010%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222019-12-05T07%3A02%3A23Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Interpretability%20and%20explainability%22%7D%2C%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22QJ5ZHJWR%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Mitchell%20et%20al.%22%2C%22parsedDate%22%3A%222019%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BMitchell%2C%20Margaret%2C%20Simone%20Wu%2C%20Andrew%20Zaldivar%2C%20Parker%20Barnes%2C%20Lucy%20Vasserman%2C%20Ben%20Hutchinson%2C%20Elena%20Spitzer%2C%20Inioluwa%20Deborah%20Raji%2C%20and%20Timnit%20Gebru.%20%26%23x201C%3BModel%20Cards%20for%20Model%20Reporting.%26%23x201D%3B%20%26lt%3Bi%26gt%3BProceedings%20of%20the%20Conference%20on%20Fairness%2C%20Accountability%2C%20and%20Transparency%20-%20FAT%2A%20%26%23x2019%3B19%26lt%3B%5C%2Fi%26gt%3B%2C%202019%2C%20220%26%23x2013%3B29.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1145%5C%2F3287560.3287596%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1145%5C%2F3287560.3287596%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DQJ5ZHJWR%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Model%20Cards%20for%20Model%20Reporting%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Margaret%22%2C%22lastName%22%3A%22Mitchell%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Simone%22%2C%22lastName%22%3A%22Wu%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Andrew%22%2C%22lastName%22%3A%22Zaldivar%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Parker%22%2C%22lastName%22%3A%22Barnes%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Lucy%22%2C%22lastName%22%3A%22Vasserman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ben%22%2C%22lastName%22%3A%22Hutchinson%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Elena%22%2C%22lastName%22%3A%22Spitzer%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Inioluwa%20Deborah%22%2C%22lastName%22%3A%22Raji%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Timnit%22%2C%22lastName%22%3A%22Gebru%22%7D%5D%2C%22abstractNote%22%3A%22Trained%20machine%20learning%20models%20are%20increasingly%20used%20to%20perform%20high-impact%20tasks%20in%20areas%20such%20as%20law%20enforcement%2C%20medicine%2C%20education%2C%20and%20employment.%20In%20order%20to%20clarify%20the%20intended%20use%20cases%20of%20machine%20learning%20models%20and%20minimize%20their%20usage%20in%20contexts%20for%20which%20they%20are%20not%20well%20suited%2C%20we%20recommend%20that%20released%20models%20be%20accompanied%20by%20documentation%20detailing%20their%20performance%20characteristics.%20In%20this%20paper%2C%20we%20propose%20a%20framework%20that%20we%20call%20model%20cards%2C%20to%20encourage%20such%20transparent%20model%20reporting.%20Model%20cards%20are%20short%20documents%20accompanying%20trained%20machine%20learning%20models%20that%20provide%20benchmarked%20evaluation%20in%20a%20variety%20of%20conditions%2C%20such%20as%20across%20different%20cultural%2C%20demographic%2C%20or%20phenotypic%20groups%20%28e.g.%2C%20race%2C%20geographic%20location%2C%20sex%2C%20Fitzpatrick%20skin%20type%29%20and%20intersectional%20groups%20%28e.g.%2C%20age%20and%20race%2C%20or%20sex%20and%20Fitzpatrick%20skin%20type%29%20that%20are%20relevant%20to%20the%20intended%20application%20domains.%20Model%20cards%20also%20disclose%20the%20context%20in%20which%20models%20are%20intended%20to%20be%20used%2C%20details%20of%20the%20performance%20evaluation%20procedures%2C%20and%20other%20relevant%20information.%20While%20we%20focus%20primarily%20on%20human-centered%20machine%20learning%20models%20in%20the%20application%20fields%20of%20computer%20vision%20and%20natural%20language%20processing%2C%20this%20framework%20can%20be%20used%20to%20document%20any%20trained%20machine%20learning%20model.%20To%20solidify%20the%20concept%2C%20we%20provide%20cards%20for%20two%20supervised%20models%3A%20One%20trained%20to%20detect%20smiling%20faces%20in%20images%2C%20and%20one%20trained%20to%20detect%20toxic%20comments%20in%20text.%20We%20propose%20model%20cards%20as%20a%20step%20towards%20the%20responsible%20democratization%20of%20machine%20learning%20and%20related%20AI%20technology%2C%20increasing%20transparency%20into%20how%20well%20AI%20technology%20works.%20We%20hope%20this%20work%20encourages%20those%20releasing%20trained%20machine%20learning%20models%20to%20accompany%20model%20releases%20with%20similar%20detailed%20evaluation%20numbers%20and%20other%20relevant%20documentation.%22%2C%22date%22%3A%222019%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1145%5C%2F3287560.3287596%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1810.03993%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222019-12-05T06%3A51%3A13Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Interpretability%20and%20explainability%22%7D%2C%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22NTL6YC8M%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Holland%20et%20al.%22%2C%22parsedDate%22%3A%222018%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BHolland%2C%20Sarah%2C%20Ahmed%20Hosny%2C%20Sarah%20Newman%2C%20Joshua%20Joseph%2C%20and%20Kasia%20Chmielinski.%20%26%23x201C%3BThe%20Dataset%20Nutrition%20Label%3A%20A%20Framework%20To%20Drive%20Higher%20Data%20Quality%20Standards.%26%23x201D%3B%20%26lt%3Bi%26gt%3BArXiv%3A1805.03677%20%5BCs%5D%26lt%3B%5C%2Fi%26gt%3B%2C%202018.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1805.03677%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1805.03677%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DNTL6YC8M%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22The%20Dataset%20Nutrition%20Label%3A%20A%20Framework%20To%20Drive%20Higher%20Data%20Quality%20Standards%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sarah%22%2C%22lastName%22%3A%22Holland%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Ahmed%22%2C%22lastName%22%3A%22Hosny%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Sarah%22%2C%22lastName%22%3A%22Newman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Joshua%22%2C%22lastName%22%3A%22Joseph%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Kasia%22%2C%22lastName%22%3A%22Chmielinski%22%7D%5D%2C%22abstractNote%22%3A%22Artificial%20intelligence%20%28AI%29%20systems%20built%20on%20incomplete%20or%20biased%20data%20will%20often%20exhibit%20problematic%20outcomes.%20Current%20methods%20of%20data%20analysis%2C%20particularly%20before%20model%20development%2C%20are%20costly%20and%20not%20standardized.%20The%20Dataset%20Nutrition%20Label%20%28the%20Label%29%20is%20a%20diagnostic%20framework%20that%20lowers%20the%20barrier%20to%20standardized%20data%20analysis%20by%20providing%20a%20distilled%20yet%20comprehensive%20overview%20of%20dataset%20%26quot%3Bingredients%26quot%3B%20before%20AI%20model%20development.%20Building%20a%20Label%20that%20can%20be%20applied%20across%20domains%20and%20data%20types%20requires%20that%20the%20framework%20itself%20be%20flexible%20and%20adaptable%3B%20as%20such%2C%20the%20Label%20is%20comprised%20of%20diverse%20qualitative%20and%20quantitative%20modules%20generated%20through%20multiple%20statistical%20and%20probabilistic%20modelling%20backends%2C%20but%20displayed%20in%20a%20standardized%20format.%20To%20demonstrate%20and%20advance%20this%20concept%2C%20we%20generated%20and%20published%20an%20open%20source%20prototype%20with%20seven%20sample%20modules%20on%20the%20ProPublica%20Dollars%20for%20Docs%20dataset.%20The%20benefits%20of%20the%20Label%20are%20manyfold.%20For%20data%20specialists%2C%20the%20Label%20will%20drive%20more%20robust%20data%20analysis%20practices%2C%20provide%20an%20efficient%20way%20to%20select%20the%20best%20dataset%20for%20their%20purposes%2C%20and%20increase%20the%20overall%20quality%20of%20AI%20models%20as%20a%20result%20of%20more%20robust%20training%20datasets%20and%20the%20ability%20to%20check%20for%20issues%20at%20the%20time%20of%20model%20development.%20For%20those%20building%20and%20publishing%20datasets%2C%20the%20Label%20creates%20an%20expectation%20of%20explanation%2C%20which%20will%20drive%20better%20data%20collection%20practices.%20We%20also%20explore%20the%20limitations%20of%20the%20Label%2C%20including%20the%20challenges%20of%20generalizing%20across%20diverse%20datasets%2C%20and%20the%20risk%20of%20using%20%26quot%3Bground%20truth%26quot%3B%20data%20as%20a%20comparison%20dataset.%20We%20discuss%20ways%20to%20move%20forward%20given%20the%20limitations%20identified.%20Lastly%2C%20we%20lay%20out%20future%20directions%20for%20the%20Dataset%20Nutrition%20Label%20project%2C%20including%20research%20and%20public%20policy%20agendas%20to%20further%20advance%20consideration%20of%20the%20concept.%22%2C%22date%22%3A%222018%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%22%22%2C%22ISSN%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Farxiv.org%5C%2Fabs%5C%2F1805.03677%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222019-12-05T06%3A55%3A35Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Interpretability%20and%20explainability%22%7D%2C%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%5D%7D%7D%2C%7B%22key%22%3A%226SEBDCE8%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Bender%20and%20Friedman%22%2C%22parsedDate%22%3A%222018%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBender%2C%20Emily%20M.%2C%20and%20Batya%20Friedman.%20%26%23x201C%3BData%20Statements%20for%20Natural%20Language%20Processing%3A%20Toward%20Mitigating%20System%20Bias%20and%20Enabling%20Better%20Science.%26%23x201D%3B%20%26lt%3Bi%26gt%3BTransactions%20of%20the%20Association%20for%20Computational%20Linguistics%26lt%3B%5C%2Fi%26gt%3B%206%20%282018%29%3A%20587%26%23x2013%3B604.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1162%5C%2Ftacl_a_00041%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.1162%5C%2Ftacl_a_00041%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3D6SEBDCE8%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Data%20Statements%20for%20Natural%20Language%20Processing%3A%20Toward%20Mitigating%20System%20Bias%20and%20Enabling%20Better%20Science%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Emily%20M.%22%2C%22lastName%22%3A%22Bender%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Batya%22%2C%22lastName%22%3A%22Friedman%22%7D%5D%2C%22abstractNote%22%3A%22In%20this%20paper%2C%20we%20propose%20data%20statements%20as%20a%20design%20solution%20and%20professional%20practice%20for%20natural%20language%20processing%20technologists%2C%20in%20both%20research%20and%20development.%20Through%20the%20adoption%20and%20widespread%20use%20of%20data%20statements%2C%20the%20field%20can%20begin%20to%20address%20critical%20scientific%20and%20ethical%20issues%20that%20result%20from%20the%20use%20of%20data%20from%20certain%20populations%20in%20the%20development%20of%20technology%20for%20other%20populations.%20We%20present%20a%20form%20that%20data%20statements%20can%20take%20and%20explore%20the%20implications%20of%20adopting%20them%20as%20part%20of%20regular%20practice.%20We%20argue%20that%20data%20statements%20will%20help%20alleviate%20issues%20related%20to%20exclusion%20and%20bias%20in%20language%20technology%2C%20lead%20to%20better%20precision%20in%20claims%20about%20how%20natural%20language%20processing%20research%20can%20generalize%20and%20thus%20better%20engineering%20results%2C%20protect%20companies%20from%20public%20embarrassment%2C%20and%20ultimately%20lead%20to%20language%20technology%20that%20meets%20its%20users%20in%20their%20own%20preferred%20linguistic%20style%20and%20furthermore%20does%20not%20misrepresent%20them%20to%20others.%22%2C%22date%22%3A%222018%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.1162%5C%2Ftacl_a_00041%22%2C%22ISSN%22%3A%222307-387X%22%2C%22url%22%3A%22https%3A%5C%2F%5C%2Fwww.mitpressjournals.org%5C%2Fdoi%5C%2Fabs%5C%2F10.1162%5C%2Ftacl_a_00041%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222019-12-05T06%3A54%3A28Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Interpretability%20and%20explainability%22%7D%2C%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%5D%7D%7D%2C%7B%22key%22%3A%22TDUZV8IL%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Bounegru%20et%20al.%22%2C%22parsedDate%22%3A%222017%22%2C%22numChildren%22%3A0%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BBounegru%2C%20Liliana%2C%20Jonathan%20Gray%2C%20Tommaso%20Venturini%2C%20and%20Michele%20Mauri.%20%26lt%3Bi%26gt%3BA%20Field%20Guide%20to%20%26%23x201C%3BFake%20News%26%23x201D%3B%20and%20Other%20Information%20Disorders%26lt%3B%5C%2Fi%26gt%3B.%20Amsterdam%3A%20Public%20Data%20Lab%2C%202017.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-ItemURL%26%23039%3B%20href%3D%26%23039%3Bhttp%3A%5C%2F%5C%2Ffakenews.publicdatalab.org%5C%2F%26%23039%3B%26gt%3Bhttp%3A%5C%2F%5C%2Ffakenews.publicdatalab.org%5C%2F%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3DTDUZV8IL%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22book%22%2C%22title%22%3A%22A%20Field%20Guide%20to%20%5Cu201cFake%20News%5Cu201d%20and%20Other%20Information%20Disorders%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Liliana%22%2C%22lastName%22%3A%22Bounegru%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Jonathan%22%2C%22lastName%22%3A%22Gray%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Tommaso%22%2C%22lastName%22%3A%22Venturini%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Michele%22%2C%22lastName%22%3A%22Mauri%22%7D%5D%2C%22abstractNote%22%3A%22A%20Field%20Guide%20to%20%5Cu201cFake%20News%5Cu201d%20and%20Other%20Information%20Disorders%20explores%20the%20use%20of%20digital%20methods%20to%20study%20viral%20news%2C%20political%20memes%2C%20trolling%20practices%20and%20their%20social%20life%20online.%20It%20is%20a%20project%20of%20the%20Public%20Data%20Lab%20with%20support%20from%20First%20Draft.%22%2C%22date%22%3A%222017%22%2C%22language%22%3A%22en%22%2C%22ISBN%22%3A%22%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Ffakenews.publicdatalab.org%5C%2F%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222020-04-03T07%3A02%3A58Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Fake%20news%22%7D%2C%7B%22tag%22%3A%22Journalism%22%7D%2C%7B%22tag%22%3A%22News%20and%20social%20media%22%7D%2C%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%5D%7D%7D%2C%7B%22key%22%3A%223I264DEX%22%2C%22library%22%3A%7B%22id%22%3A2133649%7D%2C%22meta%22%3A%7B%22creatorSummary%22%3A%22Collins%20et%20al.%22%2C%22parsedDate%22%3A%222015%22%2C%22numChildren%22%3A1%7D%2C%22bib%22%3A%22%26lt%3Bdiv%20class%3D%26quot%3Bcsl-bib-body%26quot%3B%20style%3D%26quot%3Bline-height%3A%201.35%3B%20padding-left%3A%201em%3B%20text-indent%3A-1em%3B%26quot%3B%26gt%3B%5Cn%20%20%26lt%3Bdiv%20class%3D%26quot%3Bcsl-entry%26quot%3B%26gt%3BCollins%2C%20Gary%20S.%2C%20Johannes%20B.%20Reitsma%2C%20Douglas%20G.%20Altman%2C%20and%20Karel%20G.M.%20Moons.%20%26%23x201C%3BTransparent%20Reporting%20of%20a%20Multivariable%20Prediction%20Model%20for%20Individual%20Prognosis%20Or%20Diagnosis%20%28TRIPOD%29%3A%20The%20TRIPOD%20Statement.%26%23x201D%3B%20%26lt%3Bi%26gt%3BAnnals%20of%20Internal%20Medicine%26lt%3B%5C%2Fi%26gt%3B%20162%2C%20no.%201%20%282015%29%3A%2055.%20%26lt%3Ba%20class%3D%26%23039%3Bzp-DOIURL%26%23039%3B%20href%3D%26%23039%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.7326%5C%2FM14-0697%26%23039%3B%26gt%3Bhttps%3A%5C%2F%5C%2Fdoi.org%5C%2F10.7326%5C%2FM14-0697%26lt%3B%5C%2Fa%26gt%3B.%20%26lt%3Ba%20title%3D%26%23039%3BCite%20in%20RIS%20Format%26%23039%3B%20class%3D%26%23039%3Bzp-CiteRIS%26%23039%3B%20data-zp-cite%3D%26%23039%3Bapi_user_id%3D2133649%26amp%3Bitem_key%3D3I264DEX%26%23039%3B%20href%3D%26%23039%3Bjavascript%3Avoid%280%29%3B%26%23039%3B%26gt%3BCite%26lt%3B%5C%2Fa%26gt%3B%20%26lt%3B%5C%2Fdiv%26gt%3B%5Cn%26lt%3B%5C%2Fdiv%26gt%3B%22%2C%22data%22%3A%7B%22itemType%22%3A%22journalArticle%22%2C%22title%22%3A%22Transparent%20Reporting%20of%20a%20multivariable%20prediction%20model%20for%20Individual%20Prognosis%20Or%20Diagnosis%20%28TRIPOD%29%3A%20The%20TRIPOD%20Statement%22%2C%22creators%22%3A%5B%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Gary%20S.%22%2C%22lastName%22%3A%22Collins%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Johannes%20B.%22%2C%22lastName%22%3A%22Reitsma%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Douglas%20G.%22%2C%22lastName%22%3A%22Altman%22%7D%2C%7B%22creatorType%22%3A%22author%22%2C%22firstName%22%3A%22Karel%20G.M.%22%2C%22lastName%22%3A%22Moons%22%7D%5D%2C%22abstractNote%22%3A%22The%20Transparent%20Reporting%20of%20a%20multivariable%20prediction%20model%20for%20Individual%20Prognosis%20Or%20Diagnosis%20%28TRIPOD%29%20Initiative%20developed%20a%20set%20of%20recommendations%20for%20the%20reporting%20of%20studies%20developing%2C%20validating%2C%20or%20updating%20a%20prediction%20model%2C%20whether%20for%20diagnostic%20or%20prognostic%20purposes....The%20resulting%20TRIPOD%20Statement%20is%20a%20checklist%20of%2022%20items%2C%20deemed%20essential%20for%20transparent%20reporting%20of%20a%20prediction%20model%20study.%22%2C%22date%22%3A%222015%22%2C%22language%22%3A%22en%22%2C%22DOI%22%3A%2210.7326%5C%2FM14-0697%22%2C%22ISSN%22%3A%220003-4819%22%2C%22url%22%3A%22http%3A%5C%2F%5C%2Fannals.org%5C%2Farticle.aspx%3Fdoi%3D10.7326%5C%2FM14-0697%22%2C%22collections%22%3A%5B%5D%2C%22dateModified%22%3A%222019-12-05T06%3A59%3A52Z%22%2C%22tags%22%3A%5B%7B%22tag%22%3A%22Interpretability%20and%20explainability%22%7D%2C%7B%22tag%22%3A%22Reporting%20and%20documentation%20methods%22%7D%5D%7D%7D%5D%7D
Droge, Abigail. “WE1S to Develop ‘Research-to-Action Toolkits.’” WE1S (blog), 2020. https://we1s.ucsb.edu/research_post/we1s-to-develop-research-to-action-toolkits/. Cite
Gebru, Timnit, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumeé III, and Kate Crawford. “Datasheets for Datasets.” ArXiv:1803.09010 [Cs], 2019. http://arxiv.org/abs/1803.09010. Cite
Mitchell, Margaret, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. “Model Cards for Model Reporting.” Proceedings of the Conference on Fairness, Accountability, and Transparency - FAT* ’19, 2019, 220–29. https://doi.org/10.1145/3287560.3287596. Cite
Holland, Sarah, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. “The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards.” ArXiv:1805.03677 [Cs], 2018. http://arxiv.org/abs/1805.03677. Cite
Bender, Emily M., and Batya Friedman. “Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science.” Transactions of the Association for Computational Linguistics 6 (2018): 587–604. https://doi.org/10.1162/tacl_a_00041. Cite
Bounegru, Liliana, Jonathan Gray, Tommaso Venturini, and Michele Mauri. A Field Guide to “Fake News” and Other Information Disorders. Amsterdam: Public Data Lab, 2017. http://fakenews.publicdatalab.org/. Cite
Collins, Gary S., Johannes B. Reitsma, Douglas G. Altman, and Karel G.M. Moons. “Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement.” Annals of Internal Medicine 162, no. 1 (2015): 55. https://doi.org/10.7326/M14-0697. Cite