TY - CONF TI - Quantifying the Effects of Text Duplication on Semantic Models AU - Schofield, Alexandra AU - Thompson, Laure AU - Mimno, David T2 - Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing AB - Duplicate documents are a pervasive problem in text datasets and can have a strong effect on unsupervised models. Methods to remove duplicate texts are typically heuristic or very expensive, so it is vital to know when and why they are needed. We measure the sensitivity of two latent semantic methods to the presence of different levels of document repetition. By artificially creating different forms of duplicate text we confirm several hypotheses about how repeated text impacts models. While a small amount of duplication is tolerable, substantial over-representation of subsets of the text may overwhelm meaningful topical patterns. C1 - Copenhagen, Denmark C3 - Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing DA - 2017/// PY - 2017 DO - 10.18653/v1/D17-1290 DP - DOI.org (Crossref) SP - 2737 EP - 2747 LA - en PB - Association for Computational Linguistics UR - http://aclweb.org/anthology/D17-1290 Y2 - 2019/07/10/15:47:02 KW - Topic model optimization KW - Topic modeling ER -