Follow
Area: Data and Benchmarking
Natural Language Processing critically depends on data. Yet, most existing text collections consist of individual, isolated documents. To foster research in cross-document NLP, this area develops novel corpora and unified benchmarks for the study of interconnected, changing texts. We devise an inclusive view on text that - unlike most prior work in NLP - takes both textual and non-textual elements into account to enable efficient cross-document processing.
As NLP finds its way into real-life applications, concerns regarding the provenance, quality and legal status of data arise. Collecting interconnected, living texts is coupled with additional challenges, incl. multiple authorship, confidentiality and privacy concerns. Contributing to the growing body of research on ethics in NLP, this area puts special focus on developing general-purpose methodologies and workflows for ethics-, confidentiality- and copyright-aware data collection.
Publications
Datasets and Code
A pre-release of our general-purpose library for cross-document NLP modelling and analysis. Current version of the library provides converters from several document formats into a uniform data model, as well as an API for common graph operations that facilitate cross-document analysis on varying granularity levels. The library is constantly extended to cover more document formats and cross-document relation types, star the repo to stay up-to-date with the new releases!
An open implementation of a peer reviewing data collection workflow for OpenReview.net. The code can be used to set up a licensing workflow for peer review data and paper drafts submitted to OpenReview-based venues. We provide the implementation for creating license tasks for reviewers and authors of selected submissions, as well as the code for retrieving the peer reviewing data in a privacy- and anonymity-aware fashion.
A corpus of anonymised structured peer reviews collected during the ACL-2018 reviewing campaign. ACL-2018 employed a rich reviewing schema, with each review containing a wide range of textual, binary, ternary and numerical fields, including Strengths, Weaknesses, Summary, aspect scores, overall score and confidence scores. While openly publishing the textual data is not possible due to the ethical concerns, we make numerical data publicly available to support meta-scientific study of peer reviewing in the NLP community.
