ACL 2013 Workshop: PITR 2013
The Second Workshop on Predicting and Improving Text Readability for Target Reader Populations
8 August, 2013
The Second Workshop on Predicting and Improving Text Readability for Target Reader Populations was held in conjunction with the Association for Computational Linguistics (ACL) 2013 conference, 4 - 9 August, 2013, at the National Palace of Culture, Sofia, Bulgaria.
This workshop is an ACL Special Interest Group on Speech and Language Processing for Assistive Technologies (SIG-SLPAT) sponsored workshop.
PROCEEDINGS
PITR 2013 proceedings are available here.
PROGRAMME
| 09:20-10:30 Session 1: Plenary | ||
| 09:20 | Welcome and Introduction | |
| 09:30 | Invited Talk | |
| Identifying outstanding writing: Corpus and experiments based on the
science journalism genre Annie Louis abstract
I will discuss the hitherto unexplored area of text quality prediction:
identifying outstanding pieces of writing. A system to
do this task will benefit article recommendation
and information retrieval. To do the task, we need to not only
be able to measure spelling, grammar and organization quality but also
quantify creative and engaging writing and topic. In addition, new
resources are needed as existing corpora are focused on non-native
student writing, output of text generation systems and artificial
manipulation to create texts with low quality writing. I will propose the science journalism genre as an apt one for such text quality experiments. Science journalism pieces entertain a reader as much as they teach and inform. I will introduce a corpus of science journalism articles which we have collected for use in text quality studies. The corpus contains science journalism pieces from the New York Times split into two categories---written by award-winning journalists and others. This corpus offers many desirable properties which were unavailable in previous resources. It represents realistic differences in writing quality, samples are based on professional writers rather than language learners, contains thousands of articles, and is publicly available. I will also describe automatic measures based on visual elements, surprisal and structure of these articles which are indicative of outstanding articles in the corpus and also turn out complementary to traditional metrics to quantify readability and organization quality of writing. | ||
| 10:30-11:00 Coffee break | ||
| 11:00-12:30 Session 2: Posters | ||
| 11:00 | Poster Teasers | |
| 11:20 | Poster Session | |
| Sentence Simplification as Tree Transduction Dan Feblowitz and David Kauchak abstract
In this paper, we introduce a syntax-based sentence simplifier that models simplification using a probabilistic synchronous tree substitution grammar (STSG). To improve the STSG model specificity we utilize a multi-level backoff model with additional syntactic annotations that allow for better discrimination over previous STSG formulations. We compare our approach to T3 (Cohn and Lapata, 2009), a recent STSG implementation, as well as two state-of-the-art phrase-based sentence simplifiers on a corpus of aligned sentences from English and Simple English Wikipedia. Our new approach performs significantly better than T3, similarly to human simplifications for both simplicity and fluency, and better than the phrase-based simplifiers for most of the evaluation metrics.
| ||
| Building a German/Simple German Parallel Corpus for Automatic Text Simplification David Klaper, Sarah Ebling, Martin Volk abstract
In this paper we report our experiments in creating a parallel corpus using German/Simple German documents from the web. We require parallel data to build a statistical machine translation (SMT) system that translates from German into Simple German. Parallel data for SMT systems needs to be aligned at the sentence level. We applied an existing monolingual sentence alignment algorithm. We show the limits of the algorithm with respect to the language and domain of our data and suggest ways of circumventing them.
| ||
| The C-Score - Proposing Reading Comprehension Metrics as a Common Evaluation Measure for Text Simplification Irina Temnikova and Galina Maneva abstract
This article addresses the lack of common approaches for text simplification evaluation, by presenting the first attempt for a common evaluation metrics. The article proposes reading comprehension evaluation as a method for evaluating the results of Text Simplification (TS). An experiment, as an example application of the evaluation method, as well as three formulae to quantify reading comprehension, are presented. The formulae produce an unique score, the C-score, which gives an estimation of user’s reading comprehension of a certain text. The score can be used to evaluate the performance of a text simplification engine on pairs of complex and simplified texts, or to compare the performances of different TS methods using the same texts. The approach can be particularly useful for the modern crowd-sourcing approaches, such as those employing the Amazon’s Mechanical Turk or CrowdFlower. The aim of this paper is thus to propose an evaluation approach and to motivate the TS community to start a relevant discussion, in order to come up with a common evaluation metrics for this task.
| ||
| A Language-Independent Approach to Automatic Text Difficulty Assessment for Second-Language Learners Wade Shen, Jennifer Williams, Tamas Marius and Elizabeth Slesky abstract
In this paper, we introduce a new baseline for language-independent text difficulty assessment applied to the Interagency Language Roundtable (ILR) proficiency scale. We demonstrate that reading level assessment is a discriminative problem that is best-suited for regression. Our baseline uses z-normalized shallow length features and TF-LOG weighted vectors on bag-of-words for Arabic, Dari, English, and Pashto. We compare Support Vector Machines and the Margin-Infused Relaxed Algorithm measured by mean squared error. We provide an analysis of which features are most predictive of a given level.
| ||
| Guest Paper from NLP4ITA Proceedings: A System for the Simplification of Numerical Expressions at Different Levels of Understandability Susan Bautista, Raquel Hervás, Pablo Gervás, Richard Power and Sandra Williams | ||
| 12:30-14:00 Lunch break | ||
| Session 3: Presentations | ||
| 14:00 | Text Modification for Bulgarian Sign Language Users | |
|
Slavina Lozanova, Ivelina Stoyanova, Svetlozara Leseva, Svetla Koeva and Boian Savtchev abstract
The paper discusses the main issues regarding the reading skills and comprehension proficiency in written Bulgarian of people with communication difficulties, and deaf people, in particular. We consider several key components of text comprehension which pose a challenge for deaf readers and propose a rule-based system for automatic modification of Bulgarian texts intended to facilitate comprehension by deaf people, to assist education, etc. In order to demonstrate the benefits of such a system and to evaluate its performance, we have carried out a study among a group of deaf people who use Bulgarian Sign Language (BulSL) as their primary language (primary BulSL users), which compares the comprehensibility of original texts and their modified versions. The results shows a considerable improvement in readability when using modified texts, but at the same time demonstrates that the level of comprehension is still low, and that a complex set of modifications will have to be implemented to attain satisfactory results.
| ||
| 14:20 | Modeling Comma Placement in Chinese Text for Better Readability using Linguistic Features and Gaze Information abstract
Comma placements in Chinese text are relatively arbitrary although there are some syntactic guidelines for them. In this research, we attempt to improve the readability of text by optimizing comma placements through integration of linguistic features of text and gaze features of readers.
We design a comma predictor for general Chinese text based on conditional random field models with linguistic features. After that, we build a rule-based filter for categorizing commas in text according to their contribution to readability based on the analysis of gazes of people reading text with and without commas.
The experimental results show that our predictor reproduces the comma distribution in the Penn Chinese Treebank with 78.41 in F1-score and commas chosen by our filter smooth certain gaze behaviors.
| |
| Tadayoshi Hara, Chen Chen, Yoshinobu Kano and Akiko Aizawa | ||
| 14:40 | On The Applicability of Readability Models to Web Texts | |
| Sowmya Vajjala and Detmar Meurers abstract
An increasing range of features is being used for automatic readability classification. The impact of the features typically is evaluated using reference corpora containing graded reading material. But how do the readability models and the features they are based on perform on real-world web texts? In this paper, we want to take a step towards understanding this aspect on the basis of a broad range of lexical and syntactic features and several web datasets we collected.
Applying our models to web search results, we find that the average reading level of the retrieved web documents is rela- tively high. At the same time, documents at a wide range of reading levels are identified and even among the Top-10 search results one finds documents at the lower levels, supporting the potential usefulness of readability ranking for the web. Finally, we report on generalization experiments showing that the features we used generalize well across different web sources. | ||
| 15:00 | Report from NLP4ITA | |
| Horacio Saggion | ||
| 15:30-16:00 Tea break | ||
| Session 4: Presentations and Close | ||
| 16:00 | The CW Corpus: A New Resource for Evaluating the Identification of Complex Words | |
|
Matthew Shardlow abstract
The task of identifying complex words (CWs) is important for lexical simplification, however it is often carried out with no evaluation of success. There is no basis for comparison of current techniques and, prior to this work, there has been no standard corpus or evaluation technique for the CW identification task. This paper addresses these shortcomings with a new corpus for evaluating a system's performance in identifying CWs. Simple Wikipedia edit histories were mined for instances of single word lexical simplifications. The corpus contains 731sentences, each with one annotated CW. This paper describes the method used to produce the CW corpus and presents the results of evaluation, showing its
validity.
| ||
| 16:20 | A Pilot Study of Readability Prediction with Reading Time | |
|
Hitoshi Nishikawa, Toshiro Makino and Yoshihiro Matsuo abstract
In this paper we report the results of a pilot study of basing readability prediction on training data annotated with reading time. Although reading time is known to be a good metric for predicting readability, previous work has mainly focused on annotating the training data with subjective readability scores usually on a 1 to 5 scale. Instead of the subjective assessments of complexity, we use the more objective measure of reading time. We create and evaluate a predictor using the binary classification problem; the predictor identifies the better of two documents correctly with 68.55% accuracy. We also report a comparison of predictors based on reading time and on readability scores.
| ||
| 16:40 | Final Discussion and Close | |
CALL FOR PAPERS
Many NLP systems generate or reformulate human language but how readable is the output? What makes language easy or difficult to read for different types of readers? How can existing text be manipulated to improve information access? How does writing style affect readability, comprehension, and appreciation of text? The last few years have seen a resurgence of interest in these questions amongst computational linguists as attention turns to more sophisticated techniques for textual presentation and to address the widely differing needs of end users. The relevance of this research area has spawned a number of workshops on related topics, for example, SL-PAT 2012 (slpat.org) and NLP4ITA 2012 (www.taln.upf.edu/nlp4ita/), and a new special interest group, Speech and Language Processing for Assistive Technologies (slpat.org), which sponsors this workshop.
PITR is a cross-disciplinary workshop bringing together researchers in any field concerned with the readability, accessibility and quality of text, particularly computational linguists, psycholinguists and educational researchers. We solicit papers on:
- Reformulation of existing text (text-to-text systems)
- Generation of readable language from data (data-to-text systems)
- Generation of text in specific styles and registers for readability
- Evaluation of language simplification strategies
- Evaluation of the readability of computer-generated text
- Evaluation of the readability of machine translation output
- Prediction of aspects of text style related to readability
- Prediction of the readability of documents
- Readability issues in specialist texts such as questionnaires, exam questions, safety instructions, etc.
- Novel evaluation strategies for assessing text readability
- Novel readability metrics
- Techniques for simplifying lexis
- Techniques for simplifying syntax
- Techniques for simplifying discourse properties (making text more transparent, etc.)
- Techniques for manipulating textual layout to improve accessibility
- Techniques for making descriptions of numerical quantities more accessible
- Techniques for making technical terminology more accessible
- Techniques for making descriptions of logical statements more accessible
- Techniques for explaining complex ideas through accessible text
- Systems aimed at adults with poor literacy
- Systems aimed at children learning to read
- Systems aimed at 2nd language learners
- Systems aimed at people with language deficits (aphasia, deafness, neurodegeneration, etc.)
- Systems aimed at non-experts accessing technical material
SUBMISSIONS
Papers should prepared in ACL format not exceeding 8 pages in length plus up to 2 additional pages for references. Papers should also be anonymised for blind reviewing.
Please submit your paper via the online START Conference Manager system.
Some authors will be invited to give oral presentations. All accepted authors will be expected to present a poster. Last year, the poster session was very lively, giving poster-only authors and oral-presentation authors ample opportunities to discuss their research.
IMPORTANT DATES
May 3, 2013: Deadline for paper submission
May 29, 2013: Notification of acceptance
June 14, 2013: Camera-ready deadline
August 8, 2013: PITR 2013
ORGANISERS
Sandra Williams, The Open University, UK.
Advaith Siddharthan, University of Aberdeen, UK.
Ani Nenkova, University of Pennsylvania, USA.
PROGRAMME COMMITTEE
Julian Brooke, University of Toronto, Canada.
Kevyn Collins-Thompson, Microsoft Research (Redmond), USA.
Siobhan Devlin, University of Sunderland, UK.
Micha Elsner, University of Edinburgh, UK.
Thomas François, University of Louvain, Belgium.
Caroline Gasperin, TouchType Ltd., UK.
Albert Gatt, University of Malta, Malta.
Pablo Gervás, Universidad Complutense de Madrid, Spain.
Iryna Gurevych, Technische Universitat Darmstadt, Germany.
Raquel Hervás, Universidad Complutense de Madrid, Spain.
Véronique Hoste, University College Ghent, Belgium.
Matt Huenerfauth, The City University of New York (CUNY), USA.
Iustina Ilisei, University of Wolverhampton, UK.
Annie Louis, University of Pennsylvania, USA.
Hitoshi Nishikawa, NTT, Japan.
Ehud Reiter, University of Aberdeen, UK.
Horacio Saggion, Universitat Pompeu Fabra, Spain.
Irina Temnikova, University of Wolverhampton, UK.
Ielka van der Sluis, University of Groningen, The Netherlands.
Kristian Woodsend, University of Edinburgh, UK.
Last modified: July 2013, S.H.Williams
Keynote Speaker
Annie Louis, University of Edinburgh
Annie Louis is a Newton International Fellow at the University of
Edinburgh. She completed her PhD at University of Pennsylvania
with a thesis on text quality prediction. She has also worked on
automatic summarization and discourse parsing. She is currently
working on discourse and document-level issues in machine translation.
Annie has received a EMNLP best paper award and a SIGDIAL Best Student
paper award.