NAACL HLT 2012 Workshop
Predicting and improving text readability for target reader populations (PITR2012)
June 7, 2012
Predicting and Improving Text Readability for Target Reader Populations Workshop will be be held in conjunction with the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL HLT) 2012 conference, 4 - 8 June, 2012, at Le Centre Sheraton Montréal
1201, boul. René-Lévesque ouest, Montréal, (Québec), Canada, H3B-2L7.
This workshop is an ACL Special Interest Group on Speech and Language Processing for Assistive Technologies (SIG-SLPAT) sponsored workshop.
NEWS!!
Keynote Speaker
Enriching the Web with Readability Metadata
Dr Kevyn Collins-Thompson, Microsoft Research, Redmond, WA.
abstract
Little is currently known about the nature of the Web, its users, and how users interact with content when seen through the lens of text readability. For example, a document isn't relevant to a person's information need - at least, not immediately - if they can't understand it, yet Web search engines have traditionally ignored the problem of finding or providing content at the right level of difficulty as an aspect of relevance. I'll show how computing and applying metadata based on text readability at Web scale - especially in combination with topic metadata - opens up new and sometimes surprising possibilities for enriching our interactions with the Web, from personalizing Web search results to predicting user and site expertise to estimating searcher motivation. I'll also discuss future challenges and opportunities in predicting and improving text readability, particularly in light of the rapidly growing interest in large-scale applications for online education.
Accepted Papers
Toward Determining the Comprehensibility of Machine Translations.
Tucker Maney, Linda Sibert, Dennis Perzanowski, Kalyan Gupta and Astrid Schmidt-Nielsen
abstract
Economic globalization and the needs of the intelligence community have brought machine translation into the forefront. Human translators cannot meet the need for high quality translations or “good enough” translations that suffice only to enable understanding. Much research has been done in creating translation systems and in evaluating these systems. Metrics for the latter have primarily focused on improving the overall quality of entire test sets but not on assessing the understanding of individual sentences or paragraphs. Therefore, we have focused on developing a theory of translation effectiveness by isolating a set of translation variables and measuring their effects on comprehension and readability of translations. In the following study, we focus on investigating how certain linguistic permutations, omissions, and insertions contribute to the understanding of translated texts.
Towards Automatic Lexical Simplification in Spanish: An Empirical Study.
Biljana Drndarevic and Horacio Saggion
abstract
In this paper we present the results of the analysis of a parallel corpus of original and simplified texts in Spanish, gathered for the purpose of developing an automatic simplification system for this language. The system is intended for individuals with cognitive disabilities who experiment difficulties reading and interpreting informative texts. We here concentrate on lexical simplification operations applied by human editors in the process of simplification of texts that make up our corpus, on the basis of which we derive a set of rules to be implemented automatically. For the time being we have addressed the issue of lexical units substitution, including rules concerning reporting verbs and adjectives of nationality; insertion of definitions; simplification of numerical expressions; and simplification of named entities. We have also documented instances of human simplifications which are either impossible to implement computationally or require additional resources currently unavailable for Spanish. Our future work will concentrate on crystallising the rules we have derived so far and implementing them on richer data.
Building Readability Lexicons with Unannotated Corpora.
Julian Brooke, Vivian Tsang, David Jacob, Fraser Shein and Graeme Hirst
abstract
Lexicons of word difficulty are useful for various educational applications, including readability classification and text simplification. In this work, we explore automatic creation of these lexicons using methods which go beyond simple term frequency, but without relying on age-graded texts. In particular, we derive information for each word type from the readability of the web documents they appear in and the words they co-occur with, linearly combining these various features. We show the efficacy of this approach by comparing our lexicon with an existing coarse-grained, low-coverage resource and a new crowdsourced annotation.
Making Readability Indices Readable.
Sara Tonelli, Ke Tran Manh and Emanuele Pianta
abstract
Although many approaches have been presented to compute and predict readability of documents in different languages, the information provided by readability systems often fail to show in a clear and understandable way how difficult a document is and which aspects contribute to content readability. We address this issue by presenting a system that, for a given document in Italian, provides not only a list of readability indices inspired by Coh-Metrix, but also a graphical representation of the difficulty of the text compared to the three levels of Italian compulsory education, namely elementary, middle and high-school level. We believe that this kind of representation makes readability assessment more intuitive, especially for educators who may not be familiar with readability predictions via supervised classification. In addition, we present the first available system for readability assessment of Italian inspired by Coh-Metrix.
The Contribution of NLP and Machine Learning to Readability Studies.
Thomas François and Eleni Miltsakaki
abstract
Readability formulas are methods that match readers with texts, suiting their reading abilities. Several methodological paradigms have previously been investigated in the field. The most popular of them is the classic approach, which gave birth, among others, to the popular Flesch formula. This paper compares the classic approach with an emerging paradigm, based on NLP-enable features and machine learning algorithms. Our experiments, carried on a corpus of texts for French as a foreign language, yielded four main results: (1) the new readability formula performed better than the classic formula; (2) ``non classic'' features were slightly more informative than ``classic'' features; (3) modern machine learning algorithms did not improve the explicative power of our readability model, but allowed to better classify new observations; and (4) combining ``classic'' and ``non classic'' features resulted in a significant gain in performance.
Offline Sentence Processing Measures for testing Readability with Users.
Advaith Siddharthan and Napoleon Katsos
abstract
While there has been much work on computational models to predict readability based on the lexical, syntactic and discourse properties of a text, there are also interesting open questions about how computer generated text should be evaluated with target populations. In this paper, we compare two offline methods for evaluating sentence quality, magnitude estimation of acceptability judgements and sentence recall. These methods differ in the extent to which they can differentiate between surface level fluency and deeper comprehension issues. We find, most importantly, that the two correlate. Magnitude estimation can be run on the web without supervision, and the results can be analysed automatically. The sentence recall methodology is more resource intensive, but allows us to tease apart the fluency and comprehension issues that arise.
Graphical Schemes May Improve Readability but not Understandibility for People with Dyslexia.
Luz Rello, Ricardo Baeza-Yates, Horacio Saggion and Eduardo Graells
abstract
Generally, people with dyslexia are poor readers but strong visual thinkers. The use of schemes for helping text comprehension is recommended in education manuals. This study explores the relation between text readability and the conceptual schemes which aim to make the text more transparent for these specific target readers. Our results are based on a user study with a group of twenty three dyslexic users and a control group. The data collected from our study combines qualitative data from questionnaires and quantitative data from tests carried out using eye tracking. The findings contribute to the assessment of readability for dyslexics and suggest that schemes may help to improve readability for dyslexics but are counterproductive for understandability.
Comparing human versus automatic feature extraction for fine-grained elementary readability assessment.
Yi Ma, Ritu Singh, Eric Fosler-Lussier and Robert Lofthus
abstract
Early primary children's literature poses some interesting challenges for automated readability assessment: for example, teachers often use fine-grained reading leveling systems for determining appropriate books for children to read (many current systems approach readability assessment at a coarser whole grade level). Ma et al. (2012) suggest that the fine-grained assessment task can be approached using a ranking methodology, and incorporating features that correspond to the visual layout of the page improves performance. However, their methodology for using ``found'' text (e.g., scanning in a book from the library) requires human annotation of the text regions and correction of the OCR text. In this work, we ask whether the annotation process can be automated, and also experiment with richer syntactic features found in the literature that can be automatically derived from either the human-corrected or raw OCR text. We find that automated visual and text feature extraction work reasonably well and can allow for scaling to larger datasets, but that in our particular experiments the use of syntactic features adds little to the performance of the system, contrary to previous findings.
WORKSHOP DESCRIPTION
How readable is the output of systems that generate or reformulate language? What makes language easy or difficult to read for different types of readers? How can existing text be reformulated to improve information access?
The last few years have seen a resurgence of work on text simplification and readability. Examples include learning lexical and syntactic simplification operations from Simple English Wikipedia revision histories (e.g., Zhu et al. 2010, Woodsend and Lapata 2011, Biran et al. 2011), exploring complex lexico-syntactic simplification requiring morphological changes as well as constituent reordering (Siddharthan 2010, 2011), simplifying mathematical form (Power and Williams, 2012), applications to second language learners (Peterson 2007) and low literacy adults (e.g., Gasperin et al. 2010), attempts to measure linguistic quality (Pitler et al. 2010, Nenkova et al. 2010), analyses of the use of text modification for deaf students (e.g., O'Neill 2005), and NLG research on summarising technical data for lay people (e.g., Mahamood and Reiter 2011).
This will be a cross-disciplinary workshop bringing together researchers in computational linguistics, psycholinguistics and education with an interest in text reformulation, generation of texts at different levels of difficulty, and readability measures. We solicit papers on reformulation (text-to-text), generation of readable language from data (data-to-text), user evaluations of language simplification strategies, and studies on the readability of text. We would like contributions on how to simplify:
- Lexis and Syntax
- Numerical quantities and logical relations
- Discourse Properties (making text more transparent, etc.)
We are particularly interested in research aimed at assessing the readability of machine-generated text, simplifying texts, and assessing the accessibility of texts for specific target readers such as:
- Adults with poor literacy
- 2nd language learners
- People with language deficits (Aphasia, Deafness, Neurodegeneration, etc.)
- Lay readers accessing technical material
IMPORTANT DATES
Mar 25, 2012: Deadline for paper submission
Apr 24, 2012: Notification of acceptance
May 4, 2012: Camera-ready deadline
Jun 7, 2012: Workshop date
CAMERA-READY SUBMISSION
Authors should revise their papers, check that they are in ACL style, and submit camera-ready versions at the submission site.
ACCOMMODATION
Please note: the workshop may overlap with people arriving for the Montréal Grand Prix Formula 1 race (Jun 10). This could make hotel space a bit tight, so if you are going to the workshop, you might want to book early.
Accommodation details are on the
main conference site
ORGANISERS
Sandra Williams, The Open University, UK.
Advaith Siddharthan, University of Aberdeen, UK.
Ani Nenkova, University of Pennsylvania, USA.
PROGRAMME COMMITTEE
Gregory Aist, Iowa State University, USA.
John Carroll, University of Sussex, UK.
Kevyn Collins-Thompson, Microsoft Research (Redmond), USA.
Siobhan Devlin, University of Sunderland, UK.
Noémie Elhadad, Columbia University, USA.
Micha Elsner, University of Edinburgh, UK.
Richard Evans, University of Wolverhampton, UK.
Lijun Feng, Columbia University, USA.
Caroline Gasperin, TouchType Ltd., UK.
Albert Gatt, University of Malta, Malta.
Pablo Gervás, Universidad Complutense de Madrid, Spain.
Iryna Gurevych, Technische Universitat Darmstadt, Germany.
Raquel Hervás, Universidad Complutense de Madrid, Spain.
Véronique Hoste, University College Ghent, Belgium.
Matt Huenerfauth, The City University of New York (CUNY), USA.
Iustina Ilisei, University of Wolverhampton, UK.
Tapas Kanungo, Microsoft, USA.
Mirella Lapata, University of Edinburgh, UK.
Annie Louis, University of Pennsylvania, USA.
Ruslan Mitkov, University of Wolverhampton, UK.
Hitoshi Nishikawa, NTT, Japan.
Mari Ostendorf, University of Washington, USA.
Ehud Reiter, University of Aberdeen, UK.
Lucia Specia, University of Wolverhampton, UK.
Irina Temnikova, University of Wolverhampton, UK.
Ielka van der Sluis, University of Groningen, The Netherlands.
References
Biran, O., S. Brody, and N. Elhadad. 2011. Putting it Simply: a Context-Aware Approach to Lexical Simplification. ACL 2011.
Gasperin, C., E. Maziero, & S. Alusio. 2010. Challenging choices for text simplification. Proc. Computational Processing of the Portuguese Language.
Mahamood, S. and E. Reiter. 2011. Generating Affective Natural Language for Parents of Neonatal Infants. Proc. ENLG 2011.
Nenkova, A., J. Chae, A. Louis and E. Pitler. 2010. Structural Features for Predicting the Linguistic Quality of Text: Applications to Machine Translation, Automatic Summarization and Human-Authored Text. In E Krahmer and M Theune, editors, Empirical Methods in Natural Language Generation: Data-oriented Methods and Empirical Evaluation.
O'Neill, R., 2005. Should we modify English language for deaf learners?
International Perspectives on Language Support, Cheltenham: Direct Learn Services Ltd (e-book)
Petersen, S. 2007. Natural language processing tools for reading level assessment and text simplification for bilingual education. Ph.D. thesis,
University of Washington, Seattle, WA.
Pitler, E., A. Louis and A. Nenkova. 2010. Automatic Evaluation of Linguistic Quality in Multi-Document Summarization. Proc. ACL 2010.
Power, R. and S. Williams. 2012. Generating numerical approximations. Computational Linguistics, Volume 38, No.1.
Siddharthan, A. 2010. Complex lexico-syntactic reformulation of sentences using typed dependency representations. Proc. INLG 2010.
Siddharthan, A. 2011. Text Simplification using Typed Dependencies: A Comparision of Different Generation Strategies. Proc. ENLG 2011.
Woodsend, K., & M. Lapata. 2011. Learning to Simplify Sentences with Quasi-Synchronous Grammar and Int. Programming. Proc. EMNLP 2011.
Zhu, Z., D. Bernhard, & I. Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. Proc. COLING 2010.
Last modified: April 2012, S.H.Williams