Automatic Text Simplification. Horacio Saggion

Чтение книги онлайн.

СКАЧАТЬ which contains articles from the Weekly Reader (see above) and articles from the BBCBitesize website. As in previous work, the model contains a number of different groups of features accounting for lexical and POS tag distribution information, superficial characteristics (e.g., word length) and classical readability indices (e.g., Flesch-Kincaid), age-of-acquisition word information, word ambiguity, etc., 10-fold cross-validation evaluation, using correlation and means error rate metrics, is carried out as is validation on available standard datasets from the Common Core Standards corpus⁷ (168 documents), the TASA corpus (see Vajjala and Meurers [2014] for details) (37K documents), and the Math Readability corpus⁸ (120 web pages). The model achieves high correlation in cross-validation and reasonable correlation across datasets, except in the Math corpus probably because of the rating scale used. The approach also compares very favorably with respect to several proprietary systems. Where sentence readability is concerned, the model trained on the WeeBit corpus is applied to sentences from the OneStopEnglish corpus,⁹ a dataset in which original documents (30 articles, advanced level) have been edited to obtain documents at intermediate and beginner reading levels. Experiments are first undertaken to assess whether the model is able to separate the three different types of documents, and then to evaluate a sentence readability model. To evaluate sentence readability, each pair of parallel documents (advanced-intermediate, intermediate-beginner, advanced-beginner) is manually sentence-aligned and experiments are carried out to test whether the model is able to preserve the relative readability order of the aligned sentences (e.g., advanced-level sentence less readable than beginner-level sentence). Overall, the model preserves the readability order in 60% of the cases.

2.7 READABILITY AND AUTISM

      Yaneva et al. [2016a,b] study text and Web accessibility for people with ASD. They developed a small corpus composed of 27 documents evaluated by 27 people diagnosed with an ASD. The novelty of the corpus is that, in addition to induced readability levels, it also contains gaze data obtained from eye-tracking experiments in which ASD subjects (and a control group of non-ASD subjects) were measured reading the texts, after which they were asked multiple-choice text-comprehension questions. The overall difficulty of the texts was obtained from quantitative data relating to answers given to those comprehension questions. Per each text, correct and incorrect answers were counted and text ranked based on number of correct answers. The ranking provided a way to separate texts into three difficulty levels: easy, medium, and difficult. The corpus itself was not used to develop a readability model; instead, it was used as test data for a readability model trained on the WeeBit corpus (see previous section), which was transformed into a 3-way labeled dataset (only 3 difficulty levels were extracted from WeeBit to comply with the ASD corpus).

      Yaneva et al. grouped sets of features according to the different types of phenomena that account for: (i) lexico-semantic information such as word characteristics (length, syllables, etc.), numerical expressions, passive verbs, etc.; (ii) superficial syntactic information such as sentence length or punctuation information; (iii) cohesion information such as occurrence of pronouns and definite descriptions, etc.; (iv) cognitively motivated information including word frequency, age of acquisition of words, word imagability, etc.; and (v) information arising from several readability indices such as the Flesch-Kincaid Grade Level and the FOG readability index, etc. Two decision-tree algorithms, random forest [Breiman, 2001] and reduced error pruning tree (see [Hall et al., 2009]), were trained on the WeeBit corpus (see previous section) and cross-validated in WeeBit and tested in the ASD corpus. Feature optimization was carried out using a best-first feature selection strategy which identified such features as polysemy, FOG index, incidence of pronouns, sentence length, age of acquisition, etc. The feature selection procedure yields a model with improved performance on training and test data; nonetheless, results of the test on the ASD corpus are not optimal when compared with the cross-validation results on WeeBit. Worth noting is that although some of the features selected might be representative of problems ASD subjects may encounter when reading text, these features emerged from a corpus (WeeBit) that is not ASD-specific, suggesting that the selected features model general text difficulty assessment.

      Based on the ASD corpus, a sentence readability assessment dataset was prepared composed of 257 sentences. Sentences were classified into easy-to-read and difficult based on the eye-tracker data associated with the texts. Sentences were ranked based on the average number of fixations they had during the readability assessment experiments and the set of sentences split in two parts to yield the two sentence readability classes. To complement the sentences from ASD and to control for length, short sentences from publicly available sources [Laufer and Nation, 1999] were added to the dataset. The labels for these sentences were obtained through a comprehension questionnaire which subjects with ASD had to answer. Sentences were considered easy to read if at least 60% of the subjects answered correctly the comprehension question associated with the sentence. Binary classification experiments on this dataset were performed using the Pegasos algorithm [Shalev-Shwartz et al., 2007] with features to model superficial sentence characteristics (number of words, word length, etc.), cohesion (proportion of connectives, causal expressions, etc.), cognitive indicators (word concreteness, imagability, polysemy, etc.), and several incidence counts (negation, pronouns, etc.). A cross-validation experiment achieved 0.82 F-score using a best-first feature selection strategy.

2.8 CONCLUSION

      Over the years researchers have tried to come up with models able to predict the difficulty of a given text. Research on readability assessment is important for automatic text simplification in that models of readability assessment can help identify texts or text fragments which would need some sort of adaptation in order to made them accessible for a specific audience. Readability assessment can also help developers in the evaluation of automatic text simplification systems. Although traditional formulas which rely on simple superficial proxies are still used, in recent years, the availability of sophisticated natural language processing tools and better understanding of text properties accounting for text quality, cohesion, and coherence have fueled research in readability assessment, notably in computational linguistics.

      This chapter covered several aspects of the readability assessment problem including reviewing classical readability formulas, presenting several advanced computational approaches based on machine learning techniques and sophisticated linguistic features, and pointing out current interest for readability for specific target populations as well as for texts of peculiar characteristics such as web pages.

2.9 FURTHER READING

      Readability formulas and studies have been proposed for many different languages. For Basque, an agglutinative language with rich morphology, Gonzalez-Dios et al. [2014] recently proposed using a number of Basque-specific features to separate documents with two different readability levels, achieving over 90% accuracy (note that is only binary classification). A readability formula developed for Swedish, the Lix index, which uses word length and sentence length as difficulty factors, has been used in many other languages [Anderson, 1981]. There has been considerable research on readability in Spanish [Anula Rebollo, 2008, Rodríguez Diéguez et al., 1993, Spaulding, 1956] and its application to automatic text simplification evaluation [Štajner and Saggion, 2013a].

       ¹ http://www.weeklyreader.com

       ² http://literacynet.org/

       ³ https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html

       ⁴ http://www.helsinki.fi/varieng/CoRD/corpora/FLOB/

       ⁵ http://www.dueparole.it/default_.asp

       ⁶ http://www.repubblica.it/

       ⁷ http://www.corestandards.org/ELA-Literacy/

       СКАЧАТЬ