Variations of the CorDeGen+ Method for the Languages of Northern European Countries

This study is devoted to the problem of generating text corpora for their use during the development and testing of natural language processing information systems. The CorDeGen and CorDeGen+ methods are among the approaches that address this problem. However, as shown in this paper, the application of these methods to the development and testing of information systems for processing texts in “regional” languages (less widely spoken than English) has not yet been considered, despite its challenges. In this study, the languages of Northern Europe are considered as such “regional” languages, and the issue of removing part of the terms (if they coincide with the stop words of these languages) from the generated corpora during preprocessing is solved. To address this issue, the paper proposes seven new language variations of the CorDeGen+ method, specifically for Lithuanian, Danish, Swedish, Norwegian, Northern Sami, Lule Sami, and Icelandic languages. Latvian, Estonian, Finnish, and Southern Sami languages are also considered in this study, and the results show that the use of the CorDeGen+(0-9) variation, already described in the literature, is sufficient for them. The experimental verification of the effect of removing part of the terms showed that the use of the proposed language variations and CorDeGen+(0-9) variation prevents the removal of 20–43% of all terms from the generated corpus during preprocessing.