The Portuguese B\(^2\) SG: A Semantic Test for Distributional Thesaurus
The lack of availability of gold standards for evaluation of distributional thesauri is a stumbling block that prevents a direct comparison of alternative approaches in a uniform way. Here we present B\(^2\) SG, a TOEFL-like task for Portuguese that conta
- PDF / 143,847 Bytes
- 7 Pages / 439.37 x 666.142 pts Page_size
- 58 Downloads / 160 Views
Abstract. The lack of availability of gold standards for evaluation of distributional thesauri is a stumbling block that prevents a direct comparison of alternative approaches in a uniform way. Here we present B2 SG, a TOEFL-like task for Portuguese that contains 2,875 tests with semantic relations (synonyms, antonyms and hypernyms) for nouns and verbs. The resource is validated by comparing it with lexical resources and by human judgment. The resource was used for evaluating two distributional thesauri: one built from lemmata and the other from surface forms. The evaluation of thesauri demonstrated that the use of lemmata is slightly more accurate than the use surface forms for building distributional thesauri. B2 SG is readily available for download (http://www. inf.ufrgs.br/pln/resource/B2SG.zip). Keywords: Gold standard · Semantic relations · Distributional thesauri
1
Introduction
The importance of resources such as WordNet [5] can be measured by the number of initiatives dedicated to (re)produce them in other languages, such as the EuroWordNet [15] and the Global WordNet Association [3]. For Portuguese, such initiatives include Onto.PT [7], OpenWN-PT [13], WordNet.PT [10], WordNet.Br [4]. Manual construction of such resources is costly and time-consuming, and normally ends up with low coverage. An alternative is the automatic construction of distributional thesauri, that present semantic similarity between words and are both language independent and applicable to any domain [9]. The automatic evaluation of such thesauri is a complex task, because of the lack of resources with information on the similarity of words. Moreover, due to the large scale of the resulting thesauri the evaluation cannot be done manually by judges as it would consume too much time. An alternative is the evaluation of thesauri based on their performance on a particular task. For instance, we can approximate the concept of similarity by presenting an explicit semantic relation between words, such as the TOEFL [8] and the WordNet-Based Synonymy Test (WBST) [6] for English. For Portuguese, there are no specific gold standards for the evaluation of distributional thesauri. Here we present the BabeNet-Based c Springer International Publishing Switzerland 2016 J. Silva et al. (Eds.): PROPOR 2016, LNAI 9727, pp. 333–339, 2016. DOI: 10.1007/978-3-319-41552-9 33
334
R. Wilkens et al.
Semantic Gold Standard (B 2 SG)1 , that builds upon the WBST, and we also use this resource as a means to evaluate two distributional thesauri. We discuss the methodology used for developing and validating B 2 SG in Sect. 2. The creation and evaluation of distributional thesauri is then described in Sects. 3 and 4. Conclusions and future work are presented in Sect. 5.
2
Constructing B2 SG
The BabelNet-Based Semantic Gold Standard (B2 SG) contains nouns and verbs involving antonym, hypernym and synonym relations. Similar to Toefl [8] and WBST [6], for each target word the B2 SG lists 4 alternatives: one semantically related word, and 3 unrelated words. For i
Data Loading...
