A Reservoir Computing Approach to Word Sense Disambiguation
- PDF / 947,778 Bytes
- 10 Pages / 595.224 x 790.955 pts Page_size
- 78 Downloads / 204 Views
A Reservoir Computing Approach to Word Sense Disambiguation Kiril Simov1 · Petia Koprinkova-Hristova1
· Alexander Popov1 · Petya Osenova1
Received: 11 February 2020 / Accepted: 23 July 2020 © Springer Science+Business Media, LLC, part of Springer Nature 2020
Abstract Reservoir computing (RC) has emerged as an alternative approach for the development of fast trainable recurrent neural networks (RNNs). It is considered to be biologically plausible due to the similarity between randomly designed artificial reservoir structures and cortical structures in the brain. The paper continues our previous research on the application of a member of the family of RC approaches—the echo state network (ESN)—to the natural language processing (NLP) task of Word Sense Disambiguation (WSD). A novel deep bi-directional ESN (DBiESN) structure is proposed, as well as a novel approach for exploiting reservoirs’ steady states. The models also make use of ESN-enhanced word embeddings. The paper demonstrates that our DBiESN approach offers a good alternative to previously tested BiESN models in the context of the word sense disambiguation task having smaller number of trainable parameters. Although our DBiESN-based model achieves similar accuracy to other popular RNN architectures, we could not outperform the state of the art. However, due to the smaller number of trainable parameters in the reservoir models, in contrast to fully trainable RNNs, it is to be expected that they would have better generalization properties as well as higher potential to increase their accuracy, which should justify further exploration of such architectures. Keywords Reservoir computing · Echo state network · Word sense disambiguation · Word embeddings
Introduction In the area of natural language processing (NLP), RNNs are considered a viable tool for linguistic modeling, due to their ability to keep memory traces of the context (preceding and/or succeeding text) of a given word at theoretically
This article belongs to the Topical Collection: Trends in Reservoir Computing Guest Editors: Claudio Gallicchio, Alessio Micheli, Simone Scardapane, Miguel C. Soriano Petia Koprinkova-Hristova
[email protected]; [email protected] Kiril Simov [email protected] Alexander Popov [email protected] Petya Osenova [email protected] 1
Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Sofia, Bulgaria
arbitrary distances from it. That is why BiLSTMs have been successfully applied to a number of sequence-to-sequence tasks in NLP, such as part-of-speech tagging, chunking, named entity recognition, dependency parsing [1–4]). Word sense disambiguation (WSD) is an NLP task aimed at assigning proper categories of meaning to words that are ambiguous (i.e., they can assume several related or unrelated meanings, depending on the context). For instance, the word “chair” can refer to a person who is managing some activity (“The chair gave the word to the next participant in the meeting.”), or to a piece of furniture (“The st
Data Loading...
