Intrinsic and Extrinsic Evaluation of the Quality of Biomedical Embeddings in Different Languages

Abstract

Lately, language models have been applied to several tasks in biomedical natural language processing. Some public language models are available online, each built with different corpora. In this paper, we evaluate different public word embedding models trained with both general and biomedical corpora for English and Portuguese. We present intrinsic evaluations based on semantic analogies that use word pairs extracted from the MeSH biomedical thesaurus and also from benchmarks that are available for general-domain evaluation. For extrinsic evaluations we rely on a classification task over Eletronic Health Records. Our experiments show that biomedical embeddings can better capture semantics for biomedical analogies in both languages. On the other hand for extrinsic evaluation, based on classification tasks using the language models, larger general textual corpora appeared equally or more effective.