Author

Hiroki Sakaji, Masahiro Suzuki, Kiyoshi Izumi, and Hiroyuki Mitsugi

Conference

2022 IEEE International Conference on Big Data (Big Data)

Abstract

We propose a new pre-trained architecture for economics/finance domain adaptation of language models in this paper. Pre-trained language models have become commonplace in a wide range of language processing applications. Because they learn from generic documents such as Wikipedia, many pre-trained language models are not fully adapted to the domain. As a result, there are two approaches: one is to develop a domain-specific pre-trained language model, and the other is to adapt the model learned on general documents to the domain through further pre-training with domain texts. However, no definitive better method has been discovered, and each project is working on it in different ways. As a result, this study focuses on the Japanese language’s economics/financial domains and investigates how pre-trained language models can be better adapted to domain-specific tasks.

Paper

PDF

Bibtex

@inproceedings{Sakaji-2022-bigdata,
  title = {{Gradual Further Pre-training Architecture for Economics/Finance Domain Adaptation of Language Model}},
  author = {Sakaji, Hiroki and Suzuki, Masahiro and Izumi, Kiyoshi and Mitsugi, Hiroyuki},
  booktitle = {2022 IEEE International Conference on Big Data (Big Data)},
  year = {2022},
  doi = {10.1109/BigData55660.2022.10020445},
  pages = {2352-2355},
}