TOWARDS CROSS-ATTENTION PRE-TRAINING IN NEURAL MACHINE TRANSLATION

Chỉ số đề mục

Lĩnh vực nghiên cứu

Dạng tài liệu

BB

Tác giả

Pham Vinh Khang, Nguyen Hong Buu Long, Nguyễn Hồng Bửu Long⁽¹⁾

Nhan đề

TOWARDS CROSS-ATTENTION PRE-TRAINING IN NEURAL MACHINE TRANSLATION

Nhan đề tiếng anh

Nguồn trích

Tạp chí Khoa học - Trường Đại học Sư phạm TP Hồ Chí Minh

Năm xuất bản

2022

Số

10

Trang

1749

ISSN

Từ khóa

Từ khóa tiếng anh

Tóm tắt

The advent of pre-train techniques and large language models has significantly leveraged the performance of many natural language processing (NLP) tasks. However, pre-trained language models for neural machine translation remain a challenge as little information about the interaction of the language pair is learned. In this paper, we explore several studies trying to define a training scheme to pre-train the cross-attention module between the encoder and the decoder by using the large-scale monolingual corpora independently. The experiments show promising results, proving the effectiveness of using pre-trained language models in neural machine translation.

Tóm tắt tiếng anh

Kí hiệu kho

File toàn văn

Xem toàn văn