Mask-Predict

下载模型

预处理

text=PATH_YOUR_DATA

output_dir=PATH_YOUR_OUTPUT

src=source_language

tgt=target_language

model_path=PATH_TO_MASKPREDICT_MODEL_DIR

python preprocess.py --source-lang $src--target-lang${tgt} --trainpref $text/train--validpref$text/valid --testpref $text/test--destdir${output_dir}/data-bin --workers 60 --srcdict $\text{Missing superscript or subscript argument}${src}$tgt/dict.${src}.txt --tgtdict ${model_path}/maskPredict$\text{Missing superscript or subscript argument}${tgt}/dict.${tgt}.txt

训练

model_dir=PLACE_TO_SAVE_YOUR_MODEL

python train.py $outpu{t}_{d}ir/data-bin--archber{t}_{t}ransforme{r}_{s}eq2seq--share-all-embeddings--criterionlabe{l}_{s}moothe{d}_{l}engt{h}_{c}ros{s}_{e}ntropy--label-smoothing0.1--lr5e-4--warmup-init-lr1e-7--min-lr1e-9--lr-schedulerinvers{e}_{s}qrt--warmup-updates10000--optimizeradam--adam-beta{s}^{\prime }(0.9,0.999{)}^{\prime }--adam-eps1e-6--tasktranslatio{n}_{s}elf--max-tokens8192--weight-decay0.01--dropout0.3--encoder-layers6--encoder-embed-dim512--decoder-layers6--decoder-embed-dim512--fp16--max-source-positions10000--max-target-positions10000--max-update300000--seed0--save-dir${model_dir}

评估

python generate_cmlm.py $outpu{t}_{d}ir/data-bin--path${model_dir}/checkpoint_best_average.pt --task translation_self --remove-bpe --max-sentences 20 --decoding-iterations 10 --decoding-strategy mask_predict

许可证

MASK-PREDICT 是 CC-BY-NC 4.0。该许可证适用于预训练模型。

引用

请引用如下

@inproceedings{ghazvininejad2019MaskPredict,
  title = {Mask-Predict: Parallel Decoding of Conditional Masked Language Models},
  author = {Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
}