On the basis of improving the Bidirectional Encoder Representations from Transformers (BERT) pre-training task, we add in-depth pre-training tasks, and based on the optimisation results of the LDA ...