Multi-Granular BERT: An Interpretable Model Applicable to Internet-of-Thing devices

Abstract

With the development of the Energy Internet (EI), its applications have gradually spread from industrial uses to smart homes. Specifically, home Internet of Things(IoT) devices have become popular in the field of smart homes. In this paper, we propose an interpretable model that can be applied on the IoT devices. When Chinese characters are grouped into words, the meaning may vary. Inspired by the observation, we convert character-level Bi-directional Transformer (BERT) to word-level, which we call it multi-granular BERT (MLGB). It constructs the n-gram representation of different lengths within a model. It also learns the self-Attention between n-grams during both pre-Training and task-specific fine-Tuning to learn both the word representation and word-word self-Attention at the same time. As a diagnostic task, we evaluate our model on two Chinese text pair classification tasks and observe the model's behavior. The MLGB retains the BERT's accuracy on the tasks while demonstrates more interpretable word-level self-Attention. Multi-granularity may also have served as a regularization of attention that alleviates the non-identifiability issue of self-Attention.