They do work, but transformers work better on the accuracy/compute tradeoff curve. Transformers let far away positions in the sequence exchange info in 1 step, while a dilated CNN will need many layers. Additionally, Transformers can be fairly efficiently trained.
I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.
est · · focus · HN ↗
Or is it?
DevelopingElk · · focus · HN ↗
I do think if Transformers weren't invented you would still be able to train powerful language models, but they would take more ram and be slower so nobody does this.