Letter #18 - In which we pretend to understand the words in front of us.
We are what we pretend to be, so we must be careful about what we pretend to be.
Honey you are a rock
Upon which I stand
And I come here to talk
I hope you understand
Back when they were young. Listen to the whole album.
Among the biggest benefits of language modelling is that training data comes for free with any text corpus and that potentially unlimited amounts of training data are available. This is particularly significant, as NLP deals not only with the English language. More than 4,500 languages are spoken around the world by more than 1,000 speakers. Language modeling as a pretraining task opens the door to developing models for previously underserved languages.
…The time is ripe for practical transfer learning to make inroads into NLP. In light of the impressive empirical results of ELMo , ULMFiT , and OpenAI it only seems to be a question of time until pretrained word embeddings will be dethroned and replaced by pretrained language models in the toolbox of every NLP practitioner. This will likely open many new applications for NLP in settings with limited amounts of labeled data.
I’ve always found it frustrating that most NLP research is applied to a handful of languages, as only those have enough labeled data. Being able to take a pretrained language model, tweak it for your specific use case and use it just like you would with a CV model is quite exciting.
Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.
Also by Sebastian Ruder, it’s something you should keep in mind if you’re doing any NLP work.
Not sure what you just read? Take a look at this post.
Enjoyed it? Subscribe to my tiny newsletter and get new issues straight to your inbox. Every week, on Tuesdays.