Does Generative AI make traditional NLP obsolete?

AI Discussions
1

Natural Language Processing traditionally involved using a lot of parsing, tokenization, etc., and people used LSTM’s to perform NLP. With the advent of Generative AI, is traditional NLP obsolete? Is there still a need to do manually create programs to clean and process natural language, or will Transformers and LLM’s be capable of doing everything that NLP could, by fine-tuning the LLM’s on specific datasets?

2

Hello @Nav9 ,

Traditional NLP is not obsolete with the advent of generative AI. In fact, the two approaches can be complementary. Traditional NLP techniques can be used to clean and process natural language data, which can then be used to train generative AI models. This can help to improve the accuracy and performance of generative AI models.

For example, traditional NLP techniques can be used to identify and remove noise from natural language data, such as typos, slang, and informal language. This can help to improve the accuracy of generative AI models, as they will not be trained on data that is incorrect or misleading.

Traditional NLP techniques can also be used to extract features from natural language data, such as keywords, entities, and relationships. This can help to improve the performance of generative AI models, as they will be able to generate more relevant and informative text.

Yes, finetuning LLMs increase the usecases for LLMs. However, this is not always the case. Traditional NLP techniques may be more effective than generative AI models. For example, traditional NLP techniques may be more effective at identifying and removing noise from natural language data.

Regards,
Can Koz

1 Like
3

Thanks canxkoz, but currently, LLM’s can be used to clean and process natural language data, to identify and remove noise from natural language data, such as typos, slang, and informal language. So couldn’t we just use LLM’s instead of traditional NLP techniques?
It’s something like: To build the first high-level language compiler, we needed to write the compiler in assembly level language. But once the compiler was built, we just upgraded the compiler and even wrote new compilers using the high level language. There was no longer a need to write in assembly level language.

1 Like