Creating a costome tockenizer for romanized pali text

[Vivek_anil]

How to start creating a custom nlp tokenizer for the romanized pali text?

[gent.spah]

Not very knowledgable in this but I think this article could be of help to you:

Tokenization (Building a Tokenizer and a Sentencizer)