Processing Persian text with AI: preparing the data before the model

Most AI tools are built and tested mostly in English. They work in Persian too, but Persian text often arrives in forms that can affect retrieval, counting and model inputs: two words that look identical on screen are different strings to a computer. Most of the work of processing Persian well happens before any model sees the text, and it is the same work whether the goal is search, classification, a chatbot or analytics.

Letters that look the same but are not

Persian is written in an extended Arabic script, and two of its most common letters have corresponding Arabic forms:

• Persian ی (U+06CC) and Arabic ي (U+064A)
• Persian ک (U+06A9) and Arabic ك (U+0643)

They come from Arabic keyboard layouts, older systems and text copied from elsewhere. In the middle of a word they look the same, so a reader never notices, but for a computer «کتاب» written with the Arabic kaf is a different word, and a search for one does not find the other.

Other variants need decisions too: diacritics (vowel marks and tanwin), the forms of hamza, and Arabic letters such as ة that appear in loanwords. The usual approach is to normalise: pick one form for each and convert everything to it, in the documents and in every query, in exactly the same way. One caution: آ is a letter of its own in Persian, not a decorated ا, and should not be merged with it.

Text extracted from PDFs sometimes contains Arabic presentation forms, separate character codes for each letter's shape at the start, middle or end of a word. They look right and match nothing. Unicode's NFKC normalisation turns them back into ordinary letters.

The half-space

Persian uses the zero-width non-joiner (ZWNJ, U+200C), the "half-space", inside words: between «می» and the verb in «می‌خواهم», between a noun and the plural «ها» in «کتاب‌ها». People type it as a half-space, as a full space, or not at all, so the same word appears in three spellings:

• «می‌خواهم», with the half-space
• «می خواهم», with a space, which splits it into two "words"
• «میخواهم», with the parts joined

Anything that splits text on spaces (word counts, keyword matching, many simple tokenizers) then counts and matches wrongly. Normalisers for Persian repair the common cases: the prefixes «می» and «نمی», the plural «ها», and suffixes such as «تر» and «ترین».

Three sets of digits

Numbers can arrive in Persian digits (۰ to ۹, U+06F0 to U+06F9), Arabic-Indic digits (٠ to ٩, U+0660 to U+0669) or Western digits (0 to 9), sometimes mixed in a single phone number. For processing, convert all of them to one set so that prices, dates and phone numbers can be compared, searched and validated. For display, show them as your readers expect: on a Persian page, Persian digits.

Dates need the same care. Persian documents usually use the Solar Hijri calendar, so a date has to be converted before it can be compared with a Gregorian one, and a format such as «۱۴۰۵/۰۷/۲۹» has to be recognised as a date at all.

How people actually write

Formal written Persian and the Persian people type every day differ a lot. Chats, comments, reviews and support messages are full of colloquial spelling («میخوام» for «می‌خواهم», «رو» for «را»), and some people write Persian in Latin letters, often called Finglish ("salam, chetori?"). A model or a dictionary built on formal text handles these less well. If your data comes from people rather than publishers, test on samples of it and decide which forms to map to which.

Search in Persian

Good Persian search does all of the above to both the documents and the query, and then deals with how Persian builds words:

• plurals («کتاب‌ها», «دانشجویان») and suffixes attached to nouns
• verb forms built from a stem with prefixes and endings («می‌روم», «نرفتند»)
• loanwords with more than one accepted spelling

A stemmer cuts words down to a common root; a lemmatizer maps a word to its dictionary form, which works better for verbs. An open-source Python library for Persian, Hazm, includes a normaliser, tokenizers, a stemmer and a lemmatizer, and is a reasonable place to start.

And the language model

Modern language models read and write Persian, and for many tasks they work well. Three things are worth knowing:

• Cost and length: the same text usually takes more tokens in Persian than in English, and models are priced and limited in tokens (we explain tokens in What is a large language model, and what are its limitations?).
• Normalise before you index. In a system that answers from your documents (RAG), text is turned into vectors for search. The same sentence written two ways can come out as two different vectors, and the right passage is missed. Normalise documents and questions the same way first.
• Test in Persian, with your own data. Quality differs between languages and between models, and only your own examples, written the way your users write, tell you by how much. This matters for a company chatbot in particular.

A short checklist

1. Convert Arabic ي and ك to Persian ی and ک, and decide on diacritics and hamza; keep آ.
2. Repair half-spaces in prefixes, plurals and suffixes.
3. Convert digits to one set for processing; show Persian digits to Persian readers.
4. Apply NFKC to text from PDFs and other copied sources.
5. Run the same steps on documents and on queries.
6. Test on real samples of the text your users write, colloquial and Finglish included.

Websites face the same problems in their addresses and search pages; our SEO checklist for a Persian website covers that side. Contact us to discuss a system that processes Persian text.