Text Segmenter

Split text into words, sentences, and user-perceived characters.

How to interpret Unicode and segmentation

Text Segmenter works with a layer of text that is not always visible on screen. A user-perceived character may contain several code points, emoji may use ZWJ sequences, and word rules vary by language.

  • Distinguish UTF-16 units, code points, and grapheme clusters before defining field limits or positions.
  • Use normalization when comparing equivalent text represented by different sequences.
  • Treat results as a default analysis: languages such as Thai, Chinese, and Japanese may need tailored rules.
Practical tip: Do not cut text inside a grapheme cluster. Test emojis, combining marks, and different scripts.

Como usar este módulo

Segmentar texto por palavras, frases ou grapheme clusters é mais preciso do que separar apenas por espaços. As regras padrão são sensíveis a idioma e contexto.

Dica prática: Trate a segmentação como uma aproximação padrão e teste o idioma do seu conteúdo.
Ver Word Counter Voltar para Texto