Why similarity matters
Text similarity scores how alike two texts are. It drives deduplication, search ranking, plagiarism checks, and clustering. The right metric depends on whether you care about surface form or meaning.
Surface and set based metrics
- Edit distance counts character insertions, deletions, and substitutions, good for typos and short strings.
- Jaccard similarity treats each text as a set of words and divides the intersection by the union, ignoring order.
- These are cheap but blind to meaning, so synonyms look different.
Vector and semantic metrics
- Represent each text as a vector, classically TF IDF, now a dense embedding.
- Cosine similarity measures the angle between vectors, so it ignores length and focuses on direction.
- Embedding cosine captures semantic similarity, rating a car and an automobile as close even with no shared words.
Choosing a metric
- For near duplicate detection on raw strings, use edit distance or Jaccard.
- For matching meaning across different wording, use embedding cosine similarity.
Key idea
Text similarity ranges from surface metrics like edit distance and Jaccard to semantic embedding cosine, and you pick by whether duplicate form or shared meaning is what you need to detect.