I have a large corpus of item names that may refer to the same product using aliases, abbreviations, years, or descriptive words. I don't have a canonical product list to match against. How can I discover the groups and give them useful names?
How can I group noisy product names when I don't know the canonical names yet?
Answers
Start by turning each item description into a text representation, then cluster similar entries. TF-IDF is a useful baseline for names that share words. Text embeddings can also help find similar meanings when the wording differs.
For clustering, look at hierarchical clustering or DBSCAN if you don't know the number of products in advance. K-means is another option, but you have to choose the number of clusters. Inspect a sample of each cluster and tune the representation and clustering settings based on what gets incorrectly merged or split.
Reply to this answer
First decide what counts as the same product. Is a year a separate product, a variant, or just extra description? The answer changes which records should be grouped together.
Try separating the core product from variant attributes where you can. Otherwise, a clustering model might merge distinct versions or split aliases that should be together. Check both kinds of error when reviewing results.
Reply to this answer
For a large corpus, nearest-neighbor search can help find likely matches without comparing every entry with every other entry. Use those candidates to build groups, then review uncertain cases rather than assuming every nearby result is the same product.
To label a group, you could choose a representative name from its entries and retain the other names as aliases. That gives you a starting point, not necessarily the official product name.
Reply to this answer
Look into named entity recognition and knowledge extraction too. They may help pull product mentions and attributes out of longer text before you try to group them. How useful that is depends on whether your records are short names or full descriptions.
Reply to this answer
Add your answer
Answers are reviewed before publication.
!cautiously+optimistic~:
Clustering gives you groups, but it doesn't automatically guarantee a good canonical name. You still need a separate way to select or assign one.