I'm working on a project where I need to vectorize a list of JSON objects that represent products. Since the AI Search feature only supports vectorizing a single field, I've been manually concatenating all the fields into one string formatted like this: : . : . ... This method worked for many of the products, but I've started running into issues with specific queries, especially those that involve product IDs. I'm looking for community input on two main points: 1) Is there an automated solution to enhance this process and improve search quality? 2) If not, is my current approach of concatenating everything into one string a viable method, or is it likely to lead to bigger problems down the line?
4 Answers
Have you considered asking CoPilot directly? Sometimes it’s best to get insights from the source.
If you're going ahead with concatenating keys and values, try to make it read coherently, like forming a proper sentence. Depending on your query API's capabilities, you might want to consider using different indices for each field and then search all indices with your queries. Also, vectorizing product IDs might not be necessary, but I could be missing something important about your implementation.
Just to add, the current setup runs through CoPilot/ChatGPT, so that's where a significant part of my vectorization challenge lies.
You should treat each product as its own document in your searches. The key issue here is how to effectively vectorize multiple product properties since Azure AI Search allows only one vector field. So, make sure you're structuring things correctly for the best results.
The catch is that the index serves as a knowledge source for CoPilot Studio, and I can’t control the requests that will be pulled to fetch data, which is why I have to stick with concatenation.