I'm looking to create a price comparison platform, initially focusing on quick commerce services like Zepto and Instamart, with plans to expand into ecommerce, pharmacy, and possibly services like cabs in the future. While I know there are established players in this space, I'm excited to build this project to learn and explore if I can offer a unique perspective or improvement over what's already out there.
Here's what I've considered so far:
- Reverse engineering or analyzing APIs from various quick commerce platforms.
- Developing a search orchestration layer to collect queries from multiple sources.
- Implementing product search and matching across different platforms.
- Normalizing results since naming conventions, units, and packaging can differ significantly.
- Eventually adding location-aware availability and pricing features.
I'm seeking guidance on:
- Is reverse engineering APIs the best approach, or are there cleaner alternatives?
- Can anyone recommend open-source projects or frameworks that could help me?
- What are the best practices for search orchestration, product normalization, deduplication, and managing inconsistent catalogs?
I welcome any feedback, including criticisms—I'm here to learn!
1 Answer
I’d suggest considering web scraping as a reliable alternative to reverse engineering APIs. APIs can change without warning, and some platforms actively block access. For matching products, fuzzy string matching libraries like fuzzywuzzy or rapidfuzz could be very handy since product names tend to vary a lot between platforms. It might also be a good idea to create a scoring system that weighs factors like brand, product type, and size instead of relying solely on name matching. Just a heads up, location-based pricing can get complicated quickly; availability zones shift often, and you'll need to account for different pricing on the same product across regions.
I agree with web scraping! You can use libraries like Cheerio to parse HTML and perform jQuery-like searches. It's simple because you essentially just need to execute a GET request to gather the HTML, and then parse it from there. Puppeteer is another solid option; it opens a browser and navigates the site to find what you need.