Hey everyone! I'm curious about how folks are managing to build integrations with platforms that don't offer public APIs. I've found that typical browser automations can be pretty fragile and require ongoing maintenance. Right now, I'm diving into creating reliable, low-latency API endpoints for those platforms without public APIs. If anyone has tackled this challenge, I'd love to hear your experiences or any solutions you're currently using. Thanks in advance!
4 Answers
When I worked in a distribution company, many of our suppliers didn’t have public APIs, just these awful portals. We ended up scraping and automating our interactions, and luckily, a lot of those systems were outdated from the 2000s, so there was a minimal chance they'd change. But yeah, scraping is often the solution.
Teaming up with the platform directly can sometimes yield results. They might share API documentation with you. Just keep in mind that for B2B solutions, it can get pricey—sometimes tens of thousands per month depending on the data and the industry.
You can also 'integrate' by manually interacting with the website while keeping an eye on the Network tab in the dev tools. By observing the requests sent during your actions (like URLs, request/response bodies, headers, etc.), you might replicate those in your code. Just be cautious, though—these platforms with no public APIs may not want automated access, and you could be risking a breach of their TOS, which could lead to legal issues.
A lot of the time, web scraping is the way to go. If you're lucky, some platforms provide JSON-LD structured data, which makes it easier to grab the content in a neat format.
That makes sense! But what about situations where you need to perform actions, like making a reservation?