I'm working at a small city planning firm, and we've been using ChatGPT Plus to help us monitor new requests for proposals (RFPs) from various local, state, and federal sources, as well as some nonprofits and bid aggregator sites. We've set things up so it checks specific websites daily for updates on RFPs that fit our criteria based on keywords we've provided. However, we've been running into several issues.
1. Even though we've automated some tasks across ten chats, we're not seeing consistent results. Sometimes ChatGPT doesn't run the task unless we manually prompt it, and other times it misses RFPs completely or only checks one link instead of casting a wider net.
2. For the other chats that aren't automated, we're finding that even with daily prompts, it often ignores our instructions to look for RFPs if the given link doesn't have them. Plus, many sources block ChatGPT from accessing content—would switching to Python help with this?
3. Overall, it feels like ChatGPT isn't responding to our settings effectively, especially for smaller agencies that don't have dedicated pages for RFPs and post them randomly across their sites. We're wondering if Python would allow us to directly scrape these web pages more effectively. Any advice on how to improve the prompts for ChatGPT or suggestions about transitioning to Python for web scraping would be greatly appreciated!
3 Answers
It sounds like you might be over-complicating things by trying to make ChatGPT do a job it's not designed for. LLMs like ChatGPT aren’t really suited for automating repetitive tasks like web scraping. If you need reliable results, you should definitely consider using a data engineering approach instead. Python can give you the tools to scrape structured data effectively, which fits your needs much better.
ChatGPT is great for conversations, but it’s not built for automation or real-time data aggregation. Simply adjusting your prompts won’t solve the core issue; you'll need a proper system for scraping. Building a dedicated scraper in Python will give you much more reliable and consistent results compared to using LLMs. Just keep in mind, websites can change, and maintaining your scraper systems will require some attention.
ChatGPT isn't a browser, so the issues you’re encountering are expected. Switching to Python is the smart move. For finding new pages or RFPs, think about integrating a Google Search scraping tool—it can really help with discovery!
Exactly! Using a search scrape lets you dig deeper instead of just relying on fixed links. It could give you a broader view of available RFPs.