I'm new to tech and recently left the pharma industry, which isn't known for being on the cutting edge of technology. I'm interested in scraping data about cookies from various websites, including their types, ingredients, retailers, manufacturers, recipes, tariffs, pricing, and sales information. I'm using Python for the scraping, along with tools like XLS and CSV files and Jupyter for validation. My goal is to find the most cost-effective way to make this data searchable on Google and easy for users to navigate. I'm open to setting up a server and plan to initially work on it alone, but I want to create an open-source repository and gather feedback from users for multiple iterations. Eventually, I hope to attract sponsorship from major manufacturers or retailers to support my work. I have some experience with SQL and other database systems, but I don't aim to become a coding expert. Any advice would be appreciated!
5 Answers
How extensive is the database you're planning to build? You'll need a well-organized structure since you're looking at around 75 fields for the data. Think about the user experience too, especially since traditional artists may not be tech-savvy.
Before you get too deep, have you set up your scrapers and tested the data collection? Make sure you have a reliable source that can handle daily requests without hitting your residential IP. Start small and then think about scaling up.
Totally, I'm working on the scrapers! Just worried about scalability so I don't have to redo everything later.
For SEO, it can be free if you know how to manage it after your launch. Are you looking to improve searches on your site itself, or get better visibility on Google?
Both! I want to make sure users can easily find the data they're searching for. It has to be super user-friendly.
It sounds like an interesting project! Just to clarify, when you say 'cookies,' are you referring to the edible kind or the digital data stored by browsers? That'll help me understand what you're tackling here!
Clearly the edible kind! The focus here is on recipes and ingredients.
Yeah, it's definitely about the edible cookies. Sounds like a fun niche to dive into!
Just a heads up, calling yourself a 'greenhorn' can have negative connotations. Maybe use a different term next time to avoid any misinterpretation!
Good point! I want to cover everything from color and viscosity to where to buy these paints, and I'm hoping to connect with about 3,000 global retailers.