I'm diving into the world of web scraping and had a bit of a rough start. I tried following a GitHub project to learn, but I found the methods were outdated, especially with the snscrape library. I also attempted to use Twitter's own mining API, but ran into issues with rate limits. I've come across libraries like Playwright and Selenium, but I'm primarily focused on learning how to extract data efficiently and convert it into datasets for future analysis. If you have any recommendations or resources, I'd really appreciate it!
4 Answers
What specific Twitter data do you need that you can’t find through an API? Are you aiming for sentiment analysis or something else?
To avoid API headaches, check out SNScrape. It’s a handy tool that can scrape Twitter data without relying on headless browsers. For organizing your data, pandas is a fantastic library that makes dataset conversion a breeze. Just a heads up, scraping legality varies by site, so check the site's terms and their robots.txt file before diving in—best to practice where scraping is allowed!
It seems like Twitter’s API has gotten pretty restrictive lately. If you’re interested in machine learning, you might want to skip the web scraping for now, as it's becoming increasingly complicated. You could check out existing datasets instead—here are some links to consider: [Sentiment140 Dataset](https://www.kaggle.com/datasets/kazanova/sentiment140) and [Twitter Sentiment Analysis](https://huggingface.co/datasets/carblacac/twitter-sentiment-analysis). If you really want to scrape later, you could try APIs or scrape when you're more experienced, but it’s definitely a challenging starting point.
Thanks, I’ll look into those datasets! I’m teaching myself, so I’m still figuring things out. If you have more advice, I’d love to hear it!
Don’t stress too much about the legality of scraping. It’s not illegal to scrape most webpages, but many websites don’t allow it in their terms of service and might block your IP if they notice automated scraping. Always check the robots.txt file of a site to see what's allowed. Just flooding a site with requests can land you in trouble as a Denial of Service attack, but normal scraping practices are generally fine if done responsibly.
Thanks for the clarification! I wasn't sure about the rules, so this helps a lot.
Yeah, I’m looking into AI and ML—sentiment analysis was on my list. But since scraping seems complicated and potentially illegal, I might have to find alternative resources.