I'm working on a personal portfolio project and need to confirm if my approach is legit. I'm scraping publicly available information from university academic calendars—things like program requirements, course descriptions, and prerequisites—without needing any logins or credentials. I then store this info in a database and create a graph that shows courses, prerequisites, and program structures. Additionally, I use this data to help map a student's transcript to their program, indicate which courses they can take or are blocked from, and offer basic course recommendations. I want to clarify that this project is strictly for my portfolio, I'm not making any money off it, I'm not redistributing the raw data, and all of my sources are publicly accessible pages. Is this generally considered okay or legal?
3 Answers
Don't forget to check the site's robots.txt file to see their scraping policy. If you're unsure, reaching out to the university directly could clarify their stance on scraping their information.
From what I know, as long as you're not gaining unauthorized access to a system, it's generally legal to scrape public websites. Sites that don’t require accounts are usually fair game. Just be polite with your scraping—prevent getting rate limited or IP banned.
Web scraping isn't inherently illegal, especially for publicly accessible data. Problems usually arise when you violate a site's terms of service or scrape content that requires authentication. Most of the time, it's a civil matter, and enforcement usually comes through cease-and-desist letters rather than lawsuits. For your non-commercial portfolio project, the risk is minimal. Just ensure your scraper is respectful, implement rate limits, and avoid overwhelming their servers. Legal challenges over personal projects are rare, but if a site does ask you to stop, it's best to comply.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically