I wrote a Python crawler that collects a few basic fields from several real estate agency websites: whether a listing is a house or apartment, its area in square meters, and the number of bedrooms and bathrooms. I would use the results only for aggregated statistics, such as counting how many 100-square-meter houses have two bedrooms and one bathroom. I would not republish listing text or photos, advertise the properties, sell the raw data, or present myself as a real estate agent.
The sites have terms covering reuse and republishing of their listings, and I know contacting the owners for permission would be the safest route. However, the agencies are not especially easy to work with, so I am unsure whether permission is genuinely necessary for this limited use case or whether I should simply build a polite crawler and respond if someone objects.
The statistics would also help me qualify potential buyers before referring them to the agencies, so the project could arguably benefit both sides. I would appreciate perspectives on the ethical and legal issues, especially in a European jurisdiction. Are there accepted standards for responsible crawling, and should I consult a lawyer before proceeding?
5 Answers
The risk depends heavily on the scope. A private experiment with a few carefully spaced requests is very different from repeatedly harvesting an entire catalog for an application used by many people. If the project grows beyond personal analysis, consider obtaining permission or using a licensed industry data source rather than relying on the assumption that nobody will object.
The fact that you are extracting only numbers helps, since facts such as area and bedroom counts generally are not protected by copyright in the same way as descriptions and photographs. However, in the EU a database can have separate protection for the investment involved in compiling and maintaining it. Repeatedly taking a substantial portion of a site's listings could still be an issue, and the site's terms may impose contractual restrictions even when copyright does not apply.
Treat the legal question separately from the crawling mechanics. A lawyer familiar with technology, contracts, intellectual property, and European database rights can tell you whether the sites' terms or the way you extract the data create a problem. The businesses you mentioned had legal advice and plans for handling disputes; if a lawsuit would be financially devastating, a short consultation is worthwhile.
Not publishing the original descriptions or images is an important distinction, and using aggregated results is less risky than redistributing listings. Still, public availability does not automatically mean unrestricted permission to collect data at scale. Keep the collection narrow, avoid storing unnecessary identifying details, respect removal requests, and get local legal advice if this will support a public or commercial service.
From a technical and ethical standpoint, make the crawler easy to identify, follow each site's robots.txt and published policies, respect response headers, cache results, and use conservative request rates. Do not try to evade blocking or disguise the crawler. Those steps do not guarantee permission or eliminate contractual issues, but they reduce the chance of burdening the sites and make your conduct more defensible.
That distinction is helpful. I am not trying to reproduce the listings, and the results would mainly support buyer qualification before referrals, but I will look into a short consultation before making the crawler broadly available.