Can a systems engineer realistically build a competitor price monitor?

November 13, 2026 5 answers
Asked by 7p+crayon_305

I've been asked to check competitor prices daily against roughly 20,000 items and email an alert when the difference exceeds 2%. I'm a systems engineer rather than a web developer, and some competitors use anti-scraping protection. Is this realistic to build in-house?

Answers

27p.fuse_check

It's feasible, but the comparison and email are the easy parts. The hard part is getting reliable prices for the right products every day, then keeping that data flowing when sites change or block requests.

Start by checking whether any competitors or data providers offer a permitted feed or API. Availability and cost will vary. Also check the relevant terms and get guidance on acceptable collection practices before committing to scraping protected sites.

A small proof of concept against an accessible source would show what the data looks like, but it won't prove that 20,000 items across multiple competitors will be straightforward.

Reply to this answer
dustpan.with.seniority

Product matching deserves its own workstream. A price is only useful if it's for the same item and comparable offer. Agree on which manufacturer identifiers count, how variants and pack sizes are handled, and whether shipping, tax, promotions, and availability belong in the comparison.

Define the 2% calculation too. Which price is the baseline, and should an alert fire in either direction? Bad matches or inconsistent price definitions could produce a lot of very convincing false alarms.

queue_415+open:

It would help to keep uncertain matches separate for review instead of treating every possible match as an exact one.

Reply to this answer
b6+folder_28

Scope this as an ongoing service, not a script you hand over on Friday. You'll need somewhere to run it, a daily pipeline, failure alerts, logs, and a way to spot stale or obviously wrong prices. Someone also needs time to investigate when a site changes or a data source stops working.

Ask management to compare the cost of building and supporting that service with a vendor quote. If development isn't your role, getting a developer's estimate is a sensible starting point.

Reply to this answer
z3!compass_740

AI can help draft code and explain unfamiliar parts, but it doesn't make the integration reliable or remove the need for testing and support. A prototype might be quite doable if you can script, provided you have access to a usable data source. I wouldn't promise a production deadline before testing that assumption.

Reply to this answer
tx7!align

Be careful about treating public pages as an unlimited data feed. Daily collection at this scale can put load on a site and may trigger blocking. Build the plan around permitted access and reasonable request rates, and agree on what happens when a source can't be accessed rather than assuming every price will always be available.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions