I'm looking to create a web app where I can display another website in an iframe and enable an AI to perform actions like clicking buttons and entering text. The main app will also include features like a chat window for directing the AI. I'm curious if there's any library or technology available that could facilitate this kind of integration. Most tools I found, like Puppeteer or Browserless, usually operate by launching a new browser instance rather than working within an iframe. Any suggestions?
5 Answers
Just a heads up, if you're trying to open a website in an iframe, it needs to be allowed by the original server's settings. If it's a third-party site, consider using a browser extension instead. That might give you the flexibility you're looking for!
You might want to try Puppeteer with a Puppeteer MCP server. This setup could allow your AI to integrate directly with the testing browser, which should fit your needs pretty well.
Thanks! I'll give that a shot.
Browser permissions generally block this kind of activity by default, but extensions can get around that. Check out some guides on how to set it up properly.
Thanks for the tip! I’ll definitely look into the browser extension approach.
Unfortunately, controlling things inside an iframe from your main app isn't straightforward due to browser security policies. Browser extensions might be your best bet.
Have you thought about using Cypress or Playwright for this? What you're aiming for sounds very similar to what those tools do. Just curious why you're set on using AI specifically with iframes?
I want the AI to execute actions without needing to launch a local app or separate browser instance. Ideally, I could even trigger tests from my phone, so I'm trying to see if that's possible.
I can configure the main app to whitelist the testing web app for the iframe, but I hadn't considered the browser extension yet. I'll explore that option.