I have a live website whose pages load some content dynamically through JavaScript and API calls. Once the project ends, I'm shutting down the backend and want to preserve the current site as a completely static version, including the rendered text, images, CSS, fonts, and other assets.
I tried tools such as wget and HTTrack, but they mostly captured the original HTML and missed content that appears only after JavaScript runs. Would a browser-rendering tool be the right solution, or is there a better workflow for crawling and exporting the site?
4 Answers
If the dynamic data comes from predictable, parameterless API endpoints, you may not need to save only the final rendered HTML. Crawl the site, save each API response as a local JSON file, and change the frontend to read those files instead of calling the backend. That keeps the existing UI behavior while removing the server dependency, though you’ll need to account for caching, URL paths, and any requests that depend on user state.
For a site with JavaScript-rendered pages, use Playwright or Puppeteer. Have the browser visit each URL, wait for the relevant content and network requests to finish, then save the resulting DOM and download the referenced assets. You can write each page into a directory structure that mirrors its URL. This is easy to automate for a large site, while a small number of pages could be handled manually.
Plain wget can download the original responses and linked files, but it won’t execute the JavaScript that creates the missing content. It only works by itself if the server already returns all the data in the initial HTML or if you separately fetch and reproduce the API responses. Otherwise, use a headless browser or convert the project to a static-site build process.
A browser-based capture tool is the safest option when you need exactly what visitors currently see. Render each page in a real browser, wait until lazy-loaded images and API content appear, and then export the page along with its assets. For a couple dozen pages, a tool that packages a page and its images, stylesheets, and fonts into a self-contained file may be quicker than building a pipeline.
Exactly—saving the markup after the browser has finished rendering is different from downloading the initial response. The script should also wait for the page’s API calls and lazy-loaded content before capturing it.