Timeout puppeeter

by

On the project, that I am working, a lot of times we are dealing with the timeout errors with the puppeeter.

It seems that headless puppeeter does not have the timeout for the resources, like what the more headful type implement. No bring able to download one resource, it timeout the entire page.

Since a lot of time, I am only interested in either HTML of the page or in one single resource or it very rare cases a couple of resources (and which it is depends on if they are having backend rendering/static site or JavaScript rendering), having the code fail because of (for me) irrelevant resource, makes my code very brittle.

There are two potential fixes that I could found. If the resource is not blocking, then waiting for the "domcontentloaded" (HTML is loaded and DOM build, but external resources may not be loaded yet) instead of the default "load" (every resource is loaded) event makes sure, that there is less chance of code failing.

If the resource is not blocking, then having a timeout like event is helpful. This can be done with Promise.race or something similar.

The version that I came up with is like this:

const [, resource] = await Promise.all([
    page.goto(url, { waitUntil: 'domcontentloaded' }).catch(() => {}),
    page.waitForResponse((response) => responseCheck(response)).catch(() => {}),
]);
if (!resource) {
    finishCode()
}

Sources that helped me with the solution: