Fetch web content behind a firewall.
Fetching web content from other websites from client-side usually either results in a CORS or a 403 Forbidden
error. A typical workaround for this is to fetch it via a proxy server, but this is also usually blocked due to "Are you a human?" checks.
node-proxy-fetch uses Puppeteer to get the actual page content, grabs the generated HTML, transforms and serves it.
In your proxy server code, assuming you're using Express:
// Packages:
import express from 'express'
import fetch from 'node-proxy-fetch'
// Constants:
const app = express()
// Functions:
app.get('/', async (req, res) => {
const webpage = await fetch({
targetURL: 'https://www.npmjs.com',
type: 'DOCUMENT',
puppeteerOptions: {
baseURL: 'https://www.npmjs.com/package/solid-custom-scrollbars'
}
})
res.send(webpage)
})
app.get('/image', async (req, res) => {
const image = (await fetch({
targetURL: 'https://picsum.photos/1000',
type: 'BLOB'
})).data
res.send(image)
})
app.listen(3000)
If you're using this package with Heroku, be sure to add puppeteer-heroku-buildpack
as your app's buildpack.
If you want to use this package with AWS, try out the sister package aws-proxy-fetch
, or check out this guide.
string
The target URL that you want to fetch.
FetchType = 'DOCUMENT' | 'BLOB'
The type of content you are fetching.
AxiosOptions
- OPTIONAL
Options for Axios, only used when type
is BLOB
.
AxiosRequestConfig<any>
- OPTIONAL
AxiosRequestHeaders
- OPTIONAL
PuppeteerOptions
- OPTIONAL
string
The base URL with the pattern protocol:https://domain.tld
. All relative paths in the fetched HTML is replaced with this.
number
- OPTIONAL
The number of milliseconds to wait for before scraping the HTML. This gives time for the Javascript to run on the page. Defaults to 5000
.
boolean
- OPTIONAL
Whether to transform relative paths with the baseURL
or not. Defaults to true
.
Partial<PuppeteerOptions>
- OPTIONAL
Launch options for Puppeteer.
string[]
- OPTIONAL
Launch arguments for Puppeteer.
MIT