Node Fetch

A light-weight module that brings window.fetch to Node.js.

Build status Coverage status Current version Install size Mentioned in Awesome Node.js Discord

Consider supporting us on our Open Collective:

Open Collective
--- - [Motivation](#motivation) - [Features](#features) - [Difference from client-side fetch](#difference-from-client-side-fetch) - [Installation](#installation) - [Loading and configuring the module](#loading-and-configuring-the-module) - [Upgrading](#upgrading) - [Common Usage](#common-usage) - [Plain text or HTML](#plain-text-or-html) - [JSON](#json) - [Simple Post](#simple-post) - [Post with JSON](#post-with-json) - [Post with form parameters](#post-with-form-parameters) - [Handling exceptions](#handling-exceptions) - [Handling client and server errors](#handling-client-and-server-errors) - [Handling cookies](#handling-cookies) - [Advanced Usage](#advanced-usage) - [Streams](#streams) - [Buffer](#buffer) - [Accessing Headers and other Meta data](#accessing-headers-and-other-meta-data) - [Extract Set-Cookie Header](#extract-set-cookie-header) - [Post data using a file stream](#post-data-using-a-file-stream) - [Post with form-data (detect multipart)](#post-with-form-data-detect-multipart) - [Request cancellation with AbortSignal](#request-cancellation-with-abortsignal) - [API](#api) - [fetch(url[, options])](#fetchurl-options) - [Options](#options) - [Default Headers](#default-headers) - [Custom Agent](#custom-agent) - [Custom highWaterMark](#custom-highwatermark) - [Class: Request](#class-request) - [new Request(input[, options])](#new-requestinput-options) - [Class: Response](#class-response) - [new Response([body[, options]])](#new-responsebody-options) - [response.ok](#responseok) - [response.redirected](#responseredirected) - [Class: Headers](#class-headers) - [new Headers([init])](#new-headersinit) - [Interface: Body](#interface-body) - [body.body](#bodybody) - [body.bodyUsed](#bodybodyused) - [body.arrayBuffer()](#bodyarraybuffer) - [body.blob()](#bodyblob) - [body.json()](#bodyjson) - [body.text()](#bodytext) - [body.buffer()](#bodybuffer) - [Class: FetchError](#class-fetcherror) - [Class: AbortError](#class-aborterror) - [TypeScript](#typescript) - [Acknowledgement](#acknowledgement) - [Team](#team) - [Former](#former) - [License](#license) ## Motivation Instead of implementing `XMLHttpRequest` in Node.js to run browser-specific [Fetch polyfill](https://github.com/github/fetch), why not go from native `http` to `fetch` API directly? Hence, `node-fetch`, minimal code for a `window.fetch` compatible API on Node.js runtime. See Jason Miller's [isomorphic-unfetch](https://www.npmjs.com/package/isomorphic-unfetch) or Leonardo Quixada's [cross-fetch](https://github.com/lquixada/cross-fetch) for isomorphic usage (exports `node-fetch` for server-side, `whatwg-fetch` for client-side). ## Features - Stay consistent with `window.fetch` API. - Make conscious trade-off when following [WHATWG fetch spec][whatwg-fetch] and [stream spec](https://streams.spec.whatwg.org/) implementation details, document known differences. - Use native promise and async functions. - Use native Node streams for body, on both request and response. - Decode content encoding (gzip/deflate/brotli) properly, and convert string output (such as `res.text()` and `res.json()`) to UTF-8 automatically. - Useful extensions such as redirect limit, response size limit, [explicit errors][error-handling.md] for troubleshooting. ## Difference from client-side fetch - See known differences: - [As of v3.x](docs/v3-LIMITS.md) - [As of v2.x](docs/v2-LIMITS.md) - If you happen to use a missing feature that `window.fetch` offers, feel free to open an issue. - Pull requests are welcomed too! ## Installation Current stable release (`3.x`) ```sh $ npm install node-fetch ``` ## Loading and configuring the module ```js // CommonJS const fetch = require('node-fetch'); // ES Module import fetch from 'node-fetch'; ``` If you want to patch the global object in node: ```js const fetch = require('node-fetch'); if (!globalThis.fetch) { globalThis.fetch = fetch; } ``` For versions of Node earlier than 12, use this `globalThis` [polyfill](https://mathiasbynens.be/notes/globalthis). ## Upgrading Using an old version of node-fetch? Check out the following files: - [2.x to 3.x upgrade guide](docs/v3-UPGRADE-GUIDE.md) - [1.x to 2.x upgrade guide](docs/v2-UPGRADE-GUIDE.md) - [Changelog](docs/CHANGELOG.md) ## Common Usage NOTE: The documentation below is up-to-date with `3.x` releases, if you are using an older version, please check how to [upgrade](#upgrading). ### Plain text or HTML ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://github.com/'); const body = await response.text(); console.log(body); })(); ``` ### JSON ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://api.github.com/users/github'); const json = await response.json(); console.log(json); })(); ``` ### Simple Post ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: 'a=1'}); const json = await response.json(); console.log(json); })(); ``` ### Post with JSON ```js const fetch = require('node-fetch'); (async () => { const body = {a: 1}; const response = await fetch('https://httpbin.org/post', { method: 'post', body: JSON.stringify(body), headers: {'Content-Type': 'application/json'} }); const json = await response.json(); console.log(json); })(); ``` ### Post with form parameters `URLSearchParams` is available on the global object in Node.js as of v10.0.0. See [official documentation](https://nodejs.org/api/url.html#url_class_urlsearchparams) for more usage methods. NOTE: The `Content-Type` header is only set automatically to `x-www-form-urlencoded` when an instance of `URLSearchParams` is given as such: ```js const fetch = require('node-fetch'); const params = new URLSearchParams(); params.append('a', 1); (async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: params}); const json = await response.json(); console.log(json); })(); ``` ### Handling exceptions NOTE: 3xx-5xx responses are _NOT_ exceptions, and should be handled in `then()`, see the next section. Wrapping the fetch function into a `try/catch` block will catch _all_ exceptions, such as errors originating from node core libraries, like network errors, and operational errors which are instances of FetchError. See the [error handling document][error-handling.md] for more details. ```js const fetch = require('node-fetch'); try { fetch('https://domain.invalid/'); } catch (error) { console.log(error); } ``` ### Handling client and server errors It is common to create a helper function to check that the response contains no client (4xx) or server (5xx) error responses: ```js const fetch = require('node-fetch'); const checkStatus = res => { if (res.ok) { // res.status >= 200 && res.status < 300 return res; } else { throw MyCustomError(res.statusText); } } (async () => { const response = await fetch('https://httpbin.org/status/400'); const data = checkStatus(response); console.log(data); //=> MyCustomError })(); ``` ### Handling cookies Cookies are not stored by default. However, cookies can be extracted and passed by manipulating request and response headers. See [Extract Set-Cookie Header](#extract-set-cookie-header) for details. ## Advanced Usage ### Streams The "Node.js way" is to use streams when possible. You can pipe `res.body` to another stream. This example uses [stream.pipeline](https://nodejs.org/api/stream.html#stream_stream_pipeline_streams_callback) to attach stream error handlers and wait for the download to complete. ```js const util = require('util'); const fs = require('fs'); const streamPipeline = util.promisify(require('stream').pipeline); (async () => { const response = await fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png'); if (response.ok) { return streamPipeline(res.body, fs.createWriteStream('./octocat.png')); } throw new Error(`unexpected response ${res.statusText}`); })(); ``` ### Buffer If you prefer to cache binary data in full, use buffer(). (NOTE: buffer() is a `node-fetch` only API) ```js const fetch = require('node-fetch'); const fileType = require('file-type'); (async () => { const response = await fetch('https://octodex.github.com/images/Fintechtocat.png'); const buffer = await response.buffer(); const type = fileType.fromBuffer(buffer) console.log(type); })(); ``` ### Accessing Headers and other Meta data ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://github.com/'); console.log(res.ok); console.log(res.status); console.log(res.statusText); console.log(res.headers.raw()); console.log(res.headers.get('content-type')); })(); ``` ### Extract Set-Cookie Header Unlike browsers, you can access raw `Set-Cookie` headers manually using `Headers.raw()`. This is a `node-fetch` only API. ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://example.com'); // Returns an array of values, instead of a string of comma-separated values console.log(res.headers.raw()['set-cookie']); })(); ``` ### Post data using a file stream ```js const {createReadStream} = require('fs'); const fetch = require('node-fetch'); const stream = createReadStream('input.txt'); (async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: stream}); const json = await response.json(); console.log(json) })(); ``` ### Post with form-data (detect multipart) ```js const fetch = require('node-fetch'); const FormData = require('form-data'); const form = new FormData(); form.append('a', 1); (async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: form}); const json = await response.json(); console.log(json) })(); // OR, using custom headers // NOTE: getHeaders() is non-standard API const options = { method: 'POST', body: form, headers: form.getHeaders() }; (async () => { const response = await fetch('https://httpbin.org/post', options); const json = await response.json(); console.log(json) })(); ``` ### Request cancellation with AbortSignal You may cancel requests with `AbortController`. A suggested implementation is [`abort-controller`](https://www.npmjs.com/package/abort-controller). An example of timing out a request after 150ms could be achieved as the following: ```js const fetch = require('node-fetch'); const AbortController = require('abort-controller'); const controller = new AbortController(); const timeout = setTimeout(() => { controller.abort(); }, 150); (async () => { try { const response = await fetch('https://example.com', {signal: controller.signal}); const data = await response.json(); useData(data); } catch (error) { if (error.name === 'AbortError') { console.log('request was aborted'); } } finally { clearTimeout(timeout); } })(); ``` See [test cases](https://github.com/node-fetch/node-fetch/blob/master/test/) for more examples. ## API ### fetch(url[, options]) - `url` A string representing the URL for fetching - `options` [Options](#fetch-options) for the HTTP(S) request - Returns: Promise<[Response](#class-response)> Perform an HTTP(S) fetch. `url` should be an absolute url, such as `https://example.com/`. A path-relative URL (`/file/under/root`) or protocol-relative URL (`//can-be-http-or-https.com/`) will result in a rejected `Promise`. ### Options The default values are shown after each option key. ```js { // These properties are part of the Fetch Standard method: 'GET', headers: {}, // Request headers. format is the identical to that accepted by the Headers constructor (see below) body: null, // Request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream redirect: 'follow', // Set to `manual` to extract redirect headers, `error` to reject redirect signal: null, // Pass an instance of AbortSignal to optionally abort requests // The following properties are node-fetch extensions follow: 20, // maximum redirect count. 0 to not follow redirect compress: true, // support gzip/deflate content encoding. false to disable size: 0, // maximum response body size in bytes. 0 to disable agent: null, // http(s).Agent instance or function that returns an instance (see below) highWaterMark: 16384 // the maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource. } ``` #### Default Headers If no values are set, the following request headers will be sent automatically: | Header | Value | | ------------------- | ------------------------------------------------------ | | `Accept-Encoding` | `gzip,deflate,br` _(when `options.compress === true`)_ | | `Accept` | `*/*` | | `Connection` | `close` _(when no `options.agent` is present)_ | | `Content-Length` | _(automatically calculated, if possible)_ | | `Transfer-Encoding` | `chunked` _(when `req.body` is a stream)_ | | `User-Agent` | `node-fetch` | Note: when `body` is a `Stream`, `Content-Length` is not set automatically. #### Custom Agent The `agent` option allows you to specify networking related options which are out of the scope of Fetch, including and not limited to the following: - Support self-signed certificate - Use only IPv4 or IPv6 - Custom DNS Lookup See [`http.Agent`](https://nodejs.org/api/http.html#http_new_agent_options) for more information. In addition, the `agent` option accepts a function that returns `http`(s)`.Agent` instance given current [URL](https://nodejs.org/api/url.html), this is useful during a redirection chain across HTTP and HTTPS protocol. ```js const http = require('http'); const https = require('https'); const httpAgent = new http.Agent({ keepAlive: true }); const httpsAgent = new https.Agent({ keepAlive: true }); const options = { agent: function(_parsedURL) { if (_parsedURL.protocol == 'http:') { return httpAgent; } else { return httpsAgent; } } }; ``` #### Custom highWaterMark Stream on Node.js have a smaller internal buffer size (16kB, aka `highWaterMark`) from client-side browsers (>1MB, not consistent across browsers). Because of that, when you are writing an isomorphic app and using `res.clone()`, it will hang with large response in Node. The recommended way to fix this problem is to resolve cloned response in parallel: ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://example.com'); const r1 = await response.clone(); return Promise.all([res.json(), r1.text()]).then(results => { console.log(results[0]); console.log(results[1]); }); })(); ``` If for some reason you don't like the solution above, since `3.x` you are able to modify the `highWaterMark` option: ```js const fetch = require('node-fetch'); (async () => { const response = await fetch('https://example.com', { // About 1MB highWaterMark: 1024 * 1024 }); return res.clone().buffer(); })(); ``` ### Class: Request An HTTP(S) request containing information about URL, method, headers, and the body. This class implements the [Body](#iface-body) interface. Due to the nature of Node.js, the following properties are not implemented at this moment: - `type` - `destination` - `referrer` - `referrerPolicy` - `mode` - `credentials` - `cache` - `integrity` - `keepalive` The following node-fetch extension properties are provided: - `follow` - `compress` - `counter` - `agent` - `highWaterMark` See [options](#fetch-options) for exact meaning of these extensions. #### new Request(input[, options]) _(spec-compliant)_ - `input` A string representing a URL, or another `Request` (which will be cloned) - `options` [Options][#fetch-options] for the HTTP(S) request Constructs a new `Request` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Request/Request). In most cases, directly `fetch(url, options)` is simpler than creating a `Request` object. ### Class: Response An HTTP(S) response. This class implements the [Body](#iface-body) interface. The following properties are not implemented in node-fetch at this moment: - `Response.error()` - `Response.redirect()` - `type` - `trailer` #### new Response([body[, options]]) _(spec-compliant)_ - `body` A `String` or [`Readable` stream][node-readable] - `options` A [`ResponseInit`][response-init] options dictionary Constructs a new `Response` object. The constructor is identical to that in the [browser](https://developer.mozilla.org/en-US/docs/Web/API/Response/Response). Because Node.js does not implement service workers (for which this class was designed), one rarely has to construct a `Response` directly. #### response.ok _(spec-compliant)_ Convenience property representing if the request ended normally. Will evaluate to true if the response status was greater than or equal to 200 but smaller than 300. #### response.redirected _(spec-compliant)_ Convenience property representing if the request has been redirected at least once. Will evaluate to true if the internal redirect counter is greater than 0. ### Class: Headers This class allows manipulating and iterating over a set of HTTP headers. All methods specified in the [Fetch Standard][whatwg-fetch] are implemented. #### new Headers([init]) _(spec-compliant)_ - `init` Optional argument to pre-fill the `Headers` object Construct a new `Headers` object. `init` can be either `null`, a `Headers` object, an key-value map object or any iterable object. ```js // Example adapted from https://fetch.spec.whatwg.org/#example-headers-class const Headers = require('node-fetch'); const meta = { 'Content-Type': 'text/xml', 'Breaking-Bad': '<3' }; const headers = new Headers(meta); // The above is equivalent to const meta = [['Content-Type', 'text/xml'], ['Breaking-Bad', '<3']]; const headers = new Headers(meta); // You can in fact use any iterable objects, like a Map or even another Headers const meta = new Map(); meta.set('Content-Type', 'text/xml'); meta.set('Breaking-Bad', '<3'); const headers = new Headers(meta); const copyOfHeaders = new Headers(headers); ``` ### Interface: Body `Body` is an abstract interface with methods that are applicable to both `Request` and `Response` classes. The following methods are not yet implemented in node-fetch at this moment: - `formData()` #### body.body _(deviation from spec)_ - Node.js [`Readable` stream][node-readable] Data are encapsulated in the `Body` object. Note that while the [Fetch Standard][whatwg-fetch] requires the property to always be a WHATWG `ReadableStream`, in node-fetch it is a Node.js [`Readable` stream][node-readable]. #### body.bodyUsed _(spec-compliant)_ - `Boolean` A boolean property for if this body has been consumed. Per the specs, a consumed body cannot be used again. #### body.arrayBuffer() #### body.blob() #### body.json() #### body.text() _(spec-compliant)_ - Returns: `Promise` Consume the body and return a promise that will resolve to one of these formats. #### body.buffer() _(node-fetch extension)_ - Returns: `Promise` Consume the body and return a promise that will resolve to a Buffer. ### Class: FetchError _(node-fetch extension)_ An operational error in the fetching process. See [ERROR-HANDLING.md][] for more info. ### Class: AbortError _(node-fetch extension)_ An Error thrown when the request is aborted in response to an `AbortSignal`'s `abort` event. It has a `name` property of `AbortError`. See [ERROR-HANDLING.MD][] for more info. ## TypeScript **Since `3.x` types are bundled with `node-fetch`, so you don't need to install any additional packages.** For older versions please use the type definitions from [DefinitelyTyped](https://github.com/DefinitelyTyped/DefinitelyTyped): ```sh $ npm install --save-dev @types/node-fetch ``` ## Acknowledgement Thanks to [github/fetch](https://github.com/github/fetch) for providing a solid implementation reference. ## Team | [![David Frank](https://github.com/bitinn.png?size=100)](https://github.com/bitinn) | [![Jimmy Wärting](https://github.com/jimmywarting.png?size=100)](https://github.com/jimmywarting) | [![Antoni Kepinski](https://github.com/xxczaki.png?size=100)](https://github.com/xxczaki) | [![Richie Bendall](https://github.com/Richienb.png?size=100)](https://github.com/Richienb) | [![Gregor Martynus](https://github.com/gr2m.png?size=100)](https://github.com/gr2m) | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | [David Frank](https://bitinn.net/) | [Jimmy Wärting](https://jimmy.warting.se/) | [Antoni Kepinski](https://kepinski.me) | [Richie Bendall](https://www.richie-bendall.ml/) | [Gregor Martynus](https://twitter.com/gr2m) | ###### Former - [Timothy Gu](https://github.com/timothygu) - [Jared Kantrowitz](https://github.com/jkantr) ## License [MIT](LICENSE.md) [whatwg-fetch]: https://fetch.spec.whatwg.org/ [response-init]: https://fetch.spec.whatwg.org/#responseinit [node-readable]: https://nodejs.org/api/stream.html#stream_readable_streams [mdn-headers]: https://developer.mozilla.org/en-US/docs/Web/API/Headers [error-handling.md]: https://github.com/node-fetch/node-fetch/blob/master/docs/ERROR-HANDLING.md