Node HTTP Requests: Read Streams with a Byte Limit

Choose the lower-level client for a specific reason

Node's built-in http and https modules expose request and response streams directly. They are useful when you need that level of control, but they require more lifecycle handling than a small fetch call. You must finish the request, handle errors and decide what to do with each response body.

The Node HTTP request documentation describes ClientRequest and IncomingMessage behavior. The example below uses CommonJS require statements and can be saved as a .cjs file. It selects the client from the URL protocol and deliberately supports only HTTP and HTTPS.

Collect a bounded body as bytes before decoding

const http = require("node:http");
const https = require("node:https");

function readText(urlText, maxBytes = 1048576) {
  const url = new URL(urlText);
  const client = url.protocol === "https:" ? https
    : url.protocol === "http:" ? http : null;
  if (!client) throw new Error("Use an HTTP or HTTPS URL.");
  if (!Number.isSafeInteger(maxBytes) || maxBytes < 1) {
    throw new Error("maxBytes must be a positive safe integer.");
  }
  return new Promise((resolve, reject) => {
    const request = client.request(url, {method: "GET"}, async response => {
      try {
        const chunks = [];
        let size = 0;
        for await (const chunk of response) {
          size += chunk.length;
          if (size > maxBytes) throw new Error("Response is too large.");
          chunks.push(chunk);
        }
        resolve({
          status: response.statusCode,
          headers: response.headers,
          text: Buffer.concat(chunks).toString("utf8")
        });
      } catch (error) { reject(error); }
    });
    request.on("error", reject);
    request.setTimeout(5000, () => request.destroy(new Error("Socket inactivity timeout.")));
    request.end();
  });
}

The byte limit applies while reading, so a missing or misleading Content-Length does not bypass it. The function keeps Buffer chunks and decodes the combined result once. Decoding each arbitrary chunk separately can corrupt a multibyte character split between chunks.

This helper is for UTF-8 text and does not perform automatic decompression or content-type conversion. If a server sends compressed bytes, the caller needs the corresponding decoding stage and limits for the decoded size. Do not label arbitrary binary data as readable text simply because toString returns a string.

Understand what completion and timeout mean

Calling request.end finishes the outgoing request. Without it, a server may continue waiting for a request the client never completes. The response callback indicates that response headers arrived; the body is only complete after the stream has been consumed successfully.

The timeout shown here is a socket inactivity timeout, not a strict deadline for the entire operation. A slow stream that keeps delivering data may continue longer than five seconds. If the application needs a total deadline, add and clear a separate timer that destroys the request under the application's cancellation policy.

A connection error rejects the promise. A readable HTTP 404 still resolves with status 404 and its body, allowing the caller to decide how to handle it. The helper does not automatically follow redirects, parse JSON or turn every non-success status into a thrown exception.

Set limits appropriate to the expected endpoint

The default one-mebibyte limit is an example policy for a small response. Choose a limit based on the endpoint's documented size and the application's memory budget. Increasing it to accommodate one oversized result can multiply memory pressure when many requests run concurrently.

If the task is downloading a large file, stream to an appropriate destination with backpressure rather than collecting the entire body. Still verify the status before treating the result as a successful file, and remove or isolate partial output when a stream fails.

Test stream behavior with a local fixture

Use a local server that sends a Unicode string in deliberately split byte chunks. Assert that the final text is intact. Then send a body exactly at the configured limit and another one byte above it. This tests byte accounting rather than relying on a single convenient ASCII response.

Also test a readable error status and a connection that closes before its declared body is complete. The latter must not resolve as a successful truncated result. Keep protocol-validation tests separate so that unsupported schemes fail before a connection is attempted.

For a redirect fixture, assert that the result remains a redirect response and retains its Location header. The helper should not silently request a second destination. If the caller later adds redirect support, set a hop limit and reapply its destination policy at every hop, including changes between HTTP and HTTPS.

The REST API Testing Tool can help inspect an authorized endpoint's browser-readable response. It does not reproduce Node's stream chunk boundaries or replace these local transport tests. Compare the two clients only after accounting for their different response handling and browser access restrictions.

Related guides

Send an HTTP JSON Body in Node.js with the Correct Byte Length

Build a Node HTTP POST using encoded JSON bytes, finish the request, inspect the returned body, and distinguish inactivity from a total timeout.

Header HTTP Get

A practical guide to header http get, including the key steps and checks for syntax, semantics, input data and environment-specific output.

Websocket HTTP Header

Step through websocket http header and verify syntax, semantics, input data and environment-specific output before you use or share the result.