Streams & Buffers

11 min read

Streams & Buffers

A Buffer is a fixed-size allocation of raw binary memory outside the V8 heap. Streams process data in chunks instead of loading it all into memory. Together they enable efficient handling of large files, network data, and real-time feeds.

Buffers

// Create a Buffer
const buf1 = Buffer.from("Hello, Node.js!", "utf8");
console.log(buf1);          // <Buffer 48 65 6c 6c 6f ...>
console.log(buf1.toString()); // Hello, Node.js!
console.log(buf1.length);   // 15

// Allocate empty buffer of 8 bytes
const buf2 = Buffer.alloc(8);
buf2.writeUInt32BE(42, 0);
console.log(buf2.readUInt32BE(0)); // 42

Readable & Writable Streams

const fs = require("fs");

// Read large file in chunks (memory-efficient)
const readable = fs.createReadStream("large-file.csv", { encoding: "utf8" });

readable.on("data", function(chunk) {
  console.log("Received chunk of", chunk.length, "characters");
});

readable.on("end", function() {
  console.log("Finished reading");
});

readable.on("error", function(err) {
  console.error("Stream error:", err.message);
});

Piping Streams

The .pipe() method connects a readable stream to a writable stream. This is the most common pattern for copying or transforming data — for example, compressing a file on the fly.

const fs = require("fs");
const zlib = require("zlib");

// Compress a file using gzip — memory efficient, any file size
const readStream = fs.createReadStream("input.txt");
const writeStream = fs.createWriteStream("input.txt.gz");
const gzip = zlib.createGzip();

readStream.pipe(gzip).pipe(writeStream);

writeStream.on("finish", function() {
  console.log("File compressed successfully!");
});

Prefer stream.pipeline() from stream/promises over .pipe() in modern code — it properly handles errors and backpressure across all streams in the chain.

Ready to test yourself?

Practice Node.js— quiz & coding exercises