Streams — The definitive guide | Articles | web.dev
Skip to main content English Deutsch Español Español – América Latina Français Indonesia Italiano Polski Português – Brasil Tiếng Việt Türkçe Русский עברית العربيّة فارسی हिंदी বাংলা ภาษาไทย 中文 – 简体 中文 – 繁體 日本語 한국어 Sign in Articles Modern Web Guidance is now stable! Upgrade your coding agents with expert-vetted web development best practices.
Read the announcement . Home Articles Streams — The definitive guide Stay organized with collections Save and categorize content based on your preferences. Learn how to use readable, writable, and transform streams with the Streams API. Thomas Steiner The Streams API allows you to programmatically access streams of data received over the network or created by whatever means locally and process them with JavaScript. Streaming involves breaking down a resource that you want to receive, send, or transform into small chunks, and then processing these chunks bit by bit.
While streaming is something browsers do anyway when receiving assets like HTML or videos to be shown on webpages, this capability has never been available to JavaScript before fetch with streams was introduced in 2015. Note Streaming is technically possible with XMLHttpRequest , but it's a less than optimal solution. Here's a GitHub gist showing a good example of 'XMLHttpRequest' . Previously, if you wanted to process a resource of some kind (be it a video, or a text file, etc.), you would have to download the entire file, wait for it to be deserialized into a suitable format, and then process it.
With streams being available to JavaScript, this all changes. You can now process raw data with JavaScript progressively as soon as it is available on the client, without needing to generate a buffer, string, or blob. This unlocks a number of use cases, some of which I list below: Video effects: piping a readable video stream through a transform stream that applies effects in real time. Data (de)compression: piping a file stream through a transform stream that selectively (de)compresses it.
Image decoding: piping an HTTP response stream through a transform stream that decodes bytes into bitmap data, and then through another transform stream that translates bitmaps into PNGs. If installed inside the fetch handler of a service worker, this allows you to transparently polyfill new image formats like AVIF. Browser support ReadableStream and WritableStream Browser Support 43 14 65 10.1 Source TransformStream Browser Support 67 79 102 14.1 Source Core concepts Before I go into details on the various types of streams, let me introduce some core concepts.
Chunks A chunk is a single piece of data that is written to or read from a stream. It can be of any type; streams can even contain chunks of different types. Most of the time, a chunk will not be the most atomic unit of data for a given stream. For example, a byte stream might contain chunks consisting of 16 KiB Uint8Array units, instead of single bytes. Readable streams A readable stream represents a source of data from which you can read.
In other words, data comes out of a readable stream. Concretely, a readable stream is an instance of the ReadableStream class. Writable streams A writable stream represents a destination for data into which you can write. In other words, data goes in to a writable stream. Concretely, a writable stream is an instance of the WritableStream class. Transform streams A transform stream consists of a pair of streams : a writable stream, known as its writable side, and a readable stream, known as its readable side.
A real-world metaphor for this would be a simultaneous interpreter who translates from one language to another on-the-fly. In a manner specific to the transform stream, writing to the writable side results in new data being made available for reading from the readable side. Concretely, any object with a writable property and a readable property can serve as a transform stream. However, the standard TransformStream class makes it easier to create such a pair that is properly entangled.
Pipe chains Streams are primarily used by piping them to each other. A readable stream can be piped directly to a writable stream, using the readable stream's pipeTo() method, or it can be piped through one or more transform streams first, using the readable stream's pipeThrough() method. A set of streams piped together in this way is referred to as a pipe chain. Backpressure Once a pipe chain is constructed, it will propagate signals regarding how fast chunks should flow through it.
If any step in the chain cannot yet accept chunks, it propagates a signal backwards through the pipe chain, until eventually the original source is told to stop producing chunks so fast. This process of normalizing flow is called backpressure. Teeing A readable stream can be teed (named after the shape of an uppercase 'T') using its tee() method. This will lock the stream, that is, make it no longer directly usable; however, it will create two new streams , called branches, which can be consumed independently.
Teeing also is important because streams cannot be rewound or restarted, more about this later. A pipe chain routing a readable stream going from a call, to the fetch API, through a transform stream, then teed and sent to both a browser for the first resulting readable stream and to the service worker cache for the second resulting readable stream. The mechanics of a readable stream A readable stream is a data source represented in JavaScript by a ReadableStream object that flows from an underlying source.
The ReadableStream() constructor creates and returns a readable stream object from the given handlers. There are two types of underlying source: Push sources constantly push data at you when you have accessed them, and it is up to you to start, pause, or cancel access to the stream. Examples include live video streams, server-sent events, or WebSockets. Pull sources require you to explicitly request data from them once connected to.
Examples include HTTP operations via fetch() or XMLHttpRequest calls. Stream data is read sequentially in small pieces called chunks . The chunks placed in a stream are said to be enqueued . This means they are waiting in a queue ready to be read. An internal queue keeps track of the chunks that have not yet been read. A queuing strategy is an object that determines how a stream should signal backpressure based on the state of its internal queue.
The queuing strategy assigns a size to each chunk, and compares the total size of all chunks in the queue to a specified number, known as the high water mark . The chunks inside the stream are read by a reader . This reader retrieves the data one chunk at a time, allowing you to do whatever kind of operation you want to do on it. The reader plus the other processing code that goes along with it is called a consumer . The next construct in this context is called a controller .
Each readable stream has an associated controller that, as the name suggests, allows you to control the stream. Only one reader can read a stream at a time; when a reader is created and starts reading a stream (that is, becomes an active reader ), it is locked to it. If you want another reader to take over reading your stream, you typically need to release the first reader before you do anything else (although you can tee streams).
Creating a readable stream You create a readable stream by calling its constructor ReadableStream() . The constructor has an optional argument underlyingSource , which represents an object with methods and properties that define how the constructed stream instance will behave. The underlyingSource This can use the following optional, developer-defined methods: start(controller) : Called immediately when the object is constructed.
The method can access the stream source, and do anything else required to set up the stream functi
The rest of this page is on the live site.