How to encode Node.js response from scratch
In this story, I’ll be showing how to encode responses from Node.js using the built-in zlib library with gzip, deflate and br in a simple html site as shown below
<html lang="en">
<head>
<title>Encoding and Caching</title>
</head>
<body>
<h3>Encoding & Caching Example</h3>
<img id="ramboImg" alt="Rambo Gif" src="images/rambo.gif"/>
</body>
</html>But before we jump into the weeds, I want to mention a two reasons why you would want to encode your response: smaller and faster responses. The best part is that browsers already support this on their end, so there is nothing we need to do to support this on the client side.
The majority of the browsers support 3 type of encodings gzip, deflate and br, and it’s almost always sent by the browser for any request sent to the servers, but it’s up to the server if they want to encoder the response and by doing so they need to communicate which encoding they’re using to encode using the Content-Encoding response header.
The above screenshot was taken from Chrome when making a request my local server.
The above screenshot is showing a http response from my local server sending a response with gzip encoding, oh yeah 😎.
If I would have ignored the Accept-Encoding and send the response without encoding, that would be OK, but we are surely missing out on faster response transfers.
We have to be careful and match the content with the encoding, otherwise the browser won’t know how to decode show the response. Alright let’s get to it!
First lets create a file named EncodingUtil.js, this file is going to contain all the code related to encoding, I will be exporting a function named getSupportedEncoderInfo(request) that will return an object holding the name of the supported encoding or null if the server doesn’t support such encoding, this same object will hold a function to create the zlib object that will help us encode the response. The response parameter in the method signature above, is to know which encodings the browser supports.
At the very top of the file import the zlib mobule withconst zlib = require(‘zlib’)
Then write the implementation of getSupportedEncoderInfo that will parses the Accept-Encoding sent by the browsers to determine which is the best encoding for the response, below is the method signature with export name:
exports.getSupportedEncoderInfo = function getSupportedEncoderInfo(request) {/* body */}The first thing that the function is going to do is to read the accept-encoding header from the request and see if exists:
let acceptEncoding = request.headers['accept-encoding']
let knownEncodings = [kGzip, kDeflate, kBr, kAny, kIdentity]
let acceptEncodings = []
if (!acceptEncoding || acceptEncoding.trim().length === 0) {
knownEncodings = [kIdentity]
acceptEncodings = [new ClientEncodingInfo(kIdentity, 1)]
} else {
// TODO: Parse the list of values
}If accept-encoding doesn’t exist, I’m overwriting knownEncodings and acceptEncodings arrays with kIdentity, kIdentity is a constant with 'identity' as its value, 'identity' basically means don’t encode the response, I’ explain how to parse the value correctly next, but before I do that let me show the declaration of the ClientEncodingInfo class
class ClientEncodingInfo {
constructor(name, qvalue) {
this.name = name
this.qvalue = qvalue
}
}As you can see the class declaration only holds two properties a name and a qvalue; name is used to hold the name of the encoding and qvalue is used to prioritize the encodings sent by the browser, the greater the value the higher the chance it will be picked for encoding, the values go from 0.0 to 1.0; 0 meaning don’t use this at all, and 1 meaning yes you definitely need to use this.
Next I show the class definition of EncoderInfo, I’ll be using it to return a new instance to the caller, I also show a list of constants used through the sample:
const kGzip = 'gzip'
const kDeflate = 'deflate'
const kBr = 'br'
const kAny = '*'
const kIdentity = 'identity'
class EncoderInfo {
constructor(name) {
this.name = name
}
isIdentity() {
return this.name === kIdentity
}
createEncoder() {
switch (this.name) {
case kGzip: return zlib.createGzip()
case kDeflate: return zlib.createDeflate()
case kBr: return zlib.createBrotliCompress()
default: return null
}
}
}EncoderInfo holds the name and it has two functions: createEncoder() that when called returns the appropriate encoding stream writer and isIdentity() that will be used to know if we’re going to need to encode the response or not.
Now onto the parsing code, accept-encoding can hold one value, multiple values and even key-value combinations specified priority order with qvalues and even forbidding encodings from the response by assigning 0 in its qvalue, below are a few examples:
Accept-Encoding: deflate, gzip
Accept-Encoding:
Accept-Encoding: *
Accept-Encoding: deflate;q=0.5, gzip;q=1.0
Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0qvalue is represented with q in request headers.
*means any encoding*;q=0accept only explicit encodingsdeflate;q=0.5, gzip;q=1.0acceptsdeflateandgzipbut prefersgzip.- More info here.
As you can see from the above, it can get really wild to parse the list of encodings, but below I show a snippet nailing the a parser for it, bare with me while I explain it:
if (!acceptEncoding || acceptEncoding.trim().length === 0) {
// Code already shown above.
} else {
let acceptEncodingArray = acceptEncoding.split(',')
for (let encoding of acceptEncodingArray) {
encoding = encoding.trim()
if (/[a-z*];q=0$/.test(encoding)) {
let split = encoding.split(';')
let name = split[0].trim()
if (name === kAny) {
explicit = true
}
knownEncodings.splice(knownEncodings.indexOf(name), 1)
} else if (/[a-z*]+;q=\d+(.\d+)*/.test(encoding)) {
let split = encoding.split(';')
let name = split[0].trim()
let value = split[1].trim()
value = value.split('=')[1]
value = parseFloat(value)
acceptEncodings.push(new ClientEncodingInfo(name, value))
} else {
acceptEncodings.push(new ClientEncodingInfo(encoding.trim(), 1.0))
}
}
// Order by qvalue, max to min
}// Code that goes here is explained later
let acceptEncodingArray = acceptEncoding.split(‘,’) splits the list by , making it easier to encode q values, then I loop through all it’s values in a for loop.
The first condition if (/[a-z*];q=0$/.test(encoding)) {} is to remove encodings supported by the server that are not supported by the clients, /[a-z*];q=0$/ is a regex that matches any encoding with q=0, we have these as special case because when *;q=0 is found, we want to make the encodings explicit, meaning, only encoder with the given encodings in the accept-encoding header, otherwise fail the response.knownEncodings.splice(knownEncodings.indexOf(name), 1) is used to remove the known server encodings from the array.
Next we are looking for encodings with qvalue other than 0 that also matches decimals with else if (/[a-z*]+;q=\d+(.\d+)*/.test(encoding)) within this condition we get the name and q value and push it to the acceptEncodings array.
Last part of the parser is to push encodings without any q value showed in the } else { section, I set a q value of 1 for these.
After parsing is done, I sort the encodings by q value given by the client with the following snippet:
acceptEncodings.sort((a, b) => {
return b.qvalue - a.qvalue
})Now that the encodings are sorted by priority it’s much easier to match known server encodings with the ones the client is requesting, at first match we’re done! we assign the name of the encoding to the encoding variable for later use:
let encoding = ''
for (let encodingInfo of acceptEncodings) {
if (knownEncodings.indexOf(encodingInfo.name) !== -1) {
encoding = encodingInfo.name
break
}
}If the picked encoding is * meaning any, we want to pick an encoding the server knows that has not been excluded with q=0:
// If any, pick a known encoding
if (encoding === kAny) {
for (let knownEncoding of knownEncodings) {
if (knownEncoding === kAny) {
continue
} else {
encoding = knownEncoding
break
}
}
}The last thing we need to check is if we were actually able to find a matching encoding between the server and the client, if it couldn’t and it was not explicit then we can send the response without encoding using identity otherwise we don’t know which encoding is supported by the client and we need to send http status code 406 not acceptable.
// If no known encoding was set, then use identity if not excluded
if (encoding.length === 0) {
if (!explicit && knownEncodings.indexOf(kIdentity) !== -1) {
encoding = kIdentity
} else {
console.error('No known encoding were found, return 406')
return null
}
}If by this point we still have an encoding return a new EncoderInfo object:
return new EncoderInfo(encoding)Now that we’re done with EncodingUtil.js, let’s imported to app.js (the main node.js app):
const { getSupportedEncoderInfo } = require('./EncodingUtil')As you can see, we don’t need to specify the file extension, node.js knows it’s a javascript file, but because we’re exporting the method as a named export we do have to specify the name of the method we want to export from the script with { getSupportedEncoderInfo }.
Once imported we use getSupportedEncoderInfo whenever we get a request that’s why I’m inserting it as the first line within the http.createServer() callback:
http.createServer((request, response) => {
let encoderInfo = getSupportedEncoderInfo(request)If encoderInfo is null we have to return 406 and end the request, I have included a JSON for my tests, but an empty response is better.
if (!encoderInfo) {
// Encoded not supported by this server
response.statusCode = 406
response.end()
return
}I set the response content-encoding header so that the client knows which decoder to use:
response.setHeader('Content-Encoding', encoderInfo.name)And then if the encoding is not identity I create a PassThrough stream in a variable called body, that should only be used for the contents of the body and nothing else, header and status codes should still use the response object provided by the http module.
let body = response// If encoding is not identity, encode the response =)
if (!encoderInfo.isIdentity()) {
body = new PassThrough()
pipeline(body, encoderInfo.createEncoder(), response, onError)
}
PassThorough and pipeline are exports from stream:
const { pipeline, PassThrough } = require('stream')After we have the pass through stream ready we can pipe in the server body response through it, like the following index.html site:
response.setHeader('Content-Type', 'text/html')
const stream = fs.createReadStream(`${__dirname}/index.html`)
stream.pipe(body)As you can see, I still use response to set the header and then pipe the contents of the html file into the body.
Find the complete code example below, it downloads files from an FTP server, encodes and sends it back as the body response:
Write y’all later ✌️
Reference: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
