General Purpose Webhook Proxy
@ is a payload delivery service which proxies payloads from the webhook source and transmit them to a locally running app. However, it was designed for GitHub, so customizing it for other services is necessary - here's how we did it.
WIIFM (what's in it for me)
- Learn what is a webhook
- Learn what is smee.io and how it can be used with webhooks
- Learn how to customize smee.io for your own needs
- Learn about some alternatives
Introduction
; if you want to test or develop your webhook integration locally, you need some way to expose your local host to the internet.
This is where smee.io comes in handy. Smee.io is a webhook payload delivery service that uses ." (and we thank them for it 🙏). It's free, easy to use, and works with any service that supports webhooks - or at lease, in theory it should.
Since smee.io was designed to work with
Reservation
In order to create a channel on smee.io, you should point your browser to ). However, if anyone knows your channel ID, they will be able to listen to messages sent to the channel, so you don't want to pick an easily guessable channel ID.
Security
Smee.io relies on message authentication rather than channel security. This makes a lot of sense as smee.io has no configuration and all the webhook messages GitHub sends out have a signature header (e.g. X-Hub-Signature-256: sha256=xxxxxxx...). This is actually a good practice, which Slack follows as well (different header), however, not all services do. It would have been great if you could protect your channel with a key somehow, wouldn't it? Without it, someone can spam or listen to our channel, intentionally or not. It would also mean that you would give your channels meaningful names, like /slack-integration or /github-app
Content Type
Smee.io only supports application/json content as the webhook payload. I do not think that was intentional, but it was not intentionally designed to support other content types as well. This is due to the use of a common Express.js body parsers:
app.use(express.json())
app.use(express.urlencoded())
Taking Slack webhooks as an example, . I, personally, don't understand why Slack chose to do so, but it is what it is.
Because smee.io uses the express.urlencoded body parse, when the it receives content-type: application/x-www-form-urlencoded it will automatically convert the payload (e.g. key1=Some%20Value&key2=Other%20Value) to JSON:
JSON.stringify(req.body) // -> '{"key1":"Some Value","key2":"Other Value"}'
This is very useful when you want to work with the content, which smee.io does in order to display the content on the web UI, but it is breaking the forwarding of the message to the clients (as the client expects it to arrive as URL encoded payload)
Endpoint Verification
Smee.io is a payload delivery service (
This means it automatically responds to the service with 200 OK, this happens even if there are no subscribers:
app.post('/:channel', async (req, res, next) => {
await bus.emitEvent({ ... })
res.status(200).end()
})
Unfortunately, with Slack, there's an and trade off some of the complexity, but you will probably have to ask your IT for help with this every time there's a change.
Tunneling Service
Tunneling services can be considered as a solution in some cases. Services like , create a public endpoint that tunnels communication to your local endpoint via a tunnel client.
This is great when you need to craft a special response to the webhook service, for example the Slack ownership verification mentioned above. In addition, tunneling solutions support path forwarding out of the box as it tunnels the full request.
or free-tier cloud providers) or at low cost (e.g. and get it working. And by working, I don't mean deploying it, but being able to build and run it locally to a point where you can place a break point in the code and have it pause there. This is necessary in order to customize any code. I actually had some issues doing that with the main branch as it was in and I'm not sure it was related, but I had to replace
- "node-sass": "^4.14.0",
+ "sass": "^1.58.3",
and upgrade a bunch of packages:
- "babel-loader": "^8.1.0",
+ "babel-loader": "^9.1.2",
- "mini-css-extract-plugin": "^0.9.0",
+ "mini-css-extract-plugin": "^2.7.2",
- "node-sass": "^4.14.0",
+ "sass": "^1.58.3",
- "webpack": "^4.43.0",
+ "webpack": "^5.75.0",
- "webpack-cli": "^3.3.11"
+ "webpack-cli": "^5.0.1"
I also had to add the --inspect flag to the start-dev npm script
- "start-dev": "concurrently \"nodemon --ignore src/ ./index.js\" \"webpack -w --mode development\"",
+ "start-dev": "concurrently \"nodemon --inspect --ignore src/ ./index.js\" \"webpack -w --mode development\"",
and declare a newer version of the node engine compatibility in package.json
"engines": {
- "node": "12.x.x"
+ "node": "16.x.x"
},
as well as in the Dockerfile together with specifying the platform architecture for compatibility building on MacBooks as well
-FROM node:12-alpine as bundles
+FROM --platform=linux/amd64 node:16-alpine as bundles
You can find the final package.json and Dockerfile package as I had good experience with it and it supports cascading config options.
Create the default base configuration:
// config/default.js
/**
* Mode enum
* @enum {string}
*/
const Mode = {
/** block all */
block: 'block',
/** no protection */
open: 'open',
/** only in list */
allowed: 'allowed',
/** requires password */
password: 'password'
}
module.exports = {
channels: {
/**
* Mode
* @param mode {Mode} one of the modes
*/
mode: Mode.allowed,
list: {
protected: {
password: 'password'
},
open: {
password: null
},
slack: {
password: 'slack',
handler: 'slack'
}
}
}
}
Later, during deployment I can mount a config/local.js file to my docker and it will override some items in the config/default.js file.
You'll note that the slack channel configuration defines a handler: 'slack', you'll see where that comes to play in a bit.
Add Support for URL Encoded Payloads
When a URL encoded payload is received, I want to pass it as is to the subscribers. For that, we need to keep the raw content (it will be signed as well in most cases), so I piggybacked on the body parser verifier:
app.use(express.urlencoded({
extended: true,
verify (req, res, buf, encoding) {
if (buf?.length) {
req.rawBody = buf.toString(encoding || 'utf8')
}
}
}))
This saves the original buffer into a new rawBody property based on this wonderful ! 🙏). Then we need to forward the raw body to the subscribers:
await bus.emitEvent({
channel: req.params.channel,
payload: {
...req.headers,
// forward raw body if captured (application/x-www-form-urlencoded)
// otherwise, forward the parsed body
body: req.rawBody ?? req.body,
query: req.query,
timestamp: Date.now()
}
})
Create Custom Middleware
I wanted to touch the original code as little as possible, so it would be easy to merge updates from the upstream repo. This means there should be a minimal footprint for the custom handlers and security and the changes should be encapsulated as much as possible. The best way to do that in the Express.js world is via ). That way we will be able to support all (I hope) webhook services. Even those who do not support custom headers or query params. To resolve the password^3^:
module.exports = function customMiddlewareInstaller(app) {
...
app.use('/:channel', async (req, res, next) => {
const channel = req.params.channel;
const [name, password] = channel?.split(':') || []; // Note: password cannot contain the `:` character
}
}
^3^ Do note that I'm using clear-text password in both sender (URL) and configuration for the sake of simplicity. This solution is not meant to be used in production. You can extend this code to support a more robust and secure solution, but that is outside of the scope of this post
Resolve the channel options from configuration:
module.exports = function customMiddlewareInstaller(app) {
...
app.use('/:channel', async (req, res, next) => {
...
const options = config.channels?.list?.[name];
}
}
Resolve custom handler from channel options:
module.exports = function customMiddlewareInstaller(app) {
...
app.use('/:channel', async (req, res, next) => {
...
const handler = req.method === 'POST' && options?.handler
? require(`./handlers/${options.handler}`)?.bind(null, req, res, next)
: null;
}
}
What this means is that is a handler is configured, the code will try to load it from the /handlers folder. e.g. for the configuration shown above the slack channel had the handler: 'slack' configured, which means that the code will try to resolve the module from ./handlers/slack; if it finds it, it will bind the connection params to the default export method and later execute it if it passes all the conditions.
Note that with node, loaded modules are cached, so it actually only loads it once while the application is running.
Moving on... based on the configured mode, apply conditions:
module.exports = function customMiddlewareInstaller(app) {
...
switch (mode) {
// block everything
case 'block': return forbidden(res)
// allow everything
case 'open': break;
// allow only if in list
case 'allowed': {
if (options?.password && options.password !== password) return forbidden(res)
break;
}
// allow only if in list and password protected
case 'password': {
if (!options?.password || options.password !== password) return forbidden(res)
break;
}
// misconfiguration
default: throw new Error(`Invalid mode ${mode}`)
}
// if there's no handler OR it return 'false'
// request should be handled by the next layer
if (!await handler?.(req, res, next)) return next()
}
And here's the special Slack handler:
// handlers/slack.js
module.exports = function slackHandler (req, res) {
const challenge = req.body?.challenge
console.log(`slackHandler[${challenge || 'passthrough'}]`)
if (!challenge) return false
res.send(challenge)
return true
}
With this custom handler, our flow would look like this:
as well. So maybe a follow-up?
Deployment
We decided to deploy on our AWS ECS. For simplification we created a deploy.sh script, you're welcome to use it as well, though it is outside the scope of this post.
References
- and
- The good folks of the 's , , sish
SOCIAL SHARE CARD GENERATOR