Building Developer Friendly Webhooks

Introduction

If you're a first-time technical architect tasked with building a webhook system, there are several critical aspects and best practices to consider to ensure the success and developer-friendliness of your webhook feature. Below, we'll provide comprehensive guidance, including architectural considerations, package recommendations, and additional concepts like fan-out.

Examples of Developer-Friendly Webhook Products

To inspire your design, consider successful products that provide webhook functionality that developers love, such as:

• Stripe: Stripe offers webhooks for real-time payment notifications, making it easy for developers to stay informed about payment-related events in their applications. You can register multiple webhooks, select events, use CLI for localhost routing, etc. They offer the best developer experience w.r.t consuming webhooks.
• GitHub: GitHub's webhooks notify developers about various activities, like code commits and issues, enabling seamless integration with other development tools.
• Twilio: Twilio's webhooks for SMS and voice services allow developers to receive event notifications, enabling them to build real-time communication applications.

Software Architecture Pattern

Publish-Subscribe (Pub/Sub) Pattern

The Pub/Sub architectural pattern is highly recommended for building a scalable and efficient webhook system. In this pattern, your application serves as the publisher, sending events to a central message broker that delivers these events to subscribed consumers. This approach decouples producers (your application) from consumers (external systems), ensuring scalability, reliability, and flexibility.

Webhook Authentication

Implementing proper authentication is essential to ensure the security of webhook communication. Here are popular techniques and examples:

Authentication Type	Description
API Keys	Provide consumers with unique API keys that must be included in webhook request's HTTP header. Verify these keys on the server side by the consumers. These are simple to implement, and suitable for low-security use cases.
OAuth	Use OAuth 2.0 for secure authentication, allow producers for strong security and supports fine-grained permissions.
Signature Verification	You should include a verification signature signing the request raw body. This effectively means the payload has originated from your service and is not tempered.

Technical Features - Developer Experience Matters

As a webhook feature designer, your customers are developers who possess a keen eye for spotting issues and design flaws. To ensure your webhook system meets their needs, consider the following essential technical features:

HTTP Status Codes

HTTP status codes play a crucial role in webhook communication. As a webhook event publisher, you should observer the Response Status Code and note the success or failure of the delivery. For example,

• a 200 status code indicates a successful acceptance of the event by the receiver.
• a 4xx status code indicates a receiver's issue or payload mismatch. You service shouldn't try delivery until webhook configuration/setup is updated. Retries are not recommended in this situation.
• a 5xx code signifies receiver's internal server error. Retries can be attempted.

This is a must feature to be built as a webhook publisher.

Retries

Implement retry mechanisms for failed webhook deliveries. If a consumer's endpoint is temporarily unavailable, your system should attempt redelivery at defined intervals to ensure reliability. A non 200 status code should be considered as failure.

Note that ideally a 4xx status code from consumer shouldn't be retried. In such cases, further delivery of events should be suspended for the given HTTP destination.

Rate Limits

Enforce rate limits to events delivery to prevent abuse and ensure fair usage of your webhook system. Clearly communicate these limits to consumers and provide guidelines on handling rate limiting errors. E.g. for a multi-tenant SaaS application, a tenant should be limited to delivery of 1M events per day. Anything beyond this should be throttled and deferred for next day.

Idempotency

Encourage consumers to implement idempotent handling of webhook events. This guarantees that processing the same event multiple times yields the same results as processing it once. Add a unique eventId to help the consumers. This is a must feature.

Timeouts and Failures

• Establish a transparent timeout policy when initiating a request and ensure your consumers are well-informed. For instance, promote a fixed 10-second HTTP timeout threshold. Any request exceeding this limit will be marked as failed. This approach encourages consumers to embrace asynchronous processing.
• Develop a resilient error handling strategy complete with precise error codes and descriptive messages to assist consumers in efficiently resolving issues. Implement robust logging and monitoring mechanisms to track failures and identify performance bottlenecks in real-time.

Availability

Ensure high availability by using load balancing, redundant servers, and a reliable message broker. This minimizes downtime and ensures uninterrupted service.

Out-of-Order Events

Introduce a feature to enhance the way webhook consumers manage out-of-order events. Add a timestamp to events and allowing consumers to reorder or ignore events when necessary. They way they can also measure the processing delay for better APM reporting and monitoring. This is a must feature.

Filter And Event Selection

Not every consumer has the same appetite for events. To cater to diverse preferences, your system should empower users with the ability to filter specific event types. Additionally, offering configuration options at the object or entity level, with CRUD (Create, Read, Update, and Delete) event type. This can enhance flexibility for consumers.

Fanning Out

Your system's ability to "fan out" events, duplicating and delivering them to multiple endpoints, plays a pivotal role. This feature empowers micro-service users by offering flexibility, redundancy, even load distribution, customization, and scalability, all crucial aspects for seamless integration and efficient event processing within their architecture.

In a nutshell, your webhook system should offer the flexibility of configuring multiple endpoints with the same destination URL.

Documentation

Documentation is the best experience for developers. Giving your webhook consumers detailed payloads, event names, and attribute definition will greatly improve experience. If your webhook exposes multiple event types with varying payloads, you should provide one example for each use case, simplifying your integration process.

UI Features - Easier Debugging

To assist developers in debugging webhook-related issues, include the following features in your product's user interface:

• Test Webhook Endpoint: Allow users to simulate webhook deliveries to their endpoints to verify their systems are set up correctly. In the screen where user is configuring the URL, allow him to sent sample events. They way a user gets a confidence on connectivity.
• Webhook Event Logs: Provide a log of all webhook events sent, including status codes received, response times observed, and delivery attempts. You don't need to store or show all, but at least show the last 100, or more importantly the failed once.
• Webhook Configuration Screens: Offer a user-friendly interface for configuring webhooks, specifying endpoints, and managing subscriptions. Insists on HTTPs endpoints only.

Delivering Webhooks at Scale: Key Tech-Design Considerations

Delivering webhooks at scale requires careful planning and design. In this section, let's deep dive into three main tech-design or architectural considerations for delivering webhooks at scale.

1. Asynchronous Processing

If your application allows bulk data change or insert, you are going to trigger large number of webhook requests. It's crucial to adopt an asynchronous processing model. Instead of sending webhooks synchronously, i.e. when the data is being updated, it should be deferred and become non-blocking processing. Asynchronous processing decouples your code logic and does heavy lifting without impacting the original call. This to be combined with distributed queues to build a scalable webhook delivery architecture.

Why:

• Improved system responsiveness and scalability.
• Reduces the risk of timeouts, latency increase, or delays when delivering webhooks.

Cost:

• Requires additional infrastructure for message queuing or event-driven architecture.
• Additional cost to implement, monitor, etc.

2. Message Queues

Message queuing systems, like Apache Kafka, RabbitMQ, or AWS SQS, are invaluable for managing the delivery of webhooks. They help ensure reliability and fault tolerance by storing and delivering messages in a robust and ordered manner.

Why:

• Guarantees message delivery even in the face of network or system failures.
• Enables load balancing and horizontal scaling.
• Allows you to implement retry with easy
• A temporary storage for webhook events when the consumer service is down.

Cost:

• Adds complexity to the architecture.
• Requires monitoring to avoid message backlogs, needing scaling up and down.

3. Retry Mechanism

Webhook delivery isn't always guaranteed due to network issues or consumer service downtimes. Implementing a retry mechanism is essential to ensure that failed deliveries are retried automatically.

How:

• AWS SQS: AWS SQS provides a built-in feature called "Visibility Timeout." When a message is retrieved from the queue, it becomes invisible to other consumers for a specified duration (visibility timeout). If the consumer successfully processes the message within this time frame, you should deleted from the queue. If processing fails, the message becomes visible again after the timeout, allowing you for an automatic retries.
• RabbitMQ: RabbitMQ allows you to implement custom retry logic by acknowledging or rejecting messages based on processing outcomes. You can delay requeueing messages, adjust delivery delays, and set up dead-letter queues for handling failed messages, effectively enabling retries.

Why:

Increases the chances of successful webhook delivery. Enhances the reliability of your webhook system.

Cons:

• Must be carefully configured to avoid excessive retries, and potential API rate limits.
• Sophisticated design is needed to handle exponential backup kind of retries.
• Makes these events get delivered out-of-order. This may work for notification type of use-cases. However, data-sync use-cases demand a in-order delivery along with timestamp of the change.

4. Rate Limiting

To prevent overloading your recipient system or external APIs, implement rate limiting for outgoing webhooks. This ensures that you stay within acceptable usage limits and maintain good relationships with third-party services.

Pros:

• Prevents abuse of resources and potential service disruptions.
• Keeps you compliant with API usage policies.

Cons:

• May require fine-tuning to balance delivery speed with rate limits.

5. Monitoring and Metrics (APM)

Application Performance Monitoring (APM) can help you track various aspects of webhook functionality. Here are the key metrics you should measure from a webhook working perspective:

• Delivery Success Rate: Monitor the percentage of successfully delivered webhooks versus the total number of webhooks sent. This metric provides insight into the reliability of your webhook delivery system. Keep track of the rate of failed webhook deliveries and the types of errors encountered. Understanding error patterns helps in identifying and resolving issues promptly.
• Delivery Latency: Measure the time it takes for a webhook to be delivered from the moment it's triggered to the moment it's received by the consumer. Lower latency ensures real-time updates. You should have ability to slice this data based on tenant-id, destination URL, etc.
• Retry Rate: Monitor the frequency or percentage of webhook retries. High retry rates may indicate issues with the webhook delivery process or the recipient's availability. Too much high retry rate shall increase your cost. An ideal retry rate should be less than 5%.
• Queue Length: If you're using a message queue system, track the length of the message queue. Long queues may suggest backlogs and potential delivery delays. If the queue length is high along with data change rate, you shall needs many days to drain the queue. You should consider scaling the workers/shards in such situation.
• Payload Size: Measure the size of webhook payloads. Large payloads can impact delivery times and may require special handling. E.g. largest payload should be uploaded to Blob Storage and only link should be sent in the webhook event. This is optional, but knowing helps.
• Resource Utilization (Worker/shards): Monitor the resource utilization of your webhook infrastructure, such as CPU, memory, and network usage, to ensure it can handle the load, and you can save cost.
• 4xx Error Status Codes: Things you cannot control should be built with boundaries. Measure 4xx errors individually to know the issues with destination URLs.
  • Rate Limit Percentage
  • Authentication Failures Percentage

Recommendations for Consumers

Provide consumers with comprehensive documentation and best practices, including:

• Endpoint Validation: Encourage consumers to validate and secure their webhook endpoints to prevent unauthorized access and data tampering.
• Security: Recommend implementing authentication and authorization mechanisms to ensure only trusted parties can access webhook data.
• Monitoring: Advise consumers to set up monitoring and alerting for their webhook endpoints to quickly identify and address issues.
• Testing: Emphasize the importance of thorough testing of webhook integration, including handling edge cases and error scenarios. Most common issues are around out of order execution and idempotency. You can instruct your consumers to create a temporary HTTP Bin at Beeceptor to receive all the events and route to localhost for end to end testing.