This documentation covers Fullstory’s system webhooks and v1 APIs. For streaming webhooks, please refer to v2.

This feature is paid only. To view available plans, please see this page.

Getting Started

Webhooks are like HTTP APIs, but in the opposite direction: instead of you writing code to contact Fullstory API servers, Fullstory has written code that can contact your organization’s servers.

There is a fixed list of events which can trigger webhooks (though this list is growing as Fullstory broadens and improves our support for webhooks). When such an event occurs in the Fullstory platform, and you have a webhook endpoint configured to receive that type of event, our systems will contact yours and send your servers data that describe the event that occurred.

Managing Endpoints

You can manage your endpoints — the URLs that Fullstory uses to contact your servers — in the Settings area of the Fullstory application, where you'll find a section named Webhooks.

Note that you must have an Admin or Architect key to view and manage webhook settings.

Endpoints are configured with the following properties:

1. URL: Each endpoint is identified by its URL. The URL must have a scheme of HTTP or HTTPS. This URL includes the server address that Fullstory will contact.
2. Secret: The secret is a value that should be known only to Fullstory and your endpoint server. This will be used for authentication — so you know that the request your server receives really came from Fullstory and not from some bad actor or hacker. See below for more information on how to use the secret.
3. Events: The set of events determines what kinds of events trigger webhooks to be sent to this URL. For example, you might write code that is interested when a new note is created for a session and is not interested in other kinds of events; so you would only select the "Note Created" event.

Before you save the configuration, you can try sending a test event to the endpoint by clicking on the Test endpoint button. Learn more

How Fullstory Delivers Webhook Events

When a webhook-enabled event occurs inside of the Fullstory platform, an event payload is generated, and, if there are any configured endpoints for that event type, it is then queued for delivery.

The term "webhook-enabled" just means an event or activity for which we can send a webhook. For example, when a user creates a note while viewing a session, that is webhook-enabled. However, when a user in your organization loads the "Everyone" segment in the Fullstory UI, that is not webhook enabled. The sections below catalog all of the various event types that are presently webhook-enabled.

Events are typically delivered within a few seconds. While our system attempts to deliver events in mostly FIFO order (first in, first out), there are no guarantees. In order to handle a possibly high volume of webhook events, the architecture of Fullstory’s webhook platform favors throughput over strict FIFO ordering. When events are few and far between, they will be delivered in FIFO order (excluding sources of delay like endpoint configuration errors or transient network errors). The ordering of events that happen close to the same time, however, will not be predictable.

When an event is delivered, a signature for the event payload is computed. This signature is included in the request. The actual request your endpoint will receive has the following properties:

• The HTTP method will be POST.
• The Content-Type of the request body will be "application/json; charset=utf-8".
• There will be a request header named "Fullstory-Signature"; this includes the request signature which the recipient should verify. (More details below.)
• Every JSON request body encodes a JSON object that has these properties:
  1. "eventName" — The value of this property is a string that indicates the event type. Each webhook-enabled event type described below has a unique name.
  2. "version" — The version number for the payload. Right now, all payloads have a version number of 1. But when we need to re-define an event’s data in an incompatible way, the new structure will be sent with the next integer version number (e.g. 2). Never fear: we will not send your endpoint a newer version of the payload until you configure it to receive the new version. That way, when we introduce a new version, we won’t break your existing endpoints. (See below for more details on versioning.)
  3. "data" — The value of this property will be a JSON value whose format depends on the event type. The various events described below also describe the format of the data field.
• The actual JSON payload will be no larger than 1 megabyte.

Below is an example showing the entire HTTP request:

POST /fswebhook HTTP/1.1
Host: example.com
Content-Type: application/json; charset=utf-8
Fullstory-Signature: o:TN1,t:1578598083,v:40LSCTg5FsT01HoUJrl8rI+791Z31umBNWYRIovpU9c=

{"eventName":"note.created","version":1,"data":{"id":"abcdefg", "author":"john@example.com", "text":"This is interesting"}}

Throttling

To avoid overloading your servers (or being an inadvertent party to a denial-of-service against your webhook endpoint), Fullstory will not deliver webhook events to a given subscription at a rate faster than 1000 events per minute.

If you have configured an endpoint to receive a high volume event, such as a custom event that happens extremely frequently across your users’ sessions, delivery of webhooks could be delayed. When the rate of events for an endpoint exceeds 1000 per minute, the excess events will be queued. So if the high rate of events is temporary, then after the high rate subsides, all of the events could still be delivered, though spread out over a longer period of time than in which the events were created. However, if the high rate is sustained, then some events may never be delivered. If an event is undeliverable for 24 hours, it is considered dead, and no subsequent deliveries will be attempted.

When events are being throttled, the ordering of events becomes much less predictable. Instead of only delivering older events in the queue (FIFO order), new and recent events can be allowed through, too.

The limit of 1000 events per minute is per organization. So if you use the Umbrella Management features of Fullstory and have more than one account configured to send webhooks to the same endpoint URL, that URL may receive more than 1000 events per minute (up to 1000 per minute from each account).

Errors and Retries

When Fullstory delivers an event to your endpoint, it is considered successful if the request is fully transmitted and the endpoint replies within 10 seconds with a status code between 200 and 299 (inclusive).

That leaves three scenarios in which delivery is not considered successful:

1. The message is not transmitted to the endpoint. There are numerous reasons this can happen, mostly network or configuration errors.
   • There could be an endpoint configuration error, such as a typo in the hostname. This could prevent Fullstory from being able to resolve an IP address for the endpoint host.
   • There could be a transient network problem that prevents Fullstory programs from reaching the endpoint host.
   • There could be a problem on the endpoint host or in its network such that it is not accepting connections. This could happen if the server program has crashed or if there are firewall rules that block the network connection.
   • The endpoint may be configured to use HTTPS but the server expects plain-text HTTP, or vice versa.
   • The endpoint may be configured to use HTTPS but the endpoint server presents a TLS certificate that is not trusted. Fullstory’s systems will not trust a self-signed server certificate: your server needs a certificate issued by a trusted certificate authority in order for you to use HTTPS.
   • The endpoint configured falls under the category of a reserved IP (such as private internal networks, loopback addresses, multicast etc).
2. The message is successfully sent, and the endpoint server fails to respond within 10 seconds. This is considered a timeout, and Fullstory will close its HTTP connection after this duration.
3. The message is successfully sent, the endpoint server responds within 10 seconds, but sends back a status code that is not between 200 and 299 (inclusive).
   • Note that this includes status codes less than 200 and those between 300 and 399. Even though those are not technically considered error codes in the HTTP spec, they are not accepted as valid response codes for delivering webhooks.

The first two situations will be retried, unless the event is too old and has already been retried too many times. One exception is if the endpoint resolves to a reserved IP address When this happens, the delivery will not be retried. For the third situation, whether the message can be retried or not depends on the status code. The following HTTP status codes are considered transient failures:

• All codes between 500 and 599 (inclusive)
• 302 Found
• 303 See Other
• 307 Temporary Redirect
• 429 Too Many Requests

If the server responds with one of these, delivery will be retried.

Any other status code is considered a permanent failure. Permanent failures will not be retried.

When a delivery attempt fails and is retried, the subsequent attempt will be delayed using an exponential backoff strategy. Delivery will be attempted with a minimum of a 1 minute delay increasing to a maximum of a 10 minute delay between attempts.

If no delivery attempt is successful for 24 hours, the message is considered dead and will no longer be retried. When this happens, the webhooks configuration UI will indicate this in the Status column of the webhooks settings screen.

Clicking the "Error" status will show a report that summarizes the recent dead messages that could not be delivered to the endpoint. The report describes the kinds of errors encountered, to assist troubleshooting your endpoint configuration.

If more than 10 webhooks deliveries are attempted AND the percentage of webhook deliveries that result in failure exceed 10% for a period of 24 hours, a notification email will be sent. These emails are enabled by default for admins of your organization and can be opted in (or out) via the notification settings page.

Audit Log

Audit logs provide a detailed history of every webhook attempt made in the last 30 days. They are an easy way to figure why your webhooks pipeline is not functioning as intended. They provide the ability to look at the exact data payload that was sent, the HTTP response that the destination responded with, and how fullstory reacted.

Clicking one of the endpoints that you have configured on the webhooks settings screen will display a list of the webhook attempts that were made by Fullstory to that particular endpoint. They are presented in descending order by the time of the attempt. You may also filter the attempts by result status: success, temporary failure, or permanent failure.

When the dropdown link on the attempt item is clicked, more details such as the event ID, description of the result of the attempt, and the data payload itself is displayed.

Authentication with the Shared Secret

To authenticate the incoming request, your endpoint implementation needs to compute a request signature and then compare that signature to the one actually included in the request. If the two signatures match, the request is valid. Otherwise, the request is invalid and should be rejected with a "401 Unauthenticated" HTTP status code. If the timestamp of when the request was created is too old, the request could be a replay attack, in which case the request should be rejected (even if the signature is correct).

Computing a Request Signature

In the steps listed above, the first step was to compute a request signature using the shared secret. This operation requires four different inputs:

1. The ID of your organization. This is an identifier provided by Fullstory that uniquely identifies your account. If you use the Umbrella Management feature of Fullstory, you may have multiple accounts, each with its own ID. If you were to configure multiple accounts to send webhooks to the same endpoint, you would then need to use the ID provided in the request to know from which account the event originated.
2. The timestamp at which the request was made. This is not a timestamp you can attain by looking at the current time on a clock. This, too, is provided in the request.
3. The event payload. This is the body of the POST request your endpoint receives.
4. The shared secret. This is not provided in the request. The endpoint implementation code must have access to the very same secret that was provided when the endpoint was configured in Fullstory settings.

The first three are all provided in the request. The event payload is the request body for the HTTP request. But the other two values must be parsed out of the Fullstory-Signature request header. This header is a string that is a comma-delimited list of key-value pairs. Each pair is in the format key:value, and there will be three such pairs:

1. o: The "o" stands for "organization". The value paired with this key is the ID for your account.
2. t: The "t" stands for "timestamp". The value paired with this key is an integer value that represents a unix timestamp: seconds since the epoch (midnight on January 1, 1970 UTC).
3. v: The "v" stands for "value". The header is named "Fullstory-Signature", so the value paired with this key is the actual signature value. It is a base64-encoded string of bytes.

The signature is computed by computing an HMAC keyed hash using the SHA256 algorithm. The key is the shared secret. The data being hashed is the canonical event payload. This canonical payload has the following format: {payload}:{ID}:{timestamp}. In this format, {payload} is the actual HTTP request body — the event payload; {ID} is the ID of your organization (provided in the signature header); and {timestamp} is the integer timestamp when the request was created (also provided in the signature header).

So, to compute a signature, the endpoint implementation code must parse the signature header as described above and then construct the canonical event payload by combining the request body as well as the "o" and "t" values that were encoded in the signature header. Below is an example:

Type	Value
HTTP request body	{"eventName":"note.created", "data":{"id":"abcdefg", "author":"john@example.com", "text":"This is interesting"}}
Fullstory-Signature header	o:TN1,t:1578598083,v:40LSCTg5FsT01HoUJrl8rI+791Z31umBNWYRIovpU9c=
Shared secret	a1618333f9471311g173033fcd370b8
Canonical Event Payload	{"eventName":"note.created","data":{"id":"abcdefg", "author":"john@example.com", "text":"This is interesting"}}:TN1:1578598083

After using the shared secret and canonical event payload to compute the HMAC keyed hash, the result is the signature.

Validating the Request’s Signature

The final step is to validate the request’s signature. This is done by base64-decoding the "v" value from the signature header and comparing it to the signature you just computed. If the two do not match (e.g. they don’t represent the exact same sequence of bytes), then the request’s signature is invalid, and the request should be rejected.

Checking the Request Timestamp

Even if the signature is valid, the endpoint implementation should also take the given timestamp, the "t" value from the signature header, and compare it to the current timestamp. Such a check guards against replay attacks, where old messages could be re-delivered in whole later. Replayed messages will have a valid signature, since the request body would be identical to an earlier request (unless the shared secret has since changed). So the most efficient way to discover they are not valid and don’t need to be processed is to check the timestamp.

It is likely that the two timestamps will not match since Fullstory’s servers and your servers will not have perfectly synchronized clocks, and there is some delay due to network transit and processing time. Because of the clock skew between servers, it is even possible for the time indicated in the request to be in the future.

A reasonable rule of thumb is to reject the request if the difference between your current time and the given timestamp is greater than five minutes. For this reason, it is also good practice to use NTP to keep your servers’ clocks synchronized, to help ensure that your clocks are within five minutes of ours.

Versioning

As we evolve and maintain features in the Fullstory platform, or add new features that impact or integrate with existing ones, we may have need to change the representation of objects in our API data model.

The actual payload delivered to webhook endpoints includes the version number for its payload. The version numbers are just integers. We will only introduce a new version when a backwards-incompatible change must be introduced. When we do introduce a new version, we will continue delivering the old version for endpoints that were configured to receive it.

Note that we will only introduce a new version number for incompatible changes. So below are the changes that we deem compatible:

• Adding a field. We reserve the right to add new fields to existing event types without releasing a new version of the payload.

  This doesn’t just mean a new field directly under the "data" attribute of the payload, but anywhere in the payload. So if the payload has nested JSON objects in its structure, those lower-level objects could also have new properties added to them.

• Adding a supported value. There are some fields that will only contain a fixed set of values. These are string values, where the set of allowed strings is limited. For example, the "duration" field in the Segment Threshold Alert payload can have a value of "DAILY", "WEEKLY", or "MONTHLY".

  In the future, we may add to the set of allowed values. Your endpoint should be implemented so it is resilient to unknown values. For example, if you use a switch block (a control flow statement supported by many popular programming languages including Go and Java), make sure to add a default case that will gracefully handle an unknown value.

Other changes, particularly removing fields or changing the type of a field, are considered incompatible and will be reserved for new versions of an event payload. Note that renaming a field is considered incompatible: it is effectively the same as adding a field and removing a field at the same time. So, since removing a field is not a compatible change, neither is renaming one.

Importantly, this means that your endpoint’s JSON deserialization logic should allow unrecognized fields and just ignore them. This is typically an option, if not the default, for popular JSON libraries in most languages. But this allows us to add a new field to the payload without breaking your endpoint.

Implementing Endpoints

This section contains best practices, to make sure your endpoint implementations function correctly. Many of the points below are directly related to details above, regarding how we send events to your servers.

• Prefer to use TLS ("Transport-Layer Security", often called "SSL").

  Even though the shared secret is never transmitted, there may be other information in webhook event payloads that you want to keep private between Fullstory and your servers. For this reason, it is recommended that you use URLs with the HTTPS protocol instead of plain-text HTTP. This allows fullstory to authenticate your server via its TLS certificate (to make sure a bad actor has not somehow intercepted the request). And it also means that all traffic between our systems and yours is encrypted.

• Use a cryptographically secure shared secret.

  When using HMAC with the SHA256 hashing algorithm, the recommended key (Based on these sources: FIPS 180-4, RFC 2104, RFC 4868) is a cryptographically random value between 256 and 512 bits, depending on the quality of the random number generator.

  We recommend using the openssl tool, which includes a strong random number generator. Using this tool to create a 256-bit secret can be done as easily as running the following command:

  openssl rand -base64 32

  This command generates a secret using 32 random bytes (which is 256 bits). The output can be directly used as the shared secret and pasted into the webhook endpoint config in Fullstory settings.

• Always verify the incoming request signature.

  If your endpoint fails to verify the signature, it is possible for a bad actor to send fake events that might cause your systems to take incorrect action. Always verify the signature (as described in the previous section) before taking any action or executing any business logic. Similarly, always check the timestamp that is part of the request signature, to guard against replay attacks or misconfigured systems re-sending old requests.

• Your endpoint should respond as quickly and briefly as possible.

  Our systems only wait for 10 seconds for a response from your webhook endpoint. Also, our systems never examine the body of a response, instead using only the HTTP status code to decide whether the message has been successfully delivered.

  If you need to do any lengthy processing for an event, it is recommended to use a durable queue so that you can acknowledge the webhook quickly, without waiting for the processing to complete. There are many queuing solutions available. Cloud Platforms typically provide their own such services, such as Amazon SQS, Google Cloud Pub/Sub, and Google Cloud Task Queues. There are also open-source systems that you can run, such as Apache Kafka, Apache ActiveMQ, and RabbitMQ. Less formal solutions may store incoming messages in some other durable store, like MySQL or Redis (essentially using the databases as a queue).

  In addition to replying quickly, it is also recommended to return just an empty body. A body is not necessary; anything that is written as a response is discarded when Fullstory receives it. An empty body prevents any waste, in terms of network transmission.

• Your endpoint should be idempotent.

  There are conditions under which your endpoint will receive the same event more than once. While our systems try to provide single delivery of messages, this is not always possible — that’s the nature of distributed systems. For example, a network issue could occur that prevents our systems from seeing a "200 OK" response sent by your endpoint. In that situation, you will receive the message again, as we retry delivery until we know it’s been successfully accepted (within limits).

• Your endpoint must make no assumptions about event ordering.

  A crucial point to how messages are delivered is that they are unordered. Just because one event actually occurs in our system before another does not mean that it will be delivered to your endpoint before the other is delivered.

  Because of this, some event payloads will be minimal, possibly providing only an ID that your endpoint can use to query data via Fullstory’s HTTP APIs. That way, when you are processing an event, you always know the current state of data. (If we instead included the data in the event payload and then delivered events out of order, your endpoint could be misled about both the order of changes that occurred as well as the final disposition of the data.)

• Periodically rotate the shared secret.

  For security best practices, it is suggested to periodically re-generate the signing secret. Since Fullstory will only ever use a single secret when signing its requests, this means that your endpoint should be implemented to support up to two secrets; one of them should be configured with a hard stop date. This updated configuration should be deployed to your endpoint before changing the shared secret in Fullstory webhook settings.

  For example, let’s say I want to change the secret for my webhook endpoint. We’ll call the current secret OLD and the new secret NEW. Let’s also assume that today’s date is January 1st, 2020 and that it will take at most three days to get the updated secrets deployed to my production endpoint and also configured in Fullstory settings. In this example, I should configure my endpoint so that it will first try to verify the signature using the NEW secret. If that fails and the current date is on or before January 4th, 2020 (three day window from today), then the endpoint should re-try verifying the signature, but with the OLD secret. If either of these succeeds, then authentication is successful. If both of them fail, then authentication fails. If the first check fails (with NEW secret) and the current date is after January 4th, 2020, authentication should fail.

• Your endpoint should be mindful of the HTTP status codes it returns.

  Because there are certain HTTP codes that will cause Fullstory to give up on delivering a given event, your endpoint should be mindful to never return one of those codes in response to a transient/recoverable error. When in doubt, a "500 Internal Server Error" is probably safe: it will cause our systems to keep re-trying to deliver the message.

• Don’t use the same secret in more than one place.

  It is advised to use a different secret for each endpoint you define. Furthermore, if you have a single endpoint that accepts webhooks from multiple Fullstory accounts (e.g. your account uses the Umbrella Management features in Fullstory), you should also use a different secret for each account. In the one endpoint implementation, your code will then need to choose which secret to use when verifying the signature based on the ID indicated in the request.

• Use subcategory value not display name.

  When creating or updating with an endpoint that requires a subcategory, the specified subcategory needs to be the encoded value returned from List Event Subcategories and not the display_name.

  If trying to subscribe to the event segment.trend.alert with the subcategory Everyone, instead of

  {
    "eventName": "segment.trend.alert",
    "subcategory": "Everyone"
  }

  use

  {
    "eventName": "segment.trend.alert",
    "subcategory": "CgsiCW8tSDkxLW5hMRIOCgRTZWF0GgZzeXN0ZW0SEwoHU2VnbWVudBoIZXZlcnlvbmUSEQoIU2VhcmNoaWUaBTQ5MTY4EhAKBUFsZXJ0EICAgICAqJkL"
  }