Some sites don't load correctly
Incident Report for Framer
Postmortem

On February 27th 2024, a small subset of Framer sites experienced a partial service degradation. The affected sites were served with incomplete HTML content under certain conditions.

The root cause was a code change that we deployed February 26th 2024 at 15:39 UTC. The change caused our servers to incorrectly truncate gzip compressed sites. The corrupted content was also stored into our caches. This resulted in sites being served with incomplete content when clients requested a gzip compressed site.

Modern browsers (about 97% of site visitors) use brotli compression, so were not affected. However, sites using a reverse proxy that request gzip content were affected, and possibly passed on broken content.

On February 27th 2024 around 09:05 UTC, multiple Framer employees started noticing some sites to be broken (as well as some Framer community posts reporting similar issues).

We immediately started investigating and fixed the issue by:

  • Rolling back the problematic code change.
  • Purging all cached content from our hosting systems.

This process was completed by 11:30 UTC on February 27th 2024.

We are committed to a high quality standard, and we take this incident extremely seriously. This is what we’re doing to prevent this from happening again:

  • We are reviewing our testing and vetting procedures for future code changes and deployments carefully. We’re adding additional regression testing for more common and not-so-common scenarios before a code change can be released to the production systems.
  • We are looking into improving our reaction time to such incidents, giving customers better means to alert us and improving the processes for rolling back code changes.

We apologize to all our customers and Framer site visitors for the inconvenience and disruptions caused by this incident.

Posted Feb 28, 2024 - 12:06 CET

Resolved
The fix has fully rolled out, and we've confirmed it solves the rendering issue for the subset of affected sites.
Posted Feb 27, 2024 - 12:37 CET
Identified
We're rolling out a fix to mitigate the issue.
Posted Feb 27, 2024 - 12:16 CET
Update
We have found the root cause which was related to our gzip compression, which only affected a small subset of sites because most browsers use brotli compression.

Proxied sites (especially CloudFlare) were also mostly affected because they default to gzip.
Posted Feb 27, 2024 - 11:29 CET
Update
We are continuing to investigate this issue.
Posted Feb 27, 2024 - 10:48 CET
Investigating
We're seeing that some sites are not rendering correctly.

We're currently investigating the issue.
Posted Feb 27, 2024 - 10:47 CET
This incident affected: Hosting.