Got an email yesterday with the following
Hello,
There has been an ongoing issue with our website that allowed some customer data to be viewed in the source code of public system76.com pages. My team is very security-minded and an incident of this nature is of significant concern.
Before discussing the details I want to emphasize the following: your system76.comaccount credentials and credit card numbers were not compromised at any point.
When did this occur?
It started on March 17th, 2022 and was resolved on May 26th, 2023.
What was potentially exposed?
User account information and order details including
- names
- physical addresses
- email addresses
- phone numbers
Again, your system76.com credentials and credit card were not compromised at any point. It was not the contents of our entire database, but a random selection of what happened to be in our web servers’ memory at the time. It is not possible to determine which users had their information visible over the course of time when this leak was occurring.
Can any of this data still be accessible?
This data was embedded directly into the source code of our website, so it will be included in any archives that were taken during this time period. We have already been in contact with the internet archive, and they have removed their data, but there are other potential places where it could have been stored.What pages on our site contained this data?
Due to the nature of the flaw, the data was being included to some extent on every page on our site.Why was there a delay before resolving this and making an announcement?
We wanted to do our best to clean up any residual customer data that was easily accessible before disclosing the nature of this leak. This involved working with the internet archive to delete their copy of the data before public disclosure.Is there any evidence that any of this data was misused?
To date, the only indications of misuse that we have are reports of spam emails being sent to customers.Why did it take so long to discover?
There were a number of factors that play into this, but they do not change the fact that it should not have been happening for over a year.
- You would need to look at just the right location in a very long single-line string. This is a common method for minimizing javascript and most developers are used to seeing such excessively long lines on a website. There’s usually not much reason to look into them in more detail.
- It was not something a developer would ever see in their local environment. It was also not noticeable in our staging environment due to the difference in activity there.
- It is entirely unexpected to see sensitive data exposed in this fashion.
What are the technical details?
One of the technologies that we use on our website is server-side rendering with nuxt. Several years ago a caching layer was implemented so that the server-side rendering wouldn’t have to ask our backend API every time it needed the same information. It worked fine, but front-end technologies shifted and the way it was implemented became outdated.
Last year our team made an update to a seemingly unrelated piece of code, to fix a deprecation message no less. However, this exposed the initial implementation of this cache to a new style of handling state, which it had not been written for and was not prepared to handle.
So what happened is the following: state that shouldn’t have been shared was shared between sessions. This is one of the major pitfalls of using server-side rendering in a web application. Unlike an API, the code in an SSR application doesn’t always have distinct boundaries between what happens on a single client vs. what happens on the server.What is being done to prevent similar leaks in the future?
For the time being we are running an automated scan to alert us if sensitive data is ever found on our website again. We are also continually performing manual inspections to make sure nothing else is in there that shouldn’t be, but we are confident that we have identified and fixed this issue.
In the long term, we are moving towards a static model (SSG instead of SSR) for our website, which will eliminate the majority of the risk of shared state.This is a mistake that we will learn from and we offer our sincerest apologies for leaking information that was entrusted to us. We deeply regret the incident and remain committed to safeguarding your information and trust.
If you have any questions or concerns that were not addressed here please reply to this email.
Thank you,
Mike Cifelli
Director of Web Engineering
System76