October 18th 2019 Incident Post-mortem
Published on October 24th, 2019
First, we, the team at Conference Badge, want to sincerely apologize for this incident. In this post-mortem, you will learn what happened and what measures were taken so that such an incident never happens again.
On Friday October 18th 2019 at 3:20PM ET, a security vulnerability was identified in our system. It was brought to our attention by a 3rd party security researcher who has collaborated with companies such as Spotify, Tumblr and Twitter.
On Monday October 21st 2019 at 9:46AM ET, we sent an email to affected users of our website disclosing the nature and state of the issue.
The vulnerability was due to an Amazon S3 bucket (a file hosting service) that was misconfigured as public. This allowed listing and downloading files from the bucket.
In this bucket were hosted PDF files of name badges that were generated in order to be printed and worn by event attendees. For the vast majority of badges, the information printed on these badges include the attendee’s First name and Last name. In some cases, extra fields like Job title and Company were also included, as well as QR codes encoded in the vCard format, which could include, if provided by the event organizer, the attendee’s Email address and Phone number. This affected all orders placed on Conference Badge in the past and up to October 18th 2019 at 4:35PM ET.
For a very small subset of users, the S3 bucket also included a copy of spreadsheets that failed to be imported by our servers when uploaded by the user. Upon such import failures, these problematic spreadsheets were stored in the bucket for further analysis and improvement of the system. Users affected by this specific case have been contacted with a distinct disclosure email.
It is worth stressing that NO logins, passwords, billing information or credit card information were included in the exposed data. Our database was never exposed at any point, only an S3 bucket was.
Amazon S3 Server Access Logging was not enabled on this bucket, so it is impossible for us to determine which files have been downloaded and from what source. Customers who want to know the exact information that was included on their past name badges may email us at firstname.lastname@example.org requesting a copy of their PDF files. We are keeping a copy of these PDFs available for this purpose until November 18th 2019. On this date, we will permanently delete the files. You may also contact us to request immediate deletion of your past name badge PDFs.
What We Did
The following actions were taken to secure Conference Badge user data since the breach was initially reported.
Secure bucket access
Approximately 30 minutes after receiving the initial report, we manually removed List permissions from the bucket through the AWS console. However, we quickly realized a seemingly inoffensive line in our code base was actually re-enabling List permissions on the bucket upon every file upload. This fact was discovered around 1 hour after the breach was reported. The code was immediately updated to stop this, and the bucket policy was changed again to disable List permissions.
Implement a limited AWS IAM role for our servers
To prevent any code from ever changing the bucket permissions again, we introduced a new IAM role used by our servers to interact with the bucket. This role will prevent the code base from ever performing administrative actions on the bucket. It will be limited to the following actions: downloading files, uploading files, deleting files, and presigning URLs to files.
Purge data from past events
When we were informed of the vulnerability, we promptly deleted all past event data from our database. The database was not compromised at any point; this was a precautionary measure.
On the following business day, we implemented and deployed a daily background process to eliminate all data related to events that ended 30 days prior. This process deletes attendee data from our database and badge PDFs from our S3 bucket.
We offer a service to event organizers for easily producing name badges; we have never been in the business of accumulating customer data and never will be. Hosting the minimal amount of data required to provide our service is the first line of mitigation against any potential data leak.
Use presigned URLs to serve badge PDF files
We previously were using long unguessable URLs to distribute PDF files to end-users. This means once a user had access to a file URL, they could continue requesting and distributing the file via the same URL. We now serve these files using S3 presigned URLs that automatically expire after a short time.
Enable Server Access Logging on our S3 bucket
An access log record contains details about the requests that are made to a bucket. This information includes the request type, the resources specified in the request, and the time the request was received at. Logging will help us investigate any further incidents and let us know what/when files are requested and from where.
Implement AWS CloudTrail and CloudWatch alerts
We enabled AWS CloudTrail to maintain a log of all activity happening on our AWS account as well as CloudWatch event rules to instantly alert our team upon any administrative action, including AWS Console sign-ins, S3 bucket creation / modification, etc.
Responsible Disclosure Program
We published a Security page on our website, including information about our responsible vulnerability disclosure program. We were already welcoming responsible disclosures and have offered rewards to security researchers, but this page now makes things clear and official.
What We Will Do
To further strengthen the security of our system, the following actions will be taken in the near future.
Annual security assessment
We’re in the process of partnering with an infosec firm to do an extensive security assessment of the Conference Badge website. The first assessment should be completed by the end of November. The letter of assessment will be made publicly available on our Security page when completed. The assessment will include an application penetration test and a cloud security audit. Going forward, such security assessments will be performed on a yearly basis.
User interface change to prevent users from importing unneeded data
When importing attendee lists in Conference Badge, the application will explicitly ask users what columns are needed to design their badges. Only those columns will be imported in our database.