BXTN Server Outage Notification

Event Description

BXTN was a victim of a ransomware attack at a service provider's datacenter.

Impacts

  • Both our PowerBid and Private Planroom websites, as well as our Nightly Email Update and Construction Bulletin email systems, were down for over 6 days; our public-facing website (where you are reading this message) was unaffected.

Details

  • 12/24/2019 5:00 AM EST: We initially became aware of the website outage. We later found out that the attack actually began around 2:00 AM EST.
  • 12/24/2019 7:30 AM EST: We were informed by our service provider that the outage could possibly be a ransomware attack.
  • 12/24/2019 12:30 PM EST: We were informed by our service provider that we are definitely one of their affected customers.
  • 12/25/2019 2:00 PM EST: We were informed by our service provider that all our files have been encrypted by the attackers and are completely inaccessible at this time.
  • 12/26/2019 12:30 PM EST: We were informed by our service provider that they have started the process of restoring services of affected customers. They cannot give us an ETA on the recovery of our particular servers, only that they believe it will be sometime within the next few days.
  • 12/27/2019 9:00 AM EST: We now have access to a backup copy of construction documents - please give us a call at 865-525-0443 and we can email you copies of any documents that you need. 
  • 12/29/2019 4:30 PM EST: We have restored access to some internal systems, but our PowerBid and Private Planroom websites remain down. Our service provider has made significant progress today, so we are hopeful to have more positive news to report soon. You can learn more information about this specific attack by reading these articles at Krebs on Security and CRN Magazine.
  • 12/30/2019 9:15 AM EST: We have completed a full recovery of all our systems with zero data loss and are completely back online. The Nightly Email Update system will be reactivated on 12/31 at 2:00 AM EST; that email will contain all project information and updates from the day before the attack on 12/23 as well as all new projects, addenda, etc. that our staff have processed throughout the day today. The next Construction Bulletin will also be emailed on the morning of 12/31 as usually scheduled.

We sincerely apologize for any inconvenience this outage has caused.

Final Update: 12/30/2019 9:25 AM EST