Monitoring - A fix has been implemented and we are monitoring the results.
Feb 4, 16:16 EST
Identified - Otava is aware of the remote code execution vulnerability in Apache Log4j. This vulnerability is also known as Log4shell and has the CVE assignment CVE-2021-44228

Background:
Log4j is a Java based logging audit framework. Log4j2 2.14.1 and below are susceptible to a remote code execution vulnerability where a remote attacker can leverage this vulnerability to take full control of a vulnerable machine.

IMPORTANT: This is a notice of a critical vulnerability and related threat; this is NOT a notice that we have been compromised as a result of this vulnerability.

Otava Security has been actively working with our third-party service partners and will apply critical patches immediately upon release. Otava has currently applied "known" security signatures for perimeter defense sensors, is actively scanning for such security vulnerabilities in all data centers and will continue active monitoring for all data centers for any bad actors and related malicious behavior.

If Otava does not manage your infrastructure devices we still recommend you follow vendor best practices and guidelines regarding patches for this vulnerability.

If you have questions about products or services that Otava can help provide please reach out to your account representative for further discussion.

Status updates can be viewed at Status.Otava.com where this incident is published. ALL customers can subscribe and receive real-time updates as we proceed.

Thank you,

Otava Operations

Dec 14, 19:01 EST
Investigating - VMware has released a notification of a critical vulnerability related to Apache. Otava is currently investigating the vulnerability and will be applying changes within the Otava network infrastructure in the attempt to mitigate any risk while Otava works with the vendor.

CVE: CVE-2021-44228

Dec 12, 16:17 EST
AU1 - Austin, TX Operational
Public Cloud Operational
Private Cloud Operational
Next Gen Cloud (Gen3) Operational
Networking Operational
Security Operational
Otava Cloud Backup (OCB) Operational
Otava Cloud Connect (OCC) Operational
Otava Cloud Backup for MS365 (OCB-MS365) Operational
DRaaS Operational
Data Center Facilities Operational
NV1 - Las Vegas, NV Operational
Public Cloud Operational
Private Cloud Operational
Next Gen Cloud (Gen3) Operational
Networking Operational
Security Operational
Otava Cloud Backup (OCB) Operational
Otava Cloud Connect (OCC) Operational
Otava Cloud Backup for MS365 (OCB-MS365) Operational
DRaaS Operational
Data Center Facilities Operational
IN1 - Indianapolis, IN Operational
Public Cloud Operational
Private Cloud Operational
Next Gen Cloud (Gen3) Operational
Networking Operational
Security Operational
Monitoring (OTMonitor) Operational
Otava Cloud Backup (OCB) Operational
Otava Cloud Connect (OCC) Operational
Otava Cloud Backup for MS365 (OCB-MS365) Operational
DPaaS Operational
DRaaS Operational
Data Center Facilities Operational
MM1 - Flint, MI (Mid-Michigan) Operational
Public Cloud Operational
Private Cloud Operational
Networking Operational
Security Operational
Monitoring (OTMonitor) Operational
Otava Cloud Backup (OCB) Operational
Otava Cloud Connect (OCC) Operational
Otava Cloud Backup for MS365 (OCB-MS365) Operational
DPaaS Operational
DRaaS Operational
Data Center Facilities Operational
Networking Operational
MD1 - Metro Detroit, MI Operational
Private Cloud Operational
Networking Operational
Security Operational
Monitoring (OTMonitor) Operational
Otava Cloud Backup for MS365 (OCB-MS365) Operational
DPaaS Operational
DRaaS Operational
Data Center Facilities Operational
AA2 - Ann Arbor, MI Operational
Private Cloud Operational
Networking Operational
Security Operational
Monitoring (OTMonitor) Operational
DPaaS Operational
DRaaS Operational
Data Center Facilities Operational
VA4 - Ashburn, VA Operational
Public Cloud Operational
Private Cloud Operational
Networking Operational
Security Operational
Data Center Facilities Operational
MA3 - Manchester, UK Operational
Public Cloud Operational
Private Cloud Operational
Networking Operational
Security Operational
Otava Cloud Connect (OCC) Operational
Data Center Facilities Operational
DEN1 - Denver, CO Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMWare Operational
Data Center Facilities Operational
NYC1 - New York, NY Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
DAL1 - Dallas, TX Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
PHX1 - Phoenix, AZ Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
CLE1 - Cleveland, OH Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
DUS1 - Dusseldorf, Germany Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
FRA1 - Frankfort, Germany Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
LA1 - Los Angeles, CA Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
SF1 - San Fransico, CA Operational
vCloud Operational
Networking Infrastructure Operational
Internet Backbone Operational
Security Operational
Otava Cloud Connect (OCC) Operational
DRaaS powered by Veeam Operational
DRaaS powered by VMware Operational
Data Center Facilities Operational
Client Services Operational
OTPortal (portal.otava.com) Operational
Partner Portal (partner.otava.com) Operational
Support Portal (support.otava.com) Operational
Website (otava.com) Operational
VMware Horizon View (apps.otava.com) Operational
Phone System Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Maintenance: Zerto Virtual Manager (ZVM) Patching & Restart

Maintenance Classification: Scheduled

Start Time: May 30, 2022 at 02:00 AM ET.

End Time: May 30, 2022 at 08:00 AM ET.

Type: Scheduled maintenance

Service Impacting: Yes

Description: Otava will be performing a scheduled patching & restart to the Zerto Virtual Manager (ZVM) server.

This notification is establishing a regular scheduled patching and the restart of the ZVM.

During the period of the maintenance the following will be true:

· Replication for your protected systems will NOT be impacted.

· Partial access to the Zerto user interface will not be available during this time period, however, because we will only be performing the maintenance on one ZVM at a time, an emergency DR failover is still possible.

· If during this time you need assistance with a DR failover, please contact support.

Estimated Downtime: During the maintenance period access to the Zerto user interface will be unavailable for an estimated 3 Hours. Replication will NOT be impacted by this event.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

Posted on May 2, 03:52 EDT
Past Incidents
May 27, 2022
Completed - The scheduled maintenance has been completed.
May 27, 06:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 27, 01:00 EDT
Scheduled - Start Time: 5/27 - 1:00 AM EDT

End Time: 5/27 - 6:00 AM EDT

Type: ISP network optimization

Service Impacting: Yes

Description: The ISP is performing network optimization.

Estimated Downtime: Up to 2 seconds as the circuit gracefully fails over to the redundant circuit.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 17, 16:59 EDT
May 26, 2022
Completed - The scheduled maintenance has been completed.
May 26, 06:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 26, 01:00 EDT
Scheduled - Start Time: 5/26 - 1:00 AM EDT

End Time: 5/26 - 6:00 AM EDT

Type: ISP network optimization

Service Impacting: Yes

Description: The ISP is performing network optimization.

Estimated Downtime: Up to 2 seconds as the circuit gracefully fails over to the redundant circuit.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 17, 16:55 EDT
Completed - The scheduled maintenance has been completed.
May 26, 04:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 26, 02:00 EDT
Scheduled - Start Time: 5/25 - 11:00 PM EDT

End Time: 5/26 - 02:00 AM EDT

Type: Firewall Firmware Upgrade

Service Impacting: Yes

Description: Otava network engineers will be upgrading the Infrastructure firewalls at MM. The upgrade is a three-step code upgrade that will cause the firewall in the HA stack to fail-over three times during the upgrades resulting in a few moments of packet-loss. DPaaS jobs could be interrupted during the upgrade, but should restart once complete.

Estimated Downtime: Three periods of up to 5 seconds of packet loss while firewalls fail-over.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 26, 00:48 EDT
Completed - The scheduled maintenance has been completed.
May 26, 00:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 25, 23:00 EDT
Scheduled - Start Time: 5/25 - 11:00 PM EDT

End Time: 5/25 - 11:59 PM EDT

Type: Firewall Firmware Upgrade

Service Impacting: Yes

Description: Otava network engineers will be upgrading the Infrastructure firewalls at MM. The upgrade is a three-step code upgrade that will cause the firewall in the HA stack to fail-over three times during the upgrades resulting in a few moments of packet-loss. DPaaS jobs could be interrupted during the upgrade, but should restart once complete.

Estimated Downtime: Three periods of up to 5 seconds of packet loss while firewalls fail-over.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 23, 08:45 EDT
May 25, 2022
Update - The scheduled maintenance has been completed.
May 25, 12:31 EDT
Completed - The scheduled maintenance has been completed.
May 25, 12:31 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 25, 06:30 EDT
Scheduled - Otava engineers are establishing a maintenance window to perform patching on our Otava Cloud Connect infrastructure at our following datacenter locations :
AU1-Austin
NV1-Las Vegas
MA3-Manchester
IN1-Indianapolis
MM1-Flint
These patches are required in order to continue to provide a secure and compliant environment.

Service Impacting: Yes - Running backup jobs may fail or be cancelled, new jobs may not start until all services are upgraded and maintenance is complete.

We will kindly ask you to check your backup jobs and restart them if needed once the maintenance window is complete.

Estimated Downtime: Intermittently throughout the maintenance window

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 2, 03:50 EDT
May 24, 2022
Completed - The scheduled maintenance has been completed.
May 24, 17:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 24, 08:00 EDT
Scheduled - Start Time: 5/24/22 - 8:00 AM EDT

End Time: 5/24/22 - 5:00 PM EDT

Type: UPS Annual Preventative Maintenance

Service Impacting: Low

Description: Otava engineers, working with vendor engineers, are scheduling a maintenance window to perform preventative maintenance on the Uninterruptible Power Supply.

Estimated Downtime: Due to the redundant nature of the design, no impact on power is expected.
If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

MD1 - Metro Detroit, MI - Building Systems

May 12, 08:59 EDT
May 23, 2022

No incidents reported.

May 22, 2022

No incidents reported.

May 21, 2022
Completed - The scheduled maintenance has been completed.
May 21, 16:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 21, 14:00 EDT
Scheduled - Start Time: 5/21 - 2:00 PM EDT

End Time: 5/21 - 4:00 PM EDT

Type: Firmware Update

Service Impacting: Low

Description: Otava and Vendor engineers are performing firmware updates on storage systems for added resiliency and security.

Estimated Downtime: No impact expected.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 16, 14:37 EDT
Completed - The scheduled maintenance has been completed.
May 21, 16:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 21, 14:00 EDT
Scheduled - Start Time: 5/21 - 2:00 PM EDT

End Time: 5/21 - 4:00 PM EDT

Type: Firmware Update

Service Impacting: Low

Description: Otava and Vendor engineers are performing firmware updates on tier 2 storage systems for added resiliency and security.

Estimated Downtime: No impact expected.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 13, 16:15 EDT
May 20, 2022

No incidents reported.

May 19, 2022
Resolved - This incident has been resolved.
May 19, 20:13 EDT
Monitoring - vCenter is back up and we're continuing to monitor. Please reach out to Otava support if you have any questions.
May 17, 23:47 EDT
Investigating - Otava Engineers are currently investigating an outage of the AU-PubVC3 vCenter. Running VMs are not affected as vCenter only provides management functions. We will update you as we have more information.
May 17, 23:09 EDT
Completed - The scheduled maintenance has been completed.
May 19, 20:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 19, 14:01 EDT
Scheduled - Start Time: 5/19 - 2:00 PM EDT

End Time: 5/19 - 8:00 PM EDT

Type: Urgent switch replacement

Service Impacting: Low

Description: Otava engineers will be replacing a switch in the Gen3 environment.

Estimated Downtime: No downtime expected.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 18, 16:26 EDT
Completed - The scheduled maintenance has been completed.
May 19, 18:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 19, 15:00 EDT
Scheduled - Start Time: 5/19 - 3:00 PM EDT

End Time: 5/19 - 6:00 PM EDT

Type: Urgent vCenter STS Certificate Troubleshooting

Service Impacting: Low

Description: Otava engineers will be troubleshooting vCenter STS Certificates in sCloud AU-PubVC3.

Estimated Downtime: Maintenance should not be impactful however, VM management could be interrupted during the vCenter reboot.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 18, 12:03 EDT
Completed - The scheduled maintenance has been completed.
May 19, 17:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 19, 08:02 EDT
Scheduled - Start Time: 5/19/22 - 8:00 AM EDT

End Time: 5/19/22 - 5:00 PM EDT

Type: Generator Preventative Maintenance

Service Impacting: Low

Description: Otava and vendor engineers will be performing routine annual minor maintenance inspections on the backup generators at AA2. We do not anticipate any impact as a result of this work, in the spirit of transparency we wanted to make you aware of the maintenance. Due to redundant power systems, no impact is expected.

Estimated Downtime: No impact expected If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

AA2 - Ann Arbor, MI - Building Systems

May 5, 13:20 EDT
Completed - The scheduled maintenance has been completed.
May 19, 08:01 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 17, 08:01 EDT
Update - We will be undergoing scheduled maintenance during this time.
May 11, 12:59 EDT
Scheduled - Mid-Michigan Data Center Annual Major Generator Inspection

Start Time: 5/17/22 - 8:00 AM EDT

End Time: 5/18/22 - 5:00 PM EDT

Type: Annual Major Generator Inspection and Load Bank Testing

Description: Otava and vendor engineers will be performing routine maintenance on the backup generators at the Mid Michigan data center. During the maintenance, Otava will load test each generator one at a time and perform preventative maintenance inspections. We do not anticipate any impact as a result of this work, in the spirit of transparency we wanted to make you aware of the maintenance. Due to redundant power systems, no impact is expected.

Estimated Downtime: No impact expected If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

MM1 - Genesee County, MI - Data Center Facilities

May 11, 12:57 EDT
May 18, 2022
Resolved - Start Time: 5/18/22 - 4:08 PM EDT

End Time: 5/18/22 - 6:25 PM EDT

Type: Utility Power Outage

Service Impacting: Low

Description: DTE reported equipment failure upstream from the data center.

Estimated Downtime: Due to the redundant nature of the data center design no impact.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 18, 18:37 EDT
Completed - The scheduled maintenance has been completed.
May 18, 12:30 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 18, 06:31 EDT
Scheduled - Otava engineers are establishing a maintenance window to perform patching on our Otava Cloud Backup infrastructure at our following datacenter locations :
AU1-Austin
NV1-Las Vegas
IN1-Indianapolis
MM1-Flint
These patches are required in order to continue to provide a secure and compliant environment.

Service Impacting: Yes - Running backup jobs may fail or be cancelled, new jobs may not start until all services are upgraded and maintenance is complete.

We will kindly ask you to check your backup jobs and restart them if needed once the maintenance window is complete.

Estimated Downtime: Intermittently throughout the maintenance window

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 2, 03:49 EDT
May 17, 2022
Completed - The scheduled maintenance has been completed.
May 17, 15:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 17, 14:00 EDT
Scheduled - Start Time: 5/17 - 2:00 PM EDT

End Time: 5/17 - 3:00 PM EDT

Type: Firewall lifecycle management

Service Impacting: Yes

Description: Otava engineers will be performing lifecycle maintenance on the DPaaS firewalls.

Estimated Downtime: Otava will be unable to perform restores during the maintenance window. Also, clients may experience backup failures during the maintenance window. For any failed backup jobs, the Otava Operations team will run on-demand backup jobs after the maintenance to maintain backup retention.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 10, 16:28 EDT
Completed - The scheduled maintenance has been completed.
May 17, 14:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 17, 13:00 EDT
Scheduled - Start Time: 5/17 - 1:00 PM EDT

End Time: 5/17 - 2:00 PM EDT

Type: Firewall lifecycle management

Service Impacting: Yes

Description: Otava engineers will be performing lifecycle maintenance on the DPaaS firewalls.

Estimated Downtime: Otava will be unable to perform restores during the maintenance window. Also, clients may experience backup failures during the maintenance window. For any failed backup jobs, the Otava Operations team will run on-demand backup jobs after the maintenance to maintain backup retention.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 10, 16:26 EDT
May 16, 2022
Completed - The scheduled maintenance has been completed.
May 16, 19:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 16, 18:02 EDT
Scheduled - Start Time: 5/16 - 6:00 PM EDT

End Time: 5/16 - 7:00 PM EDT

Type: Urgent vCenter STS Certificate Update

Service Impacting: Low

Description: Otava engineers will be updating the vCenter STS Certificates in sClouds AU-PubVC3 and LV-PubVC3.

Estimated Downtime: Maintenance should not be impactful however, VM management could be interrupted during the vCenter reboot.

If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

May 16, 13:29 EDT
Completed - The scheduled maintenance has been completed.
May 16, 17:00 EDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 16, 08:00 EDT
Scheduled - Start Time: 5/16/22 - 8:00 AM EDT

End Time: 5/16/22 - 5:00 PM EDT

Type: Annual Major Generator Inspection and Load Bank Testing

Description: Otava and vendor engineers will be performing routine maintenance on the backup generators at MD1. During the maintenance, Otava will load test each generator one at a time and perform preventative maintenance inspections. We do not anticipate any impact as a result of this work, in the spirit of transparency we wanted to make you aware of the maintenance. Due to redundant power systems, no impact is expected.

Estimated Downtime: No impact expected If you have any questions or concerns regarding this maintenance please contact us by logging into https://support.otava.com or calling 877-740-5028, option 3.

MD1 - Metro Detroit, MI - Building Systems

May 2, 15:45 EDT
May 15, 2022

No incidents reported.

May 14, 2022

No incidents reported.

May 13, 2022

No incidents reported.