Notice #382348
Recovering

Global Incident

Began:

There is a global wide incident related to Crowdstrike / Microsoft that is causing outage for many online services and systems. Tech-Keys does not utilize Crowdstrike and your core systems should be unaffected. However, you may run into other online systems throughout the day that are experiencing outages.

Future updates will be provided if available, or if we become aware of commonly utilized systems that are affected.

Notice #382353
Recovering

Net Health Outage

Began:

NetHealth: Our teams continue to work on both EAST and WEST data centers for Optima and Azure for all applications.

Although some clients may be able to log into the applications now, services have not been fully restored yet for those still marked as major or partial outage. Please proceed with caution (creating, documenting and saving data) until we communicate that all services are fully restored for that product.

NetHealth has reported that Value-Based Care Solutions and FOTO Analytics are now fully operational however we are still seeing some reported issues with sites loading. We will continue to monitor the situation and provide regular updates as soon as we receive any other information.

NetHealth: Our internal support team is also impacted by this Microsoft outage. There will be a slow response to calls or emails until the outage is resolved.

Our IT teams are working to implement the necessary fixes to restore services.

For more updates visit https://status.nethealth.com/

  • Tech-Keys Support
    • Email Support
    • Phone Support
    • Web Support
  • Microsoft 365 Services

    Azure AD

    • Authentication (Sign-in)
    • Teams
    • Azure Services
  • Email Services
    • Exchange Online (Office 365)
    • Mimecast (Email Filtering)
    • Avanan (Email Filtering)
    • Exclaimer (Email Signatures)
  • File Services
    • SharePoint (Office 365)
    • OneDrive for Business (Office 365)
  • Backup Services
    • Datto Backup
    • SkyKick Office 365 Backup
  • EMARsafe
    • EMARsafe Portal
    • EMARsafe Backup Download
  • Remote Access to Computer
  • Partner Services
    • Voip-Keys VoIP Services
  • Status Page
  • Microsoft Windows
  • Cloud
    • Internet
    • Power
  • PointClickCare
  • Printing
    • Printix
  • Internet Provider
    • Comcast
    • Verizon
    • Spectrum
  • Password Management
    • Keeper
Previous Incidents

[Resolved] Users may intermittently be unable to update or cancel calendar items in the Outlook desktop client

Began: Ended: Duration:
  • Email Services
  • Exchange Online (Office 365)

Microsoft: We’ve added the required components to the fix and additional testing has been completed. The fix is in the process of deployment and is on track to saturate within the next three days. We’ll be carefully monitoring telemetry in the impacted environments throughout the roll out.

As a potential workaround, users can try updating meetings using the new Outlook desktop client or Outlook on the web: https://outlook.office365.com/calendar/view/day

Microsoft: We’ve encountered a delay in the ongoing roll out of the fix. We estimate that the fix will now saturate all impacted environments by Friday, July 12, 2024, by approximately 3:00 AM UTC.

As a potential workaround, users can try updating meetings using the new Outlook desktop client or Outlook on the web: https://outlook.office365.com/calendar/view/day

Microsoft: We’ve closely monitored the service telemetry data since our last update and have verified that the Outlook calendar update and cancellation errors have significantly decreased. We’ll continue evaluating related activity to verify if any new errors appear while also examining if newly reported issues are related to previously affected calendar items, where conflicts or duplicate meetings were present.

Additionally, the investigation into the root cause of the issue remains ongoing and more information will be shared once it is confirmed.

Microsoft: We've now resolved the incident. Thanks for your patience.

[Resolved] Azure Services

Began: Ended: Duration:
  • Microsoft 365 Services
  • Azure Services

From Microsoft: Starting at approximately 21:56 UTC on 18 Jul 2024, a subset of customers may experience issues with multiple Azure services in the Central US region including failures with service management operations and connectivity or availability of services.

Current Status: We are aware of this issue and are actively investigating.

From Microsoft: Current Status: We are aware of this issue and have engaged multiple teams. We’ve determined the underlying cause. A backend cluster management workflow deployed a configuration change causing backend access to be blocked between a subset of Azure Storage clusters and compute resources in the Central US region. This resulted in the compute resources automatically restarting when connectivity was lost to virtual disks. We are currently applying mitigation. Customers should see signs of recovery at this time as mitigation applies across resources in the region.

Microsoft: We've now resolved the incident. Thanks for your patience.

No further notices from the past 7 days.