20230127 - Infinity disruptions, unexpected platform crash - Resolved

The issue is resolved as of 14:47. We keep monitoring and will take measures to avoid this from happening in the future. We apologize for the inconvenience. 

The issue is resolved as of 14:47. We keep monitoring and will take measures to avoid this from happening in the future. We apologize for the inconvenience. 

---

Services are back to normal, and this is the impact measured :

Infinity queues: no calls distributed during about 7 minutes on 50% of tenants. qboard/cdc clients disconnected for 2 minutes.
Enreach Contact: no calls distributed during about 7 minutes on 50% of tenants. client applications disconnected for 2 minutes.

Engineers are still investigating and platform constantly monitored, but the issue is solved.

---

We are having Infinity platform disruptions at the moment. Unexpectedly, the platform has crashed causing the platform reboot, and therefore all the services including Enreach Contact are affected.
The impact has been 50% of the customers having problems to make and receive calls. Investigating the root cause at the moment.

Issue is considered restored since 14:47 but we will keep the monitopring opened until root cause is found.

We will update you with a short technical summary of the cause after the resolution.

Our apologies for the inconvenience.

from:
27/01/2023 - 14:40
till:
27/01/2023 - 14:47
Last update: 27/01/2023 - 15:59
Read More
20230127 - Infinity queue's CDC, Qboard, Enreach Contact - Solved

Problems with the webservices for infinity have been resolved.

27/01/2023, 09:33
Problems with the webservices for infinity have been resolved.

----

27/01/2023, 08:38
Problems seem to be network related and impacting logging into/using webservices. Enreach Contact also seems to have been impacted

------

We are currently experiencing a disruption in the service infinity. This disruption seems to be impacting Coligo Qboard and logging in and out via Coligo Desktop on Infinity Queue's.
Our engineers are investigating the problem.

We cannot give an expected resolution time at this moment.
We will provide you with a new update on the situation at 09:00. There are no known workarounds at this moment.

Our apologies for the inconvenience.

from:
27/01/2023 - 07:30
till:
27/01/2023 - 09:15
Last update: 27/01/2023 - 09:33
Read More
20230125 - office365 issue - Microsoft Resolved

All services are back online

 

 

Last update: 25 January 2023 at 13:16

All services are back online

-------------

Users may be unable to access multiple Microsoft 365 services  

Microsoft reported the following:

User impact: Users may be unable to access multiple Microsoft 365 services.   More info: We've received reports that the following services are impacted: -Microsoft Teams -Exchange Online -Outlook -SharePoint Online -OneDrive for Business -Microsoft Graph  Current status: We've identified a potential networking issue and are reviewing telemetry to determine the next troubleshooting steps.  

Scope of impact: Any user serviced by the affected infrastructure may be unable to access multiple Microsoft 365 services.

 

 

from:
25/01/2023 - 09:18
till:
25/01/2023 - 13:10
Last update: 25/01/2023 - 13:17
Read More
20230116 - Network problem with impact on all services - Resolved

16/01/2023, 15:16
All services have been restored around 14:40. Network team is investigating the rootcause

16/01/2023, 15:16
All services have been restored around 14:40. Network team is investigating the rootcause

-----

16/01/2023, 14:42
Network disruption seems to be decreasing. Services are restoring. Engineers are investigating and monitoring

-----

We are currently experiencing a disruption in all services
Our engineers are investigating the problem.

We cannot give an expected resolution time at this moment.
We will provide you with a new update on the situation at 15:00

We will update you with a short technical summary on the cause after the resolution.

Our apologies for the inconvenience.

from:
16/01/2023 - 14:29
till:
16/01/2023 - 14:40
Last update: 16/01/2023 - 15:17
Read More
20230113 - Devices logged out from extensions on MV3 - Solved

The display updates have been all send and the delays have been resolved.

13/01/2023, 12:25
The display updates have been all send and the delays have been resolved.

----

13/01/2023, 12:08
Display updates are being send now, but can cause a delay (minutes) for users that manually login/logout via 901/902 or 8801/8802
----

13/01/2023, 11:20
All extensions that were logged out have been restored with the backup. Calls are being delivered again to the devices. The display have not been updated. For this 901 / 8801 needs to be used.
Engineers are looking into forcing a display update.

----

13/01/2023, 10:58
When a customer calls 901 / 8801 to login their extension, they can call and be called immediately. We advise to do this and not wait on the restore via the backup.

----

13/01/2023, 10:38
Engineers created a script to restore the extensions from a backup. The backup is almost ready to start the script.

----

13/01/2023, 09:17
Engineers are looking into a way to restore the extension logins in the database. There is no ETA yet, so our recommendation is still to do this manually via 901 / 8801

----

13/01/2023, 09:00
The cause has been found and affected extensions logins older then 30 days. The responsible script will be hotfixed during the day to prevent this from happening again.

----

13/01/2023, 08:00
We are currently experiencing a disruption in the service Managed Voice 3. This disruption seems to be impacting extension logins on the channels and customer need to dial 901 or 8801 to log back in.
Our engineers are investigating the problem.

We cannot give an expected resolution time at this moment.
We will provide you with a new update on the situation at 09:00. There are no known workarounds at this moment except logging back in.

We will update you with a short technical summary on the cause after the resolution.

Our apologies for the inconvenience.

from:
13/01/2023 - 07:00
till:
13/01/2023 - 12:25
Last update: 13/01/2023 - 12:25
Read More
20230111 - MV3 - delay in CDR's - We currently face a delay in the creation of CDR's for MV3. This also means call history events are delayed. Engineers are working on a fix. [UPDATE 2]

CDR's are slowly being visible again from CDR view in Operator. We still have some delay.
We are still monitoring actively,estimating that all CDRs are fully recoverable and delay mitigated in the upcoming hours.
This service announcement will be updated once we have the official confirmation that everything is back to normal from our engineers.

CDR's are slowly being visible again from CDR view in Operator. We still have some delay.
We are still monitoring actively,estimating that all CDRs are fully recoverable and delay mitigated in the upcoming hours.
This service announcement will be updated once we have the official confirmation that everything is back to normal from our engineers.

---

Impact is limited to Managed Voice 3. Fix is successful. Delay is still being processed, it is taking more time than initially estimated but in progress.
Next update soon.

---

We are currently experiencing a disruption in the service Operator. This disruption seems to be impacting CDR's are not being received in the CDR view and all customers are affected.
Fix is already rolled out. Within minutes all delayed CDR's should be processed and recovered.

We expect a fast resolution and CDR recovering in the next minutes.
We will provide you with a new update on the situation in 15 minutes or before if we have confirmation from our engineers is fixed..

We will update you with a short technical summary on the cause after the resolution.

Our apologies for the inconvenience.

from:
11/01/2023 - 11:39
till:
Unknown
Last update: 11/01/2023 - 12:25
Read More
20230109 - [Mobile] - Packet loss issues affecting connections to KPN, Vodafone and potentially VoLTE calls/data - SOLVED

The root cause of the disturbance has been found and will be resolved during an emergency maintenance in the upcoming night. Since any customer impact is already mitigated, this incident will be closed.

[10:51]

The root cause of the disturbance has been found and will be resolved during an emergency maintenance in the upcoming night. Since any customer impact is already mitigated, this incident will be closed.

[09:39]

Impact mitigating measures have been taken and from a customer point of view any issues they might have experienced should have been resolved.

[]

We are currently experiencing slight packetloss in our mobile domain because of an unknown disturbance in our network part. This packetloss might impact call quality and/or the availability of mobile data. This is currently under investigation by a joint mobile/network team.

We will provide you with a new update on the situation at 09:45. There are no known workarounds at this moment.

Our apologies for the inconvenience.

from:
09/01/2023 - 08:49
till:
09/01/2023 - 10:51
Last update: 09/01/2023 - 10:51
Read More
20230102 - Operator slow or unresponsive - resolved

02/01/2023, 14:40
Engineers have found and resolved the issue

02/01/2023, 14:40
Engineers have found and resolved the issue

----------

We are currently experiencing a disruption in the service Operator. This disruption seems to be impacting response time of Operator
Our engineers are investigating the problem.

We cannot give an expected resolution time at this moment.
We will provide you with a new update on the situation at 15:00. There are no known workarounds at this moment.

Our apologies for the inconvenience.

from:
02/01/2023 - 14:33
till:
02/01/2023 - 14:40
Last update: 02/01/2023 - 14:40
Read More
Subscribe to Enreach interruptions RSS