Partial Service Disruption
Cloud PBX, IP-PRI, MS Teams Direct Routing, SIP Trunking, TelNet UC with Webex
June 3, 2025 10:14AM EDT
[Resolved] Dear Valued Customer,
This communication is a follow-up to the service disruption experienced on June 2, 2025, impacting voice services and Cloud PBX features for a portion of our customer base.
Event Summary
TelNet experienced a voice service disruption that affected inbound and outbound call functionality, as well as Cloud PBX Auto Attendant responses. The issue began at approximately 11:05 AM, during which time customers reported symptoms such as calls failing to connect, ringing with no ability to answer, or reaching a busy signal after several seconds.
At approximately 11:17 AM, public internet routing appeared to stabilize, and general voice services began recovering without intervention. However, Cloud PBX Auto Attendant services remained impaired until 12:12 PM when full service was restored following manual intervention.
Incident Impact
Some customers were unable to place or receive voice calls during the incident window.
Cloud PBX customers, specifically those using Auto Attendants, may have received a busy signal or a failed call after a few seconds of ringing.
Root Cause
The disruption was triggered by a failover event between our Session Border Controllers (SBCs). Our active SBC was unable to reach its designated gateway - a situation likely caused by a broader (upstream) public internet routing issue. As designed, the SBC attempted to fail over to its standby counterpart. However, the standby SBC experienced the same reachability problem, resulting in continued service interruption.
Our network monitoring systems (PRTG) identified a significant increase in hop count across both of our internet backbone connections leading up to the event. This simultaneous shift across independent upstream providers strongly suggests an external internet routing anomaly beyond TelNet’s infrastructure.
At 11:17 AM, external routing appeared to normalize, allowing standard voice sessions to resume. However, Cloud PBX Auto Attendant services required manual intervention to restart a media server, which successfully restored those services by 12:12 PM.
Next Steps and Preventative Measures
To prevent recurrence, TelNet is taking the following actions:
Engaging with our internet backbone providers to gain additional insight into the upstream routing anomaly.
Evaluating additional internal monitoring and automated recovery mechanisms for media services during and after SBC failover scenarios.
We sincerely apologize for the disruption and appreciate your continued trust as we strengthen our systems and procedures to better serve you.
If you have any further questions, please contact our Support Team at 800.508.1254 Option 4 or email support@telnetww.com.
Sincerely,
TelNet Operations Team
June 2, 2025 12:29PM EDT
[Monitoring] Master Ticket: 00086291 – Service Restored
As of 12:12 PM, the issue impacting Cloud PBX Auto Attendant services has been resolved.
Due to the number of switchover events our SBCs experienced during this morning’s public internet routing instability, a media server required manual intervention to restore proper call/media handling. Once restarted, Auto Attendant functionality returned to normal.
We will continue monitoring the environment closely to ensure stability.
If you are still experiencing any issues, please don’t hesitate to contact our support team at support@telnetww.com or 800-508-1254, Option 4.
We appreciate your patience and understanding during this disruption.
June 2, 2025 12:11PM EDT
[Identified] Master Ticket: 00086291
Following this morning’s SBC-related event impacting voice sessions, we are now receiving additional reports specifically affecting our Cloud PBX customers. Some callers attempting to reach Auto Attendants are experiencing a “busy” signal after approximately 16 seconds of ringing.
Our team is actively investigating this issue and working to identify any relation to the earlier SBC event. We understand the critical nature of voice service and are treating this with high urgency.
The next update will be provided within 30 minutes, or sooner if new information becomes available.
If you need immediate assistance, please contact our support team at support@telnetww.com or call 800-508-1254, Option 4. Please be advised that call wait times may be elevated due to increased volume.
We appreciate your continued patience as we work to resolve the issue.
June 2, 2025 11:28AM EDT
[Investigating] Master Ticket: 00086291 – Service Impairment Update
We’ve identified an event involving one of our SBC (Session Border Controller) pairs that impacted active voice sessions. Affected customers may have experienced issues such as the inability to place or receive calls or calls that rang without the ability to answer.
The disruption appears to have cleared on its own around 11:17 AM without intervention. Our engineering team remains actively engaged and is continuing to investigate the root cause.
We will provide another update within the next 60 minutes, or sooner if new information becomes available.
If you’re still experiencing issues or have questions, please contact our support team at support@telnetww.com or call 800-508-1254, Option 4. Please note that we are experiencing higher than normal call volumes and hold times may be longer than usual.
We appreciate your patience and understanding.
June 2, 2025 11:07AM EDT
[Investigating] Master Ticket: 00086291
Hello TelNet Community, we want to bring attention that we are experiencing an influx of customer reports and are immediately triaging the issues to isolate any commonalities. We apologize for the disruption and we are working urgently to correct the problem. The next update will be within 30 minutes, or as new information becomes available.
If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4, please be aware that we are experiencing higher than normal call volume and you may experience a wait time.
