Active Incident

Voice Services

Cloud Contact Center

Operational

Cloud PBX

Operational

Digital Fax

Operational

IP-PRI

Operational

MS Teams Direct Routing

Operational

SIP Trunking

Operational

SMS

Operational

TDM-PRI

Operational

TelNet UC with Webex

Operational

Network Connectivity

Internet

Operational

SD-WAN

Operational

Data Center

Colocation Services

Operational

Data Center | Southfield

Operational

Contact Center

Contact Center Services

Operational

Portals

customerportal.telnetww.com

Operational

voiceview.telnetww.com

Operational

Incident Status

Operational

Components

Voice Services

Services

Cloud Contact Center, Cloud PBX, Digital Fax, IP-PRI, MS Teams Direct Routing, SIP Trunking, TelNet UC with Webex



March 13, 2024 4:48PM EDT
[Monitoring] We want to provide you with another update regarding the ongoing connectivity issues that some of our users have been experiencing. Our collaborative work with the impacted ISP continues, and their team is actively inspecting all areas of their network for potential issues that may affect packet size and/or transport protocol. This thorough examination is part of our joint commitment to identifying and addressing any challenges within their infrastructure. Please be advised that we have a few work arounds that have shown success for many users with smaller packet size relating to the UDP changes in traffic from the last update. If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4 relating case 00068881.

March 1, 2024 4:11PM EST
[Monitoring] We want to provide you with a significant update regarding the ongoing connectivity issues that some of our users have been experiencing. After careful investigation with our Network and Engineering teams, we identified that the issue undeniably lies within the impacted ISP's network. As you know, despite our continued efforts to engage with the ISP, the resolution process has been challenging. Our Findings / Confirming ISP Network Issues: During our in-depth testing, we identified that when customers migrated their services away from the impacted ISP (utilizing alternative solutions such as 4G/5G Hotspots or other ISPs), that these users experienced a restoration of their services - highlighting a direct correlation between the issue and the impacted ISP’s network. Recent Changes and Potential Improvement: In response to the challenges faced, we have implemented a change to our phones that has shown promising results in enhancing voice connectivity for several customers. Specifically, we modified the transport layer protocol for select customers from TCP to UDP. Understanding TCP vs. UDP: TCP (Transmission Control Protocol): Think of TCP like a reliable, ordered delivery service. It ensures that data packets are transmitted in a specific order and verifies that they are received correctly. This is beneficial for applications where every piece of data needs to arrive intact. UDP (User Datagram Protocol): UDP, on the other hand, is more like a speedy courier service. It delivers data packets quickly but without the same level of order or verification. How This Affects You: The change from TCP to UDP has resulted in a marked improvement in maintaining voice connectivity for the customers involved in the initial rollout. Based on this original feedback, it appears that the original TCP transmissions were being affected by potential filtering performed by the impacted ISP. By changing the transmission type to UDP, it appears that the voice traffic is not being affected to the same degree. It is, however, important to note that UDP (although expedient) may have reliability considerations, and our initial choice of TCP is based on its preferred reliability. Our intention would be to revert back to TCP as soon as the impacted ISP resolves their network issue, ensuring that the optimal method of transport is used. Encouraged that our customers are reporting an improvement in their services, we have extended the change to the remaining customers who reported trouble. Your Feedback Matters: We value your experience, and your feedback is crucial in helping us understand the full impact of this change. We kindly ask you to share your observations and insights with us. Has the modification positively impacted your voice connectivity? Are there any other aspects we should be aware of? Please reach out to our support team (at 800.508.1254 OPT 4) or support@telnetww.com with your feedback. We genuinely appreciate your cooperation and understanding as we work together to optimize your connectivity experience. Our goal throughout this issue has been neutral troubleshooting to effectively isolate the root cause of trouble - regardless of the area of responsibility. As a neutral party, we will strive to alleviate unneeded stress on our customers by owning repair efforts until the trouble is resolved. Thank you for your continued trust in TelNet Worldwide.

February 29, 2024 7:57PM EST
[Identified] After persistent efforts, we have successfully reached the escalation team at the impacted ISP, who have disclosed that a fiber cut occurred in their core network in the area on 2/27. This incident may have resulted in continued issues for customers, and the ISP is actively investigating the aftermath of the fiber cut. We appreciate your patience and understanding as the ISP works to address the underlying network challenge. Rest assured, we remain committed to staying on top of the situation and will continue to provide updates as we receive more information. We understand the impact this has on your services and sincerely thank you for your ongoing support. If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

February 27, 2024 3:51PM EST
[Identified] Despite our best efforts to address the matter, we regret to inform TelNet Voice customers that the impacted ISP has unfortunately declined our multiple requests for collaboration or investigation into the identified issue - the absence of expected returned SIP traffic. The core issue revolves around the challenge of proving a negative — the absence of anticipated traffic. In such situations, our ability to take direct action is limited, leaving us in a challenging position. We understand the frustration and inconvenience this may cause, and we want to assure you that we are actively exploring alternative avenues to find a resolution with the impacted ISP. If you have access to an alternative internet source (such as 4G/5G LTE not sourced from the impacted ISP), consider temporarily swapping your voice services to the alternate connection. This may help bypass the ongoing issue. If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

February 27, 2024 2:33PM EST
[Identified] We want to keep you informed about the ongoing connectivity issues impacting some of our users. Following our initial investigation, we've determined that all TelNet traffic successfully reaches our customers through all available routing pathways. However, there's a challenge with the expected return traffic that is specifically impacting users sharing a common ISP. It's crucial to understand that this issue is exclusive to users with a particular ISP, and unfortunately, TelNet lacks control over the routing and network configuration of other companies. Our team has initiated contact with the affected ISP to address the situation. While we face limitations in influencing external network configurations, we are very open to working directly with the impacted ISP. If any customers can connect us with the Network Operations Center (NOC) or a Network Technician from the affected ISP, we believe direct collaboration would be beneficial in resolving this matter. We appreciate your understanding and patience as we navigate through this situation and remain dedicated to providing updates and working towards a resolution. If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

February 27, 2024 11:08AM EST
[Investigating] Recently, we've observed an increased number of customer reports regarding intermittent issues such as dropped calls, no inbound or outbound calls, and loss of SIP Registration. Upon investigation, we've identified a common thread - that affected customers share the same Internet Service Provider (ISP). Our team has conducted traceroutes from several points in our network, all of which successfully reached the impacted ISP's network. These traces appear to be "dying" many hops within the ISP's network. This points us toward the possibility that the issue lies within their network infrastructure. TelNet is actively attempting to reach out to the impacted ISP to make them aware of this issue. We apologize to any TelNet customers affected by the ISP-related issues, and it's important to note that as these disruptions occur within another ISP's network - it is beyond our control as there is nothing we can do to directly address or resolve the challenges within their network. We appreciate your understanding and patience as we work collaboratively with the ISP to expedite a resolution. If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

History (Last 7 days)

Incident Status

Degraded Performance


Components

Voice Services, Network Connectivity


Services

Cloud Contact Center, Cloud PBX, Internet, IP-PRI, MS Teams Direct Routing, SIP Trunking, TelNet UC with Webex




March 13, 2024 1:59PM EDT
[Resolved] The traffic re-route resolved the issues relating to this event, 00068230. For 2 weeks after the change, we monitored no further issues with the carrier and had restored these paths for redundancy. We have had live traffic for an additional 2 weeks without issues. We are now closing this issue. If you, or any of your team members, are experience any troubles please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

February 14, 2024 3:44PM EST
[Monitoring] We have made progress in pinpointing the root cause of the reported call quality issues. Our investigation has led us to identify specific carrier trunks that seem to be contributing to the challenges experienced today. To address this matter, we are taking proactive steps to reroute both inbound and outbound calls. By doing so, we aim to circumvent the affected paths associated with the identified carrier trunks, ensuring a more stable and reliable communication experience for all users. If you, or any of your team members, are still encountering call quality issues (or you have any questions) please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

February 14, 2024 12:27PM EST
[Monitoring] As of our latest assessment, we are not receiving any new reports of trouble, and it appears that the routing policy change made at 11:26 am seems to have positively impacted the situation. However, we understand that individual experiences may vary. If you, or any of your team members, are still encountering call quality issues (such as calls "cutting in and out"), we want to ensure we address this immediately. Please reach out to our support team by emailing support@telnetww.com or calling us toll free at 800-508-1254 Option #4. so that we can investigate further and assist you in resolving any lingering concerns.

February 14, 2024 11:31AM EST
[Investigating] We are aware that some users are experiencing issues with calls "cutting in and out," and we understand the impact this may have on your communication. Our engineering team is actively investigating the root cause of these concerns. To address the issue, a routing policy change was implemented at 11:26 am today. We are closely monitoring the situation to assess whether this adjustment alleviates the reported symptoms. If you should encounter any further developments, or you have any call examples to provide us, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.

Description

MAINTENANCE TYPE: Call Recording Update DESCRIPTION: This maintenance is being performed to update our call recording hosts and servers, between 10pm and 11pm SERVICE IMPACT: No Impact is expected to the services, but the server sites will be unreachable during a reboot. call-recordings.telnetww.com If you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.


Components

Voice Services


Services

Cloud Contact Center


Schedule

March 12, 2024 10:00PM - March 12, 2024 11:00PM EDT



March 12, 2024 11:00PM EDT
[Update] Scheduled maintenance is completed

March 12, 2024 10:00PM EDT
[Update] Scheduled maintenance has started