Cloud Contact Center, Cloud PBX, Digital Fax, IP-PRI, MS Teams Direct Routing, SIP Trunking, TelNet UC with Webex
March 13, 2024 4:48PM EDT
[Monitoring] We want to provide you with another update regarding the ongoing connectivity issues that some of our users have been experiencing.
Our collaborative work with the impacted ISP continues, and their team is actively inspecting all areas of their network for potential issues that may affect packet size and/or transport protocol. This thorough examination is part of our joint commitment to identifying and addressing any challenges within their infrastructure.
Please be advised that we have a few work arounds that have shown success for many users with smaller packet size relating to the UDP changes in traffic from the last update.
If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4 relating case 00068881.
March 1, 2024 4:11PM EST
[Monitoring] We want to provide you with a significant update regarding the ongoing connectivity issues that some of our users have been experiencing.
After careful investigation with our Network and Engineering teams, we identified that the issue undeniably lies within the impacted ISP's network. As you know, despite our continued efforts to engage with the ISP, the resolution process has been challenging.
Our Findings / Confirming ISP Network Issues:
During our in-depth testing, we identified that when customers migrated their services away from the impacted ISP (utilizing alternative solutions such as 4G/5G Hotspots or other ISPs), that these users experienced a restoration of their services - highlighting a direct correlation between the issue and the impacted ISP’s network.
Recent Changes and Potential Improvement:
In response to the challenges faced, we have implemented a change to our phones that has shown promising results in enhancing voice connectivity for several customers. Specifically, we modified the transport layer protocol for select customers from TCP to UDP.
Understanding TCP vs. UDP:
TCP (Transmission Control Protocol): Think of TCP like a reliable, ordered delivery service. It ensures that data packets are transmitted in a specific order and verifies that they are received correctly. This is beneficial for applications where every piece of data needs to arrive intact.
UDP (User Datagram Protocol): UDP, on the other hand, is more like a speedy courier service. It delivers data packets quickly but without the same level of order or verification.
How This Affects You:
The change from TCP to UDP has resulted in a marked improvement in maintaining voice connectivity for the customers involved in the initial rollout. Based on this original feedback, it appears that the original TCP transmissions were being affected by potential filtering performed by the impacted ISP. By changing the transmission type to UDP, it appears that the voice traffic is not being affected to the same degree.
It is, however, important to note that UDP (although expedient) may have reliability considerations, and our initial choice of TCP is based on its preferred reliability. Our intention would be to revert back to TCP as soon as the impacted ISP resolves their network issue, ensuring that the optimal method of transport is used.
Encouraged that our customers are reporting an improvement in their services, we have extended the change to the remaining customers who reported trouble.
Your Feedback Matters:
We value your experience, and your feedback is crucial in helping us understand the full impact of this change. We kindly ask you to share your observations and insights with us. Has the modification positively impacted your voice connectivity? Are there any other aspects we should be aware of?
Please reach out to our support team (at 800.508.1254 OPT 4) or support@telnetww.com with your feedback. We genuinely appreciate your cooperation and understanding as we work together to optimize your connectivity experience.
Our goal throughout this issue has been neutral troubleshooting to effectively isolate the root cause of trouble - regardless of the area of responsibility. As a neutral party, we will strive to alleviate unneeded stress on our customers by owning repair efforts until the trouble is resolved. Thank you for your continued trust in TelNet Worldwide.
February 29, 2024 7:57PM EST
[Identified] After persistent efforts, we have successfully reached the escalation team at the impacted ISP, who have disclosed that a fiber cut occurred in their core network in the area on 2/27.
This incident may have resulted in continued issues for customers, and the ISP is actively investigating the aftermath of the fiber cut. We appreciate your patience and understanding as the ISP works to address the underlying network challenge.
Rest assured, we remain committed to staying on top of the situation and will continue to provide updates as we receive more information. We understand the impact this has on your services and sincerely thank you for your ongoing support.
If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.
February 27, 2024 3:51PM EST
[Identified] Despite our best efforts to address the matter, we regret to inform TelNet Voice customers that the impacted ISP has unfortunately declined our multiple requests for collaboration or investigation into the identified issue - the absence of expected returned SIP traffic.
The core issue revolves around the challenge of proving a negative — the absence of anticipated traffic. In such situations, our ability to take direct action is limited, leaving us in a challenging position.
We understand the frustration and inconvenience this may cause, and we want to assure you that we are actively exploring alternative avenues to find a resolution with the impacted ISP.
If you have access to an alternative internet source (such as 4G/5G LTE not sourced from the impacted ISP), consider temporarily swapping your voice services to the alternate connection. This may help bypass the ongoing issue.
If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.
February 27, 2024 2:33PM EST
[Identified] We want to keep you informed about the ongoing connectivity issues impacting some of our users. Following our initial investigation, we've determined that all TelNet traffic successfully reaches our customers through all available routing pathways. However, there's a challenge with the expected return traffic that is specifically impacting users sharing a common ISP.
It's crucial to understand that this issue is exclusive to users with a particular ISP, and unfortunately, TelNet lacks control over the routing and network configuration of other companies. Our team has initiated contact with the affected ISP to address the situation. While we face limitations in influencing external network configurations, we are very open to working directly with the impacted ISP.
If any customers can connect us with the Network Operations Center (NOC) or a Network Technician from the affected ISP, we believe direct collaboration would be beneficial in resolving this matter. We appreciate your understanding and patience as we navigate through this situation and remain dedicated to providing updates and working towards a resolution.
If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.
February 27, 2024 11:08AM EST
[Investigating] Recently, we've observed an increased number of customer reports regarding intermittent issues such as dropped calls, no inbound or outbound calls, and loss of SIP Registration. Upon investigation, we've identified a common thread - that affected customers share the same Internet Service Provider (ISP).
Our team has conducted traceroutes from several points in our network, all of which successfully reached the impacted ISP's network. These traces appear to be "dying" many hops within the ISP's network. This points us toward the possibility that the issue lies within their network infrastructure.
TelNet is actively attempting to reach out to the impacted ISP to make them aware of this issue. We apologize to any TelNet customers affected by the ISP-related issues, and it's important to note that as these disruptions occur within another ISP's network - it is beyond our control as there is nothing we can do to directly address or resolve the challenges within their network.
We appreciate your understanding and patience as we work collaboratively with the ISP to expedite a resolution.
If you should encounter any further developments, or you have any questions, please contact our support team by emailing support@telnetww.com or toll free at 800-508-1254 Option #4.