Using CRA to Troubleshoot Cisco UC
Author: Kaz Kido - CCNP, Arista ACE-A
Dear User,
Regarding the voice quality issue that you reported, could you please provide the following information so I can troubleshoot?
-
When was the call made/received?
-
What was the number that called from and to?
-
Was it an outbound call or inbound call?
-
Was it choppy, garble, or staticky?
-
If there was no audio, do you know if the person on the other side could hear you?
-
Is it happening only to your phone or is it happening to other people at your location as well?
-
Have you tried rebooting your phone?
Thanks,
Voice Engineer
If you are a voice engineer and have experience in troubleshooting voice quality issues, I believe that you have written a similar email or had a conversation with end-users asking these questions. In my experience, it is a pretty rare case that the end-users can provide accurate answers to all of your questions. In some cases, a user might even report over exaggerated statements thinking it will help to pressure and solve their issue quickly while it will just put voice engineers into a maze of voice troubleshooting labyrinth due to misleading information.
Voice engineers need to know what is actually going on at the site to start troubleshooting, but getting accurate first hand information is often one of the most difficult parts in voice troubleshooting world. Additionally, apart from those questions to an end user, we need even more information, such as codec, jitter, packet loss...etc.
How do we get the information?
Asking lines of questions to end users? RTMT Trace on CUCM? Debug on a voice gateway? Logs from a phone? Or use CUCM native CDR tool? And after all, you might not even find the call that you are looking for -- maybe the date/time provided by the end user is incorrect? Now what? Ask the user again? Send a tech to the site? Or maybe you drive to the site if it is close? Yeah, fun time begins…
Now, let me show you how Call Record Analyzer can help this.
For example Lets say that you receive an escalation from a helpdesk that a user experienced a voice quality issue yesterday. No specific time and no calling or called number provided - of course.
Take a look at the CRA reporting tool screen shot below (10 digits DID and IP have been blurred)
Look at that!
You have the called number and calling number as well as date and time (up to the second!). If you have to work with a service provider who is asking for “fresh” call samples, you can give them instantly! You just need to know the extension of the phone that you are looking to get this info. I bet all end users should be able to provide their extension pretty easily.
How sweet is it?
You have all the info on one page instead of having to ask users. Now, you have solid information to start off investigation and troubleshooting. You probably want additional information, such as jitter, packet loss, latency, codec that you would collect from different places/resources in voice environment. Well, you might have already noticed but some of that information is already displayed!
There is codec identifier, source and destination IP, and disconnect cause code. For now you have to reference Cisco documentation to find the name of the codec (“https://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/service/11_0_1/cdrdef/CUCM_BK_RFDE0054_00_cucm-cdr-administration-guide-1101/CUCM_BK_RFDE0054_00_release-1101_chapter_0110.pdf”), but it is still WAAAY better than going through traces and debugs to find it out. In addition to this information, you have access to even more information stored in CRA. You can simply click a phone number on the report to pull the information like below:
At this point, you would have most of the information that you would need for troubleshooting. The only information that you cannot probably get is a debug output on a live call, but most likely you do not even need to run a debug with the information available on CRA.
There is another really cool feature that CRA has that I want to bring up.
Ok now, we have lots of information about calls to or from a specific number. That is good, but this phone number makes or receives dozens of calls a day! Do I need to go through all of the calls on the report to see which calls, out of dozens, l had an issue with jitter, packet loss or latency? That will seriously hurt my eyes. Well, how about this!
Yeah! CRA can highlight calls with jitter, latency, or packet loss beyond normal VoIP thresholds! You can just click “Check for Network Issues” button on the top of the page.
Highlighting those calls on a report seems to be such a simple concept, yet it is extremely helpful and useful.
During troubleshooting, you probably want to find out if the reported voice issue is affecting only a specific phone(s) or a large group of phones. If it is affecting just one phone, then hey, maybe it is just a layer 1 issue or even the other side of the call, try rebooting, swap handset, and see what happens and get a cup of coffee. But, if it is a large group of phones being affected, then get a cup of coffee and start looking into the network infrastructure.
Again, this is simple information, but very useful information. Without a tool like CRA, you have to ask users if other people are experiencing the same issue. However, there is not practical or reliable information in most scenarios.
Here is an example of how you can utilize this feature:
Just by looking at this page, not even having to look at anything else, you can tell that calls in this department (yes, you can create a group and run a report for that group only) are having an issue regardless of the phone numbers involved. This indicates that the issue might be related to the network for this department, such as QoS settings, bandwidth availability...etc.
CRA provides lights for voice engineers for troubleshooting voice issues. Imagine how long and what it would take to get this much information on a voice issue if you are troubleshooting without a tool like CRA. You might still be exchanging emails or playing phone tag with an end-user trying to gather more information.