How do you approach Mobile Core Network Troubleshooting?

Hello everyone,

I’m looking for someone experienced in mobile Core Network troubleshooting who could help me improve my troubleshooting skills, particularly in:

  • CS Core — MSC, HLR, AuC, etc.
  • PS Core / EPC — MME, SGW, PGW, PCRF, HSS, etc.
  • VoLTE / IMS
  • 5G Core
  • Signaling and protocols such as SS7, Diameter, GTP, SIP, HTTP/2
  • Troubleshooting using Wireshark, logs, KPIs, alarms, and call flows

If anyone has technical documentation, troubleshooting guides, call-flow diagrams, training material, or real-world troubleshooting examples that they can share, I would really appreciate it.

I’m particularly interested in learning a structured methodology for troubleshooting issues from the UE/network symptoms → signaling → logs → root cause → resolution.

Thank you in advance to anyone willing to share knowledge or resources.

The single most useful thing I can give you is the methodology itself, because once the structure is in your head you can apply it to any node or protocol. Here is the loop I use, mapped to your symptom to resolution flow.

First, frame the symptom precisely before touching anything: what, where, when, who, how many, and since when. Scope it to one cell, one subscriber, one APN, or the whole node. A vague symptom is the number one reason RCA fails.

Second, draw the call flow for that exact service before you diagnose it. List the nodes and interfaces the procedure crosses. For a VoLTE MO call that is UE to eNB to MME to S/PGW plus IMS P/S/I-CSCF and TAS, over Gm, Mw, Cx, Sh, Gx and Rx. You cannot localise a fault you cannot draw.

Third, gather evidence classified by layer and time: alarms, KPIs and counters, node logs, and a trace or pcap of one failed attempt. Align the timestamps first, because a mis-aligned trace invents a false story.

Fourth, walk the call flow message by message and find the first place it deviates: the first error response, the first missing message, the first reject. The failure is where the flow first breaks, not where the symptom appears.

Fifth, read the exact cause value at that break: the Diameter result-code, the GTP cause, the SIP response, the NAS EMM or 5GMM cause, the MAP or ISUP error. That code names the failing procedure and which side rejected.

Sixth, apply the counterfactual before you commit: a cause is only the root cause if changing that one thing would remove the symptom. Reject the explanation that needs the most special pleading.

Seventh, fix and verify at the same layer you diagnosed: symptom gone, KPI back to baseline, and re-run the trace to confirm the flow now completes. Watch for side effects.

To practice this for free, build a lab you can break on purpose. Open5GS or free5GC give you a full EPC and 5GC; add UERANSIM or srsRAN for the UE side. Capture with Wireshark, which dissects NGAP, NAS-5GS, Diameter, GTP, SIP and HTTP/2 SBI, and use sngrep for live SIP. For the correct flows to compare your broken traces against, read TS 23.401 for EPC, TS 23.502 for 5GC procedures, TS 24.301 and 24.501 for NAS causes, TS 29.274 for GTPv2 causes, and TS 24.229 for IMS. Break one thing, trace it, find the first deviation, fix it, and confirm. That single habit builds more skill than any document.

Happy to help as you go. If you want structured one-to-one walk-throughs of real traces, that is the kind of mentoring I do, so tag me on a specific failed call flow and we can work it end to end.

Hey, this is a great goal to work on- troubleshooting is honestly something you get better at with structure + practice, not just by memorizing protocols.

Keep a personal “cause code cheat sheet”- every time you hit a Diameter/GTP/SIP cause in a real trace, note what it meant and what actually caused it. Over time it becomes faster than digging through specs mid-troubleshooting.

If a full lab feels heavy to start, practicing on public VoLTE/EPC sample pcaps in Wireshark first helps build call-flow-reading confidence.

For simplified call-flow explanations EPC, VoLTE/IMS, 5G Core alongside the specs, TechLTE World (techlteworld.com) has some beginner-friendly breakdowns that might help.

Hope it helps!

RCA LOOP
STEP
ACTION

  1. SCOPE | What, where, when, who, how many?
  2. FLOW | Draw the exact procedure and interfaces.
  3. EVIDENCE | Align logs, KPIs, alarms and one failed trace.
  4. DEVIATION | Find the FIRST missing/rejected/abnormal message.
  5. CAUSE | Read the exact SIP/Diameter/GTP/NAS/SBI cause.
  6. PROVE | Change one variable — does the failure disappear?
  7. FIX | Correct the proven fault at that layer.
  8. VERIFY | Re-run the flow and confirm recovery.

GOLDEN RULE

SYMPTOM → FLOW → EVIDENCE → FIRST DEVIATION → CAUSE → PROVE → FIX → VERIFY

The skill is not memorising every cause code.

It is learning to find the first place where the network stopped doing what it was supposed to do.

Fourat, I’d start with one successful IMS registration and annotate what each message actually proves. With IMS AKA, a 401 Unauthorized is the authentication challenge, so that response alone doesn’t mean registration failed. Follow the next REGISTER and its response. I co-founded HiCellTek; its UE-side IMS/SIP decoding lets you inspect that exchange alongside radio events. Then compare with a failed attempt and check the IMS-side trace where the sequences diverge. The handset capture won’t expose every internal core transaction, but it gives you a concrete starting point before tackling a whole VoLTE call.Fourat, I’d start with one successful IMS registration and annotate what each message actually proves. With IMS AKA, a 401 Unauthorized is the authentication challenge, so that response alone doesn’t mean registration failed. Follow the next REGISTER and its response. I co-founded HiCellTek; its UE-side IMS/SIP decoding lets you inspect that exchange alongside radio events. Then compare with a failed attempt and check the IMS-side trace where the sequences diverge. The handset capture won’t expose every internal core transaction, but it gives you a concrete starting point before tackling a whole VoLTE call.