AI-assisted development has changed how quickly we can design, implement, and modify complex systems.
Tools such as Claude Code can understand a large codebase, generate services, refactor modules, add tests, and even reason about distributed system behavior.
But there is a critical lesson I have learned while working on real-time telecom and CCaaS systems:
The production system is always the final source of truth.
A bug that looks obvious from the application code can actually be caused by TCP behavior, SIP signaling, FreeSWITCH state, Fail2Ban, Redis data, network routing, process limits, timing, or a combination of several systems.
This becomes especially difficult when the system is handling a large outbound campaign.
AI Understands the Code. Production Shows the Reality.
When developing with an AI coding agent, it is tempting to describe a problem like:
"Agents are not receiving calls. Fix the campaign allocation logic."
The AI can inspect the campaign service, Redis queues, agent state management, database queries, and call-dispatch logic.
It may even identify a perfectly reasonable bug.
But in a real production environment, the actual problem could be somewhere completely different.
For example:
Application layer
- Campaign scheduler
- Agent availability
- Redis campaign state
- PostgreSQL transactions
- Job queues
- WebSocket events
Telephony layer
- FreeSWITCH channel state
- SIP registration
- SIP INVITE
- SIP response codes
- RTP negotiation
- Gateway state
- Call timeout behavior
Operating system layer
- TCP connections
- File descriptors
- CPU saturation
- Memory pressure
- Network buffers
- Process limits
Security layer
- Fail2Ban
- Firewall rules
- IP blocking
- Rate limiting
- Unexpected connection rejection
Infrastructure layer
- Docker/Kubernetes
- Load balancers
- NAT
- Public/private IP mapping
- Redis availability
- Network latency
The application logs may say:
"Call dispatched successfully."
FreeSWITCH may say:
"No response."
And the network may show that the SIP packet never reached the destination.
All three statements can be true.
That is why debugging production telecom systems requires a different mindset.
Start From the Wire, Not the Code
One of the biggest mistakes in production troubleshooting is immediately modifying application code.
I prefer to work from the bottom of the stack upward.
A typical investigation starts with:
TCP dump → FreeSWITCH logs → SIP signaling → Fail2Ban/firewall → Redis → Application logs → Database → Business logic
The exact order can vary, but the principle is the same:
First establish what actually happened. Then determine why the software believed something different happened.
For a SIP problem, a packet capture can be more valuable than hundreds of application log lines.
For example:
Application
|
| "Call initiated"
v
FreeSWITCH
|
| INVITE
v
Network
|
X Packet dropped / rejected If the packet never leaves the server, changing the campaign algorithm will not solve the problem.
TCPDump Can Tell a Completely Different Story
When a production call fails, I often want to see the actual traffic.
For example:
tcpdump -ni any host <IP> or a SIP-focused capture:
tcpdump -ni any port 5060 The purpose is not simply to collect packets.
The objective is to answer concrete questions:
- Did the packet leave the server?
- Did the remote server respond?
- Was the response received?
- Was the response delayed?
- Was the connection reset?
- Did traffic arrive on the expected interface?
- Is NAT changing the expected source?
- Are retransmissions occurring?
- Is there asymmetric routing?
These answers can completely change the debugging direction.
Instead of:
"The dialer has a bug."
The real conclusion may be:
"The dialer created the call correctly, FreeSWITCH generated the INVITE, but the network path did not return the expected SIP response."
That is a fundamentally different problem.
FreeSWITCH Logs Add the Telephony Context
Packet capture tells us what happened on the network.
FreeSWITCH logs tell us what the telephony engine believed was happening.
For example, we may need to correlate:
Application call ID
↓
Campaign ID
↓
Agent ID
↓
FreeSWITCH UUID
↓
SIP Call-ID
↓
Gateway
↓
Carrier response Without correlation identifiers, troubleshooting becomes guesswork.
A production-grade CCaaS platform should make it possible to move from:
Campaign → Agent → Call → UUID → SIP transaction → Carrier
in seconds.
This is much more important than simply having "good logs."
Never Ignore Fail2Ban
Security tooling is another layer that developers frequently overlook.
A system can appear healthy from the application perspective while Fail2Ban has silently blocked an IP.
For example:
Application
↓
FreeSWITCH
↓
Firewall
↓
Fail2Ban
↓
BLOCKED The application may continue creating calls.
FreeSWITCH may continue running.
Redis may contain thousands of pending campaign records.
But the external communication path is effectively broken.
Therefore, during production troubleshooting I want to check:
- Fail2Ban jails
- Current bans
- Firewall rules
- SIP port accessibility
- RTP port ranges
- Recent authentication failures
- Unexpected IP blocks
A security mechanism doing exactly what it was configured to do can still look like an application failure.
Large Campaigns Change the Nature of the Problem
Small-scale testing can be extremely misleading.
Suppose we test:
10 agents
100 calls Everything works.
Then production runs:
500 agents
50,000 campaign records Suddenly the system behaves differently.
Why?
Because scale introduces state.
Redis may contain:
available agents
reserved agents
active calls
retry calls
pending calls
failed calls
campaign assignments
timeouts
dispositions Now imagine a worker crashes at exactly the wrong moment.
The application may have already written:
agent:123 = RESERVED but never completed:
agent:123 = CONNECTED The system now has a ghost reservation.
Multiply that by hundreds of agents and thousands of calls.
The system may appear to have available capacity while Redis believes the opposite.
Redis Is Not Just a Cache in a Real-Time Dialer
This is particularly important in distributed dialer architecture.
When Redis contains operational state, it becomes part of the system's control plane.
For example:
Campaign
↓
Redis Queue
↓
Agent Reservation
↓
Call Dispatch
↓
FreeSWITCH
↓
Call Completion
↓
Disposition
↓
Redis State Update If any transition fails, state can become inconsistent.
Consider:
1. Agent becomes available
2. Redis reserves agent
3. Call is requested
4. FreeSWITCH creates channel
5. Carrier rejects call
6. Application misses cleanup event The agent may remain:
RESERVED even though no real call exists.
At scale, these small inconsistencies become major capacity problems.
The Most Dangerous Bugs Are Often State Bugs
A production system can have perfectly valid code and still produce invalid state.
For example:
Database:
Agent = AVAILABLE
Redis:
Agent = RESERVED
FreeSWITCH:
No active channel
Frontend:
Agent = READY Which one is correct?
There is no universal answer.
The architecture must explicitly define the source of truth for each state.
For a real-time system, I usually want a state machine rather than scattered boolean flags.
For example:
AVAILABLE
↓
RESERVING
↓
RESERVED
↓
DIALING
↓
RINGING
↓
CONNECTED
↓
WRAP_UP
↓
AVAILABLE And every abnormal transition needs a recovery path:
DIALING
↓ timeout
AVAILABLE
RESERVED
↓ worker crash
AVAILABLE
CONNECTED
↓ channel disappears
WRAP_UP
RESERVED
↓ stale timeout
AVAILABLE This is where production engineering becomes much more than writing functions.
Why AI Can Struggle With These Bugs
AI coding agents are extremely useful, but they usually reason from the evidence available to them.
If you give the AI:
campaign.service.ts
redis.service.ts
agent.service.ts it can reason very effectively about those files.
But it cannot automatically know that:
- a firewall rule changed,
- Fail2Ban banned an address,
- packets are being dropped,
- FreeSWITCH has a different channel state,
- Redis contains stale campaign state,
- a carrier is delaying responses,
- a Kubernetes node is overloaded,
- or a production race condition only appears under high concurrency.
The solution is not to stop using AI.
The solution is to give AI production evidence.
The New Production Debugging Workflow
A strong AI-assisted troubleshooting workflow should look more like this:
Production Incident
|
v
Collect Evidence
|
+----------------+----------------+
| | |
v v v
tcpdump FreeSWITCH System
logs metrics
| | |
+----------------+----------------+
|
v
SIP / Network
Correlation
|
v
Redis State Check
|
v
Application Logs
|
v
Database State
|
v
Code Investigation
|
v
Fix + Test This changes the role of AI from:
"Find the bug in my code."
to:
"Here is what actually happened in production. Correlate the network capture, FreeSWITCH logs, Redis state and application logs. Identify the most likely failure point and explain why."
That is a much stronger use of AI.
Production Debugging Requires Evidence Correlation
The real skill is often not finding one log entry.
It is correlating multiple systems around the same event.
For example:
11:42:01.102
Campaign worker reserves Agent 287
11:42:01.115
Redis:
agent:287 = RESERVED
11:42:01.127
FreeSWITCH:
originate requested
11:42:01.139
tcpdump:
SIP INVITE transmitted
11:42:06.139
No SIP response
11:42:11.140
FreeSWITCH:
originate timeout
11:42:11.145
Application:
call marked FAILED
11:42:11.150
Redis:
agent:287 remains RESERVED Now the real bug becomes visible.
The problem may not be:
"Why didn't the call connect?"
It may actually be:
"Why was the agent not released after the originate timeout?"
That distinction is critical.
Observability Should Be Designed Before the Incident
This experience has reinforced an important architectural principle:
Observability is not an operational add-on. It is part of the system design.
For a real-time CCaaS/dialer platform, I want at minimum:
Application
- Structured logs
- Correlation IDs
- Campaign IDs
- Agent IDs
- Call IDs
- Job IDs
- Error classification
FreeSWITCH
- UUID correlation
- SIP signaling visibility
- Channel lifecycle
- Gateway status
- Call termination causes
Redis
- Queue depth
- Key TTLs
- Reservation state
- Stale keys
- Consumer/worker health
Infrastructure
- CPU
- Memory
- Network
- File descriptors
- TCP connections
- Container health
Security
- Firewall events
- Fail2Ban bans
- Authentication failures
Metrics
- Calls attempted
- Calls connected
- Calls failed
- Average call setup time
- Agent utilization
- Queue depth
- Redis latency
- FreeSWITCH channel count
- SIP response distribution
Without these signals, troubleshooting becomes archaeology.
AI + Human Expertise Is More Powerful Than Either Alone
I don't see AI coding agents as a replacement for production engineering.
I see them as an extremely powerful engineering multiplier.
AI is excellent at:
- understanding large codebases,
- tracing application logic,
- generating diagnostic scripts,
- analyzing logs,
- proposing fixes,
- writing tests,
- identifying race conditions,
- refactoring services,
- documenting architecture.
The engineer still needs to understand:
- networking,
- SIP,
- RTP,
- FreeSWITCH,
- Linux,
- Redis,
- distributed systems,
- concurrency,
- observability,
- production failure modes.
The best workflow is therefore:
Human collects reality → AI analyzes evidence → Human validates hypothesis → AI helps implement fix → Production telemetry validates the result.
The Biggest Lesson
When AI-generated software enters production, the debugging methodology has to evolve.
The question should not be:
"What does the code say should happen?"
The better question is:
"What actually happened across the entire system?"
For a real-time telecom platform, that means looking beyond the application.
Start with the wire.
Then inspect FreeSWITCH.
Check security controls.
Inspect Redis state.
Correlate application events.
Verify database state.
Then finally examine the code.
Because in large-scale real-time systems, the hardest bugs are rarely contained inside one function.
They live in the gaps between systems.
And that is exactly where production engineering, observability, distributed-systems knowledge, and AI-assisted analysis need to work together.
Final Thought
AI can write the code.
AI can review the code.
AI can even suggest the fix.
But when thousands of real calls are flowing through a production system, the packets, processes, state machines, and telemetry tell the truth.
The engineer's job is to connect those pieces together.
