Archives October 2016

Testing FreeSWITCH for memory leaks using Valgrind and SIPp

In this tutorial we will assume you have already setup SIPp on another server to stress test FreeSWITCH. We covered this in our previous article

We will need to install valgrind on the same machine running FreeSWITCH. I like using Debian for FreeSWITCH, which conveniently already has valgrind in the repository.

apt install valgrind

Now that valgrind is installed, we will need to make sure FreeSWITCH is not already running.

systemctl stop freeswitch
ps -ef | grep freeswitch

Once we stop FreeSWITCH, we want to start up FreeSWITCH using valgrind. Below is the command to do so. This will log all memory issues found by Valgrind to vg.log in the directory you run the command from.

/usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Running the FreeSWITCH from the stable repository will not result in many errors. Below is my the end of my vg.log while running FreeSWITCH 1.6.12-20-b91a0a6~64bit

==20986== LEAK SUMMARY:
==20986==    definitely lost: 0 bytes in 0 blocks
==20986==    indirectly lost: 0 bytes in 0 blocks
==20986==      possibly lost: 16,384 bytes in 2 blocks
==20986==    still reachable: 192 bytes in 1 blocks
==20986==         suppressed: 0 bytes in 0 blocks
==20986==
==20986== For counts of detected and suppressed errors, rerun with: -v
==20986== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

There are no new errors after the leak summary in this log.

When the FreeSWITCH is having memory management problems, valgrind will continue to log errors after the LEAK SUMMARY. Below is an example of a memory leak when accessing the libcrypto.so library for WebRTC.

143768 ==32354== Use of uninitialised value of size 8
143769 ==32354==    at 0x67C8B80: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
143770 ==32354==    by 0xF684D2C1E662C6C4: ???
143771 ==32354==    by 0xE16640B5E123D1DD: ???
143772 ==32354==    by 0xD68DF16DAF4386F1: ???
143773 ==32354==    by 0xB2B603F64080C703: ???
143774 ==32354==    by 0x398A1FCC7DC2FF3B: ???
143775 ==32354==    by 0xBE60B2707D98773F: ???
143776 ==32354==    by 0x375392F355F20B45: ???
143777 ==32354==    by 0x124A183B188AA4EA: ???
143778 ==32354==    by 0x5BFE669681E01F8: ???
143779 ==32354==    by 0x6A4C9E9A4AA5A4C0: ???
143780 ==32354==    by 0x29B07C509D28C49F: ???
143781 ==32354==

In this case, valgrind was logging this message to the log many times per second. You can view the memory mismanagement manifesting itself in a memory leak by viewing the allocated memory with ps. We will want to monitor the change in RSS, or Resident Set Size, which represents the ammount of bytes are allocated for a process in RAM.

ps -aux | grep '[v]algrind.bin'

On Debian 8, ps -aux will list the information in the following order:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

The output of your ps -aux command should look something like this

root     20987 36.7 13.1 631032 271356 ?       S<sl 13:33  30:47 /usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Knowing the current total allocated memory for FreeSWITCH is cool, but not very useful for seeing the memory leak in action. If FreeSWITCH has a memory leak, it will fail to release memory when it is done with it, causing the allocated space to continue to grow. A useful way to observe this is by monitoring the RSS of the FreeSWITCH process over time, then running SIPp to create an artificial load on FreeSWTICH I created a simple bash script to log FreeSWITCH’s RSS, the current time, and the number of sip channels to a file.

#!/bin/sh


while true; do

INT=60
FILE=/root/mem.log
DATE=$(date +"%F-%R")
MEM=$(ps -aux | grep '[v]algrind.bin' | awk -F ' ' '{print $6}')
CHANNELS=$(/usr/local/freeswitch/bin/fs_cli -x "show channels" | grep total | awk -F ' ' '{print $1}')

echo ${DATE},${MEM},${CHANNELS} >> ${FILE}
sleep ${INT}s
done

This script will log the time, RSS, and SIP channels to /root/mem.log. You may need to adjust the CHANNELS variable to point to wherever the fs_cli binary is stored.

Save the script to a file, such as memtest.sh, then run it with

./memtest.sh &

You can confirm it is running in the background with jobs.

Now, with FreeSWITCH running under valgrind, we will want to start up the SIPp stress test. On the SIPp server, run the following.

sipp -sf uac.xml -s 9999999  <freeswitch-ip>:5080 -trace_msg -l 15 -d 60000

This test is best done over a long period of time. I would let SIPp generate traffic on the server for a couple hours while our monitor script is running. After the system has ran under load for some time, the mem.log should look soemthing like this.

2016-10-24-15:29,271212,15
2016-10-24-15:29,270800,15
2016-10-24-15:30,270672,15
2016-10-24-15:31,270828,15
2016-10-24-15:32,270708,15
2016-10-24-15:33,270604,15
2016-10-24-15:34,270812,15
2016-10-24-15:35,270628,15
2016-10-24-15:36,270816,15

If FreeSWITCH has been running for multiple hours and the RSS has seemed to plateau, like above, FreeSWITCH is likely not showing any critical memory mismanagements. Here is an example of the RSS continuing to grow due to a memory leak.

2016-08-11-04:09,379192,60
2016-08-11-04:10,381584,60
2016-08-11-04:11,383104,60
2016-08-11-04:12,385096,60
2016-08-11-04:13,385356,60
2016-08-11-04:14,387580,60
2016-08-11-04:16,387432,60
2016-08-11-04:17,389316,60
2016-08-11-04:18,391144,60
2016-08-11-04:19,393076,60
2016-08-11-04:20,394484,61
2016-08-11-04:21,395776,60
2016-08-11-04:22,397072,60
2016-08-11-04:23,398140,59
2016-08-11-04:24,400112,60
2016-08-11-04:25,401868,60
2016-08-11-04:26,403348,60

FreeSWITCH had been running for many hours at this point, and the RSS had never stopped growing. This is the same system which was logging the libcrypto error in the valgrind log.

When you notice a memory issue with your version of FreeSWTICH, check the FreeSWTICH JIRA for known bugs. There may already be a patch out for the issue you are experiencing.

SIPp – Load Testing FreeSWITCH

In this article we will go over how to get SIPP installed and start up a basic load test for FreeSWITCH.

Installing SIPp

We recommend installing SIPp to a different machine than where you are running FreeSWITCH. This will provide the most realistic stress test. You can download SIPP here: https://sourceforge.net/projects/sipp/files/sipp/3.4/

Run the following commands once you have the SIPp tar downloaded to get it compiled.

tar -xzf sipp-3.3.990.tar.gz
cd sipp-3.3.990
apt install libncurses6-dev libncurses6
./configure; make

If you are on Debian 8, You will need to install the following ncurses libraries instead:

apt install libncurses5-dbg libncurses5 libncurses

SIPp will now be installed in the directory you ran the make command in.

FreeSWITH Dialplan

We will create a basic extension in FreeSWITCH to process the sip traffic we will be sending via SIPp.

vim dialplan/public/0000000_sipp.xml

<include>
  <extension name="sipp_test">
    <condition field="destination_number" expression="^(9999999)$">
    <action application="answer"/>
    <action application="playback" data="ivr/ivr-welcome_to_freeswitch.wav"/>
    <action application="hangup"/>
    </condition>
  </extension>
</include>

Once you have this configured, give FreeSWITCH the reloadxml command:

fs_cli -x 'reloadxml'

This will create the extension sipp_test in the public context, which is listening for incoming traffic to the number 9999999. Freeswitch will playback an ivr message to every calling matching this number.

Next we will want to make sure our SIPp server is able to reach FreeSWITCH on the sip and rtp ports. Run the following iptables commands:

iptables -A INPUT -p udp --dport 5080 -s -j ACCEPT
iptables -A INPUT -p udp -m multiport --dports 16384:32768 -s -j ACCEPT

SIPp Testing

We are going to need a scenario file. You can paste the following into an xml file called uac.xml

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE scenario SYSTEM "sipp.dtd">

<!-- This program is free software; you can redistribute it and/or      -->
<!-- modify it under the terms of the GNU General Public License as     -->
<!-- published by the Free Software Foundation; either version 2 of the -->
<!-- License, or (at your option) any later version.                    -->
<!--                                                                    -->
<!-- This program is distributed in the hope that it will be useful,    -->
<!-- but WITHOUT ANY WARRANTY; without even the implied warranty of     -->
<!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the      -->
<!-- GNU General Public License for more details.                       -->
<!--                                                                    -->
<!-- You should have received a copy of the GNU General Public License  -->
<!-- along with this program; if not, write to the                      -->
<!-- Free Software Foundation, Inc.,                                    -->
<!-- 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA             -->
<!--                                                                    -->
<!--                 Sipp default 'uac' scenario.                       -->
<!--                                                                    -->

<scenario name="Basic Sipstone UAC">
  <!-- In client mode (sipp placing calls), the Call-ID MUST be         -->
  <!-- generated by sipp. To do so, use [call_id] keyword.                -->
  <send retrans="500">
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>
  <recv response="100"
        optional="true">
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>

  <!-- By adding rrs="true" (Record Route Sets), the route sets         -->
  <!-- are saved and used for following messages sent. Useful to test   -->
  <!-- against stateful SIP proxies/B2BUAs.                             -->
  <recv response="200" rtd="true">
  </recv>

  <!-- Packet lost can be simulated in any send/recv message by         -->
  <!-- by adding the 'lost = "10"'. Value can be [1-100] percent.       -->
  <send>
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 1 ACK
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <!-- This delay can be customized by the -d command-line option       -->
  <!-- or by adding a 'milliseconds = "value"' option here.             -->
  <pause/>

  <!-- The 'crlf' option inserts a blank line in the statistics report. -->
  <send retrans="500">
    <![CDATA[

      BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 2 BYE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <recv response="200" crlf="true">
  </recv>

  <!-- definition of the response time repartition table (unit is ms)   -->
  <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

  <!-- definition of the call length repartition table (unit is ms)     -->
  <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

We can now run the following command to start the SIPp load test. This will start up an ncurses graph to show you the progress.

sipp -sf uac.xml -s 9999999 :5080 -trace_msg -l 15 -d 60000

In the next tutorial we will go over how to analyze FreeSWITCH’s memory management with valgrind while under a stress test.

FreeSWITCH, configuring the Callcenter Module

We will go over how to setup the call center module. The callcenter module is used for creating an inbound queue for connecting inbound callers with agents registered to your system. We will be assuming you followed our FreeSWITCH setup tutorial here

MOH dependency

Make sure you have installed the moh sounds, which on debian is done with apt install freeswitch-music-default.

callcenter.conf.xml

You will first need to make sure the callcenter module is enabled in modules.conf

    <load module="mod_callcenter"/>

Next open up autoload_configs/callcenter.conf.xml. A default queue is provided, which we will make some modifications to.

    <queue name="test@default">
      <param name="strategy" value="longest-idle-agent"/>
      <param name="moh-sound" value="$${hold_music}"/>
      <param name="time-base-score" value="system"/>
      <param name="max-wait-time" value="0"/>
      <param name="max-wait-time-with-no-agent" value="0"/>
      <param name="max-wait-time-with-no-agent-time-reached" value="5"/>
      <param name="tier-rules-apply" value="false"/>
      <param name="tier-rule-wait-second" value="300"/>
      <param name="tier-rule-wait-multiply-level" value="true"/>
      <param name="tier-rule-no-agent-no-wait" value="false"/>
      <param name="discard-abandoned-after" value="60"/>
      <param name="abandoned-resume-allowed" value="false"/>
    </queue>

Now we need to add an agent.

<agent name="1000@default" type="callback" contact="[call_timeout=10]user/1000" status="Available" max-no-answer="3" wrap-up-time="10" reject-delay-time="10" busy-delay-time="60" />

And specify the agent’s tier, which also puts them into the queue.

   <tier agent="1000@default" queue="test@default" level="1" position="1"/>

We can now send inbound calls to our callcenter queue in our dialplan.

<include>
  <extension name="test_ivr">
   <condition field="destination_number" expression="^(test_ivr)$">
     <action application="set" data="accountcode=test_in"/>
     <action application="callcenter" data="test@default"/>
     <action application="hangup"/>
   </condition>
  </extension>
</include>

FreeSWITCH custom CDR guide and dividing CDR logs by accountcode

We will be configuring the cdr_csv module to log all calls going through a specific extension to a seperate file. This is useful if you would like to monitor all calls hitting a certain extension in your dialplan without having to dig through the master CDR file. We will assume that you have setup your FreeSWITCH system using our previous tutorial, FreeSWITCH 1.6 Quick Install Guide for Debian Jessie

CDR

First make sure the mod_cdr_csv module is uncommented in /etc/freeswitch/autoload_configs/modules.conf

    <load module="mod_cdr_csv"/>

Next we need to create a new template in /etc/freeswitch/autoload_configs/cdr_csv.conf.xml. Note that the template will need to be named the same as whatever account code you decide to use in your dialplan later. When an accountcode matches a template name, FreeSWITCH will create a unique CDR file for all calls specified as that accountcode.

    <template name="test_in">"${caller_id_number}","${start_stamp}","${answer_stamp}","${end_stamp}","${accountcode}"</template>

This template will log the caller id, start of call time, answer time, end of call time, and accountcode name to test_in.csv in the log-base directory, which defaults to /var/log/freeswitch/cdr-csv/ You can find a list of cdr variables here

Dialplan

Now in your dialplan you will need to set the accountcode. In my dialplan, I am bridging to a user once it hits my default context, which is where I decided to set the account code. Below you will see the accountcode set before the call is bridged to the 1000 user.

     <action application="set" data="accountcode=test_in"/>
     <action application="bridge" data="USER/1000@default"/>

You will need to reload FreeSWITCH after making these changes. If you are on Debian 8, you can run systemctl restart freeswitch. Once FreeSWITCH is reloaded you should see a seperate CDR file at /var/log/freeswitch/cdr-csv/test_in.csv, which will only contain calls flagged with the accountcode test_in.

FreeSWITCH 1.6 Quick Install Guide for Debian Jessie

Our Goal

There’s a lot of FreeSWITCH material out on the web.  Our goal is to provide you a concise document that provides the necessary steps or scripts to install the latest stable release of FreeSWITCH.

What is FreeSWITCH

FreeSWITCH is an OpenSource VoIP platform that enables you to build communication platforms that span from a simple PBX to a sophisticated backend communication framework that allows thousands of calls to interconnect per second.

Assumptions

  • Debian Jessie is installed
  • You have root or sudo access via the CLI

Let’s Install (with Explanation of each command)

  • Install Public Key: We need to install the FreeSWITCH public key so that we validate the packages that we are going to install came from FreeSWITCH.org and was not tampered with.
    wget -O - https://files.freeswitch.org/repo/deb/debian/freeswitch_archive_g0.pub | apt-key add -
    
  • Setup Repository: Setup the Debian repository file with the necessary configuration to download the FreeSWITCH packages
    echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list 
    
  • Install the Packages: We will first update our local repository using apt-get update, which will obtain the metadata of the FreeSWITCH packages since we just added the FreeSWITCH repository. It will also update the metadata for other repositories as well. Lastly, we will install FreeSWITCH using the apt-get install -y freeswitch-meta-all command
    apt-get update && apt-get install -y freeswitch-meta-all
    
  • Secure the Extensions: By default all of the extensions have a password of 1234. FreeSWITCH will warn you that all extensions are using a default password of 1234 and ask that you change it by putting an entry in the /var/log/freeswitch/freeswitch.log file. Also, it will delay any outbound calls for a few seconds. You can change the default password by running these commands.
    sed -i "s/1234/10082013/" /etc/freeswitch/vars.xml
    fs_cli -x reloadxml
    

    Where the extension passwords will now be 10082013. You can change the default password by just replacing 10082013 with your password.

Register SIP Phones

Now we can validate that the install was successful. We can do this by registering 1 or more SIP phones. The default install of FreeSWITCH comes with a set of users. Let’s register extension 1000 and 1001 with two separate VoIP phones and make calls between the sip phones. Remember that your password for each extension is 10082013 or whatever you manually changed it too.