Load Balancing FusionPBX Outbound Gateways

In certain situations you want to load balance SIP traffic between multiple SIP endpoints. These SIP endpoints are typically a Session Border Controller (SBC) in your organization or an upstream SIP provider. In any case, you need some logic that will distribute the traffic between the SIP endpoints. In the FusionPBX world (which is based on FreeSWITCH) this module is called Distributor (mod_distributor). In this article we will explain how to configure it.

Assumptions

  • You have at least two gateways defined within FusionPBX, which represents the SIP endpoints that you want to distribute traffic between.

Configuration Steps

  1. Login to FusionPBX and make sure the mod_distributor module is enabled. You can do so by going under Advanced -> Modules. Distributor should be an option to choose from, but should be disabled.  It will need to be enabled. “Enabled” should read True on the distributor line once it’s enabled.
  2. Get Gateway ID’s – right click on the Gateway name and click “Copy Address”. Open up a text editor and paste the address. The Gateway ID is the string after “id=” in the URL. Do this for all of the gateways you want to distribute traffic too.
3. Changing the mod_distributor config xml:
  • SSH to your FusionPBX server, make your way into the /etc/freeswitch directory
  • Using your favorite editor, edit autoload_configs/distributor.conf.xml and add the following:
<list name="my_list_001" total-weight="2">
<node name="978fefdd-45cd-4457-8ecd-5d82eaea3b12" weight="1"/> <!-- my_gateway_001 - its just comment with actual GW name -->
<node name="1a8d448d-e06e-2dab-b6f9-4421a98e4d8e" weight="1"/> <!-- my_gateway_002 -->
</list>
4. Save and run the following commands from the FreeSWITCH cli
fs_cli > reloadxml
fs_cli > distributor_ctl reload
5. Updated your Outbound Routing setting. You have something like this now: But, you want to change the last line to something like this: The Distributor command above takes a string value that contains a list that was defined in distributor.conf.xml and returns a gateway based on the weight defined in the configuration. In this case, “my_list_001” is the list used. This module can be used for other use cases where you need to generate random/weighted values.

dSIPRouter Media Server API v1

We are introducing a new API that will enable the provisioning of extensions within a backend media server. The API will provide an abstraction layer for provisioning different backend media servers. The first release will focus on FusionPBX.

dSIPRouter has a concept of an Endpoint Group. An endpoint group can be mapped to a FusionPBX Cluster. When this happens the domains from FusionPBX is automatically sync’d with dSIPRouter. We will leverage the existing connection to enable the ability to provision and update Domains and Extensions within FusionPBX. In the future, we will support FreePBX and other media servers as well. Hence, giving users one API for provisioning users and dSIPRouter handles the complexity of translating the request into the backend media server.

The feature will be available in 0.642. You can provide comments and feedback on the API by submitting comments in the GitHub issue which can be found here

The API endpoints for this release are:

/mediaserver/domain/
/mediaserver/extension/

The Payload for this release:

Payload for Domain

{
   domain_id:
   name: string,
   enabled: boolean,
   description: string,
   config_id: endpointgroup_id | conf_id
}

Payload for Extension

{
    type: single| multiple,
    num_of_extensions: integer,
    attributes: {

      domain_id: string | null,
      account_code: string,
      extension: string,
      password: string|null ,
      outbound_caller_number: string|null,
      outbound_caller_name: string:null,
      vm_enabled: boolean,
      vm_password: string,
      vm_notify_email: string,
      enabled: boolean,
      config_id: endpointgroup_id | conf_id
    }
}

Example Usage

The following example will provision a new Endpoint Group in dSIPRouter and a Domain and Extension in FusionPBX. Note, that the API will also support updating and deleting as well.

1) The user will create an endpoint group to represent a FusionPBX standalone instance or a FusionPBX cluster

https://{{DSIP_ADDR}}:5000/api/v1/endpointgroups

They will receive an endpoint group id, which will be used for provisioning the backend media server

2) The user can then create a new domain

https://{{DSIP_ADDR}}:5000/api/mediaserver/domain/

{
   domain_id: null
   name: AprilandMackCo,
   enabled: true,
   description: "April and Mack Co,
   config_id: 64
}

The user will receive the domain_id, let’s assume its 98

3) The user can create one user or a set of users. We are going to create 10 users

https://{{DSIP_ADDR}}:5000/api/mediaserver/extension/

{
    type: "multiple",
    num_of_extensions: 10,
    attributes: {

      domain_id: 98,
      account_code: "124-24245",
      extension: "1000",
      password: "starterpassword" ,
      outbound_caller_number: "8889072085",
      outbound_caller_name: "dOpenSource",
      vm_enabled: true,
      vm_password: "94145",
      vm_notify_email: "",
      enabled: true,
      config_id: 64
    }
}

This API will create 10 extensions starting from 1000 with the same attributes. You can also create a single extension.

Configuring Yealink W60P Cordless DECT IP Phone

The purpose of this tutorial is to explain how to configure the Yealink W60P Cordless DECT IP Phone with FreePBX, Asterisk of FusionPBX.  We will assume that you have the extension already configured in one of the aforementioned systems.  Therefore, we will focus on the steps needed to configure the phone.

I must note that the experience with configuring the Yealink W60P was much better then configuring the Panasonic KX-TGP600

The high-level steps needed to complete this are listed below.  We will go into detail for each section.

  • Locate Web Management UI IP Address
  • Log into theWeb Management UI
  • Configure the phone settings via the Web Management UI

Locate Web Management UI IP Address

  1. Pickup one of the Handsets that is registered to the base station
  2. Click OK
  3. Click OK on the (i) Icon
  4. Click OK on the BASE option
  5. The IP address of the Web Management UI (aka Base station) will be shown

Log into theWeb Management GUI

  1. Open a web browser
  2. Enter http://<IP address of the Web Management UI>
  3. Enter admin/admin as the username and password

Configure the phone settings via the Web Management GUI

  1. Click Account
  2. Select the Account you want to configure.  We will use Account1
  3. Enter in the following required and optional fields:

Required Fields

Line Action
Label
Display Name
Register Name
Username Name
Password

Optional Fields
These fields are only needed if you have a proxy server in front of your media server such as Kamailio or OpenSIP’s

Enable Outbound Proxy Server
Outbound Proxy Server 1

FusionPBX: Configuring the Call Center Module

We will go over how to setup the call center module. The call center module is used for creating an inbound queue for connecting inbound callers with agents registered to your system.

  1. You will first need to log into your FusionPBX server via the gui

  2. Select the Domain in which you would like to set up the call center in.

  3. Select Apps→ Call Center→ plus sign(+) to add a call queue.

  4. Fill in the necessary information and SAVE:

  • Queue name: Reference name for the queue
  • Extension: Extension number for the queue. Make sure it is unique for the system.
  • Strategy: Strategy is the way the system will hunt for available agents. For example: if longest-idle-agent is selected, the agent who hasn’t been called for the longest time (idle) will be called first.
  • Time Base Score: This is for whether you want to add the extra seconds from the call being established to the agent call time or not. System will add it where queue will keep the time from when the queue is reached.
  • Tier Rules Apply:  Select True if you want to apply the tier rules which we will setup later

All the others fields are pretty much standard. If you would like detailed information for each field you may find it here.

 

 

  1. Click “back’ to return to the Call Center Queue then on the top right corner, Click Agent

  2. Click the plus sign (+) to add agents in the Call Center Agents screen.

  3. Fill in the necessary fields:

  • Agent Name: Select the corresponding account you have created

  • Type: two types supported, callback and uuid-standby. callback will try to reach the agent via the contact fields value. uuid-standby will try to directly bridge the call using the agent uuid.

  • Contact: Select the extension you want to associate with that agent

  • Status: You can select status from here but it will be interacting with it later so its not an important field for now

  • Agent Logout: This is where you can select a time of the day where automatically the agent will be logged out. As it says it requires a service to enforce it.

 

While in the Call Center Agent screen you may also add a Agent ID and Password if you would like the agents to be able to log in on their own.

Note: Agent ID can be a string of numbers that will be used to select via the telephone diakpad.

Agent Call Center Log in

Agents can login to call center with *22 from the phone (hardphone or softphone) or via the FusionPBX web interface. Admin and Super Admin accounts can also log other agents in or out.

Enabling G.729 Codec in FreeSWITCH

This tutorial has been created to go over whats needed to configure the non-licensed version of the g729 codec.  This codec would normally cost a license fee, but recently the patent expired, which allows the codec to be used without paying a license fee.

Adding the needed repository:

The repository below is for Centos 7 64 bits:

rpm -ivh http://repo.okay.com.mx/centos/7/x86_64/release/okay-release-1-1.noarch.rpm

Installing g729 codec:

yum install freeswitch-codec-bcg729

Please edit the following config files:

In /etc/freeswitch/autoload_config/modules.conf.xml please replace mod_g729 with mod_bcg729
vim /etc/freeswitch/autoload_config/modules.conf.xml

The file should like the following when complete – Notice the third load module statement.  It now contains the license free mod_bcg729 module versus the mod_g729 module.

In /etc/freeswitch/vars.xml please replace mod_g729 with mod_bcg729.

vim /etc/freeswitch/vars.xml

Within the same config file edit the following line following line to ensure g729 is a preferred codec along with any other codec you have installed.

If the install has been done correctly you should be able to use g729 codec without an issue.

Setting up FreeSWITCH WebRTC functionality

This tutorial will go over how to setup WebRTC on FreeSWITCH using a certificate from letsencrypt. WebRTC is a protocol which allows voip calls to be conducted over a web browser without additional plugins or software. This tutorial will assume you are Debian 8, which is the recommended OS for production FreeSWTICH servers.

FreeSWITCH WebRTC encryption using letsencrypt

We will use letsencrypt to create tls certificates for our FreeSWITCH server and automate the renewal. WebRTC requires a valid tls certificate for security purposes, and letsencrypt is a cheap and easy way to obtain one.

echo "deb http://ftp.debian.org/debian jessie-backports main" >> /etc/sources.list
apt update && apt upgrade
apt install certbot -t jessie-backports
apt install apache2

You should now have the certbot package installed, and apache2 installed with the default configuration of /var/www/html as your root directory.

You will also want to update /etc/hosts and /etc/hostname to reflect the domain name you will be using.

in /etc/hosts append your hostname to the end of the 127.0.1.1 line

127.0.1.1 <somedomain-name>

and in /etc/hostname replace the current name with your hostname.

<somedomain-name>

Next, we will create our certificate. Execute the following command and fill out any necessary fields.

certbot certonly --webroot -w /var/www/html/ -d <somehostname>

You should see the following if it was successful.

 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/pbx.somedomain.com/fullchain.pem. Your cert
   will expire on 2017-04-11. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you lose your account credentials, you can recover through
   e-mails sent to root@pbx.somedomain.com.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Next, we need to setup the auto renewal with cron. These certificates expire every 90 days, and by default the certbot application will renew the certificate if it is within 30 days of expiring. I chose to run the command every week to be safe.

crontab -e
0 0 * * 7 certbot renew

Configuring SSL for HTTP

Since we are using SSL for WebRTC, might as well use the certificate to enable HTTPS on our server as well.

We need to enable the SSL module for Apache2. On Debian, you can issue this command.

sudo a2enmod ssl

Next we need to create a virtual host in /etc/apache2/sites-available/. Create a file named your new domain name in /conf

vim /etc/apache2/sites-available/pbx.somedomain.com.conf

Then include the following configuration. Note, you will need to point the SSL certificates to the correct directory depending on your domain. You will also need to change the ServerName parameter to whatever your domain name is.

<VirtualHost *:443>
        ServerName somedomain-name
        DocumentRoot /var/www/html
        SSLEngine on
        SSLProtocol all -SSLv3 -SSLv3
        SSLCompression off
        SSLHonorCipherOrder On
        SSLCipherSuite EECDH+AESGCM:EECDH+AES:EDH+AES

        SSLCertificateFile /etc/letsencrypt/live/pbx.somedomain.com/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/pbx.somedomain.com/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/pbx.somedomain.com/chain.pem

        ServerAdmin some-email@address.com

        ErrorLog ${APACHE_LOG_DIR}/error-devel.log
        CustomLog ${APACHE_LOG_DIR}/access-devel.log combined
        DirectoryIndex index.html
</VirtualHost>

Enable the new virtual host with a2ensite pbx.somedomain.com.conf. This will create a symlink from sites-available to sites-enabled. Then Restart Apache2 to enable these changes with systemctl restart apache2.

Installing FreeSWITCH

I chose to use the repository maintained by FreeSWITCH to simplify updates, but you can also compile from source if you wish. If you compiled from source, the FreeSWTICH configuration will be in /usr/local/freeswitch/conf/ instead of inside /etc/.

Use the following commands to setup the FreeSWITCH official repo and install the FreeSWITCH packages on Debian 8.

wget -O - https://files.freeswitch.org/repo/deb/debian/freeswitch_archive_g0.pub | apt-key add -
echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list
apt-get update && apt-get install -y freeswitch-meta-all git

We are going to want the source files later to copy over the verto js demo.

cd /usr/local/src/
git clone https://freeswitch.org/stash/scm/fs/freeswitch.git -bv1.6 freeswitch

Installing the java script dependencies for the verto demo.

This will install npm and other dependencies for the verto demo to run.

apt update
apt install npm nodejs-legacy
npm install -g grunt grunt-cli bower
npm install
bower --allow-root install
grunt build

Configuring FreeSWTICH

We are going to use the letsencrypt tls certificate we installed earlier for WebRTC. A combined key needs to be created from fullchain.pem and privkey.pem.

cat /etc/letsencrypt/live/somedomain.com/fullchain.pem /etc/letsencrypt/live/somedomain.com/privkey.pem > /etc/freeswitch/tls/wss.pem

Some changes will need to be made to the /etc/freeswitch/autoload_configs/verto.conf.xml file. I removed the ipv6 profile for simplicity.

<configuration name="verto.conf" description="HTML5 Verto Endpoint">

  <settings>
    <param name="debug" value="10"/>
    <!-- seconds to wait before hanging up a disconnected channel -->
    <!-- <param name="detach-timeout-sec" value="120"/> -->
    <!-- enable broadcasting all FreeSWITCH events in Verto -->
    <!-- <param name="enable-fs-events" value="false"/> -->
    <!-- enable broadcasting FreeSWITCH presence events in Verto -->
    <!-- <param name="enable-presence" value="true"/> -->
  </settings>

  <profiles>
    <profile name="somedomain">
      <param name="bind-local" value="0.0.0.0:8081"/>
      <param name="bind-local" value="0.0.0.0:8082" secure="true"/>
      <param name="force-register-domain" value="$${domain}"/>
      <param name="secure-combined" value="/etc/freeswitch/tls/wss.pem"/>
      <param name="secure-chain" value="/etc/freeswitch/tls/wss.pem"/>
      <param name="userauth" value="true"/>
      <param name="context" value="public"/>
      <param name="dialplan" value="XML"/>
      <!-- setting this to true will allow anyone to register even with no account so use with care -->
      <param name="blind-reg" value="false"/>
      <param name="mcast-ip" value="224.1.1.1"/>
      <param name="mcast-port" value="1337"/>
      <param name="rtp-ip" value="$${local_ip_v4}"/>
      <!--  <param name="ext-rtp-ip" value=""/> -->
      <param name="local-network" value="localnet.auto"/>
      <param name="outbound-codec-string" value="opus,vp8"/>
      <param name="inbound-codec-string" value="opus,vp8"/>

      <param name="apply-candidate-acl" value="localnet.auto"/>
      <param name="apply-candidate-acl" value="wan_v4.auto"/>
      <param name="apply-candidate-acl" value="rfc1918.auto"/>
      <param name="apply-candidate-acl" value="any_v4.auto"/>
      <param name="timer-name" value="soft"/>

    </profile>
  </profiles>
</configuration>

Uncomment the following line in /etc/freeswitch/directory/default.xml, or whatever directory you would like to be able to use mod_verto for WebRTC.

      <param name="jsonrpc-allowed-event-channels" value="demo,conference,presence"/>

Now we can copy over the demo verto client to our web server directory

cp -r /usr/local/src/freeswitch/html5/verto/demo/ /var/www/html/
chown -R www-data:www-data /var/www/html/

You should now be able to go to https://<your-domain-name>/demo and you will see the demo WebRTC interface. By default, this client will register to your FreeSWITCH server as extension 1008 with the default password. Because we are registered as a user in the Default directory, our calls will be processed by the default context. You can add some dialplan to test the setup in /etc/freeswitch/dialplan/default/ Such as the following (replace the playback data with the path to a sound file to play back. When dialing 12345, you should hear the audio file played back.)

<include>
  <extension name="verto_test">
    <condition field="destination_number" expression="^(12345)$">
             <action application="answer"/>
             <action application="log" data="INFO ********* VERTO WEBRTC CALL *******" />
             <action application="playback" data="path-to-some-sound-file.wav"/>
             <action application="hangup" />
    </condition>
  </extension>
</include>

Multi-Tenant Configuration in FreeSWITCH

In this article we will be going over the basics of setting up a multi-tennant environment in FreeSWITCH

Directory

We will need to create a new directory for our second tennant. The default configuration is a good place to start from, so copy over the default.xml file and the default directory to the domain name of your new company.

cp default.xml dopensource.com.xml
cp -r default dopensource.com

Now, change the domain name, group name, and include directory with vim dopensource.com.xml

  <domain name="dopensource.com">

...

   <group name="dopensource.com">

...

   <X-PRE-PROCESS cmd="include" data="dopensoruce.com/*.xml"/>

By default, each user is set to use the default context. We will need to change this if we want each company to have a different context. You can either change this in each users include file, or remove this line from those files and set it in the global directory xml file.

To remove all of the user_context lines from the user’s include files, run this sed command.

sed -i '/<variable name="user_context" value="default"\/>/d' /etc/freeswitch/directory/dopensource.com/*

Then add the user_context setting in the directory xml file with vim dopensource.com.xml

    <variables>
      <variable name="record_stereo" value="true"/>
      <variable name="default_gateway" value="$${default_provider}"/>
      <variable name="default_areacode" value="$${default_areacode}"/>
      <variable name="transfer_fallback_extension" value="operator"/>
      <variable name="user_context" value="dopensource.com"/>
    </variables>

Now when a user registered to our system via dopensource.com, their calls will be processed in the context dopensource.com. We will need to create this context for this to work though.

Dialplan

Now that we have setup our Directory, it is time to create a seperate context for our new company. We will start from the defaul.xml file as a base.

cd /etc/freeswitch/dialplan/
cp default.xml dopensource.com.xml
cp -r default dopensource.com

Now, we will need to change the name in dopensource.com to reflect our new context with vim dopensource.com.xml

  <context name="dopensource.com">

We also want to adjust the include directory.

    <X-PRE-PROCESS cmd="include" data="dopensource.com/*.xml"/>

FreeSWITCH should now be configured to host a second company and seperate it into a new context. You can repeat this process for as many domain names as you like.

You will need to change your dns settings to make sure the domain name you use in your directory is actually routable to your FreeSWITCH server. Also, don’t forget to register to this domain name when setting up your phone.

Testing FreeSWITCH for memory leaks using Valgrind and SIPp

In this tutorial we will assume you have already setup SIPp on another server to stress test FreeSWITCH. We covered this in our previous article

We will need to install valgrind on the same machine running FreeSWITCH. I like using Debian for FreeSWITCH, which conveniently already has valgrind in the repository.

apt install valgrind

Now that valgrind is installed, we will need to make sure FreeSWITCH is not already running.

systemctl stop freeswitch
ps -ef | grep freeswitch

Once we stop FreeSWITCH, we want to start up FreeSWITCH using valgrind. Below is the command to do so. This will log all memory issues found by Valgrind to vg.log in the directory you run the command from.

/usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Running the FreeSWITCH from the stable repository will not result in many errors. Below is my the end of my vg.log while running FreeSWITCH 1.6.12-20-b91a0a6~64bit

==20986== LEAK SUMMARY:
==20986==    definitely lost: 0 bytes in 0 blocks
==20986==    indirectly lost: 0 bytes in 0 blocks
==20986==      possibly lost: 16,384 bytes in 2 blocks
==20986==    still reachable: 192 bytes in 1 blocks
==20986==         suppressed: 0 bytes in 0 blocks
==20986==
==20986== For counts of detected and suppressed errors, rerun with: -v
==20986== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

There are no new errors after the leak summary in this log.

When the FreeSWITCH is having memory management problems, valgrind will continue to log errors after the LEAK SUMMARY. Below is an example of a memory leak when accessing the libcrypto.so library for WebRTC.

143768 ==32354== Use of uninitialised value of size 8
143769 ==32354==    at 0x67C8B80: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
143770 ==32354==    by 0xF684D2C1E662C6C4: ???
143771 ==32354==    by 0xE16640B5E123D1DD: ???
143772 ==32354==    by 0xD68DF16DAF4386F1: ???
143773 ==32354==    by 0xB2B603F64080C703: ???
143774 ==32354==    by 0x398A1FCC7DC2FF3B: ???
143775 ==32354==    by 0xBE60B2707D98773F: ???
143776 ==32354==    by 0x375392F355F20B45: ???
143777 ==32354==    by 0x124A183B188AA4EA: ???
143778 ==32354==    by 0x5BFE669681E01F8: ???
143779 ==32354==    by 0x6A4C9E9A4AA5A4C0: ???
143780 ==32354==    by 0x29B07C509D28C49F: ???
143781 ==32354==

In this case, valgrind was logging this message to the log many times per second. You can view the memory mismanagement manifesting itself in a memory leak by viewing the allocated memory with ps. We will want to monitor the change in RSS, or Resident Set Size, which represents the ammount of bytes are allocated for a process in RAM.

ps -aux | grep '[v]algrind.bin'

On Debian 8, ps -aux will list the information in the following order:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

The output of your ps -aux command should look something like this

root     20987 36.7 13.1 631032 271356 ?       S<sl 13:33  30:47 /usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Knowing the current total allocated memory for FreeSWITCH is cool, but not very useful for seeing the memory leak in action. If FreeSWITCH has a memory leak, it will fail to release memory when it is done with it, causing the allocated space to continue to grow. A useful way to observe this is by monitoring the RSS of the FreeSWITCH process over time, then running SIPp to create an artificial load on FreeSWTICH I created a simple bash script to log FreeSWITCH’s RSS, the current time, and the number of sip channels to a file.

#!/bin/sh


while true; do

INT=60
FILE=/root/mem.log
DATE=$(date +"%F-%R")
MEM=$(ps -aux | grep '[v]algrind.bin' | awk -F ' ' '{print $6}')
CHANNELS=$(/usr/local/freeswitch/bin/fs_cli -x "show channels" | grep total | awk -F ' ' '{print $1}')

echo ${DATE},${MEM},${CHANNELS} >> ${FILE}
sleep ${INT}s
done

This script will log the time, RSS, and SIP channels to /root/mem.log. You may need to adjust the CHANNELS variable to point to wherever the fs_cli binary is stored.

Save the script to a file, such as memtest.sh, then run it with

./memtest.sh &

You can confirm it is running in the background with jobs.

Now, with FreeSWITCH running under valgrind, we will want to start up the SIPp stress test. On the SIPp server, run the following.

sipp -sf uac.xml -s 9999999  <freeswitch-ip>:5080 -trace_msg -l 15 -d 60000

This test is best done over a long period of time. I would let SIPp generate traffic on the server for a couple hours while our monitor script is running. After the system has ran under load for some time, the mem.log should look soemthing like this.

2016-10-24-15:29,271212,15
2016-10-24-15:29,270800,15
2016-10-24-15:30,270672,15
2016-10-24-15:31,270828,15
2016-10-24-15:32,270708,15
2016-10-24-15:33,270604,15
2016-10-24-15:34,270812,15
2016-10-24-15:35,270628,15
2016-10-24-15:36,270816,15

If FreeSWITCH has been running for multiple hours and the RSS has seemed to plateau, like above, FreeSWITCH is likely not showing any critical memory mismanagements. Here is an example of the RSS continuing to grow due to a memory leak.

2016-08-11-04:09,379192,60
2016-08-11-04:10,381584,60
2016-08-11-04:11,383104,60
2016-08-11-04:12,385096,60
2016-08-11-04:13,385356,60
2016-08-11-04:14,387580,60
2016-08-11-04:16,387432,60
2016-08-11-04:17,389316,60
2016-08-11-04:18,391144,60
2016-08-11-04:19,393076,60
2016-08-11-04:20,394484,61
2016-08-11-04:21,395776,60
2016-08-11-04:22,397072,60
2016-08-11-04:23,398140,59
2016-08-11-04:24,400112,60
2016-08-11-04:25,401868,60
2016-08-11-04:26,403348,60

FreeSWITCH had been running for many hours at this point, and the RSS had never stopped growing. This is the same system which was logging the libcrypto error in the valgrind log.

When you notice a memory issue with your version of FreeSWTICH, check the FreeSWTICH JIRA for known bugs. There may already be a patch out for the issue you are experiencing.

SIPp – Load Testing FreeSWITCH

In this article we will go over how to get SIPP installed and start up a basic load test for FreeSWITCH.

Installing SIPp

We recommend installing SIPp to a different machine than where you are running FreeSWITCH. This will provide the most realistic stress test. You can download SIPP here: https://sourceforge.net/projects/sipp/files/sipp/3.4/

Run the following commands once you have the SIPp tar downloaded to get it compiled.

tar -xzf sipp-3.3.990.tar.gz
cd sipp-3.3.990
apt install libncurses6-dev libncurses6
./configure; make

If you are on Debian 8, You will need to install the following ncurses libraries instead:

apt install libncurses5-dbg libncurses5 libncurses

SIPp will now be installed in the directory you ran the make command in.

FreeSWITH Dialplan

We will create a basic extension in FreeSWITCH to process the sip traffic we will be sending via SIPp.

vim dialplan/public/0000000_sipp.xml

<include>
  <extension name="sipp_test">
    <condition field="destination_number" expression="^(9999999)$">
    <action application="answer"/>
    <action application="playback" data="ivr/ivr-welcome_to_freeswitch.wav"/>
    <action application="hangup"/>
    </condition>
  </extension>
</include>

Once you have this configured, give FreeSWITCH the reloadxml command:

fs_cli -x 'reloadxml'

This will create the extension sipp_test in the public context, which is listening for incoming traffic to the number 9999999. Freeswitch will playback an ivr message to every calling matching this number.

Next we will want to make sure our SIPp server is able to reach FreeSWITCH on the sip and rtp ports. Run the following iptables commands:

iptables -A INPUT -p udp --dport 5080 -s -j ACCEPT
iptables -A INPUT -p udp -m multiport --dports 16384:32768 -s -j ACCEPT

SIPp Testing

We are going to need a scenario file. You can paste the following into an xml file called uac.xml

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE scenario SYSTEM "sipp.dtd">

<!-- This program is free software; you can redistribute it and/or      -->
<!-- modify it under the terms of the GNU General Public License as     -->
<!-- published by the Free Software Foundation; either version 2 of the -->
<!-- License, or (at your option) any later version.                    -->
<!--                                                                    -->
<!-- This program is distributed in the hope that it will be useful,    -->
<!-- but WITHOUT ANY WARRANTY; without even the implied warranty of     -->
<!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the      -->
<!-- GNU General Public License for more details.                       -->
<!--                                                                    -->
<!-- You should have received a copy of the GNU General Public License  -->
<!-- along with this program; if not, write to the                      -->
<!-- Free Software Foundation, Inc.,                                    -->
<!-- 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA             -->
<!--                                                                    -->
<!--                 Sipp default 'uac' scenario.                       -->
<!--                                                                    -->

<scenario name="Basic Sipstone UAC">
  <!-- In client mode (sipp placing calls), the Call-ID MUST be         -->
  <!-- generated by sipp. To do so, use [call_id] keyword.                -->
  <send retrans="500">
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>
  <recv response="100"
        optional="true">
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>

  <!-- By adding rrs="true" (Record Route Sets), the route sets         -->
  <!-- are saved and used for following messages sent. Useful to test   -->
  <!-- against stateful SIP proxies/B2BUAs.                             -->
  <recv response="200" rtd="true">
  </recv>

  <!-- Packet lost can be simulated in any send/recv message by         -->
  <!-- by adding the 'lost = "10"'. Value can be [1-100] percent.       -->
  <send>
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 1 ACK
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <!-- This delay can be customized by the -d command-line option       -->
  <!-- or by adding a 'milliseconds = "value"' option here.             -->
  <pause/>

  <!-- The 'crlf' option inserts a blank line in the statistics report. -->
  <send retrans="500">
    <![CDATA[

      BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 2 BYE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <recv response="200" crlf="true">
  </recv>

  <!-- definition of the response time repartition table (unit is ms)   -->
  <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

  <!-- definition of the call length repartition table (unit is ms)     -->
  <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

We can now run the following command to start the SIPp load test. This will start up an ncurses graph to show you the progress.

sipp -sf uac.xml -s 9999999 :5080 -trace_msg -l 15 -d 60000

In the next tutorial we will go over how to analyze FreeSWITCH’s memory management with valgrind while under a stress test.

FreeSWITCH, configuring the Callcenter Module

We will go over how to setup the call center module. The callcenter module is used for creating an inbound queue for connecting inbound callers with agents registered to your system. We will be assuming you followed our FreeSWITCH setup tutorial here

MOH dependency

Make sure you have installed the moh sounds, which on debian is done with apt install freeswitch-music-default.

callcenter.conf.xml

You will first need to make sure the callcenter module is enabled in modules.conf

    <load module="mod_callcenter"/>

Next open up autoload_configs/callcenter.conf.xml. A default queue is provided, which we will make some modifications to.

    <queue name="test@default">
      <param name="strategy" value="longest-idle-agent"/>
      <param name="moh-sound" value="$${hold_music}"/>
      <param name="time-base-score" value="system"/>
      <param name="max-wait-time" value="0"/>
      <param name="max-wait-time-with-no-agent" value="0"/>
      <param name="max-wait-time-with-no-agent-time-reached" value="5"/>
      <param name="tier-rules-apply" value="false"/>
      <param name="tier-rule-wait-second" value="300"/>
      <param name="tier-rule-wait-multiply-level" value="true"/>
      <param name="tier-rule-no-agent-no-wait" value="false"/>
      <param name="discard-abandoned-after" value="60"/>
      <param name="abandoned-resume-allowed" value="false"/>
    </queue>

Now we need to add an agent.

<agent name="1000@default" type="callback" contact="[call_timeout=10]user/1000" status="Available" max-no-answer="3" wrap-up-time="10" reject-delay-time="10" busy-delay-time="60" />

And specify the agent’s tier, which also puts them into the queue.

   <tier agent="1000@default" queue="test@default" level="1" position="1"/>

We can now send inbound calls to our callcenter queue in our dialplan.

<include>
  <extension name="test_ivr">
   <condition field="destination_number" expression="^(test_ivr)$">
     <action application="set" data="accountcode=test_in"/>
     <action application="callcenter" data="test@default"/>
     <action application="hangup"/>
   </condition>
  </extension>
</include>