All posts by nschoonover

This tutorial will go over how to setup WebRTC on FreeSWITCH using a certificate from letsencrypt. WebRTC is a protocol which allows voip calls to be conducted over a web browser without additional plugins or software. This tutorial will assume you are Debian 8, which is the recommended OS for production FreeSWTICH servers.

FreeSWITCH WebRTC encryption using letsencrypt

We will use letsencrypt to create tls certificates for our FreeSWITCH server and automate the renewal. WebRTC requires a valid tls certificate for security purposes, and letsencrypt is a cheap and easy way to obtain one.

echo "deb http://ftp.debian.org/debian jessie-backports main" >> /etc/sources.list
apt update && apt upgrade
apt install certbot -t jessie-backports
apt install apache2

You should now have the certbot package installed, and apache2 installed with the default configuration of /var/www/html as your root directory.

You will also want to update /etc/hosts and /etc/hostname to reflect the domain name you will be using.

in /etc/hosts append your hostname to the end of the 127.0.1.1 line

127.0.1.1 <somedomain-name>

and in /etc/hostname replace the current name with your hostname.

<somedomain-name>

Next, we will create our certificate. Execute the following command and fill out any necessary fields.

certbot certonly --webroot -w /var/www/html/ -d <somehostname>

You should see the following if it was successful.

 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/pbx.somedomain.com/fullchain.pem. Your cert
   will expire on 2017-04-11. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you lose your account credentials, you can recover through
   e-mails sent to root@pbx.somedomain.com.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Next, we need to setup the auto renewal with cron. These certificates expire every 90 days, and by default the certbot application will renew the certificate if it is within 30 days of expiring. I chose to run the command every week to be safe.

crontab -e
0 0 * * 7 certbot renew

Configuring SSL for HTTP

Since we are using SSL for WebRTC, might as well use the certificate to enable HTTPS on our server as well.

We need to enable the SSL module for Apache2. On Debian, you can issue this command.

sudo a2enmod ssl

Next we need to create a virtual host in /etc/apache2/sites-available/. Create a file named your new domain name in /conf

vim /etc/apache2/sites-available/pbx.somedomain.com.conf

Then include the following configuration. Note, you will need to point the SSL certificates to the correct directory depending on your domain. You will also need to change the ServerName parameter to whatever your domain name is.

<VirtualHost *:443>
        ServerName somedomain-name
        DocumentRoot /var/www/html
        SSLEngine on
        SSLProtocol all -SSLv3 -SSLv3
        SSLCompression off
        SSLHonorCipherOrder On
        SSLCipherSuite EECDH+AESGCM:EECDH+AES:EDH+AES

        SSLCertificateFile /etc/letsencrypt/live/pbx.somedomain.com/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/pbx.somedomain.com/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/pbx.somedomain.com/chain.pem

        ServerAdmin some-email@address.com

        ErrorLog ${APACHE_LOG_DIR}/error-devel.log
        CustomLog ${APACHE_LOG_DIR}/access-devel.log combined
        DirectoryIndex index.html
</VirtualHost>

Enable the new virtual host with a2ensite pbx.somedomain.com.conf. This will create a symlink from sites-available to sites-enabled. Then Restart Apache2 to enable these changes with systemctl restart apache2.

Installing FreeSWITCH

I chose to use the repository maintained by FreeSWITCH to simplify updates, but you can also compile from source if you wish. If you compiled from source, the FreeSWTICH configuration will be in /usr/local/freeswitch/conf/ instead of inside /etc/.

Use the following commands to setup the FreeSWITCH official repo and install the FreeSWITCH packages on Debian 8.

wget -O - https://files.freeswitch.org/repo/deb/debian/freeswitch_archive_g0.pub | apt-key add -
echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list
apt-get update && apt-get install -y freeswitch-meta-all git

We are going to want the source files later to copy over the verto js demo.

cd /usr/local/src/
git clone https://freeswitch.org/stash/scm/fs/freeswitch.git -bv1.6 freeswitch

Installing the java script dependencies for the verto demo.

This will install npm and other dependencies for the verto demo to run.

apt update
apt install npm nodejs-legacy
npm install -g grunt grunt-cli bower
npm install
bower --allow-root install
grunt build

Configuring FreeSWTICH

We are going to use the letsencrypt tls certificate we installed earlier for WebRTC. A combined key needs to be created from fullchain.pem and privkey.pem.

cat /etc/letsencrypt/live/somedomain.com/fullchain.pem /etc/letsencrypt/live/somedomain.com/privkey.pem > /etc/freeswitch/tls/wss.pem

Some changes will need to be made to the /etc/freeswitch/autoload_configs/verto.conf.xml file. I removed the ipv6 profile for simplicity.

<configuration name="verto.conf" description="HTML5 Verto Endpoint">

  <settings>
    <param name="debug" value="10"/>
    <!-- seconds to wait before hanging up a disconnected channel -->
    <!-- <param name="detach-timeout-sec" value="120"/> -->
    <!-- enable broadcasting all FreeSWITCH events in Verto -->
    <!-- <param name="enable-fs-events" value="false"/> -->
    <!-- enable broadcasting FreeSWITCH presence events in Verto -->
    <!-- <param name="enable-presence" value="true"/> -->
  </settings>

  <profiles>
    <profile name="somedomain">
      <param name="bind-local" value="0.0.0.0:8081"/>
      <param name="bind-local" value="0.0.0.0:8082" secure="true"/>
      <param name="force-register-domain" value="$${domain}"/>
      <param name="secure-combined" value="/etc/freeswitch/tls/wss.pem"/>
      <param name="secure-chain" value="/etc/freeswitch/tls/wss.pem"/>
      <param name="userauth" value="true"/>
      <param name="context" value="public"/>
      <param name="dialplan" value="XML"/>
      <!-- setting this to true will allow anyone to register even with no account so use with care -->
      <param name="blind-reg" value="false"/>
      <param name="mcast-ip" value="224.1.1.1"/>
      <param name="mcast-port" value="1337"/>
      <param name="rtp-ip" value="$${local_ip_v4}"/>
      <!--  <param name="ext-rtp-ip" value=""/> -->
      <param name="local-network" value="localnet.auto"/>
      <param name="outbound-codec-string" value="opus,vp8"/>
      <param name="inbound-codec-string" value="opus,vp8"/>

      <param name="apply-candidate-acl" value="localnet.auto"/>
      <param name="apply-candidate-acl" value="wan_v4.auto"/>
      <param name="apply-candidate-acl" value="rfc1918.auto"/>
      <param name="apply-candidate-acl" value="any_v4.auto"/>
      <param name="timer-name" value="soft"/>

    </profile>
  </profiles>
</configuration>

Uncomment the following line in /etc/freeswitch/directory/default.xml, or whatever directory you would like to be able to use mod_verto for WebRTC.

      <param name="jsonrpc-allowed-event-channels" value="demo,conference,presence"/>

Now we can copy over the demo verto client to our web server directory

cp -r /usr/local/src/freeswitch/html5/verto/demo/ /var/www/html/
chown -R www-data:www-data /var/www/html/

You should now be able to go to https://<your-domain-name>/demo and you will see the demo WebRTC interface. By default, this client will register to your FreeSWITCH server as extension 1008 with the default password. Because we are registered as a user in the Default directory, our calls will be processed by the default context. You can add some dialplan to test the setup in /etc/freeswitch/dialplan/default/ Such as the following (replace the playback data with the path to a sound file to play back. When dialing 12345, you should hear the audio file played back.)

<include>
  <extension name="verto_test">
    <condition field="destination_number" expression="^(12345)$">
             <action application="answer"/>
             <action application="log" data="INFO ********* VERTO WEBRTC CALL *******" />
             <action application="playback" data="path-to-some-sound-file.wav"/>
             <action application="hangup" />
    </condition>
  </extension>
</include>

In our previous article we setup google-authenticator for authenticating openssh. Now, we need a way for users to be able to login once before setting up google-authenticator. Here is a script for checking if a user has not logged in and ran google-authentication yet, runs google-authenticator, then prevents that user from logging in again without either google-authentication or an ssh public key. To setup this script do the following:

1. Edit /etc/pam.d/sshd and make the following changes after #%PAM-1.0

   auth    required        pam_sss.so try_first_pass
   auth    sufficient  pam_listfile.so item=user sense=allow file=/google-auth/authusers
   auth    sufficient      pam_google_authenticator.so
   auth       required     pam_sepermit.so
   #auth       include      password-auth
   

   We are using ldap, so our users authenticate with pam_sss.so. If you are using regular unix passwords, you should change this first required auth line to pam_unix.so instead. We setup a pam_listfile, which has a list of all the users we want to allow access to login. Our script, called from each of these users .bash_profile file, will remove the user from this file after their first login without google-authenticator setup. Notice, we commented out the password-auth include line. We do not want to call the password-auth file because we only want users to be able to log in if they are in if they can first authenticate with sss/ldad and are in our google-auth list. Otherwise, they can login with google_authenticator, or from an sshd key, which will happen before the authentication is sent to pam.

2. Create the /google-auth/authusers file, which pam will check for which users to allow without requiring further authentication. I used /home/ to see which users are actual human users who will need to login. This file just needs to be a list with one username per line. This file needs to be in a new directory, because we need to give all of these users permission to write to it so their usernames can be removed via our script once they create a google authentication file.

   a. create a group and add all of the necesarry users to this group.

   groupadd google-auth
   gpasswd google-auth -M bob,joe,smith
   

   b. Now create the authusers file and set the permissions to be owned by google-auth, then allowed write access by users in that group.

   mkdir /google-auth/
   touch /google-auth/authusers
   chgrp google-auth /google-auth/authusers
   chmod ug=rwx,o= /google-auth/authusers
   

   c. Now add the necesarry users to /google-auth/authusers

   bob
   joe
   smith
   

3. Install the script to be ran. Just add these couple lines to an .sh file in /usr/local/bin, or you can copy it over if you did a git pull. I created my script with vim /usr/local/bin/google-auth-check.sh

   #!/bin/bash
   
   if [ ! -f ~/.google_authenticator ]; then
       google-authenticator
       if [ -f ~/.google_authenticator ]; then
           sed -i "/^${USER}$/d" /google-auth/authusers
       fi
   fi
   

   And then give it execute permissions.

   chmod +x /usr/local/bin/google-auth-check.sh
   

4. Next, we need to add this script to each of the ~/.bash\_profile files in each users home directory. I will come up with a find and echo command, but for now, just add this at the end of these files in each users directory you want this to work for.

   sh /usr/local/bin/google-auth-check.sh
   

You can run this bash command to find every .bash_profile in /home/ and append the line above. Be careful before running this command. It works on my system, but you can never be too careful with recursive commands such as this.

find /home/ -name ".bash_profile" -print0 | xargs --verbose -0 -I{} sh -c "echo 'sh /usr/local/bin/google-auth-check.sh' >> {}"

This should be all you need to do. Now when a user logs in for the first time, the google-authenticator application will automatically run and create a .google\_authenticator file. The user will then be able to add the key into their phone app and have multifactor authentication to log into their account. Their name will get removed from the /google-auth/authusers file, so they will no longer be given access without a multi factor auth, or ssh key.

Sources:

https://www.digitalocean.com/community/tutorials/how-to-use-pam-to-configure-authentication-on-an-ubuntu-12-04-vps

http://www.linux-pam.org/Linux-PAM-html/sag-pam_exec.html

https://wiki.archlinux.org/index.php/PAM

Install google-authenticator

In this tutorial we will go over how to setup two factor authentication for SSH using google-authenticator on CentOS 6.

First thing we need to do is install the google-authenticator package using yum.

yum install google-authenticator

Next, run google-authenticator to genereate a key.

google-authenticator
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/nschoonover@ldap.dopensource.com%3Fsecret%3D3FO6WYXPZUMTEWUB
Your new secret key is: 3FO6WYXPZUMTEWUB
Your verification code is 799347
Your emergency scratch codes are:
  84194653
  29899626
  13794318
  84024610
  76941184

Do you want me to update your "~/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n)
By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n)

With this information, you will need to install the google authenticator application on your phone. Copy and paste the URL at the beginning of the output from google-authenticator, which should show you a QR code. Scan this code using the Google Authenticator application on your phone. You will now see a 6 digit number in your Google Authenticator app. This number will change every certain ammount of minutes. This number will be your second authentication method when logging into your server. If you ever lose your secret key and need to setup another phone, you can always run the google-authenticator command again to regenerate.

Next, we need to setup pam to use google_authenticator for authenticating users.

Open up the pam ssh configuration file.

sudo vim /etc/pam.d/sshd

Then add the following lines at the bottom.

## Google autehnticator for ssh
auth required pam_google_authenticator.so

Next, we need to add the ChallengeResponseAuthentication yes option to the /etc/ssh/sshd\_config file.

# Change to no to disable s/key passwords
ChallengeResponseAuthentication yes
#ChallengeResponseAuthentication no

Now restart sshd for the new changes to take affect.

sudo service sshd restart

Troubleshooting

The first time I tried to set this up, I was unable to login using my authentication code. I would be prompted for a code when tried to initiate an ssh connection, but I would see this message in the logs and would not be let in.

Dec 28 17:49:43 web2p sshd(pam_google_authenticator)[19784]: Invalid verification code

It turns out the problem was caused by the time on my testing server not being correct. Make sure you have ntp running and configured correctly.

To install and set ntp to auto start:

yum install ntp
chkconfig ntpd on
service ntpd start

Now run this command to manually force ntp to sync with pool.ntp.org

ntpdate pool.ntp.org

You can check the date on your server with the date command to make sure ntp is working. After syncing my time, I was able to log into my ssh server using google authenticator without a problem.

Public Key Authentication

We decided that we wanted to allow users to login using either Google Authenticator, or a regular ssh Pulic Key. We generally use public keys for ssh authentication, but occasionally we will want to login to one of our servers from a machine we will only use once. Google Authenticator is a great compromise for this as it allows us to authenticate without an ssh key, but still maintains our security.

To ensure sshd will allow authentication from either the Google Authenticator module in pam, or an ssh key, uncommend the following lines in /etc/ssh/sshd\_config

PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication yes
UsePAM yes

The changes we made to /etc/pam.d/sshd should only effect the password authentication portion of the ssh login process. If PubkeyAuthentication yes is set, ssh should approve your ssh key authentication without reaching out to pam and requiring two factor authentication.

In our next article, we will go over how to allow users to login once before google-authentication is enforced.

Installing Jungledisk

If you are installing the Jungle Disk agent to a server via command line, you will need to use wget or another cli http client to download the installer. We are installing Jungle Disk to a CentOS 6 server, but there are also packages for Debian based systems, and a tar file with the source if you would like to compile.

Download the installer, and run the rpm to setup and start the backup agent.

wget https://downloads.jungledisk.com/jungledisk/junglediskserver-3180-0.x86_64.rpm
[root@ldap ~]# sudo rpm -i junglediskserver-3180-0.x86_64.rpm
Stopping junglediskserver: [FAILED]
Starting junglediskserver:
[root@ldap ~]# ps -ef | grep jungle
root 22936 1 0 15:58 ? 00:00:00 /usr/local/bin/junglediskserver

This is the rpm package as of the time this article is written, but as there are new releases of the agent software, this will change. Make sure you download the latest version from your Jungle Disk account.

Setup the license

[root@ldap jungledisk]# cd /etc/jungledisk/
[root@ldap jungledisk]# cp junglediskserver-license-EXAMPLE.xml junglediskserver-license.xml

Now you will need to add your license to the new junglediskserver-license.xml file. You can find this license in your licenses section of the control panel. https://secure.jungledisk.com/secure/account/subscriptions.aspx

vim /etc/jungledisk/junglediskserver.xml

Enter in your license string in the licenseKey section.

<licenseKey>some_license_string</licenseKey>

Setting up the Management Client and configuring a backup.

You will need to install the Management Client to a Windows computer. Download the client here: https://secure.jungledisk.com/downloads/business/server/windows/

Once it is downloaded and installed, you will need to log in. Click Next to start the configuration.

If you already have setup Jungle Disk to backup another server, you will need to chose your new server from the server list. The client will be the hostname on the server. Now, create a new online disk.

Choose between Standard Security, or High, which will encrypt your data whil it is stored on the Jungle Disk infrastructure. (We chose standard)

Select you backup schedule. We stayed with the default daily schedule.

Next, select what files to backup on your linux server.

Hit ok, and next. You should be done! You can now try running a manual backup to get your files backed up right away, or wait for the scheduled backup to run.

In this article we will be going over the basics of setting up a multi-tennant environment in FreeSWITCH

Directory

We will need to create a new directory for our second tennant. The default configuration is a good place to start from, so copy over the default.xml file and the default directory to the domain name of your new company.

cp default.xml dopensource.com.xml
cp -r default dopensource.com

Now, change the domain name, group name, and include directory with vim dopensource.com.xml

  <domain name="dopensource.com">

...

   <group name="dopensource.com">

...

   <X-PRE-PROCESS cmd="include" data="dopensoruce.com/*.xml"/>

By default, each user is set to use the default context. We will need to change this if we want each company to have a different context. You can either change this in each users include file, or remove this line from those files and set it in the global directory xml file.

To remove all of the user_context lines from the user’s include files, run this sed command.

sed -i '/<variable name="user_context" value="default"\/>/d' /etc/freeswitch/directory/dopensource.com/*

Then add the user_context setting in the directory xml file with vim dopensource.com.xml

    <variables>
      <variable name="record_stereo" value="true"/>
      <variable name="default_gateway" value="$${default_provider}"/>
      <variable name="default_areacode" value="$${default_areacode}"/>
      <variable name="transfer_fallback_extension" value="operator"/>
      <variable name="user_context" value="dopensource.com"/>
    </variables>

Now when a user registered to our system via dopensource.com, their calls will be processed in the context dopensource.com. We will need to create this context for this to work though.

Dialplan

Now that we have setup our Directory, it is time to create a seperate context for our new company. We will start from the defaul.xml file as a base.

cd /etc/freeswitch/dialplan/
cp default.xml dopensource.com.xml
cp -r default dopensource.com

Now, we will need to change the name in dopensource.com to reflect our new context with vim dopensource.com.xml

  <context name="dopensource.com">

We also want to adjust the include directory.

    <X-PRE-PROCESS cmd="include" data="dopensource.com/*.xml"/>

FreeSWITCH should now be configured to host a second company and seperate it into a new context. You can repeat this process for as many domain names as you like.

You will need to change your dns settings to make sure the domain name you use in your directory is actually routable to your FreeSWITCH server. Also, don’t forget to register to this domain name when setting up your phone.

In this tutorial we will assume you have already setup SIPp on another server to stress test FreeSWITCH. We covered this in our previous article

We will need to install valgrind on the same machine running FreeSWITCH. I like using Debian for FreeSWITCH, which conveniently already has valgrind in the repository.

apt install valgrind

Now that valgrind is installed, we will need to make sure FreeSWITCH is not already running.

systemctl stop freeswitch
ps -ef | grep freeswitch

Once we stop FreeSWITCH, we want to start up FreeSWITCH using valgrind. Below is the command to do so. This will log all memory issues found by Valgrind to vg.log in the directory you run the command from.

/usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Running the FreeSWITCH from the stable repository will not result in many errors. Below is my the end of my vg.log while running FreeSWITCH 1.6.12-20-b91a0a6~64bit

==20986== LEAK SUMMARY:
==20986==    definitely lost: 0 bytes in 0 blocks
==20986==    indirectly lost: 0 bytes in 0 blocks
==20986==      possibly lost: 16,384 bytes in 2 blocks
==20986==    still reachable: 192 bytes in 1 blocks
==20986==         suppressed: 0 bytes in 0 blocks
==20986==
==20986== For counts of detected and suppressed errors, rerun with: -v
==20986== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

There are no new errors after the leak summary in this log.

When the FreeSWITCH is having memory management problems, valgrind will continue to log errors after the LEAK SUMMARY. Below is an example of a memory leak when accessing the libcrypto.so library for WebRTC.

143768 ==32354== Use of uninitialised value of size 8
143769 ==32354==    at 0x67C8B80: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
143770 ==32354==    by 0xF684D2C1E662C6C4: ???
143771 ==32354==    by 0xE16640B5E123D1DD: ???
143772 ==32354==    by 0xD68DF16DAF4386F1: ???
143773 ==32354==    by 0xB2B603F64080C703: ???
143774 ==32354==    by 0x398A1FCC7DC2FF3B: ???
143775 ==32354==    by 0xBE60B2707D98773F: ???
143776 ==32354==    by 0x375392F355F20B45: ???
143777 ==32354==    by 0x124A183B188AA4EA: ???
143778 ==32354==    by 0x5BFE669681E01F8: ???
143779 ==32354==    by 0x6A4C9E9A4AA5A4C0: ???
143780 ==32354==    by 0x29B07C509D28C49F: ???
143781 ==32354==

In this case, valgrind was logging this message to the log many times per second. You can view the memory mismanagement manifesting itself in a memory leak by viewing the allocated memory with ps. We will want to monitor the change in RSS, or Resident Set Size, which represents the ammount of bytes are allocated for a process in RAM.

ps -aux | grep '[v]algrind.bin'

On Debian 8, ps -aux will list the information in the following order:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

The output of your ps -aux command should look something like this

root     20987 36.7 13.1 631032 271356 ?       S<sl 13:33  30:47 /usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Knowing the current total allocated memory for FreeSWITCH is cool, but not very useful for seeing the memory leak in action. If FreeSWITCH has a memory leak, it will fail to release memory when it is done with it, causing the allocated space to continue to grow. A useful way to observe this is by monitoring the RSS of the FreeSWITCH process over time, then running SIPp to create an artificial load on FreeSWTICH I created a simple bash script to log FreeSWITCH’s RSS, the current time, and the number of sip channels to a file.

#!/bin/sh


while true; do

INT=60
FILE=/root/mem.log
DATE=$(date +"%F-%R")
MEM=$(ps -aux | grep '[v]algrind.bin' | awk -F ' ' '{print $6}')
CHANNELS=$(/usr/local/freeswitch/bin/fs_cli -x "show channels" | grep total | awk -F ' ' '{print $1}')

echo ${DATE},${MEM},${CHANNELS} >> ${FILE}
sleep ${INT}s
done

This script will log the time, RSS, and SIP channels to /root/mem.log. You may need to adjust the CHANNELS variable to point to wherever the fs_cli binary is stored.

Save the script to a file, such as memtest.sh, then run it with

./memtest.sh &

You can confirm it is running in the background with jobs.

Now, with FreeSWITCH running under valgrind, we will want to start up the SIPp stress test. On the SIPp server, run the following.

sipp -sf uac.xml -s 9999999  <freeswitch-ip>:5080 -trace_msg -l 15 -d 60000

This test is best done over a long period of time. I would let SIPp generate traffic on the server for a couple hours while our monitor script is running. After the system has ran under load for some time, the mem.log should look soemthing like this.

2016-10-24-15:29,271212,15
2016-10-24-15:29,270800,15
2016-10-24-15:30,270672,15
2016-10-24-15:31,270828,15
2016-10-24-15:32,270708,15
2016-10-24-15:33,270604,15
2016-10-24-15:34,270812,15
2016-10-24-15:35,270628,15
2016-10-24-15:36,270816,15

If FreeSWITCH has been running for multiple hours and the RSS has seemed to plateau, like above, FreeSWITCH is likely not showing any critical memory mismanagements. Here is an example of the RSS continuing to grow due to a memory leak.

2016-08-11-04:09,379192,60
2016-08-11-04:10,381584,60
2016-08-11-04:11,383104,60
2016-08-11-04:12,385096,60
2016-08-11-04:13,385356,60
2016-08-11-04:14,387580,60
2016-08-11-04:16,387432,60
2016-08-11-04:17,389316,60
2016-08-11-04:18,391144,60
2016-08-11-04:19,393076,60
2016-08-11-04:20,394484,61
2016-08-11-04:21,395776,60
2016-08-11-04:22,397072,60
2016-08-11-04:23,398140,59
2016-08-11-04:24,400112,60
2016-08-11-04:25,401868,60
2016-08-11-04:26,403348,60

FreeSWITCH had been running for many hours at this point, and the RSS had never stopped growing. This is the same system which was logging the libcrypto error in the valgrind log.

When you notice a memory issue with your version of FreeSWTICH, check the FreeSWTICH JIRA for known bugs. There may already be a patch out for the issue you are experiencing.

In this article we will go over how to get SIPP installed and start up a basic load test for FreeSWITCH.

Installing SIPp

We recommend installing SIPp to a different machine than where you are running FreeSWITCH. This will provide the most realistic stress test. You can download SIPP here: https://sourceforge.net/projects/sipp/files/sipp/3.4/

Run the following commands once you have the SIPp tar downloaded to get it compiled.

tar -xzf sipp-3.3.990.tar.gz
cd sipp-3.3.990
apt install libncurses6-dev libncurses6
./configure; make

If you are on Debian 8, You will need to install the following ncurses libraries instead:

apt install libncurses5-dbg libncurses5 libncurses

SIPp will now be installed in the directory you ran the make command in.

FreeSWITH Dialplan

We will create a basic extension in FreeSWITCH to process the sip traffic we will be sending via SIPp.

vim dialplan/public/0000000_sipp.xml

<include>
  <extension name="sipp_test">
    <condition field="destination_number" expression="^(9999999)$">
    <action application="answer"/>
    <action application="playback" data="ivr/ivr-welcome_to_freeswitch.wav"/>
    <action application="hangup"/>
    </condition>
  </extension>
</include>

Once you have this configured, give FreeSWITCH the reloadxml command:

fs_cli -x 'reloadxml'

This will create the extension sipp_test in the public context, which is listening for incoming traffic to the number 9999999. Freeswitch will playback an ivr message to every calling matching this number.

Next we will want to make sure our SIPp server is able to reach FreeSWITCH on the sip and rtp ports. Run the following iptables commands:

iptables -A INPUT -p udp --dport 5080 -s -j ACCEPT
iptables -A INPUT -p udp -m multiport --dports 16384:32768 -s -j ACCEPT

SIPp Testing

We are going to need a scenario file. You can paste the following into an xml file called uac.xml

<?xml version="1.0" encoding="ISO-8859-1" ?>
<!DOCTYPE scenario SYSTEM "sipp.dtd">

<!-- This program is free software; you can redistribute it and/or      -->
<!-- modify it under the terms of the GNU General Public License as     -->
<!-- published by the Free Software Foundation; either version 2 of the -->
<!-- License, or (at your option) any later version.                    -->
<!--                                                                    -->
<!-- This program is distributed in the hope that it will be useful,    -->
<!-- but WITHOUT ANY WARRANTY; without even the implied warranty of     -->
<!-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the      -->
<!-- GNU General Public License for more details.                       -->
<!--                                                                    -->
<!-- You should have received a copy of the GNU General Public License  -->
<!-- along with this program; if not, write to the                      -->
<!-- Free Software Foundation, Inc.,                                    -->
<!-- 59 Temple Place, Suite 330, Boston, MA  02111-1307 USA             -->
<!--                                                                    -->
<!--                 Sipp default 'uac' scenario.                       -->
<!--                                                                    -->

<scenario name="Basic Sipstone UAC">
  <!-- In client mode (sipp placing calls), the Call-ID MUST be         -->
  <!-- generated by sipp. To do so, use [call_id] keyword.                -->
  <send retrans="500">
    <![CDATA[

      INVITE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>
      Call-ID: [call_id]
      CSeq: 1 INVITE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Type: application/sdp
      Content-Length: [len]

      v=0
      o=user1 53655765 2353687637 IN IP[local_ip_type] [local_ip]
      s=-
      c=IN IP[media_ip_type] [media_ip]
      t=0 0
      m=audio [media_port] RTP/AVP 0
      a=rtpmap:0 PCMU/8000

    ]]>
  </send>
  <recv response="100"
        optional="true">
  </recv>

  <recv response="180" optional="true">
  </recv>

  <recv response="183" optional="true">
  </recv>

  <!-- By adding rrs="true" (Record Route Sets), the route sets         -->
  <!-- are saved and used for following messages sent. Useful to test   -->
  <!-- against stateful SIP proxies/B2BUAs.                             -->
  <recv response="200" rtd="true">
  </recv>

  <!-- Packet lost can be simulated in any send/recv message by         -->
  <!-- by adding the 'lost = "10"'. Value can be [1-100] percent.       -->
  <send>
    <![CDATA[

      ACK sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 1 ACK
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <!-- This delay can be customized by the -d command-line option       -->
  <!-- or by adding a 'milliseconds = "value"' option here.             -->
  <pause/>

  <!-- The 'crlf' option inserts a blank line in the statistics report. -->
  <send retrans="500">
    <![CDATA[

      BYE sip:[service]@[remote_ip]:[remote_port] SIP/2.0
      Via: SIP/2.0/[transport] [local_ip]:[local_port];branch=[branch]
      From: sipp <sip:sipp@[local_ip]:[local_port]>;tag=[pid]SIPpTag00[call_number]
      To: [service] <sip:[service]@[remote_ip]:[remote_port]>[peer_tag_param]
      Call-ID: [call_id]
      CSeq: 2 BYE
      Contact: sip:sipp@[local_ip]:[local_port]
      Max-Forwards: 70
      Subject: Performance Test
      Content-Length: 0

    ]]>
  </send>

  <recv response="200" crlf="true">
  </recv>

  <!-- definition of the response time repartition table (unit is ms)   -->
  <ResponseTimeRepartition value="10, 20, 30, 40, 50, 100, 150, 200"/>

  <!-- definition of the call length repartition table (unit is ms)     -->
  <CallLengthRepartition value="10, 50, 100, 500, 1000, 5000, 10000"/>

</scenario>

We can now run the following command to start the SIPp load test. This will start up an ncurses graph to show you the progress.

sipp -sf uac.xml -s 9999999 :5080 -trace_msg -l 15 -d 60000

In the next tutorial we will go over how to analyze FreeSWITCH’s memory management with valgrind while under a stress test.

We will go over how to setup the call center module. The callcenter module is used for creating an inbound queue for connecting inbound callers with agents registered to your system. We will be assuming you followed our FreeSWITCH setup tutorial here

MOH dependency

Make sure you have installed the moh sounds, which on debian is done with apt install freeswitch-music-default.

callcenter.conf.xml

You will first need to make sure the callcenter module is enabled in modules.conf

    <load module="mod_callcenter"/>

Next open up autoload_configs/callcenter.conf.xml. A default queue is provided, which we will make some modifications to.

    <queue name="test@default">
      <param name="strategy" value="longest-idle-agent"/>
      <param name="moh-sound" value="$${hold_music}"/>
      <param name="time-base-score" value="system"/>
      <param name="max-wait-time" value="0"/>
      <param name="max-wait-time-with-no-agent" value="0"/>
      <param name="max-wait-time-with-no-agent-time-reached" value="5"/>
      <param name="tier-rules-apply" value="false"/>
      <param name="tier-rule-wait-second" value="300"/>
      <param name="tier-rule-wait-multiply-level" value="true"/>
      <param name="tier-rule-no-agent-no-wait" value="false"/>
      <param name="discard-abandoned-after" value="60"/>
      <param name="abandoned-resume-allowed" value="false"/>
    </queue>

Now we need to add an agent.

<agent name="1000@default" type="callback" contact="[call_timeout=10]user/1000" status="Available" max-no-answer="3" wrap-up-time="10" reject-delay-time="10" busy-delay-time="60" />

And specify the agent’s tier, which also puts them into the queue.

   <tier agent="1000@default" queue="test@default" level="1" position="1"/>

We can now send inbound calls to our callcenter queue in our dialplan.

<include>
  <extension name="test_ivr">
   <condition field="destination_number" expression="^(test_ivr)$">
     <action application="set" data="accountcode=test_in"/>
     <action application="callcenter" data="test@default"/>
     <action application="hangup"/>
   </condition>
  </extension>
</include>

We will be configuring the cdr_csv module to log all calls going through a specific extension to a seperate file. This is useful if you would like to monitor all calls hitting a certain extension in your dialplan without having to dig through the master CDR file. We will assume that you have setup your FreeSWITCH system using our previous tutorial, FreeSWITCH 1.6 Quick Install Guide for Debian Jessie

CDR

First make sure the mod_cdr_csv module is uncommented in /etc/freeswitch/autoload_configs/modules.conf

    <load module="mod_cdr_csv"/>

Next we need to create a new template in /etc/freeswitch/autoload_configs/cdr_csv.conf.xml. Note that the template will need to be named the same as whatever account code you decide to use in your dialplan later. When an accountcode matches a template name, FreeSWITCH will create a unique CDR file for all calls specified as that accountcode.

    <template name="test_in">"${caller_id_number}","${start_stamp}","${answer_stamp}","${end_stamp}","${accountcode}"</template>

This template will log the caller id, start of call time, answer time, end of call time, and accountcode name to test_in.csv in the log-base directory, which defaults to /var/log/freeswitch/cdr-csv/ You can find a list of cdr variables here

Dialplan

Now in your dialplan you will need to set the accountcode. In my dialplan, I am bridging to a user once it hits my default context, which is where I decided to set the account code. Below you will see the accountcode set before the call is bridged to the 1000 user.

     <action application="set" data="accountcode=test_in"/>
     <action application="bridge" data="USER/1000@default"/>

You will need to reload FreeSWITCH after making these changes. If you are on Debian 8, you can run systemctl restart freeswitch. Once FreeSWITCH is reloaded you should see a seperate CDR file at /var/log/freeswitch/cdr-csv/test_in.csv, which will only contain calls flagged with the accountcode test_in.

In this article we will go over how to compile the asterisk 14 beta. There are a few new features to play with in this new release.

I recommend bringing your system up to date before trying out the new version of asterisk.

apt update && apt upgrade

Now we need to make sure we have all the dependencies for asterisk and the capability of compiling new software.

apt install build-essential openssl libxml2-dev libncurses5-dev uuid-dev sqlite3 libsqlite3-dev pkg-config libjansson-dev

I like to put all my source code I plan on compiling in /usr/local/src, so we will start out by downloading the source and un taring.

cd /usr/local/src/
wget http://downloads.asterisk.org/pub/telephony/asterisk/asterisk-14.0.0-beta2.tar.gz
tar -xzf asterisk-14.0.0-beta2.tar.gz
cd asterisk-14.0.0-beta2/

Now we can compile asterisk.

./configure
make
make install 
make samples

Once asterisk is finished you should have an asterisk binary in /usr/sbin/asterisk

root@asterisk-14-deb:/etc/asterisk# whereis asterisk
asterisk: /usr/sbin/asterisk /usr/lib/asterisk /etc/asterisk /usr/include/asterisk /usr/include/asterisk.h /usr/share/man/man8/asterisk.8

The configuration files are stored in /etc/asterisk/

Let’s start by adding some users to register as in /etc/asterisk/sip.conf. Add the following at the end of the file.

[100]
type=friend
username=100
secret=somenewsecret
context=test
host=dynamic
qualify=yes

[101]
type=friend
username=101
secret=somenewsecret
context=test
host=dynamic
qualify=yes

• You should replace the secret with some other long hash.

In /etc/asterisk/extensions.conf we need to add a dialplan to route calls to these new users.

Add the following at the end of /etc/asterisk/extensions.conf

[test]
exten => 100,1,Dial(SIP/100)
exten => 101,1,Dial(SIP/101)

• This is creating a new context, test and allowing either phone to dial each other.

Setup systemd for Asterisk.

We will first want to create a user to run asterisk as. It is generally a good idea to not run processes as root unless you need to.

useradd -r -s /bin/false asterisk

We will also need to give the asterisk user permissions over the needed directories.

chown -R asterisk /etc/asterisk/
chown -R asterisk /var/lib/asterisk/
chown -R asterisk /var/run/asterisk/
chown asterisk /usr/sbin/asterisk
chown -R asterisk /var/log/asterisk/

Create a new file at /etc/systemd/system/asterisk.service

vim /etc/systemd/system/asterisk.service

and add the following.

[Unit]
Description=Asterisk PBX And Telephony Daemon
After=network.target

[Service]
User=asterisk
Group=asterisk
Environment=HOME=/var/lib/asterisk
WorkingDirectory=/var/lib/asterisk
ExecStart=/usr/sbin/asterisk -f -C /etc/asterisk/asterisk.conf
ExecStop=/usr/sbin/asterisk -rx 'core stop now'
ExecReload=/usr/sbin/asterisk -rx 'core reload'

[Install]
WantedBy=multi-user.target

We should now be able to control asterisk via systemd. To start asterisk issue the following.

systemctl start asterisk

and to make asterisk start when the server boots

systemctl enable asterisk

You should now be able to register your phones to extension 100 and 101 on your new Asterisk 14 server. Once registered, you can connect to the asterisk console with asterisk -rv.

asterisk -rv
asterisk-14-deb*CLI> sip show peers
Name/username             Host                                    Dyn Forcerport Comedia    ACL Port     Status      Description                      
100/100                   192.168.1.10                            D  Auto (Yes) No             5060     OK (45 ms)                                   
101/101                   (Unspecified)                            D  Auto (No)  No             0        UNKNOWN                                      
2 sip peers [Monitored: 1 online, 1 offline Unmonitored: 0 online, 0 offline]
asterisk-14-deb*CLI>