Blog

Assumptions:

• Using chan_sip
• Using Chrome as your WebRTC client
• Asterisk 11.x
• Using FreePBX 12.0.x
• CentOS 6.x

Download sipML 5

sipML is the WebRTC Client that we are going to use. We need to download the repository

yum install git
cd /var/www/html/
git clone https://github.com/DoubangoTelecom/sipml5.git
chown -R asterisk:asterisk sipml5/

Enable SSL on Built-in HTTP Server of Asterisk

vim /etc/asterisk/http_custom.conf

tlsenable=yes
tlsbindaddr=0.0.0.0:8089
tlscertfile=/etc/pki/tls/certs/localhost.crt
tlsprivatekey=/etc/pki/tls/private/localhost.key

We also need to give asterisk permissions to read the tls certs

chown asterisk:asterisk /etc/pki/tls/certs/localhost.crt
chown asterisk:asterisk /etc/pki/tls/private/localhost.key

Enable Extension for Secure Web Sockets (WSS)

In older version of freepbx, they do not support wss transports, so this will need to be manually configured in /etc/asterisk/sip_custom.conf replacing SOME_EXTENSION and SOMESECRET. The important line is the transport=wss,udp,tcp,tls, which will have wss as the first entry. dtlscertfile and dtlsprivatekey will need to be pointed at the same key cert and key setup in http_custom.conf

[SOME_EXTENSION]
deny=0.0.0.0/0.0.0.0
secret=1234pccw
dtmfmode=rfc2833
canreinvite=no
context=from-internal
host=dynamic
trustrpid=yes
mediaencryption=yes
sendrpid=pai
type=friend
nat=force_rport,comedia
port=5060
qualify=yes
qualifyfreq=60
transport=wss,udp,tcp,tls
avpf=yes
force_avp=no
icesupport=yes
encryption=yes
callgroup=
pickupgroup=
dial=SIP/1103
permit=0.0.0.0/0.0.0.0
callerid=SOME_EXTENSION
callcounter=yes
faxdetect=no
cc_monitor_policy=generic
dtlsenable=yes
dltsverify=no
dtlscertfile=/etc/pki/tls/certs/localhost.crt
dtlsprivatekey=/etc/pki/tls/private/localhost.key

Configure sipML5 expert mode

Browse to https://<server-name>/sipml5. Make sure you include the https and click on the demo button. You should now be at a registration screen. Enter in the extension you would like to register as in the display name and private identity. The public identity will follow the following format:

sip:<Extensions>@<ip-address>

The password will be the secret set for your extensions and the realm will be the ip address or domain name of your server.

We also need to configure expert mode to set the wss address and stun settings.

Under expert mode, the WebSocket Server URL follows the following syntax:

wss://<ip-of-server>:8089/ws

Then set the ice server to the following google address:

[{stun.l.google.com:19302}]

Finally select Enable RTCWeb Breaker and hit save.

You should now be able to register to your extension. To troubleshoot, you can bring up the console in chrome by right clicking and selecting inspect. Additionally, make sure you have opened the necessary wss and rtp ports in your firewall (8089/tcp, 10000-20000/udp)

Start up two instances of Chrome and test

To make a call between two webrtc phones, you will need to install chromium, an open source version of the Google Chrome browser. You can alternatively use two computers with chrome installed. You can add a second extension in /etc/asterisk/sip_custom.conf following the same syntax as the previous extension. After an asterisk restart, you should be able to register to the new extension using the same methods and place a call between the two browsers.

We recently worked with a customer that had a requirement that their application needed to connect via Site-to-Site VPN to there clients application.  They had a few choices, but they decided to move there application to Amazon Web Services  (AWS) and connect to there clients datacenter from there.  Therefore we setup a Virtual Private Cloud (VPC) within Amazon and started down the path of setting up a Site-to-Site Virtual Private Network (VPN) connection.

There are multiple ways of implementing a VPN within Amazon as discussed here.  In most cases, it’s going to come down to using a AWS Hardware VPN or a Software VPN.  The AWS Hardware VPN can be configured within a couple clicks and it gives you the option to generate the configuration for multiple well known firewalls, which  you can use to configure your firewall or you can provide to your firewall administrator.  The Software VPN consists of running an EC2 instance that has software that implements VPN functionality.

The main factor in deciding AWS Hardware VPN versus Software VPN should be based on who’s initiating the traffic.  In our case, the customers application needed to initiate the request.  This means that we had to leverage the Software VPN approach because the AWS Hardware VPN can not initiate traffic.  It can only accept request.  So, it’s great for a company that wants to migrate systems from there datacenter to Amazon and then have there user access the systems. Hence, their users are the initiators of the traffic.

The installation and setup of a Software VPN isn’t really that difficult, but you have to have some basic understanding of how AWS networking works.  There are a few Software VPN implementations, but we selected OpenSWAN.  There’s a few good articles that we used.

• Creating a dynamic site-to-site VPN with OpenSwan on Ubuntu 10.04 on EC2

• Connecting Cisco ASA to VPC EC2 Instance (IPSec)

One of the main gotcha’s in setting up OpenSwan are to ensure that the Access Control List (ACL) defined by the far end (the Router that you are establishing the VPN with) matches the Right side configuration parameter within the setup.  Once you read thru the above articles you will know what I mean.

The average time to setup an AWS Hardware VPN is 5 hours.  This includes configuration, testing and turn-up with the far end.

The average time to setup a Software VPN is 10-20 hours.  It really depends on the complexity of the Amazon VPC and how you need traffic to be routed and represented.

About dOpenSource

We provide Amazon Web Services (AWS) consulting with a focus on DevOPs and Infrastructure Migration.
We are proud to be based in Detroit, MI with coverage from 9am-8pm ET.  We have staff on the East and
West Coast.  You can purchase support from us by going to http://dopensource.com/shop

This tutorial will go over how to setup WebRTC on FreeSWITCH using a certificate from letsencrypt. WebRTC is a protocol which allows voip calls to be conducted over a web browser without additional plugins or software. This tutorial will assume you are Debian 8, which is the recommended OS for production FreeSWTICH servers.

FreeSWITCH WebRTC encryption using letsencrypt

We will use letsencrypt to create tls certificates for our FreeSWITCH server and automate the renewal. WebRTC requires a valid tls certificate for security purposes, and letsencrypt is a cheap and easy way to obtain one.

echo "deb http://ftp.debian.org/debian jessie-backports main" >> /etc/sources.list
apt update && apt upgrade
apt install certbot -t jessie-backports
apt install apache2

You should now have the certbot package installed, and apache2 installed with the default configuration of /var/www/html as your root directory.

You will also want to update /etc/hosts and /etc/hostname to reflect the domain name you will be using.

in /etc/hosts append your hostname to the end of the 127.0.1.1 line

127.0.1.1 <somedomain-name>

and in /etc/hostname replace the current name with your hostname.

<somedomain-name>

Next, we will create our certificate. Execute the following command and fill out any necessary fields.

certbot certonly --webroot -w /var/www/html/ -d <somehostname>

You should see the following if it was successful.

 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/pbx.somedomain.com/fullchain.pem. Your cert
   will expire on 2017-04-11. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you lose your account credentials, you can recover through
   e-mails sent to root@pbx.somedomain.com.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Next, we need to setup the auto renewal with cron. These certificates expire every 90 days, and by default the certbot application will renew the certificate if it is within 30 days of expiring. I chose to run the command every week to be safe.

crontab -e
0 0 * * 7 certbot renew

Configuring SSL for HTTP

Since we are using SSL for WebRTC, might as well use the certificate to enable HTTPS on our server as well.

We need to enable the SSL module for Apache2. On Debian, you can issue this command.

sudo a2enmod ssl

Next we need to create a virtual host in /etc/apache2/sites-available/. Create a file named your new domain name in /conf

vim /etc/apache2/sites-available/pbx.somedomain.com.conf

Then include the following configuration. Note, you will need to point the SSL certificates to the correct directory depending on your domain. You will also need to change the ServerName parameter to whatever your domain name is.

<VirtualHost *:443>
        ServerName somedomain-name
        DocumentRoot /var/www/html
        SSLEngine on
        SSLProtocol all -SSLv3 -SSLv3
        SSLCompression off
        SSLHonorCipherOrder On
        SSLCipherSuite EECDH+AESGCM:EECDH+AES:EDH+AES

        SSLCertificateFile /etc/letsencrypt/live/pbx.somedomain.com/cert.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/pbx.somedomain.com/privkey.pem
        SSLCertificateChainFile /etc/letsencrypt/live/pbx.somedomain.com/chain.pem

        ServerAdmin some-email@address.com

        ErrorLog ${APACHE_LOG_DIR}/error-devel.log
        CustomLog ${APACHE_LOG_DIR}/access-devel.log combined
        DirectoryIndex index.html
</VirtualHost>

Enable the new virtual host with a2ensite pbx.somedomain.com.conf. This will create a symlink from sites-available to sites-enabled. Then Restart Apache2 to enable these changes with systemctl restart apache2.

Installing FreeSWITCH

I chose to use the repository maintained by FreeSWITCH to simplify updates, but you can also compile from source if you wish. If you compiled from source, the FreeSWTICH configuration will be in /usr/local/freeswitch/conf/ instead of inside /etc/.

Use the following commands to setup the FreeSWITCH official repo and install the FreeSWITCH packages on Debian 8.

wget -O - https://files.freeswitch.org/repo/deb/debian/freeswitch_archive_g0.pub | apt-key add -
echo "deb http://files.freeswitch.org/repo/deb/freeswitch-1.6/ jessie main" > /etc/apt/sources.list.d/freeswitch.list
apt-get update && apt-get install -y freeswitch-meta-all git

We are going to want the source files later to copy over the verto js demo.

cd /usr/local/src/
git clone https://freeswitch.org/stash/scm/fs/freeswitch.git -bv1.6 freeswitch

Installing the java script dependencies for the verto demo.

This will install npm and other dependencies for the verto demo to run.

apt update
apt install npm nodejs-legacy
npm install -g grunt grunt-cli bower
npm install
bower --allow-root install
grunt build

Configuring FreeSWTICH

We are going to use the letsencrypt tls certificate we installed earlier for WebRTC. A combined key needs to be created from fullchain.pem and privkey.pem.

cat /etc/letsencrypt/live/somedomain.com/fullchain.pem /etc/letsencrypt/live/somedomain.com/privkey.pem > /etc/freeswitch/tls/wss.pem

Some changes will need to be made to the /etc/freeswitch/autoload_configs/verto.conf.xml file. I removed the ipv6 profile for simplicity.

<configuration name="verto.conf" description="HTML5 Verto Endpoint">

  <settings>
    <param name="debug" value="10"/>
    <!-- seconds to wait before hanging up a disconnected channel -->
    <!-- <param name="detach-timeout-sec" value="120"/> -->
    <!-- enable broadcasting all FreeSWITCH events in Verto -->
    <!-- <param name="enable-fs-events" value="false"/> -->
    <!-- enable broadcasting FreeSWITCH presence events in Verto -->
    <!-- <param name="enable-presence" value="true"/> -->
  </settings>

  <profiles>
    <profile name="somedomain">
      <param name="bind-local" value="0.0.0.0:8081"/>
      <param name="bind-local" value="0.0.0.0:8082" secure="true"/>
      <param name="force-register-domain" value="$${domain}"/>
      <param name="secure-combined" value="/etc/freeswitch/tls/wss.pem"/>
      <param name="secure-chain" value="/etc/freeswitch/tls/wss.pem"/>
      <param name="userauth" value="true"/>
      <param name="context" value="public"/>
      <param name="dialplan" value="XML"/>
      <!-- setting this to true will allow anyone to register even with no account so use with care -->
      <param name="blind-reg" value="false"/>
      <param name="mcast-ip" value="224.1.1.1"/>
      <param name="mcast-port" value="1337"/>
      <param name="rtp-ip" value="$${local_ip_v4}"/>
      <!--  <param name="ext-rtp-ip" value=""/> -->
      <param name="local-network" value="localnet.auto"/>
      <param name="outbound-codec-string" value="opus,vp8"/>
      <param name="inbound-codec-string" value="opus,vp8"/>

      <param name="apply-candidate-acl" value="localnet.auto"/>
      <param name="apply-candidate-acl" value="wan_v4.auto"/>
      <param name="apply-candidate-acl" value="rfc1918.auto"/>
      <param name="apply-candidate-acl" value="any_v4.auto"/>
      <param name="timer-name" value="soft"/>

    </profile>
  </profiles>
</configuration>

Uncomment the following line in /etc/freeswitch/directory/default.xml, or whatever directory you would like to be able to use mod_verto for WebRTC.

      <param name="jsonrpc-allowed-event-channels" value="demo,conference,presence"/>

Now we can copy over the demo verto client to our web server directory

cp -r /usr/local/src/freeswitch/html5/verto/demo/ /var/www/html/
chown -R www-data:www-data /var/www/html/

You should now be able to go to https://<your-domain-name>/demo and you will see the demo WebRTC interface. By default, this client will register to your FreeSWITCH server as extension 1008 with the default password. Because we are registered as a user in the Default directory, our calls will be processed by the default context. You can add some dialplan to test the setup in /etc/freeswitch/dialplan/default/ Such as the following (replace the playback data with the path to a sound file to play back. When dialing 12345, you should hear the audio file played back.)

<include>
  <extension name="verto_test">
    <condition field="destination_number" expression="^(12345)$">
             <action application="answer"/>
             <action application="log" data="INFO ********* VERTO WEBRTC CALL *******" />
             <action application="playback" data="path-to-some-sound-file.wav"/>
             <action application="hangup" />
    </condition>
  </extension>
</include>

This quick article explains how to configure Aastra phones to register with FreePBX 13.x as an extension.

In order to manually provision phones you need the following basic information:

• Server IP
• Server Port (the default is 5060)
• Extension
• Secret (the password for the phone)

In our example, lets assume the following internal network info for the FreePBX server:

• Server IP: 172.16.16.5 |
• Server Port: 5060

You now need the Extension and Secret. This can be found by logging into the FreePBX GUI and going to extensions section of the interface. You will see a list of extensions. Below we only show one extension, which is 10000

You can click on that extension and locate the Secret, which is outlined in red.

Now that you have this information you can login to the Aastra phone and click on “Global SIP”. You will enter the Extension and Secret into the fields that are outlined at the top and the Server IP and Server Port will replace the data outlined at the bottom.

Note – you will need to reboot your phone after configuring the phone.

In our previous article we setup google-authenticator for authenticating openssh. Now, we need a way for users to be able to login once before setting up google-authenticator. Here is a script for checking if a user has not logged in and ran google-authentication yet, runs google-authenticator, then prevents that user from logging in again without either google-authentication or an ssh public key. To setup this script do the following:

1. Edit /etc/pam.d/sshd and make the following changes after #%PAM-1.0

   auth    required        pam_sss.so try_first_pass
   auth    sufficient  pam_listfile.so item=user sense=allow file=/google-auth/authusers
   auth    sufficient      pam_google_authenticator.so
   auth       required     pam_sepermit.so
   #auth       include      password-auth
   

   We are using ldap, so our users authenticate with pam_sss.so. If you are using regular unix passwords, you should change this first required auth line to pam_unix.so instead. We setup a pam_listfile, which has a list of all the users we want to allow access to login. Our script, called from each of these users .bash_profile file, will remove the user from this file after their first login without google-authenticator setup. Notice, we commented out the password-auth include line. We do not want to call the password-auth file because we only want users to be able to log in if they are in if they can first authenticate with sss/ldad and are in our google-auth list. Otherwise, they can login with google_authenticator, or from an sshd key, which will happen before the authentication is sent to pam.

2. Create the /google-auth/authusers file, which pam will check for which users to allow without requiring further authentication. I used /home/ to see which users are actual human users who will need to login. This file just needs to be a list with one username per line. This file needs to be in a new directory, because we need to give all of these users permission to write to it so their usernames can be removed via our script once they create a google authentication file.

   a. create a group and add all of the necesarry users to this group.

   groupadd google-auth
   gpasswd google-auth -M bob,joe,smith
   

   b. Now create the authusers file and set the permissions to be owned by google-auth, then allowed write access by users in that group.

   mkdir /google-auth/
   touch /google-auth/authusers
   chgrp google-auth /google-auth/authusers
   chmod ug=rwx,o= /google-auth/authusers
   

   c. Now add the necesarry users to /google-auth/authusers

   bob
   joe
   smith
   

3. Install the script to be ran. Just add these couple lines to an .sh file in /usr/local/bin, or you can copy it over if you did a git pull. I created my script with vim /usr/local/bin/google-auth-check.sh

   #!/bin/bash
   
   if [ ! -f ~/.google_authenticator ]; then
       google-authenticator
       if [ -f ~/.google_authenticator ]; then
           sed -i "/^${USER}$/d" /google-auth/authusers
       fi
   fi
   

   And then give it execute permissions.

   chmod +x /usr/local/bin/google-auth-check.sh
   

4. Next, we need to add this script to each of the ~/.bash\_profile files in each users home directory. I will come up with a find and echo command, but for now, just add this at the end of these files in each users directory you want this to work for.

   sh /usr/local/bin/google-auth-check.sh
   

You can run this bash command to find every .bash_profile in /home/ and append the line above. Be careful before running this command. It works on my system, but you can never be too careful with recursive commands such as this.

find /home/ -name ".bash_profile" -print0 | xargs --verbose -0 -I{} sh -c "echo 'sh /usr/local/bin/google-auth-check.sh' >> {}"

This should be all you need to do. Now when a user logs in for the first time, the google-authenticator application will automatically run and create a .google\_authenticator file. The user will then be able to add the key into their phone app and have multifactor authentication to log into their account. Their name will get removed from the /google-auth/authusers file, so they will no longer be given access without a multi factor auth, or ssh key.

Sources:

https://www.digitalocean.com/community/tutorials/how-to-use-pam-to-configure-authentication-on-an-ubuntu-12-04-vps

http://www.linux-pam.org/Linux-PAM-html/sag-pam_exec.html

https://wiki.archlinux.org/index.php/PAM

Install google-authenticator

In this tutorial we will go over how to setup two factor authentication for SSH using google-authenticator on CentOS 6.

First thing we need to do is install the google-authenticator package using yum.

yum install google-authenticator

Next, run google-authenticator to genereate a key.

google-authenticator
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/nschoonover@ldap.dopensource.com%3Fsecret%3D3FO6WYXPZUMTEWUB
Your new secret key is: 3FO6WYXPZUMTEWUB
Your verification code is 799347
Your emergency scratch codes are:
  84194653
  29899626
  13794318
  84024610
  76941184

Do you want me to update your "~/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n)
By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n)

With this information, you will need to install the google authenticator application on your phone. Copy and paste the URL at the beginning of the output from google-authenticator, which should show you a QR code. Scan this code using the Google Authenticator application on your phone. You will now see a 6 digit number in your Google Authenticator app. This number will change every certain ammount of minutes. This number will be your second authentication method when logging into your server. If you ever lose your secret key and need to setup another phone, you can always run the google-authenticator command again to regenerate.

Next, we need to setup pam to use google_authenticator for authenticating users.

Open up the pam ssh configuration file.

sudo vim /etc/pam.d/sshd

Then add the following lines at the bottom.

## Google autehnticator for ssh
auth required pam_google_authenticator.so

Next, we need to add the ChallengeResponseAuthentication yes option to the /etc/ssh/sshd\_config file.

# Change to no to disable s/key passwords
ChallengeResponseAuthentication yes
#ChallengeResponseAuthentication no

Now restart sshd for the new changes to take affect.

sudo service sshd restart

Troubleshooting

The first time I tried to set this up, I was unable to login using my authentication code. I would be prompted for a code when tried to initiate an ssh connection, but I would see this message in the logs and would not be let in.

Dec 28 17:49:43 web2p sshd(pam_google_authenticator)[19784]: Invalid verification code

It turns out the problem was caused by the time on my testing server not being correct. Make sure you have ntp running and configured correctly.

To install and set ntp to auto start:

yum install ntp
chkconfig ntpd on
service ntpd start

Now run this command to manually force ntp to sync with pool.ntp.org

ntpdate pool.ntp.org

You can check the date on your server with the date command to make sure ntp is working. After syncing my time, I was able to log into my ssh server using google authenticator without a problem.

Public Key Authentication

We decided that we wanted to allow users to login using either Google Authenticator, or a regular ssh Pulic Key. We generally use public keys for ssh authentication, but occasionally we will want to login to one of our servers from a machine we will only use once. Google Authenticator is a great compromise for this as it allows us to authenticate without an ssh key, but still maintains our security.

To ensure sshd will allow authentication from either the Google Authenticator module in pam, or an ssh key, uncommend the following lines in /etc/ssh/sshd\_config

PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication yes
UsePAM yes

The changes we made to /etc/pam.d/sshd should only effect the password authentication portion of the ssh login process. If PubkeyAuthentication yes is set, ssh should approve your ssh key authentication without reaching out to pam and requiring two factor authentication.

In our next article, we will go over how to allow users to login once before google-authentication is enforced.

Installing Jungledisk

If you are installing the Jungle Disk agent to a server via command line, you will need to use wget or another cli http client to download the installer. We are installing Jungle Disk to a CentOS 6 server, but there are also packages for Debian based systems, and a tar file with the source if you would like to compile.

Download the installer, and run the rpm to setup and start the backup agent.

wget https://downloads.jungledisk.com/jungledisk/junglediskserver-3180-0.x86_64.rpm
[root@ldap ~]# sudo rpm -i junglediskserver-3180-0.x86_64.rpm
Stopping junglediskserver: [FAILED]
Starting junglediskserver:
[root@ldap ~]# ps -ef | grep jungle
root 22936 1 0 15:58 ? 00:00:00 /usr/local/bin/junglediskserver

This is the rpm package as of the time this article is written, but as there are new releases of the agent software, this will change. Make sure you download the latest version from your Jungle Disk account.

Setup the license

[root@ldap jungledisk]# cd /etc/jungledisk/
[root@ldap jungledisk]# cp junglediskserver-license-EXAMPLE.xml junglediskserver-license.xml

Now you will need to add your license to the new junglediskserver-license.xml file. You can find this license in your licenses section of the control panel. https://secure.jungledisk.com/secure/account/subscriptions.aspx

vim /etc/jungledisk/junglediskserver.xml

Enter in your license string in the licenseKey section.

<licenseKey>some_license_string</licenseKey>

Setting up the Management Client and configuring a backup.

You will need to install the Management Client to a Windows computer. Download the client here: https://secure.jungledisk.com/downloads/business/server/windows/

Once it is downloaded and installed, you will need to log in. Click Next to start the configuration.

If you already have setup Jungle Disk to backup another server, you will need to chose your new server from the server list. The client will be the hostname on the server. Now, create a new online disk.

Choose between Standard Security, or High, which will encrypt your data whil it is stored on the Jungle Disk infrastructure. (We chose standard)

Select you backup schedule. We stayed with the default daily schedule.

Next, select what files to backup on your linux server.

Hit ok, and next. You should be done! You can now try running a manual backup to get your files backed up right away, or wait for the scheduled backup to run.

In this article we will be going over the basics of setting up a multi-tennant environment in FreeSWITCH

Directory

We will need to create a new directory for our second tennant. The default configuration is a good place to start from, so copy over the default.xml file and the default directory to the domain name of your new company.

cp default.xml dopensource.com.xml
cp -r default dopensource.com

Now, change the domain name, group name, and include directory with vim dopensource.com.xml

  <domain name="dopensource.com">

...

   <group name="dopensource.com">

...

   <X-PRE-PROCESS cmd="include" data="dopensoruce.com/*.xml"/>

By default, each user is set to use the default context. We will need to change this if we want each company to have a different context. You can either change this in each users include file, or remove this line from those files and set it in the global directory xml file.

To remove all of the user_context lines from the user’s include files, run this sed command.

sed -i '/<variable name="user_context" value="default"\/>/d' /etc/freeswitch/directory/dopensource.com/*

Then add the user_context setting in the directory xml file with vim dopensource.com.xml

    <variables>
      <variable name="record_stereo" value="true"/>
      <variable name="default_gateway" value="$${default_provider}"/>
      <variable name="default_areacode" value="$${default_areacode}"/>
      <variable name="transfer_fallback_extension" value="operator"/>
      <variable name="user_context" value="dopensource.com"/>
    </variables>

Now when a user registered to our system via dopensource.com, their calls will be processed in the context dopensource.com. We will need to create this context for this to work though.

Dialplan

Now that we have setup our Directory, it is time to create a seperate context for our new company. We will start from the defaul.xml file as a base.

cd /etc/freeswitch/dialplan/
cp default.xml dopensource.com.xml
cp -r default dopensource.com

Now, we will need to change the name in dopensource.com to reflect our new context with vim dopensource.com.xml

  <context name="dopensource.com">

We also want to adjust the include directory.

    <X-PRE-PROCESS cmd="include" data="dopensource.com/*.xml"/>

FreeSWITCH should now be configured to host a second company and seperate it into a new context. You can repeat this process for as many domain names as you like.

You will need to change your dns settings to make sure the domain name you use in your directory is actually routable to your FreeSWITCH server. Also, don’t forget to register to this domain name when setting up your phone.

Issue: Needed to get the IP address of a VM running on my XenServer.

Resolution: Execute this command

xe vm-list params=name-label,networks | grep -v "^$"

The command yielded me a list of the VM’s with the ip address.

name-label ( RW) : DevStackOSDomU
networks (MRO): 2/ip: 172.24.4.10; 1/ip: 10.255.255.255; 0/ip: 10.10.10.187
name-label ( RW) : Control domain on host: warrier.dopensource.com
networks (MRO): <not in database>

Note, I only had 1 VM on this Xenserver, but I needed the IP address:

In this tutorial we will assume you have already setup SIPp on another server to stress test FreeSWITCH. We covered this in our previous article

We will need to install valgrind on the same machine running FreeSWITCH. I like using Debian for FreeSWITCH, which conveniently already has valgrind in the repository.

apt install valgrind

Now that valgrind is installed, we will need to make sure FreeSWITCH is not already running.

systemctl stop freeswitch
ps -ef | grep freeswitch

Once we stop FreeSWITCH, we want to start up FreeSWITCH using valgrind. Below is the command to do so. This will log all memory issues found by Valgrind to vg.log in the directory you run the command from.

/usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Running the FreeSWITCH from the stable repository will not result in many errors. Below is my the end of my vg.log while running FreeSWITCH 1.6.12-20-b91a0a6~64bit

==20986== LEAK SUMMARY:
==20986==    definitely lost: 0 bytes in 0 blocks
==20986==    indirectly lost: 0 bytes in 0 blocks
==20986==      possibly lost: 16,384 bytes in 2 blocks
==20986==    still reachable: 192 bytes in 1 blocks
==20986==         suppressed: 0 bytes in 0 blocks
==20986==
==20986== For counts of detected and suppressed errors, rerun with: -v
==20986== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

There are no new errors after the leak summary in this log.

When the FreeSWITCH is having memory management problems, valgrind will continue to log errors after the LEAK SUMMARY. Below is an example of a memory leak when accessing the libcrypto.so library for WebRTC.

143768 ==32354== Use of uninitialised value of size 8
143769 ==32354==    at 0x67C8B80: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
143770 ==32354==    by 0xF684D2C1E662C6C4: ???
143771 ==32354==    by 0xE16640B5E123D1DD: ???
143772 ==32354==    by 0xD68DF16DAF4386F1: ???
143773 ==32354==    by 0xB2B603F64080C703: ???
143774 ==32354==    by 0x398A1FCC7DC2FF3B: ???
143775 ==32354==    by 0xBE60B2707D98773F: ???
143776 ==32354==    by 0x375392F355F20B45: ???
143777 ==32354==    by 0x124A183B188AA4EA: ???
143778 ==32354==    by 0x5BFE669681E01F8: ???
143779 ==32354==    by 0x6A4C9E9A4AA5A4C0: ???
143780 ==32354==    by 0x29B07C509D28C49F: ???
143781 ==32354==

In this case, valgrind was logging this message to the log many times per second. You can view the memory mismanagement manifesting itself in a memory leak by viewing the allocated memory with ps. We will want to monitor the change in RSS, or Resident Set Size, which represents the ammount of bytes are allocated for a process in RAM.

ps -aux | grep '[v]algrind.bin'

On Debian 8, ps -aux will list the information in the following order:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

The output of your ps -aux command should look something like this

root     20987 36.7 13.1 631032 271356 ?       S<sl 13:33  30:47 /usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Knowing the current total allocated memory for FreeSWITCH is cool, but not very useful for seeing the memory leak in action. If FreeSWITCH has a memory leak, it will fail to release memory when it is done with it, causing the allocated space to continue to grow. A useful way to observe this is by monitoring the RSS of the FreeSWITCH process over time, then running SIPp to create an artificial load on FreeSWTICH I created a simple bash script to log FreeSWITCH’s RSS, the current time, and the number of sip channels to a file.

#!/bin/sh


while true; do

INT=60
FILE=/root/mem.log
DATE=$(date +"%F-%R")
MEM=$(ps -aux | grep '[v]algrind.bin' | awk -F ' ' '{print $6}')
CHANNELS=$(/usr/local/freeswitch/bin/fs_cli -x "show channels" | grep total | awk -F ' ' '{print $1}')

echo ${DATE},${MEM},${CHANNELS} >> ${FILE}
sleep ${INT}s
done

This script will log the time, RSS, and SIP channels to /root/mem.log. You may need to adjust the CHANNELS variable to point to wherever the fs_cli binary is stored.

Save the script to a file, such as memtest.sh, then run it with

./memtest.sh &

You can confirm it is running in the background with jobs.

Now, with FreeSWITCH running under valgrind, we will want to start up the SIPp stress test. On the SIPp server, run the following.

sipp -sf uac.xml -s 9999999  <freeswitch-ip>:5080 -trace_msg -l 15 -d 60000

This test is best done over a long period of time. I would let SIPp generate traffic on the server for a couple hours while our monitor script is running. After the system has ran under load for some time, the mem.log should look soemthing like this.

2016-10-24-15:29,271212,15
2016-10-24-15:29,270800,15
2016-10-24-15:30,270672,15
2016-10-24-15:31,270828,15
2016-10-24-15:32,270708,15
2016-10-24-15:33,270604,15
2016-10-24-15:34,270812,15
2016-10-24-15:35,270628,15
2016-10-24-15:36,270816,15

If FreeSWITCH has been running for multiple hours and the RSS has seemed to plateau, like above, FreeSWITCH is likely not showing any critical memory mismanagements. Here is an example of the RSS continuing to grow due to a memory leak.

2016-08-11-04:09,379192,60
2016-08-11-04:10,381584,60
2016-08-11-04:11,383104,60
2016-08-11-04:12,385096,60
2016-08-11-04:13,385356,60
2016-08-11-04:14,387580,60
2016-08-11-04:16,387432,60
2016-08-11-04:17,389316,60
2016-08-11-04:18,391144,60
2016-08-11-04:19,393076,60
2016-08-11-04:20,394484,61
2016-08-11-04:21,395776,60
2016-08-11-04:22,397072,60
2016-08-11-04:23,398140,59
2016-08-11-04:24,400112,60
2016-08-11-04:25,401868,60
2016-08-11-04:26,403348,60

FreeSWITCH had been running for many hours at this point, and the RSS had never stopped growing. This is the same system which was logging the libcrypto error in the valgrind log.

When you notice a memory issue with your version of FreeSWTICH, check the FreeSWTICH JIRA for known bugs. There may already be a patch out for the issue you are experiencing.