Installing OpenLDAP 2.4 and Configuring Apache DS – Part 3

This web series is brought to you by: dOpenSource and Flyball Labs

This post is PART 3 of a series that details how to install Apache Directory Studio and OpenLDAP server and connect the two seamlessly. In the last post we went adding users, groups and organizations to your OpenLDAP server. We also went over modifying existing users and groups in your LDAP tree as well and how to query your LDAP database.

We have been using “.ldif” files and a terminal to manage our OpenLDAP tree thus far, but this method (although necessary to start) doesn’t provide a very nice “User Experience” or UX as we call it on the dev side of the house. So lets change that and put a nice Graphical User Interface (GUI) on top of our OpenLDAP system to manage our LDAP tree entries.

A quick note: there is a ton of information on querying your OpenLDAP tree structure that we have not yet touched on. Learning these methods will be vital to managing your users quickly and efficiently. Just to give you a glimpse forward, there are many advanced methods for searching and filtering LDAP queries such as: specifying search base, binding to a DN, specifying scope or search, advanced regular expression matching and filter output fileds, to name a few. We will go over ways to effectively get your data from your OpenLDAP server in a later post.

If you haven’t read PART 1 you can find it here: Installing OpenLDAP

If you haven’t read PART 2 you can find it here: Configuring OpenLDAP

We will now walthrough the installation of Apache Direcotry Studio and how to connect it to your existing OpenLDAP server. The examples below assume that the domain is dopensource.com but, you MUST REPLACE this with your own domain. Lets get started!

Requirements

This guide assumes you have either a Debian-based or cent-os / red-hat based linux distro.
This guide also assumes that you have root access on the server to install the software.
This guide assumes you have followed the steps in PART 1 and PART 2 of the series and have an OpenLDAP server configured with some test users and groups added to your tree.

Installing Apache Directory Studio


Switch to root user and enter your root users pw

Either using sudo:

sudo -i

Or if you prefer using su:

su

Install Oracle Java

Prior to installing Apache Directory Studio (Apache DS) we need to install Oracle Java version 8 or higher. The easiest way to do this is through a well a maintained repository / ppa.

Unistall all old versions of openjdk and oracle jdk:

On a debian based system the cmd would be:

apt-get -y remove --purge $(apt list | grep -ioP '.*open(jre|jdk).*' 'oracle-java.*' '^java([0-9]{1}|-{1}).*' '^ibm-java.*')

On a centos based system the cmd would be:

yum -y erase $(yum list | grep -ioP '(.*open(jre|jdk).*|oracle-java.*|^java([0-9]{1}|-{1}).*|^ibm-java.*)'
Remove old Java environment variables

We also need to cleanup any old java environment variables that may cause issues when we install the new version.

Check your environment for any java variables using this cmd:

env | grep "java"

If there are no matches then you can move on to the next section. Otherwise you need to manually remove those variables from your environment.

Check each one of the following files to see if there are any vars we need to remove. If you get a match remove the line that contains that variable entry (most likely an export statement):

cat /etc/environment 2> /dev/null | grep "java"
cat /etc/profile 2> /dev/null | grep "java"
cat ~/.bash_profile 2> /dev/null | grep "java"
cat ~/.bash_login 2> /dev/null | grep "java"
cat ~/.profile 2> /dev/null | grep "java"
cat ~/.bashrc 2> /dev/null | grep "java"

To remove the line just use your favorite text editor and source that file. For example if .bashrc what the culprit:

vim ~/.bashrc
... remove stuff ...
source ~/.bashrc

Now you are ready to install a recent release of Oracle Java.

Download and install Oracle JDK

For debian based systems there is a group of developers that package the latest versions of Oracle Java and provide it as a ppa repository. This is very convenient for updating Java versions. For centos based systems you will have to download the tar file of the latest release from Oracle’s website and install it manually.

For debian based systems we will add the ppa, update our packages and install the latest release of Oracle JDK.

cat << EOF > /etc/apt/sources.list.d/oracle-java.list
deb http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main
deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu trusty main
EOF

apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys EEA14886
apt-get -y update && apt-get -y install oracle-java8-installer
apt-get install oracle-java8-set-default

For centos based systems go to Oracle’s website here click on accept license agreement and copy the link of the linux .rpm file corresponding to your OS architecture. If you don’t know what architecture your machine is run the following command to find out:

uname -m

Now dowload and install the rpm using the following cmds:

JAVA_URL='the link that you copied'
JAVA_RPM=${JAVA_URL##*/}

cd /usr/local/
wget -c --no-cookies --no-check-certificate \
--header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
"$JAVA_URL"

rpm -ivh "$JAVA_RPM"
rm -f "$JAVA_RPM"

unset JAVA_URL && 
unset JAVA_RPM

At this point you should have Oracle Java JDK installed on your machine. Verify with the following cmds:

java -version
javac -version

You should see output similar to the following:

[root@box ~]# java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

[root@box ~]# javac -version
javac 1.8.0_131

If you get an error here you may need to uninstall any dependencies on your system that are linked to the old java version and redo the process. One example may be libreoffice (often installed by default and depends on default java installation) which we could remove with following:

Debian based systems:

apt-get -y remove --purge '.*libreoffice.*'

Centos based systems:

yum -y remove '.*libreoffice.*'

For those with errors; after removing any packages that depended on the old versions of java, go back through the installation steps outlines above and you should be able to print out your java version.

Set Java environment vars for new version

First we will set our environment variables using a script we will make here: /etc/profile.d/jdk.sh

cat << 'EOF' > /etc/profile.d/jdk.sh
#!/bin/sh
#
# Dynamic loading of java environment variables
# We also set class path to include the following:
# if already set the current path is included,
# the current directory, the jdk and jre lib dirs,
# and the shared java library dir.
#

export JAVA_HOME=$(readlink -f /usr/bin/javac 2> /dev/null | sed 's:/bin/javac::' 2> /dev/null)
export JRE_HOME=$(readlink -f /usr/bin/java 2> /dev/null | sed 's:/bin/java::' 2> /dev/null)
export J2SDKDIR=J2REDIR=$JAVA_HOME
export DERBY_HOME=$JAVA_HOME/db

export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin:$DERBY_HOME/bin
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JRE_HOME/lib:/usr/share/java
EOF

After our script is created make it executable, and then source the main profile.

chmod +x /etc/profile.d/jdk.sh
source /etc/profile

You should now see the environment variables we added above when running this cmd:

env | grep -oP '.*(JAVA_HOME|JRE_HOME|J2SDKDIR|J2REDIR|DERBY_HOME|CLASSPATH).*'

Lastly we are going to enable the Java client for our web browsers:

Close all instances of firefox and chrome:

kill all firefox
kill all chrome

Grab the plugin from your Oracle Java install and link it:

JAVA_PLUGIN=$(find ${JRE_HOME}/ -name 'libnpjp2.so')

mkdir -p /usr/lib/firefox-addons/plugins && cd /usr/lib/firefox-addons/plugins
ln -s ${JAVA_PLUGIN}

mkdir -p /opt/google/chrome/plugins && cd /opt/google/chrome/plugins
ln -s ${JAVA_PLUGIN}

unset JAVA_PLUGIN
cd ~

Install the Apache Directory Studio suite

Download either the 32bit or 64bit (depending on you OS) version of Apache DS with the following:

APACHE_DSx64_URL="http://www-eu.apache.org/dist/directory/studio/2.0.0.v20161101-M12/ApacheDirectoryStudio-2.0.0.v20161101-M12-linux.gtk.x86_64.tar.gz"

APACHE_DSx32_URL="http://www-eu.apache.org/dist/directory/studio/2.0.0.v20161101-M12/ApacheDirectoryStudio-2.0.0.v20161101-M12-linux.gtk.x86.tar.gz"

APACHE_DS_URL="the url for your os architecture"
APACHE_DS_TAR=${APACHE_DS_URL##*/}

cd /usr/local/
wget --no-cookies --no-check-certificate "$APACHE_DS_URL"
tar -xzf "$APACHE_DS_TAR" && rm -f "$APACHE_DS_TAR" && cd ApacheDirectoryStudio

Now you can start the gui with the following cmd:

./ApacheDirectoryStudio

If you are configuring this on a remote server (like I was) you will need to set the ssh configs properly to forward X-applications (GUI / display) over to your local machine. If you are configuring the server locally and have a display you can skip this step.

Enable X11 forwarding over ssh on the remote server:

sed -ie '0,/.*X11Forwarding.*/{s/.*X11Forwarding.*/X11Forwarding yes/}; 0,/.*X11UseLocalhost.*/{s/.*X11UseLocalhost.*/X11UseLocalhost yes/}' /etc/ssh/sshd_config

Now disconnect and allow your local machine’s ssh client to receive the forwarded display:

sed -ie '0,/.*ForwardX11.*/{s/.*ForwardX11.*/ForwardX11 yes/}; 0,/.*ForwardX11Trusted.*/{s/.*ForwardX11Trusted.*/ForwardX11Trusted yes/}' /etc/ssh/ssh_config

Then establish a an ssh connection with remote X11 forwarding enabled and run it:

ssh -Y -C -4 -c blowfish-cbc,arcfour 'your remote user@your remote server ip'
cd /usr/local/ApacheDirectoryStudio
./ApacheDirectoryStudio

You should now see the GUI appear. Lets connect to our OpenLDAP server.

Click on the buttons File –> New –> LDAP Browser –> LDAP Connection>

Enter a name for your connection and enter the hostname of the LDAP server (you can find it using the **hostname** cmd)

Make sure to check your connection to the server by pressing the “Check Network Parameter” button before moving on. If you get an error, verify connectivity to the OpenLDAP server and port that your entered and try again.

Enter the LDAP admin’s Bind DN and password. From our example in the first blog post it would look something like this:

Bind DN or user: cn=Manager,dc=dopensource,dc=com

Bind Password: ‘ldap admin password’

Make sure to click the “Check Authentication” button and verifiy your credentials before moving on.

Enter your Base DN, for our example this was: **dc=dopensource,dc=com**

Check the feature box “Fetch Operational attributes while browsing”

You can now click “finish” and your connection to your OpenLDAP server through Apache DS is complete.

To view your DN and to manage the entries we made earlier:

click the LDAP icon on the left and it will bring up the sidbar for LDAP

Alternatively you can navigate to your LDAP tree by clicking window –> show view –> ldap browser

Congratulations you have set up your Apache DS front-end application to access your OpenLDAP server!

For More Information


For more information, or for any questions visit us at:

dOpenSource

Flyball Labs

For Professional services and Ldap configurations see:

dOpenSource OpenLDAP

dOpenSource OpenLDAP Services

If you have any suggestions for topics you would like us to cover in our next blog post please leave them in the comments 🙂

Written By:

DevOpSec
Software Engineer
Flyball Labs

— Keep Calm and Code On —

Installing OpenLDAP-2.4 and Configuring Apache DS – Part 2

This web series is brought to you by: dOpenSource and Flyball Labs

This post is PART 2 of a series that details how to install Apache Directory Studio and OpenLDAP server and connect the two seamlessly. In the last post we went over installing OpenLDAP server and you should now have an operating OpenLDAP server on either a Debian based or Cent-OS based machine.

If you haven’t read PART 1 you can find it here: Installing OpenLDAP

We will now go over adding users, groups and organizations to your nice and shiny OpenLDAP server. These are all LDAP entries that will go in your OpenLDAP tree. The examples below assume that the domain is dopensource.com but, you MUST REPLACE this with your own domain. Lets get started!

Requirements

This guide assumes you have either a Debian-based or cent-os / red-hat based linux distro.
This guide also assumes that you have root access on the server to install the software.
This guide assumes you have followed the steps in PART 1 of the series and have an OpenLDAP server configured and running already.

Adding LDAP entries to your OpenLDAP tree


Switch to root user and enter your root users pw

Either using sudo:

sudo -i

Or if you prefer using su:

su

Adding an Organization Unit (OU)

Lets add an OU to our tree called ‘users’ that will hold all of our companies users.

We will modify our tree the same we did before, using tmp .ldif files and loading into ldap.

cat << EOF > /tmp/users.ldif
dn: ou=Users,dc=dopensource,dc=com
objectClass: organizationalUnit
ou: Users
EOF

Then add it to the tree using the following cmd:

ldapadd -f /tmp/users.ldif -D cn=Manager,dc=dopensource,dc=com -w < ldap admin pw >

And of course we clean up after ourselves:

rm -f /tmp/users.ldif

Lets check out our new and shiny organization in ldap:

ldapsearch -x -L -b dc=dopensource,dc=com

Adding Users to the OU

Next lets add some of our employess to the users OU:

cat << EOF > /tmp/fred.ldif
dn: cn=Fred Flintstone,ou=Users,dc=dopensource,dc=com
cn: Fred Flintstone
sn: Flintstone
objectClass: inetOrgPerson
userPassword: < user password >
uid: fflintstone
EOF

cat << EOF > /tmp/wilma.ldif
dn: cn=Wilma Flintstone,ou=Users,dc=dopensource,dc=com
cn: Wilma Flintstone
sn: Flintstone
objectClass: inetOrgPerson
userPassword: < user password >
uid: wflintstone
EOF

Like before we need to add the entry to our tree:

ldapadd -f fred.ldif -D cn=Manager,dc=dopensource,dc=com -w < ldap admin pw >
ldapadd -f wilma.ldif -D cn=Manager,dc=dopensource,dc=com -w < ldap admin pw >

And then cleanup for a nice and clean system:

rm -f /tmp/fred.ldif
rm -f /tmp/wilma.ldif

Lets make sure we didn’t miss any employees in our long list of users:

ldapsearch -x -L -b ou=Users,dc=dopensource,dc=com

Adding a Group to the OU

We have to seperate our employees because the Engineers don’t play nice with the Salesperson’s:

cat << EOF > /tmp/engineering.ldif
dn: cn=Engineering,ou=Users,dc=dopensource,dc=com
cn: Engineering
objectClass: groupOfNames
member: cn=Fred Flintstone,ou=Users,dc=dopensource,dc=com
EOF

cat << EOF > /tmp/sales.ldif
dn: cn=Sales,ou=Users,dc=dopensource,dc=com
cn: Sales
objectClass: groupOfNames
member: cn=Wilma Flintstone,ou=Users,dc=dopensource,dc=com
EOF

Lets add it to the good ‘ol christmass tree of ldap:

ldapadd -f /tmp/engineering.ldif -D cn=Manager,dc=dopensource,dc=com -w < ldap admin pw >
ldapadd -f /tmp/sales.ldif -D cn=Manager,dc=dopensource,dc=com -w < ldap admin pw >

And we have to cleanup after the holiday party:

rm -f /tmp/engineering.ldif
rm -f /tmp/sales.ldif

Lets check our groups and make sure they mingle well together:

ldapsearch -x -L -b ou=Users,dc=dopensource,dc=com cn=Engineering
ldapsearch -x -L -b ou=Users,dc=dopensource,dc=com cn=Sales

We can narrow it down even further, if we only want to see what groups exist:

ldapsearch -x -LLL -b "ou=Users,dc=dopensource,dc=com" "(&(objectclass=groupOfNames))" *

Adding an Existing User to an Existing Group

So Fred is sick and tired of R&D and wants to sell stuff, lets transfer him over to sales:

cat << EOF > /tmp/addtogroup.ldif
dn: cn=Sales,ou=Users,dc=dopensource,dc=com
changetype: modify
add: member
member: cn=Fred Flintstone,ou=Users,dc=dopensource,dc=com
EOF

Get him on the sales floor by adding into Sales group:

ldapadd -f /tmp/addtogroup.ldif -D cn=Manager,dc=dopensource,dc=com -w < ldap admin pw >

Gotta cleanup his desk before we go visit:

rm -f /tmp/addtogroup.ldif

Lets check out Fred’s new office:

ldapsearch -x -LLL -b "ou=Users,dc=dopensource,dc=com" "(&(cn=Engineering))" member

Looks like he’s right next to Wilma, hopefully they work well together 🙂

For More Information


For more information, the next post in the series or for any questions visit us at:

dOpenSource
Flyball Labs

For Professional services and Ldap configurations see:

dOpenSource OepnLDAP
dOpenSource OpenLDAP Services

Written By:

DevOpSec
Software Engineer
Flyball Labs

Installing OpenLDAP 2.4 and Configuring Apache DS – Part 1

This web series is brought to you by: dOpenSource and Flyball Labs

This post is PART 1 of a series that details how to install Apache Directory Studio and OpenLDAP server and connect the two seamlessly. OpenLDAP is what’s referred to as an “Lightweight Directory Access Protocol” or LDAP for short, and is based on X.500 standards.

OpenLDAP can be used to manage user and groups in an organization and authenticate them on your systems, through certificate validation. APache DS is the front-end GUI we will be using to interface with our OpenLDAP server to manage our users and groups.

A quick note before we begin; in this post we use angle brackets to denote that you either need to input information there, or that there will be information output there. Great, lets get started!

Requirements

This guide assumes you have either a Debian-based or cent-os / red-hat based linux distro.
This guide also assumes that you have root access on the server to install the software.

Installing OpenLDAP Server


Switch to root user and enter your root users pw

Either using sudo:

sudo -i

Or if you prefer using su:

su

Install OpenLDAP dependencies using you package manager

On Debian based systems would be:

apt-get update -q -y
apt-get install -q -y dpkg slapd ldap-utils ldapscripts

On cent-os / rhel systems:

yum -y -q update
yum -y -q install openldap compat-openldap openldap-clients \
        openldap-servers openldap-servers-sql openldap-devel

The debian package may ask you for the admin ldap password now, no worries. We are going to write over this in the next step so put the same password that you will use throughout the tutorial.

Create a password for the admin ldap user

slappasswd
<enter secret pw here>
<re-enter secret pw here>
<you will see hash of pw here>

Copy the hash that is output from this cmd to your clipboard.

Update the ldap conf file with the new password

We are going to move to the dir where our ldap configs are:

For centos / rhel that is here: /etc/openldap/slapd.d/cn\=config
For debian / ubuntu that is here: /etc/ldap/slapd.d/cn\=config

Then we need to add the root pw hash we copied from last step.

cd /etc/*ldap/slapd.d/cn\=config
echo "olcRootPW: {SSHA}<pw hash goes here>" >> olcDatabase\=\{2\}bdb.ldif

Modify the distinguished name or (DN) for short; of the olcSuffix. This can be done using sed:

You should set the suffix to your DNS domain name. This will be appended to the DN in your tree.
For example, for dopensource.com would be:

sed -i "s/olcSuffix:.*/olcSuffix: dc=dopensource,dc=com/" olcDatabase\=\{2\}bdb.ldif
sed -i "s/olcRootDN:.*/olcRootDN: cn=Manager,dc=dopensource,dc=com/" olcDatabase\=\{2\}bdb.ldif

You would replace dopensource && com with your own domain.

Now change the monitor.ldif file to match tholcRootDN we changed earlier in bdb/ldif:

sed -i 's/olcAccess:.*/olcAccess: {0}to *  by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read  by dn.base="cn=Manager,dc=dopensource,dc=com" read  by * none/' olcDatabase={1}monitor.ldif
sed -ie '7d;' olcDatabase={1}monitor.ldif

Restrict users from viewing other users’ password hashes:

echo 'olcAccess: {0}to attrs=userPassword by self write by dn.base="cn=Manager,dc=dopensource,dc=com" write by anonymous auth by * none' >> olcDatabase\=\{2\}bdb.ldif
echo 'olcAccess: {1}to * by dn.base="cn=Manager,dc=dopensource,dc=com" write by self write by * read' >> olcDatabase\=\{2\}bdb.ldif

Set OpenLDAP service to start on boot

chkconfig slapd on
service slapd start

If you get any echecksum errors, such as:

5900f369 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={0}config.ldif"
5900f369 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={1}monitor.ldif"
5900f369 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif"

Then you must run the following commands to fix your checksums before adding any entries.

Fixing checksum errors in config files

Copy the file(s) shown to have bad checksums when starting slapd to /tmp

mkdir -p /tmp/fixes &&
/bin/cp -rf /etc/openldap/slapd.d/cn=config/{olcDatabase={0}config.ldif,olcDatabase={1}monitor.ldif,olcDatabase={2}bdb.ldif} /tmp/fixes &&
cd /tmp/fixes

Remove the first 2 lines containing the old checksums:

for file in /tmp/fixes/*; do
    sed -i '1,2d' "${file}"
done

Install the zlib dev package and perl archive utils if not installed

For centos based systems:

yum -y -q install zlib-dev perl-Archive-Zip

For debian based sytems:

apt-get -y -q install zlib1g-dev libarchive-zip-perl

Now we can go ahead and calculate the new checksums:

for file in /tmp/fixes/*; do
CRC=$(crc32 "$file")

read -r -d '' INSERT_LINES << EOF
# AUTO-GENERATED FILE - DO NOT EDIT!! Use ldapmodify.
# CRC32 ${CRC}
EOF

cat << EOF > "${file}"
${INSERT_LINES}
$(cat ${file})
EOF
done

Then copy back the fixed files over the originals & delete tmp folder:

/bin/cp -rf /tmp/fixes/* /etc/openldap/slapd.d/cn=config/
cd /tmp && rm -Rf fixes

Create the root entry in our ldap tree.

The root entry is going to be configured as a special entry called a domain controller. To do this we will add the same domain name we have been using to the domain controller root entry.

Create the root entry in a temp file, should be named after your domain:

cat << EOF > /tmp/dopensource.ldif
dn: dc=dopensource,dc=com
objectClass: dcObject
objectClass: organization
dc: dopensource
o : dopensource
EOF

Add the contents of the file to your tree (pw is the amdmin ldap user):

ldapadd -f /tmp/dopensource.ldif -D cn=Manager,dc=dopensource,dc=com -w <your ldap admin pw>

Verify it was added (you should see the same info you added displayed):

ldapsearch -x -LLL -b dc=dopensource,dc=com

If you got back the same entry you put in, then delete the temp file:

rm -f /tmp/dopensource.ldif

We add in a symlink for better portable here:

ln -s /etc/openldap /etc/ldap

Allow access to the ldap server in your firewall rules:

iptables -A INPUT -p tcp --dport 389 -j ACCEPT
/sbin/service iptables save
iptables -F
service iptables restart

To ensure your iptables rule persisted you can check with:

iptables -L

To verify ldap is up check on port 389:

netstat -antup | grep -i 389 --color=auto
<terminal output here>

You should see LISTEN in your output, something like this:

tcp        0      0 0.0.0.0:389                 0.0.0.0:*                   LISTEN      22905/slapd
tcp        0      0 :::389                      :::*                        LISTEN      22905/slapd

For More Information


For more information, the next post in the series or for any questions visit us at:

dOpenSource
Flyball Labs

For Professional services and Ldap configurations see:

dOpenSource OepnLDAP
dOpenSource OpenLDAP Services

Written By:

DevOpSec
Software Engineer
Flyball Labs

Creating Custom OCF Resource Agents

A Simple Guide for Creating OCF Resource Agents with Pacemaker and Corosync


HA Clustering Explained

For those not familiar with High Availability resources & services or Cluster Computing, in this section we will go over a brief summary and explanation of OCF RA’s before we dive in to creating our own. In cluster networking multiple servers are inter-connected together on a network to provide improved delivery of services such as with HA.

The servers (referred to as nodes) are all managed by a Cluster Resource Manager (CRM), which as you may have guessed, manages cluster resources. High Availability (HA) refers to the ability to provide a resource or service with minimal to zero downtime. In layman’s terms, whatever client wants, client gets, and the store is open 24/7/365. In our case we will be using Pacemaker as our CRM.

Now each node has to have a way to communicate with other nodes in cluster for this to work, when a resource fails, or a node goes down, the CRM needs to be notified to make other resources available. This is where Corosync steps in. Corosync the communication layer that provides Pacemaker with the status of resources and nodes on the cluster. The information that Pacemaker receives can communicate many complex situations and scenarios that Pacemaker can deal with in a variety of ways.

OCF Resource Agents

Where do the Resource Agents fit into this picture you say? Glad you asked, the RA’s are deployed by pacemaker to interface with each resource that needs managed, using Corosync’s communication layer. The protocol used by the RA’s is known as the Open Cluster Framework (OCF) and defines how RA’s should be implemented and requirements to create one. Most of the time they are implemented in a shell script, but they can be written in any programming language needed. More info on HA cluster computing and OCF protocol implementation can be found at Cluster Labs and if you have any further questions feel free to email us at Flyball Labs That is it for the lecture today, on with the chlorophyll!


How to Create Your Own OCF RA


Prerequisites

This guide assumes that you have Corosync and Pacemaker setup on muli-node cluster (atleast 2).

You must also have access to the remote nodes via ssh / scp to transfer the completed files. Note that every node will need a copy of the RA script.

Design

We will be creating a simple Resource Agent that runs a supplied script to perform a health check. Our health check will simply check if the node has any hard drives that are 95% used or more. If the health check does not pass, then we must shut the node down and perform a failover to another node (including our RA).

Instructions

  1. First we are going to grab a template OCF script. You will need to install git cmd line package if you haven’t installed already. Change dir into that directory.
    
        git clone https://github.com/russki/cluster-agents.git &&
        cd cluster-agents
    
  2. Make a copy and rename the RA something pertinent like “HealthAgent”.
        cp generic-script health-agent
    
  3. Make sure to mae the copy executable.
        chmod +x HealthAgent
    
  4. Change names in the agent script to the one we renamed it to.
        sed -i 's/generic_script/health_agent/g' health-agent
        sed -i 's/generic-script/health-agent/g' health-agent
        sed -i 's/Generic Script/Health Agent/'  health-agent
    

    We now have a Resource Agent that can run and manage a shell script. Arguments will be passed through the agent to the calling script that we supply it. Now we need to copy it over to each of the cluster nodes, install and test it.

  5. We can accomplish this fairly simple with a bash script such as the one below. Note I just whipped this up so may need tweaked.

        #!/bin/bash
        #
        # Assuming user is the same on each node
        #
    
        USAGE() {
            echo "./rlogin.sh    ..."
        }
    
        if [ "$#" -eq 0 ]; then
            USAGE
            exit 1
        fi
    
        read -p "Username: " user
    
        for server in "$@"
        do
          scp health-agent "$user"@"$server":/usr/lib/ocf/resource.d/heartbeat
        done
    
        exit 0
    
  6. Create a shell script for the resource manager to run, which will check the HDD status on the node.
    Here is a sample script that I use across my cluster nodes, that checks the HDD capacity, and returns status.

    
        #!/bin/bash
        # Check if physical hard drives are full
        # Return 1 if any drive is over tolerance (percent)
        # Return 0 if all drives are under tolerance (percent)
    
        TOLERANCE=5 #default to 1)print $0}')
        TEST="$((100 - $TOLERANCE))"
    
        while read -r line; do
            if [[ "$line" -gt "$TEST" ]]; then
                echo "drive is over tolerance"
                exit 1
            fi
        done <<< "$USED"
    
        echo "all drives passed"
        exit 0
    

    Save the bash script, make it executable, then SCP it over to all the servers. Then connect to each node, perform tests, and add resource if everything checks out.

  7. Test your agent using ocf-tester to catch any mistakes (on remote nodes). We want to do this on all nodes to ensure they are configured / have correct dependencies to run our script. Node: provide it with full path to script.

        ocf-tester -n /usr/lib/ocf/resource.d/heartbeat/health-agent
    
  8. Try executing agent in a shell and make sure it executes w/o any major errors. Again like previous example we should check each of the nodes.
        export OCF_ROOT=/usr/lib/ocf; bash -x /usr/lib/ocf/resource.d/heartbeat/health-agent start
    
  9. Add the resource to Pacemaker using pcs.
    This should be done on all nodes in cluster you want that resource agent to monitor on.

        pcs resource create health-agent ocf:heartbeat:health-agent script="full/path/to/script/to/run" state="/dev/shm" alwaysrun="yes" op monitor interval=60s
    
  10. To see status of cluster:
        pcs status
    

    Which should show your newly added resource agent and some additional status information that can be helpful in debugging.

    To show detailed information about only the health-agent resource run the following:

        pcs resource show health-agent
    

    You can also run resource agent action cmds with the same cmd. This can be useful to test if a certain action isn’t working properly.

        pcs resource debug-start health-agent
        pcs resource debug-stop health-agent
        pcs resource debug-monitor health-agent
    

Summary

I hope that helped clear up a few gotchas about cluster resource management. The OCF protocol is not very intuitive and small changes can completely ruin your previously working agents. Make sure to test, test, and test again. There is also a more sophisticated testing framework for OCF resource agents called: ocft which allows for testing environments, creating test cases, and much more. You can find that tool at Linux HA Hope you enjoyed this post, more awesomeness to follow!


For More Information

For more information, questions and comments visit us at:

dOpenSource
Flyball Labs

Written By:

DevOpSec
Software Engineer
Flyball Labs

Scripting one time ssh access to allow for Google Authenticator setup

In our previous article we setup google-authenticator for authenticating openssh. Now, we need a way for users to be able to login once before setting up google-authenticator. Here is a script for checking if a user has not logged in and ran google-authentication yet, runs google-authenticator, then prevents that user from logging in again without either google-authentication or an ssh public key. To setup this script do the following:

  1. Edit /etc/pam.d/sshd and make the following changes after #%PAM-1.0
    auth    required        pam_sss.so try_first_pass
    auth    sufficient  pam_listfile.so item=user sense=allow file=/google-auth/authusers
    auth    sufficient      pam_google_authenticator.so
    auth       required     pam_sepermit.so
    #auth       include      password-auth
    

    We are using ldap, so our users authenticate with pam_sss.so. If you are using regular unix passwords, you should change this first required auth line to pam_unix.so instead. We setup a pam_listfile, which has a list of all the users we want to allow access to login. Our script, called from each of these users .bash_profile file, will remove the user from this file after their first login without google-authenticator setup. Notice, we commented out the password-auth include line. We do not want to call the password-auth file because we only want users to be able to log in if they are in if they can first authenticate with sss/ldad and are in our google-auth list. Otherwise, they can login with google_authenticator, or from an sshd key, which will happen before the authentication is sent to pam.

  2. Create the /google-auth/authusers file, which pam will check for which users to allow without requiring further authentication. I used /home/ to see which users are actual human users who will need to login. This file just needs to be a list with one username per line. This file needs to be in a new directory, because we need to give all of these users permission to write to it so their usernames can be removed via our script once they create a google authentication file.

    a. create a group and add all of the necesarry users to this group.

    groupadd google-auth
    gpasswd google-auth -M bob,joe,smith
    

    b. Now create the authusers file and set the permissions to be owned by google-auth, then allowed write access by users in that group.

    mkdir /google-auth/
    touch /google-auth/authusers
    chgrp google-auth /google-auth/authusers
    chmod ug=rwx,o= /google-auth/authusers
    

    c. Now add the necesarry users to /google-auth/authusers

    bob
    joe
    smith
    
  3. Install the script to be ran. Just add these couple lines to an .sh file in /usr/local/bin, or you can copy it over if you did a git pull. I created my script with vim /usr/local/bin/google-auth-check.sh
    #!/bin/bash
    
    if [ ! -f ~/.google_authenticator ]; then
        google-authenticator
        if [ -f ~/.google_authenticator ]; then
            sed -i "/^${USER}$/d" /google-auth/authusers
        fi
    fi
    

    And then give it execute permissions.

    chmod +x /usr/local/bin/google-auth-check.sh
    
  4. Next, we need to add this script to each of the ~/.bash\_profile files in each users home directory. I will come up with a find and echo command, but for now, just add this at the end of these files in each users directory you want this to work for.
    sh /usr/local/bin/google-auth-check.sh
    

You can run this bash command to find every .bash_profile in /home/ and append the line above. Be careful before running this command. It works on my system, but you can never be too careful with recursive commands such as this.

find /home/ -name ".bash_profile" -print0 | xargs --verbose -0 -I{} sh -c "echo 'sh /usr/local/bin/google-auth-check.sh' >> {}"

This should be all you need to do. Now when a user logs in for the first time, the google-authenticator application will automatically run and create a .google\_authenticator file. The user will then be able to add the key into their phone app and have multifactor authentication to log into their account. Their name will get removed from the /google-auth/authusers file, so they will no longer be given access without a multi factor auth, or ssh key.

Sources:

https://www.digitalocean.com/community/tutorials/how-to-use-pam-to-configure-authentication-on-an-ubuntu-12-04-vps

http://www.linux-pam.org/Linux-PAM-html/sag-pam_exec.html

https://wiki.archlinux.org/index.php/PAM

Setting up Google Authentication for SSH Multi-Factor Auth

Install google-authenticator

In this tutorial we will go over how to setup two factor authentication for SSH using google-authenticator on CentOS 6.

First thing we need to do is install the google-authenticator package using yum.

yum install google-authenticator

Next, run google-authenticator to genereate a key.

google-authenticator
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/nschoonover@ldap.dopensource.com%3Fsecret%3D3FO6WYXPZUMTEWUB
Your new secret key is: 3FO6WYXPZUMTEWUB
Your verification code is 799347
Your emergency scratch codes are:
  84194653
  29899626
  13794318
  84024610
  76941184

Do you want me to update your "~/.google_authenticator" file (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n)
By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) y

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n)

With this information, you will need to install the google authenticator application on your phone. Copy and paste the URL at the beginning of the output from google-authenticator, which should show you a QR code. Scan this code using the Google Authenticator application on your phone. You will now see a 6 digit number in your Google Authenticator app. This number will change every certain ammount of minutes. This number will be your second authentication method when logging into your server. If you ever lose your secret key and need to setup another phone, you can always run the google-authenticator command again to regenerate.

Next, we need to setup pam to use google_authenticator for authenticating users.

Open up the pam ssh configuration file.

sudo vim /etc/pam.d/sshd

Then add the following lines at the bottom.

## Google autehnticator for ssh
auth required pam_google_authenticator.so

Next, we need to add the ChallengeResponseAuthentication yes option to the /etc/ssh/sshd\_config file.

# Change to no to disable s/key passwords
ChallengeResponseAuthentication yes
#ChallengeResponseAuthentication no

Now restart sshd for the new changes to take affect.

sudo service sshd restart

Troubleshooting

The first time I tried to set this up, I was unable to login using my authentication code. I would be prompted for a code when tried to initiate an ssh connection, but I would see this message in the logs and would not be let in.

Dec 28 17:49:43 web2p sshd(pam_google_authenticator)[19784]: Invalid verification code

It turns out the problem was caused by the time on my testing server not being correct. Make sure you have ntp running and configured correctly.

To install and set ntp to auto start:

yum install ntp
chkconfig ntpd on
service ntpd start

Now run this command to manually force ntp to sync with pool.ntp.org

ntpdate pool.ntp.org

You can check the date on your server with the date command to make sure ntp is working. After syncing my time, I was able to log into my ssh server using google authenticator without a problem.

Public Key Authentication

We decided that we wanted to allow users to login using either Google Authenticator, or a regular ssh Pulic Key. We generally use public keys for ssh authentication, but occasionally we will want to login to one of our servers from a machine we will only use once. Google Authenticator is a great compromise for this as it allows us to authenticate without an ssh key, but still maintains our security.

To ensure sshd will allow authentication from either the Google Authenticator module in pam, or an ssh key, uncommend the following lines in /etc/ssh/sshd\_config

PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication yes
UsePAM yes

The changes we made to /etc/pam.d/sshd should only effect the password authentication portion of the ssh login process. If PubkeyAuthentication yes is set, ssh should approve your ssh key authentication without reaching out to pam and requiring two factor authentication.

In our next article, we will go over how to allow users to login once before google-authentication is enforced.

Backing up a Linux Server with Jungle Disk

Installing Jungledisk

If you are installing the Jungle Disk agent to a server via command line, you will need to use wget or another cli http client to download the installer. We are installing Jungle Disk to a CentOS 6 server, but there are also packages for Debian based systems, and a tar file with the source if you would like to compile.

Download the installer, and run the rpm to setup and start the backup agent.

wget https://downloads.jungledisk.com/jungledisk/junglediskserver-3180-0.x86_64.rpm
[root@ldap ~]# sudo rpm -i junglediskserver-3180-0.x86_64.rpm
Stopping junglediskserver: [FAILED]
Starting junglediskserver:
[root@ldap ~]# ps -ef | grep jungle
root 22936 1 0 15:58 ? 00:00:00 /usr/local/bin/junglediskserver

This is the rpm package as of the time this article is written, but as there are new releases of the agent software, this will change. Make sure you download the latest version from your Jungle Disk account.

Setup the license

[root@ldap jungledisk]# cd /etc/jungledisk/
[root@ldap jungledisk]# cp junglediskserver-license-EXAMPLE.xml junglediskserver-license.xml

Now you will need to add your license to the new junglediskserver-license.xml file. You can find this license in your licenses section of the control panel. https://secure.jungledisk.com/secure/account/subscriptions.aspx

vim /etc/jungledisk/junglediskserver.xml

Enter in your license string in the licenseKey section.

<licenseKey>some_license_string</licenseKey>

Setting up the Management Client and configuring a backup.

You will need to install the Management Client to a Windows computer. Download the client here: https://secure.jungledisk.com/downloads/business/server/windows/

Once it is downloaded and installed, you will need to log in. Click Next to start the configuration.

welcome

If you already have setup Jungle Disk to backup another server, you will need to chose your new server from the server list. The client will be the hostname on the server. Now, create a new online disk.

new-disk

Choose between Standard Security, or High, which will encrypt your data whil it is stored on the Jungle Disk infrastructure. (We chose standard)

standard

Select you backup schedule. We stayed with the default daily schedule.

backup_schedule

Next, select what files to backup on your linux server.

what_to_backup

Hit ok, and next. You should be done! You can now try running a manual backup to get your files backed up right away, or wait for the scheduled backup to run.

Testing FreeSWITCH for memory leaks using Valgrind and SIPp

In this tutorial we will assume you have already setup SIPp on another server to stress test FreeSWITCH. We covered this in our previous article

We will need to install valgrind on the same machine running FreeSWITCH. I like using Debian for FreeSWITCH, which conveniently already has valgrind in the repository.

apt install valgrind

Now that valgrind is installed, we will need to make sure FreeSWITCH is not already running.

systemctl stop freeswitch
ps -ef | grep freeswitch

Once we stop FreeSWITCH, we want to start up FreeSWITCH using valgrind. Below is the command to do so. This will log all memory issues found by Valgrind to vg.log in the directory you run the command from.

/usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Running the FreeSWITCH from the stable repository will not result in many errors. Below is my the end of my vg.log while running FreeSWITCH 1.6.12-20-b91a0a6~64bit

==20986== LEAK SUMMARY:
==20986==    definitely lost: 0 bytes in 0 blocks
==20986==    indirectly lost: 0 bytes in 0 blocks
==20986==      possibly lost: 16,384 bytes in 2 blocks
==20986==    still reachable: 192 bytes in 1 blocks
==20986==         suppressed: 0 bytes in 0 blocks
==20986==
==20986== For counts of detected and suppressed errors, rerun with: -v
==20986== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)

There are no new errors after the leak summary in this log.

When the FreeSWITCH is having memory management problems, valgrind will continue to log errors after the LEAK SUMMARY. Below is an example of a memory leak when accessing the libcrypto.so library for WebRTC.

143768 ==32354== Use of uninitialised value of size 8
143769 ==32354==    at 0x67C8B80: ??? (in /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0)
143770 ==32354==    by 0xF684D2C1E662C6C4: ???
143771 ==32354==    by 0xE16640B5E123D1DD: ???
143772 ==32354==    by 0xD68DF16DAF4386F1: ???
143773 ==32354==    by 0xB2B603F64080C703: ???
143774 ==32354==    by 0x398A1FCC7DC2FF3B: ???
143775 ==32354==    by 0xBE60B2707D98773F: ???
143776 ==32354==    by 0x375392F355F20B45: ???
143777 ==32354==    by 0x124A183B188AA4EA: ???
143778 ==32354==    by 0x5BFE669681E01F8: ???
143779 ==32354==    by 0x6A4C9E9A4AA5A4C0: ???
143780 ==32354==    by 0x29B07C509D28C49F: ???
143781 ==32354==

In this case, valgrind was logging this message to the log many times per second. You can view the memory mismanagement manifesting itself in a memory leak by viewing the allocated memory with ps. We will want to monitor the change in RSS, or Resident Set Size, which represents the ammount of bytes are allocated for a process in RAM.

ps -aux | grep '[v]algrind.bin'

On Debian 8, ps -aux will list the information in the following order:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

The output of your ps -aux command should look something like this

root     20987 36.7 13.1 631032 271356 ?       S<sl 13:33  30:47 /usr/bin/valgrind.bin --tool=memcheck --error-limit=no --log-file=vg.log --leak-check=full --leak-resolution=high --show-reachable=yes /usr/bin/freeswitch -vg -ncwait -nonat

Knowing the current total allocated memory for FreeSWITCH is cool, but not very useful for seeing the memory leak in action. If FreeSWITCH has a memory leak, it will fail to release memory when it is done with it, causing the allocated space to continue to grow. A useful way to observe this is by monitoring the RSS of the FreeSWITCH process over time, then running SIPp to create an artificial load on FreeSWTICH I created a simple bash script to log FreeSWITCH’s RSS, the current time, and the number of sip channels to a file.

#!/bin/sh


while true; do

INT=60
FILE=/root/mem.log
DATE=$(date +"%F-%R")
MEM=$(ps -aux | grep '[v]algrind.bin' | awk -F ' ' '{print $6}')
CHANNELS=$(/usr/local/freeswitch/bin/fs_cli -x "show channels" | grep total | awk -F ' ' '{print $1}')

echo ${DATE},${MEM},${CHANNELS} >> ${FILE}
sleep ${INT}s
done

This script will log the time, RSS, and SIP channels to /root/mem.log. You may need to adjust the CHANNELS variable to point to wherever the fs_cli binary is stored.

Save the script to a file, such as memtest.sh, then run it with

./memtest.sh &

You can confirm it is running in the background with jobs.

Now, with FreeSWITCH running under valgrind, we will want to start up the SIPp stress test. On the SIPp server, run the following.

sipp -sf uac.xml -s 9999999  <freeswitch-ip>:5080 -trace_msg -l 15 -d 60000

This test is best done over a long period of time. I would let SIPp generate traffic on the server for a couple hours while our monitor script is running. After the system has ran under load for some time, the mem.log should look soemthing like this.

2016-10-24-15:29,271212,15
2016-10-24-15:29,270800,15
2016-10-24-15:30,270672,15
2016-10-24-15:31,270828,15
2016-10-24-15:32,270708,15
2016-10-24-15:33,270604,15
2016-10-24-15:34,270812,15
2016-10-24-15:35,270628,15
2016-10-24-15:36,270816,15

If FreeSWITCH has been running for multiple hours and the RSS has seemed to plateau, like above, FreeSWITCH is likely not showing any critical memory mismanagements. Here is an example of the RSS continuing to grow due to a memory leak.

2016-08-11-04:09,379192,60
2016-08-11-04:10,381584,60
2016-08-11-04:11,383104,60
2016-08-11-04:12,385096,60
2016-08-11-04:13,385356,60
2016-08-11-04:14,387580,60
2016-08-11-04:16,387432,60
2016-08-11-04:17,389316,60
2016-08-11-04:18,391144,60
2016-08-11-04:19,393076,60
2016-08-11-04:20,394484,61
2016-08-11-04:21,395776,60
2016-08-11-04:22,397072,60
2016-08-11-04:23,398140,59
2016-08-11-04:24,400112,60
2016-08-11-04:25,401868,60
2016-08-11-04:26,403348,60

FreeSWITCH had been running for many hours at this point, and the RSS had never stopped growing. This is the same system which was logging the libcrypto error in the valgrind log.

When you notice a memory issue with your version of FreeSWTICH, check the FreeSWTICH JIRA for known bugs. There may already be a patch out for the issue you are experiencing.

Moving to Docker – Practicing What We Preach (Work in Progress)

We have made a decision to jump head first into containers with a focus on providing Enterprise Grade Docker and Kubernetes Support.  It’s only right to convert our infrastructure into containers.  We are going to start with our bread and butter, which is our phone system.  Being a support company with a strong focus on helping companies deploy VoIP means that this is huge risk for us – we have to get this right or we will literally lose money.  But, if you never jump you will not soar.  In other words, if we are going to provide support for Docker, Kubernetes and in general Microservices we need to go through the process ourselves so that we can help others navigate the waters.

Current Environment

We currently use the FreePBX distribution as our PBX and Flowroute as our carrier (we love Flowroute by  the way).  We only use an handful of the options in FreePBX.  To be precise, we use:

  • IVR
  • Extension Management
  • Voicemail
  • Follow-me
  • Queues
  • Ring Groups
  • Conferences

If you think about it…each one of these features could become a container, but does that really make sense….not really sure yet.

Project Goals

We have decided to break this project into phases and we will make each new phase better by incorporating what we learned from the previous phases.

Phase I – Create a conference container and have all conference traffic be routed to 1 or more containers that just handles conferencing.  There are times when we had multiple conferences happening all at the same time, which causes performance issues.  We want a solution that would realize that the existing conference server has too many requests and it automatically spins up another conference server and start routing requests to the newly created instance.

More phases to come…

Proposed Components

  • Kamailio – to provide load balancing and routing of calls to one or more Asterisk servers
  • Asterisk – Media server
  • Docker
  • Consul – Will keep track of all Asterisk servers that are up and available

We will also need a Web GUI for managing the conferences and the other features that we need.  FreePBX is not container aware, so we are toying with the idea of developing a simple Web GUI that knows how to deal with functionality being containerized.

Creating a Docker container from an existing image

In this tutorial we will go over how to search and pull down an existing docker image from the docker hub to run it on your server or local system.

Prerequisites

  • You will need to have docker running and installed on your system.

Searching the Docker Hub

  • You can search the Docker Hub for pre made images. These images are either made by the community, or official builds of different projects.
    docker search centos
    
  • You will get the following response. Notice what it shows you. This will include the following:

  1. A description
  2. Number of stars voted by the community
  3. Indication if the image is official and built by the organization in charge of the project
  4. Indication if the build is automated or not
  • Below is an example of what you will see when searching the Docker Hub.

docker_search

  • Notice the first image is marked as the official CentOS docker image.

Creating a container from a remote image

  • Let’s create a container based on the official CenOS build.
    docker run -dit docker.io/centos
    

docker_run

  • Note, Docker will automatically download the image from the Docker Hub if the image is not installed on the host. Once completed, the image will be ran.

  • Here are the what the flags used in the previous command do:

Flag Function
-d Run container in background
-i Keep STDIN open, even if not attached
-t Allocate a pseudo-TTY

View running containers

  • Validate the container is running by getting a list of all running containers:
    docker ps
    

docker_ps

In our next tutorial we will go over how to attach to a container.