Solving BLF issues in Kazoo with RAM

We were struggling with Kazoo BLF issues for a long time once we started to get more and more customers running on our platform. At first, we put this down to customer internet connections or some sort of SIP ALG interference but what it really was, and why it looked like a customer issue, was problems with the kazoo_db (an SQLite DB) disk performance, for customers that had many BLF buttons to monitor.

We confirmed the issue with the following command;

kamcmd stats.get_statistics db_kazoo: kazoo: script: usrloc:

It returns a bunch of statistics, but we are most interested in db_kazoo stats. Anything above zero means there’s a problem;

db_kazoo:busy1 = 53
db_kazoo:busy2 = 2
db_kazoo:busy3 = 0
db_kazoo:busy4 = 0
db_kazoo:busy5 = 0

Likewise, the kamailio logs reported a “db_kazoo is busy” event around multiple subscribe events. In our instance, the servers were running HDDs, not SSDs (though they have now been upgraded). However, given that the SQLite DB that runs kamailio is rather small and we had plenty of RAM to spare, we decided to run a RAMDISK to hold the db, which would never have performance issues.

The downside to RAMDISKS however, is when you lose power, you lose the data on the disk. Normally, this wouldn’t be an issue, because restarting Kamailio sees the db regenerated. However, we had data in that db we did want to preserve (custom trusted lists), so I implemented the following solution.

The backed-up RAMDISK

We needed a ramdisk, but we needed the data preserved between reboots or during an unexpected shutdown like power loss. The solution, was to run a service that backs up the RAMDISK every {x} minutes and also backs up the RAMDISK on shutdown. During startup, the backup is restored to the RAMDISK.

Additionally, the RAMDISK had to be restored BEFORE kamailio started, and backed up AFTER kamailio shut down.

Create the RAMDISK

Create the folder first in /mnt/

sudo mkdir /mnt/ramdisk

The following was added to the end of /etc/fstab to create a 128MB ramdisk

tmpfs                   /mnt/ramdisk            tmpfs   defaults,size=128M 0 0

Then mount it

sudo mount /mnt/ramdisk

The Script

The following script, stored at /mnt/ramdisk.sh will be used to power our ramdisk service. The start command extracts a backup. The stop command tarballs a new copy of the backup. The sync command renames the previous copy to “-old” and creates a new backup (in case there’s power loss during a backup, we can restore manually from -old).

#! /bin/sh
# /mnt/ramdisk.sh
#

case "$1" in
  start)
    echo "Copying files to ramdisk for startup"
    cd /mnt
    tar zxvf ramdisk-backup.tar.gz
    echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched from HD >> /var/log/ramdisk_sync.log
    ;;
  sync)
    echo "Synching files from ramdisk to Harddisk"
    echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched to HD >> /var/log/ramdisk_sync.log
    cd /mnt
    mv -f ramdisk-backup.tar.gz ramdisk-backup-old.tar.gz
    tar zcvf ramdisk-backup.tar.gz ramdisk
    ;;
  stop)
    echo "Copying files from ramdisk to Harddisk for shutdown"
    echo [`date +"%Y-%m-%d %H:%M"`] Ramdisk Synched to HD >> /var/log/ramdisk_sync.log
    cd /mnt
    tar zcvf ramdisk-backup.tar.gz ramdisk
    ;;
  *)
    echo "Usage: /etc/init.d/ramdisk {start|stop|sync}"
    exit 1
    ;;
 esac

 exit 0

The Service

The service file (located at /etc/systemd/system/ramdisk.service) defines the constraints for our service. It ensures it is a oneshot runner, and it executes BEFORE kazoo-kamailio.service when starting (and after when stopping).

[Unit]
Description=Ramdisk Sync and Restore for KazooDB
After=network.target
Wants=kazoo-kamailio.service
Before=kazoo-kamailio.service

[Service]
Type=oneshot
User=root
ExecStart=/bin/sh /mnt/ramdisk.sh start
ExecStop=/bin/sh /mnt/ramdisk.sh stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The Symlink

The final step is to tell Kazoo-Kamailio to use our ramdisk. This is easiest done through symlinks. OPTIONALLY, copy what you have (while kazoo-kamailio isn’t running), then create the symlink.

sudo cp /etc/kazoo/kamailio/db/* /mnt/ramdisk/db/
sudo rm -rf /etc/kazoo/kamailio/db
sudo ln -s /mnt/ramdisk/db /etc/kazoo/kamailio/db

Enable the services and sync

Finally, enable the service and run it.

sudo systemctl daemon-reload
sudo systemctl enable ramdisk
sudo systemctl start ramdisk

Then add the following to your crontab to configure the sync interval;

*/15 * * * * systemctl is-active --quiet ramdisk && /bin/sh /mnt/ramdisk.sh sync