Advanced Debugging Techniques for OpenBTS

Overview

OpenBTS uses the syslog mechanism to generate logging. This can be controlled by either syslogd or rsyslog (please consult your Linux distrabution documentation for which facility provides logging).

By using syslog for logging, we can control logging granularity by file, or globally. This allows one to turn on logging only for areas that might be producing a problem, instead of flooding logging with superfluous information that will making debugging harder and take longer.

I will now lead you through how to enable logging, how to tune logging, as well as other debugging methods for making the most of your time while tracking down issues and bugs within OpenBTS.

This documentation assumes that you are running an Ubuntu based Range Networks system (Developer's Kit or GSM Access Point). If you are not running Ubuntu, please consult your distribution documentation.

Logging

Where is output stored?

Output is generally logged to /var/log/messages. In commercial installations, OpenBTS is already configured to log to /var/log/OpenBTS.log. One can configure rsyslog to log to a separate file by (probably creating and) editing the following:

/etc/rsyslog.d/OpenBTS.conf

local7.*                        /var/log/OpenBTS.log

How do I control what information is logged, and by how much?

OpenBTS generates logging based off of the Log.Level.* configuration options. Globally, you can control the system-wide log level with the Log.Level configuration key.

Valid options are as follows:

  • EMERG
  • ALERT
  • CRIT
  • ERR
  • WARNING
  • NOTICE
  • INFO
  • DEBUG

You can also log per file. This allows you to override the global settings so that you can get more debug information for a certain part of the system without affecting other levels. This is important because setting a global log level of DEBUG can significantly reduce performance of the system, causing in absolute failures, making it impossible to reproduce a possible issue. Also, more DEBUG output means more information to sift through when inspecting the log.

To override a certain file's log level, you would use a configuration key in this format: Log.Level.<filename>. For instance, to override the logging level of the SIPEngine.cpp file, you could set Log.Level.SIPEngine.cpp to whatever level you wanted.

From the OpenBTS CLI, it would look something like this:

OpenBTS> config Log.Level INFO
changed Log.Level to "INFO"

OpenBTS> config Log.Level.SIPEngine.cpp DEBUG
defined new config Log.Level.SIPEngine.cpp as "DEBUG"

OpenBTS>

Crashing

Corefiles

Corefiles provide a valuable resource. They allow a system component (with debug symbols enabled, usually compiled in with -g) to provide a backtrace when they crash, without requiring the user to be using gdb. To enable this, you will need to do the following:

1. Add the following, as root or sudo, to /etc/security/limits.conf

*               soft    core            2000000
root            soft    core            2000000

1. Edit /etc/sysctl.conf as root or with sudo and add the following:

fs.suid_dumpable=1
kernel.core_pattern=/var/corefiles/core.%e.%p.%h.%t
kernel.core_uses_pid=1

1. Create the directory, as root or sudo, /var/corefiles/. 1. Reboot the system.

All core files will be placed in /var/corefiles/. You can read them as follows:

gdb /path/to/OpenBTS /var/corefiles/thecorefile