Advanced Debugging Techniques for OpenBTS
OpenBTS uses the syslog mechanism to generate logging. This can be controlled by either syslogd or rsyslog (please consult your Linux distrabution documentation for which facility provides logging).
By using syslog for logging, we can control logging granularity by file, or globally. This allows one to turn on logging only for areas that might be producing a problem, instead of flooding logging with superfluous information that will making debugging harder and take longer.
I will now lead you through how to enable logging, how to tune logging, as well as other debugging methods for making the most of your time while tracking down issues and bugs within OpenBTS.
This documentation assumes that you are running an Ubuntu based Range Networks system (Developer's Kit or GSM Access Point). If you are not running Ubuntu, please consult your distribution documentation.
Where is output stored?
Output is generally logged to /var/log/messages. In commercial installations, OpenBTS is already configured to log to /var/log/OpenBTS.log. One can configure rsyslog to log to a separate file by (probably creating and) editing the following:
How do I control what information is logged, and by how much?
OpenBTS generates logging based off of the Log.Level.* configuration options. Globally, you can control the system-wide log level with the Log.Level configuration key.
Valid options are as follows:
You can also log per file. This allows you to override the global settings so that you can get more debug information for a certain part of the system without affecting other levels. This is important because setting a global log level of DEBUG can significantly reduce performance of the system, causing in absolute failures, making it impossible to reproduce a possible issue. Also, more DEBUG output means more information to sift through when inspecting the log.
To override a certain file's log level, you would use a configuration key in this format: Log.Level.<filename>. For instance, to override the logging level of the SIPEngine.cpp file, you could set Log.Level.SIPEngine.cpp to whatever level you wanted.
From the OpenBTS CLI, it would look something like this:
OpenBTS> config Log.Level INFO changed Log.Level to "INFO" OpenBTS> config Log.Level.SIPEngine.cpp DEBUG defined new config Log.Level.SIPEngine.cpp as "DEBUG" OpenBTS>
Corefiles provide a valuable resource. They allow a system component (with debug symbols enabled, usually compiled in with -g) to provide a backtrace when they crash, without requiring the user to be using gdb. To enable this, you will need to do the following:
- Add the following, as root or sudo, to /etc/security/limits.conf
* soft core 2000000 root soft core 2000000
- Edit /etc/sysctl.conf as root or with sudo and add the following:
fs.suid_dumpable=1 kernel.core_pattern=/var/corefiles/core.%e.%p.%h.%t kernel.core_uses_pid=1
- Create the directory, as root or sudo, /var/corefiles/.
- Reboot the system. All core files will be placed in /var/corefiles/.
You can read them as follows:
gdb /path/to/OpenBTS /var/corefiles/thecorefile