Apache HTTPD Debugging Guide - The Apache HTTP Server Project
Essentials ¶ Download! About License FAQ Security Reports Source Repositories ¶ General Information Trunk 2.4 Documentation ¶ Version 2.4 Trunk (dev) Wiki Get Involved ¶ Mailing Lists Bug Reports Developer Info User Support Subprojects ¶ Docs Test Flood Modules mod_fcgid Related Projects ¶ Apache Traffic Server Apache Traffic Control Tomcat APR mod_perl Miscellaneous ¶ Contributors Thanks! Sponsorship Privacy Apache Debugging Guide ¶ This document is a collection of notes regarding tools and techniques for debugging Apache httpd and its modules.
Got more tips? Send 'em to docs@httpd.apache.org . Thanks! Using gdb Getting a live backtrace on unix Getting a live backtrace on Windows Debugging intermittent crashes Using truss Getting the server to dump core Solaris and coredumps Getting and analyzing a TCP packet trace Using gdb ¶ If you use the gcc compiler, it is likely that the best debugger for your system is gdb. This is only a brief summary of how to run gdb on Apache -- you should look at the info and man files for gdb to get more information on gdb commands and common debugging techniques.
Before running gdb, be sure that the server is compiled with the -g option in CFLAGS to include the symbol information in the object files. The only tricky part of running gdb on Apache is forcing the server into a single-process mode so that the parent process being debugged does the request-handling work instead of forking child processes. We have provided the -X option for that purpose, which will work fine for most cases.
However, some modules don't like starting up with -X , but are happy if you force only one child to run (using " MaxClients 1 "); you can then use gdb's attach command to debug the child server. The following example, with user input in green , shows the output of gdb run on a server executable (httpd) in the current working directory and using the server root of /usr/local/apache : % gdb httpd GDB is free software and you are welcome to distribute copies of it under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details. GDB 4.16.gnat.1.13 (sparc-sun-solaris2.5), Copyright 1996 Free Software Foundation, Inc... (gdb) b ap_process_request Breakpoint 1 at 0x49fb4: file http_request.c, line 1164. (gdb) run -X -d /usr/local/apache Starting program: /usr/local/apache/src/httpd -X -d /usr/local/apache [ at this point I make a request from another window ] Breakpoint 1, ap_process_request (r=0x95250) at http_request.c:1164 1164 if (ap_extended_status) (gdb) s 1165 ap_time_process_request(r->connection->child_num,... (gdb) n 1167 process_request_internal(r); (gdb) s process_request_internal (r=0x95250) at http_request.c:1028 1028 if (!r->proxyreq && r->parsed_uri.path) { (gdb) s 1029 access_status = ap_unescape_url(r->parsed_uri.path); (gdb) n 1030 if (access_status) { (gdb) s 1036 ap_getparents(r->uri); /* OK... (gdb) n 1038 if ((access_status = location_walk(r))) { (gdb) n 1043 if ((access_status = ap_translate_name(r))) { (gdb) n 1048 if (!r->proxyreq) { (gdb) n 1053 if (r->method_number == M_TRACE) { (gdb) n 1062 if (r->proto_num > HTTP_VERSION(1,0) && ap_... (gdb) n 1071 if ((access_status = directory_walk(r))) { (gdb) s directory_walk (r=0x95250) at http_request.c:288 288 core_server_config *sconf = ap_get_module_... (gdb) b ap_send_error_response Breakpoint 2 at 0x47dcc: file http_protocol.c, line 2090. (gdb) c Continuing.
Breakpoint 2, ap_send_error_response (r=0x95250, recursive_error=0) at http_protocol.c:2090 2090 BUFF *fd = r->connection->client; (gdb) where #0 ap_send_error_response (r=0x95250, recursive_error=0) at http_protocol.c:2090 #1 0x49b10 in ap_die (type=403, r=0x95250) at http_request.c:989 #2 0x49b60 in decl_die (status=403, phase=0x62db8 "check access", r=0x95250) at http_request.c:1000 #3 0x49f68 in process_request_internal (r=0x95250) at http_request.c:1141 #4 0x49fe0 in ap_process_request (r=0x95250) at http_request.c:1167 #5 0x439d8 in child_main (child_num_arg=550608) at http_main.c:3826 #6 0x43b5c in make_child (s=0x7c3e8, slot=0, now=907958743) at http_main.c:3898 #7 0x43ca8 in startup_children (number_to_start=6) at http_main.c:3972 #8 0x44260 in standalone_main (argc=392552, argv=0x75800) at http_main.c:4250 #9 0x449fc in main (argc=4, argv=0xefffee8c) at http_main.c:4534 (gdb) s 2091 int status = r->status; (gdb) p status $1 = 403 (gdb) There are a few things to note about the above example: the " gdb httpd " command does not include any command-line options for httpd: those are provided when the " run " command is done within gdb; I set a breakpoint before starting the run so that execution would stop at the top of ap_process_request(); the " s " command steps through the code and into called procedures, whereas the " n " (next) command steps through the code but not into called procedures. additional breakpoints can be set with the " b " command, and the run continued with the " c " command. use the " where " command (a.k.a.
" bt ") to see a stack backtrace that shows the order of called procedures and their parameter values. use the " p " command to print the value of a variable. A file in the the root directory called .gdbinit provides useful macros for printing out various internal structures of httpd like tables ( dump_table ), brigades ( dump_brigade ) and filter chains ( dump_filters ). If you are debugging a repeatable crash, simply run gdb as above and make the request -- gdb should capture the crash and provide a prompt where it occurs.
If you are debugging an apparent infinite loop, simply run gdb as above and type a Control-C -- gdb will interrupt the process and provide a prompt where it was stopped. If you are debugging a system crash and you have a core file from the crash, then do the following: % gdb httpd -c core (gdb) where and it will (hopefully) print a stack backtrace of where the core dump occurred during processing. Getting a live backtrace on unix ¶ A backtrace will let you know the hierarchy of procedures that were called to get to a particular point in the process.
On some platforms you can get a live backtrace of any process. For SVR4-based variants of Unix, the pstack command for proc can be used to display a a live backtrace. For example, on Solaris it looks like % /usr/proc/bin/pstack 10623 10623: httpd -d /usr/local/apache ef5b68d8 poll (efffcd08, 0, 3e8) ef5d21e0 select (0, ef612c28, 0, 0, 3e8, efffcd08) + 288 00042574 wait_or_timeout (0, 75000, 75000, 7c3e8, 60f40, 52c00) + 78 00044310 standalone_main (5fd68, 75800, 75c00, 75000, 2, 64) + 240 000449f4 main (3, efffeee4, efffeef4, 75fe4, 1, 0) + 374 000162fc _start (0, 0, 0, 0, 0, 0) + 5c Another technique is to use gdb to attach to the running process and then using "thread apply all bt" to print the backtrace, as in: sudo gdb /path/to/bin/httpd 10623 --batch --eval-command "where" --eval-command "thread apply all bt" --eval-command "detach" --eval-command "quit" | tee gdb-output.txt Or interactively: % gdb httpd 10623 GDB is free software and you are welcome to distribute copies of it under certain conditions; type "show copying" to see the conditions.
There is absolutely no warranty for GDB; type "show warranty" for details. GDB 4.16.gnat.1.13 (sparc-sun-solaris2.5), Copyright 1996 Free Software Foundation, Inc... /usr/local/apache/src/10623: No such file or directory. Attaching to program `/usr/local/apache/src/httpd', process 10623 Reading symbols from /usr/lib/libsocket.so.1...done. ... 0xef5b68d8 in () (gdb) where #0 0xef5b68d8 in () #1 0xef5d21e8 in select () #2 0x4257c in wait_or_timeout (status=0x0) at http_main.c:2357 #3 0x44318 in standalone_main (argc=392552, argv=0x75800) at... #4 0x449fc in main (argc=3, argv=0xefffeee4) at http_main.c:4534 (gdb) thread apply all bt (gdb) detach (gdb> quit Getting a live backtrace on Windows ¶ Unzip the -symbols.zip files (obtained from the Apache download site) in the root Apache2 direc
The rest of this page is on the live site.