This article explain how to forward Apache access logs to Apsolab-Server. There are 2 options for sending access logs to Apsolab-Server. Both options require syslog-ng.

  • Configure syslog-ng to forward access logs to Apsolab-Server in raw format.
  • Configure syslog-ng to parse access logs and format log message for Apsolab-Server. (require syslog-ng 3.0 +)

Option 1: Configure syslog-ng to forward access logs to Apsolab-Server in raw format

This example assume:

  • The access log is in /var/log/httpd/access_log
  • Apsolab-Server host name is apsolab
# syslog-ng.conf
# Declare access_log source file. Note: flags=no-parse
source s_http_access_log {
    file("/var/log/httpd/access_log" flags(no-parse));
};

# Declare template for access_log (remove any line feed) 
# Replace logger name (http-localhost) 
template t_http_access_log{
    template ("<xlog><loggertype>syslog</loggertype><loggername>http-localhost</loggername><category>http</category><timestamp>$UNIXTIME</timestamp><host>$HOST</host><sourceip>$SOURCEIP</sourceip><program>HTTPD</program><pri>$PRI</pri><msg>$MSGONLY</msg></xlog>\n"); 
    template_escape(no);
};

# Declare destination for Apsolab-Server host using previous template 
destination d_http_apsolab {     
    tcp("apsolab" port (33201) template(t_http_access_log)); 
};

# Send access_log to Apsolab-Server 
log {     
    source(s_http_access_log);     
    destination(d_http_apsolab); 
};

Restart httpd service.

Option 2: Configure syslog-ng to parse access logs and format log message for Apsolab-Server. (require syslog-ng 3.0 +)

This example assume the following LogFormat from httpd.conf: “%h %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\” %T %v”

  • The access log is in /var/log/httpd/access_log
  • Apsolab-Server host name is apsolab
# syslog-ng.conf
# Declare access_log source file. Note: flags=no-parse
source s_http_access_log {
    file("/var/log/httpd/access_log" flags(no-parse));
};

# Declare parser for access_log
# Assume log format: "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v" 
parser p_http_access_log {     
    csv-parser(columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",         
        "APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",         
        "APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
        "APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
    flags(escape-double-char,strip-whitespace)
    delimiters(" ")         
    quote-pairs('""[]')     
    ); 
};  

# Declare template for access_log (remove any line feed in template declaration) 
# Replace logger name (http-localhost) 
template t_http_access_log {
    template ("<xlog><loggertype>syslog</loggertype><loggername>http-localhost</loggername><category>http</category><timestamp>$UNIXTIME</timestamp><host>$HOST</host><sourceip>$SOURCEIP</sourceip><program>HTTPD</program><pri>$PRI</pri><msg>Client IP: ${APACHE.CLIENT_IP}\nIdent Name: ${APACHE.IDENT_NAME}\nUser Name: ${APACHE.USER_NAME}\n
         Timestamp: ${APACHE.TIMESTAMP}\nRequest URL: ${APACHE.REQUEST_URL}\nStatus: ${APACHE.REQUEST_STATUS}\n
         Length: ${APACHE.CONTENT_LENGTH}\nRefer:${APACHE.REFERER}\nAgent:${APACHE.USER_AGENT}\nProcess Time:${APACHE.PROCESS_TIME}\n
         Server Name:${APACHE.SERVER_NAME}</msg></xlog>\n");
    template_escape(no); 
};

# Declare destination for Apsolab-Server host using previous template 
destination d_http_apsolab {     
    tcp("apsolab" port (33201) template(t_http_access_log)); 
};

# Send access_log using parser to Apsolab-Server 
log {     
    source(s_http_access_log);     
    parser(p_http_access_log);
    destination(d_http_apsolab); 
};

Restart httpd service.