For the purposes of publishing metrics to AWS CloudWatch I would like to get information of the number of occurrences of some keyword (Eg., Error, Exception) within the last minute (from current system time) in my application logs.
Following are the commands that I have tried so far based on the answers from a related thread ( Filter log file entries based on date range):
awk -vDate=`date -d'now-1 minutes' +["%Y-%m-%d %H:%M:%S"` '($1 FS $2) > Date {print $3}' application.log | grep "ERROR" | uniq -c
awk -vDate=`date -d'now-1 minutes' +["%Y-%m-%d %H:%M:%S"` '{if ($1 > Date) {print $3}}' application.log | grep "ERROR" | uniq -c
awk -vDate=`date -d'now-1 minutes' +["%Y-%m-%d %H:%M:%S"` '{if ($1 == $Date) {print $3}}' application.log | grep "ERROR" | uniq -c
But I get an error like this when I try this:
awk: cmd. line:1: 13:06:17
awk: cmd. line:1: ^ syntax error
Following is the format of my log file:
2016-02-05 12:10:48,761 [INFO] from org.xxx
2016-02-05 12:10:48,761 [INFO] from org.xxx
2016-02-05 12:10:48,763 [INFO] from org.xxx
2016-02-05 12:10:48,763 [INFO] from org.xxx
2016-02-05 12:10:48,763 [ERROR] from org.xxx
2016-02-05 12:10:48,763 [INFO] from org.xxx
2016-02-05 12:10:48,764 [INFO] ffrom org.xxx
2016-02-05 12:10:48,773 [WARN] from org.xxx
2016-02-05 12:10:48,777 [INFO] from org.xxx
2016-02-05 12:10:48,778 [INFO] from org.xxx
Stuck on this for quite a while. Thanks for the help!
You're using deprecated backticks and so not quoting the date output. Do this instead:
awk -vDate="$(date -d'now-1 minutes' +"%Y-%m-%d %H:%M:%S")" '($1 FS $2) > Date { if ($3~/ERROR/) print $3}' file
Note that you don't need to pipe to grep and by not having a space between -v and Date your script is gawk-specific and if it's gawk-specific then you don't need that external call to date since gawk has it's own builtin time functions (hint: BEGIN{Date=strftime("%Y-%m-%d %H:%M:%S",systime()-60)}).
You also don't need uniq -c but without seeing your real input and expected output (doing a uniq -c given that input wouldn't make any sense vs wc -l) I'm not going to guess any more.
Oh what the heck, here's the whole script in gawk:
$ cat tst.awk
BEGIN {
#date = strftime("%Y-%m-%d %H:%M:%S",systime()-60)
date = "2016-02-05 12:10:48"
}
($1" "$2) > date {
if ($3 ~ /ERROR/) {
cnt[$3]++
}
}
END {
for (err in cnt) {
print err, cnt[err]
}
}
$
$ awk -f tst.awk file
[ERROR] 1
I assume in reality you have various flavors of "ERROR" and that's why you want the count of each. Just uncomment the strftime line and delete the hard-coded timestamp line to run on your real data.