Apache log one-liners using tail, awk, sort, etc.

Good bunch of samples with other examples found at: https://blog.nexcess.net/2011/01/21/one-liners-for-apache-log-files/

# top 20 URLs from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLS excluding POST data from the last 5000 hits
tail -5000 ./transfer.log | awk -F"[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk -F"[ ?]" '{freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 IPs from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$1]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLs requested from a certain ip from the last 5000 hits
IP=1.2.3.4; tail -5000 ./transfer.log | grep $IP | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
IP=1.2.3.4; tail -5000 ./transfer.log | awk -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 URLS requested from a certain ip excluding, excluding POST data, from the last 5000 hits
IP=1.2.3.4; tail -5000 ./transfer.log | fgrep $IP | awk -F "[ ?]" '{print $7}' | sort | uniq -c | sort -rn | head -20
IP=1.2.3.4; tail -5000 ./transfer.log | awk -F"[ ?]" -v ip=$IP ' $1 ~ ip {freq[$7]++} END {for (x in freq) {print freq[x], x}}' | sort -rn | head -20
 
# top 20 referrers from the last 5000 hits
tail -5000 ./transfer.log | awk '{print $11}' | tr -d '"' | sort | uniq -c | sort -rn | head -20
tail -5000 ./transfer.log | awk '{freq[$11]++} END {for (x in freq) {print freq[x], x}}' | tr -d '"' | sort -rn | head -20
 
# top 20 user agents from the last 5000 hits
tail -5000 ./transfer.log | cut -d  -f12- | sort | uniq -c | sort -rn | head -20
 
# sum of data (in MB) transferred in the last 5000 hits
tail -5000 ./transfer.log | awk '{sum+=$10} END {print sum/1048576}'

Mac OSX 10.6 Shell Script to Configure, Launch, and Shutdown Screen Sharing

Apple’s MacOS has a tidy built in remote access client and service generically known as “Screen Sharing”. Yay! This works great for headless machines, testing servers, and annoying your spouse. The paranoid and performance conscious user may not want these services continuously running. Rogue services hanging out, listening, and waiting for some punk to probe them in an Internet dark alley. This cobbled together shell script might be for you!

This shell script has four simple commands (I use sudo because I really should):

  • # sudo ./share-screen.sh start
  • # sudo ./share-screen.sh stop
  • # sudo ./share-screen.sh allow {username}
  • # sudo ./share-screen.sh deny {username}

I use this in terminal. Make sure you put a copy of this script on the target machine and make it executable. First I make sure that Remote Login is active on the target computer. It’s under Sharing in System Preferences. Then I login remotely using ssh. I fire up the script, activate Screen Sharing, do my business, shutdown Screen Sharing, log off, and go home.

Of course, no warranty, no guaranty. Good Luck!

#!/bin/bash

if [ $# == 0 ]; then
	echo  "Commands: start, stop, allow <username>, deny <username>"
	exit
fi

case "$1" in
	start)
		echo 'Starting Remote Access'
		# Activate Apple Remote Access with current settings
		sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate
		;;
	stop)
		echo 'Stopping Remote Access'
		# Deactivate Apple Remote Access
		sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -deactivate -stop
		;;
	allow)
		if [ -z "$2" ]; then
			echo  "ERROR: Provide a valid user"
			exit
		fi
		sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -allowAccessFor -specifiedUsers
		sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -access -on -privs -all -users ${2}
		;;
	deny)
		if [ -z "$2" ]; then
			echo  "ERROR: Provide a valid user"
			exit
		fi  
		sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -allowAccessFor -specifiedUsers
		sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -configure -access -off -privs -none -users ${2}  
		;;  
esac

I almost forgot. Credit where credit is due! Inspiration from these articles and resources:

http://www.macosxhints.com/article.php?story=20080318190503111

http://rentzsch.tumblr.com/post/515009165/starting-vnc-remotely-via-kickstart

Tommy’s diggall.sh revised

Think you know the linux digg command? Ever try to do some simple network discovery with it? This script by Tommy Johnson does some nice condensed discovery. Just pass a domain for the argument.

./diggall.sh example.com

That is all!

#!/bin/bash
echo "MX points to:"
dig MX $1 +short | sort -n
echo ""
echo "A points to:"
dig A $1 +short
echo ""
echo "www points to:"
dig A www.$1 +short
echo ""
echo "mail points to:"
dig A mail.$1 +short
echo ""
echo "SOA Nameserver declared:"
echo `dig SOA $1 +short | awk '{print $1}'`
echo ""
#echo "oldmail points to:"
#dig A oldmail.$1 +short
domain1=( $(echo $1 | awk -F '.' '{print $1}'))
domain2=( $(echo $1 | awk -F '.' '{print $2}'))
domain3=( $(echo $1 | awk -F '.' '{print $3}'))
domain2=( $(echo $domain2.$domain3 | sed 's/[:.:]$//') )
first=( $(dig NS +short $domain2 | sed 's/[:.:]$//') )
second=( $(dig +short NS $domain2 @$first | sed 's/[:.:]$//') )
#echo "Authoritative nameserver for TLD is: "$second
#echo '+===============================================+'
echo "Name Servers as reported by registrar:"
echo `dig NS $domain1.$domain2 @$second | grep "AUTHORITY SECTION" -A3 | grep -v AUTHORITY | awk '{print $5}'`
echo "IPs declared by registrar:"
echo `dig NS $domain1.$domain2 @$second | grep "ADDITIONAL SECTION" -A3 | grep -v ADDITIONAL | awk '{print $5}'`
#echo '+===============================================+'
echo ""
echo "Name Servers in zone file are:"
dig NS $1 +short
dig NS $1 | grep "ADDITIONAL SECTION" -A3 | grep -v ADDITIONAL | awk '{print $5}'
echo "txt record is:"
dig txt $1 +short