A lot of new webmasters are told that there's gold in their log files. This is a great piece of advice, but it is a little cryptic. Where is this gold and what does it look like? Unlike real gold, ours looks a little different for everyone and more often than not it's lightly dusted throughout your often large log files. This means there's no quick and easy 5 step solution to finding what you want. You'll have to work for it, but the work you put into log analysis will provide you with a roadmap of how people are using your site and most importantly, insight on improving your site.
There are a few different types of log files that different web servers will generate. They all contain important information but for this article we'll focus on the access log file generated by the apache web server. We'll get to the other log files another day. The apache access log file will contain things like the IP of the person requesting a page, what page they requested, what browser they are using, what page they came from if they followed a link in to your site, etc... This is very valubale information, but all of this information is collected for every single request made to your web server. For a moderately busy site this can mean tens of thousands of requests per day. This makes it impossible to analyze your log files manually. We'll need some tools to dig for our gold.
The types of tools webmasters usually look for are tools that will help them write HTML, manage their files and work with pictures. The kinds of tools we're going to discuss don't have well known names like Photoshop, Dreamweaver or Frontpage, but are invaluable to make your site reach its fullest potential.
The tools you will need will range from complex automated log analyzers to simple basic Unix tools like tail and grep. We'll start today with a log analyzer since everyone needs one and offers a wealth of information. There are many quality log analyzers out there, each with their different take on log analysis. I personally like awstats, I've tried some others and I've found awstats to cater to my needs best. Your mileage may vary.