Downloading Raw Log File

[GdP]

Hello,

 

I use a program to analyze my log files on my own computer (Web Log Storming - from dataland: http://www.datalandsoftware.com) and for that I need to download my raw logs every day.

I have this pretty well automated for several of my sites (not hosted by TC) using a .bat file to download them. The problem with TC is that my raw logs are only available at the end of the month. The logs from the current day is accessible through CPanel, but I don't see how I can do this through a .bat file (using FTP).

 

Any hints??

 

Gert

[annie]

I'd love to do this also. But with a cron job. Saving consecutive files in my FTP folders.

[annie]

Like I said, I'd like to do this. But without knowing where the log files reside, I don't see how I could get it done. I would also be interested in a weekly cron job as an alternative.

 

Also, it's a question of permissions. Maybe I'd need to run grep to separate out hits from my domain?

 

Lots of questions.

[annie]

Hmmm, I'm starting to wonder if I can do this...

 

Let's see, gzip the contents of http://www.domain.com:2082/getaccesslog/

 

And leave them somewhere in my area.

 

Now the question is when the logs roll over here.

 

And also a question of whether or not I can make the cron job name the files different names each day or week?

[annie]

I've sort of figured out this one. I tried it, and it almost worked...

 

I created a cron job that went like this:

gzip -c /home/username/getaccesslog > /home/username/logs/log01monday.gz

 

Theory is that I could create a cron job to run each day, naming the output files differently. That would create files for each day of the week, overwriting the one week old files?

 

The cron job made the file, but I don't think I have the right path for the log file.

 

So, any ideas on making this one actually work?

[annie]

I think I solved my own problem again, thanks to doing a search on the cpanel forums. Here's how to do it:

 

gzip -c /usr/local/apache/domlogs/domain.com > /home/username/logs/log01monday.gz

 

Then just make as many differently named ones as you need.

 

Just make sure you have the disk space. Those files can get huge, depending on how large your log files are.

 

And I'd delete them on a regular basis, just downloading what I need. This is meant as a backup, if the server suddenly barfs and starts from zero again (plenty of experience on another host that this does happen frequently towards the end of the month. Guess I'm not that trusting of servers right now).

 

Um, and I also created the logs directory, so don't forget to do that, or choose another path.

[annie]

It's also possible to grep a log file, or create downloadable files for each day. That's useful if you've got a lot of traffic. One day alone could account for lots of megabytes of uncompressed log file.

 

I'm sure there's an easier way to do this, but I've found a way that's possible without being too accomplished:

 

grep '13/May' /usr/local/apache/domlogs/domain.com | gzip -9 >> /home/username/logs/may13.gz

 

Substitute domain.com for your domain name and ending. And username for your user name.

 

Then you'd have to make a new cron job for EACH day you want to extract, or each grep job you want to make. Remember to delete those cron jobs after they've run.

[DarqFlare]

You could always setup a cron for a PHP script everyday, and then have that script execute the commands, and have it dynamically make the GZ filename.

 

I think the PHP command is exec();

 

Not entirely sure (Raul can verify), but I think it'd be like this:

 

><?

// filename and date stuff here

exec("grep '${DayThenMonthString}' /usr/local/apache/domlogs/domain.com | gzip -9 >> /home/username/logs/${FilenameToSaveAsString}.gz");

?>

[annie]

Thanks, that looks promising. I know just enough perl to see that we're heading in the right direction here! But I probably would need a template file to get me started.

[annie]

I just found something on shifting the timestamp to the day before, in perl:

http://logsoft.com/perltips/122297.html

 

Is it similar in PHP?

 

Edit: Found one for PHP as well:

http://no2.php.net/date

[annie]

I finally figured out how to return yesterday's date in the same format as the log files:

 

<?

$yesterday = mktime(0,0,0,date("m"),date("d")-1,date("Y"));

 

$yesterday = strftime("%d/%b",$yesterday);

echo "$yesterday";

?>

[annie]

I think I cracked it:

 

-----

 

<?

 

$yesterday = mktime(0,0,0,date("m"),date("d")-1,date("Y"));

$yesterday = strftime("%d/%b",$yesterday);

 

$thatday = mktime(0,0,0,date("m"),date("d")-1,date("Y"));

$thatday = strftime("%d%b",$thatday);

 

exec("grep '$yesterday' /usr/local/apache/domlogs/domain.com | gzip -9 >> /home/username/logs/$thatday.gz");

 

?>

 

----

 

Not bad for a non-programmer, eh?

 

And PLEASE tell me if I've really done something unsafe!

 

The reason I had to break it up into yesterday and thatday is because if you put / into file names, the machine gets confused. You could of course append the year as well. I just didn't bother.

[annie]

I've got several versions of that script and other useful log backup scripts on my website. Click on the image:

 

:heart