"securing" Mt.cgi -- What To Do?

[brendan]

Hi all,

 

I need advice about my MT installation, and how to "secure" mt.cgi against possible attacks.

 

Honestly, I am not sure how mt.cgi could even be attacked, since it's supposed to be executable only by people with access to my blog, but last night (wee hours of this morning, actually) it was using between 46% and 84% of server resources, and as a result, my account was suspended.

 

I assume that such resource-hogging by mt.cgi must be the result of an attack of some sort. I certainly wasn't doing anything that would have caused such numbers, and I can't imagine that my fellow bloggers were, either.

 

(Please note that it is specifically mt.cgi, not a comment or trackback script, that was hogging the server. I disabled my comment and trackback scripts in January after being suspended several times as a result of comment-spam floods, and I have been using an external comment provider (HaloScan) ever since. So unless spammers have some way of using mt.cgi directly, this is not a comment-spam issue.)

 

TCH has told me that I must come up with "a plan to secure [my] MT installation" before my account will be unsuspended. The problem is, I don't know what I am "securing" it against! I feel like I'm flying blind. Comment-spam I understood, but this is bizarre. An mt.cgi attack? Huh?

 

So anyway, I have several questions. First of all, what could possibly cause mt.cgi to reach such high server-resource levels? Does my suspicion that it was an "attack" of some sort make any sense? And what can I do to prevent it from recurring?

 

I believe I am using either MT 3.15 or 3.16 (I can't recall which, and I can't check because my account is offline!); I know it is definitely 3.1x, the only question is what the "x" is.

 

I have a MySQL back-end.

 

I'm not sure what other information is relevant, but ask and ye shall receive.

 

I would very much appreciate any advice that y'all may have.

 

(And yes, I realize that the account-suspension aspect of my problem must be dealt with through the help desk, and that is already happening. I am just explaining that to provide context. What I am looking for here is help with "securing" my MT installation.)

[chroniker]

As an MT user I would like to know the answer to this also and TCHDavid will most likely be the one to answer.

 

I set up my site with MT 3.15 back in Jan and I think that 3.16 was released in March or April and 3.17 was released June 2.

[TCH-Rick]

I do not think there was any sort of "attack" on mt.cgi in this case. I looked through the logs of the account for today and all of the access to mt.cgi was from your IP. I copied the log entries to a file in your root folder that you can look at.

 

Part of the problem is due to the way MT rebuilds the entire site when an entry is made. It doesn't matter whether the post is two lines or two hundred, the whole site is rebuilt. Some of the changes to MT after the major problems with comment spammers last year was to prevent rebuilding every time a comment was added even if the page didn't change.

 

As a blog gets bigger, the load on the server goes up every time that another entry is added.

[brendan]

I do not think there was any sort of "attack" on mt.cgi in this case. I looked through the logs of the account for today and all of the access to mt.cgi was from your IP. I copied the log entries to a file in your root folder that you can look at.

 

Thanks! I will take a look at those logs.

 

Part of the problem is due to the way MT rebuilds the entire site when an entry is made. It doesn't matter whether the post is two lines or two hundred, the whole site is rebuilt. Some of the changes to MT after the major problems with comment spammers last year was to prevent rebuilding every time a comment was added even if the page didn't change.

 

As a blog gets bigger, the load on the server goes up every time that another entry is added.

<{POST_SNAPBACK}>

 

I'm aware that MT rebuilding is a resource-intensive process (though less so under MT 3.x, as you say), but surely a normal rebuild should never take up 85% of the server's resources?! If it did, this would have happened to me many times before; I blog all the time! There must have been some other contributing circumstance... hopefully the logs will help me figure out what.

 

Also, it's actually not true that the "entire site" is rebuilt every time a new post goes up. For example, publishing a new post does not rebuild every single individual entry in the blog -- that would take a good long time, and on those rare occasions when I do a manual rebuild of "all individual entries" (which I did not do last night, FWIW) it takes MUCH longer than publishing a single post does.

 

If I'm not mistaken, publishing a post rebuilds the following things:

1) all index pages;

2) the individual entry for that post, and for the post immediately preceding it and (if applicable) the one immediately following it;

3) any relevant date-based archives (e.g. monthly, daily) for the time period of the post and the time period immediately preceding and following; and

4) the category archive for that particular post.

 

In my case, the date-based and category archives are generated dynamically, so they are never "rebuilt" by MT. Thus, the only things that are automatically rebuilt when I publish a post are all my indexes and three individual archive pages: the one for the post I'm publishing and the ones before it and after it.

 

Perhaps I can reduce the load slightly by cutting down on the number of index pages I have set to automatically rebuild... but as I said, surely 85% cannot be normal, even for someone with a large blog and multiple index pages. If it was, MovableType would be banned by every host on the Web!

Edited June 15, 2005 by brendan

[TCH-Rick]

I would suggest upgrading to the latest version as well. It is not true that all versions of MT 3 use less resources. I have not kept up with the changes that have been made as I stopped using MT several months back when one of my sites was suspended for comment spam causing server problems while I was not working.

 

MT has been banned on some hosts and if were not as popular and widely used as it is I suspect it would be banned on many more. Even on a small blog I have seen it use a lot more resources that other software. I ran a number of tests when I was looking at WordPress but that was version 2.63. Back in December I read an interview as SixApart was recommending certain changes while they looked at fixing bugs that were exploited by the comment spammers. They mentioned that they were concerned that many hosts would ban the script if a fix was not found.

[TweezerMan]

Honestly, I am not sure how mt.cgi could even be attacked, since it's supposed to be executable only by people with access to my blog, but last night (wee hours of this morning, actually) it was using between 46% and 84% of server resources, and as a result, my account was suspended.

 

I assume that such resource-hogging by mt.cgi must be the result of an attack of some sort. I certainly wasn't doing anything that would have caused such numbers, and I can't imagine that my fellow bloggers were, either.

<{POST_SNAPBACK}>

I don't believe that is necessarily a valid assumption. I'd suggest examining your server's access logs around the time your account was suspended. (If you don't know the exact time, look at the logs for yesterday and today.) See what IP addresses were accessing mt.cgi, and what mt.cgi activity was triggered by those accesses.

 

(I see that TCH-Rick beat me to this point while I was writing this post! )

 

So anyway, I have several questions. First of all, what could possibly cause mt.cgi to reach such high server-resource levels? Does my suspicion that it was an "attack" of some sort make any sense? And what can I do to prevent it from recurring?

<{POST_SNAPBACK}>

The only thing I'm aware of that would cause mt.cgi to use high levels of server resources is rebuilding your weblog. I doubt that spammers compromised your MT installation and then decided to rebuild your weblog.

 

Your weblog's home page says you have almost 8,000 entries in your weblog. Performing a full rebuild of that many entries could put a high enough load on the server to get TCH's attention. Other factors that can add to the server load during a rebuild include ineffeicient template design and the use of plugins that are not server-friendly.

 

Your weblog also indicates there 5 other MT weblogs running on your MT installation. Depending on how these weblogs are set up, it's possible that a rebuild on one of these other weblogs could have caused the server load issues (but I don't think that is likely).

 

I would very much appreciate any advice that y'all may have.

<{POST_SNAPBACK}>

1) Look at the logs TCH-Rick provided to you, to see what really happened (rather than just guess or assume).

 

2) Assuming that it's a problem with rebuilding, the template design and plugin usage for the weblog that the logs indicate is the problem should be examined, to see what improvements can be made to increase how efficiently they are rebuilt.

[brendan]

All I'm saying is, this isn't normal, because if it was, my account would have been suspended a long time ago. 85% server usage is not normal behavior, even for MovableType.

 

I'm looking at that log you uploaded into my root, and something is seriously wrong... definitely not normal MT behavior. I'll PM you about it...

Edited June 15, 2005 by brendan

[brendan]

Okay, I was going to keep the log discussion to the PM, but now that David has joined in ... basically, what the logs are telling me is that there is a "GET" command to mt.cgi emanating from my IP address (or rather, the IP address of my entire apartment complex -- but I doubt I'm being hacked by a neighbor, though I suppose it's possible!) once every minute!!

 

As for a full rebuild... I have not done a full rebuild in recent days.

 

What is the time zone of the timestamps on the log? In other words, what does "[15/Jun/2005:18:15:40 -0400]" translate to? Is 18:15 GMT or EDT or what?

Edited June 15, 2005 by brendan

[TweezerMan]

I believe the timestamps in the server log are in the server's local time zone. In your case, the "-0400" indicates the timestamp is EDT.

[TCH-Rick]

I got your PM and before answering wanted to test on my account. I am not sure what is making the requests for mt.cgi from that IP address. I ran mt.cgi on my account and left it open to see if there was some sort of auto refresh that would give log entries like that. In looking at the log there are similar entries throughout the month at the one minute intervals.

 

To test whether it is coming from your computer I would suggest turning off the computer for 15 minutes or so and then checking the logs. It might also be helpful to send the log to support at SixApart.

[brendan]

Thanks, David.

 

My account was suspended at around 4:00 AM EDT this morning. Looking at the logs at that time, the only unusual behavior is that bizarre pattern of a "GET" command emanating from my apartment complex's IP address once a minute:

 

67.133.222.170 - - [15/Jun/2005:03:30:03 -0400]"POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 17297 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:31:04 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:31:55 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:33:07 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:34:05 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:34:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:35:57 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:36:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:37:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:38:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:39:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:40:01 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 8763 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:40:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:40:59 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 17687 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:41:57 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:42:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:43:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:44:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:45:09 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 200 13720 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:46:02 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:46:26 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=list&_type=template&blog_id=13 HTTP/1.1" 200 12785 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:46:26 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=list&_type=template&blog_id=13 HTTP/1.1" 200 14244 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:46:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:47:47 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 302 5 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:47:52 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=view&_type=entry&blog_id=13&id=18222&saved_added=1 HTTP/1.1" 200 20653 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:47:57 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:48:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:49:56 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:50:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 1662 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:51:23 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 152 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=view&_type=entry&blog_id=13&id=18222&saved_added=1" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:57:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 140 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:58:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 140 "-" "-"

67.133.222.170 - - [15/Jun/2005:03:59:18 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 200 152 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:59:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 200 140 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:00:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:01:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:02:14 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 302 352 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:04:02:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:03:53 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:04:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:05:58 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:06:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:07:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:08:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:09:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:10:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:11:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:12:58 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:13:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:14:56 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

67.133.222.170 - - [15/Jun/2005:04:15:54 -0400] "GET /cgi-bin/mt/mt.cgi HTTP/1.0" 302 309 "-" "-"

 

Can you decipher anything from that pattern? What do the numbers (first 200 1662, then 200 140, then 302 309) mean?

 

Also, can you do me a favor and upload another copy of that log file into my account sometime after 8:00 PM EDT (i.e., more than 10 minutes from now)? I am shutting down the computers in my apartment one-by-one, to try and figure out whether one of them is sending out a ping to my MT installation, unbeknownst to me.

Edited June 15, 2005 by brendan

[TweezerMan]

The first number in those pairs you list is the HTTP status code sent back to the browser or script requesting the page. A "200" status code is "OK" - the page requested was found on the server and served to the client normally. A "302" status code means "Found". I believe this code indicates that the page has not changed, or that there was a redirection involved in the page request, and the page to be redirected to was "found". If you look at the first set of log entries you posted, the server is currently returning a status code of "500" (Server Error) - because mt.cgi is disabled.

 

The second number in those pairs (the "1662", "140", "309") is the number of bytes the server sent back to the browser or script in response to the page request.

[TweezerMan]

A "302" status code means "Found".  I believe this code indicates that the page has not changed, or that there was a redirection involved in the page request, and the page to be redirected to was "found".

<{POST_SNAPBACK}>

Just had a "duh" moment - the "302" status codes were being sent by the server after your account was suspended, because everything was being redirected to the "Account Suspended" page (which was of course, "found").

[brendan]

I just had a "duh" moment of my own, with regard to the source of these "GET" codes.

 

I have a program on my PowerBook called The Informant which checks the status of several websites once a minute, to make sure that both my Internet connection and my crucial website pages are up and running.

 

I will stop it from checking mt.cgi... but 1,662 bytes once a minute cannot possibly be overloading the server, and I have been running that program for months with no problems. So the culprit must be something else.

 

So let's look at the log from 3:00 - 4:30 AM last night, with those "GET" commands excluded:

 

67.133.222.170 - - [15/Jun/2005:03:27:41 -0400]"GET /mt-static/mt.js HTTP/1.1" 200 4631 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:41 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 200 13720 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:41 -0400] "GET /mt-static/images/topnav-logo.gif HTTP/1.1" 200 1722 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/styles.css HTTP/1.1" 200 18025 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/topnav-bg.gif HTTP/1.1" 200 389 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-new-entry.gif HTTP/1.1" 200 91 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-entries.gif HTTP/1.1" 200 336 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-upload.gif HTTP/1.1" 200 117 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-comments.gif HTTP/1.1" 200 91 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-trackbacks.gif HTTP/1.1" 200 102 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-templates.gif HTTP/1.1" 200 99 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-categories.gif HTTP/1.1" 200 101 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-notifications.gif HTTP/1.1" 200 88 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-commenters.gif HTTP/1.1" 200 91 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-config.gif HTTP/1.1" 200 319 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-search.gif HTTP/1.1" 200 106 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-import.gif HTTP/1.1" 200 94 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-rebuild.gif HTTP/1.1" 200 95 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:42 -0400] "GET /mt-static/images/nav-view-site.gif HTTP/1.1" 200 91 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:27:45 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=view&_type=entry&blog_id=13 HTTP/1.1" 200 16267 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:29:58 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 7974 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=view&_type=entry&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:30:03 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 17297 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi"'>http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:40:01 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 8763 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:40:59 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 17687 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:45:09 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 200 13720 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:46:26 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=list&_type=template&blog_id=13 HTTP/1.1" 200 12785 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:46:26 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=list&_type=template&blog_id=13 HTTP/1.1" 200 14244 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:47:47 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 302 5 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:47:52 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=view&_type=entry&blog_id=13&id=18222&saved_added=1 HTTP/1.1" 200 20653 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:51:23 -0400] "POST /cgi-bin/mt/mt.cgi HTTP/1.1" 200 152 "http://www.brendanloy.com/cgi-bin/mt/mt.cgi?__mode=view&_type=entry&blog_id=13&id=18222&saved_added=1" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:03:59:18 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 200 152 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

67.133.222.170 - - [15/Jun/2005:04:02:14 -0400] "GET /cgi-bin/mt/mt.cgi?__mode=menu&blog_id=13 HTTP/1.1" 302 352 "-" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en) AppleWebKit/412 (KHTML, like Gecko) Safari/412"

 

Those all look like perfectly normal commands. At 3:27, I loaded the main editing page for my blog, which caused all the /mt-static/ files to load as well. That's the first 20 entries on the log. Total bytes: 56,575. Nothing extraordinary.

 

At 3:29/3:30, I posted to the blog. Two entries on the log, total bytes: 25,271.

 

At 3:40, I posted to the blog again. Two more entries on the log, total bytes: 26,450.

 

At 3:45, I loaded mt.cgi again. One entry on the log, 13,720 bytes.

 

At 3:46, I loaded mt.cgi twice (I think this was because I was having trouble with my Internet connection). Two entries on the log, 27,029 bytes.

 

At 3:47, I made a failed attempt to post to the blog. The attempt used only 5 bytes. By this time, the server slowdown had begun. (I filed a help ticket at 3:50, ticket # BNA-69338, complaining that "My PHP photo galleries and my blog CGI scripts have been excruciatingly slow." That was 12 or 13 minutes before my site was suspended for supposedly *causing* the slowdown that I was experiencing.)

 

Five seconds later, I loaded mt.cgi again. One entry on the log, 20,653 bytes.

 

After that, all of my attempts to either load mt.cgi or post to the blog failed, because of the server slowdown. Attempted post at 3:51... 152 bytes. Attempted load at 3:59...152 bytes. In both cases, I believe those bytes came from the error message "Got an error: Bad ObjectDriver config: Connection error: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)," which I documented on my help ticket.

 

That's it. At 4:02, I attempted to load mt.cgi and I saw that my site had been suspended. (You can tell because this was my first "302" code.

 

Total number of bytes in the hour prior to my site being shut down: 170,007. If you add in the 83,660 bytes from my automated software pinging mt.cgi once a minute, the total is 253,667 bytes in an hour.

 

Nothing extraordinary, nothing that would shut down a server.

 

So, after reading the log, I'm afraid I'm more perplexed than ever.

 

Do you have access to logs of the data that led your techs to conclude that my mt.cgi was the resource hogger on the server? It couldn't have been this data. I'd love to look at those logs and see what they saw, to get a better idea of what happened. Either there's something that this log isn't picking up, or there was a mistake and my mt.cgi was not the culprit.

Edited June 16, 2005 by brendan

[brendan]

Rick, forgive me for duplicating my PM, but I'm not sure which is the most efficient.

 

My question: What exactly *is* the log that you made available to me? I know that it doesn't show my entire website, because I was also accessing my /gallery/ directory during that time, and the log didn't show that.

 

Is the log limited to my /cgi-bin/mt/ directory? If so, perhaps we're looking at the *wrong* mt.cgi. I have two other installations, /cgi-bin/mt3/ and /cgi-bin/mt_berkeley/. Bill H. said he was unsure *which* mt.cgi caused the server load early this morning. Perhaps it was one of the others (in which case this must definitely have been some kind of attack, since I never use those installations anymore). Do you have logs for those directories?

[TweezerMan]

I just had a "duh" moment of my own, with regard to the source of these "GET" codes.

 

I have a program on my PowerBook called The Informant which checks the status of several websites once a minute, to make sure that both my Internet connection and my crucial website pages are up and running.

 

I will stop it from checking mt.cgi... but 1,662 bytes once a minute cannot possibly be overloading the server, and I have been running that program for months with no problems. So the culprit must be something else.

<{POST_SNAPBACK}>

It would have been nice if your Informant program included a User-Agent string when it made its requests. The User-Agent would show in the logs, and you would have known right away what was hitting your site every minute.

 

I would not recommend checking a cgi script every minute to see if my site was up. Most CGI scripts are written in perl (including MT's), and they invoke a new perl process every time one of them scripts is called. Perl processes are fairly expensive in terms of server resources (CPU and memory usage).

 

Requesting a page every minute uses up quite a bit of bandwidth over time. Your 1,662 byte mt.cgi file, requested every minute for an entire month, uses over 70MB of bandwidth. I would probably 1) check less often, like every 5 or 10 minutes if I really need to know that often, and 2) check an extremely small file on the server created specifically for this purpose. The file could just have this text in it to be a valid HTML file:

><html><body></body></html>

26 bytes instead of over 1,600, which would consume a little over 1MB of bandwidth if you checked this file once a minute for a whole month. Even less if you check less frequently.

 

Total number of bytes in the hour prior to my site being shut down: 170,007. If you add in the 83,660 bytes from my automated software pinging mt.cgi once a minute, the total is 253,667 bytes in an hour.

 

Nothing extraordinary, nothing that would shut down a server.

 

So, after reading the log, I'm afraid I'm more perplexed than ever.

 

Do you have access to logs of the data that led your techs to conclude that my mt.cgi was the resource hogger on the server? It couldn't have been this data. I'd love to look at those logs and see what they saw, to get a better idea of what happened. Either there's something that this log isn't picking up, or there was a mistake and my mt.cgi was not the culprit.

<{POST_SNAPBACK}>

The amount of bytes sent back doesn't really tell you anything about what CPU and memory resources were required to generate and send those bytes. A drastic example of this would be what occurs when you perform a full rebuild of your weblog, The web server log will only show one page request, and a relatively small number of bytes sent back to the browser. But the log won't show how many weblog pages were actually written by MT, or how much time, CPU power, or memory it took to produce them.

 

I don't know what was seen on the server side, so I don't know if one instance of MT got hung and started consuming lots of CPU, or if one the instances was blocking or interfering with subsequent instances of mt.cgi, or if it was something else. The web server logs just don't show that part of what's going on behind the scenes.

 

My question: What exactly *is* the log that you made available to me? I know that it doesn't show my entire website, because I was also accessing my /gallery/ directory during that time, and the log didn't show that.

 

Is the log limited to my /cgi-bin/mt/ directory? If so, perhaps we're looking at the *wrong* mt.cgi. I have two other installations, /cgi-bin/mt3/ and /cgi-bin/mt_berkeley/. Bill H. said he was unsure *which* mt.cgi caused the server load early this morning. Perhaps it was one of the others (in which case this must definitely have been some kind of attack, since I never use those installations anymore). Do you have logs for those directories?

I can't speak for TCH-Rick, but I wouldn't be surprised if he extracted only the log entries specifically requesting mt.cgi.

 

You should be able to download a full copy of the web server log from the "Raw Access Logs" in your CPanel.

[brendan]

Okay, I think this finally beginning to come together and make sense. I don't know yet what the solution is, but I'm starting to understand what the problem was.

 

First of all, even though The Informant clearly was not the culprit in this particular server crisis, I see what you are saying re: long-term bandwidth use, and I have already completely disabled The Informant's checking of my CGI file. So, problem solved.

 

Secondly, as for the meaning of the log file, thanks for clearing things up. What you're saying about CPU not being the same thing as bytes transferred makes sense, even though I didn't think of it that way before.

 

Thirdly, I did look at the raw log file via CPanel, and there were no attempts to access any other mt.cgi files during the problem time period. Therefore, it was definitely /cgi-bin/mt/mt.cgi that was the culprit.

 

As I've said, and as the log demonstrates, I wasn't using mt.cgi in an unusual way when the site went down; I wasn't even doing a full rebuild or anything. I was just trying to save a couple of regular posts to my blog -- nothing special. And the problem can't be blamed on the normal way that MT functions, because as I've said, if that were the case, I would have had this problem long, long ago. My blog didn't just spout 8,000 posts overnight.

 

However, I think you hit the nail on the head when you said:

 

I don't know what was seen on the server side, so I don't know if <b>one instance of MT got hung and started consuming lots of CPU, or if one the instances was blocking or interfering with subsequent instances of mt.cgi</b>, or if it was something else. The web server logs just don't show that part of what's going on behind the scenes.

 

I suspect that your guess is exactly right -- one instance of mt.cgi interfering with another and causing a hang. That makes sense, because even prior to the server slowness issues (which I unwittingly caused, but which also affected me as they were occurring), I was having problems with my Internet connection in my apartment, and because I tend to be an impatient person when it comes to technology , I may very well have started multiple instances of mt.cgi in rapid succession when the previous ones did not seem to be working. The problem then snowballed as, not only my Internet connection, but the server became slow, and I did the same thing again. The log clearly demonstrates that I did so at least two or three times, and I think the problem may have actually started even earlier than 3:00 AM EDT. So it was probably one interference/hang piled on top of another, until the server couldn't handle it anymore.

 

So basically, this is not an issue of MT security, but rather, of MT getting hung and/or some sort of interference occurring. The central question, then, is not how to "secure" my MT installation (although that's no reason to shy away from upgrading to the newest version, installing the best security possible, etc.), but rather, how to prevent mt.cgi from "hanging" in the future.

 

Part of this may be modifying my own behavior (i.e., not being so trigger-happy to click on my "login" bookmark multiple times when it doesn't seem to be working the first time), but I will also want to look into technological solutions, or at least partial solutions. In addition, it would be extremely helpful to know how long it takes before mt.cgi "times out" -- i.e., how long I need to wait before it is safe to load a new instance of the script.

 

I'm starting a new job tomorrow morning, so I have to go to bed, but after work tomorrow evening, I will look into these issues further. I'm relieved to finally know (or think I know!) what is the problem that I am trying to solve, at least in general terms. As you're all well aware, it's very hard to troubleshoot when you have only the foggiest, vaguest idea of what the "trouble" is... But now it's starting to come into focus. I think my mt.cgi script "hung," probably because of multiple instances. So now I just need to know specific conditions led to that occurrence, and how to prevent it in the future.

 

Thanks!

Edited June 16, 2005 by brendan