Odd Site Issues Today

[OJB]

Hey guys,

 

I don't usually like posting for this sort of question because it can seem so vague and its the sort of thing that cannot be easily replicated.

 

Basically all day today I have been receiving blank white screens on various of the sites I have under my reseller account. Now blank white screen to me usually screams PHP fatal error. But there are no such errors in the logs. Not only this but the white screens are very VERY intermittent. I have received the trying to log in, then log in works, I have received them trying to view posts on my forum, then the posts work. I have received them on my wiki then the wiki works. There seems to be no rhyme or reason to it. It is strange that it is across various domains too. I assumed it must be my end, even though I was accessing my sites from both home and work (both with different ISPs). Until i got home tonight and some of my users are complaining of the same thing.

 

If I get a blank white screen I did a view source and I am getting this:

 

>
<script type="text/javascript" language="javascript"> var iuub=new Date( ); iuub.setTime(iuub.getTime( )+014*074*074*01750); document.cookie="\x6e_\x73\x65\x73s\x5f\x69\x64\x3d\x34b65f\x333\x389\x34\x39\x65c\x34cb4\x30\x62\x31\x33\x31\x64\x32\x30\x39\x61a\x328\x36f"+"\x3b\x20pat\x68\075\x2f; \x65xpir\x65s="+iuub.toGMTString( ); </script> 

 

so JS is setting some sort of cookie. What cookie I don't know as it appears to be encrypted or something. But that cookie is not being set from my code as it isn't in the source when the page loads successfully. So it may be a firefox firebug thing, I don't know, I have no idea what this cookie does or is.

 

Not only this but my firebug console also appears to be showing 301 moved permanently redirects for my site - but not every time I load the site. Which is weird, as I don't have any .htaccess rules redirecting anything.

 

I have attached an example screenshot of my firebug output. What is bizarre is it tries to GET my website and I get a response:

"Failed to load source for: http://www.ukhhf.co.uk/forum"

but then it does the GET again and it is successful, on the exact same domain, subdomain and directory (www.ukhhf.co.uk/forum).

 

The actual response header I receive when it fails to load (blank white screen) is:

>
Date	Sun, 13 Sep 2009 11:00:28 GMT
Server	Apache
X-Powered-By	PHP/5.2.6
Content-Length	342
Connection	close
Content-Type	text/html; charset=UTF-8

 

It is starting to get a little frustrating, so I contacted the helpdesk and spoke to someone there who told me the sites were coming up fine for them, so there was no issue on the server side. I explained that it was very intermittent and across several users and ISPs but he was happy with it not being server side, so now that is why I am here.

 

I have been a web developer for years now so it baffles and frustrates me when something inexplicable like this occurs and I cannot find the source of the issue.

 

Even now as I have been typing this I have been receiving issues with the following sites:

 

ukhhf.co.uk/forum

mixforme.co.uk

ukhhf.co.uk/encyclopedia

 

 

Can anyone help?

 

Regards,

OJB

[TCH-Bruce]

If you figure it out please let me know. I believe it's a Firefox issue because I get this at a lot of sites (non TCH). A simple refresh or CTRL-F5 clearing my cache loads the page just fine.

[OJB]

Well the plot seems to thicken with my site. And to be honest it is getting even more inexplicable.

 

10-20 of my regular users are reporting being presented with popups warning them of viruses (when using IE) after (for no reason at all) being forwarded to a site called fast-virus-scan4, note I have not included the link as the site appears to contain adware.

 

This does not happen on Firefox but I expect when the firefox page goes blank white this is the point that the IE browser forwards to the adware site mentioned above.

 

I have downloaded every file off my forum and scanned them all with:

 

adaware

spybot s&d

nod32

malwarebytes

 

all come up clean. I also did a load of housekeeping and removed unnecessary directories and ensured everything was chmodded securely.

 

 

What makes it strange is that I also have several other sites also hosted on Alderaan (under my reseller account) and I am seeing these same issues with those sites. So, I decided to do some digging and found other sites also on the same IP address. I found one called builderssurplusatlanta and after a few minutes of clicking around on a bunch of links in IE I got forwarded to the adware site and my nod32 kicked up a warning.

 

If this was just me I would say that it most definitely was I was infected by adware/spyware/malware. But I have virus scanned my PC with Eset NOD32, SpyBot S&D, Malwarebytes and Adaware and aside from a few tracking cookies it comes up clean. Now that my other members are getting the same issues this makes me think the issue might be elsewhere, but where?

 

I am getting major complaints from my users and it has me baffled.

 

 

It doesn't happen all the time, in fact I can go 10+ minutes without it happening and then it does again. Very mind boggling.

[nmarketers]

The same issue is there with my website too. nmarketers.com

 

Right now google has flagged my website with a warning

 

I had browsed through all the links on my page and found, the following problematic

 

Page : nmarketers.com/subscribe/function.gzinflate

Link : nmarketers.com/subscribe/

 

Contents of function.gzinflate

 

<sc ript type="text/javascript"> var iuub=new Date( ); iuub.setTime(iuub.getTime( )+014*074*074*01750); document.cookie="\x6e_\x73\x65\x73s\x5f\x69\x64\x3d\x34b65f\x333\x389\x34\x39\x65c\x34cb4\x30\x62\x31\x33\x31\x64\x32\x30\x39\x61a\x328\x36f"+"\x3b\x20pat\x68\075\x2f; \x65xpir\x65s="+iuub.toGMTString( ); </sc ript>

 

This file fails to download with wget all the time. It gives the error 301 moved permenently.

 

Any Solutions ?.

Edited September 16, 2009 by TCH-Thomas
See my follow up post.

[nmarketers]

I have Reported the same in

 

http://www.google.com/support/forum/p/Webmasters/thread?tid=41c6c3759f1ebef9&hl=en

http://badwarebusters.org/main/itemview/9093?t=6124#itemblock-9145

 

But no luck till now ;-(

[TCH-Thomas]

Welcome to the forum, nmarketers.

 

I have edited your first post to deactivate the links since

 

1) nmarketers.com isn´t hosted by TCH

 

2 Since there seems to security related problems with the site, lets not have people accidently enter it.

[gtman55]

My site has been hosted by TCH for many years and I think you guys rock. The past couple days though, I've also experienced oddness at my site. Photos load most of the time but sometimes not. There have been higher server loads than normal (dantooine) and I have the EXACT issue OJB described. Some of my members are complaining of getting popups warning them of adware/malware.. some sort of fake antivirus site. My files seem ok. I contacted your tech dept. and they said the site loaded fine but I tried to tell them that wasn't the issue. Anyway, Pete Bishop said he found an error and fixed it and all should be well but... again the occasional fake antivirus rogue page tries to pop up. I'm wondering if this is some sort of problem affecting many sites on TCH? What could be causing this and how do we fix it?

[gtman55]

I just went to my site's forum index page (elantraclub.com/forum) and my antivirus stopped the page from loading. This is what it looked like:

 

 

Each time it does this it's a different address but the same basic thing. This only started happening two days ago.

[TCH-Bruce]

gtman55 you are running an older version of Invision Power Board. Could be it's been compromised.

[OJB]

Hey.

 

I was told by someone on the helpdesk that some of my files for my forum were 777 so were insecure. I went through these files and found no compromises that I could see. I set them all to 755.

 

Problem is I do not think this is a forum issue, the reason for that is I managed to get screen recorder footage of this virus popup on two other domains. One of them is one of my domains that is completely locked down permissions wise and is built upon the latest stable CakePHP framework. I built it with security in mind and made sure everything was locked down. The other domain I managed to get the popup on was the one I mentioned in one of my earlier posts which is clearly not built on a forum. I can't say what their permissions etc are like because I don't own or control that domain.

 

Here is video of it occurring on my forum UKHHF, it happened as soon as I loaded the site, then didn't happen again. I tried to test mixforme and builderssurplus but couldn't get it to happen in the same video:

 

http://www.screentoaster.com/watch/stVE1XRk1IR1xcQVpcXltYU1BS/ukhhf_virus

 

 

Here is a video of it occurring on my CakePHP build of mixforme. It happened when I clicked a link on the page. Then when I reloaded the page and clicked the same link it did not happen.

 

http://www.screentoaster.com/watch/stVE1XRk1IR1xcQVpdW1NfXlFS/mixforme_virus

 

 

Here is a video of it happening on builderssurplus (not my site but on the same IP address!). It happened when I clicked on a link and then like mixforme did not happen again when I clicked the same link upon reload:

 

http://www.screentoaster.com/watch/stVE1XRk1IR1xcQVpcUllaVVRU/builders_surplus_virus

 

 

I have scanned my PC to death with online scanners, NOD32, Adaware, SpyBot S&D, MalwareBytes and cleaned it up with CCleaner and there are no trojans or the like on my system. I have also scanned all my files for mixforme and ukhhf again with no issues.

 

I hate to keep bringing this up, but I have spent literally hours on end trying to work this one out and I am no closer. It seems odd that 3 sites, built by different people using different software would all suffer from this same issue.

[gtman55]

I have checked virtually all my files via FTP. I've tested it on various online sites looking for malicious code. My PC is clean. And yet, from time to time the popups occur. Sometimes only once a day sometimes more. Is it possible that some of TCH hosting's servers have been hit with malware? I think OJB and myself seem to be dealing with the same thing and is it just possible our sites are clean and this is server side? I do use IPB 2.3.2 and I have ALL the security patches.

 

There was something in the news the other day about the NY Times being hit by this and some other sites. This seem to have only stated a couple of days ago. I've checked all my recently modified files and have found nothing suspicious. Again, I'm a big fan of TCH and been here 4.5 years.

Edited September 17, 2009 by gtman55

[TCH-Thomas]

I haven´t been able to verify this but I believe that IPB is now on version 3.x only and Bruce suggests that using version 2.x may be the problem.

 

With verify I mean that some forums, cms´s etc may have two versions (for example joomla 1.0.x and 1.5.x) but I haven´t been able to find out if this is the case with IPB.

[TCH-Bruce]

I've been doing some checking and this isn't a server side issue.

 

A lot of sites are now being attacked with JavaScript Trojans the server is not attacked by virus , this happens to both Linux and windows server sites.

 

So it is doesnt look like a server based attack

 

How this happens?

 

When the computer from where you upload data thru FTP/fp is infected , it injects some JavaScript to all html files.

 

So how you can prevent this happening is keep your pc up to date by having recent antivirus, antispyware and then change your ftp logins.

[klibreck]

Glad to see I'm not the only person with this problem!

 

I too am on dantooine, like gtman55, and am experiencing the exact same problem.

 

This seems a very new and widespread problem, seemingly appearing in only the past week or so. I'm not sure that it is so called infected computers uploading by FTP that are causing it, as my system is clean and I haven't uploaded through FTP in quite a while.

 

Can anybody offer advice on how to clean up?

 

I asked support but they said they didn't find anything and it all looked ok. But the pop-ups/redirects to the fraudulent websites are still occurring.

[gtman55]

Finally!! Yes I've been in contact with the tech support at TCH and given them info from a security site (badwarebusters) I recently joined. BB seems convinced this issue is with the server. I'm not here trying to make a stink. I like TCH. All i ask is they read the links I sent them that might help fix the "backdoor" script that apparently has entered the server. The standard response is "It came from a client's PC via FTP". Well, I am extremely security concious on my home pc and it's always clean. I use NOD32 and many other anti-trojan, anti-spyware programs. It is possible it did come from someone on dantooine with an infected machine though. But now it resides in the server. Apparently this does not change any files on any site.

 

The attack is most likely part of what is known as "Goscanpark" a server backdoor that intermittently and at random sends people to the fake anti-scanner site. This is a new type of attack most hosts don't know about and I even sent TCH tech literature that would help them remove it and what to look for.

 

Here's what I've been told and a link to the detailed information:

 

That’s a typical “Goscanpark” code and a typical “Goscanpark” behavior (intermittent activity and redirects to fake anti-virus sites).

 

Luckily it can be easily detected by server admins if they search for a backdoor script. Show your hosting provider the article about Goscanpark and let them read comments to that article where other server admins share their experience about how they found the backdoor script.

 

If your hosting provider simply denies the problem, consider moving your site to another server.

 

Denis – www.UnmaskParasites.com

 

goscanpark

 

I suspect the code you found is an indication of a random insertion of code as a result of a server with a problem. Chances are that other sites on your server are experiencing the same random problem.

 

The following discussion is about what seems to be a similar issue.

 

http://badwarebusters.org/main/itemview/9093

http://wordpress.org/support/topic/310791

 

Edit: Corrected the Wordpress link – Be sure you check this out

 

I would suggest that you keep proof of the web-sniffer.net results, each time you hit the site. Is that code there every time you check? Is it totally replacing your content?

 

If you can show your hosting provider what is happening, they may be more open to reviewing the documentation that is referenced in the WordPress forum thread and consider that there may be a server problem. Edit: This could be a variation of the “goscanpark” situation.

 

http://blog.unmaskparasites.com/2009/07/23/goscanpark-13-facts-about-malicious-server-wide-meta-redirects/

 

 

These are independent people trying to help.

 

And here's something from a TCH customer:

 

I’m also at TCH, and from the threads I’ve followed in their forum, their customer support seems to be generally quite responsive to issues such as this. However, if your only inquiry so far has been in the forum (and you haven’t actually filed a support request), I’d go ahead and file a support request, except that you might wait until you’ve been able to gather a fair amount of documentary evidence about what’s going on. As Denis says in his articles and forum posts, if this is goscanpark, it can be extremely intermittent and elusive, and if a host doesn’t immediately see the types of compromise that they’re used to seeing, it can help if you have convincing evidence that they need to look more closely (I’m not talking about TCH in particular).

 

Your site is currently returning a page sometimes, and sometimes not. Here is the log of a download with wget:

 

C:\>wget http://www.elantraclub.com'>http://www.elantraclub.com

<del>-19:27:16</del>- http://www.elantraclub.com/'>http://www.elantraclub.com/'>http://www.elantraclub.com/

=> `index.html’

Resolving www.elantraclub.com… 208.76.80.81

Connecting to www.elantraclub.com|208.76.80.81|:80… connected.

HTTP request sent, awaiting response… Read error (Connection reset by peer) in headers.

Retrying.

 

It does that 3 more times, then…

 

<del>-19:27:33</del>- http://www.elantraclub.com/

(try: 5) => `index.html’

Connecting to www.elantraclub.com|208.76.80.81|:80… connected.

HTTP request sent, awaiting response… Read error (Connection reset by peer) in headers.

Giving up.

 

But then other times, it returns a page:

 

C:\>wget http://www.elantraclub.com

<del>-19:35:47</del>- http://www.elantraclub.com/

=> `index.html’

Resolving www.elantraclub.com… 208.76.80.81

Connecting to www.elantraclub.com|208.76.80.81|:80… connected.

HTTP request sent, awaiting response…

HTTP/1.1 200 OK

Date: Sat, 19 Sep 2009 02:35:45 GMT

Server: Apache/2.2.11 (Unix) mod_ssl/2.2.11 OpenSSL/0.9.7a mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635

Last-Modified: Thu, 17 Sep 2009 14:13:11 GMT

ETag: “e80094-db7-473c69d8fabc0”

Accept-Ranges: bytes

Content-Length: 3511

Connection: close

Content-Type: text/html

Length: ignored [text/html]

 

[ <=>

19:35:49 (91.58 MB/s) – `index.html’ saved [ 3511 ]

 

This is a status message I haven’t seen before, so I don’t know what it means:

“Read error (Connection reset by peer) in headers.”

It “sounds” like something is intermittently messing with the outgoing headers, modifying them to say “go away”.

 

So, unfortunately, one step to do is find some of your neighbor sites and see if they’re doing the same thing.

 

Sorry for the long post but this is an urgent situation and hopefully TCH will take the information to good use and fix the issue. To klibrek... are your redirects to various things like indianapolis-sales.com and best-virus-scanners5.com and stuff like that? I think the fact that you have the same problem is starting to make it clear the server is infected.

Edited September 19, 2009 by gtman55

[klibreck]

Hi gtman55, yes the redirects appearing on my site are to the likes of best-virus-scan05, total-virus-scan05, etc.

 

Looks like you've done some good research. Hopefully TCH will respond quickly (as they always have in the past)

[gtman55]

I have worked hard and put a lot of work into my site over the past 5 years.

 

I think we all like TCH and want this fixed and hopefully this can be prevented in the future.

 

TCH has always been very good with support.

Edited September 19, 2009 by gtman55

[SteveW]

Last night I downloaded the home pages from the sites I could identify that are on the server and searched thru the logs and the retrieved pages, about 320 page requests from probably about 150 sites (without noticing I had the www and non-www version of each, which I'll fix today). The sites of gtman55 and klibreck were two of them. (OJB, feel free to send me your domain name by PM, if you like).

 

I only got 3 of the "Read error (Connection reset by peer) in headers." messages, on 2 sites, neither of them yours.

 

I only found 2 suspicious iframes, except it was on the www and non-www versions of the same site, so it's just 1.

 

This morning, I did the same downloads. Unfortunately, I accidentally overwrote the log file, so can't count "Connection reset" messages. I only got the same 1 iframe on the same 1 site. I'll try to look into that further.

 

I also crawled about 500 pages from gtman55's site this morning (sorry, but at least it was only at dial-up speeds). I got no "Connection reset" messages and no virus alerts. I haven't completed the text searches, but a quick search finds no suspicious iframes.

 

Later today I'll try to do the all-sites crawl again to see if those "Connection reset" messages occur on different sites than previously.

 

-----

 

I'd suggest that anyone getting reports of strange behavior or viruses try to get detailed information about the circumstances: what page they were viewing, what they were trying to do (only reading, or posting a message?, etc.).

 

Anything this intermittent is hard to pin down. It's hard to fix something if it won't act broken during the time you're looking at it.

 

It actually makes it more difficult that no site so far as been flagged at Google with "This site may harm your computer", because we therefore don't have Google's machinery and their reports to help investigate.

 

If someone (someone who is immediately affected by this, meaning someone in this thread or on dantooine, not just anybody) wants to look up the 150 sites in the Google SafeBrowsing Database to see if any are flagged, I can provide the list, but it's against Google TOS to do it programmatically, so it has to be one-at-a-time lookup.

 

At the moment, there is very little evidence to go on. I downloaded more than 1,100 pages from this server, and only found evidence of 1 possibly hacked site, and I'm not even sure about that one, and it's not one of the ones being discussed here. So even though it can be said that the behavior being reported sounds at least similar to a "goscanpark" type compromise, there is very little of that behavior, and it's especially strange that no site is flagged at Google, so it's way too soon to make an assumption, or even have a belief, that that's the case.

 

Anyone affected should carefully go through all the steps that they would ordinarily do if they were certain that it was only their site that got hacked, and not try to "make a case" that it's the server, because usually it isn't. Going by statistics, individual site compromise has to be ruled out before it's justified to wonder if it might be a problem in the server. That being said, comparing notes on what people are seeing, as is being done in this thread, is exceptionally useful.

 

If admins want to take a pre-emptive look to see if anything seems out of place, one of the things in the useful comments here:

http://blog.unmaskparasites.com/2009/07/23/goscanpark-13-facts-about-malicious-server-wide-meta-redirects/

is the discussion of malicious crontabs being found running in memory but that have been deleted from disk so they can't be found by searching files.

[gtman55]

Steve the only thing is that this odd script that shows up was mentioned as something that set some kind of intermittent cookie or something and this was what I saw one time on my site.... It's so similar to the others posted here and this type of script is mentioned as part of the server exploit. Again I am no expert... just trying to give TCH as many clues as possible to figure things out.

 

<script type="text/javascript" language="javascript"> var pocjosm=new Date( ); pocjosm.setTime(pocjosm.getTime( )+014*074*074*01750); document.cookie="n\x5fs\x65s\x73\x5fid=\x66\x63c\x32c9d\x30e7d\x353\x66\x63\x324\x38\x35\x360\x38c\x36\x61\x38\x65\x374\x65\x358"+"\x3b\x20pat\x68\075\x2f; \x65xpir\x65s="+pocjosm.toGMTString( ); </script>

 

I did notice the server was down for a short time this afternoon and I wonder if there was some work being done behind the scenes by TCH to correct something.

Edited September 19, 2009 by gtman55

[TCH-Carl]

We have been reviewing this server extensively and will post the findings later tonight. Please give us some more time. Thank you for your understanding and patience.

[gtman55]

Thanks Carl. That's all we can ask for.

Edited September 19, 2009 by gtman55

[Paul Squires]

This problem has been affecting my site for about a week. I'm "happy" to know that I am not the only one affected. I wiped my site and reinstalled Simple Machines Forum as well as Mantis Bug Tracker (both php based apps). The problem went away for a couple of days but now it is back again. After surfing through a few pages on the site you will eventually get redirected to an "antivirus warning page".

 

I am on alderaan.

 

Domain is PlanetSquires.com

 

I really hope that someone can figure this out. Lots of my customers are emailing.

 

Thanks,

 

Paul

[gtman55]

We have been reviewing this server extensively and will post the findings later tonight. Please give us some more time. Thank you for your understanding and patience.

Any update?

[klibreck]

gtman55 (and any others on dantooine), I had a reply from support last night:

 

"We have located the compromised account and have taken corrective measures. The server and security settings have been tweaked to ensure that such attempts are brought to our notice if it happens again. Kindly accept our apologies for any inconveniences this may have caused and we thank you for your patience and understanding."

 

Way to go guys - huge thank you to Carl, Dick and anybody else that was involved in sorting this

 

Look forward to reading the official announcement detailing their findings.

 

Paul Squires - I'm guessing that the corrective measures will be rolled out across other affected servers. Best way of finding out would be to put in a support ticket.

[SteveW]

Another good reason why I'm happy to be hosted here. Great work, techs. Nice work, everybody.

[OJB]

Wow, I went away for the weekend and come back to this! Awesome. I haven't had a chance to check my sites yet but I am very keen to do so now.

 

I love it here at TCH.

[OJB]

Unfortunately I am still getting them on my Alderaan hosted sites. Any idea what server the compromised account was found on, and if it was not Alderaan whether these corrective measures will be rolled out to other servers too?

[klibreck]

OJB - as I said in my above post, I was referring to dantooine. Raise a support ticket and ask there about alderaan.

[Head Guru]

Hey Gang,

 

Please let me know if your still having these js inserts occuring, I do believe we kicked the server enough times to have finally fixed it.

 

I will give you a complete explanation of the events and how we corrected it once I get some feed back.

[OJB]

Thanks for the update, Bill.

 

The issues have stopped on all my sites hosted on Alderaan now, which is something I and my users are very relieved about. Thank you for your hard work looking into this for us!

 

Hey Gang,

 

Please let me know if your still having these js inserts occuring, I do believe we kicked the server enough times to have finally fixed it.

 

I will give you a complete explanation of the events and how we corrected it once I get some feed back.

[gtman55]

No issues in the last day and a half. Fingers crossed...

[TCH-Dick]

As promised Ryan has now posted an article explaining this issue. You can find this article in the TCH Forums or if you wish to comment on the artcle you can do so on the TCH Blog.

 

Thanks to everyone for being so patient during this and for being persistent in sharing any data you found.

[klibreck]

Thanks for the link, Dick.

 

However I just went to browse through OJB's site (ukhhf.co.uk) for the first time, and within a few minutes of browsing I experienced the problem there (I believe it's on alderaan?).

Edited September 23, 2009 by klibreck

[OJB]

I can confirm it has indeed flared back up on my site for myself and several users.

[TCH-Thomas]

Please report this at the helpdesk, so they can investigate it further if needed.

The techs don´t read the forums as often as we mods do.

[simhomer]

alderaan appears to still have issues. Ticket submitted.

[Head Guru]

Were looking into this again.

[Head Guru]

Hey Gang,

 

How are things looking ?

[OJB]

Things have calmed down on Alderaan again. Not seen or heard about any issues today.