Jump to content
StevenTing

Events Of 5/27/2012

Recommended Posts

To those whos sites are back but cpanel won´t let you in. Please wait a bit more as the techs are still working on the issues.

Share this post


Link to post
Share on other sites

pkronhert,

 

I would wait a bit more if possible, as I guess the techs still have a lot to do with this issue.

However, you could reopen the ticket, but expect some waiting to hear from the techs.

 

I would wait, but I was told by TCH-Dick above, specifically in regard to my server, "Please check now and let us know if you still have issues." I did update my ticket. Hoping some one can help with this soon. I fixed as many sites as I could myself, but some sites are customer sites that I didn't create, so I don't have the index.htm page for them to restore it myself.

Share this post


Link to post
Share on other sites

Ah, I see.

I am sure though that the techs are working as fast as they can to resolve this for everyone. :)

Share this post


Link to post
Share on other sites

I would wait, but I was told by TCH-Dick above, specifically in regard to my server, "Please check now and let us know if you still have issues." I did update my ticket. Hoping some one can help with this soon. I fixed as many sites as I could myself, but some sites are customer sites that I didn't create, so I don't have the index.htm page for them to restore it myself.

 

I have fixed and responded to your ticket. Please check now

Share this post


Link to post
Share on other sites

Make sure you select the Cpanel login from the drop down and enter your server name as well. It should work.

 

Thank you. Now I have accessed the CDP server, but the most recent restore files are dated May 10. Have the more recent ones been deleted?

Share this post


Link to post
Share on other sites

hi,

please reload my index.html (www.haatami.com) if possible, my index.html deleted and i upload a temporary index.html

 

This should be fixed now. Please check

Share this post


Link to post
Share on other sites

server: worf

site was hacked today - don't when it started - fixed within two hours of calling support

cpanel works - email works - fix was good (and quickly done ... good job)

 

my question is:

for everyone that was affected by this hack, is there anything we should be doing to our accounts to ensure our security for the future? should we be changing (updating) passwords, updating our website pages, scan our website for any residue malware or hacks? was this a problem with our sites alone or with the server our accounts were on?

 

I am sure I speak for everyone in saying thanks for providing a quick fix to a major problem but we would feel much better if you provided feedback on how this happened and if this was something we might have caused (through faulty webdesign and security) or a problem with your service (being hacked by an individual)

 

Again, thanks for all your hard work

Edited by mmddkkthakd

Share this post


Link to post
Share on other sites

- Make sure you have every script you use updated to it´s newest version.

- If you code it yourself, go through your code to see that everything is correct and has no security vulnerabilities

- Change passwords for cpanel, e-mail accounts and such regular.

Share this post


Link to post
Share on other sites

PHP e-mailing doesn't seam to work anymore (although, not to cause alert, it could be just me, as I am developing my new website, so I might have broken my tested code).

>
		$to = $email; //$email received from the form submission
		$subject = "Welcome";
		$message = 'some message';

		// To send HTML mail, the Content-type header must be set
		$headers  = 'MIME-Version: 1.0' . "rn";
		$headers .= 'Content-type: text/html; charset=iso-8859-1' . "rn";

		// Additional header
		$headers .= 'From: [redacted] <noreply@[redacted].com>';

		// Mail.... that... e-mail!
		if (mail($to, $subject, $message, $headers)) {
			// victory!
		} else {
			// Oh noes! <- I get here 
		}

 

I am on bespin server

 

Thanks

Edited by GoodBYtes

Share this post


Link to post
Share on other sites

Update - We are still working on restoring specific client sites that were effected and not restored from our first round.

 

Once we get thru this phase of the chaos, I will give more updates.

 

BK

Share this post


Link to post
Share on other sites

GoodBYtes,

 

Please hold on as the techs are still working on the issues.

If the code still not working later today, please post the code problem in the Scripting Talk forum and we can have a look. :)

Share this post


Link to post
Share on other sites

I also had a custom file on a vBulletin forum called start.php that was affected.

Share this post


Link to post
Share on other sites

I have read that this TiGER-M@TE hack (which has been around for a while, unless this one is different) changes CPanel passwords and gets in that way. However, my CPanel passwords do not seem to be changed. I did change the one for my main account to protect WHM. If you can update us at some point on whether we should change all our CPanel passwords due to this, would appreciate. Thanks.

Share this post


Link to post
Share on other sites

Just finding out about this now. Neen with TCH so long and never having had a problem like this, I just don't monitor my site that often Anyway, it looks like the index.php restore script worked. However, any index.htm file was deleted, either by the script or by the hacker. I've restored my site from my personal backup files.

 

I still can't send any email: access denied...

 

Server is stormtrooper

 

Regards,

Dan Cumpian

Edited by dcumpian

Share this post


Link to post
Share on other sites

On server Phoenix..

 

I think more than just the index pages were affected.

 

My sites contact forms have stopped working; some are perl (cgi) scripts and others are php form processors.

 

However, I seem to be getting regular emails on some of the accounts affected.

 

Is anybody else experiencing this as well?

Share this post


Link to post
Share on other sites

I wasn't home when someone emailed me about the hacked message. I don't have an index.php, but apparently they either created one or targeted my index.htm.

 

When I got home, I checked my site (on sidious) and found that the index.htm had been removed, so I uploaded a copy from local backup and site is working now.

 

If FTP is accurate, my other files at the public-html are fine based on date-stamps. Should I be checking deeper to make sure all is well?

Share this post


Link to post
Share on other sites

Changing passwords every now and then are always good.

Not so easy to change your WHM password. TCH requires you to submit a support ticket and they will change it.

However, sub-accounts can be changed through WHM.

Share this post


Link to post
Share on other sites

Were still here guys. We have moved into another phase of this situation and are working non stop.

 

Will update when we have more to report.

Share this post


Link to post
Share on other sites

My site is working, but my contact forms are not working either. I get "Could not instantiate mail function." when testing form.

Share this post


Link to post
Share on other sites

Ray_Bman,

 

I was referring to all individual cpanels.

 

As for WHM, same goes there, changing password every now and then are good for saftey, even though it takes a little more time to go through the help desk to have whm password changed than logging in to your cpanel.

Share this post


Link to post
Share on other sites

I think we got the php mail issues fixed.

 

If anyone can try that and let me know here, that would be great.

Share this post


Link to post
Share on other sites

My site is working, but my contact forms are not working either. I get "Could not instantiate mail function." when testing form.

 

Give it a try now please and let me know.

Share this post


Link to post
Share on other sites

My contact form is still not working. Still get - Could not instantiate mail function. error.

Share this post


Link to post
Share on other sites

 

If FTP is accurate, my other files at the public-html are fine based on date-stamps. Should I be checking deeper to make sure all is well?

 

I would check a couple folders deep. If you use frontage extensions it also copied files in there. I never found anything more than 2 folders deep.

Share this post


Link to post
Share on other sites

Thanks for the quick work on this issue. I have two sites that are back without any issues however one of the sites is back but not functioning correctly, it uses Wordpress - some of the pages are missing completely and some of the functionality is not functioning and the home page goes to an error page.

 

photomelange.com is the site. the index files are not the index files left by the hackers. But if this was restored from a recent backup it was not restored correctly. I'm not sure at this point what to do...

Share this post


Link to post
Share on other sites

Thanks for the quick work on this issue. I have two sites that are back without any issues however one of the sites is back but not functioning correctly, it uses Wordpress - some of the pages are missing completely and some of the functionality is not functioning and the home page goes to an error page.

 

photomelange.com is the site. the index files are not the index files left by the hackers. But if this was restored from a recent backup it was not restored correctly. I'm not sure at this point what to do...

 

We are looking at your site now. Give us a few moments for an update.

Share this post


Link to post
Share on other sites

Full restore works pretty well. However it does not remove additional files that were added. It will overwrite existing files which fixes a majority of problems. For example the front page extension folders didn't originally have index.php files but were newly creates. The restore didn't remove those but the script from TCH may have removed those.

Share this post


Link to post
Share on other sites

Full restore works pretty well. However it does not remove additional files that were added. It will overwrite existing files which fixes a majority of problems. For example the front page extension folders didn't originally have index.php files but were newly creates. The restore didn't remove those but the script from TCH may have removed those.

 

Good to know. Thanks StevenTing.

Edited by clydejsn

Share this post


Link to post
Share on other sites

Is this directed to mycat2 only?

 

Your site has been restored from backups. If your still having issues, please open a ticket and give us a full explanation of what issues your having and we can take a look at it for you. However, at this time the restore is completed.

Share this post


Link to post
Share on other sites

PHP Mail issues have been resolved as far as we can tell.

 

If your having any issues still let me know.

Share this post


Link to post
Share on other sites

As of 1:29pm PDT still no php mail on my site: genosis, www.salemharvest.org.

Edited by rayates55

Share this post


Link to post
Share on other sites

As of 1:29pm PDT still no php mail on my site: genosis, www.salemharvest.org. No error message generated, but no email is sent.

 

We are working on that now.

Share this post


Link to post
Share on other sites

Thanks. Forms still aren't working. Will submit a ticket.

 

What is your domain?

Share this post


Link to post
Share on other sites

My contact forms are now working! Yay! Thank you!

Edited by clydejsn

Share this post


Link to post
Share on other sites

I am unable to locate your forums, please give me a direct link.

 

Sorry for the confusion. Forms not forums. All is good; site seems to be working correctly. Thanks for all of your help.

Share this post


Link to post
Share on other sites

Mail form fix is rolling out to all servers now. Updates coming shortly.

Share this post


Link to post
Share on other sites

Forms still aren't working here. (phoenix server)

 

I guess we have to submit a request for a restore..?

Share this post


Link to post
Share on other sites

You guys are awesome. Not how you planned on spending your Memorial Day weekend I'm sure. Go ahead, Bill, have a Mountain Dew. You deserve one!

Share this post


Link to post
Share on other sites

Forms still aren't working here. (phoenix server)

 

I guess we have to submit a request for a restore..?

 

Mail issue should be corrected, please check it out and let us know if it doesn't work.

Share this post


Link to post
Share on other sites

I'm not sure if you were still working on it or not... but I was able to go in and fix some of the issues on my site. If I continue to have issues I will submit a support ticket, thank you!

Share this post


Link to post
Share on other sites

GoodBYtes,

 

Please hold on as the techs are still working on the issues.

If the code still not working later today, please post the code problem in the Scripting Talk forum and we can have a look. :)

Right, thanks! :D

 

 

I think we got the php mail issues fixed.

 

If anyone can try that and let me know here, that would be great.

Yes it wors perfectly fine on my side now. Huge Thanks!

 

Man this was a lousy month to stop drinking Mountain Dew

lol.

It's ok, a couple more days left!

Share this post


Link to post
Share on other sites

Okay, all seems to be working now (phoenix server).

 

Thank TCH guys for being on top of this.

 

It would nice to know how this all happened.

 

Maybe after all the dust settles...?

Edited by Ray_Bman

Share this post


Link to post
Share on other sites

TCH Family,

 

I have tons of emails from many of our concerned clients. Please understand that we have all been working non-stop since this am to get things corrected on our servers. I will personally reply to each and everyone of you that have emailed me. Just asking that you all be a little patient on my replies. Anything urgent should be sent thru our help desk as we have full 24/7 support waiting to help you.

 

Thanks and we will update again shortly.

Share this post


Link to post
Share on other sites

I've been out for a while, and just checked response forms on our sites on Columbus and Ft Worth. Everything works now. Many thanks to all of the TCH Gurus for all of your hard work today.

Share this post


Link to post
Share on other sites

I have replied to all the emails from clients. I am now working on returning phone calls.

 

Things are running smooth on the servers. We have identified the method of this hack and we will be releasing a post sometime tonight with full details.

 

We are all still monitoring, watching and working any tickets that come in.

 

I still have not got to a Mountain Dew, however I have run out of milk and Marlboro lights. lol

Share this post


Link to post
Share on other sites

I look forward to the report.

 

On a side note I say go back to the dew and drop the marlboro's. It'll be better for all of us.

Share this post


Link to post
Share on other sites

Concerning events today, I am wondering about the difference between index.html and index.php. I really didn't think that I had an index.php on my website. Is it possible to have both file types but not know it? Is an index.php really needed? If it is, where is it located? I can't see it and I've looked for invisible files...

Share this post


Link to post
Share on other sites

vaneram,

 

Are you talking about the backups that Bill mentions you could restore from in his answer here?

http://www.totalchoi...ndpost&p=250135

 

Someone said we could restore our sites ourselves by going to "R1Soft Restore Backups" from cpanel. The most recent file set I find is dated May 10. It may be moot because my site works find, but this seems strange.

Share this post


Link to post
Share on other sites

This is why I love this web hosting service. Super duper fast response (I don't think it could have been faster), help everyone fixing their web site and even do it for them, continuous monitoring of servers, transparent, super helpful even on crazy time like this, on forum active very knownegeable staff, always professional (even on their pictures they were a tie or well dressed.. see... ALWAYS professional), acknowledge a possible discovery of security breach, and solve it so it doesn't happen again. A very long day for TCH, but for us, users, it's like if nothing happened.

 

I think TCH deserve a big round of applauce for their great effort, and huge thanks. I don't think they could have handled such suprise situation any better. :)

Once everything is cleaned up, let's all dance! :D

Share this post


Link to post
Share on other sites

Due to this issue we are going ahead with kernel reboots that we had scheduled for June. These reboots are rolling out now 10 servers at at time, A-Z by host name and should have a downtime of less than 5 minutes per server. As always, if there reboot takes an excessive amount of time, we will post to the appropriate server forum.

Share this post


Link to post
Share on other sites

While our site is online and apparently without problem we still can not access cpanel. Is it "normal" (i.e. you are still working on and this is a problem common to many sites) or should we update our ticket?

Our site is on Arlington

Share this post


Link to post
Share on other sites

My guess is that the techs are still working on this, however you could follow Bills advice and submit a ticket.

 

TCH Family,

 

I have tons of emails from many of our concerned clients. Please understand that we have all been working non-stop since this am to get things corrected on our servers. I will personally reply to each and everyone of you that have emailed me. Just asking that you all be a little patient on my replies. Anything urgent should be sent thru our help desk as we have full 24/7 support waiting to help you.

 

Thanks and we will update again shortly.

Share this post


Link to post
Share on other sites

Most of my sites showing index of page this morning so am manually uploading the current index page.

 

Not just index.php files affected - index.htm pages too.

 

I'm hoping TCH doesm't now overwrite them with old back up index pages.

Share this post


Link to post
Share on other sites

We have identified the method of this hack and we will be releasing a post sometime tonight with full details.

 

Is that post going to be here, or was it made somewhere else? I'm very curious what happened.

 

A client of mine also reported being hacked (my reseller account is on jandoon), and her password isn't obvious. Based on the extent of the damage, I'd guess they got access to the server as a whole, not individual accounts. I suppose that this loser could have guessed a lot of client passwords (I've seen a hack on a discussion board that I used to help on where they accessed accounts whose passwords were the same as their user names), but I don't think that happened in this case given that my client's password isn't that poor.

 

By the way, you can use a service like my-ip-neighbors.com to find out who else is on your server.

 

Steve

Edited by TCH-Thomas
See follow up post

Share this post


Link to post
Share on other sites

I have deactivated your link Steve as it´s not hosted by TCH and also may not be good to have as active link in a public forum due to what have happened.

 

By the way, that service does not seems to show all sites. I´ve tried a few now that I know are hosted on a specific server, sometimes they showed up, sometimes not.

Share this post


Link to post
Share on other sites

Are index pages still being reinstated from backup? I still have a large number of sites showing Index of / pages.

 

Do I need to manually reinstate each one (I've spent the morning doing most important ones).

Edited by georgem

Share this post


Link to post
Share on other sites

I would open a ticket with the help desk so they can have a look. Link in my signature.

Share this post


Link to post
Share on other sites

Good morning TCH Family,

 

What a long day yesterday was. Things are much calmer today, the help desk is not filled with tickets and my staff appears to be humming along and drinking mountain dews as normal.

 

I have completed a overview report and am making some final reversions and will be posting it shortly.

Share this post


Link to post
Share on other sites

Quick Synopsis: Our network was the target of a large scale website defacing attack on Sunday, September 28th commencing at around 8:30am. A large number of servers were involved in this attack; however our internal servers and infrastructure were not vulnerable to this attack. We were able to get control of the situation by 10:00am and had identified the method of attack. The attack hole was closed and we started a methodical review of what happened. Once we learned that index files were the attack focus, we wrote a custom script that would review each index.* file on an effected server and look for certain keywords that were present in the hackers file placed on the effected servers. If this file was present our script deleted it and restored your last available backup from our weekly backups. In theory a simple method of restore but it took us a bit to get it working correct. I am confident that there are still some users on the effected servers that will wake up tomorrow and find that their index pages are blank. I should say during the chaos we ended up breaking php mail handlers on some servers, however this was fixed and was a simple mistake on our end.

 

What the attacker did: At this time, the attack does not appear to have been any more malicious than replacing the web site's home page. The defacement worked by replacing index files in all public_html directories with the attacker's index.php file.

  • The hacker’s main goal was to deface websites.
  • Our entire internal infrastructure, including Domain Management, DNS Clusters, Billing, Forums, Help Desk or any Client Credit cards or personal information was not targeted or accessed. Furthermore, the entry method that was used was not an available entry point on any of our Internal Infrastructure.
  • The attacker did not obtain user passwords. This apparently was not the hacker’s goal. The hacker used a system exploit to allow him to access index files. As always though, it is prudent that you update your cPanel and FTP passwords.
  • The exploit used to gain entry has been blocked.
  • The issue was with an authentication system that was in use for an application controlling cPanel. The exploit itself was something not seen before and has been forwarded to the prospective people for review.

Where we are at this point: We have a complete understanding of the attack method used and we have already taken needed steps to block future similar attacks.

 

In the future: While not related, we are going to start phasing out our dual version PHP platform servers. We have many servers that are running two versions of PHP. This was done for clients that asked for us to allow them time to upgrade their scripts to the new 5.3.x versions of PHP. The time has come and passed and we will be removing the dual versions of PHP and will only be rolling with the latest version of PHP on all our servers. Like I said this is not related to the web site defacements, however it is something that we need to address. We also ran software and system updates on all servers last night and rebooted every shared / reseller server. These updates were originally slated for June to coincide with the coming PHP upgrades, however in light of this incident; we choose the precautionary approach and completed them now.

 

In addition: Even though our team responded quickly to disable the attack and limit the exposure to our customers, and even though the damage done was straightforward, this was a serious incident. At TotalChoice we are very aware of the threats that every website faces on a daily basis and are committed to protecting our server farm and all of our clients. Our security record up until this point has been excellent. Whilst we do view this incident as a failure, please know that this security hole was not known to anyone and if it had been we would have taken prior action.

 

Final thoughts: I want to personally thank each staff member for the amazing work done during this incident. More importantly I want to thank each and every client for their understanding and support after all without the clients we would have nothing. Please let us know if you have any questions and or concerns about this matter and we will gladly answer them for you. Once again, thank you for your business and support.

Share this post


Link to post
Share on other sites

Well, personally, I'm totally impressed by the rapid response of TCH.

 

Imagine my thoughts as we were on the way out the door to Church on Sunday, when the phone rang with another of the Church members telling me of the horrible things on our Church website.....

 

I told my wife as we left that TCH would handle it, and they did.

 

Thank You !!!

Share this post


Link to post
Share on other sites

GoodBYtes's post above (#157) expressed much the same as I was thinking, better than I can ^.

 

Especially appreciate the prompt status announcement in the forum. The notification email was how I learned of the incident. I found it a non-jarring way to be notified.

 

I want to personally thank each staff member for the amazing work done during this incident.

 

Truly amazing, skillful, and fast. Thank you.

Edited by SteveW

Share this post


Link to post
Share on other sites

I found this problem on my server (tyson) several weeks ago. I use a cron job to run a diff between the previous day's dir listing (of everything) and the current dir listing, and that's how I caught the problem. I worried that my own laptop had been compromised and they got my password. I'm glad this thread showed up, so now I know it's unlikely that my password is compromised.

 

Database passwords are nearly always stored in code, which I believe was accessible to the hacker, so I'd recommend changing them.

 

There were files other than index.php in my account that I had to restore from my local copy.

 

I also created a self-healing mechanism so that if the write time on my main index.php file changed, it would restore itself from a backup.

 

I can post more info about these two solutions if anyone is interested. Oh, and I'm also available if anyone needs a good software guy.

 

Dave.

Share this post


Link to post
Share on other sites

Thanks to all of your reports and further review, we were able to identify what files we didn't restore during our initial fix. Which as you have seen would be any index file that was not named index.php.

We have now completed a new run to restore these files.

 

In order to not undo changes or restores you have already done, this run to into account the following before restoring.

-was the file one we logged during our initial fix

-if so, was the file missing in public_html

-if so, was the file not equal to index.php

-if so, was the backup file clean of the defacement

-if so, restore

 

Thank you all again for your patience and please let us know if you continue to have issues.

Share this post


Link to post
Share on other sites

Don't forget that any "start.php" file was also affected.

 

Newer versions of vBulletin forums use "forum.php" for their home page but there is still an "index.php" that redirects to it for old bookmarks. My "index.php" was affected but it didn't cause any problems for members.

Share this post


Link to post
Share on other sites

we've been HACKED AGAIN!!! is anyone else in the same boat???

Share this post


Link to post
Share on other sites

They are aware of it and are working on the problem right now.

Share this post


Link to post
Share on other sites

thanks.

is there anyway to keep this guy out? if his goal is to cripple US business, he is succeeding.

very frustrating.

Share this post


Link to post
Share on other sites

yep, my site has been hacked as well. this time appears to be worse, as my cpanel password has been changed and i have no ftp access. argh.

Share this post


Link to post
Share on other sites

I think this is happening again 5/30/12. Client sites are going down left and right and seems to be isolated to index.php

Share this post


Link to post
Share on other sites

Happened to one of my sites too but it was just fixed.

 

Another symptom,I cannot long into cpanel for any of my sites. Not sure if this is TCH related or hack related. I'll worry about it in a couple of hours.

Share this post


Link to post
Share on other sites

WOW. No sooner do I click the post button for the previous post and my site is fixed. Now that's customer service!

Share this post


Link to post
Share on other sites

This is badness. I really hope totalchoice is able to rectify the situation.

 

My sites are on kalee, and my forum users were getting the tigermate hack notice again today.

Edited by orkan

Share this post


Link to post
Share on other sites