Jump to content

Events Of 5/27/2012


StevenTing

Recommended Posts

  • Replies 236
  • Created
  • Last Reply

Top Posters In This Topic

pkronhert,

 

I would wait a bit more if possible, as I guess the techs still have a lot to do with this issue.

However, you could reopen the ticket, but expect some waiting to hear from the techs.

 

I would wait, but I was told by TCH-Dick above, specifically in regard to my server, "Please check now and let us know if you still have issues." I did update my ticket. Hoping some one can help with this soon. I fixed as many sites as I could myself, but some sites are customer sites that I didn't create, so I don't have the index.htm page for them to restore it myself.

Link to comment
Share on other sites

I would wait, but I was told by TCH-Dick above, specifically in regard to my server, "Please check now and let us know if you still have issues." I did update my ticket. Hoping some one can help with this soon. I fixed as many sites as I could myself, but some sites are customer sites that I didn't create, so I don't have the index.htm page for them to restore it myself.

 

I have fixed and responded to your ticket. Please check now

Link to comment
Share on other sites

Make sure you select the Cpanel login from the drop down and enter your server name as well. It should work.

 

Thank you. Now I have accessed the CDP server, but the most recent restore files are dated May 10. Have the more recent ones been deleted?

Link to comment
Share on other sites

server: worf

site was hacked today - don't when it started - fixed within two hours of calling support

cpanel works - email works - fix was good (and quickly done ... good job)

 

my question is:

for everyone that was affected by this hack, is there anything we should be doing to our accounts to ensure our security for the future? should we be changing (updating) passwords, updating our website pages, scan our website for any residue malware or hacks? was this a problem with our sites alone or with the server our accounts were on?

 

I am sure I speak for everyone in saying thanks for providing a quick fix to a major problem but we would feel much better if you provided feedback on how this happened and if this was something we might have caused (through faulty webdesign and security) or a problem with your service (being hacked by an individual)

 

Again, thanks for all your hard work

Edited by mmddkkthakd
Link to comment
Share on other sites

- Make sure you have every script you use updated to it´s newest version.

- If you code it yourself, go through your code to see that everything is correct and has no security vulnerabilities

- Change passwords for cpanel, e-mail accounts and such regular.

Link to comment
Share on other sites

PHP e-mailing doesn't seam to work anymore (although, not to cause alert, it could be just me, as I am developing my new website, so I might have broken my tested code).

>
		$to = $email; //$email received from the form submission
		$subject = "Welcome";
		$message = 'some message';

		// To send HTML mail, the Content-type header must be set
		$headers  = 'MIME-Version: 1.0' . "rn";
		$headers .= 'Content-type: text/html; charset=iso-8859-1' . "rn";

		// Additional header
		$headers .= 'From: [redacted] <noreply@[redacted].com>';

		// Mail.... that... e-mail!
		if (mail($to, $subject, $message, $headers)) {
			// victory!
		} else {
			// Oh noes! <- I get here 
		}

 

I am on bespin server

 

Thanks

Edited by GoodBYtes
Link to comment
Share on other sites

Update - We are still working on restoring specific client sites that were effected and not restored from our first round.

 

Once we get thru this phase of the chaos, I will give more updates.

 

BK

Link to comment
Share on other sites

I have read that this TiGER-M@TE hack (which has been around for a while, unless this one is different) changes CPanel passwords and gets in that way. However, my CPanel passwords do not seem to be changed. I did change the one for my main account to protect WHM. If you can update us at some point on whether we should change all our CPanel passwords due to this, would appreciate. Thanks.

Link to comment
Share on other sites

Just finding out about this now. Neen with TCH so long and never having had a problem like this, I just don't monitor my site that often Anyway, it looks like the index.php restore script worked. However, any index.htm file was deleted, either by the script or by the hacker. I've restored my site from my personal backup files.

 

I still can't send any email: access denied...

 

Server is stormtrooper

 

Regards,

Dan Cumpian

Edited by dcumpian
Link to comment
Share on other sites

On server Phoenix..

 

I think more than just the index pages were affected.

 

My sites contact forms have stopped working; some are perl (cgi) scripts and others are php form processors.

 

However, I seem to be getting regular emails on some of the accounts affected.

 

Is anybody else experiencing this as well?

Link to comment
Share on other sites

I wasn't home when someone emailed me about the hacked message. I don't have an index.php, but apparently they either created one or targeted my index.htm.

 

When I got home, I checked my site (on sidious) and found that the index.htm had been removed, so I uploaded a copy from local backup and site is working now.

 

If FTP is accurate, my other files at the public-html are fine based on date-stamps. Should I be checking deeper to make sure all is well?

Link to comment
Share on other sites

Changing passwords every now and then are always good.

Not so easy to change your WHM password. TCH requires you to submit a support ticket and they will change it.

However, sub-accounts can be changed through WHM.

Link to comment
Share on other sites

Ray_Bman,

 

I was referring to all individual cpanels.

 

As for WHM, same goes there, changing password every now and then are good for saftey, even though it takes a little more time to go through the help desk to have whm password changed than logging in to your cpanel.

Link to comment
Share on other sites

 

If FTP is accurate, my other files at the public-html are fine based on date-stamps. Should I be checking deeper to make sure all is well?

 

I would check a couple folders deep. If you use frontage extensions it also copied files in there. I never found anything more than 2 folders deep.

Link to comment
Share on other sites

Thanks for the quick work on this issue. I have two sites that are back without any issues however one of the sites is back but not functioning correctly, it uses Wordpress - some of the pages are missing completely and some of the functionality is not functioning and the home page goes to an error page.

 

photomelange.com is the site. the index files are not the index files left by the hackers. But if this was restored from a recent backup it was not restored correctly. I'm not sure at this point what to do...

Link to comment
Share on other sites

Thanks for the quick work on this issue. I have two sites that are back without any issues however one of the sites is back but not functioning correctly, it uses Wordpress - some of the pages are missing completely and some of the functionality is not functioning and the home page goes to an error page.

 

photomelange.com is the site. the index files are not the index files left by the hackers. But if this was restored from a recent backup it was not restored correctly. I'm not sure at this point what to do...

 

We are looking at your site now. Give us a few moments for an update.

Link to comment
Share on other sites

Full restore works pretty well. However it does not remove additional files that were added. It will overwrite existing files which fixes a majority of problems. For example the front page extension folders didn't originally have index.php files but were newly creates. The restore didn't remove those but the script from TCH may have removed those.

Link to comment
Share on other sites

Full restore works pretty well. However it does not remove additional files that were added. It will overwrite existing files which fixes a majority of problems. For example the front page extension folders didn't originally have index.php files but were newly creates. The restore didn't remove those but the script from TCH may have removed those.

 

Good to know. Thanks StevenTing.

Edited by clydejsn
Link to comment
Share on other sites

Is this directed to mycat2 only?

 

Your site has been restored from backups. If your still having issues, please open a ticket and give us a full explanation of what issues your having and we can take a look at it for you. However, at this time the restore is completed.

Link to comment
Share on other sites

Forms still aren't working here. (phoenix server)

 

I guess we have to submit a request for a restore..?

 

Mail issue should be corrected, please check it out and let us know if it doesn't work.

Link to comment
Share on other sites

GoodBYtes,

 

Please hold on as the techs are still working on the issues.

If the code still not working later today, please post the code problem in the Scripting Talk forum and we can have a look. :)

Right, thanks! :D

 

 

I think we got the php mail issues fixed.

 

If anyone can try that and let me know here, that would be great.

Yes it wors perfectly fine on my side now. Huge Thanks!

 

Man this was a lousy month to stop drinking Mountain Dew

lol.

It's ok, a couple more days left!

Link to comment
Share on other sites

Okay, all seems to be working now (phoenix server).

 

Thank TCH guys for being on top of this.

 

It would nice to know how this all happened.

 

Maybe after all the dust settles...?

Edited by Ray_Bman
Link to comment
Share on other sites

TCH Family,

 

I have tons of emails from many of our concerned clients. Please understand that we have all been working non-stop since this am to get things corrected on our servers. I will personally reply to each and everyone of you that have emailed me. Just asking that you all be a little patient on my replies. Anything urgent should be sent thru our help desk as we have full 24/7 support waiting to help you.

 

Thanks and we will update again shortly.

Link to comment
Share on other sites

I have replied to all the emails from clients. I am now working on returning phone calls.

 

Things are running smooth on the servers. We have identified the method of this hack and we will be releasing a post sometime tonight with full details.

 

We are all still monitoring, watching and working any tickets that come in.

 

I still have not got to a Mountain Dew, however I have run out of milk and Marlboro lights. lol

Link to comment
Share on other sites

Concerning events today, I am wondering about the difference between index.html and index.php. I really didn't think that I had an index.php on my website. Is it possible to have both file types but not know it? Is an index.php really needed? If it is, where is it located? I can't see it and I've looked for invisible files...

Link to comment
Share on other sites

This is why I love this web hosting service. Super duper fast response (I don't think it could have been faster), help everyone fixing their web site and even do it for them, continuous monitoring of servers, transparent, super helpful even on crazy time like this, on forum active very knownegeable staff, always professional (even on their pictures they were a tie or well dressed.. see... ALWAYS professional), acknowledge a possible discovery of security breach, and solve it so it doesn't happen again. A very long day for TCH, but for us, users, it's like if nothing happened.

 

I think TCH deserve a big round of applauce for their great effort, and huge thanks. I don't think they could have handled such suprise situation any better. :)

Once everything is cleaned up, let's all dance! :D

Link to comment
Share on other sites

Due to this issue we are going ahead with kernel reboots that we had scheduled for June. These reboots are rolling out now 10 servers at at time, A-Z by host name and should have a downtime of less than 5 minutes per server. As always, if there reboot takes an excessive amount of time, we will post to the appropriate server forum.

Link to comment
Share on other sites

While our site is online and apparently without problem we still can not access cpanel. Is it "normal" (i.e. you are still working on and this is a problem common to many sites) or should we update our ticket?

Our site is on Arlington

Link to comment
Share on other sites

My guess is that the techs are still working on this, however you could follow Bills advice and submit a ticket.

 

TCH Family,

 

I have tons of emails from many of our concerned clients. Please understand that we have all been working non-stop since this am to get things corrected on our servers. I will personally reply to each and everyone of you that have emailed me. Just asking that you all be a little patient on my replies. Anything urgent should be sent thru our help desk as we have full 24/7 support waiting to help you.

 

Thanks and we will update again shortly.

Link to comment
Share on other sites

Most of my sites showing index of page this morning so am manually uploading the current index page.

 

Not just index.php files affected - index.htm pages too.

 

I'm hoping TCH doesm't now overwrite them with old back up index pages.

Link to comment
Share on other sites

We have identified the method of this hack and we will be releasing a post sometime tonight with full details.

 

Is that post going to be here, or was it made somewhere else? I'm very curious what happened.

 

A client of mine also reported being hacked (my reseller account is on jandoon), and her password isn't obvious. Based on the extent of the damage, I'd guess they got access to the server as a whole, not individual accounts. I suppose that this loser could have guessed a lot of client passwords (I've seen a hack on a discussion board that I used to help on where they accessed accounts whose passwords were the same as their user names), but I don't think that happened in this case given that my client's password isn't that poor.

 

By the way, you can use a service like my-ip-neighbors.com to find out who else is on your server.

 

Steve

Edited by TCH-Thomas
See follow up post
Link to comment
Share on other sites

I have deactivated your link Steve as it´s not hosted by TCH and also may not be good to have as active link in a public forum due to what have happened.

 

By the way, that service does not seems to show all sites. I´ve tried a few now that I know are hosted on a specific server, sometimes they showed up, sometimes not.

Link to comment
Share on other sites

Are index pages still being reinstated from backup? I still have a large number of sites showing Index of / pages.

 

Do I need to manually reinstate each one (I've spent the morning doing most important ones).

Edited by georgem
Link to comment
Share on other sites

Good morning TCH Family,

 

What a long day yesterday was. Things are much calmer today, the help desk is not filled with tickets and my staff appears to be humming along and drinking mountain dews as normal.

 

I have completed a overview report and am making some final reversions and will be posting it shortly.

Link to comment
Share on other sites

Quick Synopsis: Our network was the target of a large scale website defacing attack on Sunday, September 28th commencing at around 8:30am. A large number of servers were involved in this attack; however our internal servers and infrastructure were not vulnerable to this attack. We were able to get control of the situation by 10:00am and had identified the method of attack. The attack hole was closed and we started a methodical review of what happened. Once we learned that index files were the attack focus, we wrote a custom script that would review each index.* file on an effected server and look for certain keywords that were present in the hackers file placed on the effected servers. If this file was present our script deleted it and restored your last available backup from our weekly backups. In theory a simple method of restore but it took us a bit to get it working correct. I am confident that there are still some users on the effected servers that will wake up tomorrow and find that their index pages are blank. I should say during the chaos we ended up breaking php mail handlers on some servers, however this was fixed and was a simple mistake on our end.

 

What the attacker did: At this time, the attack does not appear to have been any more malicious than replacing the web site's home page. The defacement worked by replacing index files in all public_html directories with the attacker's index.php file.

  • The hacker’s main goal was to deface websites.
  • Our entire internal infrastructure, including Domain Management, DNS Clusters, Billing, Forums, Help Desk or any Client Credit cards or personal information was not targeted or accessed. Furthermore, the entry method that was used was not an available entry point on any of our Internal Infrastructure.
  • The attacker did not obtain user passwords. This apparently was not the hacker’s goal. The hacker used a system exploit to allow him to access index files. As always though, it is prudent that you update your cPanel and FTP passwords.
  • The exploit used to gain entry has been blocked.
  • The issue was with an authentication system that was in use for an application controlling cPanel. The exploit itself was something not seen before and has been forwarded to the prospective people for review.

Where we are at this point: We have a complete understanding of the attack method used and we have already taken needed steps to block future similar attacks.

 

In the future: While not related, we are going to start phasing out our dual version PHP platform servers. We have many servers that are running two versions of PHP. This was done for clients that asked for us to allow them time to upgrade their scripts to the new 5.3.x versions of PHP. The time has come and passed and we will be removing the dual versions of PHP and will only be rolling with the latest version of PHP on all our servers. Like I said this is not related to the web site defacements, however it is something that we need to address. We also ran software and system updates on all servers last night and rebooted every shared / reseller server. These updates were originally slated for June to coincide with the coming PHP upgrades, however in light of this incident; we choose the precautionary approach and completed them now.

 

In addition: Even though our team responded quickly to disable the attack and limit the exposure to our customers, and even though the damage done was straightforward, this was a serious incident. At TotalChoice we are very aware of the threats that every website faces on a daily basis and are committed to protecting our server farm and all of our clients. Our security record up until this point has been excellent. Whilst we do view this incident as a failure, please know that this security hole was not known to anyone and if it had been we would have taken prior action.

 

Final thoughts: I want to personally thank each staff member for the amazing work done during this incident. More importantly I want to thank each and every client for their understanding and support after all without the clients we would have nothing. Please let us know if you have any questions and or concerns about this matter and we will gladly answer them for you. Once again, thank you for your business and support.

Link to comment
Share on other sites

Well, personally, I'm totally impressed by the rapid response of TCH.

 

Imagine my thoughts as we were on the way out the door to Church on Sunday, when the phone rang with another of the Church members telling me of the horrible things on our Church website.....

 

I told my wife as we left that TCH would handle it, and they did.

 

Thank You !!!

Link to comment
Share on other sites

GoodBYtes's post above (#157) expressed much the same as I was thinking, better than I can ^.

 

Especially appreciate the prompt status announcement in the forum. The notification email was how I learned of the incident. I found it a non-jarring way to be notified.

 

I want to personally thank each staff member for the amazing work done during this incident.

 

Truly amazing, skillful, and fast. Thank you.

Edited by SteveW
Link to comment
Share on other sites

I found this problem on my server (tyson) several weeks ago. I use a cron job to run a diff between the previous day's dir listing (of everything) and the current dir listing, and that's how I caught the problem. I worried that my own laptop had been compromised and they got my password. I'm glad this thread showed up, so now I know it's unlikely that my password is compromised.

 

Database passwords are nearly always stored in code, which I believe was accessible to the hacker, so I'd recommend changing them.

 

There were files other than index.php in my account that I had to restore from my local copy.

 

I also created a self-healing mechanism so that if the write time on my main index.php file changed, it would restore itself from a backup.

 

I can post more info about these two solutions if anyone is interested. Oh, and I'm also available if anyone needs a good software guy.

 

Dave.

Link to comment
Share on other sites

Thanks to all of your reports and further review, we were able to identify what files we didn't restore during our initial fix. Which as you have seen would be any index file that was not named index.php.

We have now completed a new run to restore these files.

 

In order to not undo changes or restores you have already done, this run to into account the following before restoring.

-was the file one we logged during our initial fix

-if so, was the file missing in public_html

-if so, was the file not equal to index.php

-if so, was the backup file clean of the defacement

-if so, restore

 

Thank you all again for your patience and please let us know if you continue to have issues.

Link to comment
Share on other sites

Don't forget that any "start.php" file was also affected.

 

Newer versions of vBulletin forums use "forum.php" for their home page but there is still an "index.php" that redirects to it for old bookmarks. My "index.php" was affected but it didn't cause any problems for members.

Link to comment
Share on other sites

This is badness. I really hope totalchoice is able to rectify the situation.

 

My sites are on kalee, and my forum users were getting the tigermate hack notice again today.

Edited by orkan
Link to comment
Share on other sites

Guest
This topic is now closed to further replies.

×
×
  • Create New...