TCH-Ryan

As part of our commitment to maintaining our servers with the latest technologies available, we have developed an implementation of PHP 5.3 that also allows full retention of PHP 5.2 support. The deployment will make PHP 5.3 the servers default PHP handler and a simple htaccess option is available that provides the ability to revert to PHP 5.2. However, in the interest of avoiding likely issues of sites breaking due to incompatibilities with PHP 5.3, what we will be doing is setting the htaccess option on all sites to force PHP 5.2 usage, which will effectively mean nothing will have changed for you compatibility wise. Then, when and if you decide that PHP 5.3 is something you would like to use or experiment with, you may remove the entry from your sites htaccess file. During our installation of the combined PHP 5.3/5.2 setup, we will check to see if a .htaccess file exists under your users public_html ( /home/user/public_html/.htaccess ), if one does not exist, we will create it for you. Then, the new PHP 5.2 handler type will be added/appended to the .htaccess file ( AddType application/x-httpd-php52 .php ), this will force the account to utilize PHP 5.2, it will be inherited by all paths under your public_html That is it! If at any time you want to test your domain with PHP 5.3, you simply remove the entry from the htaccess file -- if the content you are testing breaks, just re-add it and you will be back on 5.2. As always, if confused or in doubt about any of this, please reply here or open a support ticket and we will be glad to assist you. PHP 5.2 handler type: AddType application/x-httpd-php52 .php PHP 5.3 handler type: AddType application/x-httpd-php5 .php TIPS: - If you have the 5.2 type set in your public_html/.htaccess, and would like to test 5.3 on PHP scripts in, for example, public_html/test/, then you would create a public_html/test/.htaccess and add 'AddType application/x-httpd-php5 .php' to it. Now, the test/ path will be forced to 5.3. - If you would like to have 5.3 as the default across your site, you only need to remove the 'AddType application/x-httpd-php52 .php' entry from public_html/.htaccess and the entire site will then default to 5.3. As with above , you can set the .htaccess in any specific path and add the 5.2 handler to force that version in just that path. - For PHP options specific to 5.2 and 5.3, you can create php.ini files as needed in the applicable paths, for example, if you have the 5.2 handler type set under public_html/oldapplication/, you can drop a php.ini file into that path and its options will apply to PHP requests under that path. Once again, the bottom line here is that, be default, everything will be transparent and PHP 5.2 will remain the version your sites run unless you explicitly remove the 5.2 handler from your .htaccess file. Further reading on features that have changed in PHP 5.3 can be found at: http://www.php.net/manual/en/migration53.deprecated.php

This afternoon at roughly 4:00PM EST the TCH network began receiving an inbound DDoS attack that measured in the order of 600mbit, the larger factor of this attack was not so much the amount of traffic as the amount of packets, upward of 700,000 packets per second. The traffic combined with excessively high rate of packet flow created a crippling situation on our network edge that required considerable effort to source the target of the attack, isolate the attackers/target and then mitigate the attack off our network. At the moment the attack is currently still ongoing however it is being successfully filtered at our network edge and should present no further issues. It is important that we make clear the DDoS attack of Jan. 26th and this attack are in no way related. The distinction being that the attack on Jan. 26th was targeted at our carriers, not us directly, as part of a larger string of Internet attacks that occurred on that day which took down many sites across the Internet. This attack however was directly targeted at us which allowed us the ability to immediately take action and filter it out as promptly as possible. We regret the downtime this situation has caused and understand that in light of the recent network outage on Jan. 26th along with this afternoons outage, this may naturally cause some concerns to our customers. Let me again assure you these are isolated and unrelated incidents, however we are not one to hide from issues and we will take away any lessons we can learn from today's event that could help us restore service more promptly in future situations like this. Here at TCH, network attacks are not new to us, we have dealt with them in many shapes and sizes for years and most of the time when we receive a (D)DoS attack, it is stopped at our network edge and filtered without it ever becoming visible to any customers. We have in place at our network edge advanced firewall, intrusion prevention and bypass hardware that detects an array of malicious attacks, we are constantly tuning these systems to provide better results but sometimes even the best infrastructure will still fall victim to large and concerted network attacks. We will undertake a full review of our network protections and if need be, perform upgrades or revisions to better handle attacks of this magnitude in the future. We apologize for the clear inconvenience this situation has caused and if there are any further updates, we will immediately post them.

Jason, You are correct indeed in what your friend tells you, to a very limited extent. The Apache Web Server version 2.2.16 does have three notable low-impact vulnerabilities, however I will explain why and how they do not apply to our servers. The first two are vulnerabilities CVE-2009-3720 and CVE-2009-3560 are issues are specific in a certain apache feature, the threading option MPM Worker which TCH does not use, we use MPM Prefork for threading on TCH web servers thus mitigating these two vulnerabilities right off, in other words, they do not at all apply to us. Finally, the vulnerability CVE-2010-1623 is related to an addon module for apache that TCH does not use (http://httpd.apache.org/docs/trunk/mod/mod_reqtimeout.html) which again, mitigates this threat for us as the vulnerable software is not in use on our servers. We take security very seriously here at TCH, protecting your web sites, our servers and network at large is important to us and we always strive to do better. If you have any further questions or concerns never hesitate to drop in a help desk ticket or throw up a forum post and we will be glad to address whatever it is you may be inquiring about. Thank you for choosing TCH and I hope you have a great weekend.

I have tested this module personally and though it provides some very nice benefits on a per site basis, it has some serious draw backs on what it does to a server as a whole. The first is that it increases the servers i/o through heavy caching of modified page content, though this is one-time disk writes, it is nevertheless another cache path that needs to be maintained on the server and will receive lots of writes with dynamic content. Further, the module also increases CPU usage through heavy optimization/compression of images, js scripts and static pages, which though provides good benefits is a high price to pay when we can just use mod_deflate to compress with less load. Then there is the memory increase the module causes which adds as much as 1MB to the resident memory footprint of each web server (apache) thread, which can quickly on a loaded server add up to hundreds of MB extra memory used. All told, modpagespeed provides some nice features but it is still a very new module with very real draw backs that need to be addressed before it can find a place in any serious production environment. We will keep an eye on the development of the module and at such time that it is appropriate and will provide meaningful benefits to users without compromising the performance of the server, it will then receive strong consideration for deployment.

The failed switch has been corrected and the affected subnet is now back online, we apologize for the clear inconvenience this has caused and thank everyone for their patience.

We are currently aware of and investigating a switch failure on our network that is affecting the 208.76.83.1/25 subnet. We ask for your continued patience while this matter is further investigated, an update will be posted as soon as possible, thank you.

If you go ahead and check the website now you should find that the error is now gone. I have set the configure.php file chmod 444 for you, since your user still does own the file you can at anytime set the file as 644 if you need to edit it, this can be done with most FTP applications without much fuss. If you require further assistance on this matter, please feel free to open a help desk ticket at https://www.tchhelp.com and we will be glad to assist.

Our conversion process expects that htaccess files will not already have <IfModule php_config> elements in it, these will be added for existing php_flag/php_value options in htaccess files so adding these in advance will create unforeseen problems, so I would discourage it strongly. Permissions are handled during the conversion process, any files/paths owned by user nobody will be set as the account user and any world writable (777) files/paths will be set to 644/755 as standard.

Though we would rather see .htaccess files used to configure PHP, we do fully support the use of php.ini files on suPHP servers.

We have in place a conversion process that will take any existing php_flag/php_value options from htaccess files and wrap them in the <IfModule php_config> element tag, this allows for a seamless migration and avoids any errors. Specifically though, once a server is migrated to suPHP, without the element tag surrounding the php_flag/php_value options, the site will throw error 500 pages which is why we are putting so much time and effort into ensuring that our conversion process for .htaccess files is one that is reliable and consistent.

This morning at aprox. 2:40AM EST, our edge router suffered a failure in one of its gigabit line cards which serves traffic to our rack mounted distribution switches. The issue was quickly identified and the line card power cycled in an attempt to recover it. Though this did work and we restored service to the line card with roughly 10 minutes of downtime, it quickly became apparent there was a series of errors coming off the card indicating that another failure was possible if not highly likely. After some preparation, at 3:30AM EST we removed the line card exhibiting issues and replaced it with a spare card, this process took 5 minutes and service was quickly restored with the new line card appearing to operate as intended and no further errors being reported on the device. I should stress that we have on hand multiple spare parts for our networking equipment including a spare router in an identical configuration as the primary unit, should there have been the need, we could have and would have immediately switched service over to the standby router. This is the first hardware failure on our router in the 3 years since it has went into production, which we are certainly more than pleased with but will continue to strive for that same or better excellence going forward in our network infrastructure. We apologize for any inconvenience these outages may have caused and thank you for your understanding.

Thank you for your input cbutler, at this moment we have no intention to push on php 5.3. It is on our agenda for things to explore and we do intend to get there eventually but there is no clear need or demand for it at this time, till that changes or a security issue emerges that compels us otherwise, we will likely remain on the 5.2.x stable tree. When we do upgrade, everyone will be given plenty of notice through the forums along with a road-map of what servers will be upgraded with specific dates, allowing everyone to transition as seamless as possible.

Network maintenance concluded on schedule and new hardware on our network edge is operating stable. We apologize for the delayed follow-up but we wanted to make sure everything was performing as it should have been before any final updates were made.

We will be conducting network maintenance on the evening of Tues, February the 16th between the hours of 10 to 11PM EST (-0500), this maintenance will be the final in a series of fail over tests to validate new network hardware at our network edge. We do not expect any substantial down time from this testing however a 5-10 minute outage window is likely, we will provide further updates as the tests are underway. Additionally, we will be upgrading our backup2 server from 500gb hard disks to 1tb, bringing the raid 6 array size up to 12tb. This upgrade will have no service impact on any users apart from network backups being unavailable from this unit while the upgrade is underway.

The maintenance has concluded with no issues and no adverse down time, we thank everyone for there patience.

We will be conducting a routine fail-over test on Monday, February the 8th between the hours of 11:00PM through 12:00AM (EST -0500). We expect no downtime or at worst a 5 minute outage window, further updates will be posted as the maintenance is under way, thank you in advance for your patience. P.S: Dick will be kept clear of all networking equipment on this maintenance:

All network maintenance has been completed and there should be no further outages, we thank everyone for there understanding and again apologize for any inconvenience this may have caused.

We are currently running behind schedule and the maintenance has been bumped to 11:30 through 1:30AM, we apologize for the inconvenience and thank you in advance for your understanding.

We will be conducting reboots again this evening to push out a revised version of last nights kernel that corrects issues with r1backup agent, local firewall services and the network driver on certain servers. In addition, this new kernel revision is binary compatible with CentOS/RHEL 4 kernels being that it was built off the same kernel source tree as the standard kernels. If you would like to use this patched kernel on servers outside of TCH you are free to do so, the packages can be downloaded from: http://bala.tchmachines.com/kernel-2.6.9-78.0.30.tch.EL/

We have the last 3 days been tracking a local root vulnerability in the Linux kernel, the core element of all Linux operating systems. This vulnerability is unprecedented in scope, effecting Linux versions going as far back as 8 years which prompted extra consideration in how we handle it. Here at TCH we operate a network that is dominated by Linux, to say we took this matter very seriously would be an understatement. It was decided after evaluating the threat this vulnerability poses to our network, dedicated servers, and shared/reseller clients, that waiting any longer on an upstream update was not reasonable. Originally there was an estimate of Saturday 1900GMT for upstream updates but this fell through prompting us to take action. In addition to a lack of a reliable upstream update for this issue is the fact that this vulnerability is being actively exploited in the wild with publicly available attack code on many security and underground web sites. At this moment, we are rolling out to all Linux servers on our network an updated kernel version that will close this vulnerability while maintaining version compatibility with future upstream software updates. This effort in retaining version support will allow our dedicated clients in addition to our own support team to resume normal update practices with tools such as 'yum' or 'apt-get' and not have to worry about conflicting versions against our in-house kernel update. Please do not be alarmed if you experience an outage temporarily on dedicated, shared or reseller servers, we thank everyone for understanding the urgency of this matter and if you have any questions or comments please feel free to submit a help desk ticket at https://www.tchhelp.com. Reference: http://www.theregister.co.uk/2009/08/14/critical_linux_bug/

As a developer myself, sometimes it is hard to strike a balance between legacy feature support and moving forward with changes that will enhance performance, reliability or security (or all of). Let me be clear, I do develop php code and extensively use ereg in allot of my stuff, however I do think that going forward and given sufficient time people can safely transition to perl regexp functions with minimal effort. The actual transition to php6 is a bit of a ways off in any production environment and likely will resemble transitions of the past in that for a short time, php5 compatible binaries will still be available on servers to allow legacy applications to work without major overhauls in the short term. Identify where you use ereg functions, start making it habit in new development projects to stop using it and over time back track older code and replace it. At the end the day, php developers are just trying to do what they feel is right for the community at large, which as a matter of preference most agree perl regexp is far superior than maintaining a stand alone regexp implementation.

The nature of shared and reseller accounts is such that the onus is on us to remedy any inbound attack, for the sake of every client housed on that particular server. We can take any number of actions to mitigate attacks, all with varying degrees of success dependent on the type of attack that is occurring. Generally however if an attacker is persistent enough the ultimate outcome may be that we are required to shut off the site in question being attacked or renumerate it as so attackers are no longer hitting a live address. The bottom line is, our goal is to ensure the integrity of our network, servers and as a whole all clients we host on those assets, this sometimes means that we must make hard decisions against those being attacked in the interest of preserving uptime, performance and stability for our clients.

At present, there is no method to allow the creation of stored procedures without granting super privileges, which in a shared environment is a very bad thing to do. This can be overcome by looking at semi-dedicated or fully dedicated server where you get full root access to the server.

The servers effected by this are now back online and should all be responding to services. If you have any issues please do let us know. We are sorry for the issue and are taking investigative steps to determine the cause of this short outage and will report our findings as soon as we know more. Thank you.

This upgrade has been completed. The new APC power distribution unit has been put in place and all the servers on this PDU have been powered back on. Downtime was less than 5 minutes for the effected servers. Thank you and good night.