6 May 2010

MAIL SYSTEM PROTECTIVE OUTAGE.

There was a mail outage from approx 2:00am to 4:00am on 5 May 2010 [Server times].  This was a protective response to an apparent attack on the mail system. At approx 1:50am there commenced a series of probes of the mail system, followed by an intense attempt at cracking passwords. The mail system responded by blocking ALL of the mail system to prevent any further attempt at corruption. This left ALL users with requests to reset the password, but that mostly did not resolve any problems as the mail system also blocked all POP3 access.

Attacks continued for another 2 hours, but could not do anything as the system was 100% protected by the simple blocking of all access.

Once the attacks stopped, we then re-enabled access to all users. This was done in a slow and careful process and all users were running normally by 6am on 5 May 2010 [Server times].

If you had changed your mail password any time during the outage, that password change is wrong. You should revert back to your password that was being used before the attack. If it is not possible to do this, then please request a password reset.

We are sorry for the inconvenience, but in this instance, the protection system worked to avoid an even bigger problem for all users.

WHY WERE WE ATTACKED?
This is of course a difficult question to answer. HOWEVER, in the hours prior to the attack, we had received a number of emails from customers forwarding a message [which seems to have originated in Rumania] advising customer of a mail system change and to click on a link to update their settings. NEVER EVER DO THAT. It is probable that this started the avalanche which led to the attack.

WE WILL NEVER SEND YOU AN EMAIL,
WITH A LINK,
ADVISING A CHANGE IN SYSTEM SETTINGS.

 

31 March 2010

SERVER MIGRATION [VMWARE Corruption concluded].

Following on from the previous problems yesterday. we have now completed migration from a backup configuration on 30 March 2010. This should complete the maintenance which first detected the system problems in VMWare hypervisor.

The server system was unavailable from 13:00 30 March 2010 GMT-5:00 until 16:00 30 March 2010 GMT-5:00. All nail services for this period were queued. There was no data loss during this period.

Again, we apologize for any inconvenience during the disruption. We are working on recovery of mail for the period 10:00 29 March 2010 GMT-5:00 and 15:00 29 March 2010 GMT-5:00. for those users who have subscribed to our mail archiving services.

30 March 2010

VMWARE Corruption

At approximately 15:00 Monday 29 March 2010 GMT-5:00 there was a corruption in the VMWARE Hypervisor in our Data Center. The Hypervisor is the software that manages the virtual machine environment. Normally these kinds of problems have no effect on user operations, but in this very rare instance there was a problem.

Our technical staff attempted a recovery, but the backups for Sunday evening were also corrupted resulting in a recovery back to the backups of Saturday.

Customers will note the following issues:

Any ticket created between 10:00 27 March 2010 GMT-5:00 and 15:00 29 March 2010 GMT-5:00 is lost.
Any Web Site updates between 10:00 27 March 2010 GMT-5:00 and 15:00 29 March 2010 GMT-5:00 is lost.
Any mail WHICH WAS NOT COLLECTED OR DELIVERED between 10:00 27 March 2010 GMT-5:00 and 15:00 29 March 2010 GMT-5:00 is lost.

Mail Issues - additional notes:

Any mail which was already delivered prior to 10:00 27 March 2010 GMT-5:00 will have reached the intended recipients without problems. Any mail which was queued for delivery was lost. There were about 10 mails queued at time of failure and these were junk return mails of no importance.

Any mail which had been received by our system between 10:00 27 March 2010 GMT-5:00 and 15:00 29 March 2010 GMT-5:00 AND HAD NOT BEEN COLLECTED by a mail client is lost.

General notes:

It is extremely rare that such a sequence of events would occur and result in lost data. We have an extremely careful backup, disaster backup and recovery blueprint which has been tested regularly to ensure validity of the process. What we did not expect was a triple failure, which seems to be what happened in this instance. We are well covered for duplicate failures.

Today we are examining carefully all procedures to see how to avoid data loss in this extremely rare situation.

It is worth mentioning that ALL of our systems run in a Virtual Environment with current technology provided by all leading hardware and software manufacturers. We follow the standard procedures as recommended by all vendors to ensure 99.9999% uptime and ZERO DATA LOSS due to equipment/software failure. We use RAID6 storage to allow duplicate concurrent disk failures and the disk systems are backed up on a continual rotating basis every 24 hours. Snapshots are taken at 24 hour intervals of all virtual machines. We have full ON-LINE UPS power systems which are backed up by TRIPLE in-house generators. The electrical system is lightning proof. The network system is fiber, triplicated. We have remote storage facility connected by high speed fiber. We are not military strength, but it is certainly close.

We will resolve this current incident to further ensure reliability of customer systems.

1 February 2010

WINDOWS XP and INTERNET EXPLORER 6 or 7 or 8

 

If you are using Windows XP and any version of IE you are at risk. If you are using IE6 you are at EXTREME RISK and will be attacked no matter what level of other security you are using.

 

There is no longer any good reason to remain with Windows XP. You should be planning a move, or better yet upgrading, to Windows 7. The basic model of Windows VISTA was extremely sound. In Windows 7 many of the issues which were attributed to VISTA have been resolved, enhanced or rendered more friendly, resulting in a MUCH BETTER user experience. Windows 7 comes with IE8, which also is much improved in security when run in the Windows VISTA/Windows 7 memory model.

However, IE is and will remain a troublesome product and should be avoided where possible. Firefox is the best logical alternative to IE, but Chrome and Safari are also good choices.

USE THESE BROWSERS AS YOUR PRIMARY BROWSER and only use IE where web sites demand use of IE.

Unfortunately some of the web sites which demand IE are the very sites which should not be using IE. These tend to be sites from ON-LINE BANKING and are locked in to using ACTIVE-X components which tend to be the most insecure area of a system. Of course these are not the only security issues, but forcing the use of IE in itself is a security issue.

DO NOT FOR ONE MINUTE assume that users of LINUX [or other systems] are immune from these browser problems. Any system which uses a linear memory model can be subject to corruption via hacking. Protected memory, paged memory, various hardware protections DO NOT resolve the problems of poor coding, which is what allows these attacks to occur. In Windows VISTA and WINDOWS 7, the linear memory model was changed to a random memory model and this makes it extremely difficult for hackers to take advantage of "knowing" where code will be located.

 

The message remains. If you are using Windows XP, you are vulnerable. Upgrade to Windows 7 AS QUICKLY AS POSSIBLE.

 

Corkscrew Technical