Lessons Learned While My Website Was Down

by Luke Muehlhauser on May 19, 2010 in News

Andy Walters, Professional Genius

Hi, there. Common Sense Atheism is back!

I lost one day of data, but I’m glad to be online again, mostly thanks to freelance web developer Andy Walters. Seriously, Andy is a genius and he saved the day.

I’m sure there are a few glitches here and there, so please comment to let me know what you find. Anything: broken links, weird display, etc.

Here’s the story: Literally hours before I was going to shut down my XP laptop for the last time and never turn it on again (because I’ve switched to Mac), I got an infection. Hours later, I got reports that my website had a virus on it. It seems the virus stole my FTP password and uploaded a virus to my site, as has happened to thousands and thousands of people these past few months.

Alas, if I knew then what I know now, this whole issue could have been solved in 24 hours with no loss of data by purchasing a $150 service from WPSecurityLock.

But no. I figured this was a good opportunity to get more acquainted with the backend of Apache Web Server and SQL and WordPress. Wrong!!!!

So here’s my first lesson learned: When your website goes down, that is not the right time to try to become an expert in web development. The correct time to become an expert is (1) before your site goes down, or, more likely (2) never, and then call the experts first you dumbass.

I tried to figure stuff out on my own and ended up only digging myself in deeper, which is when Andy Walters swooped in to save the day. Unfortunately, this whole process was dragged out by many days because I work a full-time job and didn’t have much time to work on the site.

And though I had initially configured backups, I found out during this time that they had been failing since March. So the other lesson learned was: When you’ve invested tons of time into some project, it’s worth a little time to make sure your backups are working.

Finally, if you visited my website from the 14th to the 17th on a Window computer, you may have been infected by that Javascript injection attack my website suffered. You may want to scan your machine with Combofix and MalwareBytes.

That is all for now. Your regular programming from Common Sense Atheism will resume shortly. Thanks for your patience.

Previous post:

Next post:

{ 52 comments… read them below or add one }

Jason Berberich May 19, 2010 at 12:02 pm

Great to see that you’re back up and running. One backup solution I highly recommend is a WordPress plugin called Automatic Wordpress Backup. You install it, setup an account on Amazon’s S3 storage service, and tell it to backup daily, weekly, monthly, or manually. Then you forget about it (unless you end up needing it).

Your monthly bill for S3 storage would probably be less than a couple of dollars – well worth the security of knowing you always have a current backup of your site.

  (Quote)

Grumpy Bob May 19, 2010 at 12:11 pm

Does the Javascript injection attack you suffered affect Linux machines?

  (Quote)

al friedlander May 19, 2010 at 12:15 pm

Welcome back.

Yeah I was always confused by this, because it seemed to be affecting a lot of people. Nothing ever infected my computer though; I suppose I have Norton to thank for this.

  (Quote)

lukeprog May 19, 2010 at 12:16 pm

Grumpy Bob,

Good question. The answer is no.

  (Quote)

RA May 19, 2010 at 12:32 pm

I don’t know if I got infected or not because I’m a computer idiot. I did unleash Spybot Search and Destroy just in case. What’s the opinion of the tech intelligentsia here on that program for this sort of thing?

  (Quote)

Chris K May 19, 2010 at 12:51 pm

I’m pretty sure I went to college with Andy – and lived a few doors down from him in the dorms one semester!

  (Quote)

Andy Walters May 19, 2010 at 1:42 pm

Chris King? Haha, I remember you, bro.

  (Quote)

Atheist.pig May 19, 2010 at 2:01 pm

RA

If you havn’t noticed anything strange on your pc by now your probably OK. Spybot won’t pickup a bad virus anyway. But if you wanna make sure download avast and schedule a Boot time scan.

I got infected and nearly lost 300GB of data (avira anti-virus let the poxy virus through), had to rebuild the partition table in linux and get my data back. Finished with windows for good now!

  (Quote)

Chris K May 19, 2010 at 2:04 pm

Yep! good to see that you’re a thriving web genius!

  (Quote)

Andy Walters May 19, 2010 at 2:29 pm

lol, Luke overplayed it a bit. I shot you a fb friend request.

  (Quote)

lukeprog May 19, 2010 at 2:32 pm

RA, Spybot hasn’t been significantly updated in years.

  (Quote)

MauricXe May 19, 2010 at 2:35 pm

Good to hear everything is back and ok :)

I don’t know where to put this, but I thought this would be good to display atheist charity:

http://www.kiva.org/blog/2010/05/18/first-kiva-lending-team-reaches-2.html

  (Quote)

Bill Maher May 19, 2010 at 3:25 pm

I didn’t have anything to do in between looking at porn.

WB to the enternets Luke.

  (Quote)

Jeff H May 19, 2010 at 3:49 pm

Good to see the site back. I did a little fist pump when I saw that. Do you know what the virus was called? I know they typically have funky names, but I’d like to make sure that my antivirus has the definitions for it….

  (Quote)

Hermes May 19, 2010 at 4:24 pm

What’s the opinion of the tech intelligentsia here on that program for this sort of thing?

It’s a good tool, yet tattoo this to your brain;

* Security is a process, not a product.

Also;

* Loss of physical security means loss of assurance of any security.

Meaning: If the system is not under your direct physical control, or has been compromised remotely, all bets are off. You can’t be sure that any bit on it has not been compromised. This is one of the reasons why epoxy is often injected into USB ports on secure systems, and why many of them run on independent networks that not only don’t connect to the Internet but actively monitor those connections. On any system, even your own private ones, it is a good policy to lock down and limit as much as you can.

While Windows (servers and clients) can be secured, it typically is not. I remember spending two weeks way back when learning enough to moderately secure a XP system for my father, and this is after knowing quite a bit of Windows security at the time as well as having a general Unix mentality towards security.

The culture at Microsoft and of 3rd party Windows developers is often to ‘make things simple’ first, and to apply security second if at all. (Note that isn’t an issue with OSX, so the ‘make things simple’ mantra is actually not correct and covers up design limitations to Windows as well as cultural aspects that are being addressed more seriously in general by Microsoft and Windows developers over the last few years. A secure system by design should not need a virus scanner as viruses become highly impractical as a means of attack.)

The products made for Windows often don’t follow good security practices and often enough require that security be thwarted; you are encouraged to do the wrong thing. In some cases, you are given no choice and must do the wrong thing to use the software at all or effectively. Someone who is savvy with Windows security not just with using tools can deal with those issues, but this takes a great deal of knowledge to do properly. Once again, we’re back to process over product.

Similarly, regardless of operating system, some other tools such as languages or libraries (example: PHP) do not encourage good security practices. Can a PHP app be written with security in mind? Like Windows, the answer is yes. Yet, it’s not the default and if you are using Windows or PHP or other systems (OS or development) that encourage ignoring implementing a secure system or working around existing security, then the chance that the system will have security holes will be drastically increased. That’s why OSX and Linux (and other Unix or Unix-like operating systems) seemingly have very few security issues, but Windows systems are often drenched in security add-ons that scan the system looking for known ‘bad patterns’.

Well, consider this. If the bad guys have already set up shop on your computer — how do you know your security software is effective or even actually on your side? Additionally, if your security software is addressing only those issues it already knows about, what will happen when something new — and unknown — shows up? If you are the first victim, or the only target, and you rely on a pattern matching scanner programs to be your only defense — you may be nuked and may never find out about it.

There are only a few ways to deal with this, and that includes (but is not limited to) cutting off as many entries into your system as possible.

Bottom line: Spybot and other tools are excellent and effective. In the case where your system has been taken over, it may give you a false sense of security.

  (Quote)

Hermes May 19, 2010 at 4:30 pm

Luke: Did some of the blog posts get nuked as well?

During the down time, I wrote up detailed replies to comments made to these posts …

http://commonsenseatheism.com/?p=9047

http://commonsenseatheism.com/?p=7894

… and now can’t find them and can’t remember what the blog topics originally were.

FWIW, I wrote responses to these people;

Al Moritz (large reply)

Alex (short comment)

  (Quote)

Hermes May 19, 2010 at 5:56 pm

Luke, one for the list;

Missing: http://commonsenseatheism.com/cpbd.rss

  (Quote)

isom May 19, 2010 at 7:29 pm

1. Glad you’re back up!
2. Glad you’re now a Mac user
3. More importantly, glad I’m a Mac user when I came to your site while it was down ;-)

  (Quote)

Geeky Atheist May 19, 2010 at 7:44 pm

Hey, welcome back. I’m glad you were able to recover so much. I was worried how devastating it would have been to lose everything.

I just wanted to point out that the podcast-only rss feed seems to be broken right now. It returns a 404.

  (Quote)

Evolution SWAT May 19, 2010 at 8:31 pm

Glad you didn’t lose everything. This is a good reminder for everyone to make sure their backups are working, because this happens to lots of people.

  (Quote)

lukeprog May 19, 2010 at 8:57 pm

Jeff H,

Sorry, I don’t know, as I never actually saw it. I only saw the bad code, and got reports from others about a virus.

  (Quote)

lukeprog May 19, 2010 at 8:59 pm

Hermes,

Alas, one or two days of data was lost, including maybe 2 new posts and probably 100 comments.

  (Quote)

lukeprog May 19, 2010 at 9:04 pm

isom,

Lol @ #3.

  (Quote)

Hermes May 19, 2010 at 9:27 pm

Luke, got it. While I’ll likely toss my comments unread, I’m sorry to see you go through any pain.

If anything, it is one way people in general learn what is normally not obvious even outside a very small group of security minded techs. It’s a hard lesson. That said, even the experts make mistakes.

Another rule to keep in mind;

* Never assume your resources have a value that has anything to do with your data.

Meaning: If you have data that is valuable to the world … or just valuable to yourself … the resources you have may be what is most enticing and your data is entirely irrelevant. You are (usually) not a unique snowflake to the attacker. You are an unlocked door, food in the fridge, and a hook with keys on it to the car outside. As it’s not a personal attack on you (usually), but a scattershot attack on dozens to millions, that may lead to attacks that secure yet more resources.

Here’s the attitude: You could have a van Gogh on the wall. Big deal. Do you have an address book? Do you have a phone? Where’s your wallet? Can you show who’s place I should go to next, and can I abuse them too by not even going anywhere?

  (Quote)

Briang May 19, 2010 at 10:41 pm

Luke,
If you ever decide to go back to the PC. I’d recommend giving Linux a try. I’ve been using it for years and it doesn’t have the virus problems like Windows.

  (Quote)

lukeprog May 19, 2010 at 10:57 pm

Briang,

I try Linux every now and then. I’m still running a VM of Win7 on Mac because there are about a dozen Windows programs I use for which I haven’t yet found Mac equivalents. And Linux has come a long way but I just don’t have time to find ways of doing everything I do in THREE operating systems! I like Ubuntu; though. Very user-friendly.

  (Quote)

Hermes May 20, 2010 at 3:24 am

I like Ubuntu; though. Very user-friendly.

It’s good; I’m using it now. That said, OSX is still a form of unix, so the two are close cousins and neither have the same problems that Windows tends to have.

  (Quote)

exrelayman May 20, 2010 at 11:21 am

I am not very computer savvy so I don’t know how significant this may or may not be. When I clicked on the link to combofix in the post, Firefox crashed. I am using windows xp. I do a full system scan with avira, malawarebytes, and trend micro on WE, Fr, and Su respectively and none has reported any problem. Am reporting this incident per your request.

  (Quote)

Hermes May 20, 2010 at 2:13 pm

Firefox — especially older versions of it — crashes. The current release is 3.6.3 and seems to be quite stable. I recommend the good but not perfect NoScript add-on if you browse heavily or visit aggressive web sites.

  (Quote)

Haukur May 20, 2010 at 2:21 pm

Finally, if you visited my website from the 14th to the 17th on a Window computer, you may have been infected by that Javascript injection attack my website suffered.

Yeah, I got it. Cute little infection, actually. Had some fun fighting it.

  (Quote)

Hermes May 20, 2010 at 3:41 pm

Being curious, I checked a little into WordPress security hardening. Since it’s written in PHP, my previous comments apply; tread lightly and spend extra effort to nail everything down you can. This article has some handy tips — many of them basic though some are close to cringe worthy. WordPress.org has a good set of tips as well and seems to be written by someone who knows what they are doing.

One measure of how hardened a system is, is to have it so that even if most or all of the service and server passwords are known to the bad guys, your servers still can not be trivially compromised. By all means, choose good passwords. Keep them safe. Just don’t rely on that secret to be what keeps them out. Chances are if it is, there are many holes you aren’t even considering.

Back to reality: As a regular person renting part of a shared server with shared services this is difficult or impossible to achieve, let alone if you aren’t managing the instance of the server yourself. It becomes much more manageable when you manage the box or the VM itself and not just the services. Compromise, compromise.

  (Quote)

lukeprog May 20, 2010 at 4:53 pm

Haukur,

Do you remember it’s name?

  (Quote)

Haukur May 21, 2010 at 2:18 am

The malware I got when visiting your site last Saturday was “Internet Security 2010″. It’s a funny thing – it tells you (correctly!) that your computer has been infected by malware and needs to be cleaned up. But of course you shouldn’t believe the things your malware infection tells you – it’s almost like a philosophical counterexample to something. Then it tries to sell you some bogus anti-malware software. Better not give it your credit card number.

It’s not too hard to remove if you do the right thing right away. Unfortunately, I was a bit too gung-ho and deleted the malware files without realizing that the malware had changed a registry setting, making those files necessary for login. I ended up having to use the Windows CD to boot up and copy a file to the name specified in the registry yada yada, cost me a couple of hours but left no permanent damage.

  (Quote)

Steelman May 21, 2010 at 11:13 am

Luke, glad to see you’re back up and running. I’m working my way through your CPBD podcasts from the beginning. Great stuff. Keep it up!

@Haukur: My boss brought in his similarly infected Vista laptop for me to clean. The fake AV software blocks any real AV software, web pages where AV software can be downloaded, and even the USB ports so you can’t load anything from an external drive. Bad stuff.

Booting into safe mode allowed me to copy RKill and MalwareBytes onto his system from a flash drive, and eradicate the infection. Then re-enabled Windows Update (automatic) and Windows firewall, and installed Microsoft Security Essentials.

I highly recommend the Firefox browser with the NoScript and Adblock Plus extensions. There were some Java based advertisements that caused havoc a couple of months ago. You didn’t even have to click on the ad, just view a web page in which it was embedded.

  (Quote)

Gil S. May 21, 2010 at 11:58 am

Congratulations! You’ll begin to realize that a Mac is about as enchanting as switching from theism to naturalism. You’ve found the truth, don’t stray from it xD Anyways, I’ve been using the Mac for 6 years now and I think I can safely say I’m a power user. If you need any help, feel free to contact me ;)

  (Quote)

Hermes May 21, 2010 at 4:54 pm

I highly recommend the Firefox browser with the NoScript and Adblock Plus extensions. There were some Java based advertisements that caused havoc a couple of months ago. You didn’t even have to click on the ad, just view a web page in which it was embedded.

I second the recommendation on NoScript. Thanks for the comment on the Java based adds; I wasn’t aware of that problem/exploit(?). I’d appreciate some details or a link on that one if it’s not easy to find.

  (Quote)

Michael May 22, 2010 at 9:58 am

I got infected but was able to remove the virus. However, I recently started a blog on blogger and had linked to your website on it. I think I might have accessed your website using that link once or twice. Could blogger websites be infected, and could they transfer the virus to those that access those blogs? If so, is there a way to tell if the virus is there and remove it? My new blog is http://www.evaluatingarguments.blogspot.com/

  (Quote)

lukeprog May 22, 2010 at 10:18 am

Michael,

I don’t know of any way the virus could jump blogs. It would have to steal your password, too, somehow. But I’m not an expert on website infection.

  (Quote)

Hermes May 22, 2010 at 10:51 am

Michael, linking from your blog to another blog won’t do anything.

Your local Windows infection, though, opens up the possibility that you might have a problem. If you manage your Blogger account from the infected machine then it is possible. Without knowing the details of the virus and your Blogger configuration, though, I would expect that it is *possible* that the Windows infection you have could scrape up passwords and harm some of your other accounts.

The details matter, though.

Someone who deals with specific WordPress and Blogger security might know off hand if there is a risk and would likely be able to ask a few basic questions about your Windows and/or Blogger configuration. Unfortunately, I’m not that guy.

  (Quote)

Michael May 22, 2010 at 10:58 am

Thanks for your help! I appreciate it!

  (Quote)

Steelman May 22, 2010 at 12:10 pm

Hermes said:

Thanks for the comment on the Java based adds; I wasn’t aware of that problem/exploit(?). I’d appreciate some details or a link on that one if it’s not easy to find.

Here’s a March 22, 2010 Cnet report about malware in ads being served by major ad providers. Apparently, this first started happening last year.

From the article:
“Viruses and other malware were found to be lurking in ads last year on high-profile sites like The New York Times and conservative news aggregator Drudge Report.com, and this year on Drudge, TechCrunch and WhitePages.com. The practice has been dubbed ‘malvertising.’

“Now, researchers at Avast are pointing fingers at some large ad delivery platforms including Yahoo’s Yield Manager and Fox Audience Network’s Fimserve.com, which together cover more than 50 percent of online ads, and to a much smaller degree Google’s DoubleClick. In addition, some of the malicious ads ended up on Yahoo and Google sites, Avast claims.”
[...]
“Users don’t need to click on anything to get infected; a computer becomes infected after the ad is loaded by the browser, Avast said.”

I’m not sure what has been learned by the AV community regarding this malware vector in the last two months, or if Avast’s information has been corroborated.

  (Quote)

Hermes May 22, 2010 at 12:26 pm

Thanks! That rings some bells, though I thought (wrongly) that those abuses were dealt with severely.

  (Quote)

lukeprog May 22, 2010 at 2:19 pm

Stelman,

Oof! That sucks. Glad I’m on a Mac now.

  (Quote)

Haukur May 23, 2010 at 4:59 am

I highly recommend the Firefox browser with the NoScript and Adblock Plus extensions. There were some Java based advertisements that caused havoc a couple of months ago. You didn’t even have to click on the ad, just view a web page in which it was embedded.

Yup, it was Java alright. I do run Adblock Plus but I haven’t got in the habit of running NoScript.

  (Quote)

Hermes May 23, 2010 at 6:28 am

FWIW, NoScript can cause problems itself at times because it is fairly aggressive. Because those problems require a knack in dealing with software problems, I can’t recommend it to everyone. That said, after setting a few sites in your white list, it will be transparent most of the time especially if your habits are consistent.

If you find yourself scratching your head wondering why something doesn’t work as you expect, just select “Temporarily allow all this page” from the NoScript menu (S icon lower right), and reload the page. If that doesn’t do it, NoScript may have to be disabled or another browser or instance of Firefox should be used for that page.

  (Quote)

RA May 24, 2010 at 9:08 am

FYI-I think lukeprog.com has the same problem.

  (Quote)

lukeprog May 24, 2010 at 12:09 pm

RA,

Are you getting a warning there? I’m not.

  (Quote)

RA May 24, 2010 at 12:29 pm

Warning: Visiting this site may harm your computer!

The website at http://www.lukeprog.com contains elements from the site westcountry.ru, which appears to host malware – software that can hurt your computer or otherwise operate without your consent. Just visiting a site that contains malware can infect your computer.

I also experienced the same Java icon running that I did on this site when it was hacked. I had the same experience on both sites.

I definitely think it’s got a problem.

  (Quote)

RA May 24, 2010 at 12:50 pm

What had happened was:

I went to lukeprog to check out your post on keeping your computer safe. This was right after you took the site down. That’s when I had the Java icon running. And I thought to myself: “I wonder if this is hacked, too? It’s doing the same thing.”

When I checked it out today, the site comes up and then comes the big Warning message from Google.

  (Quote)

lukeprog May 24, 2010 at 1:19 pm

D’oh! I’ll have to take care of that tonight. Evil, evil, Javascript injectors!

I wonder why I don’t see those warnings, even when I’m at work…

  (Quote)

RA May 24, 2010 at 1:35 pm

You probably know this. But just in case, the edit option for these comments is no longer available. If that can be re-enabled, it would be useful.

  (Quote)

lukeprog May 24, 2010 at 2:26 pm

RA,

Okay, I’ll add it back when I have a chance.

  (Quote)

Leave a Comment