Uh-oh: was the company site hacked?

Anyone who runs a web site knows that they’re constantly under attack. You only have to look at your log files to know that hackers running site-scanners are constantly hitting your servers looking for unpatched vulnerabilities to exploit.

One of the servers I wrote for Guild Wars 1 — named AcctHttpSrv — was designed to behave like a web server. When you received an email from the Guild Wars account management system and clicked on the link it contained, AcctHttpSrv would convert that click into a database transaction to update your account information. Pretty standard stuff.

For debugging purposes I configured AcctHttpSrv to log all unknown requests it received for later review. It turned out the number of errors logged was larger by far than the number of valid transactions. Peeking at the error log provided a view into an entirely different universe, a firsthand look at the dark side of the Internet, each line an exploit that had likely been successful on other sites in the past.

Just the other day one of my readers pointed out that this site started misbehaving: the articles RSS feed stopped functioning. Since I hadn’t logged in for several weeks it likely wasn’t a change I had made. My first guess was a new WordPress hack as there have been several recently, so I ran some security scans (nothing found) and rolled back the site to a previous backup, which fixed the bug. My thanks to Steven Y. for pointing out the problem.

So I thought I’d use the occasion share a story about a web deploy that went awry.

At a previous company I worked at the web team was getting ready to roll out a completely new game site. The old site was in need of a revamp, but rather than re-use the crufty and difficult-to-use web content-management system (CMS) that had been built by an external web agency our team chose a new CMS that would make it easy for non-developers to post game news and updates with no programming required.

The web team was under the gun to deliver the site on a deadline because the game’s platform and services were cross-linked with it, and consequently those services could not be rolled out until the main web site went live. Several missed deadlines, late nights, heroic efforts and so forth, but eventually the site content was ready.

The web team pushed the content to the servers and started forwarding traffic from old servers to new. We were all happy to see the results of their efforts live at last, with everything running smoothly.

Then one of the web developers made a startling announcement. “Greg” (not his real name) announced that he was seeing an “adult” image displayed on the site. Ack! We’ve only been live a few minutes and it has been hacked already?!?

The web team, operations team, and security team jump on the problem. Let’s verify this — RIGHT NOW!

Oddly, nobody besides Greg can reproduce the problem. Should we switch back to the old servers? Brazen it out and call it a feature?

While it wouldn’t be impossible for the site to be hacked, it would be challenging. See, we weren’t actually using the CMS yet — it wasn’t ready in time. We were actually hosting the entire site as static files. Instead of running the site on a smart web server capable of dynamically generating content, we ran the site on the equivalent of a dumb file server.

As we approached the deadline it became clear that, while the site’s content was complete, there were some CMS issues that prevented it from being deployed. But we HAD to deploy the site! So I hacked together a trivial solution which scraped all the web pages from a QA-test server and we posted those files on the public web server. For the technically inclined:

wget --input-file list-of-urls-to-scrape.txt --mirror

Since we weren’t running a dynamic language on the web server, the attack surface was small enough that we could verify the configuration was correct pretty quickly. Hmmm… what are we missing; the clock is ticking!

After a few minutes, Greg sheepishly announced that he had discovered the problem. It turned out that the web browser he was using on his mobile phone (Opera) sometimes rendered pages incorrectly, leaving images from a previously viewed page visible while downloading a new page. Well … you can guess the kind of pages he was checking out at lunch just before the site was deployed.

I have to give him props, though. A lot of folks would not have said anything and we all would have been held in suspense for months. He owned up and saved us all a lot of grief, and we didn't have to roll back to the old site. Problem solved!