For all the talk about education and how to help students learn, I sometimes forget that outside our school district is a cold world. I was reminded how cold it is recently.
We had two big problems over the last couple of days with network connectivity outside the building. I worked with Illinois Century Network, Google and DerbyTech today to narrow down to the problem. The two issues were two different degrees of severity:
- Mail messages were not being delivered from the archive and spam filter to the mailboxes hosted at Google.
- We were subject to, for lack of a better term, a cyberattack that exploited our proxy web content filter.
The mail service was only partially functioning as of late Wednesday night as Google appeared to have changed the address where our mail was directed. As a result, when an email message was passed to someone in the school district, it got to our mail archive/spam filter and got stuck as our filter could not pass on that message to Google to be placed in the appropriate mailbox. Around 10AM Thursday morning I was able to isolate that problem with the help of DerbyTech and a backlog of messages then started to pour out of the archive to Google.
However, I noticed that this emptying of the archive was taking a lot longer than normal as there was more than normal emails coming in to my mailbox alerting me to someone tripping the content filter.
As it turns out the proxy used by students to filter their laptops at home was compromised. In my attempt to make things a little more transparent for kids to get online at home with their school issued and filtered laptops, I removed the necessity for a username and password. Normally, students would have to, while at home, enter in a username and password to access our filter. If they did not enter in that username and password, then they could not access the filter. Instead, they did not have to enter in that username and password with the changes I made. Their traffic was still getting filtered just like it does here at school. The term for this is an open proxy. Closed proxy would be if they had to enter the username and password.

Unfortunately, the open proxy was available for others to use. My thought was why would someone use a proxy that has a filter on it? However, I have evidence in the form of our firewall sending messages of inappropriate filter hits to my email. As it stands now, there are over 16,000 filter hits from outside our district from places like Dover, New York City, Frankfort (Germany), Berlin, Belgrade, Beijing and Caracas. I was able to trace to these locations from the addresses these hits were coming from. What these people were able to do was run their internet traffic through our network to make it appear that their traffic was coming from our district instead of, say, China. We openly proxied their traffic to other websites making it appear we were the ones requesting access to certain sites.

Here is why spammers would use our open proxy. A known spammer in Belgrade cannot simply post comment spam on a blog because the blog knows that spammer address and will immediately reject it. However, my district is not a known spammer, and has an open proxy. The spammer will send their spam through their firewall to launder the comment and make the blog think this comment is legit. The good news is that our filter blocked over 16,000 attempts to access certain inappropriate sites. Unfortunately these were spam computers programmed to send out several hundred attempts a minute to post comment spam to blogs around the world.
Because of the number of requests being processed through our filter the network loads were very very heavy starting mid day on Wednesday and through to mid day Thursday. That would explain the problems with Google searches coming up with an error. This also explains why there were intermittent outages at our three bulidings and from parents outside the district trying to access online grades as all these people were trying to get down the same pipe as the spammers. The junior high head end did not suffer as much because they only had to go out of the network-- not come into and then go out of the network. Google saw the incredible number of requests for information from our district and saw that as a possible attack or virus. Thus, some people would get a screen like Google was forbidding them to access their site, and making them input a random set of letters into a box to prove they were not a spammer or virus.
Though the earlier inbound mail problem (problem #1) was actually something totally separate, the problem with email messages coming back as undeliverable starting around noon on Thursday was totally related to this cyberattack (problem #2). As spam messages continued to mount around the world on blogs, originating from our address, the realtime filters began to tag us as a spammer. Comcast, AT&T and Barracuda began bouncing messages from us. The error messages received on the bounce I had never seen before, but clearly provided links that we were now beginning to be known as a spammer address.
With this information, and with help from Illinois Century Network, Google and DerbyTech, I was able to close the proxy (DerbyTech), ensure the attacks were slowing down (Illinois Century Network), and begin to restore our address as a legit non-spammer (Google). As it stands now, I have re-instituted the need to input a username and password to use our proxy. I have taken the steps to get our district cleared on any blacklists that would reject our emails, and it looks like most are again accepting our emails. Our network loads are back to normal and ICN will continue to monitor for the next few days to ensure there isn't another attack. The spammers will continue to attempt, but as they need to enter a username and password they should begin to remove us from their list of open proxies.
The good that has come from this incident has me looking more at ways to prevent this in the future. Setting up an SPF record for our domain is a start. I also cleaned up the accepted mail relays from our email archive system. And I am now considering implementing more stringent authentication for the proxy which is available to us, but in my limited capacities I have to find the time to set this up.
Hmm. I wonder if those students in my district are still learning?
P.S. Getting off these blacklists on the surface seems simple, but it truly is a painstaking process. I even found one blacklister in Germany who wanted to charge me 50 Euros to be removed from their list OR I had to wait 7 days since the last indicated malicious activity. THAT sounds like robbery.
Labels: derbytech, Google, ICN, network, open proxy, security