Friday, August 26, 2011

MAPI Connections Exceeded

I remember when I worked as a consultant one of my clients suddenly saw a rash of disconnects from Exchange 2003. The culprit was narrowed down to event ID 9646. and ultimately I believe we adjusted the number of connections allows via the registry and all was well. The situation has reared its ugly head again in my very small business environment and has gotten me to revisit both the problem and the solution. Quick investigation of Google hits reveals that this wasn't simply an issue for older versions of Exchange/Outlook either, so it appears it's all still relevant. Here's how I approached it this time.

First of all, Outlook 2010 doesn't show you an error message indicating the problem. It simply shows that the mailbox is disconnected in the bottom right of the window. I tried the standard send/receive to jolt it online, exited Outlook (making sure no processes remained in Task Manager), and checked the profile settings. The user had just rebooted her computer so I didn't bother trying that again. I verified that the computer could ping the server using the FQDN. I tried to launch Outlook using the /rpcdiag switch. It flickered but never actually connected to the server, which helped me to understand that the program wasn't getting a connection to the server at all. I checked the event logs on the machine, which yielded no information. Then I went to the server.

Immediately at the top of the system log was Event ID 9646. I rolled my eyes at this familiar face and began to refresh myself on my options. I knew I could simply increase the max connections but I wanted to take some time and make sure there weren't better alternatives out there, so I took some time to do a little extra reading (one of the luxuries of being in-house IT). I couldn't take too long though because the end user was standing over my shoulder, clearly more than a little anxious to get back to work. I could have also just used TCPView to kill some MAPI connections, but this may have only been a short-term solution and I wanted to be done with this for a bit.

What I wound up doing to solve the issue on a more permanent basis (as well as provide an easy resolution should others begin to experience the same thing) was create a security group, and give it "View Information Store Status" privileges via ESM as per the suggestion in this Microsoft KB. I have also installed exmon to be able to view more detail about connections in the future should this come back up.

Tuesday, August 16, 2011

HTML Not Rendering Properly at Localhost

I use a Mac. I wanted to test out some changes to a website before uploading the edited documents, so I saved a copy of the site locally to my MAMP installation (MAMP is a Mac version of WAMP, which is Windows/Apache/MySQL/PHP rolled into one). I edited the page using TextEdit and tried to view it locally in my browser. It showed the actual content of the HTML file. I checked the file extension and everything was fine. Other html files I'd created previously also worked properly.

Turns out that if you create an HTML file using TextEdit, you have to go into the Format menu and select the option "Make Plain Text". 'Nuff said.

Monday, August 15, 2011

SYN Attacks

Today while checking my Apache logs I saw a message that said: "possible SYN flooding on port 80. Sending cookies.". I'm familiar enough with DoS attacks that this made me immediately concerned so I went online to find out what I could. Googling this message turned up a number of good security articles about what it means and what can be done to try and protect oneself. Essentially a SYN flood is when a malicious host sends the first part of a TCP handshake (a segment with the SYNC flag set) and never responds to the SYN+ACK that it receives. The attacked server sits waiting for a response, holding the half-open connection in its backlog queue and attempting to re-send the SYN segment. Over time the backlog queue fills up and the server starts rejecting any new connection attempts.

Documented ways to confirm that you are actually under a SYN flood attack include using netstat to view half-open connections (listed as SYN_RECV) and using wget to try and load your site from the actual host to see how long it takes (doing it from the local host takes out some network considerations). In general though slow performance of your website or an inability to connect at all along with these kinds of logged messages are a good indication.

In my case, I ran netstat a few times over a number of minutes and while at one point I did see upwards of 8 SYN_RECV entries, for the most part there weren't a lot of half-open connections. I also checked netstat to get a total list of open connections by IP and didn't see anything suspicious there. Even though I knew that the source IPs were likely spoofed, I also used Geo IP tool to look up where the attacks were "coming from" just to satisfy my own curiosity.

Even though it seemed that I wasn't likely the victim of a SYN flood attack at this point (maybe some kid playing around somewhere and trying to get his/her newb hack on), I was very interested in what I was reading and also thought it best to use this as an opportunity to be proactive. I read about some standard settings to the kernel and TCP stack options that can be implemented in both Windows and Linux to help mitigate problems caused by this kind of activity. One of the basic ones involved implementing SYN cookies.

By enabling SYN cookies, the sequence number being sent back to the malicious host is used to complete the handshake. Usually this number is randomly generated, but in this case it is constructed according to specific rules. If your server then receives an ACK with that sequence number, it completes the connection. However it does this without relying on the backlog queue so you don't have to worry about it filling up and rejecting new legitimate connections. You can tell that this is effect when you see the message: possible SYN flooding on port 80. Sending cookies" in your logs.

So, it seems like I had this turned on on my server already, although in the file that controls this setting, /etc/sysctl.conf, the line for this was commented out. I uncommented it just to be on the safe side.

Other things that can be done are to increase the backlog queue size, change the timeout values for the SYN (in other words decreasing the number of retries and consequently how long it keeps half-opened connections in the backlog queue).

Resources I used for getting this information include:

Saturday, August 13, 2011

Google Password Militia

I had the unfortunate experience of finding out that Google is very strict and a little heavy-handed with their password policy. I had forgotten my password for logging in to YouTube, but it turned out that I had a session open in a browser window that I hadn't closed so I could go in and change my password to one that I would hopefully remember. I changed it to something random, thinking it was solely for YouTube. It turned out that it changed my Gmail password as well since the two are now linked (quite against my will mind you). This meant that my email password for Gmail was now different from the rest of my email accounts, which was an important thing for me. I use a password for email, another for social networking, another for financial, etc. I tried to go back and change my Gmail back to my old password and Google won't let me. No warning about how it's bad to reuse the same password, just a line that says you can't. Period. Not that you can't use the last 3 even. You cannot use a previously used password. Crap.

I then tried to set it to a few different passwords, hoping maybe I could age it out. Turns out Google is a little overbearing about your password. Too short, not strong enough... I'm definitely all for security, and normally I wouldn't have any issue with creating a strong password. But, I don't really need Google forcing it on me. It's my account, my security. Give me a couple of warnings and stern admonishments, then let me make my own decision. Don't force it on me.

So now I can't use my favorite password, and it's made me cranky.

Friday, August 12, 2011

VMWare Fusion Lock Files

Here's a neat one.

I used to work off a Macbook provided by my previous employer, complete with VMWare Fusion for running Windows VMs. When I left I backed up my computer with Time Machine. A year later I was able to get a new Mac (finally; they're darn expensive!) and I restored my old backup to it so that I could have all of my original settings. It worked like a charm, except I couldn't actually launch my saved Windows 7 VM. I tried building a new one using the old files with no success.

I finally figured out what the issue was. I didn't realize that when you suspended a VM, the host puts a lock on the files so that it can't be used by another disk at the same time. When I did my backup, the VM was suspended, not shut down, so there was a lock on it. Therefore every time I tried to spin it up on the new machine it thought the VM was already in use. All I needed to do was find the .lck file and delete it.

The next step was finding the lock file. On a Mac (not sure about Windows), the virtual machines are stored in your Documents under a folder called Virtual Machines, and each vm is a single file. I tried using spotlight to search for .lck files with no luck. I knew these files existed because when trying to import I was able to point to a .vmdk file. A thoughtless right-click brought up an option to "Show Package Contents", and there they were! I deleted the .lck file, and my vm powered up with no problem. I love it when a plan comes together.