Tuesday, September 4, 2012

World of Discovery

I started a new job recently. I can't tell you much about the company or what we do. I signed a NDA you know, and I don't want to risk any information being a giveaway. I seriously doubt that would happen, but the market my company is in is a pretty small one as far as I can tell. Well, I suppose it could simply be small to me since I had no idea that there was such a market before I started my job search. Kind of the same way I was surprised to find that there was a market for what my last company did.

So, I'm a Linux Systems Administrator for this company, and it's been a pretty interesting and challenging ride right out of the gate. One of the most challenging aspects has been the fact that two of the team members with whom I'd interviewed quit within a couple of weeks of my starting. One guy quit (as in walked out) two weeks before my first day, and the second guy gave his two weeks notice during my first week there. Suddenly there was a lot of work to be done and a lot fewer people to do it, so the luxury of time was no longer there. This network and the technologies in use are different from anything I've ever encountered before so I really wanted that time to familiarize myself.

In getting to know the network I looked for some sort of documentation or diagram that would give me a high-level view of what was in play. I'd come to expect these kinds of things from my time as a consultant, and it was one of the first things I put together when I started at Gravel, my previous company. I'm a visual person in addition, so Visio diagrams are my bestest friends. With Gravel the infrastructure was small enough that I was able to piece together what was going on in short order. I mean, in our production system we had all of two network devices: a firewall and a switch. Not very difficult to map out. In this new environment however the network is labyrinthine, to say the least. There are switches and firewalls everywhere, and no clear documentation as to how it's all laid out.

My first instinct was to use some sort of utility to walk the network and find the devices out there. I started out thinking of common tools I had used before, but quickly found that the layout of the network was not conducive to using these kinds of tools because this wasn't a simple layer 2 network. There were several layer 3 devices scattered throughout, which meant that the standard tools I might have used wouldn't work.

Ultimately I wound up using a combination of manual tools. I used Cisco's built-in CDP tool because CDP was enabled on all of the devices, which was luck on my part since lots of admins tend to disable this feature for security reasons. With CDP I was able to get a rough layout of the network. What it led me to realize was that this was a network unlike any I had encountered before. In my admittedly limited experience, networks tend to look like this:

Very basic stuff. Routing is handled by the ISP, the firewall acts as a layer 3 device and bridges any VLANs in existence (of which there are usually only the 2 configured by default on the ASA 5505), and there's a basic layer 2 switch at the access layer. The network I've inherited looks more like this:

What's different? For starters, the ISP uplink comes in to a switch instead of a firewall. The switch is layer 3 capable, having a basic license that enabled it to do RIP routing, but it's not setup that way. It's simply a layer 2 device configured for 2 VLANs: VLAN 99 (outside) and VLAN 1 (inside). There's a load balancer hanging off one interface that has virtual IPs for the machines that are to be accessed from the outside. This load balancer sits outside of the firewall. There are two firewalls in a active/standby configuration that appear to only have a role in protecting machines like the mail server. There are two layer 3 switches configured in HSRP mode that act as the default gateway for all the servers and switches. And then there are a ton of additional layer 2 switches scattered throughout the environment. 

I used CDP to sneaker my way around this network and as its complexity became obvious my mouth dropped open in awe. My brain pretty much fizzled and refused to process this. I couldn't for the life of me understand why a layer 2 switch was serving as the point of ingress, why the default gateway was not performing that role, and why the firewalls were sitting off to the side protecting only select devices instead of the entire network. Figuring out the answers to these questions required ramping up on some technologies and advanced networking that I hadn't had a need to deal with. 

To add to the confusion, the network was a flat Class B address but there were ostensibly 2 VLANs in place: VLAN 99 which was the VLAN used for the external connection (the link to the ISP) and VLAN 1 throughout the rest of the network. I'm on a new adventure, folks! Everything I thought I knew about networking is being challenged. In fact, I find myself sometimes over-analyzing a segment of the network to the point that I forget the basics. For example, in trying to understand how in the world a particular bit of traffic is possible I totally gloss over a basic networking principle like, say, ARP, just making myself more confused for no reason whatsoever. As I really dig in to this new environment I'll be posting snippets instead of the long rambling posts I used to do, which were the results of projects. My current project here is to facilitate us moving out of one network cage into another at the data center that we're using, which has many smaller parts involved. If I tried to cover it all in one post I'd essentially be writing a long, unorganized book! 

Stay tuned...

No comments:

Post a Comment