Monitoring - too often overlooked

Outages, unplanned downtime, critical events...

It is not a matter of IF they will happen, its a matter of when.

Outages are bad.
        Outages during business or production hours is very bad.
                 A user or client knowing before IT about an outage... inexcusable.

You would think this is just common sense yet it is one of the most common issues I find when starting a new engagement. (and probably one reason why the position opened up)

What to do?

IT is in the business of technology and providing information to the business units. This is where IT needs to be a customer to their service (See post on IT groups, the NOC is an IT group that is a customer to IT).

Some useful tools for varying budgets:

Budget Friendly:

• MRTG - "Multi Router Traffic Grapher"  (open source)
• Tool provides network utilization graphs.
• Useful to see if WAN links are saturated where users would complain the network is slow.
• Simple to complex dashboards can be created to give a view of the entire network landscape
• Zenoss Open Source Edition
• Can monitor all the devices on the network
• Sends an alert so you know something is down
• Web status page
• Install can be difficult... read the install guide well and you'll be OK
• Nagios (open source)
• A long used monitoring tool that has many plug-ins to improve the product like Cacti
• Sends alerts and can be setup with some very nice web status pages
• configuration can be challenging to some
• Servers Alive (low cost)
• Very easy to setup monitoring
• Free edition can monitor up to 10 items, standard or enterprise editions come with a small fee
• Email alerting and additional plug-ins (such as POP check for monitoring email flow)

Enterprise class:

These systems may not be budget friendly, but you definitely get what you pay for. Hopefully the company you are with either has these in place or can fit them into the budget.

Note:
Don't be surprised if the place you starting at owns monitoring software licenses yet haven't found the time to deploy the solution.

Setting up monitoring the right way takes time and often projects for IT are set to lower priority than projects to the business. In one way, it's great to improve productivity and provide great business intelligence reports; however, if your not maintaining the house it can fall down around you. If it does fall, where will the company be then?

• Microsoft System Center Operations Manager - aka SCOM
• In my opinion, the Mercedes of monitoring.
• The 2012 version can monitor Windows, Unix, Linux, Network devices and many software and hardware vendors are providing management packs to allow their solutions to also be monitored.
• Talk to your MS reseller as there are Windows server bundle deals which can save you a bundle by rolling up server licensing, endpoint protection and the System Center suite.
• Multiple methods to deliver alerts (email, sms, etc)
• Full robust dashboards
• Performance tracking and trending reports
• Direct ties into other System Center Suite modules that if all are fully deployed is a wonderful sight to see (and probably a great IT environment to work in)
• Alternative options to Microsoft
• NetIQ App Manager Suite
• Dell OpenManage
• SolarWinds
• CorrelSense Sharepath
• Monitors more than just a server's performance or a network link... this is end-to-end application monitoring of the user experience.
• Bottlenecks are instantly identified and ends finger pointing between teams
• Reports on performance out of normal operation with really impressive drill down dashboards

Of coarse there are many other solutions out there that are on-premise or offered as a cloud service. The key factors in any monitoring is that:

1. Minimize the false positives
• If too many alerts are generated and safely ignored, does anyone see the critical alerts or do they go into a junk mail folder?
2. IT uses the tools wisely.
• If no one is watching the alerts, its has no purpose
• If you need 24x7 support, you need a NOC
3. The staff can easily support and maintain the software 

Whats in your toolkit?

What are some tools in your bag of tricks?

     In the 2000-2010 years I was lucky to have lead many projects to merge IT teams due to corporate M&A. Each one unique to what needed to be done and the people I had the privledge to work with. One of the first things we always did do was to compare our toolkits. 3rd party products which helped us work magic at low costs.

Below are a few examples:

• PAL - Performance Analyzer of Logs
• Perfmon is a great tool but not many sysadmins use it to troubleshoot as results are hard to interpret or due to uncertainty in what counters to select
• PAL gives you pre-built templates to load into perfmon and then, the greatest feature of all.... import the trace output logs & PAL analyzes it and presents the results in an informative report
• Great for analyzing Exchange, AD, IIS, and Sharepoint servers
• Knoppix
• A bootable CD or USB drive with free emergency data and system recovery tools
• An ERD (emergency recovery disk) is critical for any support person.
• Reset the local admin password
• Recover data from a failed Windows sytem
• Repartition the hard drive or extend volumes
• Network analysis and administration tools
• ERD Commander is another nice alternative
• Dameware mini-remote
• A nice utility that allows you to take control of a users PC so you can fix their problem without leaving your desk.
• Handy if you dont have a corporate remote administration package
• Hyena
• An administration tool which can connect to multiple domains.
• Useful in exporting AD objects and their properties.
• Searches are often easier than using AD users and computers to display custom AD attributes for multiple objects at the same time. If your not good at LDAP querries, this makes it simple.
• Ability to edit multiple objects at the same time
• Ghost
• Utility that is very common. Many times where a hard drive is failing and wont stay online long enough for Windows to let you copy everything off, Ghost may be able to help. Put the drive into an external USB drive enclosure & you can grab a image of it to recover someones data.
• Powershell Plus by Idera
• If you dont know powershell yet, you will need to learn a little which this tool can help.
• Powershell Plus has a large library of scripts built-in to instantly help even PS veteran that includes scripts for AD, Exchange, SQL & Sharepoint
• Idera has many other useful tools for SQL and Sharepoint
• RedGate SQL Toolbelt
• Fantastic backup with compression tool
• Monitoring and SQL admin tools (too many to list)
• Veeam One, Veeam backup
• Comes in a free version so if your budget is tight, it can help you greatly with your virtual environment to montior and backup.
• The full licensed versions win awards year after year at VMWorld
• Training videos are provided on the Veeam website
• NetIQ Domain Migration Suite
• This product may have been discontinued but it was the best utility I found for migrating users, groups, computers and email.

So.. what do you have in your own bag of tricks?

Interview Questions

Prepare for the Interview

The interview is the time to make an impression and most candidates do this by preparing their answers to potential questions. My tip is to turn it around by playing offense.

When it comes to the point in the interview with the hiring manager where you are asked if you have any questions, have a few of these questions ready. 

 

What would be my typical day and who would I be working most closely with?

This question is multi purposed.

1. The question is designed to have the interviewer visualize you in the position as part of the team.
2.  It gives you a good glimpse into what your work day would be and allows you to judge if this position is what you are looking for.

Who are the 'stars' on the team that I would learn the most from and what would be the best traits to pick up?

1. This question is also designed to have the interviewer visualize you as part of the team.
2. Demonstrates that you are committed to learning and self improvement by learning from the team's best people.
3. Gain  

What are some of the current challenges the team is facing that I may be able to help resolve?

1. How the interviewer responds will give you a good indication if they are seriously considering you or not. If the interviewer gives you a few items that you can contribute on, its a good sign.

Continual development is critical for people in IT and something that is important to me. How do you feel I could best improve?

1. Even if you do not get the position, you gain some valuable information that can help you prepare for the next interview.
2. The question emphasises your commitment to your career. If you are not learning, you are falling behind the curve.
3. This questions landed me one of my early IT positions. The director interviewing me at the time was noticeable surprised and he gave me one of the most important pieces of advice I have received. He thought for a minute and then told me;

 "Keep your mouth shut and your ears open"
 

Sound advice in such a concise response.

• As IT, we have access to very confidential information and there is no room for someone who leaks info.
• Listen to not only what is being said but also between the lines, you can deduce what people are challenged or concerned with most. Resolve a problem before the users ask IT for help and suddenly you are perceived as being innovative and a true asset.
• Listen to what you IT colleagues have to say. There are many ways to approach a project or problem and bright ideas can come from anyone. Collaboration also is what makes the strongest teams.

The director may have been trying to tell me I simply talk too much, but I like my interpretation better. To this day, we do remain friends.

A day in the life of: Helpdesk Support

The Helpdesk

Group Role:
The helpdesk is THE face of IT. Users look to the helpdesk for answers to their problems as well as to help improve their productivity by installing new software or getting a new computer/laptop. As a helpdesk technician, you need to appear as always having the right answer and be completely trustworthy. After all, a user who hands over their laptop with all of their critical and confidential data needs to be confident you wont lose their files or look at the contents.

• Hardware support:
• Desktops and laptops from multiple vendors such as HP, Dell, IBM, and Apple. Replace failed components and issue new systems. 
• Printers, scanners and sometimes basic mobile device support depending on the environment. Typically hardware support is minimal triage before calling a 3rd party for more advanced support (IE setup a new printer on the network, change toner and rollers but not take the thing apart)
• Software support:
• Microsoft Windows or Mac OS: Install, troubleshoot, patch, upgrade
• Microsoft Office, Adobe, print devices: install, troubleshoot, patch, upgrade
• Install and Other 3rd party software the user may require
• Troubleshoot errors the user may see
• Internet Explorer alerts or pages not loading
• No network available (wired/wireless)
• Unexpected crashes of an application or OS
• Account support
• Account creation and removal
• May be the responsibility of the helpdesk or as organizations get larger, there is more segregation of duties. The task may be assigned to the security team or even be delegated to HR and created when someone is on boarded or off boarded.
• Password resets, Unlock accounts
• Add/Remove groups to Active Directory or update memberships to a group
• Sometimes this is delegated to the Helpdesk or managed by the Wintel (server) team.
• Computer accounts in Active Directory
• New PCs are joined to the domain when issued to a user
• Old PCs are removed from the domain when decommissioned.

Start of Day:

1. Arrive early.
• Coming in every day 15 minutes early shows your manager you are dependable and committed. It also helps you get the jump on any critical ticket that came in. Most executives come in early and leave late. Being the first one in translates into your becoming the go-to person for executives. This goes a long way.
2. Check the helpdesk ticket system for any open tickets that are critical.
• Pay close attention to not only the ticket request/problem but also to WHO put in the request. A medium request by your CFO warrants more than just a medium response.
• Many managers allow their support teams to self assign tickets. If so, don't cherry pick the easy tickets. Be sure to take on the more difficult tickets if you are capable and become more valuable to the team.
• If there are a number of tickets with the same issue, talk to the associated engineering team or application team if this is a broader issue occurring. [IE 20 people not able to login to Sharepoint]
3. Check with the NOC and other teams on any open issues such as outages or planned maintenance.
4. Phone support / Deskside Support
• Some helpdesk groups assign specific technicians for phone support and others to deskside support. Often its a combination of both. Depending on how your department is setup, at this point of the day begin manning the phones and responding to ticket requests.
5. Tickets
• Most small to medium groups utilize some helpdesk system while large enterprises its a must for managers and support teams.
• Follow your departments procedures on creating, assigning and updating tickets.
• Always enter detailed information when updating a ticket. When your manager reviews your work, he/she can fully understand how well you are doing and what is going on.

End of day

• IT positions never end on time so don't count on running out at exactly 5PM. Very often you may be in the middle of installing some software for a user or transferring their data to a new PC.
• Not watching the clock and staying late are good ways to get noticed by your manager and builds their confidence in you.  (Note: CIOs and business leaders users also notice)

Common tickets to expect to be assigned:

• Account is locked out
• Issue (load) a computer or laptop for a new hire
• Need software installed
• Computer wont boot or computer crashed 
• Cant access/send an email (Outlook)
• Not connected to the wireless network
• Office relocation - setup the user's PC in their new office or cube

Supplemental:
Deskside positions can lead to other opportunities such as Helpdesk manager, Server engineer, Network Engineer, Systems administrator, Active Directory Admin, and others.

Certifications often held by a support technician:

• MCSA - Microsoft Certified Solutions Associate
• CompTIA A+ 
• MOS - Microsoft Office Specialist
• Often considered a pluss as Outlook and Excel is used heavily by business users
• CCNA - Cisco Certified Network Associate
• not often required but provides a great technical skills foundation

 Qualities as a hiring manager that I look for:

1. Integrity
• I must be able to trust the candidate to a certain ethical and professional level. IT positions come with a lot of responsibility and access to sensitive information. This trust must be earned not only from the manager, but the end users that IT supports.
2. Diagnostics
• No matter how much experience someone has, if they cant instinctively troubleshoot than fixing an executives laptop while he/she is pressing you for answers and immediate resolution.. lets just say it just wont end well.
• Be aware of how to research issues and weed out the false or damaging resolution suggestions.
3. Commitment
• As an IT professional, problems will come up that you dont have the answer to...yet. There are those types of folks who hit the first obstacle and look to someone else to give them the answer. Others folks dont give up so easily and are committed to finding the solution. Note that there is a point where you should ask for help.Never asking for help is almost as bad as always asking for it.
• Commitment also includes staying ahead of the curve on technology. Businesses look to IT for answers. If the business users know more about a new gadget or IT solution coming out than you do, it is a good indication your skills and knowledge are behind the curve.
4. Interpersonal skills
• Members of the department or team must get along. Personality conflicts causes friction which leads to bigger problems.
• Having a group of people who always has each others back and are very collaborative leads to success for IT as a whole.
• A person who can effectively communicate via email, phone or in-person with IT and non-IT personell is a plus. For the helpdesk technicians, this is critical.

A day in the life of: NOC agent

NOC: Network Operations Center

Group Role:

• Monitor system health
• Monitor network connections
• Monitor batch job completion, job scheduling (rerun, pause, execute)
• Tier 1 responses often documented in a 'run book'
• Escalation of priority alerts to the appropriate support team

Note: A good place to get started in IT can be as a NOC agent

Start of day:

1. Get a lot of coffee.
• The NOC runs 24x7 and staring at a screen waiting for an alert can be a challenge.
• Some NOCs run 12 hour shifts with agents working 3-4 days per week.
2. Status transition
• As each shift changes, status of open items need to be communicated to the agents coming in, such as:
• Current outages and actions taken so far in the run book
• Systems under maintenance and when to resume monitoring
• Open requests (ie  start job #2456 @ 2:15AM)
• Changes to on-call coverage
• Confirm the information being handed off is correct (check the status board for open alarms)
3. Monitor
• Most NOCs have several dashboards that show the current state of the environment.
• Alarms are usually very easy to recognize with big red indicators, warnings are yellow
• A mailbox is usually used as well for alert messages, batch job failure or success notices and requests for batch jobs to be run/held.
4. Alert!
• An alert comes up. Follow the response checklist which may look like this:
1. Confirm if the alert is a false positive
2. Enter a ticket to record the event and assign to the NOC
3. Perform any remediation steps documented for such an alert (if present)
4. If the alert doesn't clear, follow the escalation procedures
• P1 or Critical priority response is typically a voice hand-off to the on-call engineer who handles the system or application the alarm is for
• Medium to low priority alarms, such as a QA sytem being offline are typically an email notice to the support team and ticket being assigned to them
5. Update the ticket and reassign it to the appropriate support team/person.
5. Batch Request
• Users will request a batch job to run or be held very often. If one fails, re-running the job is the normal action taken.
• Multiple failures: submit a ticket to the support team for remediation.
6. Notices
• Often engineers will notify the NOC to ignore alarms for a system as maintenance will be performed. Depending on the monitoring software, typically a NOC agent will put the system into 'maintenance mode' which suppresses alerts during a specified time window.
• On-call coverage changes occur frequently. These changes to the default rotation should be noted and also communicated to the next shift.
7. End of shift
• Transition status of systems, maintenance windows and other details to the next shift.

End of Day.


Supplemental:
There is a lot of quiet time. NOC agents can best make use of this time towards training, independent study and certifications. A natural advancement is into one of the Infrastructure support teams.

Who are all these IT people??

Most folks outside of IT do not really know what IT does and how diverse it has become. Very often business users wont even know who you are, at least until something breaks and then everyone knows your name. As an IT Pro or soon to be Pro, there are many roles to choose from for your career.

Lets take a look at a typical IT department:

Executive
It all starts with the CIO who sets overall IT directives and budgets based upon input from the business units [I.E. Marketing, Finance, HR, and the other members of the executive team]. Often reporting directly to the CIO is a purchasing or financial agent who manages procurements and adherence to the budget.

Infrastructure
The groups that make up the infra teams provide the foundation of IT services by helping the business be more productive, keeping systems secure and available and enabling the mobile workforce.

• Network Service - Manage & maintains network switches, firewalls, routers and data circuits connecting all the office to each other and the Internet.
• Storage team - Manages & maintains large SAN and NAS storage devices which can range from a few hundred terabytes to petabytes.
• Unix/Linux team - Supports all Unix or Linux servers which other groups applications or services reside upon. [IE Apache(web) servers, SAP, Oracle, DNS, compute services]
• Wintel (Windows) team - Supports all Windows servers and typically most Microsoft back office services [IE Email, Active Directory, DHCP, Print & file services, etc]. This group typically also maintain virtual environments (public & private clouds)
• Helpdesk - Deskside and phone support for the end user's desktops, laptops, tablets, Macs and other devices as well as the software installed.
• Telco - Sometimes included as part of the networking group, the Telco group maintains voice circuits and the PBX (phone system).
• NOC - The Network Operations Center monitors systems and network connections health status as well as batch jobs(automated tasks, backups and data transers). Alerts are triaged by criticality and response actions include tier 1 remediation on up to escalations to on-call engineers or network carriers and vendors to report outages.

Applications - The groups that make up the Application teams provide solutions which are directly used by the business. Such solutions may be an E-Commerce application, Reporting app, Supply Chain, ERP, CRM, Financials or HR solution. Applications are what help the business generate revenue or provide effective and timely information to business decision makers. Depending on what application platforms a company uses will determine what specialized groups are needed. Some examples are:

• Sharepoint - A key collaborative web portal which requires .NET developers and SQL DBAs. In smaller organizations, the Wintel team is often tasked with maintaining Sharepoint.
• SAP - The market leader in business solutions with very specialized and in-demand IT Pros. This solution often has multiple teams to support it [Oracle, Basis, Reports, B2B portals]
• Web Development - Once known as web masters, Web admins maintain internal and external web pages using IIS or Apache. This group will work very closely with Marketing, Sales, HR and Executives and has expanded greatly with maintaining social media.
• Application Development - Companies often develop their own applications or expand upon 3rd party or open source solutions. DBAs and programmers will round out this group.
• DBA - Database admins may be members of other applications groups or simply be their own small group. A DBA will typically specialize in one solution, most common are SQL or Oracle. Other examples are DB2 and Sybase.

Security, Program Office, Risk Management - This group is called many things but each with the same purpose; ensure the companies data is secure. In some organizations, this group may report to the CIO and others may elect to have the reporting structure outside of IT to avoid any conflict of interest. The skill sets remain the same for the group members.

• Audit team - performs internal audits to ensure everyone is adhering to corporate policies. Some companies may also need to be in compliance with Sarbane-Oxley(sox), HIPPA, PCI,  ISO####, SSAE16 and many others.
• Change Control - Tracks change requests/approvals and communicates these to the business and the other IT groups.
• Disaster Recovery - Maintains all documentation in regards to DR and conducts regular tests (normally at least once per year). Corporations have learned over the years that disasters do happen and if they are not prepared to recover within a certain timeframe, a disaster can be terminal for the business.
• Risk Management - Security team which analyzes corporate risk such as hacking threats, DLP (data loss/leakage prevention), and corporate IT policy. This group sometimes includes internal e-hack or penetration testers or uses 3rd party services to identify risks and plan counter-measure requirements. Risk Assessments, Business Impact Analysis (BIA) and Continuity of Business (CoB) each start with the Risk (security) team.

Salary info for in demand certifications

Certifications:
     If you hold a number of certifications and list them on your resume, it provides any potential employer a certainty that you have reached a standardized skill/knowledge level. They are a good watermark and are used as filter criteria by HR in scanning for possible candidates.

     Each certification comes with its own requirements which you will need to meet. Some certs take a lot of time to attain while others are quick. To determine what certifications will help you the most in your career, review job posting for your current job target as well as the next step or two in your career.

How much do certifications weight in being considered for a position?
    It depends on the employer. Some companies are required to have a minimum number of certified employees, such as Microsoft Gold Partners. Other companies use it only as a watermark on minimum skill level where more value is placed on what experience the candidate has. Either way, certifications can help you.

Top Certifications for 2013:

1. PMP - Project Management Professional - $105,750
2. CISSP - Certified Information Systems Security Professional - $103,299
3. MCSD - Microsoft Certified Solutions Developer - $97,849
4. MCDBA - Microsoft Certified Database Administrator - $95,950
5. CCDA® - Cisco Certified Design Associate - $94,799
6. MCAD - Microsoft Certified Application Developer - $93,349
7. VCP-DV: VMware Certified Professional Datacenter Virtualization - $92,400
8. CNE - Certified Novell Engineer - $91,350
9. ITIL v3 Foundation - $90,900
10. CCA - Citrix Certified Administrator - Citrix XenServer 6 - $90,850

 Thanks to Randy Muller and Global Knowledge for publishing the report. The full report and certifications 11-15 can be found on the Global Knowledge site.




Muller, Randy. 15 Top Paying Certifications for 2013. Rep. Global Knowledge, Jan. 2013. Web. 03 Mar. 2013.