Saturday, March 23, 2013

A day in the life of: The Systems Administrator


Systems Administrator:


A Systems admin can go by many names: Server Admin, NT Administrator, Systems Engineer, Unix/Linux admin, Network Administrator, etc. Refer to job descriptions to more clearly understand the company’s definition for their environment and the name they assign to this role. In the end, an IT admin supports infrastructure systems and devices.

In this “day in the life”, I'll focus on a Windows server administrator.

I would also like to mention that in my career, I have specialized in helping companies where the executive management team has reached their boiling point with their IT Infra teams. They reached out to me for help come in and assess the environment, develop a design and action plan then put things into place. The approach I have fine-tuned over the years may be the subject of a series of other posts if there is interest.

Windows is by far the most common server operating system you will see in the enterprise. The Linux/Unix admins have similar workdays however there are some differences that I will cover in another post. Network Admin (for switches and routers) is yet another posting.

Group Role:


The Windows server support group, along with the network group, is the foundation of the entire business. The physical and virtual server that the team installs, configures, secures and essentially keeps ‘alive’ and running at peak performance hosts the critical application which allows the company to do business.

In addition to the keeping the servers running and secured, the team typically supports applications such as: Email, Active Directory, ADFS, File & Print, Group Policy (automation and policy enforcement), Systems Management tools (IE SCCM, MDM, SIEM, IDMS), SharePoint, Corporate Antivirus management and  Helpdesk ticket systems.

Start of day: 


1.      Wake up, rub your eyes and grab your smartphone to check your work email if any alerts or critical requests have been sent to you.
·         No I’m not kidding… server admins are a special breed who really cares about their systems and are always keeping an eye on things 

2.      If any alerts or emails need immediate attention (IE an alert a server’s CPU has been running at 99% or a drive is near full capacity)
·         jump on your laptop and connect to the VPN
·         Remotely connect to the server in question and remedy the problem
·         If a ticket wasn’t created for the alert, create one. Update accordingly and close the ticket
·         Log off 

3.      Go through your morning routine then head to work.
·         Note: It’s always a good idea to arrive early to get a jump on any issues before users are calling about some server issue preventing them from doing their work. 

4.      Arrive at work and check the ‘state of the union’ by checking in with the NOC (if you have one), check the helpdesk tickets assigned to you and your group & the monitoring dashboard(s).
·         If any critical issues are open, crack open the laptop and remediate the problem.
·         Update and close the ticket related to the the issue you resolved. If a ticket doesn’t exist, create one. 

5.      If all is good in the world of servers, head for your morning coffee.
·         Coffee in the morning is a perfect time to interface with the colleagues within your IT group as well as other groups.
o Collaboration strengthens IT as a whole and gives you some insight if there are some other issues going where you may be able to help.
·         Coffee time is also a great time to talk to end users (non-IT) and find out how things are going
o Listen to what users say closely and read between the lines. If you pick up on any pain points like an app is slow today or challenges that some new software could increase their productivity or quality of life…
o Make a mental note and look into it. If its new software solutions, discuss it with your manager. If you fix a problem for the user, follow it up in an email to let them know otherwise they think the problem just went away.
·         Pet peeve note: Careful with extended coffee runs.
o Many managers interpret long coffee breaks as a negative unless they are aware of your 'recon' motive
o Users who get the impression IT is always on break or arent busy is bad for IT PR.
o Find the right balance of showing time to listen to people and lingering too long

6.      Return to your desk and again, check on the ‘state of the union’ (Dashboard and tickets) 

7.      Begin your normal work assignments
·         This may vary depending on how your manager assigns duties and priorities. What I found works well is in this following order:
1) Resolve any critical incident tickets  (something broken)
2) Resolve any critical requests (something needed)
3) Proactive health checks (20% of your day)
·        Each engineer should have a list of systems and app to review on a daily, weekly and monthly basis to resolve problems before they become a risk to an outage.
4) Resume resolving tickets (60% of your day)
5) Project work (20% of your day)
·        Some projects you may be dedicated 100% of your day for the duration of the project. This will be determined by your manager based on criticality and the project completion date 

8.      Throughout the day, expect interruptions. These will cause you to shift from one assignment to another.
·         System engineers juggle a lot of tasks concurrently during the day as all other IT groups rely on the infrastructure teams. Their critical events translate to a critical response by Infra groups in order for IT to respond as a whole to the ever changing requirements and needs of the business. 

9.      End of shift
·         Don’t expect to leave exactly on time. What we do as system admins doesn’t exactly finish on our schedule. It is part of the life we chose in supporting servers that don’t turn off. It is our responsibility to keep them and the business going.
·         Bring your laptop home with you every day. If you are not on call, one of your team members is and may need help. Great teams always back each other up.
·         Check the monitoring dashboards one last time before leaving 

10.  End of day

·         The end of the day doesn’t really end when your shift does as on-call rotations are common. Avoid becoming burnt out by keeping up with other hobbies outside of tech but definitely do not disconnect completely when going home for the day.


Server admins are in their position because they are never really disconnected (on-call or not). Part of this is because they become personally connected to their systems and the company. We take great pride in what systems we have built.  

Common tickets for System Administrators:

·         Email [spam, delivery problem, mailbox needed]
·         Need a new server for “xyz” project
·         Patch or software deployment needed to [servers, pc/laptop, users with “X” installed]
·         Mailbox or distribution list needed
·         Alert received for server “X”  [connectivity lost, cpu pinned @99%, free space is low, unauthorized access attempt, URL check failed, etc]
·         Data management [file restore, access request/removal, new share]
·         New solution or service project [Proof of concept, test, pilot, deploy]
·         Server or application performance
·         “How do I…” or how to questions 

Supplemental


Becoming a Systems Administrator

Server admins typically have their careers start within the helpdesk or NOC where a great amount of experience is gained, more so from the helpdesk. Many companies promote from within when a helpdesk technician shows the traits and technical initiative needed for server side support.

Skills and traits of a good admin

·         Integrity – First and foremost this trait is absolutely required.

·         Commitment – Servers can crash at any time and the business depends on IT to do whatever it takes to get services restored.
o   Taking part of an on-call rotation is very common
o   Helping the on-call person respond to an alert builds team unity
o   Seasoned engineers can all share a story of at least one time they had to work 24+ hours straight on restoring a server or recovering all the systems in the DR site (disaster recovery site). This should be a rarity but it will definitely happen in your career. This level of commitment is rewarded greatly, not just monetarily but also in the business relationships you develop with your executive management team.

·         Stay ahead of the curve
o   At this level of IT, you are the one who should have a solution to every challenge of at least know how to find it quickly. Stay ahead on your certifications and tech news

·         Vision
o   See the forest, not just the trees.
o   For example:
§  If every day you need to reboot a server to correct an issue; that is not a fix but a Band-Aid. Find the real problem so you no longer have to do this daily manual task.
§  The helpdesk keeps getting calls to set a new users homepage, desktop shortcuts and mapped network drives to match what the rest of their group has. Create a GPO (group policy object) to automatically create these settings and save the desktop group hundreds of hours over the next couple years.
§  An application owner or even a Microsoft support rep requests you to make a change the server that is having a problem. Will this cause a problem now or in the future? Will it violate any corporate standard or policy? Knowing the answers to these questions is critical.

·         Diagnostic skills
o   A professor I once had the pleasure of having once described diagnostic or troubleshooting skills as a talent that “you either got it or you don’t”. I agree with him however I do believe these skills need to be developed through use to become stronger.
§  Pay particular attention to how components or pieces come together. What dependencies are there between components and where are the failure points.
§  Identify what symptoms relate to different failure points. For example, if Internet Explorer displays a message “page cannot be displayed’; what’s wrong (if you can name only 1 or 2 things, it’s time for you to do some studying)

·         Ability to work with little supervision

o   Depends on your manager’s style however most good managers do not attempt to micromanage the systems engineers who are often relied upon to be subject matter experts (SMEs).

o   If you have built the confidence your manager has in you and judgment, they will normally loosen the reigns. If you find you are being managed more closely than your peers, evaluate how you can improve by using your peers as examples as well as speaking to your manager. In order to advance, your manager needs to really trust in you. 

Sorry this was such a long post folks but this job role manages a considerable amount of the overall IT presence within a company. The success of the server team has a direct impact upon the productivity of the business and should be a leader of innovation and security initiatives. Carreers as System Administrators can be very rewarding with talented engineers always in high demand.

 

Thursday, March 21, 2013

Monitoring - too often overlooked

Outages, unplanned downtime, critical events...

It is not a matter of IF they will happen, its a matter of when.

Outages are bad.
        Outages during business or production hours is very bad.
                 A user or client knowing before IT about an outage... inexcusable.

You would think this is just common sense yet it is one of the most common issues I find when starting a new engagement. (and probably one reason why the position opened up)

What to do?

IT is in the business of technology and providing information to the business units. This is where IT needs to be a customer to their service (See post on IT groups, the NOC is an IT group that is a customer to IT).

Some useful tools for varying budgets:

Budget Friendly:

  • MRTG - "Multi Router Traffic Grapher"  (open source)
    • Tool provides network utilization graphs.
    • Useful to see if WAN links are saturated where users would complain the network is slow.
    • Simple to complex dashboards can be created to give a view of the entire network landscape
  • Zenoss Open Source Edition
    • Can monitor all the devices on the network
    • Sends an alert so you know something is down
    • Web status page
    • Install can be difficult... read the install guide well and you'll be OK
  • Nagios (open source)
    • A long used monitoring tool that has many plug-ins to improve the product like Cacti
    • Sends alerts and can be setup with some very nice web status pages
    • configuration can be challenging to some
  • Servers Alive (low cost)
    • Very easy to setup monitoring
    • Free edition can monitor up to 10 items, standard or enterprise editions come with a small fee
    • Email alerting and additional plug-ins (such as POP check for monitoring email flow)

Enterprise class:

These systems may not be budget friendly, but you definitely get what you pay for. Hopefully the company you are with either has these in place or can fit them into the budget.

Note:
Don't be surprised if the place you starting at owns monitoring software licenses yet haven't found the time to deploy the solution.

Setting up monitoring the right way takes time and often projects for IT are set to lower priority than projects to the business. In one way, it's great to improve productivity and provide great business intelligence reports; however, if your not maintaining the house it can fall down around you. If it does fall, where will the company be then?

  • Microsoft System Center Operations Manager - aka SCOM
    • In my opinion, the Mercedes of monitoring.
    • The 2012 version can monitor Windows, Unix, Linux, Network devices and many software and hardware vendors are providing management packs to allow their solutions to also be monitored.
    • Talk to your MS reseller as there are Windows server bundle deals which can save you a bundle by rolling up server licensing, endpoint protection and the System Center suite.
    • Multiple methods to deliver alerts (email, sms, etc)
    • Full robust dashboards
    • Performance tracking and trending reports
    • Direct ties into other System Center Suite modules that if all are fully deployed is a wonderful sight to see (and probably a great IT environment to work in)
    • Alternative options to Microsoft
  • CorrelSense Sharepath
    • Monitors more than just a server's performance or a network link... this is end-to-end application monitoring of the user experience.
    • Bottlenecks are instantly identified and ends finger pointing between teams
    • Reports on performance out of normal operation with really impressive drill down dashboards
Of coarse there are many other solutions out there that are on-premise or offered as a cloud service. The key factors in any monitoring is that:
  1. Minimize the false positives
    • If too many alerts are generated and safely ignored, does anyone see the critical alerts or do they go into a junk mail folder?
  2. IT uses the tools wisely.
    • If no one is watching the alerts, its has no purpose
    • If you need 24x7 support, you need a NOC
  3. The staff can easily support and maintain the software 



Saturday, March 9, 2013

Whats in your toolkit?

What are some tools in your bag of tricks?


     In the 2000-2010 years I was lucky to have lead many projects to merge IT teams due to corporate M&A. Each one unique to what needed to be done and the people I had the privledge to work with. One of the first things we always did do was to compare our toolkits. 3rd party products which helped us work magic at low costs.

Below are a few examples:
  • PAL - Performance Analyzer of Logs
    • Perfmon is a great tool but not many sysadmins use it to troubleshoot as results are hard to interpret or due to uncertainty in what counters to select
    • PAL gives you pre-built templates to load into perfmon and then, the greatest feature of all.... import the trace output logs & PAL analyzes it and presents the results in an informative report
    • Great for analyzing Exchange, AD, IIS, and Sharepoint servers
  • Knoppix
    • A bootable CD or USB drive with free emergency data and system recovery tools
    • An ERD (emergency recovery disk) is critical for any support person.
      • Reset the local admin password
      • Recover data from a failed Windows sytem
      • Repartition the hard drive or extend volumes
      • Network analysis and administration tools
    • ERD Commander is another nice alternative
  • Dameware mini-remote
    • A nice utility that allows you to take control of a users PC so you can fix their problem without leaving your desk.
    • Handy if you dont have a corporate remote administration package
  • Hyena
    • An administration tool which can connect to multiple domains.
    • Useful in exporting AD objects and their properties.
    • Searches are often easier than using AD users and computers to display custom AD attributes for multiple objects at the same time. If your not good at LDAP querries, this makes it simple.
    • Ability to edit multiple objects at the same time
  • Ghost
    • Utility that is very common. Many times where a hard drive is failing and wont stay online long enough for Windows to let you copy everything off, Ghost may be able to help. Put the drive into an external USB drive enclosure & you can grab a image of it to recover someones data.
  • Powershell Plus by Idera
    • If you dont know powershell yet, you will need to learn a little which this tool can help.
    • Powershell Plus has a large library of scripts built-in to instantly help even PS veteran that includes scripts for AD, Exchange, SQL & Sharepoint
    • Idera has many other useful tools for SQL and Sharepoint
  • RedGate SQL Toolbelt
    • Fantastic backup with compression tool
    • Monitoring and SQL admin tools (too many to list)
  • Veeam One, Veeam backup
    • Comes in a free version so if your budget is tight, it can help you greatly with your virtual environment to montior and backup.
    • The full licensed versions win awards year after year at VMWorld
    • Training videos are provided on the Veeam website
  • NetIQ Domain Migration Suite
    • This product may have been discontinued but it was the best utility I found for migrating users, groups, computers and email.

So.. what do you have in your own bag of tricks?



Interview Questions

Prepare for the Interview


The interview is the time to make an impression and most candidates do this by preparing their answers to potential questions. My tip is to turn it around by playing offense.

When it comes to the point in the interview with the hiring manager where you are asked if you have any questions, have a few of these questions ready. 

 

What would be my typical day and who would I be working most closely with?

This question is multi purposed.
  1. The question is designed to have the interviewer visualize you in the position as part of the team.
  2.  It gives you a good glimpse into what your work day would be and allows you to judge if this position is what you are looking for.

Who are the 'stars' on the team that I would learn the most from and what would be the best traits to pick up?

  1. This question is also designed to have the interviewer visualize you as part of the team.
  2. Demonstrates that you are committed to learning and self improvement by learning from the team's best people.
  3. Gain  

What are some of the current challenges the team is facing that I may be able to help resolve?

  1. How the interviewer responds will give you a good indication if they are seriously considering you or not. If the interviewer gives you a few items that you can contribute on, its a good sign.
Continual development is critical for people in IT and something that is important to me. How do you feel I could best improve?

  1. Even if you do not get the position, you gain some valuable information that can help you prepare for the next interview.
  2. The question emphasises your commitment to your career. If you are not learning, you are falling behind the curve.
  3. This questions landed me one of my early IT positions. The director interviewing me at the time was noticeable surprised and he gave me one of the most important pieces of advice I have received. He thought for a minute and then told me;
 "Keep your mouth shut and your ears open"
 
Sound advice in such a concise response.
  • As IT, we have access to very confidential information and there is no room for someone who leaks info.
  • Listen to not only what is being said but also between the lines, you can deduce what people are challenged or concerned with most. Resolve a problem before the users ask IT for help and suddenly you are perceived as being innovative and a true asset.
  • Listen to what you IT colleagues have to say. There are many ways to approach a project or problem and bright ideas can come from anyone. Collaboration also is what makes the strongest teams.
The director may have been trying to tell me I simply talk too much, but I like my interpretation better. To this day, we do remain friends.




Sunday, March 3, 2013

A day in the life of: Helpdesk Support

The Helpdesk


Group Role:
The helpdesk is THE face of IT. Users look to the helpdesk for answers to their problems as well as to help improve their productivity by installing new software or getting a new computer/laptop. As a helpdesk technician, you need to appear as always having the right answer and be completely trustworthy. After all, a user who hands over their laptop with all of their critical and confidential data needs to be confident you wont lose their files or look at the contents.
  • Hardware support:
    • Desktops and laptops from multiple vendors such as HP, Dell, IBM, and Apple. Replace failed components and issue new systems. 
    • Printers, scanners and sometimes basic mobile device support depending on the environment. Typically hardware support is minimal triage before calling a 3rd party for more advanced support (IE setup a new printer on the network, change toner and rollers but not take the thing apart)
  • Software support:
    • Microsoft Windows or Mac OS: Install, troubleshoot, patch, upgrade
    • Microsoft Office, Adobe, print devices: install, troubleshoot, patch, upgrade
    • Install and Other 3rd party software the user may require
    • Troubleshoot errors the user may see
      • Internet Explorer alerts or pages not loading
      • No network available (wired/wireless)
      • Unexpected crashes of an application or OS
  • Account support
    • Account creation and removal
      • May be the responsibility of the helpdesk or as organizations get larger, there is more segregation of duties. The task may be assigned to the security team or even be delegated to HR and created when someone is on boarded or off boarded.
    • Password resets, Unlock accounts
    • Add/Remove groups to Active Directory or update memberships to a group
      • Sometimes this is delegated to the Helpdesk or managed by the Wintel (server) team.
    • Computer accounts in Active Directory
      • New PCs are joined to the domain when issued to a user
      • Old PCs are removed from the domain when decommissioned.


Start of Day:
  1. Arrive early.
    • Coming in every day 15 minutes early shows your manager you are dependable and committed. It also helps you get the jump on any critical ticket that came in. Most executives come in early and leave late. Being the first one in translates into your becoming the go-to person for executives. This goes a long way.
  2. Check the helpdesk ticket system for any open tickets that are critical.
    • Pay close attention to not only the ticket request/problem but also to WHO put in the request. A medium request by your CFO warrants more than just a medium response.
    • Many managers allow their support teams to self assign tickets. If so, don't cherry pick the easy tickets. Be sure to take on the more difficult tickets if you are capable and become more valuable to the team.
    • If there are a number of tickets with the same issue, talk to the associated engineering team or application team if this is a broader issue occurring. [IE 20 people not able to login to Sharepoint]
  3. Check with the NOC and other teams on any open issues such as outages or planned maintenance.
  4. Phone support / Deskside Support
    • Some helpdesk groups assign specific technicians for phone support and others to deskside support. Often its a combination of both. Depending on how your department is setup, at this point of the day begin manning the phones and responding to ticket requests.
  5. Tickets
    • Most small to medium groups utilize some helpdesk system while large enterprises its a must for managers and support teams.
    • Follow your departments procedures on creating, assigning and updating tickets.
    • Always enter detailed information when updating a ticket. When your manager reviews your work, he/she can fully understand how well you are doing and what is going on.
End of day
    • IT positions never end on time so don't count on running out at exactly 5PM. Very often you may be in the middle of installing some software for a user or transferring their data to a new PC.
    • Not watching the clock and staying late are good ways to get noticed by your manager and builds their confidence in you.  (Note: CIOs and business leaders users also notice)
Common tickets to expect to be assigned:
  • Account is locked out
  • Issue (load) a computer or laptop for a new hire
  • Need software installed
  • Computer wont boot or computer crashed 
  • Cant access/send an email (Outlook)
  • Not connected to the wireless network
  • Office relocation - setup the user's PC in their new office or cube


Supplemental:
Deskside positions can lead to other opportunities such as Helpdesk manager, Server engineer, Network Engineer, Systems administrator, Active Directory Admin, and others.

Certifications often held by a support technician:
 Qualities as a hiring manager that I look for:
  1. Integrity
    • I must be able to trust the candidate to a certain ethical and professional level. IT positions come with a lot of responsibility and access to sensitive information. This trust must be earned not only from the manager, but the end users that IT supports.
  2. Diagnostics
    • No matter how much experience someone has, if they cant instinctively troubleshoot than fixing an executives laptop while he/she is pressing you for answers and immediate resolution.. lets just say it just wont end well.
    • Be aware of how to research issues and weed out the false or damaging resolution suggestions.
  3. Commitment
    • As an IT professional, problems will come up that you dont have the answer to...yet. There are those types of folks who hit the first obstacle and look to someone else to give them the answer. Others folks dont give up so easily and are committed to finding the solution. Note that there is a point where you should ask for help.Never asking for help is almost as bad as always asking for it.
    • Commitment also includes staying ahead of the curve on technology. Businesses look to IT for answers. If the business users know more about a new gadget or IT solution coming out than you do, it is a good indication your skills and knowledge are behind the curve.
  4. Interpersonal skills
    • Members of the department or team must get along. Personality conflicts causes friction which leads to bigger problems.
    • Having a group of people who always has each others back and are very collaborative leads to success for IT as a whole.
    • A person who can effectively communicate via email, phone or in-person with IT and non-IT personell is a plus. For the helpdesk technicians, this is critical.

A day in the life of: NOC agent

NOC: Network Operations Center

Group Role:
  • Monitor system health
  • Monitor network connections
  • Monitor batch job completion, job scheduling (rerun, pause, execute)
  • Tier 1 responses often documented in a 'run book'
  • Escalation of priority alerts to the appropriate support team
Note: A good place to get started in IT can be as a NOC agent

Start of day:
  1. Get a lot of coffee.
    • The NOC runs 24x7 and staring at a screen waiting for an alert can be a challenge.
    • Some NOCs run 12 hour shifts with agents working 3-4 days per week.
  2. Status transition
    • As each shift changes, status of open items need to be communicated to the agents coming in, such as:
      • Current outages and actions taken so far in the run book
      • Systems under maintenance and when to resume monitoring
      • Open requests (ie  start job #2456 @ 2:15AM)
      • Changes to on-call coverage
    • Confirm the information being handed off is correct (check the status board for open alarms)
  3. Monitor
    • Most NOCs have several dashboards that show the current state of the environment.
    • Alarms are usually very easy to recognize with big red indicators, warnings are yellow
    • A mailbox is usually used as well for alert messages, batch job failure or success notices and requests for batch jobs to be run/held.
  4. Alert!
    • An alert comes up. Follow the response checklist which may look like this:
      1. Confirm if the alert is a false positive
      2. Enter a ticket to record the event and assign to the NOC
      3. Perform any remediation steps documented for such an alert (if present)
      4. If the alert doesn't clear, follow the escalation procedures
        • P1 or Critical priority response is typically a voice hand-off to the on-call engineer who handles the system or application the alarm is for
        • Medium to low priority alarms, such as a QA sytem being offline are typically an email notice to the support team and ticket being assigned to them
      5. Update the ticket and reassign it to the appropriate support team/person.
  5. Batch Request
    • Users will request a batch job to run or be held very often. If one fails, re-running the job is the normal action taken.
    • Multiple failures: submit a ticket to the support team for remediation.
  6. Notices
    • Often engineers will notify the NOC to ignore alarms for a system as maintenance will be performed. Depending on the monitoring software, typically a NOC agent will put the system into 'maintenance mode' which suppresses alerts during a specified time window.
    • On-call coverage changes occur frequently. These changes to the default rotation should be noted and also communicated to the next shift.
  7. End of shift
    • Transition status of systems, maintenance windows and other details to the next shift.

End of Day.


Supplemental:
There is a lot of quiet time. NOC agents can best make use of this time towards training, independent study and certifications. A natural advancement is into one of the Infrastructure support teams.



Who are all these IT people??

Most folks outside of IT do not really know what IT does and how diverse it has become. Very often business users wont even know who you are, at least until something breaks and then everyone knows your name. As an IT Pro or soon to be Pro, there are many roles to choose from for your career.

Lets take a look at a typical IT department:

Executive
It all starts with the CIO who sets overall IT directives and budgets based upon input from the business units [I.E. Marketing, Finance, HR, and the other members of the executive team]. Often reporting directly to the CIO is a purchasing or financial agent who manages procurements and adherence to the budget.

Infrastructure
The groups that make up the infra teams provide the foundation of IT services by helping the business be more productive, keeping systems secure and available and enabling the mobile workforce.
    • Network Service - Manage & maintains network switches, firewalls, routers and data circuits connecting all the office to each other and the Internet.
    • Storage team - Manages & maintains large SAN and NAS storage devices which can range from a few hundred terabytes to petabytes.
    • Unix/Linux team - Supports all Unix or Linux servers which other groups applications or services reside upon. [IE Apache(web) servers, SAP, Oracle, DNS, compute services]
    • Wintel (Windows) team - Supports all Windows servers and typically most Microsoft back office services [IE Email, Active Directory, DHCP, Print & file services, etc]. This group typically also maintain virtual environments (public & private clouds)
    • Helpdesk - Deskside and phone support for the end user's desktops, laptops, tablets, Macs and other devices as well as the software installed.
    • Telco - Sometimes included as part of the networking group, the Telco group maintains voice circuits and the PBX (phone system).
    • NOC - The Network Operations Center monitors systems and network connections health status as well as batch jobs(automated tasks, backups and data transers). Alerts are triaged by criticality and response actions include tier 1 remediation on up to escalations to on-call engineers or network carriers and vendors to report outages.
Applications - The groups that make up the Application teams provide solutions which are directly used by the business. Such solutions may be an E-Commerce application, Reporting app, Supply Chain, ERP, CRM, Financials or HR solution. Applications are what help the business generate revenue or provide effective and timely information to business decision makers. Depending on what application platforms a company uses will determine what specialized groups are needed. Some examples are:
    • Sharepoint - A key collaborative web portal which requires .NET developers and SQL DBAs. In smaller organizations, the Wintel team is often tasked with maintaining Sharepoint.
    • SAP - The market leader in business solutions with very specialized and in-demand IT Pros. This solution often has multiple teams to support it [Oracle, Basis, Reports, B2B portals]
    • Web Development - Once known as web masters, Web admins maintain internal and external web pages using IIS or Apache. This group will work very closely with Marketing, Sales, HR and Executives and has expanded greatly with maintaining social media.
    • Application Development - Companies often develop their own applications or expand upon 3rd party or open source solutions. DBAs and programmers will round out this group.
    • DBA - Database admins may be members of other applications groups or simply be their own small group. A DBA will typically specialize in one solution, most common are SQL or Oracle. Other examples are DB2 and Sybase.
Security, Program Office, Risk Management - This group is called many things but each with the same purpose; ensure the companies data is secure. In some organizations, this group may report to the CIO and others may elect to have the reporting structure outside of IT to avoid any conflict of interest. The skill sets remain the same for the group members.
    • Audit team - performs internal audits to ensure everyone is adhering to corporate policies. Some companies may also need to be in compliance with Sarbane-Oxley(sox), HIPPA, PCI,  ISO####, SSAE16 and many others.
    • Change Control - Tracks change requests/approvals and communicates these to the business and the other IT groups.
    • Disaster Recovery - Maintains all documentation in regards to DR and conducts regular tests (normally at least once per year). Corporations have learned over the years that disasters do happen and if they are not prepared to recover within a certain timeframe, a disaster can be terminal for the business.
    • Risk Management - Security team which analyzes corporate risk such as hacking threats, DLP (data loss/leakage prevention), and corporate IT policy. This group sometimes includes internal e-hack or penetration testers or uses 3rd party services to identify risks and plan counter-measure requirements. Risk Assessments, Business Impact Analysis (BIA) and Continuity of Business (CoB) each start with the Risk (security) team.