Experiences Using SNOW in IT

CERN IT Department

CH-1211 Genève 23

Switzerlandwww.cern.ch/

it

Experiences Using SNOW in IT

Emmanuel Ormancey (IT/OIS)

Maite Barroso Lopez (IT/PES)

Massimo Lamanna (IT/DSS)

Feedback on Snow - 2

• Introduction• ITIL, the process• Service catalogue• SNOW, the tool• Conclusions


Introduction

• After more than one year of using SNOW in production, present the experience of 3 groups in IT– DSS (Data & Storage Services)– OIS (Operating System & Infrastructure Services)– PES (Platform & Engineering Services)

• All three heavily using SNOW in different ways• This is not a complete survey but more a tool to

trigger a department-wide discussion to gather feedback from other groups

• And to discuss the next steps to be taken

http://information-technology.web.cern.ch/about/organisation/data-storage-services

http://information-technology.web.cern.ch/about/organisation/operating-system-infrastructure-services

http://information-technology.web.cern.ch/about/organisation/platform-engineering-services


ITIL: the process

• Processes being in use:– Incident management– Request fulfillment

• Basic principles already known to IT• Main changes:

– Introduction of CERN-wide service desk• Easy transition

– Difference between incident and request• Users not ITIL-compliant, still not clear distinction

– Different tool (will be covered later)

• Still some aspects to be implemented: – SLAs, KPIs, usage of priority/impact/urgency:

• E.g.: We don’t have a set of agreed KPIs per process, with automated monitoring, so FE and SE owners can easily see progress


ITIL: the process

• Other processes discussed:– Problem management– Change management

• Introduction of new processes:– Discussion with all IT groups (usually 1-2 meetings)

where the process is described, with little implementation details (a demo)

– Feedback from groups (through individual mails?)– One day the process will be implemented in SNOW

• We need a plan, with milestones, dates, details of what will be implemented when– A pilot phase with 1-2 groups to get early feedback?

Service catalogue

• It should be the only place where a comprehensive set of services being delivered are declared. – Not the only: we still have SDB– Complex matrix of CS, SE, FE– Built over a long period, before real usage– Originally designed to hide IT internal structure (FEs) to

users, only high level services – Finally FEs are exposed to users at the service portal,

through the search facility• They have become the real services we work with, both for users

and service managers• The high level view, or user view, that was supposed to come out

from the SEs and CSs, is not widely used.



Service catalogue

• User perspective not clear– Structure too complex for the end user

– What is the difference between FE and SE for the user ?

S-NOW Portal

• One-stop entry point for our services– Locating information

• Please note: incomplete list of tools:– Services web sites (old and new Drupal infrastructure,

twikis), Savannah, Remedy PRMS, Remedy ITCM, GGUS, GOCDB, ITSSB, ServiceDB, Lemon, SLS, …

• Working environment to do support– Ticket handling– Reporting


S-NOW, the tool

• Important investment– By the support team

• In-house team responsive to bugs

– By the services adopting S-NOW• Migration + workarounds during transition

• One of major reasons for this choice was “out of the box tool used elsewhere”– Usual dilemma between: “use it as it is (and adapt yourself)” and

“endless adapt it to your needs”– Unclear where we are

• Additional modules providing more functionality exists

– Lightweight integration desirable• API?

• Smooth transition– Common effort to “protect” the users!


S-NOW, Locating information

Search for ‘Mail Android’

FE/SE mail

Android/I-Phone

Create Egroups

Report Spam

Problem with Outlook

Email web access

The portal offers nice functionality for users

– “Search for” functionality

– Some glitches are the reflection of the complex CS/SE/FE network

• e.g. “A+” service

S-NOW, general interface

• Usability: frame layout and general URL schema– Difficult to navigate between pages, open links in

separate windows, copy/paste/save links etc. using standard web browser features/navigation• Inefficient. Needs investment

• SSO:– Was the SSO standard compliance a requirement ?

• Partially implementation of the SSO standards SAML2 & WS-Federation

– Login works, logout fails• User is redirected to SNOW instead (close the browser)• Hacks made by IT (no S-NOW support for this)

– IT hacks are being evaluated, but not yet successful


S-NOW as a reporting tool

• Too much emphasis to the shiny reporting capabilities:– What is really useful?

• Indicators like Time_to_Solve, Time_to_Route, “#3rd level”/”#2nd level” are interesting– OK for auditing the Service Desk– Much less interesting for day-by-day service managements– Custom reporting (e.g. CastorTape) missing

• Simple indicators not sensitive to the quality delivered and to the contribution of the various teams– Example from CASTOR 2nd level support:

• Tickets are from expert users• 2nd level helps us to solve by collecting information• Which is great!• But strictly speaking no ticket is closed by them

– Hence their contribution is not visible in the indicators» But greatly appreciated anyway


S-NOW as a content management system (1/2)

• Knowledge Base library– Full life cycle: editing, workflow, feedback, …

• Some glitches: IMHO the main one is the shaky editor

– More important:• Difficult to organise/present (the only structure is the FE

association)

– On the other hand very convenient for 2nd level


S-NOW as a content management system (2/2)

• Entities like CS/SE/FE have their respective web page– Not useful to give a basic overview our services

• Not reach enough

– Duplication of Drupal web pages

• SLA/SLD– SLA/SLD as description of the service– Now in the IT web infrastructure

• SLA is not ticket handling tim

Presentation title - 14

S-NOW: ticket handling

• Superior features compared to Remedy– Web application so no client to install– Better interface, easier to use– Transparency, re-assignment of tickets, comments field and

visibility– Interfaced to GGUS– Record producers– More granularity

• More SE / FE• Better case assignment to the FE

– Good idea of urgency/priority even if not used

• To be improved:– Although no show stopper a long list of improvements exists

(see later) Feedback on Snow - 16

S-NOW, the tool

• Implementation of incident management:– Lacking rota feature, last meeting in July, as of today still no

date, known plan• We have survived with tricks, in all this time we could have learnt to

work otherwise?

– No usage of urgency/impact/priority• Only impact can be changed by service manager• Priorities come through other sources

– No SLAs implemented• Some incidents can stay there for a long time without proactive

checking, no reminders sent– When the feature was enabled, S-NOW was spamming us

• No time limit for user to respond (Waiting for user)

– Only the FE is compulsory to be filled, so we work with FEs and no SEs


S-NOW: ticket handling

• Integration with other systems– It turned out to be complex and not yet completed

• The integration with GGUS is OK– But REMEDY ITCM is still with us

• Note that #GGUS tickets for a typical service is several a week across different services, but it can easily reach several per day on a single FE

• We still have ITCM alive, tickets get duplicated/lost in between


S-NOW and ITCM integration

• Not done at the beginning

• Main tool for groups to handle large hardware

• Collaboration tool between sysadmins and Service managers

• Critical on delivery timely and efficient hardware repairs

Sysadmin FE manager

Can I reboot node abc?

Yes! Tell me when done

OK!

Can I reboot node abc?

Thanx, node back in prod

ITCM3641

INC118240

INC118960

SNOWREMEDY

Context loss

Simple (but real) case.Complex cases force us to use email...

S-NOW, Request Fulfilment

• Implementation of Request fulfillment:– Often just a complex implementation for a simple flow:

many steps to achieve the same, at the end is the same person signing/agreeing the different steps

– No way to close a request after a first dialogue with the user, before escalating to fulfilment• It is possible to reject the request, but this is not nice for the user

– Not possible to convert incidents into requests without sending it back to first level (service desk)

• A large majority of issues would be better dealt with INC…


S-NOW: complex interface

• UI not always efficient• High number of clicks/tear-down menus

– Why choose “Resolved” and the “Restored”/”Not Restored”/”Spam”/”...”?

– Maybe a “click through” exercise could be nice to enhance the usability

• To act on a ticket one should belong to an egroup– And then you get all the mails

• “4th level” experts feel it as a “spam”

• Power users should have power commands– Bulk operations are needed

• Why the system does not offer an API?– E.g. programmatically query #open tickets etc..– It will greatly encourage light-weight integration


S-NOW, the case logs

• Case Logs are not understandable– This induce back and forward between second and third

level to understand what is happening– Important loss of efficiency, waste of time

Example of internal part of log of an incident


S-NOW, the mails

• Awaited improvements:– Service owner

• Need for an escalation system• Need to be able to generate reports

– Reduce the number of mails• For each case assigned to someone:

– One mail from NO-REPLY-service-desk – One mail from service-desk

Conclusions

• The process– IT Service Managers made a big effort to adapt to the ITIL process, and to adapt to the

new tools• The evolution of the process should be discussed step by step with the Service Managers• Define ‘where things should be’: KBs, SLA, SLD, etc.

– Avoid duplication!S-NOW is not the best place for everything

• Implements a Service Catalogue maintenance lightweight process– Goals: updated service list and clear for users

• The tool– IT services invested a lot in it

• The tool is good, better than the previous one.• Many improvements are needed to improve efficiency:

– Simplify the views and usual actions– Proper rotas implementation– SSO, KBs, text editor.

• The bottom line– A closer collaboration between IT Service Managers and ITIL team would improve the

whole process.– Service Managers experience should be taken more into consideration, in a constructive

manner instead of opening RFQs or running individual negotiations.


Questions?

Documents

Experiences Using SNOW in IT