Document control

AreaISRM
Procedure status
FINALISED
Owner
ApproversProcess owner
Approval status
Approved version and date

v.52  

Statement

Procedure describing how an incident or service request should be recorded, classified , prioritized, escalated, resolved, closed

Table of contents

Overview

Procedure describes how an incident or service request from customers should be recorded, classified , prioritized, escalated, resolved, close.

The tool used for recording and handling incidents and services requests is EGI Helpdesk (GGUS). The tool provides full functionality:

  • recording
  • change history
  • categorization
  • prioritization
  • classification 
  • notification

which supports the procedure.

Operational incidents are handled according to procedure PROC01 EGI Infrastructure Oversight escalation

Definitions

Please refer to the EGI Glossary for the definitions of the terms used in this procedure.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", “MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Other terms used in this procedure adopted from EGI Helpdesk (GGUS) terminology are:

TermDescriptionValuesComments
Ticket categoryThis term indicates the very first category of the ticket, aiming at indicating if the ticket is reporting about an incident or a service request. Also "Documentation" is used to request creation and update on documentation. "Release" is used when a new version of the operational tools is ready to be tested (see RDM2 Regular release process). "CMS Internal" is for cms VO specific tickets, out from the scope of this procedure. Coordination/Planning is used for activities that don't fall in the list of service requestsIncident, Service requests, Documentation, Release, CMS Internal, EGI Coordination/Planning, WLCG Coordination/Planning, Test
Ticket type

The ticket type defaults to USER in the case of an EGI user opening the ticket, resulting in a typical GGUS ticket. Other special ticket types are possible in GGUS:

  • TEAM and ALARM tickets have a reduced number of issue types in the drop-down menu, and can be opened only by members of the following VOs:
    • ALICE,
    • ATLAS,
    • CMS,
    • LHCb,
    • Biomed,
    • Belle.
  • The CMS VO has an own view of GGUS system providing some specific issue types. CMS issue types can only be selected during ticket submit by registered CMS users.
When submitting the ticket, USER, ALARM, TEAM, CMS as specified here GGUS-IssueType-Values

Only tickets with Ticket type = USER (and Scope=EGI) are considered in this procedure. TEAM, ALARM, CMS tickets are not effectively handled by EGI and are considered "VO business" (directly managed by user communities).

The Scope can be EGI or WLCG. For more information: Helpdesk Features

Type of issueThis represents a way to categorize more deeply the ticket. Here the user can specify the scope of the ticket. This field helps the TPM in classifying the ticket in terms of responsibility and assigning to the correct Support Unit to get proper support.GGUS-IssueType-ValuesThe drop-down list of issue types varies depending on the ticket type.
PriorityThis term suggests the level of criticality of an incident, depending on the urgency and/or the impact the incident (or the service request) has for the user.

GGUS Tickets Priorities

EGI DMSU Ticket Priorities


Support UnitThis term indicates a group of people that is in charge of handling and resolving issues (or working and completing service requests) related to their specific services.SUs FAQs

Further explanations are in GGUS-Support-Staff-Guide.

Entities involved in the procedure

  • Customer/user - create a ticket in EGI Helpdesk tool to record incident or service request
  • 1st line support  (TPM), 2nd line support - EGI teams providing support
  • 3rd line support team - Product teams providing support
  • GGUS ticket monitoring - responsible for oversight longstanding tickets and its escalation

Triggers




Steps

Step# ResponsibleActionComment
1
Record, Classify, Prioritize


Customer or user

Ticket is created with providing in minimum (see Definitions for the explanation of the terms)

  • Subject
  • Description of the issue
  • Ticket category: Incident, service request, Documentation.
  • Type of issue (drop-down menu). IMPORTANT: The drop-down list of issue types varies depending on the ticket type (see Definitions).

  • Priority: (top priority, very urgent, urgent, less urgent) default: less urgent
    • If customer/user plan to adjust the ticket priority from the default value to 'urgent', 'very urgent' or 'top priority' is asked to make sure to add a justification for this change in the problem description field.
  • assigning the proper Support Unit is optional, as , if known; setting the Support Unit is possible through a drop-down menu, showing the Support Units as specified in FAQ_Responsible_Units_(GGUS). Automatically ticket is assigned to TPM support group. IMPORTANT: Selecting either a site from the "Notify SITE" drop-down menu or a support unit from the "Assign to support unit" drop-down menu routes the ticket directly to the selected support unit, bypassing the TPM. If selecting a site name the NGI/ROC the site belongs to is set automatically. Hence the ticket is assigned to the relevant NGI/ROC. Additionally the site will receive a notification about the ticket.

EGI Helpdesk tool submission form: https://ggus.eu/?mode=ticket_submit


2

Re-Categorization, Re-prioritization, Resolution


11st line support  (TPM)
  • Only tickets with Ticket type = USER are considered in this procedure. TEAM, ALARM, CMS tickets are not effectively handled by EGI and are considered "VO business" (directly managed by user communities).
  • Try to resolve ticket. If this is not possible, reassign to proper 2nd line Support Unit FAQ_Responsible_Units_(GGUS)
  • The TPM team evaluates the priority suggested by the user when the ticket is opened, checks if it is alligned with the rules for priority assignment (see Definitions) and if needed it corrects the priority; this means that the user can suggest a priority, but the authoritative unit to set the priority is always the HelpDesk.
  • The TPM team members feed the knowledge base of the Known Error Database provided and maintained by the Problem Management process.
  • The TPM reacts with time specified in EGI_DMSU_Ticket_Priorities
  • If needed, the TPM can make corrections on the other parameters of the ticket set by the user if needed (type of issue, ticket category, etc.)
  • TPM decides whether the ticket contains:
    • operational incident: service degradation that's still within acceptable level according to the SLA. (e.g. outage of a site for a short period of time). In this case TPM continues with the procedure.
    • a complaint: degradation of a service at a level beyond what is tolerated by the SLA. (e.g. outage of a site 50% of the time). In this case the TPM team assigns the ticket to the EGI Operations directly. EGI Operations will then execute CRM4 Manage service complaints from the customer on behalf of the user, and close the ticket providing a reference to the service complaint opened with CRM and when it confirmed that the complained is properly handled by CRM.
    • a service request: request for support to use a specific service. In this case the TPM team assigns the ticket to the EGI User Community Support Team (ucst@egi.eu). This team will (following CRM1) will initiate customer support, providing response to the user through the helpdesk ticket.

The first line support is provided by an organisation called TPM – Ticket Processing Manager. The TPM team has members who have very good general knowledge. It is an organisation populated by people provided from the Czech NGI. This organisation is responsible for the routing and processing of all active tickets.


22nd line support team (DMSU)

The DMSU reacts with the time specified in EGI_DMSU_Ticket_Priorities

As assigned in step 2.1. For tickets that need developers intervention tickets are assigned to 3rd level support.


The second line support is formed by many support units. Each support unit is formed from members who are specialists in various areas of grid middleware, or NGI/ROC supporters for operations problems, or VO specific supporters. The membership of the support units is maintained on mailing lists.

33rd line support teamAs assigned in step 2.2. The third line support is formed by many support units. Each support unit is formed from members who are developers of various software.
 3
Escalation

 1Customer or userA ticket can be escalated within the tool

Three escalation levels are available in EGI Helpdesk for users/customers:

  • Escalating ticket to the support unit it is assigned to,
  • Escalating the ticket to the support unit and the TPM on shift,
  • Escalating the ticket to the support unit, the TPM and the GGUS ticket monitoring.

The escalation levels are reached one by one. It is not possible to choose one of them.


 2GGUS ticket monitoring

Longstanding tickets are a subject of escalation.

  • Operations Tickets (type OPS)
  • Middleware Tickets
    • The monitoring of middleware tickets follows the processes described in this FAQ
    • If there are doubts about tickets, please submit a ticket to Operations SU referencing the tickets in question. Operations SU will then escalate this to Technology Collaboration Board
  • Other Tickets
4
Closure


1st line support  (TPM)

2nd line support

3rd line support team 

Each support unit can set a ticket to following terminate statuses:

  • solved: a solution is found
  • unsolved: for tickets that can not be solved due to any reason. 



Customer/User

Each customer can set a ticket to following terminate statuses:

  • solved: a solution is found
  • unsolved: for tickets that can not be solved due to any reason

When a ticket is set to 'solved' status user/customer may verify the ticket: This status indicates that a user/customer is happy with the provided solution.

"Verified" tickets cannot be further updated, nor re-opened.




EGI Helpdesk toolSolved or unsolved tickets not verified by the submitter are set to “closed” status automatically after 10 working days.



Selecting either a site from the "Notify SITE" drop-down menu or a support unit from the "Assign to support unit" drop-down menu routes the ticket directly to the selected support unit. If selecting a site, the name of the NGI/ROC the site belongs to is set automatically. Hence the ticket is assigned to the relevant NGI/ROC. Additionally the site will receive a notification about the ticket.