PROC02 Operations Centre creation

Created by Alessandro Paolini, last modified on 2023 Jul 31

Document control

Area	EGI Federation Operations
Procedure status	FINAL
Owner	Matthew Viljoen
Approvers	Operations Management Board
Approval status	APPROVED
Approved version and date	v7, 26 Oct 2010
Statement	The purpose of this document is to clearly describe the actions and the relative steps to be undertaken for integrating an Operations Centre into the EGI Infrastructure.
Next procedure review	upon request

Procedure reviews

The following table is updated after every review of this procedure.

Date	Review by	Summary of results	Follow-up actions / Comments
16 Aug 2021	Alessandro Paolini	copy from PROC02_Operations_Centre_creation in EGI Wiki. Updated some information.
31 Jul 2023	Alessandro Paolini	updated some old links; it is not needed any more to open a ticket to the Operations Portal: the NGI and the sites are added automaticaly to the dashboard when they are created and certified on GOCDB.

Overview

The purpose of this document is to clearly describe the actions and the relative steps to be undertaken for integrating an Operations Centre into the EGI Infrastructure.

Definitions

The Integration Process Coordinator (IPC) is the entity responsible for integrating a new Operations Centre within EGI.

Please refer to the EGI Glossary for the definitions of the terms used in this procedure.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", “MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

Validation

First step for validating an Operations Centre is to be politically approved as formed from an official partner of the EGI infrastructure.

After this step, the Operations Centre needs to be technically validated. The Operations team is responsible for performing this validation.

Political Validation

CASE 1. If the organization running an Operations Centre is already represented within the EGI Council, the respective council member notifies the Director (director<AT>egi.eu) and the Service Deploy team of the EGI Foundation (operations<AT>egi.eu) that the respective Operations Centre is entering its validation cycle. At this point, the technical validation can start.

CASE 2. If the organization running an Operations Centre is not represented within the EGI Council, director<AT>egi.eu has to be contacted formalize the collaboration (e.g. through a EGI membership or MoU). When the type of collaboration is agreed, CASE 1 applies.

Technical Validation

The EGI Foundation Director informs operations<AT>egi.eu about the agreed partnership. The IPC is appointed by the Service Delivery Lead. The Operations Centre Operations manager(s) is notified about the IPC contact.
The EGI Service Delivery team opens a GGUS ticket to the IPC to start the validation process.

Start of the integration

The integration of a new Operations Centre starts when the Service Delivery Team opens an Operations Centre validation ticket to the IPC (via GGUS).

Once the ticket is filed, the IPC can start the validation process. In order to trigger the actions described in this document the IPC creates a set of new child tickets that are assigned to the individual partners that are responsible for the various validation steps. Thereby, the integration process should be as transparent as possible to all parties involved. The required actions are described below.

An example/template for the Operations Centre creation ticket is provided here:

Subject: Creation of <Operations Centre name>


Required information for the creation of the Operations Centre:
Management mailing list : management@xxx.org
Operations Centre Operations manager contact data : Person Surname (email) +phone contact,
Deputy: Person Surname (email) +phone contact,

Operations Centre security officer contact data : Person Surname (email) +phone contact,
Operations Centre security mailing list : abuse@xxx.org

ROD team mailing list : ngi-support@xxx.org
Mailing list for GGUS tickets if using GGUS directly : xxxticket@xxx.org

The FAQ document for the Operations Centre provided by the GGUS team as described in Helpdesk SUs FAQs

Pre-requisites

Before opening the GGUS ticket, the Operations Centre should:

Make sure your operational organization is able to fulfill the Resource Provider (RP) OLA
Decide about the Operations Centre name. In case of national European Operations Centres, the name should start with "NGI_"
Decide whether to use the Operations Centre's own help desk system or use GGUS directly. If the Operations Centre wants to set up their own system they need to provide an interface for interaction to GGUS with the local ticketing system and follow the recommendations available at https://ggus.eu/index.php?mode=docu#interface.
Set the following contact points:
1. Management mailing list
2. Operations Centre operations manager contact data
3. Operations Centre security officer contact data
4. Operations Centre security mailing list
5. ROD team mailing list (including people responsible for monitoring and supporting the Operations Centre infrastructure)
6. Mailing list for GGUS tickets IF GGUS is used directly for the helpdesk system
All certified Operations Centre sites need to be monitored by ARGO Monitoring Service operated centrally by EGI.
Fill the FAQ document for the Operations Centre by creating a child page in Helpdesk SUs FAQs similarly to the other FAQs
Staff in the Operations Centre that should be granted a management role (manager, deputy manager, security officer) should first register a user account into GOCDB. The user registration procedure is described in the GOCDB user documentation, section 3.1.2.
Staff in Operations Centre is familiar with Operational Procedures (these documents are quite old. We can either point to specific procedures or to other docs.)
People who are responsible for operations should be subscribed to following mailing lists (unless differently specified):
1. Operations Centre manager:
  noc-managers [at] mailman.egi.eu - registration through https://mailman.egi.eu/mailman/listinfo
2. ROD team:
  All-operator-on-duty [at] mailman.egi.eu - list which integrate all Operations Centres ROD mailing list.
  Mailing list is populated automatically from GOCDB. New Operations Centres should make sure to record accurate information in GOCDB.
3. Security staff:
  ngi-security-contacts [at] mailman.egi.eu, NGI or OC security officers subscribe to this mailing list
  site-security-contacts [at] mailman.egi.eu, Site (only certified site) security officers subscribe to this mailing list
  Both mailing lists are populated automatically from GOCDB. New Operations Centres should make sure to record accurate information in GOCDB.
4. (Recommended) Site administrators:
  LCG-ROLLOUT [at] jiscmail.ac.uk - list gathers all site admins and is designed for technical discussions - membership not mandatory but useful
  Subscription is possible through https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=LCG-ROLLOUT&A=1

Steps

Some steps of the process can be done in parallel as they are independent, so all steps grouped within the same task number can be performed concurrently (several different child tickets will be created in order to speed up the process). The general idea is that these tickets must be closed before being able to move on to the next step.

Step	Substep	Action on	Action	Required data	Notes
1		IPC	Verification of the validity of the request (were all needed data provided?)		To check if all needed information is provided to make the process quick
2		IPC	Create child tickets in the order given as follows:
	1	GOCDB	*Creation of a new Operations Centre entry in the EGI Project (with no site attached).*	Operations Centre name: <Operations Centre name> Operations Centre management mailing list: foo@bar.org Operations Centre security mailing list: Operations Centre Operations manager: Operations Centre security officer:	To register new OC in official EGI database
	2	GGUS	Create a new support unit in GGUS	OC should fill in https://wiki.egi.eu/wiki/GGUS:NGI_XXX_FAQ or https://wiki.egi.eu/wiki/GGUS:XXX_FAQ Instruction how to create GGUS wiki page Helpdesk SUs FAQs	To make possible to create tickets in official EGI ticketing system to new OC
	3	Operations	Certification of new ROD team Procedure_to_handle_new_ROD_certification_GGUS_tickets the linked pages need some updates	Include in the GGUS ticket for : Country Operations Centre acronym ROD email list	To verify if the future ROD team members are properly trained to perform their duties
3		The newly created Operations Centre	Confirmation that Operations Centre read, accept and agree to fulfill the RP OLA
4		Dteam VO manager	Create a branch/group in Dteam VO for Operations Centre and assign DN of people who will be dteam VO representative for the Operations Centre, Operations Centre Group owner and Operations Centre Group manager. Responsibilities and terminology VO representative: A person that can approve or deny dteam VO membership requests. This person is selected by the applicant during the registration phase. Group owner: A person that can approve or deny Group membership requests. In addition he can create subgroups within his Group. Group manager: A person that can approve or deny Group membership requests. These responsibilities can be assigned to the same person(s). How to assign the child ticket: assign the ticket to "VO support" after the selection of "dteam" in the concerned VO field (the ticket can also be assigned to EGI Operations since the members of SDIS team are VO managers) This step is not blocking the process and can be done in parallel. It is required to finish this step before closing parent ticket.	Following data about each responsible person: Name and Surname DN email	To make possible to register operations staff from new OC to register in Dteam VO. This VO gather all operations staff and allow them to access the grid.
5		The newly created Operations Centre	Final confirmation that the new Operations Centre can start the operations		The confirmation means that OC will provide all services required in OLA
6		GOCDB	[If the Operations Centre was part of a OC] GOCDB transfers related sites from the source ROC to the new Operations Centre structure.		The sites moving across to the new OC in the ticket indicated in the ticket.
6		The newly created Operations Centre	[If the Operations Centre wasn't part of a OC] Newly created Operations Centre should insert at least one site		Nagios system need at least one site to be validated.
7		The newly created Operations Centre	[If the Operations Centre was part of a OC] Transfer all open operational tickets to the new Operations Centre in GGUS.		To ensure that non of the GGUS tickets were forgotten durign the process.
8		The newly created Operations Centre	Check that all the sites are visible in ARGO and that alarms show up in Operations portal
9	OPTIONAL	The newly created Operations Centre	All sites should configured GIIS according to the instructions at: MAN1_How_to_publish_Site_Information GlueSiteOtherInfo: EGI_NGI=<Operations Centre name> GlueSiteOtherInfo: GRID=EGI This step can be performed at any time from this point.		To confirm that all sites publish proper data in information system about new NGI.
10		The newly created Operations Centre	[If the Operations Centre was part of a OC] Inform managers of regional VOs to change the VO scope of their VOs to the relevant Operations Centre (national). This action require only confirmation from Operations Centre manager that information was passed. Information which should be pass to VO managers: "The Vo scope can be changed by creating a ticket to Operations portal SU in GGUS."		To spread among Vo information
11	OPTIONAL	The newly created Operations Centre	NGI_XX_SERVICES service group / XX_SERVICES service group The newly created Operations Centre should create in GOC DB NGI_XX_SERVICES (if national entity) or XX_SERVICES (if international entity) service group and attached services listed on page: NGI services in GOCDB
12		The newly created Operations Centre	(depending on the previous one) Create a GGUS ticket to "Operations" SU with information about newly created service group and official OC Top-BDII, SAM, Argus instance. Ask to add OC manager to OC managers mailing list.
13		IPC	Final checks by the IPC.		Were all steps taken and finished properly?
14		Operations	Final checks should be verified. The information that the Operations Centre is ready should be sent in Monthly broadcast and announced during OMB by EGI Operations team. (This broadcast should be sent to VO managers (except Ops and Dteam VO) and NOC/ROC managers) See the template below for an indication of the message content. Subject: <Operations Centre name> is operational Dear All, We would like to announce that <Operations Centre name> is now fully operational and that we have finished its integration procedure. All necessary operational teams and tools are established in our Operations Centre and we are ready for production. This Operations Centre is visible in all operational tools as <Operations Centre name> and is responsible for all <COUNTRY> sites. Best regards,

Page tree