General information
- the Operations meeting will be on the 2nd Monday of the month
- the EGI Operations Meeting schedule for first half of 2016 is available on Indico: https://indico.egi.eu/indico/categoryDisplay.py?categId=32 and on the new summary page: https://wiki.egi.eu/wiki/Operations_Meeting
Middleware
CMD
- CMD-OS (OpenStack) released http://repository.egi.eu/category/os-distribution/cmd-os-1/
- a known issue has been documented with respective workaround, to be fixed in the next update
- update planned by March
- Keystone-VOMS 9.0.3
- ooi 0.3.2
- gridsite 2.3.3
- Cloud BDII Information provider 0.6.12
- starting working on CMD for OpenNebula
UMD
- working on UMD 4.4.0 (February release)
- FTS 3.5.7, ARC 15.03.10, DPM 1.8.11 (SL6) and 1.9 (C7), more coming
- Xrootd in EPEL-stable ( 4.5.0) looking for sites to test it
- Update to frontier-squid-3 in UMD4
- major upgrade and it has some incompatibilities with frontier-squid-2 based versions, as detailed here: https://twiki.cern.ch/twiki/bin/view/Frontier/InstallSquid#Upgrading
- the two versions will have different package names
Preview repository
Released on 2017-01-19:
- Preview 1.8.0 AppDB info (sl6): dpm-dsi 1.9.11, frontier-squid 3.5.23-2.1, LFC 1.9.0
- Preview 2.8.0 AppDB info (CentOS 7): dpm-dsi 1.9.11, frontier-squid 3.5.23-2.1, LFC 1.9.0
Note: EGI provides the preview repository without any additional quality assurance process, but the products are released as they are provided by the product team. EGI recommends the use of the UMD repositories, which contain software verified through the quality assurance process of UMD.
Operations
Feedback from Helpdesk
- [2016-12-13] Services using JGlobus fail with RFC proxies from certificates from some CAs
- Affecting dCache < v2.14, BeStMan
- Services using JGlobus fail with RFC proxies having Non-Repudiation key usage flag set, e.g. those created by usual voms-proxy-init from Grid Canada certificate
- https://ggus.eu/?mode=ticket_info&ticket_id=124650
IPv6 readiness plans
- December OMB presentation: https://indico.egi.eu/indico/event/2815/ WLCG is going to deploy their services under dual-stack mode (v4+v6) by April 2018
- EGI Operations started checking the core services against IPv6 compatibility
- EGI Operations is going to assess the IPv6 readiness of the EGI infrastructure
- early draft plan
- Technology Providers: assess the IPv6 readiness of the middleware products
- UMD team: perform verification of all services under IPv6 setup
- EGI core services developers: assess the IPv6 readiness of the products
- EGI core services hosts: assess the IPv6 readiness of the services
- Resource Centres: assess the IPv6 readiness of the site infrastructure (real machines, cloud managers)
- NGIs/ROCs please start discussing with sites and provide suggestions for the overall plan
Decommissioning of dCache 2.10
- support for the dCache 2.10 ended at December 2016
- according to EGI policies, dCache 2.10 must be decommissioned https://wiki.egi.eu/wiki/PROC16_Decommissioning_of_unsupported_software
- broadcast sent on Feb2 https://operations-portal.egi.eu/broadcast/archive/1631 + email sent on Feb7 to noc-managers mailing list
- sites to upgrade their 2.10 endpoints to a newer "golden release" of dCache
- 2.13, whose support ends on July 2017, which means in about 7 months from now, or
- 2.16, whose support ends on May 2018: here take care that the dCache team does not support the upgrade from 2.10 directly to 2.16; only 2.10->2.13 and 2.13->2.16 transitions are supported.
- decommissioning campaign will be started by EGI Operations to monitor the upgrade of the dCache 2.10 instances and follow up with the NGIs/sites; tickets will be opened this week
- deadline is end of April, having all the sites more than 2 months to plan and perform the upgrade
- probe will be WARNING for two months until April 17th, when it will switch to CRITICAL
- in May EGI Operations will open tickets against sites still publishing dCache 2.10 and follow up on the upgrade plans
- reference: https://www.dcache.org/downloads/1.9/index.shtml
Testing of the storage accounting
As discussed during the January OMB, the APEL team would need one site per NGI for testing the storage accounting. The eligible sites are the ones providing either dCache or DPM storage elements.
More information can be found in the following wiki: https://wiki.egi.eu/wiki/APEL/Storage
List of sites available for test.
Proposal to modify the declaration of scheduled interventions
Currently (see MAN02 Service intervention management) scheduled interventions (of any duration) MUST be declared at least 24 hours in advance, specifying reason and duration; any intervention declared less than 24 hours in advance will be considered unscheduled.
WLCG proposed the following modification:
- a scheduled intervention shorter than 5 days must be declared at least 24 hours in advance
- a scheduled intervention longer than 5 days must be declared at least 1 month in advance
- any other intervention that don't fulfill the rules above will be considered unscheduled
Monthly Availability/Reliability
- Underperformed sites in the past A/R reports with issues not yet fixed:
- AsiaPacific GGUS 125427
- TW-NCUHEP: site-bdii unstable
- NGI_DE GGUS 125430
- UNI-SIEGEN-HEP: waiting for the fix for CREAM probe.
- NGI_NL: GGUS 123532
- BelGrid-UCL: UNKNOWN status returned by CREAM probes, waiting for the fix for CREAM probe.
- NGI_UA:
- UA-NSCMBR GGUS 125839: on nagios the ARC-CE tests are OK, on ARGO it is reported an UNKNOWN status
- AsiaPacific GGUS 125427
- Sites suspended after past A/R reports:
- TUDresden-ZIH (NGI_DE)
- Underperformed sites after 3 consecutive months and underperformed NGIs:
- WUT (NGI_PL) GGUS 126367
ARGO proposal to use GOCDB as the only source of topology information
- slides in October Operations Meeting agenda
- ARGO Proposal (September OMB)
- ARGO and GOC-DB updates from November OMB
- Timescale:
- New GOC-DB release on Dec 7th including a boolean ‘monitored’ flag for the service endpoints
- Then creation of a web UI view for uncertified sites in ARGO
- Uncertified sites will be asked to fill in the service endpoints information. Follow the How to add URL service endpoint information into GOC-DB
- (OPTIONAL) use the GOC-DB test instance for testing the procedure
- As information is added in the GOCDB, uncertified sites/services will be picked up by the ARGO Monitoring Engine and they will start to be monitored
- By Q2 2017: support for multiple service endpoints
VAPOR
- VAPOR 2.1 released in September, it replaced GSTAT
- important for presenting the amount of computing and storage resources of the infrastructure
- new version 2.2 is about to be released:
- please test it going on the dev instance http://operations-portal.egi.eu/vapor_dev
- working on some known issues of the previous version
- each NGI should review the information provided by their sites and let us know any inconsistency: https://operations-portal.egi.eu/vapor_dev/resources/GL2ResSummary
- we need your feedback to improve the service
- report any comment into https://ggus.eu/index.php?mode=ticket_info&ticket_id=124872
AOB
Next meeting
- EGI is working on bringing up a pilot EGI-branded dockerhub service
- Mar 13th, 2017 https://indico.egi.eu/indico/event/3141/
- new calendar available until June 2017 https://indico.egi.eu/indico/category/32/