General information (TO UPDATE)
Middleware (TO UPDATE)
UMD
- Next scheduled release is February's UMD 4.10.0: https://wiki.egi.eu/wiki/UMD_Release_Schedule
- delayed due to difficulties with StagedRollout communications for some products
- plans on CentOS8 STARTED
- integration of HTCondor-CE still finalizing
- HTCondor-CE released in UMD 4.9.1 http://repository.egi.eu/2019/12/04/release-umd-4-9-1/ pointing to the original documentation (the documentation is OK)
- still needed to contribute to the WN package
- Are NGIs willing to investigate among their site admins about condor/HTCondor-CE skills and contribute with available recipes or help? -> no feedback at last EGI Ops meeting
CMD
- CMD-OS released http://repository.egi.eu/2020/02/18/release-cmd-os-1-3-0/
Preview repository
- released on 2020-01-21:
- Preview 1.26.0 AppDB info (sl6): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1
- Preview 2.26.0 AppDB info (CentOS 7): ARC 6.4.1, CVMFS 2.7.0, dCache 5.2.13, gfal2 2.17.1, STORM 1.11.17, xrootd 4.11.1
Operations
ARGO/SAM
- Status of glue-validator probe https://ggus.eu/index.php?mode=ticket_info&ticket_id=146053
- it seems this probe is no more useful
- for us the relevant information are: OS and middleware version, benchmark, number of cores and amount of storage
- Status of UNICORE probes https://ggus.eu/index.php?mode=ticket_info&ticket_id=146054
- unicore probes can be removed
- Status of various probes https://ggus.eu/index.php?mode=ticket_info&ticket_id=146055
- probe for desktop grid jobs can be removed
- probes for argus, onedata, webdav will be kept
FedCloud
cASO upgade campaign on OpenStack sites
Feedback from DMSU
Monthly Availability/Reliability
- Under-performed sites in the past A/R reports with issues not yet fixed:
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145457
- MA-01-CNRST: IGTF failures
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142591
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145458
- GARR-01-DIR: failures during the DPM upgrade
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144715
- INFN-CATANIA-STACK: failures with IGTF and OpenStack probes
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143945
- ICM: qcg and SRM failures; statistics are improving...
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=140557
- TASK: QCG problems was fixed; some CREAM-CE failures; the SRM tests are still failing and the DPM upgrade wasn't completed yet
- NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=142160
- RO-02-NIPNE: Cooling system problem, migration to CentOS7, long downtime; to return in production soon, SRM decommissioned...
- NGI_UK:
- UKI-SOUTHGRID-SUSX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=144720 Migration from CREAM to ARC, WN migration to CentOS7; SRM and ARC-CE started to fail...
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145457
- Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (Feb 2020):
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145817
- T2-TH-SUT: hardware problems, host CAs expired; improving...
- NGI_CH: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145818
- UNIGE-DPNC: only the site-bdii is in production. The site is undergoing re-organisation, the new ARC CE service will be back in production shortly.
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145815
- INFN-BARI
- NGI_SI: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145811 (SOLVED)
- ARNES: SRM failures due to incorrect information published
- ROC_LA https://ggus.eu/index.php?mode=ticket_info&ticket_id=145812
- GRID-UNAM
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=145817
- sites suspended:
IPv6 readiness plans
- INFORMATION ARE 6 MONTHS OLD, please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
- if any relevant, information will be summarised at OMB
LCGDM end of support and migration to / enabling of DOME
- The DPM team has agreed to extend support for security updates for the DPM legacy functionality until 30 September 2019. However, affected service providers should still plan to disable legacy mode well before this date.
- more details: https://wiki.egi.eu/wiki/DPM_End_of_legacy-mode_support
- since DPM 1.10.3 release, it is possible enabling the non-legacy mode DOME (Disk operations Management Engine, see documentation)
- latest DPM release 1.13.1 has been also released in UMD 4.8.5
- EGI Operations sent a broadcast to site-admins(first bunch and second bunch) and VO managers with a survey to fill in:
- site-admins: to know the upgrade plans: https://www.surveymonkey.com/r/2PBZSNB
- VO Managers: to know any barriers in moving away from using SRM protocol https://www.surveymonkey.com/r/2ZMZJVG
- Deployment statistics (Mar 16th):
$ ldapsearch -x -LLL -H ldap://egee-bdii.cnaf.infn.it:2170 -b "GLUE2GroupID=grid,o=glue" '(&(objectClass=GLUE2Manager)(GLUE2ManagerProductName=DPM))' GLUE2ManagerProductVersion GLUE2ManagerID | grep GLUE2ManagerProductVersion | sort | uniq -c
1 GLUE2ManagerProductVersion: 1.10.0
1 GLUE2ManagerProductVersion: 1.12.0
66 GLUE2ManagerProductVersion: 1.13.0
2 GLUE2ManagerProductVersion: 1.13.1
8 GLUE2ManagerProductVersion: 1.13.2
5 GLUE2ManagerProductVersion: 1.8.10
1 GLUE2ManagerProductVersion: 1.8.8
1 GLUE2ManagerProductVersion: 1.8.9
5 GLUE2ManagerProductVersion: 1.9.0
Liasing with WLCG to follow-up the upgrade. Opened GGUS tickets asking the following:
- all the sites with older DPM versions than 1.12 are suggested to upgrade to the latest DPM version , following the guide DPM upgrade (chapter 1 Upgrade to DPM 1.10.0 "Legacy Flavour" and chapter 2 Upgrade to DPM 1.10.0 "Dome Flavour")
- DOME and the old LCGDM (srm protocol) will coexist
- Monitoring: sites should enable the monitoring of the HTTP/WebDav and/or GridFTP endpoints
- register the storage service endpoint as webdav and/or globus-GRIDFTP service type, with production flag disabled, providing respectively the URL field and the Extension Properties information as explained in the HOWTO21
- check if the tests are ok
- switch the production flag to "yes"
List of tickets
- DPM upgrade, DOME enabling, and monitoring (33)
- enabling DOME and proper monitoring (6)
- tickets opened by WLCG: 13 and 7
HTCondorCE integration
Link to procedure: https://wiki.egi.eu/wiki/PROC19
GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139377
Remaining steps:
- (To do) OPS Dashboard:
- To do when Conf Mgmt and Monitoring are completed.
- (in progress) Information System: the info-provider is already available: https://github.com/opensciencegrid/htcondor-ce/tree/master/contrib/bdii
- to test during the staged rollout
- (in progress) Monitoring: some probes are already available that submit jobs to CondorCe, CREAM and ARC-CE:
- https://gitlab.cern.ch/etf/jess
- ticket to ARGO to deploy the probes on the test instance: https://ggus.eu/index.php?mode=ticket_info&ticket_id=141177
- there is a bug on CentOS6 due to python version; the probe will be tested on the CentOS7 nagios instance
SECMON failures
Several CEs are failing the job submission tests, preventing pakiti to check the vulnerabilities fixes on the WNs.
- original ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=143837
- List of tickets to the sites
- https://ggus.eu/index.php?mode=ticket_info&ticket_id=144732
AOB
Next meeting
March 9th https://indico.egi.eu/event/4897/