General information
Middleware
UMD
- CentOS8 discussion still ongoing
- The UMD 4.12.6 release contains the following products:
- ARGO 0.5.4 - ams-library py2 RPM also packages py3 specific modules https://github.com/ARGOeu/argo-ams-library/releases/tag/V0.5.4-1
- APEL-SSM 3.2.0 - bug fixes and improvements https://github.com/apel/ssm/releases/tag/3.2.0-1
- The UMD 4.13.0 release includes the following products:
- APEL 1.9.0 - Added AMS support. Requires at least SSM version 3.2.0. Bug fixes and improvements. https://github.com/apel/apel/releases/tag/1.9.0-1
- VOMS admin server 3.8.1, VOMS Admin client 2.0.21, VOMS server 2.0.16, VOMS C/C++ APIs 2.0.16 - bug fixes and enhancements http://italiangrid.github.io/voms/release-notes/voms-admin-server/3.8.1/
- CVMFS 2.8.1 - Bug Fixes and Improvements https://cvmfs.readthedocs.io/en/2.8/cpt-releasenotes.html
- The UMD team is currently working on restoring the repository frontend that hosts the web pages, which hopefully will be soon available
Preview repository
- released on 2021-04-09
- Preview 2.32.0 AppDB info (CentOS 7): APEL Client/Server 1.9.0, APEL-SSM 3.2.0, xrootd 5.1.1
- next release: ARC 6.11.0, STORM 1.11.20, VOMS 04-21
Operations
ARGO/SAM
- HTCondor-CE probes
- deployed on secmon and pakiti: GGUS 150006
- working on the probe for the host certificate validity check: GGUS 147386
- With 8.9.12 installed (expected the week of Mar 15), you should be able to query remote HTCondor-CEs for their host certificate using the following:
$ python -c 'import htcondor; ad = htcondor.Collector("collector2.opensciencegrid.org:9619").locate(htcondor.DaemonTypes.Schedd, "hosted-ce10.opensciencegrid.org"); print htcondor.SecMan().ping(ad, "READ")["ServerPublicCert"]' | openssl x509 -noout -subject -enddate subject= /CN=hosted-ce10.opensciencegrid.org notAfter=Apr 26 12:26:42 2021 GMT
- a new version of HTCondor (9.0.0) will be added to the UMD Test repo and then the probe can be deployed on the testing instance of ARGO
FedCloud
Feedback from DMSU
New Know Error Database (KEDB)
The KEDB has been moved to Jira+Confluence: https://confluence.egi.eu/display/EGIKEDB/EGI+Federation+KEDB+Home
- problems are tracked with Jira tickets to better follow-up their evoulution
- problems can be registered by DMSU staff and EGI Operations team
Monthly Availability/Reliability
- Under-performed sites in the past A/R reports with issues not yet fixed:
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150816
- GoeGrid: miscellaneous issues with ARC-CE have been solved
- NGI_HR: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148518
- egee.irb.hr: major upgrade from CentOS 6 to CentOS 7; tests currently fail due to UNKNOWN status returned
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150818
- INFN-PISA: HTCondorCE and SRM failures
- ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148956
- CBPF: DPM updated; SRM failures due to information not properly published, fixed; other SRM failures due to available space
- ROC_LA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150817
- ICN-UNAM: replaced CREAM-CE; SE certificate expired; new failures with HTCondorCE.
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=150816
- Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (April 2021):
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148468
- ZA-WITS-CORE: SRM certificate expired and failures after DPM upgrade, the information wasn't properly published.
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=151846
- KR-KISTI-GSDC-01: firewall issues
- NGI_IT: https://ggus.eu/index.php?mode=ticket_info&ticket_id=151844
- CNR-ILC-PISA: downtime for CREAM-CE decommission and migration to ARC-CE; issues in configuring torque/maui.
- INFN-BARI
- INFN-ROMA3: failures occurred during migration from CREAM-CE to HTCondorCE, and with migration to CentOS7; fixed SRM configuration.
- INFN-TRIESTE: the site was in downtime for migration to HTCondorCE.
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=151847
- UA-IFBG: CE configuration issues, DNS misconfiguration.
- Russia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=151845
- ITEP
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148468
- sites suspended:
IPv6 readiness plans
- please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment
- if any relevant, information will be summarised at OMB
APEL migration from ActiveMQ to ARGO Message Service (AMS)
- Migration insructions: https://github.com/apel/ssm/blob/dev/migrating_to_ams.md
- ActiveMQ is going to be dismissed at the end of May
- Releasing a new version of Apel Client (1.9.0) compatible with the new AMS protocol when used to trigger the publication of the accounting records
- APEL SSM works fine since 2.4.0 version
- The accounting component of ARC-CE still uses the STOMP protocol to send the message records
- The developers are working on a new version compatible with AMS
- some sites will be asked to test the new version when available
- Cloud accounting campaign:
- HTCondorCE and Storage accounting campaign:
- Most common issues:
- mismatch between the host certificate subject registered in GOCDB and the real DN
- SAN field missing / wrongly defined in the host certificate
- DNS entries not completely defined
- same host used to send different types of accounting records
- a new version of ARGO Message Service mitigates the problems related to DNS entries and the SAN field:
Prerequisites for using AMS
- A valid host certificate from an IGTF Accredited CA.
- A GOCDB 'Site' entry flagged as 'Production'.
- A GOCDB 'Service' entry of the correct service type flagged as 'Production'. The following service types are used:
- For Grid accounting use 'gLite-APEL'.
- For Cloud accounting use 'eu.egi.cloud.accounting'.
- For Storage accounting use 'eu.egi.storage.accounting'.
- The 'Host DN' listed in the GOCDB 'Service' entry must exactly match the certificate DN of the host used for accounting. Make sure there are no leading or trailing spaces in the 'Host DN' field.
CREAM-CE Decommission
- End of Security Updates and Support: 31st Dec 2020
- Original broadcast: https://operations-portal.egi.eu/broadcast/archive/2293
- Decommissioning deadline: 31st Jan 2021
- PROC16 Decommission of unsupported software
- Decommissioning start date: Oct 1st 2020
- a probe detecting CREAM-CE endpoints will be run, returning WARNING status
- GGUS ticket: https://ggus.eu/index.php?mode=ticket_info&ticket_id=148715
- eu.egi.sec.CREAMCE
- Nov 1st: probe returns CRITICAL status, alarms created on the ROD dashboard, ROD teams start to create tickets
- 1st Feb 2021: EGI Ops will start chasing the sites still providing CREAM-CE endpoints
- By this time service end-points which couldn't be upgraded should be put into downtime by site admin or ROD
- 1st March 2021: Sites still deploying unsupported service endpoints risk suspension, unless documented technical reasons prevent a Site Admin from updating these endpoints.
- Tickets opened: 49
- link to the list
- Please note that at least one CE endpoint should be associated to the APEL service type in order to monitor the publication of the accounting data, as explained here
- If the CE you are going to remove was also registered as APEL service type, do not forget to move the APEL service type to a different CE endpoint.
VOMS upgrade to CentOS 7
- VOMS for CentOS 7 released Nov 23rd with UMD 4.12.13
- VOMS Admin 3.8.0, VOMS Server 2.0.15
- VOMS endpoints registered on GOCDB as production and monitored: 41
- Provided by 33 sites
- list of ticket opened: GGUS
- total: 31. Solved: 20.
- the VOMS servers need to be published in the BDII in order to easily collect the deployed version
AOB
Next meeting
14th Jun 2021