General information

Middleware

Software repository maintenance

  • unscheduled downtime from 03-Mar-21 17:00:00 (UTC) to 05-Mar-21 14:30:06 (UTC)
  • Broadcasts announcing the unexpected downtime and the recovery
  • Migration from IASA to IBERGRID
  • Warning downtime until 09-Mar-21 14:30:00 (UTC)
  • repository is completely online; the related website not yet (only some very simple html pages)
  • more details of the incident will be circulated soon

UMD

  • CentOS8 discussion still ongoing
  • migration of Software Provisioning infrastructure to IBERGRID still ongoing
    • in particular, administration portal used for release creation done successfully
  • still planning next release, delay due to repo migration

Preview repository

  • released on 2021-02-22
    • Preview 2.31.0 AppDB info (CentOS 7): APEL-SSM 3.1.1, ARC 6.10.1, CVMFS 2.8.0 and egi-cvmfs-3-1.13, davix 0.7.6, dCache 5.2.38, gfal2 2.18.2

Operations

ARGO/SAM

  • Site-BDII metrics org.bdii.Entries and org.bdii.Freshness removed from ARGO_MON_CRITICAL profile
    • the metrics are still kept in the ARGO_MON_OPERATORS profiles
    • it is still an important service to support infrastructure oversight activities
  • HTCondor-CE probes
    • deployed on secmon and pakiti: GGUS 150006
    • working on the probe for the host certificate validity check: GGUS 147386
      • With 8.9.12 installed (expected the week of Mar 15), you should be able to query remote HTCondor-CEs for their host certificate using the following:
$ python -c 'import htcondor; ad = htcondor.Collector("collector2.opensciencegrid.org:9619").locate(htcondor.DaemonTypes.Schedd, "hosted-ce10.opensciencegrid.org"); print htcondor.SecMan().ping(ad, "READ")["ServerPublicCert"]' | openssl x509 -noout -subject -enddate
subject= /CN=hosted-ce10.opensciencegrid.org
notAfter=Apr 26 12:26:42 2021 GMT
  • CREAM-CE metrics removed from ARGO_MON, ARGO_MON_OPERATIONS and ARGO_MON_CRITICAL (GGUS 149778)
    • emi.cream.CREAMCE*
    • eu.egi.CREAM*

FedCloud

Feedback from DMSU

Monthly Availability/Reliability

IPv6 readiness plans

APEL migration from ActiveMQ to ARGO Message Service (AMS)

  • Migration insructions: https://github.com/apel/ssm/blob/dev/migrating_to_ams.md
  • ActiveMQ is going to be dismissed at the end of March with the end of EOSC-hub
  • Currently an issue with apel client prevent SSM to send properly the records through AMS
    • it doesn't affect cloud and storage accounting
    • ARC-CE might not work if using an old bundled version of SSM - but new ARC versions may work if set to use standalone SSM
    • With CondorCE it may work, we will find some sites to test it
    • by mid-March a fix will be released; then the sites with ARC-CE/HTCondorCE can implement the change
  • starting the migration with FedCloud sites

Feedback from NGI_FRANCE

On the Cloud infra, several tickets have been open to switch to the new messaging system. It would nice to have the following RPMs made available from CMD repo, and not only from UMD:

  • apel-ssm-2.4.1-1.el7.noarch
  • python-argo-ams-library-0.5.1-1.el7.noarch

In addition, many Cloud sites are now using OpenStack Stein or newer. These version are provided with python-daemon = 2.2.3-1.el7. It conflicts with the requirement of apel-ssm ( python-daemon < 2.2.0)

Feedback from URT:

  • there is new apel-ssm 3.0.0 version under untested repo and this new version solves the dependency issue of requirement python-daemon <= 2.2.0.
  • there is also the new python-argo-ams 0.54 library

ARC-CE probe failing due to UMD repositories being down

Job terminated as Failed. - Failed in data staging: Failed checking source replica http://repository.egi.eu:80/sw/production/cas/1/current/meta/ca-policy-egi-core.list: Failed to obtain information about file: Failed to connect to repository.egi.eu(IPv4):80 - JID: gsiftp://alex4.nipne.ro:2811/jobs/yq0NDmskJcynuvw3Vp3UrRNqABFKDmABFKDm8hJKDmABFKDmxx7PPm

CREAM-CE Decommission

  • End of Security Updates and Support: 31st Dec 2020
  • Decommissioning deadline: 31st Jan 2021
  • PROC16 Decommission of unsupported software
  • Decommissioning start date: Oct 1st 2020
  • Nov 1st: probe returns CRITICAL status, alarms created on the ROD dashboard, ROD teams start to create tickets
  • 1st Feb 2021: EGI Ops will start chasing the sites still providing CREAM-CE endpoints
    • By this time service end-points which couldn't be upgraded should be put into downtime by site admin or ROD
  • 1st March 2021: Sites still deploying unsupported service endpoints risk suspension, unless documented technical reasons prevent a Site Admin from updating these endpoints.
  • Tickets opened: 49
  • Please note that at least one CE endpoint should be associated to the APEL service type in order to monitor the publication of the accounting data, as explained here
    • If the CE you are going to remove was also registered as APEL service type, do not forget to move the APEL service type to a different CE endpoint.

VOMS upgrade to CentOS 7

  • VOMS for CentOS 7 released Nov 23rd with UMD 4.12.13
    • VOMS Admin 3.8.0, VOMS Server 2.0.15
  • VOMS endpoints registered on GOCDB as production and monitored: 41
    • Provided by 33 sites
  • list of ticket opened: GGUS
  • the VOMS servers need to be published in the BDII in order to easily collect the deployed version

AOB

Next meeting

12th Apr 2021

  • No labels