General information
TopBDII flooding unnecessary connections
It has been reported to EGI Operations that the TopBDII is trying many unnecessary connections to 2180/tcp because of how it's configured. This is basically due to the OSG flag still configured to "true" in /etc/glite/glite-info-update-endpoints.conf, resulting in a hourly cronjob to try establishing many useless connections.
It is enough to set OSG=false in order to prevent this useless traffic. Sites providing TopBDII are kindly asked to apply OSG=false to their configuration. An update to the glite-info-update-endpoints package will be released in UMD4 in the next days together with the regular UMD update.
Middleware
- UMD 4.8.0 released on November 29th http://repository.egi.eu/2018/11/29/release-umd-4-8-0/ including APEL-SSM 2.3.0, XRootD 4.8.4, CernVM-FS 2.5.1, CERN Frontier squid 3.5.28-3.1, DPM 1.10.4, Dynafed 1.3.2, BDII 5.2.25, glite-info-update-endpoint 3.0.1, GRAM5 14.1.0, GridFTP 13.8.1
- UMD 4.8.1 is an emergency fix of the fourth Major release of UMD. It adds a fix for FTS-Monitoring in CentOS7 that updates dependencies from python-django to python-django16 http://repository.egi.eu/2018/12/12/release-umd-4-8-1/
Preview repository
- 2018-11-16
- Preview 1.20.0 AppDB info (sl6): ARC 15.03 update 19, davix 0.6.9, dCache 3.2.38, DPM 1.10.4 and dpm-xrootd 3.6.6
- Preview 2.20.0 AppDB info (CentOS 7): ARC 15.03 update 19, davix 0.6.9, dCache 3.2.38, DPM 1.10.4 and dpm-xrootd 3.6.6
Operations
ARGO/SAM
FedCloud
Feedback from DMSU
Notifications from ARGO about the nagios probes failures
In the process of implementing the notification system in ARGO, it was introduced the following changes in GOC-DB:
- Notifications flag at the site level
- Notifications flag at the service endpoint level
In this way ARGO will retrieve from GOC-DB the information about the sites and services whom sending the email notification and the related recipients.
The logic of the notifications is the following:
Site | Service | Notify? |
---|---|---|
Y | Y | Y |
Y | N | N |
N | Y | N |
N | N | N |
If there isn't any email contact defined at the service endpoint level, it will be used the site contact. Please review your contacts.
You can enable the notifications.
NOTE: It is not mandatory for the sites
Monthly Availability/Reliability
- Under-performed sites in the past A/R reports with issues not yet fixed:
- NGI_FI: https://ggus.eu/index.php?mode=ticket_info&ticket_id=138670
- FI_TUT: problem with org.nordugrid.ARC-CE-IGTF which is returning "Script exited with code 1."
- NGI_PL:
- ICM: https://ggus.eu/index.php?mode=ticket_info&ticket_id=138062 very long downtime, recommended the suspension
- WCSS64: SRM not in BDII https://ggus.eu/index.php?mode=ticket_info&ticket_id=138672 (SOLVED)
- NGI_FI: https://ggus.eu/index.php?mode=ticket_info&ticket_id=138670
- Under-performed sites after 3 consecutive months, under-performed NGIs, QoS violations: (December 2018):
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139056
- ZA-WITS-CORE: CE and SE unreachable
- AsiaPacific: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139057
- KR-KNU-T3 (preventive): SRM certificate
- NGI_DE: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139058
- FZK-LCG2: CAs not properly updated
- LRZ: GRAM5 failures
- NGI_NL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139059
- BEgrid-BELNET: bug in OpenNebula, as reported here https://ggus.eu/index.php?mode=ticket_info&ticket_id=138504
- AfricaArabia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=139056
suspended sites: AEGIS04-KG
Decommissioning EMI WMS
WMS servers can be decommissioned. Please follow the procedure PROC12. The plan is:
- Starting from January 2018, put the WMS servers in draining: this will block the submission of new jobs and will allow the jobs previously submitted to finish
- inform in advance your users that you are going to put in draining and then dismiss the WMS servers (as per PROC12)
- there might be several VOs enabled on your WMS servers: in case only few of them need to use the service for few weeks more, you might disable the other VOs
- On Dec 14th EGI Operations sent a new broadcast to the VOs reminding the users the forthcoming WMS decommission
- From March, the nagios probe eu.egi.sec.WMS will return a CRITICAL status for the servers still in production, and the ROD teams will open a ticket to the sites that haven't finished the decommission process yet
WMS servers in downtime on GOC-DB
5 WMS still registered on GOC-DB as production and monitored
Tickets opened Oct 4th:
- BUDAPEST: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137553 (SOLVED)
- WCSS64: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137555 (SOLVED)
- CYFRONET-LGC2: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137557 (to decide if replacing it with DIRAC or shut it down)
- AEGIS01-IPB-SCL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137558 (SOLVED)
- MA-01-CNRST: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137559 (downtime expired, to check the WMS status)
- SARA-MATRIX: https://ggus.eu/index.php?mode=ticket_info&ticket_id=137560 (planned to set-up a national DIRAC server for replacing the WMS, Q1 2019)
IPv6 readiness plans
- please provide updates to the IPv6 assessment (ongoing) https://wiki.egi.eu/w/index.php?title=IPV6_Assessment information will be summarised at OMB (probably February)
webdav probes in OPERATORS profile
The webdav probes was included in the ARGO_MON_OPERATORS profile after the approval in the January 2018 OMB: in this way the failures will generate an alarm on the dashboard, and the ROD teams can open a ticket to the failing sites. If no particular issue occurs, and if at least 75% of webdav endpoint are passing the tests, the probes will be added in the ARGO_MON_CRITICAL profile, so the results of these probes will be taken into account for the A/R figures.
- webdav endpoints registered in GOC-DB: https://goc.egi.eu/gocdbpi/public/?method=get_service&&service_type=webdav
- link to nagios results: https://argo-mon.egi.eu/nagios/cgi-bin/status.cgi?servicegroup=SERVICE_webdav&style=detail
- 9 CRITICAL out of 34 (OK status: 73.5%)
List of sites that not have completed the configuration yet:
- NGI_HR: egee.irb.hr https://ggus.eu/index.php?mode=ticket_info&ticket_id=134458 (webdav URL missing in GOC-DB...)
List of sites that disabled webdav: UNIGE-DPNC, GR-01-AUTH, HG-03-AUTH, CETA-GRID, WUT
Storage accounting deployment
During the September 2017 meeting, OMB has approved the full-scale deployment of storage accounting. The APEL team tested it with a group of early adopters sites, and the results prove that storage accounting is now production-ready.
Storage accounting is currently supported only for the DPM and dCache storage elements therefore only the resource centres deploying these kind of storage elements are requested to publish storage accounting data.
In order to properly install and configure the storage accounting scripts, please follow the instructions reported in the wiki: https://wiki.egi.eu/wiki/APEL/Storage
IMPORTANT: be sure to have installed the star-accounting.py script v1.0.4 (http://svnweb.cern.ch/world/wsvn/lcgdm/lcg-dm/trunk/scripts/StAR-accounting/star-accounting.py)
After setting up a daily cron job and running the accounting software, look for your data in the Accounting Portal: http://goc-accounting.grid-support.ac.uk/storagetest/storagesitesystems.html. If it does not appear within 24 hours, or there are other errors, please open a GGUS ticket to APEL who will help debug the process.
Test portal: http://accounting-devel.egi.eu/storage.php
IMPORTANT: Do not encrypt the storage records with your host certificate, please comment out the “server_cert” variable in sender.cfg
List of sites already publishing and of tickets opened is reported here.
Several sites are not publishing the storage accounting data yet. NGIs please follow-up with the sites the configuration of the script in order to speed-up the process.
- AsiaPacific: TW-NCUHEP https://ggus.eu/index.php?mode=ticket_info&ticket_id=131426
- NGI_AEGIS: AEGIS04-KG https://ggus.eu/index.php?mode=ticket_info&ticket_id=131431
- NGI_CH:
- CSCS-LCG2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131432 (it needs some help from dCache support)
- NGI_DE:
- NGI_IT:
- INFN-ROMA1-CMS https://ggus.eu/index.php?mode=ticket_info&ticket_id=131482 (postponed)
- NGI_NDGF: FI_HIP_T2 https://ggus.eu/index.php?mode=ticket_info&ticket_id=131491
AOB
Next meeting
- February 11th, 2019 https://indico.egi.eu/indico/event/4320/
- Check meetings for the first half of 2019 in the calendar https://indico.egi.eu/indico/category/32/