General information
- the Operations meeting will be on the 2nd Monday of the month
- the EGI Operations Meeting schedule for first half of 2016 is available on Indico: https://indico.egi.eu/indico/categoryDisplay.py?categId=32 and on the new summary page: https://wiki.egi.eu/wiki/Operations_Meeting
UMD/CMD
- UMD 3.14.3 to be released, fixing problem with gpgkeys, urgent
- UMD 3.14.2 released http://repository.egi.eu/2016/06/16/release-umd-3-14-2/
- StoRM 1.11.11, dCache 2.10.56, edg-mkgrid 4.0.3 (fix), Globus Network Manager on SL6
- UMD 4.1.1 released on 27-06-2016
- edg-mkgrid 4.0.3 (missing), Globus Network Manager on SL6
- UMD 4.2.0 to be released by July
- including on CentOS7: FTS3, QCG, dCache, ARGUS
- include on CentOS7 if possible: DPM, Globus/GFAL (fix issue with udt driver)
- add regular updates on SL6
Preview repository
- 2016-07-07:
- Preview 2.3.0 AppDB info (CentOS 7): release fix for ARGUS 1.7.0
- 2016-06-20:
- Preview 1.3.0 AppDB info (sl6)
- Preview 2.2.0 AppDB info (CentOS 7)
Generic information about Preview repository: https://wiki.egi.eu/wiki/Preview_Repository
Note: EGI provides the preview repository without any additional quality assurance process, but the products are released as they are provided by the product team. EGI recommends the use of the UMD repositories, which contain software verified through the quality assurance process of UMD.
Operations
EGI central monitoring instance (ARGO)
Since July 1st, the EGI infrastructure is being monitored by two monitoring instances that can be found on these addresses:
https://argo-mon.egi.eu/nagios https://argo-mon2.egi.eu/nagios
Both instances are running the same set of tests and results provided are equivalent.
Starting from the same date, the central ARGO Web UI (http://argo.egi.eu/lavoisier ) provides information from these two instances and the Operations Portal was reconfigured to raise alarms based on information from ARGO central instances.
Results coming from NGI SAM instances are no longer consumed by the central ARGO or Operations Portal so NGIs can eventually decommission them following the standard decommissioning procedures (https://wiki.egi.eu/wiki/PROC12 ).
The FedCloud sites will be monitored by the new system starting from Aug 1st.
New set of CREAM probes
A new set of probes is being used for monitoring the CREAM CEs and the A/R computation: https://wiki.italiangrid.it/twiki/bin/view/CREAM/DjsCreamProbeNew This set of probe doesn't make use of the BDII, WMS and the messaging infrastructure like instead did the old WN monitoring framework.
RFC proxy will be default
- moving to RFC proxy instead of legacy proxy
- in production since a while, everybody is using RFC
- we will ask VOMS TP to make a little modification on VOMS client, changing the default
New configuration for DTEAM VO
The HellasGrid Certification Authority changed its DN from "/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2006" to "/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2016"
Since it is also changed the certificate of the 2 voms servers hosting dteam VO, the settings of this VO need to be updated accordingly on *ALL THE (grid and cloud) SERVICES*
- New yaim settings (for the ../vo.d/dteam file):
VOMS_SERVERS='vomss://voms.hellasgrid.gr:8443/voms/dteam?/dteam/' VOMSES="'dteam voms.hellasgrid.gr 15004 /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr dteam 24' 'dteam voms2.hellasgrid.gr 15004 /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr dteam 24'" VOMS_CA_DN="'/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2016' '/C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2016'"
- .lsc files:
# cat /etc/grid-security/vomsdir/dteam/voms.hellasgrid.gr.lsc /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms.hellasgrid.gr /C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2016
# cat /etc/grid-security/vomsdir/dteam/voms2.hellasgrid.gr.lsc /C=GR/O=HellasGrid/OU=hellasgrid.gr/CN=voms2.hellasgrid.gr /C=GR/O=HellasGrid/OU=Certification Authorities/CN=HellasGrid CA 2016
- configuration information:
https://voms.hellasgrid.gr:8443/voms/dteam/configuration/configuration.action https://voms2.hellasgrid.gr:8443/voms/dteam/configuration/configuration.action
Monthly Availability/Reliability
A/R report on ARGO: http://argo.egi.eu/lavoisier/ngi_reports?accept=html
List of the underperforming RCs for (at least) 3 consecutive months:
- AfricaArabia https://ggus.eu/?mode=ticket_info&ticket_id=117094: main problems with the monitoring system, waiting for the release of the central one
- ASRT
- EG-ZC-T3: unresponsive since too months, must be suspended
- ZA-UJ
- AsiaPacific: (since February) https://ggus.eu/index.php?mode=ticket_info&ticket_id=121222
- KR-UOS-SSCC
- MY-UPM-BIRUNI-01
- CERN: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122596 SRM issues
- NGI_DE: https://ggus.eu/?mode=ticket_info&ticket_id=121975
- UNI-SIEGEN-HEP: flapping behaviour
- NGI_IBERGRID: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122599 (SOLVED)
- UB-LCG2 site suspended
'* NGI_MARGI https://ggus.eu/index.php?mode=ticket_info&ticket_id=118465 no monitoring data since January
- NGI_NL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122607 (SOLVED)
- EENet: site suspended
- NGI_PL: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122611 (SOLVED) monitoring data missing for:
- ICM
- IFJ-PAN-BG
- NGI_RO: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122612
- RO-14-ITIM miscellaneous issues, site is recovering
- NGI_UA: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122613 (SOLVED)
- UA_KNU: many power cuts
- Russia: https://ggus.eu/index.php?mode=ticket_info&ticket_id=122615
- Ru-Troitsk-INR-LCG2
Decommissioning SL5
- Tracked on SL5_retirement wiki
- Sites still deploying unsupported service end-points risk suspension, unless documented technical reasons prevent a Site Admin from updating these end-points https://wiki.egi.eu/wiki/PROC16_Decommissioning_of_unsupported_software#Escalation_phase see step 7
- Status https://wiki.egi.eu/wiki/SL5_retirement#2016-06-13_Overall_status reported below.
- from this week on EGI Operations can suspend sites that host SL5 services in production and not set under downtime
- tickets will be opened
Status and actions (Jul 14th)
- 1 Top-BDII bdii.hpgcc.finki.ukim.mk
- 11 Site-BDII apel.indiacms.res.in arc.hpc-mhi.org ce-enmr.chemie.uni-frankfurt.de ce.hpgcc.finki.ukim.mk ce01.grid.etf.rtu.lv glite-bdii.scai.fraunhofer.de is.biruni.upm.my sbdii.grid.uni-sofia.bg uagrid.org.ua uosaf0006.sscc.uos.ac.kr west.icmp.lviv.ua
- 4 MyProxy kek2-px.cc.kek.jp myproxy.cat.cbpf.br wipp-rb.weizmann.ac.il
- 7 WMS and 6 LB glite-wms.scai.fraunhofer.de graspol.nikhef.nl graszode.nikhef.nl graskant.nikhef.nl grasveld.nikhef.nl lb.biruni.upm.my kek2-wms.cc.kek.jp kek2-lb.cc.kek.jp mb-enmr.chemie.uni-frankfurt.de marwmsmr.in2p3.fr (downtime)
- 1 VOMS glite-io.scai.fraunhofer.de
- 1 ARGUS argus.indiacms.res.in
- 7 CREAM-CE ce-enmr.chemie.uni-frankfurt.de ce.hpgcc.finki.ukim.mk ce2.particles.ipm.ac.ir glite-cream.scai.fraunhofer.de haitham.biruni.upm.my kek2-ce01.cc.kek.jp razi.biruni.upm.my
- 0 QCG Computing
- 1 STORM grid-se2.pr.infn.it (downtime)
RCs about to be suspended
Site | Hostname | Service | Downtime | Ticket | Note |
MK-03-FINKI | bdii.hpgcc.finki.ukim.mk, ce.hpgcc.finki.ukim.mk, se.hpgcc.finki.ukim.mk | Top-BDII, Site-BDII/CREAM-CE, MyProxy/SRM | No | https://ggus.eu/?mode=ticket_info&ticket_id=122885 | upgrade by Jul 22nd. SOLVED |
INDIACMS-TIFR | apel.indiacms.res.in, argus.indiacms.res.in | Site-BDII, ARGUS | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122886 | downtime for upgrade for these services for Saturday and Sunday ( 16th and 17th); UPGRADED |
UA-MHI | arc.hpc-mhi.org | Site-BDII/ARC-CE | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122887 | upgrade in a couple of weeks. Jul 28th: UPGRADED |
BMRZ-FRANKFURT | ce-enmr.chemie.uni-frankfurt.de, mb-enmr.chemie.uni-frankfurt.de | Site-BDII/CREAM-CE, WMS/LB | No | https://ggus.eu/?mode=ticket_info&ticket_id=122891 | the site will be suspended and decommissioned. Jul 21st: SUSPENDED. SOLVED |
RTUEF | ce01.grid.etf.rtu.lv | Site-BDII | No | https://ggus.eu/?mode=ticket_info&ticket_id=122893 | Site suspended by NGI |
SCAI | glite-bdii.scai.fraunhofer.de, glite-io.scai.fraunhofer.de, glite-cream.scai.fraunhofer.de, glite-wms.scai.fraunhofer.de | Site-BDII, VOMS, CREAM-CE, WMS/LB | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122894 | some services will be decommissioned, some service will be upgraded |
MY-UPM-BIRUNI-01 | is.biruni.upm.my, haitham.biruni.upm.my and razi.biruni.upm.my, px.biruni.upm.my, lb.biruni.upm.my | Site-BDII, CREAM-CE, MyProxy, LB | No | https://ggus.eu/?mode=ticket_info&ticket_id=122895 | the upgrade will last a couple of weeks. Sep 22th: site suspended for running sl5 software. |
BG05-SUGrid | sbdii.grid.uni-sofia.bg | Site-BDII | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122896 | upgrade scheduled; UPGRADED |
UA_ICYB_ARC | uagrid.org.ua | Site-BDII/ARC-CE | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122897 | upgrade scheduled for next week. Jul 22nd: UPGRADED |
KR-UOS-SSCC | uosaf0006.sscc.uos.ac.kr | Site-BDII/CREAM | No | https://ggus.eu/?mode=ticket_info&ticket_id=122898 | will update the machine |
UA_ICMP_ARC | west.icmp.lviv.ua | Site-BDII/ARC-CE | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122899 | planning to start upgrade next week and the corresponding downtime is scheduled starting from Sundays evening (Jul 17th 20:00) for 2 weeks. Aug 8th: DONE |
IR-IPM-HEP | ce2.particles.ipm.ac.ir | CREAM-CE | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122900 | upgrading... Jul 20th: DONE |
JP-KEK-CRC-02 | kek2-ce01.cc.kek.jp, kek2-px.cc.kek.jp, kek2-wms.cc.kek.jp, kek2-lb.cc.kek.jp | CREAM-CE, MyProxy, WMS, LB | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122901 | will be out of service on Friday 5th Augst 2016 at 4:00 UTC. SOLVED |
CBPF | myproxy.cat.cbpf.br | MyProxy | yes | https://ggus.eu/?mode=ticket_info&ticket_id=122902 | will be decommissioned together nagios, downtime created https://goc.egi.eu/portal/index.php?Page_Type=Downtime&id=21306 . SOLVED |
WEIZMANN-LCG2 | wipp-rb.weizmann.ac.il | WMS/LB/MyProxy | No | https://ggus.eu/?mode=ticket_info&ticket_id=122905 | site-admin on vacation; the WMS service is used only by their nagios server, they try to migrate to sl6 but found some issues in the interaction with ARGUS. SOLVED |
NIKHEF-ELPROD | graspol.nikhef.nl graszode.nikhef.nl, graskant.nikhef.nl grasveld.nikhef.nl | WMS, LB | No | https://ggus.eu/?mode=ticket_info&ticket_id=122903 | they want to keep the services, ticket UNSOLVED |
FedCloud status
AOB
Next meeting
- 08th Aug 2016 https://indico.egi.eu/indico/event/3004/ -> cancelled in favour of 12 September https://indico.egi.eu/indico/event/3005/
- new calendar available until end of 2016 https://indico.egi.eu/indico/category/32/