The eddy covariance data are 10Hz measurements of wind (3 direction) and gases concentration collected at sites over different ecosystems. To calculate the net exchange of gases, energy and temperature between ecosystems and atmosphere as 30 minutes time resolution it is needed to perform a number of calculations and processing steps that will be the basis of the Use Case.
|Background||Contact Person||Organization||Contact email|
|UNITUS - ICOS|
|University of Amsterdam|
Data Service Provision
<please identify the relevant community behavior of this use case based on the model in http://confluence.envri.eu:8090/display/ERM/SV+Community+Behaviours. Please highlight any important roles e.g., as suggested in http://confluence.envri.eu:8090/display/ERM/SV+Community+Roles>[DP1]
The main objective is to optimize the processing of the eddy covariance data in order to establish a service that can be used by different RIs that use this technique. The processing path is defined and codes are ready and tested, however the performances and robustness of the full processing path is something to build and test. The possibility to try this in the ENVRI context is an important possibility that could make a general service available to other RIs. The original measurements will be also stored in services in the context of EUDAT.
The processing per se is not challenging and it is robust (the interpretation is difficult but it is not part of the Use Case). However the quantity of data involved (18000 record for about 50 variables for each half hour), the need of a timely processing (10 hours after data submission) and the multiple and complex data path (measurements submitted from site to a central repository, metadata for data processing submitted from site to ICOS Ecosystem Thematic Centre, all should converge in the processing place and results transmitted back to the ICOS Ecosystem Thematic Centre) are the challenges to address.
The figure below summarize the workflow. Data are collected at the site and submitted to ICOS ETC (metadata) and to a trusted repository (data) where they get a PID. Both data and a processing parameter file (based on metadata) are then available for the processing, that is based on multiple runs with different settings (defined in the parameter file) in order to calculate uncertainty. The final result is a set of halfhourly files (few Mb) that must be available for ICOS ETC and CP. All the processing should happen between midnight and 8 AM, every day, with 40 to 60 sites submitting data. Once per year special processing (manually started) of the whole dataset is needed.
All the data are in ASCII format and also the processing setup file. The processing code is in Fortran and it is called and controlled by a common script that should be re-organized (can be in C, R, Python or Matlab, as you prefer). One run takes on out machines about 15 minutes, four run per sites are needed every morning. The “once per year” run takes few hours per site. Input data are few Mb and will be stored in other servers so not needed to store them.
<Please propose an agenda for implementing the case. This need not be particularly detailed at this point in time, but some aspects worth considering include: 1) timeline, 2) milestones, 3) involved RIs, 4) links to ENVRIPlus work packages / tasks, 5) allocation of resources (staff time and budgets) from RIs and ENVRIPlus. >
We are ready to start a testing phase with few sites. It is difficult to define a timeline because must be clear what is requested to us in terms of code adaptation and setup. Milestones (without date) can be:
M1: setup of the code on the system and test for functionality
M2: setup of the link system to exchange data between processing system and repositories involved
M3: test with two different sites
M4: final version of the system ready for routine processing.
The RI involved are ICOS and possibly LTER, ANAE + any other RI using Eddy Covariance.
This activity is part of WP7, Task 7.1 (development and optimization) and linked to WP9 Task 9.2 (operational system)
Resources are mainly from ENVRI+ but 20% are from the ICOS ETC work.
The expected output is a consolidated and robust processing procedure for eddy covariance data that can become operational. Two different evaluations: