Data Fusion Grid Infrastructure

   January, 2007 - December, 2008

Project Structure and Milestones

Project structure

Establishing the assimilation of products retrieved from satellite data into NWP model (T1.1)

Team: IKI RAN

To establish data, software and hardware capabilities to create initial conditions for regional WRF NWP model. To choose the most appropriative set of satellite data with regards to objective criteria. Available satellite data for NWP data assimilation will be investigated. To choose optimal set of data sources required for NWP data assimilation it is required to solve optimization problem. This problem will be solved using genetic algorithm approach.

Result, milestones :

  1. Implementation of automatic data providing system for retrieval of optimal set of satellite data products from IKI Archive by request from remote NSAU regional (East Europe) NWP system,
  2. Elaboration of automatic retrieval software tools for real-time obtaining the predefined satellite data products from operational GIDROMET special information centers by request from remote regional (East Europe) NWP system.
  3. Implementation of special client software for data assimilation purposes based on IKI packages, throughput testing of data assimilation software based on IKI packages, performance measurement and preparation of recommendation for harmonization with T2.1, T.2.2 tasks.

Development of method for automated adaptation of complex models (T1.2)

Team: SRI NASU-NSAU

To develop an intelligent method of adaptation of complex models to particular region and/or particular situations. Most environmental models cannot be used directly in regions or situations different to those they were originally developed or tested. Indeed they generally provide a set of adjustable parameters. By tuning these parameters it is possible to adapt a particular model to region in question. Within this task the method of automated model adaptation will be developed. The method will be based on Genetic Algorithms methodology. From the adaptation point of view model is thought as function of inputs (input/border condition etc.) and tunable parameters. Having set of pairs of input and expected model output one can define global optimization criteria as integral of disparity measure between actual and expected model outputs. The target of adaptation is to find minimum of such objective function with respect to model parameters. To test generalization ability of developed method the full set of inputs and expected outputs will be divided on training and test sets. The result of adaptation using training set will be judged against test set. The method will developed using simple, computation nonintensive models and than applied to adaptation of WRF NWP model.

Result, milestones :

  1. Class of models will be identified for which genetic algorithms approach is appropriate.
  2. For this class of models general method of adaptation will be developed.

Adaptation of Numerical Weather Prediction model (T1.3)

Team: SRI NASU-NSAU

To approbate developed method of model adaptation by applying it for adaptation of WRF NWP model to specific region (in particular for Ukraine). The method developed in task T1.2 will be applied for adaptation of WRF limited area weather model. In this case inputs will be provided by global weather model such as GFS and data assimilation subsystem. Tunable parameters include physics and diffusion options (such as microphysics options or cumulus parameterization). The model forecasts will be compared with result of analysis from data assimilation subsystem and with conversional observations (such as provided by ground stations) and satellite observations (such as QuikScat, Aqua, MSG data). Disparity measures will be based on existing criteria for verification of spatial fields [I. T. Jolliffe, D. B. Stephenson Forecast Verification: A Practitioner's Guide in Atmospheric Science//Wiley, 2003]. Observations that cannot be compared with model output directly (for instance MSG radiances) will be accounted with specialized models such as radiative transfer models.

Result, milestones :

  1. Method of adaptation of limited area NWP models.

Improvement of existed yield prediction methods (T1.4)

Team: IKI RAN

To improve existing yield prediction methods by assimilation NWP data and adaptation of developed methods to particular region. Developed by IKI team approach exploits the sequence of NDVI for yield prediction purposes as well. But it features by omitting any a priori model assumption. Instead it searches for direct link between NDVI evolution characteristics and harvest outcome [http://www.agrocosmos.gvc.ru/satdata/adm_index/2predict_harvest.pl]. Those characteristics reveals from statistic analysis of NDVI maps (retrieved from MODIS data) over region under observation. IKI team took part in development and software implementation of this technique applied for all-Russia yield prediction. It required 5-year set of MODIS data and land use mask for all Russian agriculture regions. NDVI retrieval and statistic analysis were conducted by IKI specialists. In order to increase MIDS model performance meteorological fields will be used.

Result, milestones :

  1. Land use maps for Ukrainian agriculture regions will be prepared.
  2. Complete statistical analysis of NDVI evolution during vegetation season of Ukraine and Ukrainian harvest outcome will be performed and regression dependencies between them will be retrieved.

Geospatial data archive (T2.1)

Team: SRI NASU-NSAU

To develop Grid-service that will provide access to various geospatial data in different formats by standard interface specified by OGC WCS [OpenGISŪ Web Coverage Service (WCS) Implementation Specification.] Data archive will be developed in the framework of OGSA-DAI [The OGSA-DAI Project]. It will include data accessors to deal with different data storage formats (at the moment NetCDF, HDF and GeoTIFF are foreseen) and specific activities (analogue to functions in common programming languages) to implement the superset of OGC WCS interface. Extensions to interface are required to enable access to multidimensional data. Extensions will be kept in coherence with development trends of OGC WCS. The developed software will be deployed and tested on the Satellite Data Storage System (SDSS) developed in IKI RAN.

Result, milestones :

  1. The OGSA-DAI Grid archive of geospatial data will provide WCS interface to underlying data independently of their storage format. Geospatial data should be interoperable with data from other OGSA-DAI sources such as relational databases, XML files and so on.

Geospatial data visualization (T2.2)

Team: IKI RAN

To develop interactive service that will allow to visualize data of different formats (modelling results data are foreseen) and provide standard OGC WMS [OpenGISŪ Web Map Service (WMS) Implementation Specification] and OGC WCS [OpenGISŪ Web Map Service (WMS) Implementation Specification] interfaces and some multidimensional extensions to retrieve visualization result from a program. Data visualization is important component of computer-assisted scientific research. Current WMS compliant software such as UMN MapServer and ESRI ArcIMS provide solid implementation of standard interface defined by OGC but doesn't include any kind of extensions for visualization of multidimensional geospatial data. Necessary extensions for WMS interfaces will be developed in the framework of this project and implemented on the basis of open-source solution UMN MapServer. Interactive web-based service for data retrieval will be implemented. Data for visualization will be taken either from file (traditional way) either from OGSA-DAI archive developed in task T2.1. Integration of MapServer and OGSA-DAI will be carried by data access adapter. Visualization service will be hosted on the server of SRI NASU-NSAU.

Result, milestones :

  1. The Web visualization of geospatial data will provide WMS interface to underlying data independently of their storage format. User will have web-interface that allow performing basic operations on visualization (zooming, panning), measurements (distance, area) and combination of different layers of data. Basic means of temporal data visualization (implemented by animation) also should be provided. Access to data will be implemented by the means of geospatial data archive developed in task T2.1.

Harmonization with WAG (T2.3)

Team: CNES

To include existing Ukrainian resources and Grid testbed into emerging Wide Area Grid project. Large-scale virtual organisation should have a set of agreements governing common access do data and services and ensuring that users get expectable results for their requests. These agreements are of extreme importance for Wide Area Grid project as it aims at developing without top overseeing organization. By its nature WAG project should deal with existing national and field-specific systems. So there are many aspects that need harmonization to establish interoperability. The most important of them are formats for data and metadata exchange and security policy. In the scope of this task these issues will be studied both from organizational and technical points of view. The study will include comparing of various solutions by established set of technical criteria, obtaining information from users of solution and checking applicability of this solutions to existing infrastructure.

Result, milestones :

  1. The following aspects of system interoperability will be harmonized:
    standards for metadata representation;
    data formats that will be used to exchange data between different sites;
    common security policy.

Legal and financial issues (T3.1)

Team: CNES

The Coordinator will be in charge of the overall coordination of legal and contractual aspects related to either contract amendments or partners legal status. The Coordinator will take the necessary measures to ensure the appropriate use of INTAS-CNES-NSAU grant between the participants by providing a time schedule for transferring of funds allocated by the INTAS, CNES, and NSAU within the consortium. Timesheet templates will be forwarded to the participants and regular reporting of manpower status will be required. Identification all major costs incurred during the project will be done and recorded in an appropriate manner (registration procedure) to facilitate the processing of audit certificates. Financial follow up tables (costs, manpower effort, and amounts of money transferred) will be handled by the coordinator.

Result, milestones :

  1. Final report

Organize meetings (T3.2)

Team: SRI NASU-NSAU

In collaboration with the partners, the Coordinator will set agenda and organize the official meetings to be held during the project. Minutes of the official meetings will be drafted for further approval. At least two meetings per year will be organized at partner's institutions.

Result, milestones :

  1. Final report

Prepare periodic progress reports and interface for reporting, review meetings and auditing (T3.3)

Team: II-SAS

Templates will be diffused for official communication inside the consortium. The Coordinator will be in charge to notify due dates of the reporting to the partners and will ensure the diffusion and collection of partners' contributions before submission to INTAS-CNES-NSAU representatives. A yearly (technical and financial) progress reporting will be applied to the proposed project.

Result, milestones :

  1. First year progress report
  2. Second year progress report
  3. Final report

Assess the project deliverables (T3.4)

Team: II-SAS

The quality and performance of the project deliverables will be controlled. Internal management rules and methodologies will be defined (e.g. identification and quality procedures).

Result, milestones :

  1. Final report

Project's website development and maintenance (T4.1)

Team: SRI NASU-NSAU

To develop Web site that will present the project objectives, results, status, consortium publications and useful links to the topics related to the project. To enable access of publicity to the current state and results of the project the web site will be developed. For Web site development standard open-source CMS frameworks will be used, such as WordPress.

Result, milestones :

  1. Web site consisting several sections, covering information on the following topics:
    - Project objectives and tasks
    - Latest news concerning project's state
    - Current results and reached milestones
    - Description of participating teams
    - Collaboration with third parties
    - Project's publications
    - Announcements of participation in conferences and workshops
    - Announcements of conferences and workshops organized by team members
    Web site will provide download area where project's result in form of software packages and documents could be downloaded.

Raising the community interest (T4.2)

Team: SRI NASU-NSAU

To rise public awareness of project's achievements. The project intends to present the objectives, methodology, developed tools, instruments and results in several scientific and/or business conferences and/or journals. This will also help partners to support the exploitation phase by increasing international awareness in their capabilities and know-how. Several participations to conferences are envisioned as part of this project. These activities will also be disseminated and promoted by means of the project website (T4.1) which will host a wealth of information and dissemination resources, including for instance brochures and presentations of the concept, and so forth. The task will be finalized by a report on raising public participation and awareness. Feedback is expected to be provided to standardization bodies and consortiums on the integration of all these technologies, their applicability, their completeness, their optimization and their further developments. Awareness of this consortium and its promotion would be achieved also through national and international associations and their events.

Result, milestones :

  1. Feedbacks obtained from scientific communities. The number of publications and presentations made on conferences and workshops, the use of tools and instruments developed within the proposed project, contribution to standards.

Foster collaboration with third parties (T4.3)

Team: II-SAS

The list of all entities that might have interest in the project outcomes will be registered and permanently updated. Contacts with these entities will be established.

Result, milestones :

  1. The results of the project will give synergetic effect with other projects such as Degree (example of successful application of Grid technologies for Earth observation problems) and DRAGOON (cooperative activities between ESA and China) projects.
  2. The results of the project will provide the basis for WAG project initiated by CEOS WGISS.
  3. The further WAG developed on the basis of created components within the proposed project could become a demonstration project promoting the wider use, in other disciplines, of ensemble-based techniques originally developed for weather forecasting (GEO task DA-06-03 in 2006 GEO Work Plan).

Software distribution (T4.4)

Team: SRI NASU-NSAU

Distribution of developed software should be considered as a separate task. It includes packaging of software by special tools such as APT and RPM, developing build scripts for source-based distribution systems such as Ports and Portage and writing user documentation. The resulted packages should be tested on targeted architectures and operating systems. Beside releases, versions of access should also be provided to development versions of software: tagged alpha and beta versions, daily snapshots and current-state source tree. Tagged versions and snapshots will be provided as a simple source code bundles. Access to current-state source tree will be done by anonymous access to development version control system (SVN is foreseen). The infrastructure of software distribution in UNIX environment is very mature and well developed. The basis of infrastructure is laid by so called Autotools [http://sourceware.org/autobook/] that consists of three programs: Autoconf [http://www.gnu.org/software/autoconf/], Automake [http://www.gnu.org/software/automake/] and Libtool [http://www.gnu.org/software/libtool/]. These tools provide abstraction level above OSes specific distinctions.

Result, milestones :

  1. Software packages that are tested on most widespread Linux distribution will be provided for users download. This packages will include both source and binary forms of distribution. Documentation on software installation and usage will also be provided.
  2. Software developed within the proposed project will also be distributed among WAG participants and other Grid-related project, such as Earth Observation Grid (ESA/ESRIN) and EGEE/DEGREE.
 
Space Research Institute::Department of Space Information Technologies and Systems