Oral Abstract

Oral Contribution (O6.4) Rees Williams (University of Groningen)

Theme: Local and global cloud infrastructure for processing and storage

The Euclid Archive Processing and Data Distribution Systems: a distributed infrastructure for Euclid and associated data

The Euclid Archive System is an ambitious information system, which sits at the heart of the Euclid Science Ground Segment. It is a joint development between the Euclid Consortium and the ESAC Science Data Centre. It encompases both Euclid data and the large volume of associated ground based data (e.g. KiDS, DES and LSST).

The Euclid Science Ground Segment consists of the Euclid Science Operations Centre and nine national Science Data Centres. The large data volumes, demand that data transfer is minimized and that the processing is taken to the data. This is supported by the Euclid Archive Data Processing System and the Euclid Archive Distributed Data System. The Data Processing System consists of a central metadata repository, which contains the information necessary to process any data item and full data lineage of any data product created. The Distributed Data System provides a cloud solution with a node at each of the national Science Data Centres, which controls data storage and transfer. It supports a large number of storage types, including POSIX file systems, iRODS, gridftp and XROOTD. No limitations is are placed on the storage implemented at an individual SDC. Further more, the user of the system needs no knowledge of where data is located. Jobs will be star
ted at the most appropriate locations, or transferred if necessary.

The Euclid Archive will also provides convenient tools for the astronomical community to access Euclid science ready data. This is implemented in the Euclid Archive Science Analysis System, which is tightly interfaces with the Data Processing System, allowing data to be accessed by either system, limited only by the defined access policy. Details of the Science Archive System interfaces, which include VO compatible interfaces and a HIPS viewer, are given in a separate paper.