Speaking Slot: 17 June 2019 – 16:40 till 17:00
(Hall 3 – booth N-210)

Subject: Design and Operation of a 3 MW Datacenter based on Prefabricated Modules – CERN’s Reply to the Computing and Storage Challenges related to the Large Hadron Collider’s (LHC) Upgrade Program

CERN, in cooperation with LHCb, is currently building a new 3MW data center. The purpose of this data center is the consolidation and filtering, in real time, of a 5 TB/s data stream coming from a High Energy Physics experiment.

The datacenter is divided into 2 logical units.

• Unit 1, consisting of two 18m SAFE (*) Prefabricated modules, receives the data stream on approximately 17000 optical links and assembles and distributes data sets to the filter processing unit.
• Unit 2 consists of four 18m SAFE Prefabricated modules housing the bulk of the computation power which is used to reduce the immense data volume to more manageable quantities.

The datacenter modules themselves are cooled via indirect fresh air cooling with an additional adiabatic cooling stage during warm periods. Rack power density is 22 kw per rack.

At the time of this presentation CERN will have been operating the first stage of this datacenter for nearly 6 months and will present first results and lessons learned from its construction and operation.

About CERN

CERN is the renowned European Organization for Nuclear Research. It employs just over 2500 people. The laboratory’s scientific and technical staff design and build the particle accelerators and ensure their smooth operation. They also help to prepare, run, analyse and interpret data from complex scientific experiments.

Two experiments need an upgrade: ALICE and LHCb

• As of 2021 the ALICE O2 facility will be a high-throughput computing system which will include heterogeneous computing platforms and disk storage systems.
• The upgraded LHCb experiment, scheduled to start taking data in 2020, will collect data at higher luminosity compared to the present running conditions and with a significantly increased selection efficiency.

CERN has decided not to build new/additional (traditional) datacenters but to rely on Prefabricated Datacenter Modules to execute the upgrades.

Why? The advantages of Automation’s Prefabricated Datacenter Modules are completely in line with their requirements:

• Fast, flexible and efficient solution to achieve the highest quality standards
• Supports the high performance requirements of CERN’s HPC
• Factory build lead-time of 12 weeks
• Performance configurations optimized for power and density, energy efficient
• ‘Near source’: close to the experiment

Moreover, when consulting the ‘CERN openlab White Paper of September 2017 on Future ICT Challenges in Scientific Research’ – Page 5: R&D Topic 1: Data-Centre Technologies and Infrastructures, Automation is completely aligned. The same goes with the Keynote Speech at ISC of Dr. Maria Girone – CERN Openlab Chief Technology Officer on 25/6/2018 explaining ‘Tackling tomorrow’s computing challenges today at CERN’.

CERN openlab White Paper of September 2017 on Future ICT Challenges in Scientific Research’ – Page 5:

R&D Topic 1: Data-Centre Technologies and Infrastructures

Designing and operating distributed data infrastructures and computing centres poses challenges in areas such as networking, architecture, storage, databases, and cloud. These challenges are amplified and added to when operating at the extremely large scales required by major scientific endeavours.

CERN is evaluating different models for increasing computing and data-storage capacity, in order to accommodate the growing needs of the LHC experiments over the next decade. All models present different technological challenges. In addition to increasing the capacity of the systems used for traditional types of data processing and storage, explorations are being carried out into a number of alternative architectures and specialized capabilities. These will add heterogeneity and flexibility to the data centres, and should enable advances in resource optimization.

Rainer Schwemmer speaks on 17 June 2019 from 16:40 till 17:00
Hall 3 – booth N-210

Want to discuss more in detail: come and visit us in booth F-912

* What is SAFE?

SAFE was developed with over 20 years of experience in designing containerized shelters which evolved into the SAFE concept, aligned with today’s ICT equipment requirements.

SAFE is a carefully chosen acronym reflecting our design philosophy and stands for Secure, Autonomous, Flexible and Efficient.

SECURE covers

• Physical security, with 4mm steel wall fully welded and additional layers of protective material and resistance class 4 and higher entrance doors.
• Reliability in term of operations and longevity. We still today have 20-year-old SAFE units in the field which are ticking away!
• Financial security: delivery at a fixed cost is part of our engineering and design concept.
• Secure investment in the short and longer term with the option to even move the SAFE in case of relocation.

AUTONOMOUS

• The SAFE’s are designed for autonomous operation. By combining all necessary techniques into a single frame the unit can be fully standalone and installed within a few hours.
• This adds additional discipline in designing our solutions in the most optimal way.

FLEXIBLE

• SAFE’s are engineered with flexibility in mind:
• Financial flexibility, in phased deployments spreading CAPEX and OPEX in line with business growth.
• Automation’s customized design approach offers customers additional flexibility.
• There is flexibility in deploying or relocating the SAFE prefabricated datacenters.

EFFICIENT

• Efficient in terms of financial investment, we see in practice at least 30% lower capex versus brick and mortar or on-site construction and assembly.
• Efficiency in terms of technical design adapted to the customer’s needs.
• Efficient in terms of energy consumption and optimization.