Global Networks for High Energy Physics and Data Intensive Sciences: Past, Present and Future
SpeakerHarvey Newman
TrackTrack 2 Nordia Room
DescriptionHighly capable and reliable networks have been critical to the science discoveries in high energy physics for the last decades, and this has been increasingly true in other data intensive fields ranging from astrophysics and photon science to genomics in recent years. While the global, national, and campus networks essential to the success of the LHC and other major science programs have kept pace until now, these networks are increasingly challenged as the data volumes and the ability to move data over long range networks continue to grow exponentially, and as ubiquitous access to data files and object collections generating complex multi-domain flow patterns is increasingly widespread.
I will review the state, evolution and outlook of the world’s research and education networks relative to the needs, along with the rapid progress in server, storage and network interface technologies and data transfer applications driving the ability of users to fully match the capacity of the major networks at relatively low cost.
I will also briefly summarize the state and evolution of the research and education networks as well as global network traffic trends and projections, and server, storage and network interface technology trends and drivers shaping the ability of physics groups throughout the world to use current and next generations of networks effectively.
A major movement of the last few years has been towards software-defined networks and intelligent services able to allocate and load balance bandwidth use across multiple paths. I will review the emerging “system level” concept of an end-to-end network-integrated system that manages workflow using machine learning optimization methods coupled to system modeling, in order to reach equilibrium among the very large and other classes of flows. The need for such systems is driven by the requirement to ensure the continued compatibility of the growing traffic generated by the LHC and other competing upcoming programs with the overall network capacity, and with the vast number of smaller flows supporting the work of the at-large research and education community, whose volume also is increasing rapidly.
Longer term target areas for these emerging systems include (1) the HL LHC program and the other projects whose data volumes and network needs are projected to exceed the current usage by one to two orders of magnitude, and (2) the use of pre-exascale (circa 2020) and exascale (2023) Leadership HPC facilities for a wide range of data intensive applications, where meeting the computing needs of the HL LHC may entail the transport of petabyte data chunks drawn from multi-exabyte data stores. Systems meeting these needs, by completing petabyte-scale transactions in hours, will entail terabit/sec flows making full use of the networks foreseen for the middle of the next decade.
I will review the state, evolution and outlook of the world’s research and education networks relative to the needs, along with the rapid progress in server, storage and network interface technologies and data transfer applications driving the ability of users to fully match the capacity of the major networks at relatively low cost.
I will also briefly summarize the state and evolution of the research and education networks as well as global network traffic trends and projections, and server, storage and network interface technology trends and drivers shaping the ability of physics groups throughout the world to use current and next generations of networks effectively.
A major movement of the last few years has been towards software-defined networks and intelligent services able to allocate and load balance bandwidth use across multiple paths. I will review the emerging “system level” concept of an end-to-end network-integrated system that manages workflow using machine learning optimization methods coupled to system modeling, in order to reach equilibrium among the very large and other classes of flows. The need for such systems is driven by the requirement to ensure the continued compatibility of the growing traffic generated by the LHC and other competing upcoming programs with the overall network capacity, and with the vast number of smaller flows supporting the work of the at-large research and education community, whose volume also is increasing rapidly.
Longer term target areas for these emerging systems include (1) the HL LHC program and the other projects whose data volumes and network needs are projected to exceed the current usage by one to two orders of magnitude, and (2) the use of pre-exascale (circa 2020) and exascale (2023) Leadership HPC facilities for a wide range of data intensive applications, where meeting the computing needs of the HL LHC may entail the transport of petabyte data chunks drawn from multi-exabyte data stores. Systems meeting these needs, by completing petabyte-scale transactions in hours, will entail terabit/sec flows making full use of the networks foreseen for the middle of the next decade.
Presentation documents
All talks
- A Hybrid Cloud for Norwegian HE
- A Virtual Global Infrastructure - Maximising infrastructure investment.
- Advances in the Global Network Architecture
- Benefits of student driven ISPs
- Building an Identity Federation using OpenID Connect
- Building an NREN without a Network
- Building an open HSM - The cryptech.is Project
- Building low-cost measurement infrastructure with perfSONAR
- CBF-ring in Northern Sweden, Finland and Norway
- Can we bridge the gap between average and peak loads in science networks?
- Certificate Transparency
- CodeRefinery: teaching scientists in modern software development
- Cost and benefit of NREN Cyber Risk Compliance
- DDoS Attacks on Research and Education: Stopping, Tracing and Prosecuting the Culprits
- Dynamic network provisioning over the NSI-enabled infrastructures: feedback and lessons learned from the FELIX project
- ESnet's Software Journey - From Networks to Visualization
- EUDAT – the pan-European Collaborative Data Infrastructure.
- EduKEEP: Towards a User-Centric Identity Management Model
- Emerging Software Defined Network Exchanges (SDXs)
- Forget SDN controllers, think Configuration Management
- Future Architecture of Federations
- GEANT Community Delivering Copernicus Mission Data
- Gaming e-Infrastructures to improve Interfederation Readiness
- Glenna: The Nordic Cloud Project
- Global Networks for High Energy Physics and Data Intensive Sciences: Past, Present and Future
- GÉANT Network Evolution: moving to a Software Defined Network
- Improving eduroam and WLAN in general
- InfiniCortex and InfiniCloud : a distributed way to exascale
- Information Security Management at DeiC
- Information Security Management at UNINETT
- Invitation to NDN2018
- Key management from the trenches and why you want an open HSM
- Make Security Great (Again)!
- Networking the Square Kilometre Array
- Nordic Media Group Report: Leveraging Media Services in the 21st Century
- Old data. Digital research infrastructure and archaeological data
- On Carrots and Sticks: Moving from Exams to Drills with Cryptocurrency Rewards
- Our new lecture capture setup
- Performance Monitoring of DWDM-links
- Pervasive Monitoring, 3 Years In
- Releasing Attributes for Science! - Why does this seem so difficult?
- SNIC Science Cloud (SSC): A National Cloud Infrastructure for Swedish Research
- Science DMZ's in the Philippines
- Scientific applications on an OpenStack cloud
- Securing the Internet
- Share, steward and reuse research data - European Open Science Cloud
- Software Defined Wavelengths: maximize photonic infrastructure usage and secure services
- Somewhere in the cloud is not enough
- SunetC – a story about fiber
- Supporting Student Mobility is impossible if we don't recognize our students
- The Changing NREN Landscape
- The Global Research Platform (GRP)
- The GÉANT SIG-ISM Group
- The LMS is dead, long live the digital learning platform
- The Networking Grand Challenge: Can We Rise to It?
- The role of YANG in network programming (for fun and profit)
- Three FUAS-universities of applied Sciences using Kaltura-service together in Finland
- Towards Smarter Security Analytics for the Internet of People
- UNINETT Certified Infrastructure Room
- Welcome
- What is an UnConference? How does it work?
- ePouta - A Cloud Platform for Sensitive Data