Global Networks for High Energy Physics and Data Intensive Sciences: Past, Present and Future

Harvey Newman
TrackTrack 2 Nordia Room
DescriptionHighly capable and reliable networks have been critical to the science discoveries in high energy physics for the last decades, and this has been increasingly true in other data intensive fields ranging from astrophysics and photon science to genomics in recent years.  While the global, national, and campus networks essential to the success of the LHC and other major science programs have kept pace until now, these networks are increasingly challenged as the data volumes and the ability to move data over long range networks continue to grow exponentially, and as ubiquitous access to data files and object collections generating complex multi-domain flow patterns is increasingly widespread.

I will review the state, evolution and outlook of the world’s research and education networks relative to the needs, along with the rapid progress in server, storage and network interface technologies and data transfer applications driving the ability of users to fully match the capacity of the major networks at relatively low cost.

I will also briefly summarize the state and evolution of the research and education networks as well as global network traffic trends and projections, and server, storage and network interface technology trends and drivers shaping the ability of physics groups throughout the world to use current and next generations of networks effectively.

A major movement of the last few years has been towards software-defined networks and intelligent services able to allocate and load balance bandwidth use across multiple paths. I will review the emerging “system level” concept of an end-to-end network-integrated system that manages workflow using machine learning optimization methods coupled to system modeling, in order to reach equilibrium among the very large and other classes of flows. The need for such systems is driven by the requirement to ensure the continued compatibility of the growing traffic generated by the LHC and other competing upcoming programs with the overall network capacity, and with the vast number of smaller flows supporting the work of the at-large research and education community, whose volume also is increasing rapidly.

Longer term target areas for these emerging systems include (1) the HL LHC program and the other projects whose data volumes and network needs are projected to exceed the current usage by one to two orders of magnitude, and (2) the use of pre-exascale (circa 2020) and exascale (2023) Leadership HPC facilities for a wide range of data intensive applications, where meeting the computing needs of the HL LHC may entail the transport of petabyte data chunks drawn from multi-exabyte data stores. Systems meeting these needs, by completing petabyte-scale transactions in hours, will entail terabit/sec flows making full use of the networks foreseen for the middle of the next decade.  

Presentation documents

All talks