A Smart Data Transfer Node

Rajkumar Kettimuthu
TrackLightning Talks -- Store Scene
DescriptionAs more and more science workflows get complex and span distributed resources and involve a distributed team of researchers, the demand on the research and education networks keep increasing. In addition to expanding the capacity, Research and education networks continue to add intelligent services to increase the productivity of the science community. Research and education networks support advanced capabilities such as bandwidth reservation and automated diagnostics. Software defined approaches are being adopted to provide intelligent capabilities. The data transfer nodes are compute systems dedicated for data transfers in distributed science environments. We explore the architecture, methods, and algorithms needed for a smart data transfer node (DTN) that self-tune and self-optimize. The condition of network and other resources that data transfers depend on change dynamically. We employ machine learning methods to make the DTN learn how to respond to environmental conditions. For example, if the load on the network is high, and if there is enough compute resource available on the source and destination DTN, a smart DTN will send compressed data instead of raw data to get a higher overall throughput. Similarly, if the probability for corruption on a network link is high, a smart DTN will automatically enable additional integrity verification. In this talk, I will describe our prototype smart DTN that uses deep reinforcement learning to learn the relationship between system (including the network) state, transfer parameters and overall throughput. Our preliminary results show that it can identify transfer parameter values that achieve ~15% higher overall performance than simple heuristics.

Presentation documents

All talks