Integrating big data tools for X-ray science

Synchrotron x-ray facilities, such as the Advanced Photon Source and
Advanced Light Source, use a variety of scattering, imaging, and
spectroscopic techniques to address a range of problems in materials
science. The increasing brightness of such x-ray sources, coupled to
recent developments in detector technologies, means that they must
handle significantly greater data rates than in the past, both on
individual beam lines and across facilities as a whole.  Here we
describe our approaches for for organizing both raw experimental data
and subsequent derived data, managing and interacting with such large
data over networks, and integrating large-scale high-performance
computers.  Our toolkit includes catalog, transfer, workflow, and GUI
components.  We will discuss the development, integration, and
application of these techniques to accelerate real-time analysis of
x-ray scattering data.