The CASC Return on Investment Working Group has investigated the value of research computing and data services to their campuses and communities. The working group has conducted surveys of the CASC community and reported its findings in the following conference publications.
Results from a second longitudinal survey of academic research computing and data center usage: expenditures, utilization patterns, and approaches to return on investment
Sharon Broude Geva, Alan Chalker, Curt Hillegas, Donald Petravick, Alan Sill, and Craig Stewart. 2021. In Practice and Experience in Advanced Research Computing (PEARC ’21). Association for Computing Machinery, New York, NY, USA, Article 41, 1–4. https://doi.org/10.1145/3437359.3465589
Availability of cloud-based resource delivery modes is transforming many areas of computing. Academic research computing and data (RCD) support largely remains based on on-premises delivery and has adopted commercial clouds more slowly than the private sector for a variety of stated reasons including factors related to cost efficiency, return on investment, institutional requirements, high costs for bulk commercial cloud computing usage, and funding patterns. Other factors involved in selection of computing resource delivery modes include capabilities and applications that are available only in or best adapted to specific computing environments. It is important for the higher education and research communities to be able to learn from each other as institutions and individuals to make optimum use of appropriate modes of delivery for RCD resources. This paper reports an overview of results from the second annual community-wide survey conducted by the Coalition for Advanced Scientific Computation on patterns of funding, usage, and return on investment for academic research computing and data resources. The results show that on-premises delivery continues to remain the preferred mode for RCD resources for most responding institutions as found in the first survey, but that commercial cloud usage is beginning to be reported for production use by a small number of respondents to the survey. Reasons for these preferences are further explored in the survey and initial high-level results are reported here.
Cloud and on-premises data center usage, expenditures, and approaches to return on investment: A survey of academic research computing organizations
Alan Chalker, Curtis W. Hillegas, Alan Sill, Sharon Broude Geva, and Craig A. Stewart. 2020. In Practice and Experience in Advanced Research Computing (PEARC ’20). Association for Computing Machinery, New York, NY, USA, 26–33. https://doi.org/10.1145/3311790.3396642
Critically important findings from this first survey include the following: many of the respondents are engaged in some form of analysis of return in research computing investments, but only a minority currently report the results of such analyses to their upper-level administration. Most respondents are experimenting with use of commercial cloud resources but no respondent indicated that they have found use of commercial cloud services to create financial benefits compared to their current methods. There is clear correlation between levels of investment in research cyberinfrastructure and the scale of both cpu core-hours delivered and the financial level of supported research grants. Also interesting is that almost every respondent indicated that they participate in some sort of national cooperative or nationally provided research computing infrastructure project and most were involved in academic computing-related organizations, indicating a high degree of engagement by institutions of higher education in building and maintaining national research computing ecosystems. Institutions continue to evaluate cloud-based HPC service models, despite having generally concluded that so far cloud HPC is too expensive to use compared to their current methods.
Cloud and on-premises data center usage, expenditures, and approaches to return on investment: A survey of academic research computing organizations
Alan Chalker, Curt Hillegas, Alan Sill, Sharon Broude Geva, and Craig Stewart. 2020. In Practice and Experience in Advanced Research Computing (PEARC ’20). Association for Computing Machinery, New York, NY, USA, Pages 26–33. https://doi.org/10.1145/3311790.3396642
The landscape of research in science and engineering is heavily reliant on computation and data processing. There is continued and expanded usage by disciplines that have historically used advanced computing resources, new usage by disciplines that have not traditionally used HPC, and new modalities of the usage in Data Science, Machine Learning, and other areas of AI. Along with these new patterns have come new advanced computing resource methods and approaches, including the availability of commercial cloud resources. The Coalition for Academic Scientific Computation (CASC) has long been an advocate representing the needs of academic researchers using computational resources, sharing best practices and offering advice to create a national cyberinfrastructure to meet US science, engineering, and other academic computing needs. CASC has completed the first of what we intend to be an annual survey of academic cloud and data center usage and practices in analyzing return on investment in cyberinfrastructure.
Assessment of non-financial returns on cyberinfrastructure: A survey of current methods
Stewart, C.A., A. Apon, D.Y. Hancock, T. Furlani, A. Sill, J. Wernert, D. Lifka, N. Berente, T. Cheatham, S.D. Slavin. 2019. Assessment of non-financial impacts of investment in cyber-infrastructure: A survey of current methods. In HARC ’19: Proceedings of the PEARC ’19 Workshop: Humans in the Loop: Enabling and Facilitating Research on Cloud Computing. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3355738.3355749.
In recent years, considerable attention has been given to assessing the value of investments in cyberinfrastructure (CI). This paper focuses on assessment of value measured in ways other than financial benefits – what might well be termed impact or outcomes. This paper is a companion to a paper presented at the PEARC’19 conference, which focused on methods for assessing financial returns on investment. In this paper we focus on methods for assessing impacts such as effect on publication production, importance of publications, and assistance with major scientific accomplishments as signified by major awards. We in particular focus on the role of humans in the loop – humanware. This includes a brief description of the roles humans play in facilitating use of research cyberinfratructure – including clouds – and then a discussion of how those impacts have been assessed. Our conclusion overall is that there has been more progress in the past very few years in developing methods for the quantitative assessment of financial returns on investment than there has been in assessing non-quantitative impacts. There are a few clear actions that many research institutions could take to start better assessing the non-financial impacts of investment in cyberinfrastructure. However, there is a great need for assessment efforts to turn more attention to the assessment of non-financial benefits of investment in cyberinfrastructure, particularly the benefits of investing in humans and the benefits to humans who are involved in supporting and using cyberinfrastructure, including clouds.
Assessment of financial returns on investments in cyberinfrastructure facilities: A survey of current methods
Craig A. Stewart, David Y. Hancock, Julie Wernert, Thomas Furlani, David Lifka, Alan Sill,
Nicholas Berente, Donald F. McMullen, Thomas Cheatham, Amy Apon, Ron Payne, Shawn D. Slavin. 2019. In Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (PEARC ’19). Pages 1–8. https://doi.org/10.1145/3332186.3332228
In recent years, considerable attention has been given to assessing the value of investments in cyberinfrastructure (CI). This paper includes a survey of current methods for the assessment of financial returns on investment (ROI) in CI. Applying the financial concept of ROI proves challenging with regard to a service that, in most academic environments, does not generate a “sold amount” such as one would find in the buying and selling of stocks. The paper concludes with a discussion of future research directions and challenges in the assessment of financial ROI in CI. This work is intended less as a definitive guide than as a starting point for further exploration in the assessment of CI’s value for scientific research.