Tuli, S; Bojarczuk, K; Gucevska, N; Harman, M; Wang, XY; Wright, G; (2023) Simulation-Driven Automated End-to-End Test and Oracle Inference. In: Proceedings - 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). (pp. pp. 122-133). IEEE

Preview

Text RPS5.pdf - Accepted Version Download (705kB) | Preview

Abstract

This is the first work to report on inferential testing at scale in industry. Specifically, it reports the experience of automated testing of integrity systems at Meta. We built an internal tool called ALPACAS for automated inference of end-to-end integrity tests. Integrity tests are designed to keep users safe online by checking that interventions take place when harmful behaviour occurs on a platform. ALPACAS infers not only the test input, but also the oracle, by observing production interventions to prevent harmful behaviour. This approach allows Meta to automate the process of generating integrity tests for its platforms, such as Facebook and Instagram, which consist of hundreds of millions of lines of production code. We outline the design and deployment of ALPACAS, and report results for its coverage, number of tests produced at each stage of the test inference process, and their pass rates. Specifically, we demonstrate that using ALPACAS significantly improves coverage from a manual test design for the particular aspect of integrity end-to-end testing it was applied to. Further, from a pool of 3 million data points, ALPACAS automatically yields 39 production-ready end-to-end integrity tests. We also report that the ALPACAS-inferred test suite enjoys exceptionally low flakiness for end-to-end testing with its average in-production pass rate of 99.84%.

Download activity - last month

Export as XMLJSONCSV

Download activity - last 12 months

Export as XMLJSONCSV

Downloads by country - last 12 months

Export as CSVJSONXML

Archive Staff Only

View Item