eprintid: 10204577
rev_number: 8
eprint_status: archive
userid: 699
dir: disk0/10/20/45/77
datestamp: 2025-02-12 11:58:58
lastmod: 2025-02-12 12:00:37
status_changed: 2025-02-12 11:58:58
type: article
metadata_visibility: show
sword_depositor: 699
creators_name: Wen, Jinfeng
creators_name: Chen, Zhenpeng
creators_name: Sarro, Federica
creators_name: Wang, Shangguang
title: Unveiling overlooked performance variance in serverless computing
ispublished: pub
divisions: UCL
divisions: B04
divisions: F48
keywords: Serverless computing; Performance variance; 
Empirical study; 
Cloud computing
note: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
abstract: Serverless computing is an emerging cloud computing paradigm for developing applications at the function level, known as serverless functions. Due to the highly dynamic execution environment, multiple identical runs of the same serverless function can yield different performance, specifically in terms of end-to-end response latency. However, surprisingly, our analysis of serverless computing-related papers published in top-tier conferences highlights that the research community lacks awareness of the performance variance problem, with only 38.38% of these papers employing multiple runs for quantifying it. To further investigate, we analyze the performance of 72 serverless functions collected from these papers. Our findings reveal that the performance of these serverless functions can differ by up to 338.76% (44.28% on average) across different runs. Moreover, 61.11% of these functions produce unreliable performance results, with a low number of repetitions commonly employed in the serverless computing literature. Our study highlights a lack of awareness in the serverless computing community regarding the well-known performance variance problem in software engineering. The empirical results illustrate the substantial magnitude of this variance, emphasizing that ignoring the variance can affect research reproducibility and result reliability.
date: 2025-03
date_type: published
publisher: Springer Verlag
official_url: https://doi.org/10.1007/s10664-025-10615-3
full_text_type: other
language: eng
verified: verified_manual
elements_id: 2358498
doi: 10.1007/s10664-025-10615-3
lyricists_name: Sarro, Federica
lyricists_name: Chen, Zhenpeng
lyricists_id: FSSAR72
lyricists_id: ZCHEM62
actors_name: Flynn, Bernadette
actors_id: BFFLY94
actors_role: owner
funding_acknowledgements: 62032003 [National Natural Science Foundation of China]
full_text_status: restricted
publication: Empirical Software Engineering
volume: 30
number: 2
article_number: 59
pages: 26
issn: 1382-3256
citation:        Wen, Jinfeng;    Chen, Zhenpeng;    Sarro, Federica;    Wang, Shangguang;      (2025)    Unveiling overlooked performance variance in serverless computing.                   Empirical Software Engineering , 30  (2)    , Article 59.  10.1007/s10664-025-10615-3 <https://doi.org/10.1007/s10664-025-10615-3>.      
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10204577/1/2305.04309v2.pdf