%X We propose a novel deep fusion architecture, CaloriNet, for the online estimation of energy expenditure for free living monitoring in private environments, where RGB data is discarded and replaced by silhouettes. Our fused convolutional neural network architecture is trainable end-to-end, to estimate calorie expenditure, using temporal foreground silhouettes alongside accelerometer data. The network is trained and cross-validated based on a publicly available dataset, SPHERE-Calorie, linking RGB-D, inertial and calorific measurements. Results show state-of-the-art minimum error on the estimation of energy expenditure (calories per minute), outperforming alternative, standard and single-modal techniques. %D 2019 %J British Machine Vision Conference 2018, BMVC 2018 %A A Masullo %A T Burghardt %A D Damen %A S Hannuna %A V Ponce-Lopez %A M Mirmehdi %T Calorinet: From silhouettes to calorie estimation in private environments %O © 2018. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms %L discovery10111762