Large memory consumption when runn… | Apple Developer Forums
Large memory consumption when running Core ML model on A13 GPU Machine Learning & AI Core ML iOS Core ML You’re now watching this thread. If you’ve opted in to email or web notifications, you’ll be notified when there’s activity. Click again to stop watching or visit your profile to manage watched threads and notifications. You’ve stopped watching this thread and will no longer receive emails or web notifications when there’s activity.
Click again to start watching. FrankSchlegel OP Created Dec ’22 Replies 3 Boosts 2 Views 3.0k Participants 3 We recently had to change our MLModel 's architecture to include custom layers, which means the model can't run on the Neural Engine anymore. After the change, we observed a lot of crashes being reported on A13 devices. It turns out that the memory consumption when running the prediction with the new model on the GPU is much higher than before, when it was running on the Neural Engine.
Before, the peak memory load was ~350 MB, now it spikes over 2 GB, leading to a crash most of the time. This only seems to happen on the A13. When forcing the model to only run on the CPU, the memory consumption is still high, but the same as running the old model on the CPU (~750 MB peak). All tested on iOS 16.1.2. We profiled the process in Instruments and found that there are a lot of memory buffers allocated by Core ML that are not freed after the prediction.
The allocation stack trace for those buffers is the following: We ran the same model on a different device and found the same buffers in Instruments, but there they are only 4 KB in size. It seems, Core ML is somehow massively over-allocating memory when run on the A13 GPU. So far we limit the model to only run on CPU for those devices, but this is far from ideal. Is there any other model setting or workaround that we can use to avoid this issue?
Replies 3 Boosts 2 Views 3.0k Participants 3 FrankSchlegel OP Dec ’22 Also reported as FB11871301. Thanks! Add comment gudgud96 OP Sep ’24 Hi @FrankSchlegel , I am facing the same issue whereby memory allocation with .cpuAndGpu is much higher than .cpuOnly . Also have the same observation that memory is not properly deallocated when running on GPU, even with some autoreleasepool tricks. Do you have any updates on this?
Add comment e_elephant OP Jul ’26 I am running a CoreML model (vision transformer, fixed input size) with hardware units cpuAndNeuralEngine on both iPhone 15 and iPhone SE 2. The memory usage is normal on iPhone 15. On iPhone SE 2 [A13], it is more than double. The model loading on SE2 takes forever, and puts a few of these warnings in the console: [Espresso::handle_ex_plan] exception=ANECF error: failed to load ANE model [...] FAILED: err=( CompilationFailure ) doUnloadModel:options:qos:error:: nil _ANEModel ANECF error: Unable to build plan. @ PrepareOpForEncode E5RT: Unable to build plan. (2) Before seemingly falling back on CPU because inference completes successfully.
But it looks as though the failed attempt at Espresso compilation leaves behind a ~800MB memory leak. What's interesting is, if I set CoreML to use cpuOnly and not cpuAndNeuralEngine, the memory usage is lower by 800MB to around ~100MB, less than what I would expect. Add comment