For tunneled inference upload bandwidth dominates not GPU speed
Comparing transcription pipeline throughput between Metal-only vs CoreML+ANE on Apple Silicon when the workload is offloaded over an SSH tunnel from a remote VM
Upgrading the local whisper inference path from Metal-only to CoreML+Apple Neural Engine (about 5x faster compute on small models) only improved end-to-end pipeline throughput by ~18 percent (17 min to 14 min for a 37 hour audiobook). The reason: when the workload is offloaded over a tunnel, the bottleneck shifts from compute to data transfer. Storyteller WhisperServerSTT logs showed 95% of per-chunk wall time was upload, 5% was conversion+inference. A 5x speedup on the 5% slice maps to ~4 percent end-to-end gain, plus some scheduling overlap with the upload pipeline gave us 18%. This generalizes: any time you offload ML inference to a remote machine over a slow link (home internet upload, VPN, SSH tunnel), profile transport vs compute before optimizing the inference path. The biggest improvements come from reducing what you ship (compress audio aggressively, drop sample rate, send only voiced segments via VAD) NOT from upgrading the inference hardware. On a fast LAN or local socket the GPU upgrade would have been transformative; over a 25 Mbps home upload it is marginal.
When advising on offloaded ML pipeline speedups over tunnels/slow links, ALWAYS ask for the wall-clock breakdown (upload vs decode vs inference) before recommending GPU/hardware changes. If upload is >50 percent of wall time, focus on payload size reduction (audio codec, sample rate, VAD pre-filter, batch sending) before any compute-side optimization.