Note

Dependency layers before source layers

ML images are large. Layer ordering decides whether a build is thirty seconds or eight minutes.

  • MLOps
  • Docker

Docker caches per layer, and invalidates every layer after the first change. Copying source before installing dependencies means every source edit reinstalls the dependency tree.

Copy the dependency manifest, install, then copy source. Dependencies change weekly, source changes hourly, so almost every build reuses the expensive layer and rebuilds only the cheap one.

This is well-known and still routinely wrong in ML images, where the dependency install is the slowest step in the whole build and therefore the one it hurts most to repeat.

All notes →Longer writing →