kaniko-build: explain when a build hits the runner's memory limit
All checks were successful
Self-test kaniko-build / build-without-push (push) Successful in 6s

Every job on the scotty-git runner is now limited to 1.5 GB of memory
(--memory=1536m, set on the runner) so no workflow can take down the git
server it shares a VM with. When a build step is stopped by that limit,
Kaniko only reports "exit status 137"; the action now prints a clear error
naming the limit and what to do. README gains a "Runner limits" section that
explains exit code 137 for every workflow, not just image builds.

Tested on a temporary branch: a 2 GB step is stopped at ~1.5 GB, the Kaniko
message appears, and a normal build still succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Vishnu Prasd P S 2026-10-02 14:29:27 +05:30
parent 21e961d4f0
commit 5ca003ad5c
2 changed files with 23 additions and 1 deletions

View file

@ -100,6 +100,18 @@ repository, never at organisation or instance level:
tool update can never change your builds without a reviewed change here.
- **Versions:** use `@v1` to get compatible fixes. Breaking changes get a new major tag.
### Runner limits (all workflows on scotty-git)
Every job on the shared scotty-git runner is limited to **1.5 GB of memory**, so that no workflow can
take down the git server it runs next to. Each job can read the limit from the environment variable
`SCOTTY_GIT_JOB_MEMORY_LIMIT`.
**If a step fails with `exit code 137`**, it was stopped because it used more than 1.5 GB. Common
fixes: run tests with fewer parallel workers, avoid loading whole datasets into memory, or split the
job. `kaniko-build` prints this explanation itself. For other steps the runner can only report the
exit code, so this is the place to look it up. If your workload genuinely needs more, ask the
scotty-git admins (every memory stop is logged on the runner: `journalctl -t ci-memory-limit`).
### Changing this action
Open a pull request. The `Self-test kaniko-build` workflow builds a small test image with your

View file

@ -71,7 +71,17 @@ IFS=$OLDIFS
if [ -n "$EXTRA_ARGS" ]; then set -- "$@" $EXTRA_ARGS; fi
echo "Building $DOCKERFILE (context: $CONTEXT)$( [ "$PUSH" = "true" ] && echo " -> $DESTS" || echo ", not pushing")"
/kaniko/executor "$@"
# Keep a copy of the output: when a RUN step (not Kaniko itself) is killed for memory,
# Kaniko exits 1 and only its message says "exit status 137".
{ /kaniko/executor "$@" 2>&1; echo "$?" > /kaniko/exit-code; } | tee /kaniko/build.log
RC=$(cat /kaniko/exit-code 2>/dev/null || echo 1)
if [ "$RC" -ne 0 ] && grep -q 'exit status 137' /kaniko/build.log; then RC=137; fi
if [ "$RC" -eq 137 ]; then
LIMIT="${SCOTTY_GIT_JOB_MEMORY_LIMIT:-1.5 GB}"
fail "Build stopped: this job hit scotty-git's per-job memory limit ($LIMIT) and was killed (exit code 137). The limit protects the shared git server. Reduce the build's memory use (e.g. fewer parallel steps in the Dockerfile, smaller dependencies), or ask the scotty-git admins for a larger runner."
elif [ "$RC" -ne 0 ]; then
fail "Kaniko build failed (exit code $RC). See the Kaniko output above for the failing step."
fi
# ---- Output ----
if [ "$PUSH" = "true" ] && [ -f /kaniko/digest ]; then