kaniko-build: explain when a build hits the runner's memory limit
All checks were successful
Self-test kaniko-build / build-without-push (push) Successful in 6s
All checks were successful
Self-test kaniko-build / build-without-push (push) Successful in 6s
Every job on the scotty-git runner is now limited to 1.5 GB of memory (--memory=1536m, set on the runner) so no workflow can take down the git server it shares a VM with. When a build step is stopped by that limit, Kaniko only reports "exit status 137"; the action now prints a clear error naming the limit and what to do. README gains a "Runner limits" section that explains exit code 137 for every workflow, not just image builds. Tested on a temporary branch: a 2 GB step is stopped at ~1.5 GB, the Kaniko message appears, and a normal build still succeeds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
21e961d4f0
commit
5ca003ad5c
2 changed files with 23 additions and 1 deletions
12
README.md
12
README.md
|
|
@ -100,6 +100,18 @@ repository, never at organisation or instance level:
|
|||
tool update can never change your builds without a reviewed change here.
|
||||
- **Versions:** use `@v1` to get compatible fixes. Breaking changes get a new major tag.
|
||||
|
||||
### Runner limits (all workflows on scotty-git)
|
||||
|
||||
Every job on the shared scotty-git runner is limited to **1.5 GB of memory**, so that no workflow can
|
||||
take down the git server it runs next to. Each job can read the limit from the environment variable
|
||||
`SCOTTY_GIT_JOB_MEMORY_LIMIT`.
|
||||
|
||||
**If a step fails with `exit code 137`**, it was stopped because it used more than 1.5 GB. Common
|
||||
fixes: run tests with fewer parallel workers, avoid loading whole datasets into memory, or split the
|
||||
job. `kaniko-build` prints this explanation itself. For other steps the runner can only report the
|
||||
exit code, so this is the place to look it up. If your workload genuinely needs more, ask the
|
||||
scotty-git admins (every memory stop is logged on the runner: `journalctl -t ci-memory-limit`).
|
||||
|
||||
### Changing this action
|
||||
|
||||
Open a pull request. The `Self-test kaniko-build` workflow builds a small test image with your
|
||||
|
|
|
|||
|
|
@ -71,7 +71,17 @@ IFS=$OLDIFS
|
|||
if [ -n "$EXTRA_ARGS" ]; then set -- "$@" $EXTRA_ARGS; fi
|
||||
|
||||
echo "Building $DOCKERFILE (context: $CONTEXT)$( [ "$PUSH" = "true" ] && echo " -> $DESTS" || echo ", not pushing")"
|
||||
/kaniko/executor "$@"
|
||||
# Keep a copy of the output: when a RUN step (not Kaniko itself) is killed for memory,
|
||||
# Kaniko exits 1 and only its message says "exit status 137".
|
||||
{ /kaniko/executor "$@" 2>&1; echo "$?" > /kaniko/exit-code; } | tee /kaniko/build.log
|
||||
RC=$(cat /kaniko/exit-code 2>/dev/null || echo 1)
|
||||
if [ "$RC" -ne 0 ] && grep -q 'exit status 137' /kaniko/build.log; then RC=137; fi
|
||||
if [ "$RC" -eq 137 ]; then
|
||||
LIMIT="${SCOTTY_GIT_JOB_MEMORY_LIMIT:-1.5 GB}"
|
||||
fail "Build stopped: this job hit scotty-git's per-job memory limit ($LIMIT) and was killed (exit code 137). The limit protects the shared git server. Reduce the build's memory use (e.g. fewer parallel steps in the Dockerfile, smaller dependencies), or ask the scotty-git admins for a larger runner."
|
||||
elif [ "$RC" -ne 0 ]; then
|
||||
fail "Kaniko build failed (exit code $RC). See the Kaniko output above for the failing step."
|
||||
fi
|
||||
|
||||
# ---- Output ----
|
||||
if [ "$PUSH" = "true" ] && [ -f /kaniko/digest ]; then
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue