TEMP: memory limit test (branch will be deleted)
Some checks failed
Memory limit test (temporary branch) / limit-visible (push) Successful in 3s
Memory limit test (temporary branch) / plain-step-uses-2gb (push) Failing after 8s
Memory limit test (temporary branch) / kaniko-run-step-uses-2gb (push) Failing after 27s
Memory limit test (temporary branch) / kaniko-normal-build (push) Successful in 6s

This commit is contained in:
Vishnu Prasd P S 2026-10-02 14:27:52 +05:30
parent 21e961d4f0
commit 8341c98e38
4 changed files with 59 additions and 1 deletions

View file

@ -0,0 +1,33 @@
name: Memory limit test (temporary branch)
on:
push:
branches: [test/memory-limit]
jobs:
limit-visible:
runs-on: ubuntu-latest
steps:
- run: |
echo "SCOTTY_GIT_JOB_MEMORY_LIMIT=$SCOTTY_GIT_JOB_MEMORY_LIMIT"
echo "kernel memory limit for this job: $(cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes)"
plain-step-uses-2gb:
runs-on: ubuntu-latest
steps:
- run: node -e 'const a=[]; for (let i=0;i<40;i++){ a.push(Buffer.alloc(50*1024*1024, 1)); console.log((i+1)*50, "MB held"); }'
kaniko-run-step-uses-2gb:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./kaniko-build
with:
context: kaniko-build/test-memhog
dockerfile: kaniko-build/test-memhog/Dockerfile
push: "false"
kaniko-normal-build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ./kaniko-build
with:
context: kaniko-build/test
dockerfile: kaniko-build/test/Dockerfile
push: "false"

View file

@ -100,6 +100,18 @@ repository, never at organisation or instance level:
tool update can never change your builds without a reviewed change here. tool update can never change your builds without a reviewed change here.
- **Versions:** use `@v1` to get compatible fixes. Breaking changes get a new major tag. - **Versions:** use `@v1` to get compatible fixes. Breaking changes get a new major tag.
### Runner limits (all workflows on scotty-git)
Every job on the shared scotty-git runner is limited to **1.5 GB of memory**, so that no workflow can
take down the git server it runs next to. Each job can read the limit from the environment variable
`SCOTTY_GIT_JOB_MEMORY_LIMIT`.
**If a step fails with `exit code 137`**, it was stopped because it used more than 1.5 GB. Common
fixes: run tests with fewer parallel workers, avoid loading whole datasets into memory, or split the
job. `kaniko-build` prints this explanation itself. For other steps the runner can only report the
exit code, so this is the place to look it up. If your workload genuinely needs more, ask the
scotty-git admins (every memory stop is logged on the runner: `journalctl -t ci-memory-limit`).
### Changing this action ### Changing this action
Open a pull request. The `Self-test kaniko-build` workflow builds a small test image with your Open a pull request. The `Self-test kaniko-build` workflow builds a small test image with your

View file

@ -71,7 +71,17 @@ IFS=$OLDIFS
if [ -n "$EXTRA_ARGS" ]; then set -- "$@" $EXTRA_ARGS; fi if [ -n "$EXTRA_ARGS" ]; then set -- "$@" $EXTRA_ARGS; fi
echo "Building $DOCKERFILE (context: $CONTEXT)$( [ "$PUSH" = "true" ] && echo " -> $DESTS" || echo ", not pushing")" echo "Building $DOCKERFILE (context: $CONTEXT)$( [ "$PUSH" = "true" ] && echo " -> $DESTS" || echo ", not pushing")"
/kaniko/executor "$@" # Keep a copy of the output: when a RUN step (not Kaniko itself) is killed for memory,
# Kaniko exits 1 and only its message says "exit status 137".
{ /kaniko/executor "$@" 2>&1; echo "$?" > /kaniko/exit-code; } | tee /kaniko/build.log
RC=$(cat /kaniko/exit-code 2>/dev/null || echo 1)
if [ "$RC" -ne 0 ] && grep -q 'exit status 137' /kaniko/build.log; then RC=137; fi
if [ "$RC" -eq 137 ]; then
LIMIT="${SCOTTY_GIT_JOB_MEMORY_LIMIT:-1.5 GB}"
fail "Build stopped: this job hit scotty-git's per-job memory limit ($LIMIT) and was killed (exit code 137). The limit protects the shared git server. Reduce the build's memory use (e.g. fewer parallel steps in the Dockerfile, smaller dependencies), or ask the scotty-git admins for a larger runner."
elif [ "$RC" -ne 0 ]; then
fail "Kaniko build failed (exit code $RC). See the Kaniko output above for the failing step."
fi
# ---- Output ---- # ---- Output ----
if [ "$PUSH" = "true" ] && [ -f /kaniko/digest ]; then if [ "$PUSH" = "true" ] && [ -f /kaniko/digest ]; then

View file

@ -0,0 +1,3 @@
# Test only: a RUN step that tries to hold 2 GB in memory (tail buffers a stream with no newlines).
FROM alpine:3.20
RUN head -c 2000m /dev/zero | tail > /dev/null