107 lines
4.6 KiB
Markdown
107 lines
4.6 KiB
Markdown
# buildah-bud tests under podman
|
|
|
|
This directory contains tools for running 'buildah bud' tests
|
|
under podman. The key concept of the workflow is:
|
|
|
|
* Pull buildah @ version specified in go.mod
|
|
* Apply a small set of patches to buildah's tests directory, such that
|
|
* BATS will use 'podman build' instead of 'buildah bud'; and
|
|
* some not-applicable-under-podman tests are skipped
|
|
|
|
It's a teeny bit more complicated than that, but that's really most of
|
|
what you need to know for most purposes. The tests run in podman CI,
|
|
and for the most part are expected to just pass.
|
|
|
|
## Troubleshooting
|
|
|
|
If you're reading this, it's probably because something went wrong.
|
|
At the time of this writing (March 2021, initial commit) it is
|
|
impossible to foresee what typical failures will look like, but
|
|
my prediction is that they will fit one of two categories:
|
|
|
|
* Failure when vendoring new buildah (e.g., by dependabot)
|
|
* Other failure
|
|
|
|
Let's examine those in reverse order:
|
|
|
|
## Failure when not vendoring
|
|
|
|
Aside from flakes, my only guess here is that you broke 'podman build'.
|
|
If this is the case, it is very likely that you are aware of what you
|
|
did; and if this is the case, your change likely falls into one of
|
|
these two categories:
|
|
|
|
* "OOPS! I didn't mean to break that". Solution: fix it!
|
|
* "Uh, yeah, this is deliberate, and we choose to be incompatible with buildah". In this case, you'll need to skip or edit the failing test(s); see below.
|
|
|
|
If neither of those is the case, then I'm sorry, you're on your own.
|
|
When you figure it out, please remember to update these instructions.
|
|
|
|
|
|
## Failure when vendoring new buildah
|
|
|
|
This is what I predict will be the usual case; and I predict that
|
|
failures will fall into one of two bins:
|
|
|
|
* failure to apply the patches; and/or
|
|
* failure because there are new buildah tests for functionality not in podman
|
|
|
|
In either case, the process for solving is the same:
|
|
|
|
* Start with a checked-out podman tree with the failing PR applied
|
|
* run `./test/buildah-bud/run-buildah-bud-tests`
|
|
|
|
Presumably, something will fail here. Whatever the failure, your next step is:
|
|
|
|
* `cd test-buildah-v<TAB>` (this is a new directory created by the script)
|
|
|
|
Now there are three possible failures:
|
|
|
|
### Failure in `git am`
|
|
|
|
If the failure was in `git am`, it probably means that buildah
|
|
`tests/helpers.bash` got updated in such a way as to cause a conflict
|
|
with the patches we apply. Your best bet is to:
|
|
|
|
* Look at `tests/*.rej`
|
|
* For each rejected patch, try to figure out where it should go and how to apply it. Do so.
|
|
* `git add tests/helpers.bash` - this is for `git am`, next
|
|
* `git am --continue` - this continues the failed patch. Make sure it succeeds.
|
|
* `./make-new-buildah-diffs` - this updates your podman working directory
|
|
* `cd ..; git diff test/buildah-bud`. This will show you a diff of a .diff file, which is really painful to read. I'm sorry. Just try to confirm that the changes look like what you expect.
|
|
|
|
Proceed with 'In all cases' below.
|
|
|
|
### Failure when applying podman-custom deltas
|
|
|
|
Failure in the `apply-podman-deltas` script means that one of the
|
|
hand-crafted exceptions was not found, e.g., there's a `skip` or
|
|
`errmsg` looking for a specific `@test` in `bud.bats` that is
|
|
no longer there.
|
|
|
|
Solution:
|
|
* Inspect the error message(s) from `apply-podman-deltas`. Each message will list a specific `@test` name.
|
|
* Look at the diffs in `tests/bud.bats` between main and your PR. (I'm really sorry; there's no quick easy command-line way to do that. You will need a checked-out buildah tree, and you will need to know the old and new buildah tags).
|
|
* In those diffs, look for changes related to each `@test` listed as an error. For example, a test being renamed or even removed.
|
|
* Update `test/buildah-bud/apply-podman-deltas` accordingly.
|
|
|
|
Proceed with 'In all cases' below.
|
|
|
|
### Failure when running tests
|
|
|
|
If the failure was in tests run, and you're vendoring, your only real choice is to add a new `skip`:
|
|
|
|
* Identify the failing test(s)
|
|
* File a new podman issue, e.g. "podman build fails buildah XYZ test"
|
|
* Edit `test/buildah-bud/apply-podman-deltas`. Search for "actual podman bugs" near the bottom, and add a new `skip` line with the reason (INCLUDE THE ISSUE NUMBER!) and the test name.
|
|
|
|
### In all cases
|
|
|
|
You will probably want to rerun `run-buildah-bud-tests` to save yourself
|
|
the hassle of having it fail in CI. (`rm -rf test-buildah-v<TAB>` first).
|
|
If you're debugging problems that run on a specific test, you can
|
|
use `--filter="pattern"` to run only tests that match "pattern".
|
|
|
|
If everything passes, `git commit --amend` your PR, adding the
|
|
files you changed under `test/buildah-bud`, then `git push --force`.
|