Commit Graph

91 Commits

Author SHA1 Message Date
Rabin Vincent
f9dfb65b05 ktest: Check parent options for iterated tests
Let's say we have "OUTPUT_DIR = build/${TEST_NAME}", and we're iterating
a test.  In the second iteration of a test, the TEST_NAME of the test
we're repeating is not used.  Instead, ${TEST_NAME} appears literally:

   touch /home/rabin/kernel/test/build/${TEST_NAME}/.config ... SUCCESS

Fix this by making __eval_option() check the parent test options
for a repeated test.

Link: http://lkml.kernel.org/r/1321616131-21352-2-git-send-email-rabin@rab.in

Signed-off-by: Rabin Vincent <rabin@rab.in>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-11-18 11:07:24 -05:00
Steven Rostedt
815e2bd7d6 ktest: Evaluate variables entered on the command line
When ktest.pl is called without any arguments, or if the config
file does not exist, ktest.pl will ask the user for some information.
Some of these questions are code paths. Allowing the user to type
${PWD} for the current directory greatly simplifies these entries.

Add variable processing to the entered values.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-28 07:01:40 -04:00
Steven Rostedt
7bf5107347 ktest: Add variable ${PWD}
Adding the variable ${PWD} that equals `pwd` makes the config files
much simpler.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-22 09:07:03 -04:00
Steven Rostedt
9f7424cc86 ktest: Add another monitor flush before installing kernel
On some tests that do multiple boots (patchcheck, bisect, etc), the build
of the next kernel to run may finish before the stable kernel has finished
booting. Then the install of the new kernel will fail when it tries to connect
as the machine has not finished the boot process.

Do one more monitor flush to make sure the machine is up and running before
trying to connect to it again.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-22 08:58:19 -04:00
Steven Rostedt
c54367f9d6 ktest: Do not opencode reboot in grub setting
When setting the next kernel to boot to with grub, do not opencode
the reboot operation.  The normal reboot operation can be modified by
config options (namely POWERCYCLE_AFTER_REBOOT). This needs to affect
all reboots. Remove the opencoded reboot to make sure that any changes
to the reboot code also affect all reboots.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-20 09:56:41 -04:00
Steven Rostedt
8d735212e4 ktest: Add processing of complex conditionals
The IF statements for DEFAULTS and TEST_START sections now handle
complex statements (&&,||)

Example:

  TEST_START IF (DEFINED ALL_TESTS || ${MYTEST} == boottest) && ${MACHINE} == gandalf

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:58:18 -04:00
Steven Rostedt
a9f84424be ktest: Fix parsing of config section lines
The order for some of the keywords on a section line
(TEST_START or DEFAULTS) does not really matter. Simply need
to remove the keyword from the line as we process it and
evaluate the next keyword in the line. By removing the keywords
as we find them, we do not need to keep track of where on the
line they were found.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:58:15 -04:00
Steven Rostedt
ac6974c76e ktest: Sort make_min_config configs by dependecies
The make_min_config test will turn off one config at a time and check
if the config boots or not, and if it does, it will remove that config
plus any config that depended on that config.

ktest already looks if a config has a dependency and will try the
dependency config first. But by sorting the configs and trying the
config with the most configs dependent on it, we can shrink the
minconfig faster.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:12 -04:00
Steven Rostedt
9900b5dc06 ktest: Add DEFINED keyword for IF statements
Have IF statements process if a config variable or option has been
defined or not. Can use NOT DEFINED in the case for telling if
a variable or option has not been defined.

DEFAULTS IF NOT DEFINED SSH_USER
SSH_USER = root

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
3d1cc41432 ktest: Add OVERRIDE keyword to DEFAULTS section
The OVERRIDE keyword will allow options defined in the given
DEFAULTS section to override options defined in previous DEFAULT
sections.

Normally, options will error if they were previous defined.
The OVERRIDE keyword allows options that have been previously
defined to be changed in the given section.

Note, the same option can not be defined in the same DEFAULT section
even if that section is marked as OVERRIDE.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
0050b6bbef ktest: Consolidate TEST_TYPE and DEFAULT code
The code that handles parsing the TEST_TYPE and DEFAULT code share
a lot of common functionality. Combine the two and add a if statement
that does what is different between them.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
2ed3b16128 ktest: Add INCLUDE keyword to include other config files
Have the reading of the config file allow reading of other config
files using the INCLUDE keyword. This allows multiple config files
to share config options.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
ab7a3f52ce ktest: Let IF keyword take comparisons
Allow ==, !=, <=, >=, <, and > to be used in IF statements
to compare if a section should be processed or not.

For example:

BITS := 32

DEFAULTS IF ${BITS} == 32
MIN_CONFIG = ${CONFIG_DIR}/config-32
ELSE
MIN_CONFIG = ${CONFIG_DIR}/config-64

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
45d73a5d8a ktest: Add IF and ELSE to config sections
Add IF keyword to sections within the config. Also added an ELSE
keyword that allows different config options to be set for a given
section.

For example:

TYPE := 1
STATUS := 0

DEFAULTS IF ${TYPE}
[...]
ELSE IF ${STATUS}
[...]
ELSE
[...]

The above will process the first section as $TYPE is true. If it
was false, it would process the last section as $STATUS is false.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
4ab1cce5bd ktest: Do not reboot on config or build issues
Even if REBOOT_ON_ERROR is set, it becomes annoying that the target
machine is rebooted when a config option is incorrect or a build
fails. There's no reason to reboot the target for host only issues.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:11 -04:00
Steven Rostedt
2b803365a6 ktest: Add option REBOOT_SUCCESS_LINE to stop waiting after a reboot
When ktest.pl reboots, it will usuall wait SLEEP_TIME seconds of idle
console before starting the next test. By setting the
REBOOT_SUCCESS_LINE, ktest will not wait SLEEP_TIME when it detects the
line while rebooting to a new kernel.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Steven Rostedt
e0a8742e3d ktest: Add NO_INSTALL option to not install for a test
There's cases where running the same kernel over and over again
is useful, and being able to not install the same kernel can
save time between tests.

Add a NO_INSTALL option that tells ktest.pl to not install the
new kernel.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Steven Rostedt
eaa1fe25ea ktest: Fail when grub menu not found
Currently if the grub menu that is supplied is not found, it will
just boot into the last grub menu in menu.lst. Fail instead of
confusing the user why their kernel is not booting.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Andrew Jones
2728be418d ktest: Include monitor in reboot code
Several places that call reboot do the same thing with respect to the
monitor. By adding this code into the reboot code, redundant code is
removed and it paves the way for the the reset time patch.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Link: http://lkml.kernel.org/r/1313155932-20092-4-git-send-email-drjones@redhat.com
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Andrew Jones
134882311c ktest: Only need to save .config when doing mrproper
Only save the .config file if we're doing mrproper

Signed-off-by: Andrew Jones <drjones@redhat.com>
Link: http://lkml.kernel.org/r/1313155932-20092-3-git-send-email-drjones@redhat.com
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Andrew Jones
a908a6659b ktest: Create outputdir if it does not exist
Signed-off-by: Andrew Jones <drjones@redhat.com>
Link: http://lkml.kernel.org/r/1313155932-20092-2-git-send-email-drjones@redhat.com
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Steven Rostedt
cd8e368f47 ktest: Add TEST_TYPE install option
In testing one of my boxes, I found that I only wanted to build and
install the kernel. I wanted to manually reboot the box and test it.
Adding a TEST_TYPE option "install" allows this to happen.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-10-17 11:54:10 -04:00
Steven Rostedt
250bae8be0 ktest: Fix bug when ADD_CONFIG is set but MIN_CONFIG is not
The MIN_CONFIG is a single config that is considered to have all the
configs that are required to boot the box.

ADD_CONFIG is a list of configs that we add that may contain configs
known to be broken (set off) or just configs that we want every box to
have and this can include shared configs.

If a config has no MIN_CONFIG defined, but has multiple files defined
for the ADD_CONFIG, the test will die, because the MIN_CONFIG will
default to ADD_CONFIG. The problem is the code to open MIN_CONFIG
expects a string of one file, not multiple, and the open will fail.

Since the real minconfig that is used is a concatination of MIN_CONFIG
and ADD_CONFIG files, we change the code to open that instead of
whatever MIN_CONFIG defaults to.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-07-15 22:05:59 -04:00
Steven Rostedt
43d1b6518e ktest: Keep fonud configs separate from default configs
The IGNORE_CONFIG file holds the configs that we don't want to change
(with their proper settings). But on start up, the make noconfig is
executed, and the configs that are on are also put into the ignore
config category. But these are configs that were forced on by the
kconfig scripts and not something that we found must be enabled to boot
our machine. By keeping the configs that are forced on by default,
separate from the configs we found that are required to boot the box, we
can get a much more interesting IGNORE_CONFIG. In fact, the
IGNORE_CONFIG can usually end up being the must have configs to boot,
and only have 6 or 7 configs set.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-07-15 22:01:56 -04:00
Steven Rostedt
35ce5952e6 ktest: Add prompt to use OUTPUT_MIN_CONFIG
If the defined OUTPUT_MIN_CONFIG in the make_min_config test exists,
then give a prompt to ask the user if they want to use that config
instead, as it is very often the case, especially when the test has been
interrupted. The OUTPUT_MIN_CONFIG is usually the config that one wants
to use to continue the test where they left off.

But if START_MIN_CONFIG is defined (thus the MIN_CONFIG is not the
default), then do not prompt, as it will be annoying if the user has
this as one of many tests, and the test pauses waiting for input, while
the user is sleeping.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-07-15 21:57:25 -04:00
Steven Rostedt
b9066f6c0e ktest: Use Kconfig dependencies to shorten time to make min_config
To save time, the test does not just grab any option and test
it. The Kconfig files are examined to determine the dependencies
of the configs. If a config is chosen that depends on another
config, that config will be checked first. By checking the
parents first, we can eliminate whole groups of configs that
may have been enabled.

For example, if a USB device config is chosen and depends on
CONFIG_USB, the CONFIG_USB will be tested before the device.
If CONFIG_USB is found not to be needed, it, as well as all
configs that depend on it, will be disabled and removed from
the current min_config.

Note, the code from streamline_config (make localmodconfig)
was copied and used to find the dependencies in the Kconfig file.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-07-15 21:29:09 -04:00
Steven Rostedt
4c4ab1204f ktest: Add test type make_min_config
After doing a make localyesconfig, your kernel configuration may
not be the most useful minimum configuration. Having a true minimum
config that you can use against other configs is very useful if
someone else has a config that breaks on your code. By only forcing
those configurations that are truly required to boot your machine
will give you less of a chance that one of your set configurations
will make the bug go away. This will give you a better chance to
be able to reproduce the reported bug matching the broken config.

Note, this does take some time, and may require you to run the
test over night, or perhaps over the weekend. But it also allows
you to interrupt it, and gives you the current minimum config
that was found till that time.

Note, this test automatically assumes a BUILD_TYPE of oldconfig
and its test type acts like boot.

TODO: add a test version that makes the config do more than just
      boot, like having network access.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-07-15 21:19:44 -04:00
Steven Rostedt
0df213ca31 ktest: Require one TEST_START in config file
There has been too many times that I put in one too many SKIP
TEST_STARTs and start the test with the default randconfig by accident
that I added this to have ktest ask the user for which test they want to
run if no TEST_START is specified.

Now if I accidently start the test with all TEST_STARTs skipped, ktest
asks what test do I want to run, and I now have a chance to kill it
before it does a make mrproper on my build directory.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-14 20:51:37 -04:00
Steven Rostedt
ddf607e5f8 ktest: Add helper function to avoid duplicate code
Several places had the following code:

    get_grub_index;
    get_version;
    install;

    start_monitor;
    return monitor;

Creating a function "start_monitor_and_boot()" replaces these mulitple
uses with a single call.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-14 20:49:13 -04:00
Steven Rostedt
1990207d53 ktest: Add IGNORE_WARNINGS to ignore warnings in some patches
Doing a patchcheck test, there may be warnings that gcc produces which
may be OK, and the test should not fail on that commit. By adding a
IGNORE_WARNINGS option to list a space delimited SHA1s that are ignored
lets the user avoid having the test fail on certain commits.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-14 20:46:25 -04:00
Steven Rostedt
e7b1344189 ktest: Fix tar extracting of modules to target
The tar command to create the module directory is cjf, but the
extraction only had xf. This works on most versions of tar, but some
versions of tar require xjf for extraction as well.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-14 20:44:36 -04:00
Steven Rostedt
4892063043 ktest: Have the testing tmp dir include machine name
As multiple tests may be executed by the same server, have the test
machine name add uniqueness to the value of the temp directory.
Otherwise the temp directories may overwrite each other's tests.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-14 20:43:59 -04:00
Steven Rostedt
0bd6c1a38f ktest: Add POST/PRE_BUILD options
There are some cases that a patch may be needed to apply to the kernel
in patchcheck or bisect tests. Adding a PRE_BUILD option to apply the
patch and POST_BUILD to remove it, allows for this to be done easily.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-14 20:39:31 -04:00
Steven Rostedt
db05cfefce ktest: Allow initrd processing without modules defined
When a config is set with CONFIG_MODULES=n, it does not mean that the
kernel does not need an initrd to boot. For systems that depend on LVM
and such, an initrd must run first.

If POST_INSTALL is defined, then run the post install regardless if
modules are needed or not.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 11:09:22 -04:00
Steven Rostedt
23715c3c9a ktest: Have LOG_FILE evaluate options as well
The LOG_FILE variable needs to evaluate the $ options as well.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 11:03:34 -04:00
Steven Rostedt
ecaf8e5213 ktest: Have wait on stdio honor bug timeout
After a bug is found, the STOP_AFTER_FAILURE timeout is used to
determine how much output should be printed before breaking out
of the monitor loop. This is to get things like call traces and
enough infromation about the bug to help determine what caused it.

The STOP_AFTER_FAILURE is usually much shorter than the TIMEOUT
that is used to determine when to quit after no more stdio is given.

But since the stdio read uses a wait on I/O, the STOP_AFTER_FAILURE is
only checked after we get something from I/O. But if the I/O does
not return any more data, we wait the TIMEOUT period instead, even
though we already triggered a bug report.

The wait on I/O should honor the STOP_AFTER_FAILURE time if a bug has
been found.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 10:48:10 -04:00
Steven Rostedt
fcb3f16a4f ktest: Implement our own force min config
Using the build KCONFIG_ALLCONFIG environment variable to force
the min config may not always work properly. Since ktest is
written in perl, it is trivial to read and replace the current
config with the configs specified by the min config.

Now the min config (and add configs) are read by perl and before
a make is done, these configs in the .config file are replaced
by the version in the min config.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 10:40:58 -04:00
Steven Rostedt
9064af5206 ktest: Add TEST_NAME option
Searching through several tests, it gets confusing which test result
is for which test. By adding the TEST_NAME option, the user can tell
which test result belongs to which test.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 10:38:48 -04:00
Steven Rostedt
30f75da5ff ktest: Add CONFIG_BISECT_GOOD option
Currently the config_bisect compares the min config with the
CONFIG_BISECT config. There may be another config that we know
is good that we want to ignore configs on. By passing in this
config it will ignore the options that are set in the good config.

Note: This only ignores the config, it does not (yet) handle
options that are different between the two configs. If the good
config has "SLAB" set and the bad config has "SLUB" it will not
find the bug if the bug had to do with changing these two options.

This is something that I intend to implement in the future.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 10:35:35 -04:00
Steven Rostedt
f1a5b96219 ktest: Add detection of triple faults
When a triple fault happens in a test, no call trace nor panic
is displayed. Instead, the system reboots to the good kernel.
Since the good kernel may display a boot prompt that matches the
success string, ktest may think that the test succeeded, when it
did not.

Detecting triple faults is tricky because it is hard to generalize
what a reboot looks like. The best that we can come up with for now
is to examine the Linux banner. If we detect that the Linux banner
matches the test we want to test, then look to see if we hit another
Linux banner with a different kernel is booted. This can be assumed
to be a triple fault.

We can't just check for two Linux banners because things like
early printk may cause the Linux banner to be displayed twice. Checking
for different kernel versions should be the safe bet.

If this for some reason detects a false triple boot. A new ktest
config option is also created:

 DETECT_TRIPLE_FAULT

This can be set to 0 to disable this checking.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 10:30:00 -04:00
Steven Rostedt
cd4f1d536c ktest: Notify reason to break out of monitoring boot
Different timeouts can cause the ktest monitor to break out of the
loop. It becomes annoying that one does not know the reason why
it exited the monitor loop. Display the cause of the reason why
the loop was exited.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-13 10:26:27 -04:00
Steven Rostedt
9bf7174949 ktest: Ignore unset values of the minconfig in config_bisect
By ignoring the unset values of the minconfig in deciding
what to test in the config_bisect can cause the problem
config from being tested too.

Just do not test the configs that are set in the minconfig.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-02 11:57:57 -04:00
Steven Rostedt
4da46da2d2 ktest: Fix result of rebooting the kernel
The command that is called that reboots the kernel may fail
but the return code is not passed back to the ktest.pl script.
This is because a ';' is used between the two commands and
if the second command fails, only the first command's return
code is returned. Using a '&&' between the two commands fixes
this.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-02 11:57:56 -04:00
Steven Rostedt
4c8cc55b3c ktest: Fix off-by-one in config bisect result
Because in perl the array size returned by $#arr, is the last
index and not the actually size of the array, we end the config
bisect early, thinking there is only one config left when there
are in fact two. Thus the result has a 50% chance of picking
the correct config that caused the problem.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-06-02 11:57:41 -04:00
Steven Rostedt
2a62512bce ktest: Allow options to be used by other options
There are cases where one ktest option may be used within another
ktest option. Allow them to be reused just like config variables
but there are evaluated at time of test not config processing time.

Thus having something like:

MAKE_CMD = make ARCH=${ARCH}

TEST_START
ARCH = powerpc

TEST_START
ARCH = arm

Will have the arch defined for each test iteration.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-05-20 15:48:59 -04:00
Steven Rostedt
77d942ceac ktest: Create variables for the ktest config files
I found that I constantly reuse information for each test case.
It would be nice to just define a variable to reuse.

For example I may have:

TEST_START
[...]
TEST = ssh root@mybox /path/to/my/script

TEST_START
[...]
TEST = ssh root@mybox /path/to/my/script

[etc]

The issue is, I may wont to change that script or one of the other
fields. Then I need to update each line individually.

With the addition of config variables (variables only used during parsing
the config) we can simplify the config files. These variables can
also be defined multiple times and each time the new value will
overwrite the old value.

The convention to use a config variable over a ktest option is to use :=
instead of =.

Now we could do:

USER := root
TARGET := mybox
TEST_SCRIPT := /path/to/my/script
TEST_CASE := ${USER}@${TARGET} ${TEST_SCRIPT}

TEST_START
[...]
TEST = ${TEST_CASE}

TEST_START
[...]
TEST = ${TEST_CASE}

[etc]

Now we just need to update the variables at the top.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-05-20 15:26:26 -04:00
Steven Rostedt
27d934b287 ktest: Reboot after each patchcheck run
The patches being checked may not leave the kernel in a state
that the next run will allow the new kernel to be copied to the
machine. Reboot to a known good kernel before continuing to the
next kernel to test.

Added option PATCHCHECK_SLEEP_TIME for the max time to sleep between
patchcheck reboots.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-05-20 09:21:02 -04:00
Steven Rostedt
4025bc62dd ktest: Reboot to good kernel after every bisect run
Reboot after each bisect run regardless if the bisect passed
or failed. The test may just be to boot the kernel and that kernel
may not have a way to copy the next kerne to it. Reboot to a known
good kernel after each bisect run.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-05-20 09:16:29 -04:00
Steven Rostedt
4d62bf51ac ktest: If test failed due to timeout, print that
If the test failed due to timeout for boot, print a message saying
so. Otherwise the user will be confused to why their test just failed.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-05-20 09:14:35 -04:00
Steven Rostedt
ca6a21f874 ktest: Fix post install command
The command to run post install (for those that want initrds) was
broken. Instead of doing a substitution for the $KERNEL_VERSION
variable. It was replacing the entire command with nothing.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
2011-05-20 09:11:58 -04:00