| commit | author | age | ||
| bc9b99 | 1 | Summary. |
| AP | 2 | -------- |
| 3 | ||
| 4 | Testing of a userland component that provides tests is performed by hooking | |
| 5 | those tests up to the 'test' target and running 'gmake test'. This should | |
| 6 | generally pass, as failing tests may indicate things you have to fix, or | |
| 7 | upstream tests that aren't applicable or need modifications on OpenIndiana. | |
| 8 | ||
| 9 | 'gmake test' is often run when a component is upgraded or otherwise | |
| 10 | intentionally changed, but sometimes it would be useful to rerun the tests | |
| 11 | after something else has changed (such as the system being upgraded, or a | |
| 12 | change in compilers; see the 'System Test' section below) and see if that | |
| 13 | has affected the tests. | |
| 14 | ||
| 15 | We do this by having a 'master test file' that contains the expected results, | |
| 16 | and having a compare target that runs the tests and compares them with the | |
| 17 | results. | |
| 18 | ||
| 19 | Note: because the initial test run and the current test run may have a | |
| 20 | different environment (different users, different locales, different machines, | |
| 21 | different compilers...) the results need to have all such output dependencies | |
| 22 | removed or abstracted. | |
| 23 | ||
| 24 | ||
| 25 | Setting up a master test file for a Userland component. | |
| 26 | ------------------------------------------------------- | |
| 27 | ||
| 28 | When setting up a test-and-compare run for a new component, you will need to | |
| 29 | have master test file(s) that are identical for both the x86 and SPARC | |
| 30 | platforms. It is suggested that they should initially be created by doing | |
| 31 | something like the following in your x86 Userland workspace, then copied to | |
| 32 | the same locations in your SPARC workspace and retested there. | |
| 33 | ||
| 34 | When you run "gmake test", a check is made to see if there is a master | |
| 21b154 | 35 | file of test results. If there is, then a test-and-compare run is performed. |
| bc9b99 | 36 | If there isn't, then just a "normal" run of the test suite is performed. |
| AP | 37 | |
| 38 | The name of the master test file (or files to be exact), will depend upon | |
| 39 | whether you have 32-bit, 64-bit, 32-and-64-bit and whether this is for a | |
| 40 | Python or Perl component. | |
| 41 | ||
| 42 | The default master file name is defined in | |
| 43 | make-rules/shared-macros.mk and is: | |
| 44 | ||
| 21b154 | 45 | COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-all.master |
| bc9b99 | 46 | |
| AP | 47 | where COMPONENT_TEST_RESULTS_DIR is: |
| 48 | ||
| 49 | COMPONENT_TEST_RESULTS_DIR = $(COMPONENT_DIR)/test | |
| 50 | ||
| 21b154 | 51 | so that means one single common master test results file will be used. |
| MT | 52 | |
| 53 | If the component needs different test result file based on bitness (32-bit vs. | |
| 54 | 64-bit) then you should simply set USE_COMMON_TEST_MASTER to "no" in the | |
| 55 | component's Makefile before shared-macros.mk is included. Then the master test | |
| 56 | result file names will be: | |
| bc9b99 | 57 | |
| AP | 58 | 32-bit: components/<component-name>/test/results-32.master |
| 59 | ||
| 60 | 64-bit: components/<component-name>/test/results-64.master | |
| 61 | ||
| 62 | both: components/<component-name>/test/results-32.master | |
| 63 | components/<component-name>/test/results-64.master | |
| 64 | ||
| 21b154 | 65 | For both Python and Perl the common master test result file |
| MT | 66 | $(COMPONENT_TEST_RESULTS_DIR)/results-all.master is used by default. |
| 67 | ||
| 68 | If USE_COMMON_TEST_MASTER is set to "no" then for Python COMPONENT_TEST_MASTER | |
| 69 | is overridden in make-rules/setup.py.mk to be: | |
| bc9b99 | 70 | |
| AP | 71 | COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PYTHON_VERSION)-$(BITS).master |
| 72 | ||
| 73 | so that means it's looking for one or more of: | |
| 74 | ||
| 75 | 2.7: components/python/<component-name>/test/results-2.7-32.master | |
| 76 | 2.7: components/python/<component-name>/test/results-2.7-64.master | |
| 77 | 3.5: components/python/<component-name>/test/results-3.5-64.master | |
| 78 | ||
| 79 | depending upon which versions of Python this component supports. | |
| 80 | ||
| 21b154 | 81 | Perl is similar, when USE_COMMON_TEST_MASTER is set to "no" the |
| MT | 82 | COMPONENT_TEST_MASTER is being overridden in make-rules/makemaker.mk to be: |
| bc9b99 | 83 | |
| 21b154 | 84 | COMPONENT_TEST_MASTER = $(COMPONENT_TEST_RESULTS_DIR)/results-$(PERL_VERSION).master |
| bc9b99 | 85 | |
| AP | 86 | so that means it's looking for one or more of: |
| 87 | ||
| 21b154 | 88 | 5.34: components/perl/<component>/test/results-5.34.master |
| MT | 89 | 5.36: components/perl/<component>/test/results-5.36.master |
| bc9b99 | 90 | |
| AP | 91 | depending upon which versions of Perl this component supports. |
| 92 | ||
| 21b154 | 93 | In order to do a test-and-compare run rather than just run the component |
| bc9b99 | 94 | test suite, initially just create an empty master test file (or files). |
| AP | 95 | |
| 96 | For example, for elinks, which just has 64-bit tests, do: | |
| 97 | ||
| 98 | $ cd components/elinks | |
| 99 | $ mkdir test | |
| 100 | $ touch test/results-64.master | |
| 101 | $ gmake test | |
| 102 | ||
| 103 | At this point, you have a set of test results in | |
| 104 | components/elinks/build/test/$(MACH64)/test-64-results. | |
| 105 | ||
| 106 | Even better, there are a set of "global" regular expressions that are | |
| 107 | run on those test results to try to normalize them. The output from that | |
| 108 | is placed in components/elinks/build/test/$(MACH64)/results-64.snapshot | |
| 109 | ||
| 110 | You can now use the contents of that file as a first cut at the master results. | |
| 111 | ||
| 112 | $ cp build/test/$(MACH64)/results-64.snapshot test/results-64.master | |
| 113 | ||
| 114 | Now run the tests again. Note that you have to get back to a clean start | |
| 115 | just in case the test process tries to compile any code. | |
| 116 | ||
| 117 | $ gmake clean | |
| 118 | $ gmake test | |
| 119 | ||
| 120 | At this point, it will again compare the test results against the master(s), | |
| 121 | and if there are still differences, they will be placed in | |
| 122 | components/elinks/build/test/$(MACH64)/test-64-diffs | |
| 123 | ||
| 124 | Typically these differences will be for things like usernames, temporary | |
| 125 | filenames, timings etc. If you have some differences, then you are going | |
| 126 | to have to write special regexp expressions in your component Makefile to | |
| 127 | "normalize" them, and adjust the master test results file so that it | |
| 128 | matches what the normalized version looks like. | |
| 129 | ||
| 130 | Transform examples. | |
| 131 | ------------------- | |
| 132 | ||
| 133 | 1/ Handling temporary filenames. | |
| 134 | ||
| 135 | See the transform in the asciidoc Makefile: | |
| 136 | ||
| 137 | COMPONENT_TEST_TRANSFORMS += \ | |
| 138 | '-e "s|/tmp......|/tmpxxxxxx|g" ' | |
| 139 | ||
| 140 | for how to "normalize" any temporary filenames that appear in the test results. | |
| 141 | ||
| 142 | 2/ Build related output. | |
| 143 | ||
| 144 | Items (such as paths to compilers) should be removed from master test files. | |
| 145 | For example, in testing the xz component, we get the following different | |
| 146 | output for compiling a .c file for 32-bit and 64-bit: | |
| 147 | ||
| 148 | x86 32-bit: | |
| 149 | ||
| 150 | /usr/gcc/6/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests -I.. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/common -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/liblzma/api -I../lib -D_REENTRANT -pthreads -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -m32 -O3 -MT test_check.o -MD -MP -MF .deps/test_check.Tpo -c -o test_check.o /export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests/test_check.c | |
| 151 | ||
| 152 | x86 64-bit: | |
| 153 | ||
| 154 | /usr/gcc/6/bin/gcc -DHAVE_CONFIG_H -I. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests -I.. -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/common -I/export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/src/liblzma/api -I../lib -D_REENTRANT -pthreads -fvisibility=hidden -Wall -Wextra -Wvla -Wformat=2 -Winit-self -Wmissing-include-dirs -Wstrict-aliasing -Wfloat-equal -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wlogical-op -Waggregate-return -Wstrict-prototypes -Wold-style-definition -Wmissing-prototypes -Wmissing-declarations -Wmissing-noreturn -Wredundant-decls -m64 -O3 -MT test_check.o -MD -MP -MF .deps/test_check.Tpo -c -o test_check.o /export/home/alp/srcs/oi-userland/components/archiver/xz/xz-5.2.4/tests/test_check.c | |
| 155 | ||
| 156 | These can all be normalized (so that a single master test file can be used), | |
| 157 | by adding the following line to COMPONENT_TEST_TRANSFORMS in the xz component | |
| 158 | Makefile: | |
| 159 | ||
| 160 | '-e "s|\(^$(CC)\).*|\\$$(CC)|" ' | |
| 161 | ||
| 162 | The equivalent compile line in the master test file is now set to just: | |
| 163 | ||
| 164 | $(CC) | |
| 165 | ||
| 166 | Note that we are using $(CC) here which will pick up the current C compiler. | |
| 167 | Should the C compiler or build options change in the future, then there will | |
| 168 | be no need to adjust the master test results file to match. It's all done | |
| 169 | automatically. | |
| 170 | ||
| 171 | It is recognized that an alternate approach to solving this problem is to | |
| 172 | create a transform to just remove lines like this from the master test files, | |
| 173 | but there is currently no automatic way to pick up the default compiler. | |
| 174 | ||
| 175 | ---- | |
| 176 | ||
| 177 | There will be other examples as more components are adjusted to test | |
| 178 | against master results files. | |
| 179 | ||
| 180 | If a lot of people start generating the very same ones, then we can | |
| 181 | consider adding them to the "global" set of regexps in | |
| 182 | make-rules/shared-macros.mk which currently looks like: | |
| 183 | ||
| 184 | COMPONENT_TEST_TRANSFORMS = \ | |
| 185 | '-e "s|$(@D)|\\$$(@D)|g" ' \ | |
| 186 | '-e "s|$(PERL)|\\$$(PERL)|g" ' \ | |
| 187 | '-e "s|$(SOURCE_DIR)|\\$$(SOURCE_DIR)|g" ' | |
| 188 | ||
| 189 | Note that in some cases, gsed is not powerful enough. E.g., if you need to | |
| 190 | do more than massage or remove certain lines, but only compare certain sections | |
| 191 | of the file, then you may need to set COMPONENT_TEST_TRANSFORMER to something | |
| 192 | other than $(GSED) such as $(NAWK). In this case, you will want to set | |
| 193 | COMPONENT_TEST_TRANSFORMS using "=" instead of "+=", as the default sed | |
| 194 | transforms will likely need to be tossed aside in favor of whatever arguments | |
| 195 | your favorite transformer might need. | |
| 196 | ||
| 197 | When your master test file(s) are in good shape, then you should "git add" | |
| 198 | them to your workspace. | |
