| 1 | \input texinfo @c -*- texinfo -*- |
| 2 | |
| 3 | @settitle FFmpeg Automated Testing Environment |
| 4 | @titlepage |
| 5 | @center @titlefont{FFmpeg Automated Testing Environment} |
| 6 | @end titlepage |
| 7 | |
| 8 | @node Top |
| 9 | @top |
| 10 | |
| 11 | @contents |
| 12 | |
| 13 | @chapter Introduction |
| 14 | |
| 15 | FATE is an extended regression suite on the client-side and a means |
| 16 | for results aggregation and presentation on the server-side. |
| 17 | |
| 18 | The first part of this document explains how you can use FATE from |
| 19 | your FFmpeg source directory to test your ffmpeg binary. The second |
| 20 | part describes how you can run FATE to submit the results to FFmpeg's |
| 21 | FATE server. |
| 22 | |
| 23 | In any way you can have a look at the publicly viewable FATE results |
| 24 | by visiting this website: |
| 25 | |
| 26 | @url{http://fate.ffmpeg.org/} |
| 27 | |
| 28 | This is especially recommended for all people contributing source |
| 29 | code to FFmpeg, as it can be seen if some test on some platform broke |
| 30 | with their recent contribution. This usually happens on the platforms |
| 31 | the developers could not test on. |
| 32 | |
| 33 | The second part of this document describes how you can run FATE to |
| 34 | submit your results to FFmpeg's FATE server. If you want to submit your |
| 35 | results be sure to check that your combination of CPU, OS and compiler |
| 36 | is not already listed on the above mentioned website. |
| 37 | |
| 38 | In the third part you can find a comprehensive listing of FATE makefile |
| 39 | targets and variables. |
| 40 | |
| 41 | |
| 42 | @chapter Using FATE from your FFmpeg source directory |
| 43 | |
| 44 | If you want to run FATE on your machine you need to have the samples |
| 45 | in place. You can get the samples via the build target fate-rsync. |
| 46 | Use this command from the top-level source directory: |
| 47 | |
| 48 | @example |
| 49 | make fate-rsync SAMPLES=fate-suite/ |
| 50 | make fate SAMPLES=fate-suite/ |
| 51 | @end example |
| 52 | |
| 53 | The above commands set the samples location by passing a makefile |
| 54 | variable via command line. It is also possible to set the samples |
| 55 | location at source configuration time by invoking configure with |
| 56 | `--samples=<path to the samples directory>'. Afterwards you can |
| 57 | invoke the makefile targets without setting the SAMPLES makefile |
| 58 | variable. This is illustrated by the following commands: |
| 59 | |
| 60 | @example |
| 61 | ./configure --samples=fate-suite/ |
| 62 | make fate-rsync |
| 63 | make fate |
| 64 | @end example |
| 65 | |
| 66 | Yet another way to tell FATE about the location of the sample |
| 67 | directory is by making sure the environment variable FATE_SAMPLES |
| 68 | contains the path to your samples directory. This can be achieved |
| 69 | by e.g. putting that variable in your shell profile or by setting |
| 70 | it in your interactive session. |
| 71 | |
| 72 | @example |
| 73 | FATE_SAMPLES=fate-suite/ make fate |
| 74 | @end example |
| 75 | |
| 76 | @float NOTE |
| 77 | Do not put a '~' character in the samples path to indicate a home |
| 78 | directory. Because of shell nuances, this will cause FATE to fail. |
| 79 | @end float |
| 80 | |
| 81 | To use a custom wrapper to run the test, pass @option{--target-exec} to |
| 82 | @command{configure} or set the @var{TARGET_EXEC} Make variable. |
| 83 | |
| 84 | |
| 85 | @chapter Submitting the results to the FFmpeg result aggregation server |
| 86 | |
| 87 | To submit your results to the server you should run fate through the |
| 88 | shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs |
| 89 | to be invoked with a configuration file as its first argument. |
| 90 | |
| 91 | @example |
| 92 | tests/fate.sh /path/to/fate_config |
| 93 | @end example |
| 94 | |
| 95 | A configuration file template with comments describing the individual |
| 96 | configuration variables can be found at @file{doc/fate_config.sh.template}. |
| 97 | |
| 98 | @ifhtml |
| 99 | The mentioned configuration template is also available here: |
| 100 | @verbatiminclude fate_config.sh.template |
| 101 | @end ifhtml |
| 102 | |
| 103 | Create a configuration that suits your needs, based on the configuration |
| 104 | template. The `slot' configuration variable can be any string that is not |
| 105 | yet used, but it is suggested that you name it adhering to the following |
| 106 | pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file |
| 107 | itself will be sourced in a shell script, therefore all shell features may |
| 108 | be used. This enables you to setup the environment as you need it for your |
| 109 | build. |
| 110 | |
| 111 | For your first test runs the `fate_recv' variable should be empty or |
| 112 | commented out. This will run everything as normal except that it will omit |
| 113 | the submission of the results to the server. The following files should be |
| 114 | present in $workdir as specified in the configuration file: |
| 115 | |
| 116 | @itemize |
| 117 | @item configure.log |
| 118 | @item compile.log |
| 119 | @item test.log |
| 120 | @item report |
| 121 | @item version |
| 122 | @end itemize |
| 123 | |
| 124 | When you have everything working properly you can create an SSH key pair |
| 125 | and send the public key to the FATE server administrator who can be contacted |
| 126 | at the email address @email{fate-admin@@ffmpeg.org}. |
| 127 | |
| 128 | Configure your SSH client to use public key authentication with that key |
| 129 | when connecting to the FATE server. Also do not forget to check the identity |
| 130 | of the server and to accept its host key. This can usually be achieved by |
| 131 | running your SSH client manually and killing it after you accepted the key. |
| 132 | The FATE server's fingerprint is: |
| 133 | |
| 134 | @table @option |
| 135 | @item RSA |
| 136 | d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 |
| 137 | @item ECDSA |
| 138 | 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 |
| 139 | @end table |
| 140 | |
| 141 | If you have problems connecting to the FATE server, it may help to try out |
| 142 | the @command{ssh} command with one or more @option{-v} options. You should |
| 143 | get detailed output concerning your SSH configuration and the authentication |
| 144 | process. |
| 145 | |
| 146 | The only thing left is to automate the execution of the fate.sh script and |
| 147 | the synchronisation of the samples directory. |
| 148 | |
| 149 | |
| 150 | @chapter FATE makefile targets and variables |
| 151 | |
| 152 | @section Makefile targets |
| 153 | |
| 154 | @table @option |
| 155 | @item fate-rsync |
| 156 | Download/synchronize sample files to the configured samples directory. |
| 157 | |
| 158 | @item fate-list |
| 159 | Will list all fate/regression test targets. |
| 160 | |
| 161 | @item fate |
| 162 | Run the FATE test suite (requires the fate-suite dataset). |
| 163 | @end table |
| 164 | |
| 165 | @section Makefile variables |
| 166 | |
| 167 | @table @option |
| 168 | @item V |
| 169 | Verbosity level, can be set to 0, 1 or 2. |
| 170 | @itemize |
| 171 | @item 0: show just the test arguments |
| 172 | @item 1: show just the command used in the test |
| 173 | @item 2: show everything |
| 174 | @end itemize |
| 175 | |
| 176 | @item SAMPLES |
| 177 | Specify or override the path to the FATE samples at make time, it has a |
| 178 | meaning only while running the regression tests. |
| 179 | |
| 180 | @item THREADS |
| 181 | Specify how many threads to use while running regression tests, it is |
| 182 | quite useful to detect thread-related regressions. |
| 183 | |
| 184 | @item THREAD_TYPE |
| 185 | Specify which threading strategy test, either @var{slice} or @var{frame}, |
| 186 | by default @var{slice+frame} |
| 187 | |
| 188 | @item CPUFLAGS |
| 189 | Specify CPU flags. |
| 190 | |
| 191 | @item TARGET_EXEC |
| 192 | Specify or override the wrapper used to run the tests. |
| 193 | The @var{TARGET_EXEC} option provides a way to run FATE wrapped in |
| 194 | @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets |
| 195 | through @command{ssh}. |
| 196 | |
| 197 | @item GEN |
| 198 | Set to @var{1} to generate the missing or mismatched references. |
| 199 | @end table |
| 200 | |
| 201 | @section Examples |
| 202 | |
| 203 | @example |
| 204 | make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate |
| 205 | @end example |