Commit | Line | Data |
---|---|---|
2ba45a60 DM |
1 | \input texinfo @c -*- texinfo -*- |
2 | ||
3 | @settitle FFmpeg Automated Testing Environment | |
4 | @titlepage | |
5 | @center @titlefont{FFmpeg Automated Testing Environment} | |
6 | @end titlepage | |
7 | ||
8 | @node Top | |
9 | @top | |
10 | ||
11 | @contents | |
12 | ||
13 | @chapter Introduction | |
14 | ||
15 | FATE is an extended regression suite on the client-side and a means | |
16 | for results aggregation and presentation on the server-side. | |
17 | ||
18 | The first part of this document explains how you can use FATE from | |
19 | your FFmpeg source directory to test your ffmpeg binary. The second | |
20 | part describes how you can run FATE to submit the results to FFmpeg's | |
21 | FATE server. | |
22 | ||
23 | In any way you can have a look at the publicly viewable FATE results | |
24 | by visiting this website: | |
25 | ||
26 | @url{http://fate.ffmpeg.org/} | |
27 | ||
28 | This is especially recommended for all people contributing source | |
29 | code to FFmpeg, as it can be seen if some test on some platform broke | |
30 | with their recent contribution. This usually happens on the platforms | |
31 | the developers could not test on. | |
32 | ||
33 | The second part of this document describes how you can run FATE to | |
34 | submit your results to FFmpeg's FATE server. If you want to submit your | |
35 | results be sure to check that your combination of CPU, OS and compiler | |
36 | is not already listed on the above mentioned website. | |
37 | ||
38 | In the third part you can find a comprehensive listing of FATE makefile | |
39 | targets and variables. | |
40 | ||
41 | ||
42 | @chapter Using FATE from your FFmpeg source directory | |
43 | ||
44 | If you want to run FATE on your machine you need to have the samples | |
45 | in place. You can get the samples via the build target fate-rsync. | |
46 | Use this command from the top-level source directory: | |
47 | ||
48 | @example | |
49 | make fate-rsync SAMPLES=fate-suite/ | |
50 | make fate SAMPLES=fate-suite/ | |
51 | @end example | |
52 | ||
53 | The above commands set the samples location by passing a makefile | |
54 | variable via command line. It is also possible to set the samples | |
55 | location at source configuration time by invoking configure with | |
56 | `--samples=<path to the samples directory>'. Afterwards you can | |
57 | invoke the makefile targets without setting the SAMPLES makefile | |
58 | variable. This is illustrated by the following commands: | |
59 | ||
60 | @example | |
61 | ./configure --samples=fate-suite/ | |
62 | make fate-rsync | |
63 | make fate | |
64 | @end example | |
65 | ||
66 | Yet another way to tell FATE about the location of the sample | |
67 | directory is by making sure the environment variable FATE_SAMPLES | |
68 | contains the path to your samples directory. This can be achieved | |
69 | by e.g. putting that variable in your shell profile or by setting | |
70 | it in your interactive session. | |
71 | ||
72 | @example | |
73 | FATE_SAMPLES=fate-suite/ make fate | |
74 | @end example | |
75 | ||
76 | @float NOTE | |
77 | Do not put a '~' character in the samples path to indicate a home | |
78 | directory. Because of shell nuances, this will cause FATE to fail. | |
79 | @end float | |
80 | ||
81 | To use a custom wrapper to run the test, pass @option{--target-exec} to | |
82 | @command{configure} or set the @var{TARGET_EXEC} Make variable. | |
83 | ||
84 | ||
85 | @chapter Submitting the results to the FFmpeg result aggregation server | |
86 | ||
87 | To submit your results to the server you should run fate through the | |
88 | shell script @file{tests/fate.sh} from the FFmpeg sources. This script needs | |
89 | to be invoked with a configuration file as its first argument. | |
90 | ||
91 | @example | |
92 | tests/fate.sh /path/to/fate_config | |
93 | @end example | |
94 | ||
95 | A configuration file template with comments describing the individual | |
96 | configuration variables can be found at @file{doc/fate_config.sh.template}. | |
97 | ||
98 | @ifhtml | |
99 | The mentioned configuration template is also available here: | |
100 | @verbatiminclude fate_config.sh.template | |
101 | @end ifhtml | |
102 | ||
103 | Create a configuration that suits your needs, based on the configuration | |
104 | template. The `slot' configuration variable can be any string that is not | |
105 | yet used, but it is suggested that you name it adhering to the following | |
106 | pattern <arch>-<os>-<compiler>-<compiler version>. The configuration file | |
107 | itself will be sourced in a shell script, therefore all shell features may | |
108 | be used. This enables you to setup the environment as you need it for your | |
109 | build. | |
110 | ||
111 | For your first test runs the `fate_recv' variable should be empty or | |
112 | commented out. This will run everything as normal except that it will omit | |
113 | the submission of the results to the server. The following files should be | |
114 | present in $workdir as specified in the configuration file: | |
115 | ||
116 | @itemize | |
117 | @item configure.log | |
118 | @item compile.log | |
119 | @item test.log | |
120 | @item report | |
121 | @item version | |
122 | @end itemize | |
123 | ||
124 | When you have everything working properly you can create an SSH key pair | |
125 | and send the public key to the FATE server administrator who can be contacted | |
126 | at the email address @email{fate-admin@@ffmpeg.org}. | |
127 | ||
128 | Configure your SSH client to use public key authentication with that key | |
129 | when connecting to the FATE server. Also do not forget to check the identity | |
130 | of the server and to accept its host key. This can usually be achieved by | |
131 | running your SSH client manually and killing it after you accepted the key. | |
132 | The FATE server's fingerprint is: | |
133 | ||
134 | @table @option | |
135 | @item RSA | |
136 | d3:f1:83:97:a4:75:2b:a6:fb:d6:e8:aa:81:93:97:51 | |
137 | @item ECDSA | |
138 | 76:9f:68:32:04:1e:d5:d4:ec:47:3f:dc:fc:18:17:86 | |
139 | @end table | |
140 | ||
141 | If you have problems connecting to the FATE server, it may help to try out | |
142 | the @command{ssh} command with one or more @option{-v} options. You should | |
143 | get detailed output concerning your SSH configuration and the authentication | |
144 | process. | |
145 | ||
146 | The only thing left is to automate the execution of the fate.sh script and | |
147 | the synchronisation of the samples directory. | |
148 | ||
149 | ||
150 | @chapter FATE makefile targets and variables | |
151 | ||
152 | @section Makefile targets | |
153 | ||
154 | @table @option | |
155 | @item fate-rsync | |
156 | Download/synchronize sample files to the configured samples directory. | |
157 | ||
158 | @item fate-list | |
159 | Will list all fate/regression test targets. | |
160 | ||
161 | @item fate | |
162 | Run the FATE test suite (requires the fate-suite dataset). | |
163 | @end table | |
164 | ||
165 | @section Makefile variables | |
166 | ||
167 | @table @option | |
168 | @item V | |
169 | Verbosity level, can be set to 0, 1 or 2. | |
170 | @itemize | |
171 | @item 0: show just the test arguments | |
172 | @item 1: show just the command used in the test | |
173 | @item 2: show everything | |
174 | @end itemize | |
175 | ||
176 | @item SAMPLES | |
177 | Specify or override the path to the FATE samples at make time, it has a | |
178 | meaning only while running the regression tests. | |
179 | ||
180 | @item THREADS | |
181 | Specify how many threads to use while running regression tests, it is | |
182 | quite useful to detect thread-related regressions. | |
183 | ||
184 | @item THREAD_TYPE | |
185 | Specify which threading strategy test, either @var{slice} or @var{frame}, | |
186 | by default @var{slice+frame} | |
187 | ||
188 | @item CPUFLAGS | |
189 | Specify CPU flags. | |
190 | ||
191 | @item TARGET_EXEC | |
192 | Specify or override the wrapper used to run the tests. | |
193 | The @var{TARGET_EXEC} option provides a way to run FATE wrapped in | |
194 | @command{valgrind}, @command{qemu-user} or @command{wine} or on remote targets | |
195 | through @command{ssh}. | |
196 | ||
197 | @item GEN | |
198 | Set to @var{1} to generate the missing or mismatched references. | |
199 | @end table | |
200 | ||
201 | @section Examples | |
202 | ||
203 | @example | |
204 | make V=1 SAMPLES=/var/fate/samples THREADS=2 CPUFLAGS=mmx fate | |
205 | @end example |