| 1 | <?xml version="1.0" encoding="ISO-8859-1"?> |
| 2 | <!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN" |
| 3 | "http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [ |
| 4 | <!ENTITY % defs SYSTEM "/xserver/doc/xml/xserver.ent"> %defs; |
| 5 | ]> |
| 6 | <article> |
| 7 | |
| 8 | <articleinfo> |
| 9 | <!-- Title information --> |
| 10 | <title>Scaled Window Support in DMX</title> |
| 11 | <authorgroup> |
| 12 | <author><firstname>Kevin E.</firstname><surname>Martin</surname></author> |
| 13 | <author><firstname>Rickard E.</firstname><surname>Faith</surname></author> |
| 14 | </authorgroup> |
| 15 | <pubdate>15 October 2003 (created 19 September 2003)</pubdate> |
| 16 | <releaseinfo>X Server Version &xserver.version;</releaseinfo> |
| 17 | <abstract> |
| 18 | <para> |
| 19 | This document investigates the possibility of adding scaled window |
| 20 | support to the DMX X server, thereby allowing a window or some |
| 21 | selected part of the logical DMX area to be displayed using a |
| 22 | scaling factor. For example, this might allow the contents of a |
| 23 | window to be magnified for easier viewing. In particular, scaling |
| 24 | for the VNC client is explored. <emphasis remap="it">Copyright 2003 |
| 25 | by Red Hat, Inc., Raleigh, North Carolina</emphasis> |
| 26 | </para> |
| 27 | </abstract> |
| 28 | </articleinfo> |
| 29 | |
| 30 | <!-- Begin the document --> |
| 31 | <sect1><title>Introduction</title> |
| 32 | <sect2><title>DMX</title> |
| 33 | <para> |
| 34 | The DMX X server (Xdmx) is a proxy server that is designed |
| 35 | to allow X servers on multiple machines to be combined into |
| 36 | a single multi-headed X server. Combined with Xinerama, |
| 37 | these heads can appear as a single very high-resolution |
| 38 | screen. Typical applications include the creation of a |
| 39 | video wall with 16 1280x1024 displays arranged in a |
| 40 | rectangle, for a total resolution of of 5120x4096. |
| 41 | </para> |
| 42 | </sect2> |
| 43 | <sect2><title>Problem Statement</title> |
| 44 | <para> |
| 45 | Applications displayed on a physically large video wall that |
| 46 | provides high pixel-resolution may be difficult to see, |
| 47 | especially if the application is designed for use on a |
| 48 | typical desktop computer with a relatively small display |
| 49 | located close to the human operator. The goal of this paper |
| 50 | is to describe and discuss solutions to this problem. |
| 51 | </para> |
| 52 | <para> |
| 53 | The original driving problem for this work is to provide |
| 54 | scaling for the <command>vncviewer</command> application when |
| 55 | displayed using DMX (VNC scaling is currently available only |
| 56 | with the Windows client, and there is no plan to extend that |
| 57 | capability to other clients). While this specific problem |
| 58 | will be addressed in this paper, the general solution space |
| 59 | will also be explored, since this may lead to a good |
| 60 | solution not only for <command>vncviewer</command> but also for |
| 61 | other applications. |
| 62 | </para> |
| 63 | </sect2> |
| 64 | <sect2><title>Task</title> |
| 65 | <para> |
| 66 | For reference, here is the original description of the task |
| 67 | this paper addresses: |
| 68 | <itemizedlist> |
| 69 | <listitem><para>Scaled window support (for VNC) |
| 70 | <itemizedlist> |
| 71 | <listitem><para> |
| 72 | Investigate possibility of implementing a "scaled |
| 73 | window" extension: |
| 74 | <itemizedlist> |
| 75 | <listitem><para> |
| 76 | Add XCreateScaledWindow call that could be used |
| 77 | in place of XCreateWindow |
| 78 | </para></listitem> |
| 79 | <listitem><para> |
| 80 | All primitives drawn to scaled window would be |
| 81 | scaled by appropriate (integral?) scaling factor |
| 82 | </para></listitem> |
| 83 | </itemizedlist> |
| 84 | </para></listitem> |
| 85 | <listitem><para> |
| 86 | Alternate approach: special case VNC support |
| 87 | </para></listitem> |
| 88 | </itemizedlist> |
| 89 | </para></listitem> |
| 90 | </itemizedlist> |
| 91 | </para> |
| 92 | </sect2> |
| 93 | </sect1> |
| 94 | |
| 95 | <sect1><title>Previous Work</title> |
| 96 | <para> |
| 97 | This section reviews relevant previous work. |
| 98 | </para> |
| 99 | <sect2><title>VNC</title> |
| 100 | <sect3><title>Scaling under VNC</title> |
| 101 | <para> |
| 102 | When using the <command>vncviewer</command> program for Windows, it |
| 103 | is possible to specify a scaling factor (as numerator and |
| 104 | denominator). When scaling is in effect, the viewer |
| 105 | software uses StretchBlt (instead of BitBlt) to display |
| 106 | the pixels for the user. When this call is made, the |
| 107 | viewer already has received all of the pixel information |
| 108 | (at full unscaled resolution). |
| 109 | </para> |
| 110 | <para> |
| 111 | The scaling in VNC is primitive. It does not conserve |
| 112 | bandwidth, it does not treat textual information |
| 113 | differently (i.e., by using a suitably scaled font), and |
| 114 | it does not provide any anti-aliasing other than that |
| 115 | provided by the underlying (Windows-only) system library. |
| 116 | </para> |
| 117 | </sect3> |
| 118 | </sect2> |
| 119 | <sect2><title>The X Video Extension</title> |
| 120 | <para> |
| 121 | The X Video Extension is a widely-available extension to the |
| 122 | X11 protocol that provides support for streaming video. |
| 123 | Integral to this support is the ability to arbitrarily scale |
| 124 | the output. In version 2.2 of the X Video specification, |
| 125 | support for scaled still images was provided, using both |
| 126 | shared memory and traditional transport. The API for this |
| 127 | support uses calls that are quite similar to XCreateWindow, |
| 128 | XPutImage, and XShmPutImage. Currently, most of the drivers |
| 129 | implemented in XFree86 only support data in various YUV |
| 130 | formats. However, several modern video adaptors support RGB |
| 131 | as well. |
| 132 | </para> |
| 133 | <para> |
| 134 | Note, though, that the target output for this scaling is an |
| 135 | overlay plane -- so X Video provides functionality that is |
| 136 | fundamentally different from that provided by the Windows |
| 137 | StrechBlt call. |
| 138 | </para> |
| 139 | </sect2> |
| 140 | </sect1> |
| 141 | |
| 142 | <sect1><title>Possible Solutions</title> |
| 143 | <para> |
| 144 | This section briefly discusses possible solutions, including |
| 145 | major advantages and disadvantages from both the |
| 146 | implementation and the end-user programmer standpoint. |
| 147 | </para> |
| 148 | <sect2><title>VNC-like Scaling</title> |
| 149 | <sect3><title>Software Scaling</title> |
| 150 | <para> |
| 151 | The <command>vncviewer</command> application could be modified to |
| 152 | provide software scaling. This is not a general solution, |
| 153 | but it does solve one of the goals of this work. |
| 154 | </para> |
| 155 | <para> |
| 156 | A prototype of this solution was implemented and a patch |
| 157 | against <filename>vnc-3.3.7-unixsrc</filename> is available in the |
| 158 | <filename>dmx/external</filename> directory. Because of limited time |
| 159 | available for this work, all of the edge cases were not |
| 160 | considered and the solution works well mainly for integer |
| 161 | scaling. |
| 162 | </para> |
| 163 | <para> |
| 164 | Currently, <command>vncviewer</command> writes to the X display |
| 165 | with XPutImage, XCopyArea, and XFillRectangle. All |
| 166 | instances of these calls have to be aware of scaling |
| 167 | and must round correctly. In the prototype solution, |
| 168 | rounding is incorrect and can cause artifacts. |
| 169 | </para> |
| 170 | <para> |
| 171 | A better solution would be to cache all updates to the |
| 172 | desktop image in <command>vncviewer</command> and only send the |
| 173 | damaged area to the X display with XPutImage. This would |
| 174 | allow the damaged area to be computed so that rounding |
| 175 | errors do not create artifacts. This method is probably |
| 176 | similar to what is used in the Window client. (The whole |
| 177 | VNC suite is being re-written in C++ and the forthcoming |
| 178 | version 4 has not been evaluated.) |
| 179 | </para> |
| 180 | </sect3> |
| 181 | <sect3><title>Scaling with the X Video Extension</title> |
| 182 | <para> |
| 183 | The scaling in the Windows <command>vncviewer</command> application |
| 184 | makes use of a scaled blit that is supplied by the |
| 185 | underlying system library. Several video cards currently |
| 186 | provide support for a scaled blit, and some X servers |
| 187 | (including XFree86) expose this capability to applications |
| 188 | via the XvPutImage interface of the X Video Extension. |
| 189 | The capability exposed by XvPutImage results in the scaled |
| 190 | image being drawn to an overlay plane. Most video cards |
| 191 | also provide support for a scaled blit into the normal |
| 192 | output planes, but this is not exposed via XvPutImage. |
| 193 | </para> |
| 194 | <para> |
| 195 | The <command>vncviewer</command> program could be modified to use |
| 196 | the X Video Extension to provide scaling under X11 that is |
| 197 | similar to the scaling currently provided under Windows. |
| 198 | Unfortunately, Xdmx does not currently export the X Video |
| 199 | Extension, so this would not provide an immediate solution |
| 200 | usable with DMX. |
| 201 | </para> |
| 202 | <para> |
| 203 | A very early-stage proof-of-concept prototype was |
| 204 | implemented and a preliminary patch against |
| 205 | <filename>vnc-3.3.7-unixsrc</filename> is available in the |
| 206 | <filename>dmx/external</filename> directory. This prototype was |
| 207 | implemented to better understand the problems that must be |
| 208 | solved to make this solution viable: |
| 209 | <itemizedlist> |
| 210 | <listitem><para> |
| 211 | As noted under the software scaling section above, |
| 212 | <command>vncviewer</command> writes to the X display with |
| 213 | several different calls. These calls write to the |
| 214 | normal output planes and are compatible with |
| 215 | XvPutImage, which writes to an overlay plane. To |
| 216 | eliminate artifacts caused by this problem, |
| 217 | <command>vncviewer</command> should be modified so that a cached |
| 218 | copy of the desktop is available, either as a |
| 219 | client-side image or a server-side off-screen pixmap, |
| 220 | so that XvPutImage would be the only method for |
| 221 | writing to the X display. |
| 222 | </para></listitem> |
| 223 | <listitem> |
| 224 | <para> |
| 225 | Although several modern graphics adaptors support |
| 226 | hardware scaling using an RGB format (e.g., ATI |
| 227 | Radeon, nVidia, etc.), XFree86 drivers typically |
| 228 | only implement YUV formats. YUV generally compress |
| 229 | the pixel information in some way. For example, two |
| 230 | commonly implemented formats, YUY2 and UYVY provide |
| 231 | intensity information for every RGB pixel, but only |
| 232 | provide chroma and luminance information for pairs |
| 233 | of horizontal pixels. Since VNC uses |
| 234 | pixel-resolution for communicating updates on the |
| 235 | wire, additional artifacts are introduced (because |
| 236 | there may not be enough information from the wire to |
| 237 | update a pair of pixels). |
| 238 | </para> |
| 239 | <para> |
| 240 | Further, the well-known problem with YUV encoding |
| 241 | is even more evident when the image is a desktop |
| 242 | instead of a movie. For example, consider a |
| 243 | 1-pixel-wide vertical window border. If the border |
| 244 | changes in color but not intensity (e.g., because a |
| 245 | window manager uses color to indicate focus), there |
| 246 | may or may not be a change in the YUY2 image, |
| 247 | depending on the algorithm used for RGB to YUV |
| 248 | conversion and on how the border pixel is ordered in |
| 249 | the pair of pixels used by the algorithm. |
| 250 | </para> |
| 251 | <para> |
| 252 | Many of these artifacts could be eliminated if |
| 253 | <command>vncviewer</command> cached a complete RGB image of |
| 254 | the desktop, and only did the conversion to YUV for |
| 255 | properly aligned areas of damage. The remaining artifacts |
| 256 | could be eliminated if an RGB format was used with X |
| 257 | Video (which may require the extension of existing |
| 258 | XFree86 drivers to support RGB). |
| 259 | </para> |
| 260 | </listitem> |
| 261 | <listitem><para> |
| 262 | Most modern video cards support exactly one overlay |
| 263 | plane that is suitable for use with X Video. |
| 264 | Therefore, only one application can use X Video at any |
| 265 | given time. This is a severe limitation in a desktop |
| 266 | environment. |
| 267 | </para></listitem> |
| 268 | </itemizedlist> |
| 269 | </para> |
| 270 | <sect4><title>Implementing the X Video Extension for DMX</title> |
| 271 | <para> |
| 272 | The user-level API for X Video is fairly simple, but the |
| 273 | underlying support required for the full specification |
| 274 | is large. However, since the API provides a method to |
| 275 | query supported capabilities, a usable subset of X |
| 276 | Video can be implemented that would support XvPutImage |
| 277 | and little else. This would require support for the |
| 278 | following: |
| 279 | <itemizedlist> |
| 280 | <listitem><para> |
| 281 | X Video Extension API calls, including the |
| 282 | following: |
| 283 | <itemizedlist> |
| 284 | <listitem><para>XvQueryExtension</para></listitem> |
| 285 | <listitem><para>XvQueryAdaptors</para></listitem> |
| 286 | <listitem><para>XvQueryPortAttributes</para></listitem> |
| 287 | <listitem><para>XvFreeAdaptorInfo</para></listitem> |
| 288 | <listitem><para>XvListImageFormats</para></listitem> |
| 289 | <listitem><para>XvGrabPort</para></listitem> |
| 290 | <listitem><para>XvCreateImage</para></listitem> |
| 291 | <listitem><para>XvPutImage</para></listitem> |
| 292 | <listitem><para>XvShmCreateImage</para></listitem> |
| 293 | <listitem><para>XvShmPutImage</para></listitem> |
| 294 | </itemizedlist> |
| 295 | </para></listitem> |
| 296 | <listitem><para> |
| 297 | Support for querying back-end X Video Extension |
| 298 | capabilities. |
| 299 | </para></listitem> |
| 300 | <listitem><para> |
| 301 | Support for sending the image to the back-ends. |
| 302 | Because X Video requires sending full images, there |
| 303 | may be a trade-off between bandwidth limitations and |
| 304 | additional complexity to divide the image up such |
| 305 | that is scales properly. |
| 306 | </para></listitem> |
| 307 | <listitem><para> |
| 308 | Possible support for a software fall-back. For |
| 309 | example, if all of the back-ends do not support the X |
| 310 | Video Extension, software scaling can be implemented |
| 311 | such that the image is sent to the back-end with |
| 312 | XPutImage. This pathway would have poor |
| 313 | performance. |
| 314 | </para></listitem> |
| 315 | </itemizedlist> |
| 316 | </para> |
| 317 | </sect4> |
| 318 | <sect4><title>Supporting RGB formats for the X Video Extension</title> |
| 319 | <para> |
| 320 | Assuming an XFree86 driver already supports the X Video |
| 321 | Extension, and assuming the target hardware supports an |
| 322 | RGB format, then adding support for that format is |
| 323 | relatively simple and straightforward. |
| 324 | </para> |
| 325 | </sect4> |
| 326 | </sect3> |
| 327 | <sect3><title>Scaling with an XPutImageScaled Extension</title> |
| 328 | <para> |
| 329 | Instead of (or in addition to) implementing the X Video |
| 330 | Extension in DMX, one obvious solution would be to |
| 331 | implement a new extension that provides access to |
| 332 | hardware-assisted scaled blits, similar to the StretchBlt |
| 333 | call available under Windows. This call would scale RGB |
| 334 | images and would not use the overlay plane (unlike the X |
| 335 | Video Extension). |
| 336 | </para> |
| 337 | <para> |
| 338 | This approach has many of the same advantages and |
| 339 | disadvantages as the XCopyAreaScaled Extension, discussed |
| 340 | in the next section. Discussion of XPutImageScaled is |
| 341 | deferred in favor of XCopyAreaScaled for the following |
| 342 | reasons: |
| 343 | <itemizedlist> |
| 344 | <listitem><para> |
| 345 | XPutImageScaled can be emulated with XCopyAreaScaled |
| 346 | by first using XPutImage to copy the image to an |
| 347 | off-screen pixmap, and then calling XCopyAreaScaled |
| 348 | between that off-screen pixmap and the target |
| 349 | drawable. |
| 350 | </para></listitem> |
| 351 | <listitem><para> |
| 352 | Since XCopyAreaScaled would copy between two areas of |
| 353 | on-screen or off-screen memory, it has additional uses |
| 354 | and can be viewed as efficiently providing a superset |
| 355 | of XPutImageScaled functionality. |
| 356 | </para></listitem> |
| 357 | </itemizedlist> |
| 358 | </para> |
| 359 | </sect3> |
| 360 | <sect3><title>Scaling with an XCopyAreaScaled Extension</title> |
| 361 | <para> |
| 362 | As noted in the previous section, because XCopyAreaScaled |
| 363 | provides a superset of the functionality provided by |
| 364 | XPutImageScaled, we will consider this extension instead. |
| 365 | </para> |
| 366 | <para> |
| 367 | First, XCopyAreaScaled would provide for RGB scaling |
| 368 | between pixmaps (i.e., on-screen or off-screen areas of |
| 369 | memory that reside on the video card). Unlike the X Video |
| 370 | Extension, which writes into an overlay plane, |
| 371 | XCopyAreaScaled would write into the non-overlay areas of |
| 372 | the screen. Key points to consider are as follows: |
| 373 | <itemizedlist> |
| 374 | <listitem><para> |
| 375 | Because different planes are involved, the two scaling |
| 376 | operations are usually implemented in hardware |
| 377 | differently, so an XCopyAreaScaled extension could be |
| 378 | added in a manner that would neither conflict with nor |
| 379 | interact with the X Video extension in any way. |
| 380 | </para></listitem> |
| 381 | <listitem><para> |
| 382 | The XCopyAreaScaled extension provides new |
| 383 | functionality that the X Video Extension does not |
| 384 | provide. Based on anecdotal feedback, we believe that |
| 385 | many people outside the DMX and VNC communities would |
| 386 | be excited about this extension. |
| 387 | </para></listitem> |
| 388 | <listitem><para> |
| 389 | The main drawback to this extension is that it is new |
| 390 | and needs to be implemented at the driver level in |
| 391 | XFree86 for each video card to be supported. At the |
| 392 | present time, it is more likely that the X Video |
| 393 | Extension will be implemented for a particular piece |
| 394 | hardware because the X Video extension has multimedia |
| 395 | uses. However, over time, we would expect the |
| 396 | XCopyAreaScaled extension to be implemented along with |
| 397 | the X Video extension, especially if it becomes |
| 398 | popular. |
| 399 | </para></listitem> |
| 400 | <listitem><para> |
| 401 | Another drawback is that not all modern cards provide |
| 402 | support for a simple scaled blit operation. However, |
| 403 | these cards usually do provide a 3D pipeline which |
| 404 | could be used to provide this functionality in a |
| 405 | manner that is transparent to the client application |
| 406 | that is using the XCopyAreaScaled extension. However, |
| 407 | this implementation pathway would make this extension |
| 408 | somewhat more difficult to implement on certain cards. |
| 409 | </para></listitem> |
| 410 | </itemizedlist> |
| 411 | </para> |
| 412 | </sect3> |
| 413 | <sect3><title>Scaling with OpenGL</title> |
| 414 | <para> |
| 415 | Another general solution to the scaling problem is to use |
| 416 | the texture scaling found in all 3D hardware. This |
| 417 | ability is already exposed through OpenGL and can be |
| 418 | exploited by clients without X server modification (i.e., |
| 419 | other than the ability to support OpenGL). An application |
| 420 | using OpenGL would transmit the non-scaled image to the X |
| 421 | server as a texture, and would then display a single |
| 422 | non-transformed rect using that texture. This also works |
| 423 | around the single overlay problem with the X Video |
| 424 | Extension as well as the need to implement additional |
| 425 | scaled primitive extensions. |
| 426 | </para> |
| 427 | <para> |
| 428 | The downside is that most OpenGL implementations require |
| 429 | power of 2 texture sizes and this can be very wasteful of |
| 430 | memory if, for example, the application needs to scale a |
| 431 | 1025x1025 image, which would require a 2048x2048 texture |
| 432 | area (even a 640x480 image would require a 1024x512 |
| 433 | texture). Another downside is that some OpenGL |
| 434 | implementations have a limited about of texture memory and |
| 435 | cannot handle textures that are very large. For example, |
| 436 | they might limit the texture size to 1024x1024. |
| 437 | </para> |
| 438 | </sect3> |
| 439 | </sect2> |
| 440 | <sect2><title>Application-transparent Scaling for DMX |
| 441 | </title><sect3><title>Back-end Scaling Without Disconnect/Reconnect</title> |
| 442 | <para> |
| 443 | VNC does scaling on the client side (in the |
| 444 | <command>vncviewer</command> application). Implementing a similar |
| 445 | solution for DMX would require support in the back-end X |
| 446 | servers and, therefore, is not a general solution. |
| 447 | </para> |
| 448 | <para> |
| 449 | XFree86 already implements some support for "scaling" that |
| 450 | could be used with DMX: if, in the XF86Config file, |
| 451 | multiple Modes are listed in the Display Subsection of the |
| 452 | Screen Section, then pressing Ctrl-Alt-Plus and |
| 453 | Ctrl-Alt-Minus can be used to iterate through the listed |
| 454 | modes. The display dimensions will change to the |
| 455 | dimensions in the Modes line, but the logical dimensions |
| 456 | of the X server (i.e., the dimensions that Xdmx knows |
| 457 | about) will not change. |
| 458 | </para> |
| 459 | <para> |
| 460 | Further, the dimensions of the XFree86 display are under |
| 461 | software control (via the XFree86-VidModeExtension), so |
| 462 | the Xdmx server could change the screen dimensions on a |
| 463 | per-display basis, thereby scaling the information on part |
| 464 | of that display. |
| 465 | </para> |
| 466 | <para> |
| 467 | However, this scaling appears to have limited use. For |
| 468 | example, assume a 4 by 4 display wall consisting of 16 |
| 469 | 1280x1024 displays. If all of the back-end servers were |
| 470 | simultaneously configured to display 640x480, the left |
| 471 | hand corner of each display would be magnified, but the |
| 472 | composite result would be unreadable. Magnifying one |
| 473 | display at a time could be usable, but could have limited |
| 474 | utility, since the result would still be no larger than a |
| 475 | single display. |
| 476 | </para> |
| 477 | </sect3> |
| 478 | <sect3><title>Back-end Scaling With Disconnect/Reconnect</title> |
| 479 | <para> |
| 480 | Disconnect and reconnect features are not currently |
| 481 | supported in DMX, but are scheduled to be implemented in |
| 482 | the future. These features, combined with the |
| 483 | XFree86-VidModeExtension Extension, would allow an |
| 484 | application to do the following: |
| 485 | <itemizedlist> |
| 486 | <listitem><para> |
| 487 | Disconnect a specific back-end server (via the DMX |
| 488 | Extension), |
| 489 | </para></listitem> |
| 490 | <listitem><para> |
| 491 | reconfigure the XFree86 back-end server resolution, |
| 492 | and |
| 493 | </para></listitem> |
| 494 | <listitem><para> |
| 495 | reconnect the back-end server to DMX -- at a new |
| 496 | origin with the new screen resolution. |
| 497 | </para></listitem> |
| 498 | </itemizedlist> |
| 499 | </para> |
| 500 | <para> |
| 501 | For example, consider a display wall consisting of 16 |
| 502 | 1280x1024 displays with a total resolution of 5120x4096. |
| 503 | All of the screens could be disconnected, repositioned, |
| 504 | and reconnected each at a resolution of 640x480. The |
| 505 | total resolution of the display wall would be 2560x1920, |
| 506 | allowing a view of a selected area approximately |
| 507 | one-fourth of the size of the DMX display. This change |
| 508 | would be completely application independent (except, |
| 509 | perhaps, for a DMX-aware window manager). When work at |
| 510 | the increased resolution was completed, the back-end |
| 511 | servers could be disconnected, reconfigured, and |
| 512 | reconnected for the original 5120x4096 view. |
| 513 | </para> |
| 514 | <para> |
| 515 | Support for this type of scaling can be implemented in a |
| 516 | DMX-aware X11 client assuming the DMX server support |
| 517 | arbitrary disconnect and reconnect semantics. Because |
| 518 | this application cannot be written before |
| 519 | disconnect/reconnect is implemented, this solution will |
| 520 | not be discussed further in this paper. |
| 521 | </para> |
| 522 | </sect3> |
| 523 | <sect3><title>Server-side Scaling</title> |
| 524 | <para> |
| 525 | In earlier versions of DMX, a frame buffer was maintained |
| 526 | on the server side, and XPutImage was used to move the |
| 527 | information from the server to the client (similar to some |
| 528 | early VNC implementations). The use of a server-side |
| 529 | frame buffer would allow the server to do scaling, but is |
| 530 | not a recommended solution because of overall performance |
| 531 | issues and server-side memory issues (i.e., the frame |
| 532 | buffer would be very large for large display walls). |
| 533 | </para> |
| 534 | <para> |
| 535 | Exploration of this path is not recommended. |
| 536 | </para> |
| 537 | </sect3> |
| 538 | </sect2> |
| 539 | <sect2><title>XCreateScaledWindow API</title> |
| 540 | <para> |
| 541 | The implementation of X Video Extension in DMX, and the use |
| 542 | of XvPutImage by applications requiring scaling requires |
| 543 | significant changes in DMX Further, XvPutImage is, |
| 544 | essentially a scaled blit, and it is only useful for |
| 545 | applications which are already using (or can be modified to |
| 546 | use) XPutImage. Therefore, a more general API will be |
| 547 | discussed as another possibility. |
| 548 | </para> |
| 549 | <para> |
| 550 | X applications typically create windows with the |
| 551 | XCreateWindow call. A new extension could provide an |
| 552 | XCreateScaledWindow call that could be used in place of the |
| 553 | XCreateWindow call and be otherwise transparent to the |
| 554 | application. This would allow applications, even those that |
| 555 | do not depend on XPutImage, to take advantage of window |
| 556 | scaling. In this section we describe how the call would |
| 557 | work, what transparency it provides, and how to solve the |
| 558 | potential problems that transparency creates. |
| 559 | </para> |
| 560 | <sect3><title>XCreateWindow</title> |
| 561 | <para> |
| 562 | The XCreateWindow call takes width and height as |
| 563 | parameters. An XCreateScaledWindow call could take all |
| 564 | the same parameters, with the addition of a scaling factor. |
| 565 | </para> |
| 566 | </sect3> |
| 567 | <sect3><title>XSetWindowAttributes</title> |
| 568 | <para> |
| 569 | An X11 window has several attributes that would have to be |
| 570 | scaled: |
| 571 | <itemizedlist> |
| 572 | <listitem><para>Background and border pixmaps</para></listitem> |
| 573 | <listitem><para>Border width</para></listitem> |
| 574 | <listitem><para>Cursor</para></listitem> |
| 575 | </itemizedlist> |
| 576 | </para> |
| 577 | </sect3> |
| 578 | <sect3><title>XGetWindowAttributes, XGetGeometry</title> |
| 579 | <para> |
| 580 | For transparency, calls that query the window attributes |
| 581 | should return unscaled information. This suggests that |
| 582 | all unscaled pixmaps and window attributes should be |
| 583 | cached. |
| 584 | </para> |
| 585 | <para> |
| 586 | Unfortunately, a window manager requires the scaled |
| 587 | geometry to properly decorate the window. The X server |
| 588 | can probably determine which client is acting as the |
| 589 | window manager (e.g., because that client will select |
| 590 | events that are used exclusively by the window manager). |
| 591 | However, other Scaled Window Extension aware clients may |
| 592 | also need to determine the scaled geometry. Therefore, at |
| 593 | least two additional extension calls should be |
| 594 | implemented: XGetScaledWindowAttributes and |
| 595 | XGetScaledGeometry. |
| 596 | </para> |
| 597 | </sect3> |
| 598 | <sect3><title>Popup and Child window positions</title> |
| 599 | <para> |
| 600 | Some applications may position popup and child windows |
| 601 | based on an unscaled notion of the main window geometry. |
| 602 | In this case, additional modifications to the client would |
| 603 | be required. |
| 604 | </para> |
| 605 | </sect3> |
| 606 | <sect3><title>Events</title> |
| 607 | <para> |
| 608 | Most events (e.g., for mouse motion) return information |
| 609 | about the coordinates at which the even occurred. These |
| 610 | coordinates would have to be modified so that unscaled |
| 611 | values were presented to the client. |
| 612 | </para> |
| 613 | </sect3> |
| 614 | <sect3><title>Implementation</title> |
| 615 | <para> |
| 616 | There are many implementation issues, some of which are |
| 617 | similar to the issues involved in implementing the X Video |
| 618 | Extension for DMX. The window contents must be scaled, |
| 619 | either by performing all operations to a frame buffer and |
| 620 | then writing the image to the display (perhaps using |
| 621 | hardware scaling support), or by modifying all of the |
| 622 | various drawing operations to perform scaling. Because of |
| 623 | the complexity involved, the frame buffer option is |
| 624 | recommended. |
| 625 | </para> |
| 626 | </sect3> |
| 627 | </sect2> |
| 628 | </sect1> |
| 629 | |
| 630 | <sect1><title>Conclusion and Recommendations |
| 631 | </title><para> |
| 632 | We recommend a three phase implementation strategy, based on |
| 633 | how an application could be written to take advantage of |
| 634 | scaling: |
| 635 | <orderedlist> |
| 636 | <listitem> |
| 637 | <para> |
| 638 | The XCopyAreaScaled extension should be implemented, since |
| 639 | this is the ideal solution for applications like VNC, and |
| 640 | since making use of this extension will require minimal |
| 641 | changes to applications that already use XPutImage or |
| 642 | XCopyArea. |
| 643 | </para> |
| 644 | <para> |
| 645 | The initial implementation work would include the design |
| 646 | of the X protocol extension, writing this up in the |
| 647 | usual format for extension documentation, implementation |
| 648 | of the protocol transport pieces in XFree86, |
| 649 | implementation of a software fall-back in XFree86 and |
| 650 | DMX, one example hardware implementation for XFree86, |
| 651 | and implementation of support for this extension in DMX. |
| 652 | </para> |
| 653 | <para> |
| 654 | We suggest implementing the extension first on the ATI |
| 655 | Radeon cards. However, since these cards do not provide |
| 656 | a 2D scaled blit primitive, the implementation would |
| 657 | have to make use of the 3D texture engine to emulate a |
| 658 | scaled blit. This is recommended, since other modern |
| 659 | graphics cards also do not provide a simple 2D scaled |
| 660 | blit operation and an example of the more difficult |
| 661 | implementation pathway would be helpful to others. |
| 662 | </para> |
| 663 | </listitem> |
| 664 | <listitem> |
| 665 | <para> |
| 666 | Until XCopyAreaScaled is widely supported, applications |
| 667 | that require scaling will have to fall back to another |
| 668 | scaling method. We suggest OpenGL as the first fall-back |
| 669 | method because it is widely available and supported by |
| 670 | DMX. |
| 671 | </para> |
| 672 | <para> |
| 673 | A project centered around OpenGL-based scaling would |
| 674 | implement this scaling in VNC as an example. This work |
| 675 | would include re-writing the <command>vncviewer</command> |
| 676 | rendering engine to cache a master copy of the desktop |
| 677 | image for all operations. |
| 678 | </para> |
| 679 | </listitem> |
| 680 | <listitem> |
| 681 | <para> |
| 682 | Since OpenGL is not implemented everywhere, and may not |
| 683 | provide hardware-assisted performance in every |
| 684 | implementation, an application that requires scaling |
| 685 | should also fall back to using the X Video Extension. |
| 686 | </para> |
| 687 | <para> |
| 688 | This project would add support for the X Video Extension |
| 689 | to DMX and would add support to VNC to take advantage of |
| 690 | this extension without introducing artifacts. This |
| 691 | would require modifying the <command>vncviewer</command> rendering |
| 692 | engine to cache a master copy of the desktop image for |
| 693 | all operations. This project should also add support |
| 694 | for the RGB format to at least one XFree86 driver (e.g., |
| 695 | ATI Radeon). |
| 696 | </para> |
| 697 | <para> |
| 698 | The X Video Extension is one of the few popular |
| 699 | extensions that DMX does not support. We recommend |
| 700 | implementing the X Video Extension even if scaling is |
| 701 | the specific goal of that work. |
| 702 | </para> |
| 703 | </listitem> |
| 704 | </orderedlist> |
| 705 | </para> |
| 706 | <para> |
| 707 | We do <emphasis>not</emphasis> recommend implementation of the |
| 708 | XCreateScaledWindow extension because of the complexity |
| 709 | involved. We do <emphasis>not</emphasis> recommend implementation of the |
| 710 | XPutImageScaled extension because it requires the same amount |
| 711 | of work as the XCopyAreaScaled extension, but provides less |
| 712 | functionality. Further, server-side scaling with a large |
| 713 | frame buffer is <emphasis>not</emphasis> recommended because of the |
| 714 | performance implications. |
| 715 | </para> |
| 716 | <para> |
| 717 | The back-end scaling, especially with disconnect/reconnect |
| 718 | support should be explored in the future after |
| 719 | disconnect/reconnect is implemented, but not at the present |
| 720 | time. |
| 721 | </para> |
| 722 | </sect1> |
| 723 | |
| 724 | </article> |
| 725 | <!-- Local Variables: --> |
| 726 | <!-- fill-column: 72 --> |
| 727 | <!-- End: --> |