Advanced Computer Graphics Techniques for Accurately Reconstructing 3-D Flow
Visualizations from Multiple Wind Tunnel Images and Measurements
Daniel F. Keefe (Student)
Kurt Severance (Mentor)
Internal Operations Group
Information Systems and Services Division
Data Analysis and Imaging Branch
Abstract
Software which is capable of accurately projecting multiple two-dimensional images onto a three-dimensional surface has been developed. This software, developed in Langley Research Center’s Data Visualization and Animation Lab, supports fusion of high-resolution images from multiple cameras, correlation of these images with three-dimensional color-mapped data, and output of the results in standard Virtual Reality Modeling Language format. This paper describes the image projection software in detail. The reverse projection scheme it uses is explained and contrasted with the forward projection technique implemented in the previous version of the software. The multiple camera assignment scheme is also presented and described in detail. The software was used to visualize flow characteristics and the pressure signature on a four-wheel landing gear which was tested in Langley Research Center’s Basic Aerodynamics Research Tunnel. The results of this visualization are presented, and the capability to combine images from multiple cameras on any three-dimensional surface is demonstrated.
Introduction
The ability to visualize three-dimensional (3-D) characteristics of a surface is essential in the field of aerodynamics. Data on air flow and pressure around surfaces are collected regularly by aeronautical researchers. At Langley Research Center, a common technique used to collect air flow data is to take in-flight and wind tunnel photographs. These images often show oil and paint mixtures streaking across a surface or tufts of fabric blowing in the direction of air flow at different points on a surface. Two-dimensional (2-D) air flow images can be difficult to interpret. For example, images taken during in-flight experiments often suffer from large perspective distortions and unconventional camera angles because of the practical limitations imposed on the experiment. These factors can make gathering air flow information directly from experimental photographs difficult for researchers. Image projection software accounts for perspective distortions and produces a 3-D model which enables researchers to inspect their data from any vantage point. The software has proven to be an invaluable tool for visualizing air flow and other surface characteristics at Langley Research Center.
The purpose of this paper is to describe advances in the image projection software available in the Data Visualization and Animation Lab (DVAL). The main objective in improving the previous version of this software was to provide support for projecting multiple images onto one surface. Usually, several different cameras and camera angles are used when collecting experimental data. The ability to intelligently fuse together portions of images taken from multiple cameras to produce a composite image of an entire 3-D surface is a major enhancement to the image projection software. Other objectives in improving this software were to support the extremely high-resolution images which are now available from digital cameras, to combine image projection data with color mapped function data, and to increase the speed of the software. This paper describes the new reverse projection scheme used in the software and contrasts it with the previous approach. The camera assignment scheme is also explained in detail. In addition, the results of a visualization of air flow and pressure signature on a four-wheel landing gear are presented.
Symbols
AH - camera horizontal aperture
AV - camera vertical aperture
c - effective focal length of the camera lens
M - 3 by 3 rotation matrix
MT - transpose of a 3 by 3 rotation matrix
u,v - texture coordinates
x,y,z - image space coordinates
X,Y,Z - object space coordinates
w , F , k - angles for camera orientation, deg
Subscripts:
C - camera lens center
I - unrefined image coordinates which lie in the z = -c plane
R - image coordinates refined to account for nonlinear lens
distortions
Abbreviations:
2-D - two-dimensional
3-D - three-dimensional
BART - Basic Aerodynamics Research Tunnel
DVAL - Data Visualization and Animation Laboratory
SGI - Silicon Graphics, Incorporated
VRML - Virtual Reality Modeling Language
Image Projection Software
There are several ways to project a 2-D image onto a 3-D surface. The most recently created software in DVAL uses a projection technique which is the inverse of the previously used method. This new, reverse image projection technique simplifies the multiple image projection algorithm and allows the software to take advantage of new advances in computer texture mapping hardware and software. Since the concepts and coordinate system transformations in both projection schemes are extremely similar, it is worthwhile to briefly describe the forward projection technique used in the previous version of image projection software developed at DVAL.
Original Image Projection Scheme
In a forward image projection scheme, each pixel in the image is first refined to account for nonlinear lens distortions. Figure 1c illustrates an image before and after this refinement. The refinement takes place in the image coordinate system. In this coordinate system, the camera lens center is located at the origin, and the image is located below the xy-plane in the plane z = - c, where c is the effective focal length of the lens that locates the image plane. The x- and y-axes exit to the right and top of the camera lens respectively. The image coordinate system is illustrated in figure 1a.
In order to correctly project an image onto a 3-D surface, the image must be in the same coordinate system as the surface. A coordinate system transformation is therefore applied to the coordinates of each image pixel. This transformation has the effect of moving the image coordinate system to the point (XC, YC, ZC) in object coordinates. This point is the camera lens center. The orientation of the image coordinate system is defined by the rotation angles w , F , and k . The orientation transformation is applied through the transpose of the 3 by 3 rotation matrix M = (mij). M is defined as
(1)
The rotation angles are defined as positive for counterclockwise rotation when observed from the positive end of an axis looking toward the origin. The complete coordinate transformation from the image to the object system is therefore given by
(2)
Note that the matrix used in equation (2) is MT, the transpose of the rotation matrix M defined in equation (1). The object coordinate system is illustrated in Figure 1b.
Once the coordinates of each image pixel have been transformed into object coordinates, the principles of photogrammetry are applied. The collinearity condition of photogrammetry requires that an object, its image, and the camera lens center lie on a line. To determine where on the surface geometry a particular image pixel lies, the software calculates the equation of the line originating at the camera lens center and passing through the image pixel’s 3-D coordinates. Each polygon in the surface geometry is then tested to see whether this line intersects it. If it does intersect a polygon, then the exact point of intersection on the surface is calculated and stored along with the color of the image pixel. This process is repeated for each pixel in the image. When the resulting points are plotted and each point is drawn according to the color of its corresponding image pixel, the image appears projected onto the surface geometry. Figure 2 is a visualization of one of the projection lines. Note that one of these lines is drawn for every pixel in the image.
The forward projection scheme described up to this point is reliable and accurate, but it has some limitations. Figure 3 is an image of a mesh created by a forward projection scheme. Each vertex of the small, irregular polygons on the right of the image is the projection of one pixel in the original image. Notice the large number of vertices required to project an entire image. This requirement makes this version of the software impractical to use with the extremely high-resolution images available from today’s digital cameras. The software is also limited by the irregularity of the mesh created by projecting each image pixel. If images from two cameras at different locations were projected onto the same surface using this scheme, the two resulting meshes would have nothing in common. Each would be irregular in its own way, since this irregularity is a function of camera location and orientation with respect to the surface. Thus, the two meshes would be nearly impossible to combine to form a composite mesh and image over the surface. In addition to these limitations, the fact that the calculated line for each pixel in the image must be checked for an intersection with each polygon in the surface geometry, makes a forward projection scheme extremely slow. This can become a problem when very complex surface models are used.
Advantages to Using a Reverse Image Projection Scheme
A reverse projection scheme starts with a surface geometry and projects each vertex of this geometry back through the image plane and to the camera lens center. Figure 4 illustrates the conceptual difference between reverse and forward projection schemes. Note the direction of projection highlighted in the figure for each scheme. In a reverse projection only the vertices of the surface geometry are associated with an image pixel. No secondary, irregular mesh is created. This is in sharp contrast to a forward projection scheme in which every image pixel is projected onto a part of the surface and these projected pixels form an irregular mesh of points on the object surface. Since no irregular mesh is created in a reverse projection scheme, the task of combining multiple images on a single surface is simplified. This is the most important advantage to using a reverse projection scheme.
Another advantage to using a reverse over a forward projection scheme is that the final image resolution can be extremely high. Reverse projection schemes have been implemented in the past. However, a major criticism of these implementations has actually been that the final image resolution is extremely poor. This criticism is accurate. In previous implementations of reverse image projection, there is no image data within each surface polygon. The only image data available on the surface itself is at the polygon vertices. The relatively recent introduction of texture mapping hardware and software has provided a solution to this problem. A reverse projection scheme is now able to project polygon vertices onto an image plane and then call on texture mapping hardware or software to fill in the rest of the polygon with the appropriate part of the image. Using this method, image resolution is a function of a machine’s texture mapping capability, not a function of the number of image pixels projected onto a surface. Thus, extremely high resolution images can be mapped onto surfaces by using relatively few projection points.
Steps in Reverse Image Projection
The reverse image projection procedure is the inverse of the forward image projection procedure. Recall that in the forward image projection scheme, the locations of image pixels were first refined to account for nonlinear lens distortions and then transformed into object coordinates. The pixels were then projected along a line to a point on the 3-D surface. The reverse projection scheme implements the inverse of this process. The first step in the reverse projection scheme is to transform the coordinates of the vertices of the 3-D surface into the image coordinate system.
The object to image coordinate transformation is the inverse of the image to object coordinate system transformation described above. First, the coordinates of each 3-D surface vertex are translated. This has the effect of moving the origin of the object coordinate system to the camera lens center (XC, YC, ZC). Then, a rotation is applied. This rotation is the inverse of the rotation applied in the forward projection scheme. It has the effect of aligning the object coordinate system axes with the image coordinate system axes. When this transformation is complete, the surface vertices are defined with respect to the image coordinate system. The rotation matrix used in the object to image system transformation is the rotation matrix M, defined in equation (1). The complete object to image coordinate system transformation is, therefore,
(3)
Note that the transformation in equation (3) uses M as its rotation matrix while the transformation in equation (2) uses MT. The transpose of the matrix M is needed in the first coordinate system transformation because the image to object system transformation is the inverse of the object to image system transformation, and the inverse of a rotation matrix is the transpose of that matrix.
Once each surface vertex is transformed to a point (xV, yV, zV) in image coordinates, a line is drawn from each point to the camera center. In the image coordinate system, the camera center is located at the origin. The parametric equations for a line from the point (xV, yV, zV) to the origin are
(4)
Recall that, in the image coordinate system, the image is located in the plane z = -c. The point of intersection of the line defined by the equations above with this plane reveals, in image coordinates, the refined point in the image, (xR, yR, -c), to be mapped onto the 3-D surface vertex. Since z = -c at the point of intersection, xR and yR can be determined using the following two equations:
(5)
Once the point (xR, yR, -c) is found, the unrefined coordinates xI and yI are determined by inverting the camera lens nonlinear distortion equations.
Camera lens distortions are modeled using high-order nonlinear equations. It is necessary to model these distortions because the refined image points are calculated assuming perfectly straight projection lines. In real cameras, projection lines actually curve slightly because they pass through a circular camera lens. The effect that nonlinear lens distortions have on an image can be seen in figure 1c. The highlighted, unrefined point (xI, yI) in the figure is associated with the refined point (xR, yR). For a reverse projection technique, the inverse of each of the nonlinear lens distortion equations is required because an unrefined point must be calculated from a refined point. Since these equations are highly nonlinear, a closed-form solution for the inverse equations could not be determined. An iterative solution technique is, therefore, employed in the image projection software. A modified Newton-Raphson method for solving nonlinear systems is applied to the lens distortion equations. For each surface vertex (xR, yR, -c), an inverse Jacobian matrix with estimated partial derivatives is used to quickly converge on an unrefined point (xI, yI, -c).
The last step in associating each 3-D surface vertex with a 2-D image point is a transformation of the unrefined x- and y-coordinates of each surface vertex (xI, yI) to the texture coordinates (u,v). The texture coordinate system is illustrated in figure 1d. Texture coordinates are 2-D coordinates which range from 0.0 to 1.0. As seen in figure 1d, (0.0, 0.0) represents the lower-left corner of an image in texture coordinates, and (1.0, 1.0) represents the upper-right corner. An image is texture mapped by using texture coordinates to select a portion of it and stretching or shrinking this portion of the image to fit on a particular shape. Figures 5 and 6 illustrate the texture mapping process. Figure 5 shows an image, taken in a wind tunnel, of air flow data on the surface of a tire. The region outlined in red in the figure is defined by a list of four texture coordinates. This region has been selected by the image projection software to be mapped onto the polygon highlighted in figure 6a. Figure 6b shows the same polygon after the texture image has been applied to it. Texture mapping immensely increases the image resolution possible in a reverse image projection scheme. Without it, there would be no image data within the polygon in figure 6b. Only the vertices of the polygon would be colored according to the data in the image. A great deal of the experimental image is lost, in a reverse image projection scheme, if image texture mapping is not used.
The texture coordinates to use for a particular surface polygon are derived through a simple coordinate system transformation. The unrefined image coordinates (xI, yI) which lie within the image plane are transformed so that their texture coordinates fall within the range 0.0 to 1.0. Recall that the image is located only within this range in texture coordinates. The following equations are used to perform this transformation:
(6)
These equations produce texture coordinates which lie outside of the 0.0 to 1.0 range when the unrefined image coordinates (xI, yI) fall outside of the image plane. This situation indicates that the point on the 3-D surface specified by the vertex is not in the image taken by the camera. In this case, the image is not mapped onto the polygon in question.
The reverse projection process is repeated until all of the 3-D surface vertices have a set of texture coordinates associated with them for each camera image input into the software. Then the process of determining which image has the best data on a particular section of the 3-D surface begins.
Camera Assignment
A camera is assigned to each polygon in the 3-D surface geometry. This assignment is made according to a set of rules which determine which camera has the best view of the polygon in question. By assigning different cameras to different surface polygons, the image projection software meets its main objective of providing support for projecting multiple images onto a single 3-D surface. The procedure for determining which camera has the best view of a surface polygon is illustrated in figure 7. A unit normal vector to the polygon surface is first calculated. This vector is represented by the red arrow in figure 7. A unit vector which originates at the center of the polygon and points toward the lens center is then calculated for each camera looking at the 3-D surface. In figure 7, the green arrow depicts the unit vector pointing toward camera one. Likewise, the blue arrow depicts the unit vector pointing toward camera two. A separate dot product is then calculated for each of these vectors with the unit normal vector of the polygon. The value of each dot product is a measure of the normality of that camera’s viewpoint with respect to the plane of the polygon. Since the dot product of any two orthogonal vectors is zero and the dot product of any two parallel, unit vectors is one, the largest dot product resulting from this procedure is associated with the camera with the most normal view of the polygon. In figure 7, camera one’s line of sight to the polygon is almost perpendicular to the plane of the polygon. In contrast, camera two can see the polygon only at a very oblique angle. The image projection software chooses to map a portion of the image captured by camera one onto this polygon and discard the portion of camera two’s image which corresponds to this section of the 3-D surface. By repeating this process for every polygon in the 3-D surface, a composite image of the entire surface can be obtained in which the data in each polygon is mapped using the image from the most appropriate camera.
Results
Figure 8 is a picture taken in the Basic Aerodynamics Research Tunnel (BART) at Langley Research Center. The four-wheel landing gear apparatus shown was tested in BART using an oil flow technique. A paint and oil mixture was applied to the tires in the wind tunnel. Pictures of the paint and oil mixture streaking across the tire surfaces were then taken using a digital camera. These pictures were used as inputs for the image projection software. The software combined these images with pressure data, which was also collected in BART, to produce a 3-D model of the landing gear setup. Figures 9 and 10 are snapshots of this model. The ability, exhibited here, to visually correlate two data sets by examining them on the same 3-D model has proven to be a great aid to researchers.
Equipment
A Silicon Graphics, Incorporated (SGI) computer in DVAL was used to create and run the image projection software. The VRML 1.0 files produced from the software were viewed using a variety of VRML viewers. These are widely available as stand-alone programs and as internet browser plug-ins on a variety of computer platforms.
Conclusions
Research has demonstrated that the advanced image projection software available in Langley Research Center’s Data Visualization and Animation Lab (DVAL) is an invaluable tool in the pursuit of accurate visualization of 3-D surface flow characteristics. The software has been significantly extended to support fusion of high-resolution images from multiple cameras, correlation of these images with 3-D color-mapped data, and output of the results in standard VRML format. These enhancements have made the image projection technique even more desirable as a means of visualizing air flow and pressure related data from 3-D surfaces analyzed in wind tunnels or during in-flight testing.
References
Figures:
