(also available as AVI)
(also available as AVI)
|
|
|
Click on a thumbnail to watch the corresponding image sequence (MOV file).
(also available as AVI) |
Note that both sequences were computed from a single snapshot by the camera.
(also available as AVI) |
| Our first prototype light field microscope (LFM). The eyepiece has been removed and replaced with a microlens array (circled in red). Above the array is a camera and 1:1 relay lens. Click here to see some more recent, portable versions. | A light field as recorded by the optical arrangement at left. The specimen is a 200-micron-tall "tower" of fluorescent crayon wax. The objective is 20x/0.5 (dry), used without a cover slip. | Perspective pan. Since microscopes are telecentric, they are normally only capable of producing orthographic views. Computed in real time. | Focal stack. Using deconvolution, this 3D stack can be converted into a 3D (cross-sectional) volume. See below for more examples. |
Light field photography is an image-based technique for recording scene appearance. Unlike conventional photography, light fields permit manipulation of viewpoint and focus after the imagery has been recorded. Devices for capturing light fields range in scale from room-size arrays of cameras to a handheld camera in which a microlens array has been inserted between the main lens and sensor plane.
In this project, we are exploring light field photography at the microscopic scale. Specifically, by inserting a microlens array into the optical train of a conventional optical microscope, we can capture light fields of biological specimens in a single snapshot. Although diffraction places a limit on the product of spatial and angular resolution in these light fields, we can nevertheless produce useful perspective flyarounds and 3D focal stacks from them. Applying standard deconvolution algorithms to these focal stacks, we can reconstruct 3D volumes. Since microscope optics produce orthographic views, perspective flyarounds represent a new way to look at microscopic specimens. Focal stacks are not new, but manual techniques for capturing them are time-consuming and hence not applicable to moving or light-sensitive specimens.
Like any new scientific imaging instrument, we expect the light field microscope to have many applications in science, medicine, and industry. For example, since the light field microscope separates image acquisition from the selection of viewpoint and focus. it can be used as a "digital viewfinder" for a conventional microscope. Applying computer vision algorithms to these selection procedures may lead to greater automation of microscope operation when analyzing large numbers of specimens, such as in clinical pathology.
|
2005 - our first light field micrographs, including insect legs, silkworm mandibles, and mouse lungs. Also included are volumetric reconstructions using 3D deconvolution, and some experimental all-focus images. |
|
2006 - light field micrographs captured at the Marine Biological Laboratory, including onion skin (under DIC illumination), squid skin (under grazing illumination), a light field scatterometry experiment, and mouse embryos. |
|
2007 - light field micrographs captured at
the Hopkins Marine Station, including fern spores and cone snails. Also included is a description of our portable light field microscope and real-time software viewer. |
|
2008 - our light field illuminator (LFI)
is used to examine a human hair fiber under changing illumination. Also, functional neural imaging of larval zebrafish, recorded in Florian Engert's lab at Harvard University. |
February 14, 2008
Microscope design: Marc Levoy
Specimen: Shinya Inoue (Marine Biological Laboratory)
|
|
| Compact light field microscope | Prototype high-resolution light field microscope |
At left above is our most compact light field microscope (LFM). As introduced on this web page, it consists of an ordinary research microscope (Nikon Eclipse 80i) and cooled scientific camera (Retiga 4000R, 2048 x 2048 pixels) with a microlens array inserted between the two (red circle). The image formed by the microlens array is conveyed to the sensor in the camera by a 1:1 relay lens system consisting of two nose-to-nose Nikon 50mm/1.4 photographic lenses. With this arrangement, instead of capturing an ordinary 2048 x 2048 pixel image, the camera captures a 2048 x 2048 sample light field, capable of producing 17 x 17 different oblique views, each about 120 x 120 pixels in size. We have fabricated several copies of this arrangement, which have been used by microscopists in several departments at Stanford University, at the Hopkins Marine Institute (Monterey, CA), the Marine Biological Laboratory (Woods Hole, MA), and ETH (Zurich).
At right above is a prototype high-resolution light field microscope. A 45-degree mirror (A) sends the microscope image through a 2:1 telecentric relay lens (B), our microlens array (C), a 1:1 telecentric relay lens (D), and a Canon 5D full-frame digital SLR (E). Since doubling the lateral magnification using a 2:1 relay lens roughly halves the angular aperture (this is called the LaGrangian invariant), we use an f/30 microlens array rather than the f/20 array we more commonly use. The camera was manually microstepped to place a green pixel beneath each pixel site, thereby avoiding color demosaicing and the consequent loss in spatial resolution. The light fields captured using this arrangement (see below) contain approximately 4500 x 3000 samples, capable of producing 15 x 15 oblique views, each about 300 x 200 pixels in size. The field of view is approximately 1.8mm horizontally for the 20x light field and 0.9mm for the 40x. For the objectives we used and the pixel resolution of our camera, these light fields are also diffraction-limited, meaning that no further spatial or angular resolution can be squeezed from the wavefront propagating through the microscope, at least not using conventional image capture protocols.
Light field (click on image for full size) 20x/0.75NA (dry) 
Flyaround (click on image to watch movie) (or reduced to CIF size in this .mp4 file) |
Light field (click on image for full size) 40x/1.3NA (oil) 
Flyaround (click on image to watch movie) (or reduced to CIF size in this .mp4 file) |
|
Click here to view these light fields interactively in your browser,
using a
Flash-based light field viewer, and click here to view them using LFDisplay, our hardware-accelerated viewer for microscope light fields. |
The two columns above represent two light fields captured using this high-resolution microscope. The specimen is a Golgi-stained slice of rat brain (courtesy of Shinya Inoue, Marine Biological Laboratory). The light field at left was photographed using a 20x/0.75NA objective, and the light field at right using a 40x/1.3NA oil immersion objective. Note the spatial detail in both movies; you can easily see the fine dendrites. Note also the large amount of parallax visible in the 40x movie, especially on the looping capillaries in the lower-left part of the field of view. This much parallax is available because a 1.3NA objective can capture rays leaving the specimen at angles up to 59 degrees on either side of the optical axis. For photographers, this would correspond to a lens having a relative aperture of f/0.38 - far faster than any commercially available photographic lens.