Vision in an Era of Human–Machine Coexistence: Computational Imaging

An era of humans and machines living together

Intelligent robots, autonomous driving, brain–computer interfaces, the low-altitude economy, AI hardware… Behind all these much-discussed topics is an underlying trend: an era in which humans and machines live and interact with one another is on its way.

Looking at it from the human side, we have both a mind and a body.

The release of ChatGPT at the end of 2022 marked machines' entry into the realm of thought. The rapid development of generative AI has also shown that machine thought is possible. We were taught in school that thought and language are what set humans apart from everything else. Perhaps that reasoning isn't right. Meanwhile, the arrival of humanoid robots, robot dogs, and other physical forms means that machines will have bodies, too.

Mind, body, and the physical world all depend on one another. None can be left out. At present, large language models have no perception of the physical world. A large language model is like a human brain, and the brain perceives the physical world through the body. The three are inseparable. For example, why can't we run fast in dreams?

Because in a dream, only the brain is working. The body cannot feel the ground, so it receives none of the feedback the ground gives us. When knowing and doing are disconnected, how can we run fast?

Looking at it from the machine side, the internet has brought about three forms of interaction: people looking for information (or machines), information finding people, and humans and machines interacting and living together.

In the internet era, people looked for information. Whatever you needed, you went to Google or Baidu and searched for the relevant keywords. In the mobile internet era, machines automatically recommend things based on your interests. Perhaps, in the future, we will live alongside machines, communicating and sharing with one another. Who knows?

The mobile internet era brought a whole series of changes to everyday life. As the internet and mobile devices became widespread, “mobile internet +” transformed one setting after another, making interactions between people and machines more convenient and reliable. Internet + finance made mobile payments commonplace. Internet + taxis gave rise to ride-hailing platforms such as Didi. Internet + shopping produced e-commerce giants such as Taobao. Riding this wave, information began to find people, with the rise of platforms such as Toutiao, Douyin, and Xiaohongshu. As robotics, autonomous driving, brain–computer interfaces, and other emerging technologies develop, perhaps we will enter a new era of humans and machines interacting and living together. We will see.

Building the eyes for this era: a revolution in vision through computational imaging

Vision is the most important part of perceiving the physical world. To adapt to an era shared by humans and machines, machine vision will inevitably have to change, too. In that change, computational imaging has a role to play as a new way of forming images. Machine vision will be about more than simply seeing a target clearly—the traditional imaging idea of “what you see is what you get.” It may be more about capturing information: a wider field of view, sharper images, or a greater depth of field. Or it may serve a particular task, such as object detection or segmentation. I hope that when these changes arrive, those of us working in the field will be ready.

What is computational imaging?

Imaging is more than a way of recording the world. It is also an important way of understanding it. Traditional optical imaging, however, can no longer meet our growing needs. This is where computational imaging comes in. So what is it, and how might it change our lives? Let's start with its definition and the different kinds of computational imaging.

Computational imaging is the process of indirectly forming images from measurements using algorithms that rely on a significant amount of computing.

— Wikipedia

In other words, computational imaging uses algorithms to form images indirectly from measurements, and those algorithms rely on substantial computation. Put simply: using computation to make imaging possible, or to make it better.

The core idea: Computational imaging combines optical hardware and algorithms to acquire, process, and reconstruct images. It goes beyond taking a photograph, using computation to overcome the limitations of traditional optical imaging and produce images that carry more information.

Related concepts: Computational imaging, computational optics, and computational photography are closely related, but they have different emphases and applications. All three use algorithms and computation to improve or extend traditional imaging. Computational imaging is the broader concept.

	Computational imaging	Computational optics	Computational photography
Main goal	Acquire high-quality images or information through hardware and algorithms working together	Optimize optical system design through computation	Improve photographic capabilities and image quality through algorithms
Focus	Joint optimization of hardware and algorithms	Design and modeling of optical hardware	Image processing and additional photographic capabilities
Typical applications	Super-resolution, light-field imaging, medical imaging	Optical lens design, adaptive optics, microscopy optics	Smartphone photography, HDR, panorama stitching
Main fields	Science, medicine, industry	Optical engineering, physics	Consumer electronics, photography
Dependence on hardware	High	High	Lower
Algorithmic emphasis	Deep learning, convex optimization	Convex optimization, signal processing	Deep learning, signal processing

Types of computational imaging

💡 Based on the needs and tasks they address, computational imaging methods can be roughly divided into two groups: imaging for better performance and imaging for specific functions.

Imaging for better performance

Accuracy: resolution, coverage, and reconstruction quality

Higher resolution: high-speed imaging, super-resolution imaging, hyperspectral imaging, and 3D imaging.
Broader coverage: long-duration imaging, wide fields of view, all-in-focus imaging, hyperspectral imaging, and continuous-viewpoint imaging.
Better reconstruction quality: restoring degraded images through denoising, super-resolution, deblurring, rain and haze removal, reconstruction, and related methods.

Efficiency: bandwidth, memory, and transmission

Compressive imaging: single-exposure imaging, single-pixel imaging, sparse-view reconstruction, and related methods.
Model quantization: model compression and deployment on mobile devices.
High-dimensional data representations: 3D NeRF, Gaussian splatting, and related approaches.

Imaging under special conditions

Low-light and high-dynamic-range imaging.
Imaging through scattering: adaptive imaging in astronomy and imaging through scattering in microscopy.
Seeing around corners: non-line-of-sight and single-photon imaging.

Imaging for specific functions

Autofocus.
Optical flow and event cameras.
Fluorescence lifetime imaging and photon counters.
Multimodal imaging: phase imaging, photoacoustic imaging, CT, terahertz imaging, MRI, ultrasound, fluorescence imaging, and more.
Image recognition, detection, segmentation, motion capture, privacy and security, and related tasks.
Portrait beautification, filters, animation effects, image generation, digital humans, and more.

Where computational imaging could be used

💡 Computational imaging has potential applications in consumer electronics, machine vision, biomedicine, and aerospace.

The following examples are from consumer electronics:

October 20, 2011: Lytro officially introduced its first consumer light-field camera.
September 12, 2017: Apple released the iPhone X, using 3D structured-light technology for the first time.
July 29, 2021: Huawei officially launched its P50 flagship series, introducing the concept of “computational optics.”
October 28, 2021: Facebook announced its change of name to Meta and said it would focus on metaverse technologies and platforms, including AR and VR.
January 17, 2023: “Computational optical imaging” was named one of Alibaba DAMO Academy's top ten technologies for 2023.
September 27, 2023: Meta and Ray-Ban launched Ray-Ban Meta smart glasses, setting off a wave of interest in AI glasses and AI hardware more broadly.
November 26, 2024: Huawei launched the Mate 70 series with “Red Maple” true-color imaging, moving from multispectral sensing—not imaging—to multispectral imaging.

💡 The categories above cover most computational imaging approaches. Some of these technologies are already widely used in industry. Below are a few imaging technologies that industry is particularly interested in.

Consumer electronics

Smartphone image signal processing (ISP) and low-level vision: denoising, deblurring, super-resolution, rain and haze removal, low-light imaging, night photography, and more.
Color reproduction, multispectral imaging, and related technologies.
3D reconstruction technologies, including multi-camera systems, structured light, time-of-flight (ToF) cameras, and lidar.
Video encoding and decoding.
Lighting estimation, depth estimation, 3D human pose estimation, digital humans, and related applications.

Biomedicine

OCT, CT, MRI, PET, ultrasound, digital pathology, and more.

Machine vision

3D reconstruction: stereo vision, photometric stereo, structured light, ToF, phase-measuring deflectometry, and related methods.
Optical measurement of dimensions, surface shape, vibration, and more.
Multispectral and hyperspectral imaging.
Thermal imaging and related technologies.

Resources for learning computational imaging

Computational imaging is a highly interdisciplinary field. It draws on optics, mathematical optimization, signal processing, computer vision, computer graphics, and more. As an engineering student with a background in optics, how can you find your way into this field quickly? There are a few foundations to learn.

💡 Theory and textbooks

Optics

Geometrical optics: I recommend Engineering Optics (《工程光学》) by Daoyin Yu (郁道银).
Physical optics: I recommend Physical Optics (《物理光学》) by Quanting Liang (梁铨廷).
Fourier optics.

Convex optimization and signal processing

Convex optimization: I recommend Steven Boyd.
Digital image processing: I recommend the Chinese edition of Gonzalez's textbook.
Computer vision: I recommend the Chinese edition of Richard Szeliski's textbook.

Computational optics, imaging, and photography

Computational Imaging and Computational Photography: Mastering New Techniques for Lenses, Lighting, and Sensors by Ramesh Raskar.
Optical Imaging and Spectroscopy by David J. Brady.
Computational Photography (《计算摄影学》) by Boxin Shi (施柏鑫).

💡 Public learning resources

Here are some openly available resources for getting started. I can't provide links here, but you can find them by searching for the titles and names below.

Video lectures

First Principles of Computer Vision — Shree Nayar · YouTube.
Image and Visual Signal Processing (图像与视觉信号处理) — Yingpin Chen (陈颖频).
Introduction to Computational Imaging (计算成像导论), GAMES204 — Qilin Sun (孙启霖) · Bilibili.

Courses, including lecture slides

Computational Photography, Carnegie Mellon University — Ioannis (Yannis) Gkioulekas.
Computational Camera and Photography, MIT — Ramesh Raskar.
Computational Imaging, Stanford University — Gordon Wetzstein.
Computational Photography, Brown University — James Tompkin.
Computational Imaging, Cornell University — Kristina Monakhova.