Week 1, Lecture 3: Image Formation

Week 1, Lecture 3: Image Formation#

NPTEL Deep Learning for Computer Vision | Prof. Vineeth N Balasubramanian, IIT Hyderabad

Companion notebook for §1.3 Image Formation.

What you will learn

The four factors that decide what an image looks like.
How light reflects off surfaces: albedo, diffuse and specular reflection, and the BRDF.
Why colour at the sensor depends on both the light source and the surface.
How a camera turns photons into an array of numbers: the Bayer grid, demosaicing, and CCD versus CMOS.

Contents

What decides an image
Models of reflection
Colour
From photons to pixels

No code in this lecture. This is a discussion lecture, so there is nothing to run here. The page follows the slides and ends with questions to think about. Hands-on notebooks begin at Lecture 4.

Review#

The previous lecture traced the field from Hubel and Wiesel through Marr to the Neocognitron, and noted that for decades features were designed by hand. Before designing anything that reads an image, it is worth knowing where the numbers in that image came from. That is this lecture.

1. What decides an image#

Four factors together determine the image that reaches you:

Light source strength and direction
Surface geometry, material, and nearby surfaces
Sensor capture properties
Image representation and colour

Each opens onto a large topic, and the course touches the parts it needs:

Area	Topics
Geometry	2D and 3D transformations, camera calibration, distortion
Photometry	Lighting, reflectance and shading, optics
Colour	Physics of colour, human colour, colour representation
Sensor	Human perception, camera design, sampling and aliasing, compression

For a full treatment, the slides point to Chapters 1–5 of Computer Vision: A Modern Approach by Forsyth and Ponce.

2. Models of reflection#

When light strikes a surface, several things happen at once.

Some is absorbed. The fraction reflected is the albedo \(\rho\); the fraction absorbed is \(1 - \rho\). Low-albedo surfaces absorb more.

Some is reflected diffusely, independent of viewing direction. Brick, cloth and rough wood behave this way. Lambert’s cosine law says the amount of reflected light is proportional to \(\cos\theta\), where \(\theta\) is the angle of illumination. This is why intensity depends on the illumination angle: less light arrives per unit area at oblique angles.

Some is reflected specularly, and that part does depend on where you view it from. A mirror is the extreme case. Most real surfaces have both diffuse and specular components.

Other effects the simple model leaves out: transparency, refraction, subsurface scattering, fluorescence and phosphorescence.

All of this is summarised by the BRDF, the Bidirectional Reflectance Distribution Function: a model of local reflection that says how bright a surface appears when viewed from one direction while light falls on it from another.

3. Colour#

Light is composed of a spectrum of wavelengths. The coloured light arriving at the sensor depends on two things multiplied together:

the colour of the light source, and
the colour of the surface.

This is why the same object photographs differently under daylight and tungsten light, and why “what colour is this object” is not a question the sensor can answer on its own.

A question from the slides. If the visible spectrum is VIBGYOR, why is colour represented with just R, G and B? The answer is not in the physics of light; it is in the three cone types of the human eye. Week 1 Lecture 4 and Week 2 Lecture 7 return to this.

4. From photons to pixels#

The Bayer grid. A camera sensor measures intensity, not colour, so colour filters are laid over the pixels in a Bayer arrangement: 50% green, 25% red, 25% blue. Green gets half the sites because human vision is most sensitive to it.

Each pixel therefore records only one of the three channels. Demosaicing algorithms reconstruct the full-colour image by estimating the two missing values at every pixel from its surrounding neighbours. Every photograph you have ever seen from a consumer camera is the output of this interpolation.

CCD versus CMOS. Two ways of getting charge out of the sensor:

	How it works
CCD	Photogenerated charge is moved from pixel to pixel to an output node, converted to voltage there, and an analog-to-digital converter turns each pixel into a digital value
CMOS	Charge is converted to voltage inside each element, using several transistors per pixel to amplify and move it

Key takeaways#

Concept	Key takeaway
Four factors	Light source, surface, sensor and representation together decide the image
Albedo	\(\rho\) is the reflected fraction; \(1 - \rho\) is absorbed
Diffuse reflection	Independent of viewing direction; follows Lambert’s cosine law, \(\propto \cos\theta\)
Specular reflection	Depends on viewing direction; most surfaces mix both kinds
BRDF	How bright a surface looks from one direction when lit from another
Colour	Light source colour times surface colour; the sensor cannot separate them alone
Bayer grid	50% green, 25% red, 25% blue; one channel measured per pixel
Demosaicing	The other two channels at each pixel are interpolated from neighbours
CCD vs CMOS	Charge converted at an output node, versus inside each pixel element

Exercises#

Why is green favoured? The Bayer grid devotes half its sites to green. Give the reason in terms of the human visual system, and predict what would degrade if the split were equal thirds.
VIBGYOR versus RGB. Answer the slide’s question in your own words. Why do three primaries suffice to reproduce a continuous spectrum, and when does the trick fail?
Diffuse or specular? For a matte wall, a polished floor, a leaf and human skin, say which component dominates and what would give it away in a photograph.
Lambert’s law in practice. A flat surface is lit from directly overhead and photographed. Sketch how measured intensity varies across the surface as it tilts away from the light, and explain why a shape-from-shading algorithm could use this.
Where demosaicing shows. Demosaicing interpolates two of every three colour values. Where in an image would you expect its errors to be most visible, and what artefact would they produce?