Removing unwanted objects from pictures with Cleanup.pictures

Ever been on vacation somewhere, and wanted to take a picture of something, only to be thwarted by the hordes of tourists? Typically for me it’s buildings of architectural interest, or wide-angle photos in towns. It’s quite a common occurrence, especially in places where tourists tend to congregate. There aren’t many choices – if you can come back at a quieter time that may be the best approach, but often you are at a place for a limited time-frame. So what to do?

Use software to remove the offending objects, or people. Now this type of algorithm designed to remove objects from an image has been around for about 20 years, known in the early years as digital inpainting, akin to the conservation process where damaged, deteriorating, or missing parts of an artwork are filled in to present a complete image. In its early forms digital inpainting algorithms worked well in scenes where the object to be removed was surrounded by fairly uniform background, or pattern. In complex scenes they often didn’t fair so well. So what about the newer generation of these algorithms?

There are many different types of picture cleaning software, some stand-alone such as the AI-powered IOS app Inpaint, others in the form of features in photo processing software such as Photoshop. One new-comer to the scene is web-based, open-source, Cleanup.pictures. It is incredibly easy to use. Upload a picture, choose the size of the brush tool, paint over the unwanted object with the brush tool, and voila! a new image, sans the offending object. Then you can just download the “cleaned” image. So how well does it work? Below are some experiments.

The first image is a vintage photograph of Paris, removing all the people from the streets. The results are actually quite exceptional.

The second image is a photograph taken in Glasgow, where the people and passing car have been erased.

The third image is from a trip to Norway, specifically the harbour in Bergen. This area always seems to have both people and boats, so it is hard to take clear pictures of the historical buildings.

The final image is a photograph taken by Prokudin-Gorskii Collection at the Library of Congress. The image is derived from a series of glass plates, and suffers from some of the original glass plates being broken, with missing pieces of glass. The result of cleaning up the image, actually has done a better job than I could ever have imagined.

The AI used in this algorithm is really good at what it does, like *really good*, and it is easy to use. You just keep cleaning up unwanted things until you are happy with the result. The downsides? It isn’t exactly perfect all the time. In regions to be removed where there are fine details you want to retain, they are often removed. Sometimes areas become “soft” because they have to be “created” because they were obscured by objects before – especially prevalent in edge detail. Some examples are shown below:

Creation of detail during inpainting

Loss of fine detail during inpainting

It only produces low-res images, with a maximum width of 720 pixels. You can upgrade to the Pro version to increase resolution (2K width). It would be interesting to see this algorithm produce large scale cleaned images. There is also the issue of uploading personal photos to a website, although they do make the point of saying that images are discarded once processed.

For those interested in the technology behind the inpainting, it is based on an algorithm known as large mask inpainting, developed by a group at Samsung, and associates [1]. The code can be obtained directly from github for those who really want to play with things.

  1. Suvorov, R., et al. Resolution-robust Large Mask Inpainting with Fourier Convolutions (2022)

Demystifying Colour (viii) : CIE colour model

The Commission Internationale de l’Eclairage (French for International Commission on Illumination) , or CIE is an organization formed in 1913 to create international standards related to light and colour. In 1931, CIE introduced CIE1931, or CIEXYZ, a colorimetric colour space created in order to map out all the colours that can be perceived by the human eye. CIEXYZ was based on statistics derived from extensive measurements of human visual perception under controlled conditions.

In the 1920s, colour matching experiments were performed independently by physicists W. David Wright and John Guild, both in England [2]. The experiments were carried out with 7 (Guild) and 10 (Wright) people. Each experiment involved a subject looking through a hole which allowed for a 2° field of view. On one side was a reference colour projected by a light source, while on the other were three adjustable light sources (the primaries were set to R=700.0nm, G=546.1nm, and B=435.8nm.). The observer would then adjust the values of three primary lights until they can produce a colour indistinguishable from a reference light. This was repeated for every visible wavelength. The result of the colour-matching experiments was a table of RGB triplets for each wavelength. These experiments were not about describing colours with qualities like hue and saturation, but rather just attempt to explain how combinations of light appear to be the same colour to most people.

Fig.1: An example of the experimental setup of Guild/Wright

In 1931 CIE amalgamated Wright and Guild’s data and proposed two sets of of colour matching functions: CIE RGB and CIE XYZ. Based on the responses in the experiments, values were plotted to reflect how the average human eye senses the colours in the spectrum, producing three different curves of intensity for each light source to mix all colours of the colour spectrum (Figure 2), i.e. Some of the values for red were negative, and the CIE decided it would be more convenient to work in a colour space where the coefficients were always positive – the XYZ colour matching functions (Figure 3). The new matching functions had certain characteristics: (i) the new functions must always be greater than or equal to zero; (ii) the y function would describe only the luminosity, and (iii) the white-point is where x=y=z=1/3. This produced the CIE XYZ colour space, also known as CIE 1931.

Fig.2: CIE RGB colour matching functions

Fig.3: CIE XYZ colour matching functions

The CIE XYZ colour space defines a quantitative link between distributions of wavelengths in the electromagnetic visible spectrum, and physiologically perceived colours in human colour vision. The space is based on three fictional primary colours, X, Y, and Z, where the Y component corresponds to the luminance (as a measure of perceived brightness) of a colour. All the visible colours reside inside an open cone-shaped region, as shown in Figure 4. CIE XYZ is then a mathematical generalization of the colour portion of the HVS, which allows us to define colours.

Fig.4: CIE XYZ colour space (G denotes the axis of neutral gray).
Fig.5: RGB mapped to CIE XYZ space

The luminance in XYZ space increases along the Y axis, starting at 0, the black point (X=Y=Z=0). The colour hue is independent of the luminance, and hence independent of Y. CIE also defines a means of describing hues and saturation, by defining three normalized coordinates: x, y, and z (where x+y+z=1).

x = X / (X+Y+Z)
y = Y / (X+Y+Z)
z = Z / (X+Y+Z)
z = 1 - x - y

The x and y components can then be taken as the chromaticity coordinates, determining colours for a certain luminance. This system is called CIE xyY, because a colour value is defined by the chromaticity coordinates x and y in addition to the luminance coordinate Y. More on this in the next post on chromaticity diagrams.

The RGB colour space is related to XYZ space by a linear coordinate transformation. The RGB colour space is embedded in the XYZ space as a distorted cube (see Figure 5). RGB can be mapped onto XYZ using the following set of equations:

X = 0.41847R - 0.09169G - 0.0009209B
Y = -0.15866R + 0.25243G - 0.0025498B (luminance)
Z = -0.082835R + 0.015708G + 0.17860B

CIEXYZ is non-uniform with respect to human visual perception, i.e. a particular fixed distance in XYZ is not perceived as a uniform colour change throughout the entire colour space. CIE XYZ is often used as an intermediary space in determining a perceptually uniform space such as CIE Lab (or Lab), or CIE LUV (or Luv).

  • CIE 1976 CIEL*u*v*, or CIELuv, is an easy to calculate transformation of CIE XYZ which is more perceptually uniform. Luv was created to correct the CIEXYZ distortion by distributing colours approximately proportional to their perceived colour difference.
  • CIE 1976 CIEL*a*b*, or CIELab, is a perceptually uniform colour differences and L* lightness parameter has a better correlation to perceived brightness. Lab remaps the visible colours so that they extend equally on two axes. The two colour components a* and b* specify the colour hue and saturation along the green-red and blue-yellow axes respectively.

In 1964 another set of experiments were done allowing for a 10° field of view, and are known as the CIE 1964 supplementary standard colorimetric observer. CIE XYZ is still the most commonly used reference colour space, although it is slowly being pushed to the wayside by CIE1976. There is a lot of information on CIE XYZ and its derivative spaces. The reader interested in how CIE1931 came about in referred to [1,4]. CIELab is the most commonly used CIE colour space for imaging, and the printing industry.

Further Reading

  1. Fairman, H.S., Brill, M.H., Hemmendinger, H., “How the CIE 1931 color-matching functions were derived from Wright-Guild data”, Color Research and Application, 22(1), pp.11-23, 259 (1997)
  2. Service, P., The Wright – Guild Experiments and the Development of the CIE 1931 RGB and XYZ Color Spaces (2016)
  3. Abraham, C., A Beginners Guide to (CIE) Colorimetry
  4. Zhu, Y., “How the CIE 1931 RGB Color Matching Functions Were Developed from the Initial Color Matching Experiments”.
  5. Sharma, G. (ed.), Digital Color Imaging Handbook, CRC Press (2003)

How does high-resolution mode work?

One of the tricks of modern digital cameras is a little thing called “high-resolution mode” (HRM), which is sometimes called pixel-shift. It effectively boosts the resolution of an image, even though the number of pixels used by the camera’s sensor does not change. It can boost a 24 megapixel image into a 96 megapixel image, enabling a camera to create images at a much higher resolution than its sensor would normally be able to produce.

So how does this work?

In normal mode, using a colour filter array like Bayer, each photosite acquires one particular colour, and the final colour of each pixel in an image is achieved by means of demosaicing. The basic mechanism for HRM works through sensor-shifting (or pixel-shifting) i.e. taking a series of exposures and processing the data from the photosite array to generate a single image.

  1. An exposure is obtained with the sensor in its original position. The exposure provides the first of the RGB components for the pixel in the final image.
  2. The sensor is moved by one photosite unit in one of the four principal directions. At each original array location there is now another photosite with a different colour filter. A second exposure is made, providing the second of the components for the final pixel.
  3. Step 2 is repeated two more times, in a square movement pattern. The result is that there are four pieces of colour data for every array location: one red, one blue, and two greens.
  4. An image is generated with each RGB pixel derived from the data, the green information is derived by averaging the two green values.

No interpolation is required, and hence no demosaicing.

The basic high-resolution mode process (the arrows represent the direction the sensor shifts)

In cameras with HRM, it functions using the motors that are normally dedicated to image stabilization tasks. The motors effectively move the sensor by exactly the amount needed to shift the photosites by one whole unit. The shifting moves in such a manner that the data captured includes one Red, one Blue and two Green photosites for each pixel.

There are many benefits to this process:

  • The total amount of information is quadrupled, with each image pixel using the actual values for the colour components from the correct physical location, i.e. full RGB information, no interpolation required.
  • Quadrupling the light reaching the sensor (four exposures) should also cut the random noise in half.
  • False-colour artifacts often arising in the demosaicing process are no longer an issue.

There are also some limitations:

  • It requires a very steady scene. It doesn’t work well if the camera is on a tripod, yet there is a slight breeze, moving the leaves on a tree.
  • It can be extremely CPU-intensive to generate a HRM RAW image, and subsequently drain the battery. Some systems, like Fuji’s GFX100 uses off-camera, post-processing software to generate the RAW image.

Here are some examples of the high resolution modes offered by camera manufacturers:

  • Fujifilm – Cameras like the GFX100 (102MP) have a Pixel Shift Multi Shot mode where the camera moves the image sensor by 0.5 pixels over 16 images and composes a 400MP image (yes you read that right).
  • Olympus – Cameras like the OM-D E-M5 Mark III (20.4MP), has a High-Resolution Mode which takes 8 shots using 1 and 0.5 pixel shifts, which are merged into a 50MP image.
  • Panasonic – Cameras like the S1 (24.2MP) have a High-Resolution mode that results in 96MP images. The Panasonic S1R at 47.3MP produces 187MP images.
  • Pentax – Cameras like the K-1 II (36.4MP) use a Pixel Shift Resolution System II with a Dynamic Pixel Shift Resolution mode (for handheld shooting).
  • Sony – Cameras like the A7R IV (61MP) uses a Pixel Shift Multi Shooting mode to produce a 240MP image.

Further Reading:

Do you need 61 megapixels, or even 102?

The highest “native” resolution camera available today is the Phase One FX IQ4 medium format camera at 150MP. Higher than that there is the Hasselblad H6D-400C at 400MP, but it uses pixel-shift image capture. Next in line is the medium format Fujifilm GFX 100/100S at 102 MP. In fact we don’t get to full-frame sensors until we hit the Sony A7R IV, at a tiny 61MP. Crazy right? The question is how useful are these sensors for the photographer? The answer is not straightforward. For some photographic professionals these large sensors make inherent sense. For the average casual photographer, they likely don’t.

People who don’t photograph a lot tend to be somewhat bamboozled by megapixels, like more is better. But more megapixels does not mean a better image. Here are some things to consider when thinking about when considering megapixels.

Sensor size

There is a point when it becomes hard to cram any more photosites into a particular sensor – they just become too small. For example the upper bound with APC-S sensors seems to be around 33MP, with full-frame it seems to be around 60MP. Put too many photosites on a sensor and the density of the photosites increases, as the size of the photosites decreases. The smaller the photosite, the harder it is for it to collect light. For example Fuji APS-C cameras seem to tap out at around 26MP – the X-T30 has a photosite pitch of 3.75µm. Note that Fuji’s leap to a larger number of megapixels also means a leap to a larger sensor – the medium format sensor with a sensor size of 44×33mm. Compared to the APS-C sensor (23.5×15.6mm), the medium format sensor is nearly four times the size. A 51MP medium format sensor has photosites which are 5.33µm in size, or 1.42 times of size of the 26MP APS-C sensor.

The verdict? Squeezing more photosites onto the same size sensor does increase resolution, but sometimes at the expense of how light is acquired by the sensor.

Image and linear resolution

Sensors are made up of photosites that acquire the data used to make image pixels. The image resolution of an image describes the number of pixels used to construct an image. For example a 16MP sensor with a 3:2 aspect ratio has an image resolution of 4899×3266 pixels – the dimensions are sometimes termed the linear resolution. To obtain twice the image resolution we need a 64MP sensor, rather than a 32MP sensor. A 32MP sensor has 6928×4619 photosites, which results in a 1.4 times increase in the linear resolution of the image. The pixel count has doubled, but the linear resolution has not. Upgrading from a 16MP sensor to a 24MP sensor means a ×1.5 increase in the pixel count, and a ×1.2 increase in linear resolution. The transition from 16MP to 64MP is a ×2 increase in linear resolution, and a ×4 increase in the number of pixels. That’s why the difference between 16MP and 24MP sensors is also dubious (see Figure 1).

Fig.1: Different image resolutions and megapixels within an APS-C sensor

To double the linear resolution of a 24MP sensor you need a 96MP sensor. So the 61MP sensor provides about double the linear resolution of a 16MP sensor, as the 102MP sensor doubles the 24MP sensor.

The verdict? Doubling the pixel count, i.e. image resolution, does not double the linear resolution.

Photosite size

When you have more photosites, you also have to ask what their physical size is. Squeezing 41 million photosites on the same size sensor as one which previously had 24 million pixels means that each pixel will be smaller, and that comes with its own baggage. Consider for instance the full-frame camera, the full-frame Leica M10-R, which has a 7864×5200 photosites (41MP) meaning the photosite size is roughly 4.59 microns. The full-frame 24MP Leica M-E has a photosite size of 5.97 microns, so 1.7 times the area. Large photosites allow more light to be captured, while smaller photosites gather less light, so when their low signal strength is transformed into a pixel, more noise is generated.

The verdict? From the perspective of photosite size, 24MP captured on a full-frame sensor will be better than 24MP on an APS-C sensor, which in turn is better than 24MP on a M43 sensor (theoretically anyways).

Optics

Comparing the quality of a 16MP lens to a 24MP lens, we might determine that the quality, and sharpness of the lens is more important than the number of pixels. In fact too many people place an emphasis on the number of pixels and forget about the fact that light has to pass through a lens before it is captured by the sensor and converted into an image. Many high-end cameras already provide an in-camera means of generating a high-resolution images, often four times the actual image resolution – so why pay more for more megapixels? Is a 50MP full-frame sensor any good without optically perfect (or near-perfect) lenses? Likely not.

The verdict? Good quality lenses are just as important as more megapixels.

File size

People tend to forget that images have to be saved on memory cards (and post-processed). The greater the megapixels, the greater the resulting file size. A 24MP image stored as a 24-bit/pixel JPEG will be 3.4MB in size (at 1/20). As a 12-bit RAW the file size would be 34MB. A 51MP camera like the Fujifilm GFX 50S II would have a 7.3MB JPEG, and a 73MB 12-bit RAW. If the only format used is JPEG it’s probably, fine, but the minute you switch to RAW it will use way more storage.

The verdict? More megapixels = more megabytes.

Camera use

The most important thing to consider may be what the camera is being used for?

  • Website / social media photography – Full-width images for websites are optimal at around 2400×1600 (aka 4MP), blog-post images max. 1500 pixels in width (regardless of height), and inside content max 1500×1000. Large images can reduce website performance, and due to screen resolution won’t be visualized to their fullest capacity anyways.
  • Digital viewing – 4K televisions have roughly 3840×2160 = 8,294,400 pixels. Viewing photographs from a camera with a large spatial resolution will just mean they are down-sampled for viewing. Even the Apple Pro Display XDR only has 6016×3384=20MP view capacity (which is a lot).
  • Large prints – Doing large posters, for example 20×30″ requires a good amount of resolution if they are being printed at 300DPI, which is the nominal standard. So this needs about 54MP (check out the calculator). But you can get by with less resolution because few people view a poster at 100%.
  • Average prints – An 8×10″ print requires 2400×3000 = 7.2MP at 300DPI. A 26MP image will print maximum size 14×20″ at 300DPI (which is pretty good).
  • Video – Does not need high resolution, but rather 4K video at a descent frame rate.

The verdict? The megapixel amount really depends on the core photographic application.

Postscript

So where does that leave us? Pondering a lot of information, most of which the average photographer may not be that interested in. Selecting the appropriate megapixel size is really based on what a camera will be used for. If you commonly take landscape photographs that are used in large scale posters, then 61 or 102 megapixels is certainly not out of the ballpark. For the average photographer taking travel photos, or for someone taking images for the web, or book publishing, then 16MP (or 24MP at the higher end) is ample. That’s why smartphone cameras do so well at 12MP. High MP cameras are really made more for professionals. Nobody needs 61MP.

The voverall erdict? Most photographers don’t need 61 megapixels. In reality anywhere between 16 and 24 megapixels is just fine.

Further Reading

Photosites – Quantum efficiency

Not every photo that makes it through the lens ends up in a photosite. The efficiency with which photosites gather incoming light photons is called its quantum efficiency (QE). The ability to gather light is determined by many factors including the micro lenses, sensor structure, and photosite size. The QE value of a sensor is a fixed value that depends largely on the chip technology of the sensor manufacturer. The QE is averaged out over the entire sensor, and is expressed as the chance that a photon will be captured and converted to an electron.

Quantum efficiency (P = Photons per μm2, e = electrons)

The QE is a fixed value and is dependent on a sensor manufacturers design choices. The QE is averaged out over the entire sensor. A sensor with an 85% QE would produce 85 electrons of signal if it were exposed to 100 photons. There is no way to effect the QE of a sensor, i.e. you can’t change things by changing the ISO.

The QE is typically 30-55% meaning 30-55% of the photons that fall on any given photosite are converted to electrons. (front illuminated sensors). In back illuminated sensors, like those typically found on smartphones, the QE is approximately 85%. The website Photons to Photos has a list of sensor characteristics for a good number of cameras. For example the sensor in my Olympus OM-D E-M5 Mark II has a supposed QE of 60%. Trying to calculate the QE of a sensor in non-trivial.

Fixing the “crop-factor” issue

We use the term “cropped sensor” only due to the desire to describe a sensor in terms of the 35mm standard. It is a relative term which compares two different types of sensor, but it isn’t really that meaningful. Knowing that a 24mm MFT lens “behaves” like a 48mm full-frame lens is pointless if you don’t understand how a 48mm lens behaves on a full-frame camera. All sensors could be considered “full-frame” in the context of their environment, i.e. a MFT camera has a full-frame sensor as it relates to the MFT standard.

As mentioned in a previous post, the “35mm equivalence” is used to relate a crop-factor lens to its full-frame equivalent. The biggest problem with this is the amount of confusion it creates for novice photographers. Especially as focal lengths on lenses are always the same, yet the angle-of-view changes according to the sensor. However there is a solution to the problem, and that is to stop using the focal length to define a lens, and instead use AOV. This would allow people to pick a lens based on its angle-of view, both in degrees, but also from a descriptive point of view. For example, a wide angle lens in full-frame is 28mm – its equivalent in APS-C in 18mm, and MFT is 14mm. It would be easier just to label these by the AOV as “wide-74°”.

It would be easy to categorize lenses into six core groups based on horizontal AOV (diagonal AOV in []) :

  • Ultra-wide angle: 73-104° [84-114°]
  • Wide-angle: 54-73° [63-84°]
  • Normal (standard): 28-54° [34-63°]
  • Medium telephoto: 20-28° [24-34°]
  • Telephoto: 6-20° [8-24°]
  • Super-telephoto: 3-6° [4-8°]
Lenses could be advertised using a graphic to illustrate the AOV (horizontal) of the lens. This effectively removes the need to talk about focal length.

They are still loosely based on how AOV related to 35mm focal lengths. For example 63° relates to the AOV of a 35mm lens, however it no longer really relates to the focal length directly. A “normal-40°” lens would be 40° no matter the sensor size, even though the focal lengths would be different (see table below). The only lenses left out of this are fish-eye lenses, which in reality are not that common, and could be put into a
specialty lens category, along with tilt-shift etc.

Instead of brochures containing focal lengths they could contain the AOV’s.

I know most lens manufacturers describe AOV using diagonal AOV, but this is actually more challenging for people to perceive, likely because looking through a camera we generally look at a scene from side-to-side, not corner-to-corner.

AOV98°84°65°
MFT8mm10mm14mm
APS-C10mm14mm20mm
FF16mm20mm28mm
Wide/ultra-wide angle lenses

AOV54°49°40°
MFT17mm20mm25mm
APS-C24mm28mm35mm
FF35mm40mm50mm
Normal lenses

AOV28°15°10°
MFT35mm70mm100mm
APS-C45mm90mm135mm
FF70mm135mm200mm
Telephoto lenses

Demystifying Colour (vi) : Additive vs. subtractive

In the real world there are two ways to produce colours, and they are very relevant because they deal with opposite ends of the photographic spectrum – additive and subtractive. Additive colours are formed by combining coloured light, whereas subtractive colours are formed by combining coloured pigments.

Additive colours are so-called because colours are built by combining wavelengths of light. As more colours are added, the overall colour becomes whiter. Add green and blue together and what you get is a washed out cyan. RGB is an example of an additive colour model. mixing various amounts of red, green, and blue produces the secondary colours: yellow, cyan, and magenta. Additive colour models are the norm for most systems of photograph acquisition or viewing.

Additive colour
Subtractive colour

Subtractive colour works the other way, by removing light. When we look at a clementine, what we see is the orange light not absorbed by the clementine, i.e. all other wavelengths are absorbed, except for orange. Or rather the clementine is subtracting the other wavelengths from the visible light, meaning there is only orange left to reflect off. CMYK and RYB (Red-Yellow-Blue) are good examples of subtractive colour models. Subtractive models are for most systems for printed material.

Different colour inks absorb and reflect specific wavelengths. CMYK (0,0,0,0) looks like white (no ink is laid down, so no light is absorbed), whereas (0,0,0,100) looks like black (maximum black is laid down meaning all colours are absorbed). CMYK values range from 0-100%. Below are some examples.

Ink colourabsorbsreflectsappears
cyanredgreen, bluecyan
magentagreenred, bluemagenta
yellowbluegreen, redyellow
magenta + yellowblue, greenredred
cyan + yellowred, bluegreengreen
cyan + magentared, greenblueblue

The effect of crop sensors on lenses

Lenses used on crop-sensor cameras are a little different to those of full-frame cameras. Mostly this has to do with size – because the sensor is smaller, the image circle doesn’t need to be as large, and therefore less glass is needed in their construction. This allows crop-sensor lenses to be more compact, and lighter. The benefit is that for lenses like telephoto, a smaller size lens is required. A 300mm FF equivalent in MFT only needs to be 150mm. But what does focal-length equivalency mean?

Focal-Length Equivalency

The most visible effect of crop-sensors on lenses is the angle-of-view (AOV), which is essentially where the term crop comes from – the smaller sensor’s AOV is a crop of the full frame. Take a photograph with two cameras: one with a full-frame and another with an APS-C sensor, from the same position using lens with the same focal lengths. The camera with the APS-C sensor will have a more narrowed AOV. For example a 35mm lens on a FF camera has the same focal length as a FF on an MFT or APS-C camera, however the AOV will be different on each. An example of this is shown in Fig.1 for a 35mm lens (showing horizontal AOV).

Fig.1: AOV for 35mm lenses on FF, APS-C, and MFT

Now it should be made clear that none of this affects the focal length of the lens. The focal length of a lens remains the same – regardless of the sensor on the camera. Therefore a 50mm lens in FF, APS-C or MFT will always have a focal length of 50mm. What changes is the AOV of each of the lenses, and consequently the FOV. In order to obtain the same AOV on a cropped-sensor camera, a new lens with the appropriate focal length must be chosen.

Manufacturers of crop-sensors like to use the term “equivalent focal length“. Now this is the focal length AOV as it relates to full-frame. So Olympus says that a MFT lens with a focal length of 17mm has a 34mm FF equivalency. It has an AOV of 65° (diagonal, as per the lens specs), and a horizontal AOV of 54°. Here’s how we calculate those (21.64mm is the diagonal of the MFT sensor, which is 17.3×13mm in size):

  • 17mm MFT lens → 2*arctan(21.64/(2*17)) = 65° (diag)
  • 17mm MFT lens → 2*arctan(17.3/(2*17)) = 54° (hor)
  • 34mm FF lens → 2*arctan(36/(2*34)) = 55.8° (hor)

So a lens with a 17mm focal length on a camera with a 2.0× crop factor MFT sensor would give an AOV equivalent of to that of a 34mm lens. An APS-C sensor has a crop factor of ×1.5, so a 26mm lens would be required to give an AOV equivalent of the 34mm FF lens. Figure 2 depicts the differences between 50mm FF and APS-C lenses, and the similarities between a 50mm FF lens and a 35mm APS-C lens (which give approximately the same AOV/FOV).

Fig.2: Example of lens equivalencies: FF vs. APS-C (×1.5)

Interchangeability of Lenses

On a side note, FF lenses can be used on crop-sensor cameras because the image circle of the FF lens is larger than the crop sensor. The reverse is however not possible, as a CS lens has a smaller image circle than a FF sensor. The picture below illustrates the various combinations of FF/MFT sensor cameras, and FF/MFT lenses.

Fig.3:The effect of interchanging lenses between FF and crop sensor cameras.

Of course all this is pointless if you don’t care about comparing your crop-sensor camera to a full-frame camera.

NOTE: I tend to use horizontal AOV rather than the manufacturers more typical diagonal AOV. It makes more sense because I am generally viewing a scene in a horizontal context.

From photosites to pixels (iii) – DIP

DIP is the Digital Image Processing system. Once the ADC has performed its conversion, each of the values from the photosite has been converted from a voltage to a binary number representing some value in its bit depth. So basically you have a matrix of integers representing each of the original photosites. The problem is that this is essentially a matrix of grayscale values, with each element of the matrix representing with a Red, Green of Blue pixel (basically a RAW image). If a RAW image is required, then no further processing is performed, the RAW image and its associated metadata are saved in a RAW image file format. However to obtain a colour RGB image and store it as a JPEG, further processing must be performed.

First it is necessary to perform a task called demosaicing (or demosaiking, or debayering). Demosaicing separates the red, green, and blue elements of the Bayer image into three distinct R, G, and B components. Note a colouring filtering mechanism other than Bayer may be used. The problem is that each of these layers is sparse – the green layer contains 50% green pixels, and the remainder are empty. The red and blue layers only contain 25% of red and blue pixels respectively. Values for the empty pixels are then determined using some form of interpolation algorithm. The result is an RGB image containing three layers representing red, green and blue components for each pixel in the image.

The DIP process

Next any processing related to settings in the camera are performed. For example, the Ricoh GR III has two options for noise reduction: Slow Shutter Speed NR, and High-ISO Noise Reduction. In a typical digital camera there are image processing settings such as grain effect, sharpness, noise reduction, white balance etc. (which don’t affect RAW photos). Some manufacturers also add additional effects such as art effect filters, and film simulations, which are all done within the DIP processor. Finally the RGB image image is processed to allow it to be stored as a JPEG. Some level of compression is applied, and metadata is associated with the image. The JPEG is then stored on the memory card.

Demystifying Colour (v) : colour gamuts

Terms used to describe colours are often confusing. If a colour space is a subset of a colour model, then what is a colour gamut? Is it the same as a colour space? How does it differ from a colour profile? In reality there is often very little difference between the terms. For example, depending on where you read it sRGB can be used to describe a colour space, a colour gamut, or a colour profile. Confused? Probably.

Colour gamuts

A gamut is a range or spectrum of some entity, for example “the complete gamut of human emotions“. A colour gamut describes a subset of colours within the entire spectrum of colours that are identifiable by the human eye, i.e. the visible colour spectrum. More specifically a gamut is the range of colours a colour space can represent.

While the range of colour imaging devices is very broad, e.g. digital cameras, scanners, monitors, printers, the range of colours they produce can vary considerably. Colour gamuts are designed to reconcile colours that can be used in common between devices. The term colour gamut is usually used in association with electronic devices, i.e. the devices range of reproducible colours, or the range of different colours that can be interpreted by a colour model. A colour gamut can therefore be used to express the difference between various colour spaces, and to illustrate the extent of coverage of a colour space.

Fig.1: CIE XYZ 2D Chromaticity Diagram depicting various colour spaces as gamuts

The colour gamut of a device is sometimes visualized as a volume of colours, typically in CIELab or CIELuv colour spaces, or as a project in the CIEXYZ colour space producing a 2D xy chromaticity diagram (CD). particularly the luminance of the primary colours. Typically a colour space specifies three (x,y) coordinates to define the three primary colours it uses. The triangle formed by the three coordinates encloses the gamut of colours that the device can reproduce. The table below shows the RGB coordinates for various colour spaces in the CIE chromaticity diagram, shown on the 2D diagram in Figure 1.

NameR(x)R(y)G(x)G(y)B(x)B(y)%CIE
sRGB0.640.330.30.60.150.0635
Adobe RGb0.640.330.210.710.150.0650
ProPhoto0.73470.26530.15960.84040.03660.000191
Apple RGB0.62500.340.280.59500.15500.0733.5
NTSC RGB0.670.330.210.710.140.0854
CIE RGB0.73460.26650.28110.70770.17060.0059

Note that colour gamuts are 3D which is more informative than the 2D CD – it captures the nuances of the colour space, particularly the luminance of the primary colours. However the problem with 3D is that it is not easy to plot, and hence the reason a 2D representation is often used (the missing dimension is brightness).

Two of the most common gamuts in the visual industry are sRGB, and Adobe RGB (which are also colour spaces). Each of these gamuts references a different range of colours, suited to particular applications and devices. sRGB is perhaps the most common gamut used in modern electronic devices. It is gamut that covers a good range of colours for average viewing needs, so much so that it is the default standard for the web, and most images taken using digital cameras. The largest RGB working space, ProPhoto is an RGB color space developed by Kodak, and encompasses 90% of the possible colours in the CIE XYZ chromaticity diagram.

Gamut mapping is the conversion of one devices colour space to another. For example the case where an image stored as sRGB is to be reproduced on a print medium with a CMYK colour space. The objective of a gamut mapping algorithm is to translate colours in the input space to achievable colours in the output space. The gamut of an output device depends on its technology. For example, colour monitors are not always capable of displaying all colours associated with sRGB.

Colour profiles

On many systems the colour gamut is described as a colour profile, and more specifically is associated with an ICC Color Profile, which is a standardized system put in place by the international colour consortium. Such profiles help convert the colours in the designated colour space associated with an image to the device. For example the standard profile on Apple laptops is “Color LCD”.Some of the most common RGB ICC profiles are sRGB (sRGB IEC61966-2.1).