image processing

Image enhancement (2) : the fine details (i.e. sharpening)

/ spqr / Leave a comment

More important than most things in photography is acuity – which is really just a fancy word for sharpness, or even image crispness. Photographs can be blurry for a number of reasons, but usually they are all trumped by lack of proper focusing, which adds a softness to an image. Now in a 3000×4000 pixel image, this blurriness may not be that apparent – and will only manifest itself when an enlargement is made of a section of the image. In terms of photographing landscapes, the overall details in the image may be crisp, however small objects may “seem” blurry, because they are small, and lack detail in any case. Sharpening will also fail to fix large blur artifacts – i.e. it’s not going to remove defocus from a photograph which was not properly focused. It is ideal for making fine details crisper.

Photo apps and “image editing” software often contains some means of improving the sharpness of images. Usually by means of the “cheapest” algorithm in existence – “unsharp masking”. It works by subtracting  a “softened” copy of an image from the original. And by softened, I mean blurred. It basically reduces the lower frequency components of the image. But it is no magical panacea. If there is noise in an image, it too will be attenuated. The benefit of sharpening can often be seen best on images containing fine details. Here are examples of three different types of sharpening algorithms on an image with a lot of fine detail.

Sharpening: original (top-L); USM (top-R); CUSM (bot-L); MS (bot-R)

Three filters are shown here are (i) Unsharp masking (USM), (ii) Cubic Unsharp masking (CUSM) and (iii) Morphological sharpening (MS). Each of these techniques has its benefits and drawbacks, and the final image with improved acuity can only really be judged through visual assessment. Some algorithms may be more attune to sharpening large nonuniform regions (MS), whilst others (USM, CUSM) may be more aligned with sharpening fine details.

Image enhancement (1) : The basics

/ spqr / Leave a comment

Image enhancement involves improving the perceived quality of an image, either for the purpose of aesthetic appeal, or for further processing. Therefore you are either enhancing features within an image, or suppressing artifacts. The basic forms of enhancement include:

  • contrast enhancement: enhancing the overall contrast of an image, to improve dynamic range of intensities.
  • noise suppression: reducing the effect of noise contained within an image
  • sharpening: improving the acuity of features within an image.

These relate to both monochromatic grayscale and colour images (there are additional mechanisms for colour images to deal with enhancing colour). The trick with these enhancement mechanisms is determining when they have achieved the required effect. In image processing this is often a case of the rest being “in the eye of the beholder”. A photograph who’s colour has been enriched may seem pleasing to one person, and over saturated to another. To illustrate, consider the following example. This image is an 8-bit image that is 539×699 pixels in size.

Original Image

Here is the histogram of its pixel intensities:

From both the image and histogram, it is possible to discern that the image lacks contrast, with the majority of gray intensities situated between values 25 and 195. So one of the enhancements could be to improve its contrast. Here is the result of a simple histogram stretching:

Contrast enhancement by stretching the histogram

It may then be interesting to smooth noise in the image or, sharpen the image to enhance the letters in the advertising. The sub-image extracted from the above shows three different techniques.

Forms of image enhancement (sub-image extracted from contrast enhanced image): original (top-left), noise suppression using a 3×3 mean filter (top-right), image sharpening using unsharp masking (bottom-left), and unsharp masking applied after mean filtering (bottom-right).

Can blurry images be fixed?

/ spqr / 1 Comment

Some photographs contain blur which is very challenging to remove. Large scale blur, which is the result of motion, or defocus can’t really be suppressed in any meaningful manner. What can usually be achieved by means of image sharpening algorithms is that finer structures in an image can be made to look more crisp. Take for example the coffee can image shown below, in which the upper lettering on the label in almost in focus, while the lower lettering has the softer appearance associated with de-focus.

The problem with this image is partially the fact that the blur is not uniform. Below are two regions enlarged:containing text from opposite ends of the blur spectrum.

Reducing blur, involves a concept known as image sharpening(which is different from removing motion blur, a much more challenging task). The easiest technique for image sharpening, and the one most often found in software such as Photoshop is known as unsharp masking. It is derived from analog photography, and basically works by subtracting a blurry version of the original image from the original image. It is by no means perfect, and is problematic in images where there is noise, as it tends to accentuate the noise, but it is simple.

Here I am using the “Unsharp Mask” filter from ImageJ. It subtracts a blurred copy of the image and rescales the image to obtain the same contrast of low frequency structures as in the input image. It works in the following manner:

  1. Obtain a Gaussian blurred image, by specifying a blur radius (in the example below the radius = 5).
  2. Filter the blurred image using a “Mask Weight, which determines the strength of filtering. A value from 0.1-0.9. (In the example below, the mask weight =0.4)
  3. Subtract the filtering image from the original image.
  4. Divide the resulting image by (1.0-mask weight) – 0.6 in the case of the example.
1. Original image; 2. Gaussian blurred image (radius=5); 3. Filtered image (multiplied by 0.4); 4. Subtracted image (original-filtered); 5. Final image (subtracted image / 0.6)

If we compare the resulting images, using an enlarged region, we find the unsharp masking filter has slightly improved the sharpness of the text in the image, but this may also be attributed to the slight enhancement in contrast. This part of the original image has less blur though, so let’s apply the filter to the second image.

The original image (left) vs. the filtered image (right)

Below is the result on the second portion of the image. There is next to no improvement in the sharpness of the image. So while it may be possible to slightly improve sharpness, where the picture is not badly blurred, excessive blur is impossible to “remove”. Improvements in acuity may be more to the slight contrast adjustments and how they are perceived by the eye.

More on Mach bands

/ spqr / Leave a comment

Consider the following photograph, taken on a drizzly day in Norway with a cloudy sky, and the mountains somewhat obscured by mist and clouds.

Now let’s look at the intensity image (the colour image has been converted to 8-bit monochrome):

If we look at a region near the top of the mountain, and extract a circular region, there are three distinct regions along a line. To the human eye, these appear as quite uniform regions, which transition along a crisp border. In the profile of a line through these regions though, there are two “cliffs” (Aand B) that marks the shift from one region to the next. Human eyes don’t perceive these “cliffs”.

The Mach bands is an illusion that suggests edges in an image where in fact the intensity is changing in a smooth manner.

The downside to Mach bands is that they are an artificial phenomena produced by the human visual system. As such, it might actually interfere with visual inspection to determine the sharpness contained in an image.

Mach bands and the perception of images

/ spqr / Leave a comment

Photographs, and the results obtained through image processing are at the mercy of the human visual system. A machine cannot interpret how visually appealing an image is, because aesthetic perception is different for everyone. Image sharpening takes advantage of one of the tricks of our visual system. Human eyes see what are termed “Mach bands” at the edges of sharp transitions, which affect how we perceive images. This optical illusion was first explained by Austrian physicist and philosopher Ernst Mach (1838–1916) in 1865. Mach discovered how our eyes leverage the use of contrast to compensate for its inability to resolve fine detail. Consider the image below containing ten squares of differing levels of gray.

Notice how the gray squares appear to scallop, with a lighter band on the left, and a darker band on the right of the squares? This is an optical illusion, in fact the gray squares are all uniform in intensity. To resolve the brain/eyes deficiency in being able to resolve detail, incoming light gets processed in such a manner than the contrast between two different tones is exaggerated. This gives the perception of more detail. The dark and light bands seen on either side of the gradation are the Mach bands. Here is an example of what human eyes see:

What does this have to do with manipulation techniques such as image sharpening? The human brain perceives exaggerated intensity changes near edges – so image sharpening uses this notion to introduce faux Mach bands by amplifying intensity edges. Consider as an example the following  image, which basically shows two mountain sides, one behind the other. Without looking too closely you can see the Mach bands.

Taking a profile perpendicular to the mountain sides provides an indication of the intensity values along the profile, and shows the edges.

The profile shows three plateaus, and two cliffs (the cliffs are ignored by the human eyes). The first plateau is the foreground mountainside, the middle plateau is the mountainside behind that, and the uppermost plateau is some cloud cover. Now we apply an unsharp masking filter to the image, to sharpen the image (radius=10, mask weight=0.4)

Notice how the UM filter has the effect of adding a Mach band to each of the cliff regions.

Creating art-like effects in photographs

/ spqr / Leave a comment

Art-like effects are easy to create in photographs. The idea is to remove textures, and sharpen edges in a photograph to make it appear more like abstract art. Consider the image below. An art-like effect has been created on this image using a filter known as Kuwahara. It has the effect of homogenizing regions of colour, hence you will notice a loss of detail within the image, and colours within a region. It was originally designed to process angiocardiographic images. The usefulness of filters such as Kuwahara is that they remove detail and  increase abstraction. Another example of such a filter is the bilateral filter.

Image (before) and (after)

The Kuwahara is based on local area “flattening”, removing detail in high-contrast regions while protecting shape boundaries in low-contrast areas. The only issue with Kuwahara is that is can produce somewhat “blocky” results. Choosing a different shaped “neighbourhood” will have a different affect on the image. A close-up view of the beetle in the image above shows the distinct edges of the processed image. Note also how some of the features have changed colour slightly (the beetles legs have transformed from dark brown to a pale brown colour), due to the influence of the surrounding pink petal colour.

Close-up detail (before) and (after)

Filters like Kuwahara are also used to remove noise from images.

The perception of enhanced colour images

/ spqr / Leave a comment

Image processing becomes more difficult when you involve colour images. That’s primarily because there is more data involved. With monochrome images, there is really only intensity. With colour images comes chromaticity – and the possibility of modifying the intrinsic colours within an image whilst performing some form of image enhancement. Often, image enhancement in colour images is challenging because the impact of the enhancement is very subjective.

Consider this image of Schynige Platte in Switzerland. It is very colourful, and seems quite vibrant.

The sky however seems too aquamarine. The whole picture seems like some sort of “antique photo filter” has been applied to it. How do we enhance it, and what do we want to enhance? Do we want to make the colours more vibrant? Do we want to improve the contrast?

In the first instance, we merely stretch the histogram to reduce the gray tonality of the image. Everything becomes much brighter, and there is a slight improvement in contrast. There are parts of the image that do seem too yellow, but it is hard to know whether this is an artifact of the original scene, or the photograph (likely an artifact of dispersing yellow flower petals).

Alternatively, we can improve the images contrast. In this case, this is achieved by applying a Retinex filter to the image, and then taking the average of the filter result and the original image. The resulting image is not as “bright”, but shows more contrast, especially in the meadows.

Are either of these enhanced images better? The answer of course is in the eye of the beholder. All three images have certain qualities which are appealing. At the end of the day, improving the aesthetic appeal of a colour image is not an easy task, and there is no “best” algorithm.

How many colours are in a photograph?

/ spqr / Leave a comment

The number of colours in a 24-bit colour image is 256³ or 16,777,216 colours. So how many colours are there in a 8 MP photo? Consider the following beautiful photograph:

Picture of a flower on a Japanese quince tree.
A picture of a flower from a Japanese quince

In this image there are 515,562 unique colours. Here’s what is looks like as a 3D RGB histogram:

Most photographs will not contain 16 million colours (obviously if they have less than 16 MP, that’s a given). If you want to check out some images that do, try allrgb.com. Here is another image with more colours: 1,357,892 to be exact. In reality, very few real everyday photographs contain that amount of hue varieties.

Stained glass window at Metro Charlevoix in Montreal
Stained glass window at Metro Charlevoix in Montreal

Now as the average number of colours humans can perceive is only around a million, having 16 million colours in an image is likely overkill.

Photographic blur you can’t get rid of

/ spqr / Leave a comment

Photographs sometimes contain blur. Sometimes the blur is so bad that it can’t be removed, no matter the algorithm. Algorithms can’t solve everything, even those based on physics. Photography ultimately exists because of the existence of glass lenses – you can’t make any sort of camera without them. Lenses have aberrations (although lenses these days are pretty flawless) – some of these can be dealt with in-situ using corrective algorithms.

Some of this blur is attributable to vibration – no one has hands *that* steady, and tripods aren’t always convenient. Image stabilization, or vibration reduction has done a great job in retaining image sharpness. This is especially important in low-light situations where the photograph may require a longer exposure. The rule of thumb is that a camera should not be hand-held at shutter speeds slower than the equivalent focal length of the lens. So a 200mm lens should not be handheld at speeds slower than 1/200 sec.

Sometimes though, the screen on a digital camera doesn’t tell the full story either. The resolution may be too small to appreciate the sharpness present in the image – and a small amount of blur can reduce the quality of an image. Here is a photograph taken in a low light situation, which, with the wrong settings, resulted in a longer exposure time, and some blur.

Another instance relates to close-up, or macro photography, where the depth-of-field can be quiet shallow. Here is an  example of a close-up shot of the handle of a Norwegian mangle board. The central portion of the horse, near the saddle, is in focus, the parts to either side are not – and this form of blur is impossible to suppress. Ideally in order to have the entire handle in focus, one would have to use a technique known as focus stacking (available in some cameras).

Here is another example of a can where the writing at the top of the can is almost in focus, whereas the writing at the bottom is out-of-focus – due in part to the angle the shot was taken, and the shallow depth of field. It may be possible to sharpen the upper text, but reducing the blur at the bottom may be challenging.

What is a grayscale image?

/ spqr / Leave a comment

If you are starting to learn about image processing then you will likely be dealing with grayscale or 8-bit images. This effectively means that they contain 2^8 or 256 different shades of gray, from 0 (black), to 255 (white). They are the simplest form of image to create image processing algorithms for. There are some image types that are more than 8-bit, e.g. 10-bit (1024 shades of grey), but in reality these are only used in specialist applications. Why? Doesn’t more shades of grey mean a better image? Not necessarily.

The main reason? Blame the human visual system. It is designed for colour, having three cone photoreceptors for conveying colour information that allows humans to perceive approximately 10 million unique colours. It has been suggested that from the perspective of grays, human eyes cannot perceptually see the difference between 32 and 256 graylevel intensities (there is only one photoreceptor with deals with black and white). So 256 levels of gray are really for the benefit of the machine, and although the machine would be just as happy processing 1024, it is likely not needed.

Here is an example. Consider the following photo of the London Blitz, WW2 (New Times Paris Bureau Collection).

blitz

This is a nice grayscale image, because it has a good distribution of intensity values from 0 to 255 (which is not always easy to find). Here is the histogram:

blitzHST

Now consider the image, reduced to 8, 16, 32, 64, and 128 intensity levels. Here is a montage of the results, shown in the form of a region extracted form he original image.

The same image with differing levels of grayscale.

Not that there is very little perceivable difference, except at 8 intensity levels, where the image starts to become somewhat grainy. Now consider a companion of this enlarged region showing only 256 (left) versus 32 (right) intensity levels.

blitz256vs32

Can you see the difference? There is very little difference, especially when viewed in the over context of the complete image.

Many historic images look like they are grayscale, but in fact they are anything but. They may be slightly yellowish or brown in colour, either due to the photographic process, or due to aging of the photographic medium. There is no benefit to processing these type of photographs as colour images however, they should be converted to 8-bit.