A robust combination interpolation method for video super-resolution

This paper presents an efficient method for video super-resolution (SR) based on two main ideals: Firstly, input video frames can be separated into two components, nontexturing image and texturing image. Then each component image is applied to a compatible interpolation method to improve the quality of high-resolution (HR) reconstructed frame. Secondly, based on the approach that border regions of image details are the most lossy information regions from the sampling process. Therefore, a task of compensation interpolation is essential to increase the quality of the reconstructed HR images. From these discussions, we proposed an efficient method for video SR by combining the spatial interpolation in different texturing regions and the sampling compensation interpolation to improve the quality of video super-resolution. Our results shown that, the quality of HR frames, reconstructed by the proposed method, is better than that of other methods, , and in recently. The significant contribution is the low complexity of the proposed method. Hence, it is possible to apply the proposed algorithm to real-time video super-resolution applications.


INTRODUCTION
Video super-resolution is to reconstruct and create HR video frames from the input lowresolution (LR) video frames.According to the purpose of increasing in quality of image information, video SR is recently interested as an important research direction.Up to now, there are many authors with their methods for image SR reconstructions, as described in technical overview of Park in 2003.In general, there are two types of SR methods, single-frame SR and multi-frame SR.
In single-frame SR, these methods use interpolation techniques in spatial or frequency domain to upscale the input LR frame.Then the reconstructed HR image is applied by filtering, smoothing and reshaping techniques to decrease noises and increase quality of the reconstructed HR image.There are some typical studies.Li in 2001 used New Edge-Directed Interpolation (NEDI) to interpolate HR images in the wavelet domain.Takeda in 2007 developed a frame work for SR image using Multi-Dimension Kernel Regression Interpolation (KRI).In this method, each pixel in the video frame sequence is approximated with a 2-D local Taylor series.Mallat in 2010 used Sparse Mixing Estimators (SME) to define coefficients for interpolating in the wavelet transform.W. Dong [4] in 2011 used adaptive sparse domain selection and adaptive regularization (ASDS) to interpolate HR images in spatial domain.
In multi-frame SR, the input frames are registered the motion between them.Then based on the registered parameters, the input frames are rearranged in the same co-ordinate.The image information missed in the sampling process will be combined to recover the HR original image.There are some typical studies in multi-frame SR.Keren in 1988 based on the first order Taylor expansion to solve the registration equations.Vandewalle in 2006 and Bui-Thu in 2009 are based on the fact that two shifted images, which are different in the frequency domain only by a phase shift, can be found the shifts from their correlation in the Fourier transform.Lui , in 2011, also has achieved significant progress results.The author has proposed a Bayesian approach for adaptive video super-resolution.The proposed algorithm estimates simultaneously the motion of the details, noise kernel and noise level, while reconstructing the HR frame.
There are many input data for solving the reconstruction problems.Therefore, the multiframe methods are usually more efficient than the single frame methods, and they are possible to reconstruct HR frames in higher quality.However, multi-frame SR methods take more time than single-frame methods for processing time.Thus, it is impossible to apply multi-frame SR for video applications, which demands realtime processing.
Although there are many researches in singleframe SR with advanced results, but they still exist two key problems.Firstly, single-frame SR methods usually create degradation at the ledge of texturing details, as what we see in Figure 1.Therefore, the advanced algorithms have to solve enough good for this problem.Secondly, most recent advanced algorithms can provide high quality in reconstructed HR frame.However, they also take too much time for SR process, so it is impractical for applying the recent SR algorithms to real-time SR video processing.

Our Approach
With the aim of our study for video superresolution, we have to solve the two key problems by decreasing the degradation at the ledge of texturing details and directing to realtime processing for the proposed SR algorithm.
In order to decrease the degradation and increase the quality of the reconstructed HR frame, this paper presents an efficient method for single-frame video SR based on two main ideas: Firstly, input video frames are separated into two components, low-frequency image and high frequency image.Then, compatible interpolation method is applied to each component to improve the quality of HR reconstructed image.Secondly, border regions of image details are the most lossy information regions from the sampling process.Therefore, a task of compensation interpolation is essential to increase the quality of reconstructed HR image.Based on these ideas, we proposed an efficient method for video SR by combining the spatial interpolation in different frequency domains and the sampling compensation interpolation for improving the quality of SR video images.
To directing to real-time processing, the proposed algorithm is innovated from the Cubic interpolation technique.

Overview of Paper
The structure of the paper organizes as follow: in section II, we propose the Spatial Interpolation in Different Texturing Regions method (SIDTR).Next, to increase the accuracy, in section III, we propose the Sampling Compensation Interpolation method (SCI) for the HR image, reconstructed in the section II.In the section IV, we propose the Combining Spatial Interpolation methods (CSI) by combining SIDTR and SCI.The results are present by comparing to different algorithms.In section V, we release the conclusion.

Bilinear interpolation
In mathematics, Bilinear interpolation [10] is an extension of linear interpolation for a uniform 2-dimensional image space.
It is supposed that we want to explain the value of a function f at unknown points P (x, y), but we know 4 points which belong the function: The method is linear interpolation in one direction, then further interpolated for the remaining directions, and the f function is developed as follow: (1)

Cubic interpolation
To overcome the shortcomings of linear interpolation, cubic interpolation [10] is developed.It is based on the concept that the relationship between gray level values of pixels is nonlinear, and expressed as a polynomial of type 3, as follows: (2) One of the characteristics of the cubic polynomial interpolation, we need 16 points surrounding a point (x, y) to solve out the parameters a ij .Based on these parameters, we determine f (x, y).Solving for the images on the large size requires a lot operation, as well as time consuming for processing.
In practical, to increase speed of the algorithm also as grow quality of the interpolated image, the cubic algorithms were developed follow as piecewise cubic polynomials.The piecewise functions, with format as follows: (3) With s = s(x,y) is co-ordinate of the pixels, P i is piecewise polynomials, which are describes as follow: (4) Based on their ability of derivative and continuous boundary conditions between the adjacent pixels we can solve and determine the parameters of P (x).
There are two efficient cubic interpolation algorithms which have been developed in Matlab [10].They are Bicubic Interpolation and Cubic Spline Interpolation.

Bicubic interpolation in Matlab use Piecewise Cubic Interpolation Hermic (Pchip). The algorithm is presented as follows:
Set as the distance of the k th subpixel, Let the first order different of P(s), we have: Let the slope of the interpolated function, P(s).We get: (7) The piecewise cubic interpolation of P(s) in space is: With and in range of , and (3.16) has to satisfy four conditions, as follow: (9) From ( 8) and four conditions of (9), we can find out the parameters of and .Based on the parameters we can define the Pchip interpolation function of .

Cubic Spline interpolation
Cubic Spline interpolation was developed on the basis of interpolation Pchip.This method is added smoothing by using continuous conditions at the curving points.The contents of the following methods: From (8), we have the second derivative of P(s), ( 10)

At
, we have the second derivative in negative direction of P(s) is, (11)

At
, we have the second derivative in positive direction of P(s) at , Similarly, we have the second derivative in negative direction of P(s) at is, The continuous condition at the curving point or also called the curving condition, at , From ( 8), ( 9) and ( 14), we can solve out the values of parameters: d k , d k+1 .

Evaluations about the spatial interpolations
Bilinear interpolation is shown that the simplicity of its algorithm with the linear relationship between the gray levels of pixels.So when the image is interpolated, the detail regions which have the gray values varying linearly have results better than the detail regions which have the gray level values varying non-linearly.Bicubic interpolation has been developed to overcome the defect of Bilinear interpolation.It is good mapping ability for image space.However, Bicubic interpolation is not enough good for smoothing image while Cubic Spline interpolation is stronger than Pchip interpolation for smoothing image by using the curving condition at each pixels.Both Bicubic and Cubic Spline interpolation have low complexity and fast processing time.These methods have been using in pratical for real-time processing video applications.
It can be seen in Figure 2, which illustrates the response of the spatial interpolation techniques.In the area of detail which has gray level variable brokenly, the Cubic Spline interpolation reconstructs of the signal curve better than Bicubic (Pchip) interpolation.In other words, Cubic Spline interpolation allows restoring high frequency components from sampled images better than Bicubic interpolation.However, when applied to the texturing details of image, Cubic Spline interpolation will get the results less than Bicubic (Pchip) interpolation.We can see this illustration in Figure 3.In the border areas of texturing details, where there is mutation of the gray values, the Cubic Spline interpolation created degradation.I, the quality of the advanced algorithms is not much higher than that of Bicubic algorithm.However, the processing time of Bicubic algorithm is very fast to compare with that of the others.The average processing time for upscaling 30 frame sequences in size 144x176 pixels, by CPU Core 3i 2.53 GHz is 600 seconds for NEDI [2] , 1 seconds for Bicubic, 200 seconds for KRI [3], and 1200 seconds for SME [4].
Base on the above evaluations, we proposed a robust spatial interpolation algorithm by spatial interpolation in different texturing regions (SIDTR).The proposed algorithm uses low-pass filter to separate the texturing detail image from the image frame.We get texturing image and non-texturing image.Next it is used the Linear interpolation for texturing image, and the Cubic Spline interpolation for none-texturing image.Then, combining two interpolated images, we get a HR reconstructed image.The proposed method is implemented in the block diagram at Figure 4.For each mono color space of the frame input, firstly, the input video frames are filtered by Low-pass Gaussian Filter.The output image of Low-pass Filter is nontexturing image, f L .Then subtracting the original frame with the non-texturing image, f L , we get the texturing image f H . Next, the texturing image is interpolated by using linear interpolation method, and the non-texturing image is interpolated by using Cubic Spline method, with scale of 2x2.Finally, the two interpolated images are added to create a nature HR image.
To find the optimum cutoff frequency for the Low-pass Gaussian Filter we implemented the SIDTR method for nine standard video sequences.The results are shown in Figure 5.The optimum cutoff frequency is seleted about 20 to 30.

SAMPLING COMPENSATION INTERPOLATION
When sampled, images usually loose much the detail information at border pixels.As illustrated in Figure 6.a), for the sampled image, the sampled positions of pixels are in red points.Figure 6.b) show the image after being sampled.The sub-pixels in green points are lossy information of border regions.It is easy to realize that if the sampled image is zoomed in then the visual quality at the sub-pixels of border regions will be degraded.Consequently, to increase the quality for the upscaled image, we have to interpolate compensation for sampling process.
Through the experimental statistics, we proposed four types of sampling compensation interpolation.For the type I, as shown in Figure 7 a) & b), the above border pixels are in light blue and the below border pixels are in dark blue.The gray levels of pixels, which are called in the same border, are approximate to each other and far different from the gray levels of the opposite pixels.Position 1 and 2 are base-points to interpolate.The condition of border pixels, at the base-point position 1, P(x, y), is present as follow, The thresholds are defined based on the standard mean and deviation of gray level differentiation, as following: , and With, C is a threshold to discriminate the border region between details of the image.Refer to intra prediction algorithms for video compression.We selected C by 10 (for the range of gray levels from 0-255).Figure 9 shows type III and IV of border, with the base-points at the positions 5, 6, 7 and 8. Similarly, It is easy for us to find out the border conditions and border interpolation algorithms of the other base-point pixels.Figure 10 presents the sampling compensation interpolation algorithm for the sub-pixels in region of the below border pixels, at the position 1 of Figure 7.The input LR frame, f (x,y,k), in dark-blue grid, is interpolated into HR frame, f H , in light-blue grid.The k is present for the processing mono color space (R,G,B or Y,U,V).The sub-pixels are interpolated in directions, which are in orange vectors, corresponding to parameters, p.The maximum value of p is 4, which is selected from practice about discriminating ability of the eye for straight edge.

RESULTS OF WORK
To evaluate the result of the proposed interpolation method, we implemented practical experiments on eight standard sequences, as shown in Figure 10.To present power of the proposed algorithm, the video standard sequences were selected in form variety of real image details, from less detail sequences as: Foreman, Soccer and Pamphlet, to more detail sequences as: Mobile, Paris, Stefan, Flower-garden and Husky.The more details video sequence has in, the more complexity program has to solve.
Firstly, the video frame sequences are downsampled, with scale of 2x2, to create input LR frames.Then the LR frames are up-scaled, with scale of 2x2, by the proposed method, SIDTR, as present in section 3, to create HR frame.Next, the reconstructed HR frames are interpolated for sampling compensation by using SCI method to increase the quality of the final reconstructed HR frames, as presented in section 4. To evaluate the quality of HR reconstructed frame, we use PSNR and SSIM [11] measurement between the original HR frames and the HR frames reconstructed by different algorithms.PSNR Results of the proposed methods are present in two columns, SIDTR and CSI, as shown in Table 1.The SIDTR column is the results of the proposed method in section 3. The CSI column is the results of the final proposed method, combining of spatial interpolation methods, SIDTR and SCI.The results of the proposed method, SIDTR, are better to compare that of other algorithms.Furthermore, the SCI method is proved even more efficient by higher the PSNR than that of SIDTR.The average PSNR values of the proposed method CSI are higher than that of other algorithms, 0.27 dB to Bicubic method, 1.57 dB to KRI' results, 0.09 dB to SME' results, and 0.14 dB to ASDS' results.Although our results are sligthly higher than that of the SME algorithm, but our processing time is much faster than that of SME algorithm.The average processing time for one frame, in size 144x176 pixels, is about 0.36 seconds for the proposed algorithm (CSI), 0.3 seconds for Bicubic, while it takes about 243 seconds with the SME algorithm, and 39 seconds with the ASDS algorithm.

Figure 1 .
Figure 1.Illustrates the degradation at ledges of texturing details of the reconstructed HR image by typical singleframe SR algorithms

Figure 2 .Figure 3 .
Figure 2. Illustrate the respond of spatial interpolation techniques

Figure 4 .
Figure 4. Illustration of the spatial interpolation in different frequence domains method

Figure 5 .
Figure 5. Statistic the gain of PSNR versus the frequency cutoff D 0

Figure 6 .Figure 7 .
Figure 6.a) Sampled pixels at the red points, b) the loss information at the green points.

Figure 7 .
Figure 7.a) illustrates the interpolation directions of type I, at two base-point 1 and 2, in orangle vectors.At the base-point 1, in region of below border pixels, have interpolation directions: 45 0 , 26.5 0 , 18.4 0 , 14 0 , and 11.3 0 .At the base-point 2, in region of above border pixels,

Figure 8
Figure 8 presents type II of border, with the base-points to interpolate at the position 3 and 4.Figure9shows type III and IV of border, with the base-points at the positions 5, 6, 7 and 8. Similarly, It is easy for us to find out the border conditions and border interpolation algorithms of the other base-point pixels.

Figure 8 .Figure 10 .
Figure 8. Illustrates the interpolation directions of type II in orange vectors.At the base-point 3, in region of the below border pixels, have the interpolation directions: 135 o , 153.5 o , 161.6 o , 166 o , and 168.7 o .At the base point 4, in region of above border pixels, have

Figure 10 .
Figure 10.Standard video sequences are used for the practical experiments.

Figure 11 .Figure 12 .
Figure 11.Comparison of SR results, applied on Foreman sequency at frame 11.From the top to botton, they are the original frame, the LR frame, the HR frame reconstructed by SME algorithm and the HR frame reconstructed by the proposed method, CSI.