CN105957008A - Panoramic image real-time stitching method and panoramic image real-time stitching system based on mobile terminal - Google Patents
Panoramic image real-time stitching method and panoramic image real-time stitching system based on mobile terminal Download PDFInfo
- Publication number
- CN105957008A CN105957008A CN201610303650.2A CN201610303650A CN105957008A CN 105957008 A CN105957008 A CN 105957008A CN 201610303650 A CN201610303650 A CN 201610303650A CN 105957008 A CN105957008 A CN 105957008A
- Authority
- CN
- China
- Prior art keywords
- present frame
- transformation matrix
- panoramic picture
- frame
- panoramic image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 239000011159 matrix material Substances 0.000 claims abstract description 92
- 230000009466 transformation Effects 0.000 claims abstract description 61
- 230000008569 process Effects 0.000 claims abstract description 9
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 230000008859 change Effects 0.000 claims description 16
- 238000001514 detection method Methods 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 13
- 230000008878 coupling Effects 0.000 claims description 7
- 238000010168 coupling process Methods 0.000 claims description 7
- 238000005859 coupling reaction Methods 0.000 claims description 7
- 238000013507 mapping Methods 0.000 claims description 7
- 238000000205 computational method Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 238000013501 data transformation Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 2
- 230000004044 response Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a panoramic image real-time stitching method and a panoramic image real-time stitching system based on a mobile terminal. The method comprises the following steps: detecting the feature points of a current frame of an acquired video stream; carrying out feature point matching on an obtained feature point attribute sequence of the current frame and a feature point attribute sequence of a previous frame of the video stream; calculating a transformation matrix between the current frame and the previous frame according to the result of matching; and finally, stitching the current frame and the previous frame in real time according to the transformation matrix, repeating the cycle, sequentially getting a preliminary panoramic image, an intermediate panoramic image and a final panoramic image, and previewing and displaying the preliminary panoramic image or the intermediate panoramic image or the final panoramic image in real time. The algorithm is simple, and there is a small amount of calculation. A panoramic image can be stitched and previewed in real time in the process of panoramic shooting. The method and the system have low requirement on hardware configuration, and are particularly applicable to high-resolution real-time panoramic shooting in mobile terminals with medium or low configuration such as mobile phones.
Description
Technical field
The present invention relates to technical field of image processing, a kind of panoramic picture based on mobile terminal is real
Time joining method and application the method system.
Background technology
Panoramic mosaic technology in current mobile device is generally by the image of the multiple angles of collection, and root
According to characteristic point matching method, original high-resolution image is carried out the behaviour that image registration and fusion etc. are computationally intensive
Make, generate panoramic picture.
Firstly, since image registration and fusion is computationally intensive, so can only be by the side of interval sampling
Formula gathers image.Owing to the interval of image acquisition is relatively big, first cause lap less, to characteristic point
Alignment is disadvantageous, easily produces the ghost image of panorama sketch.
Secondly as the scenery difference of sampling is relatively big, the white balance of usual photographic head and exposure parameter are
There is bigger change, so post processing part typically also needs to use some exposure compensatings and complexity
Method mixed image, so that image transition is natural, thus it is complete to allow user need to wait a longer time acquisition
Scape image.
Again, such acquisition mode determines and is difficult to allow mobile device generate some intermediate object program for user
Live preview, makes the unpredictable final splicing result of user.
Even if it addition, provide the preview of splicing result, owing to it is computationally intensive, it is impossible to obtain the pre-of smoothness
Look at effect, and the real-time of Preview results cannot be met, final fusion results generally have some not from
Right sawtooth and integration region smear sense, and, require higher for hardware configuration.
Computationally intensive due to real time panoramic, so at mobile terminals such as the mobile phone of low and middle-end configuration or cameras
The middle high-resolution real time panoramic that realizes needs to overcome bigger technical difficulty.
Summary of the invention
The present invention solves the problems referred to above, it is provided that a kind of panoramic picture based on mobile terminal splices in real time
Method and system, algorithm is simple, and real-time is good, it is possible to realize splicing live preview, Consumer's Experience in real time
More preferably.
For achieving the above object, the technical solution used in the present invention is:
First, the present invention provides a kind of real-time joining method of panoramic picture based on mobile terminal, and it includes
Following steps:
10. obtain the video flowing of preview in real time, and the present frame of the video flowing obtained is carried out feature spot check
Survey, obtain the characteristic point sequence of attributes of this present frame；
The characteristic point of the characteristic point sequence of attributes of described present frame with the former frame of described video flowing is belonged to by 20.
Property sequence carries out Feature Points Matching；
30. calculate the change between described present frame and described former frame according to the result of described Feature Points Matching
Change matrix；
Described present frame is spliced in real time with described former frame, at the beginning of obtaining by 40. according to described transformation matrix
Step panoramic picture, and this preliminary panoramic picture is carried out preview show；
50. repeat steps 10,20,30, and according to described transformation matrix by described present frame with described at the beginning of
Step panoramic picture splices in real time, obtains intermediate panoramic image, and carries out pre-to this intermediate panoramic image
Look at display；
60. repeat step 50, and according to described transformation matrix by described present frame and described intermediate panoramic figure
As splicing in real time, obtain final panoramic picture, and this final panoramic picture is carried out preview show.
Preferably, in described step 10, described feature point detection farther includes:
11. pairs of described present frames carry out gray processing process, generate gray-scale map；
12. pairs of gray level images carry out down-sampled, obtain the gray-scale map under one group of different scale, form an ash
Degree figure pyramid；
The gray-scale map of 13. pairs of different scales carries out FAST feature point detection, obtains described characteristic point attribute sequence
Row.
Preferably, described characteristic point sequence of attributes includes: position attribution, direction attribute, scale properties with
And description of correspondence.
Preferably, in described step 20, described Feature Points Matching farther includes:
21. utilize KNN algorithm, previous by the characteristic point sequence of attributes of described present frame and described video flowing
The characteristic point sequence of attributes of frame is mated, obtain between described present frame with described former frame mate right
Quantity；
The 22. pairs of described couplings to quantity carry out threshold calculations, if described quantity is more than or equal to presetting threshold
Value, then the match is successful；Otherwise it fails to match.
Preferably, in described step 30, calculate described present frame according to the result of described Feature Points Matching
And the transformation matrix between described former frame, its computational methods are:
31. if the result of described Feature Points Matching is that the match is successful, then calculate institute by RANSAC algorithm
State mate to optimum homography matrix, and the internal reference matrix combining demarcation carries out being calculated described change
Change matrix；
If the result of 32. described Feature Points Matching is that it fails to match, then calculate described according to gyro data
Transformation matrix between present frame and described former frame.
Preferably, in described step 40 or 50 or 60, according to described transformation matrix by described present frame
Splice in real time with described former frame or described preliminary panoramic picture or described intermediate panoramic image, enter one
Step includes:
41. carry out forward projection according to described transformation matrix to described present frame, and calculate described present frame
Characteristic point mapping coordinates on the face of cylinder that focal length is radius or sphere；
42. carry out back projection according to described transformation matrix to described mapping coordinates, and calculate described pinup picture
Coordinate 2D on the tangent perspective plane of the described face of cylinder or described sphere is projected in described present frame
Interpolated coordinates；
43. obtain, by the method for line sampling, the pixel that described interpolated coordinates is corresponding in described present frame
Value；
44. according to described interpolated coordinates and the pixel value of correspondence thereof, by described present frame and described former frame or
Described preliminary panoramic picture or described intermediate panoramic image splice.
Preferably, in described splicing, the most further to described present frame and described former frame or described
Preliminary panoramic picture or described intermediate panoramic image carry out emergence mixed processing, the side of this emergence mixed processing
Method includes:
71. calculate the side-play amount between described present frame and described former frame；
72. calculate the mixed weight-value between described present frame and described former frame according to described side-play amount；
73., according to described mixed weight-value and described interpolated coordinates and the pixel value of correspondence thereof, are calculated
The end value of mixing eventually.
Secondly, the present invention provides a kind of real-time splicing system of panoramic picture based on mobile terminal, comprising:
Feature point detection module, for the video flowing of acquisition preview in real time, and working as the video flowing obtained
Front frame carries out feature point detection, obtains the characteristic point sequence of attributes of this present frame；
Feature Points Matching module, for by the characteristic point sequence of attributes of described present frame and described video flowing
The characteristic point sequence of attributes of former frame carries out Feature Points Matching；
Transformation matrix computing module, for according to the result of described Feature Points Matching calculate described present frame with
Transformation matrix between described former frame；
Panoramic mosaic module, for carrying out described present frame with described former frame according to described transformation matrix
Splicing, obtains preliminary panoramic picture in real time；Or, according to described transformation matrix by described present frame and institute
State preliminary panoramic picture to splice in real time, obtain intermediate panoramic image；Or according to described transformation matrix
Described present frame is spliced in real time with described intermediate panoramic image, obtains final panoramic picture；
Panorama preview module, for this preliminary panoramic picture or described intermediate panoramic image or described
Final panoramic picture carries out preview and shows.
Preferably, described transformation matrix computing module farther includes the first transformation matrix computing module and
Two transformation matrix computing modules:
If the result of described Feature Points Matching is that the match is successful, then calls described first transformation matrix and calculate mould
Block, by RANSAC algorithm calculate described coupling to optimum homography matrix, and combine demarcation
Internal reference matrix carries out being calculated described transformation matrix；
If the result of described Feature Points Matching is that it fails to match, then calls described second transformation matrix and calculate mould
Block, calculates the transformation matrix between described present frame and described former frame according to gyro data.
Preferably, also include emergence mixing module, in splicing to described present frame and described
Former frame or described preliminary panoramic picture or described intermediate panoramic image carry out emergence mixed processing.
The invention has the beneficial effects as follows:
The real-time joining method of a kind of based on mobile terminal panoramic picture of the present invention and system, it is by real
Time obtain the video flowing of preview, and the present frame of video flowing obtained is carried out feature point detection, is somebody's turn to do
The characteristic point sequence of attributes of present frame, then by the characteristic point sequence of attributes of described present frame and described video
The characteristic point sequence of attributes of the former frame of stream carries out Feature Points Matching, and according to the knot of described Feature Points Matching
Fruit calculates the transformation matrix between described present frame and described former frame, will finally according to described transformation matrix
Described present frame splices in real time with described former frame, so moves in circles, and obtains preliminary panorama successively
Image, intermediate panoramic image, final panoramic picture, and in real time to preliminary panoramic picture or intermediate panoramic
Image or final panoramic picture carry out preview and show, not only algorithm is simple, amount of calculation is little, it is possible to realize
Every frame resolution is 5,000,000 pixels (2560*1920), the processing speed of frame per second 30fps, thus realizes
Splicing, the live preview in real time of panoramic picture is carried out during pan-shot, and for hardware configuration
Require relatively low, be particularly well-suited to realize high-resolution in the mobile terminal such as mobile phone of low and middle-end configuration the most complete
Scape shoots；Further, make spliced panoramic picture more smooth by emergence mixed processing and do not lose
The definition of original image.
Accompanying drawing explanation
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes of the present invention
Point, the schematic description and description of the present invention is used for explaining the present invention, is not intended that the present invention's
Improper restriction.In the accompanying drawings:
Fig. 1 is the general flow chart of the present invention real-time joining method of panoramic picture based on mobile terminal；
Fig. 2 is the structural representation of the present invention real-time splicing system of panoramic picture based on mobile terminal；
Fig. 3 is the general flow chart of the real-time splicing of panoramic picture of the present invention one specific embodiment.
Detailed description of the invention
For making the purpose of the embodiment of the present invention, technical scheme and advantage clearer, below in conjunction with this
Accompanying drawing in bright embodiment, is clearly and completely described the technical scheme in the embodiment of the present invention,
Obviously, described embodiment is a part of embodiment of the present invention rather than whole embodiments.Should
Understanding, specific embodiment described herein only in order to explain the present invention, is not intended to limit the present invention.
Based on the embodiment in the present invention, those of ordinary skill in the art are not under making creative work premise
The every other embodiment obtained, broadly falls into the scope of protection of the invention.
As it is shown in figure 1, the real-time joining method of a kind of based on mobile terminal panoramic picture of the present invention, its
Comprise the following steps:
10. obtain the video flowing of preview in real time, and the present frame of the video flowing obtained is carried out feature spot check
Survey, obtain the characteristic point sequence of attributes of this present frame；
The characteristic point of the characteristic point sequence of attributes of described present frame with the former frame of described video flowing is belonged to by 20.
Property sequence carries out Feature Points Matching；
30. calculate the change between described present frame and described former frame according to the result of described Feature Points Matching
Change matrix；
Described present frame is spliced in real time with described former frame, at the beginning of obtaining by 40. according to described transformation matrix
Step panoramic picture, and this preliminary panoramic picture is carried out preview show；
50. repeat steps 10,20,30, and according to described transformation matrix by described present frame with described at the beginning of
Step panoramic picture splices in real time, obtains intermediate panoramic image, and carries out pre-to this intermediate panoramic image
Look at display；
60. repeat step 50, and according to described transformation matrix by described present frame and described intermediate panoramic figure
As splicing in real time, obtain final panoramic picture, and this final panoramic picture is carried out preview show.
In described step 10, described feature point detection farther includes:
11. pairs of described present frames carry out gray processing process, generate gray-scale map；
12. pairs of gray level images carry out down-sampled, obtain the gray-scale map under one group of different scale, form an ash
Degree figure pyramid；
The gray-scale map of 13. pairs of different scales carries out FAST feature point detection, obtains described characteristic point attribute sequence
Row.
Wherein, described characteristic point sequence of attributes includes: position attribution, direction attribute, scale properties and
Corresponding description.The obtaining step of this feature point sequence of attributes specifically includes:
Step1. gray level image pyramid is set up；
Step2. the target detection dividing each pyramid maximum is counted, wherein, in pyramid each layer with
The quantity ratio of last layer is consistent with image scaling ratio；
Step3.FAST Corner Detection goes out candidate feature point；
Step4. first round screening is done according to FAST response；
Step5. can be poor due to FAST response value representativeness, on a kind of circumference, τ inspection, more smart
True is the response of gradient harris, needs the angle point detected, recalculates its response value；
Step6. according to the response value calculated in step5, candidate angular is carried out last and takes turns screening；
Step7. by IC (x square and y square), angle point direction is calculated；
Step8. each tomographic image is carried out Gaussian Blur and antinoise processes, obtain denoising image；
Step9., in the denoising image basis of step8, description of a length of 64 is selected to extract position.
The position extracted is centered by angle point.
In described step 20, described Feature Points Matching farther includes:
21. utilize KNN algorithm, previous by the characteristic point sequence of attributes of described present frame and described video flowing
The characteristic point sequence of attributes of frame is mated, obtain between described present frame with described former frame mate right
Quantity；
The 22. pairs of described couplings to quantity carry out threshold calculations, if described quantity is more than or equal to presetting threshold
Value, then the match is successful；Otherwise it fails to match.
Specific as follows:
Step1. the hamming distance describing son is calculated as the distance measure of two characteristic points, it is assumed that figure
A and figure B is two two field pictures subject to registration, and in figure A, characteristic point i is to the distance of characteristic point j in figure B,
Go through characteristic point in figure B, obtain in figure B in figure A nearest two characteristic points Minj of i characteristic point,
SndMinj, note distance is MinDistance, sndMinDistance, wherein, MinDistance <
sndMinDistance。
If step2. meeting MinDistance < sndMinDistance* (1-threadhold),
Threshold is predetermined threshold value, and its value is that 0-1. then Minj matches i, preserves matching result,
Otherwise it fails to match.
Step3. repeat step step1-step2, go through all characteristic points in figure A, obtain the spy of coupling
Levy a little.
In described step 30, according to the result of described Feature Points Matching calculate described present frame with described before
Transformation matrix between one frame, its computational methods are:
31. if the result of described Feature Points Matching is that the match is successful, then calculate institute by RANSAC algorithm
State mate to optimum homography matrix, and the internal reference matrix combining demarcation carries out being calculated described change
Change matrix；
If the result of 32. described Feature Points Matching is that it fails to match, then calculate described according to gyro data
Transformation matrix between present frame and described former frame.
Wherein, described step 31 farther includes:
Step1. utilize the feature point pairs (mate to) matched, obtain optimal list by RANSAC algorithm
Answer mapping matrix H.
Step2. obtain the camera internal reference matrix K of demarcation, and combine described optimal homography matrix H,
Obtaining the transformation matrix R of correspondence, its computing formula is:
R=Kinv*H*K；
In above-mentioned formula, K represents described camera internal reference matrix, and H represents described optimal homography matrix,
Kinv represents the inverse matrix of described internal reference matrix K, and R represents the spin moment of described present frame and described former frame
Battle array.
Described step 32 farther includes:
Step1. mobile terminal attitudes vibration information from described present frame to described former frame is obtained,
Described attitudes vibration information includes: under mobile terminal local Coordinate System, and present frame attitude is relative to benchmark
Three axle anglecs of rotation α of attitude ', β ', γ ', former frame attitude relative to reference attitude three axles rotate
Angle [alpha] ", β ", γ ", and the time interval Δ t between present frame to former frame；
Step2. according to α ', β ', γ ', α ", β ", γ " and Δ t be calculated described gyro
Instrument in described present frame attitude relative to the transformation matrix between described former frame attitude.
In described step 40 or 50 or 60, according to described transformation matrix by described present frame with described before
One frame or described preliminary panoramic picture or described intermediate panoramic image splice in real time, farther include:
41. carry out forward projection according to described transformation matrix to described present frame, and calculate described present frame
Characteristic point mapping coordinates on the face of cylinder that focal length is radius or sphere；
42. carry out back projection according to described transformation matrix to described mapping coordinates, and calculate described pinup picture
Coordinate 2D on the tangent perspective plane of the described face of cylinder or described sphere is projected in described present frame
Interpolated coordinates；
43. obtain, by the method for line sampling, the pixel that described interpolated coordinates is corresponding in described present frame
Value；
44. according to described interpolated coordinates and the pixel value of correspondence thereof, by described present frame and described former frame or
Described preliminary panoramic picture or described intermediate panoramic image splice.
Wherein, the forward projection (map forward) of described step 41 and the reverse of described step 42 are thrown
Shadow (map backward) is all that a kind of input picture maps to the location of pixels between output image
(warp), for two dimensional image, it is simply that calculate original image each pixel in the target image
Position one to one, as a example by the face of cylinder:
Forward projection planar view as, in upright projection to the face of cylinder, obtaining each point on view plane
(x, y, z=1) project to the face of cylinder coordinate (u, v, w), wherein (xw,yw,zw) it is point on view plane
At the coordinate that camera coordinates is fastened, its computing formula is as follows:
V=scale × (π-arccos (w))；
Rear orientation projection is the inverse process of forward projection in fact, by coordinate on the face of cylinder (u, v), upright projection
On view plane, obtain coordinate corresponding to artwork (x, y), its computing formula is as follows:
xw=sin (u/scale)；
zw=cos (u/scale)；
yw=v/scale；
In above-mentioned formula, R is spin matrix, and K is the internal reference matrix of photographic head, and scale is cylindrical radius
(generally focal length).
Further, in the present embodiment, also by OpenGL or OpenCL or OpenGL and OpenCL
Described forward projection and described back projection are carried out hardware-accelerated by collaborative work mode, can carry greatly
The execution efficiency of high algorithm, thus realize splicing in real time and rendering.
Also include step 70, in described splicing further to described present frame and described former frame or
Described preliminary panoramic picture or described intermediate panoramic image carry out emergence mixed processing (blend), this emergence
The method of mixed processing includes:
71. calculate the side-play amount between described present frame and described former frame；
72. calculate the mixed weight-value between described present frame and described former frame according to described side-play amount；
73., according to described mixed weight-value and described interpolated coordinates and the pixel value of correspondence thereof, are calculated
The end value of mixing eventually.
Wherein, the computational methods of the mixed weight-value in described step 72 are as follows:
weightcnt=X/L；
weightpre=1-weightcnt；
In above-mentioned formula, weightcntRepresent the mixed weight-value of described present frame, weightpreRepresent institute
Stating the mixed weight-value of former frame, L represents described side-play amount, and X represents described present frame and described former frame
Between splicing line position.
The end value of the final mixing of described step 73, its computational methods are as follows:
Valueout=weightcnt*Valuecnt+weightpre*Valuepre；
In formula, ValueoutRepresent the end value of calculated final mixing, weightcntRepresent institute
State the mixed weight-value of present frame, weightpreRepresent the mixed weight-value of described former frame, ValuecntRepresent
The pixel value that the interpolated coordinates of described present frame is corresponding, ValuepreRepresent the interpolated coordinates of described former frame
Corresponding pixel value.
As in figure 2 it is shown, the present invention provides a kind of real-time splicing system of panoramic picture based on mobile terminal,
Comprising:
Feature point detection modules A, for the video flowing of acquisition preview in real time, and to the video flowing obtained
Present frame carries out feature point detection, obtains the characteristic point sequence of attributes of this present frame；
Feature Points Matching module B, for by the characteristic point sequence of attributes of described present frame and described video flowing
The characteristic point sequence of attributes of former frame carry out Feature Points Matching；
Transformation matrix computing module C, calculates described present frame for the result according to described Feature Points Matching
And the transformation matrix between described former frame；
Panoramic mosaic module D, for entering described present frame with described former frame according to described transformation matrix
Row splicing in real time, obtains preliminary panoramic picture；Or, according to described transformation matrix by described present frame with
Described preliminary panoramic picture splices in real time, obtains intermediate panoramic image；Or according to described conversion square
Described present frame is spliced in real time by battle array with described intermediate panoramic image, obtains final panoramic picture；
Panorama preview module E, for this preliminary panoramic picture or described intermediate panoramic image or institute
State final panoramic picture to carry out preview and show.
Described transformation matrix computing module C farther includes the first transformation matrix computing module C1 and second and becomes
Change matrix calculus module C2:
If the result of described Feature Points Matching is that the match is successful, then calls described first transformation matrix and calculate mould
Block C1, by RANSAC algorithm calculate described coupling to optimum homography matrix, and combine mark
Fixed internal reference matrix carries out being calculated described transformation matrix；
If the result of described Feature Points Matching is that it fails to match, then calls described second transformation matrix and calculate mould
Block C2, calculates the transformation matrix between described present frame and described former frame according to gyro data.
In the present embodiment, also include emergence mixing module F, be used in splicing described present frame
Emergence mixed processing is carried out with described former frame or described preliminary panoramic picture or described intermediate panoramic image,
So that do not have obvious crenellated phenomena in splicing effect.
Described mobile terminal includes: mobile phone, digital camera or panel computer etc. are configured with the equipment of photographic head.
It should be noted that each embodiment in this specification all uses the mode gone forward one by one to describe, each
What embodiment stressed is all the difference with other embodiments, identical similar between each embodiment
Part see mutually.For system embodiment, due to itself and embodiment of the method basic simlarity,
So describe is fairly simple, relevant part sees the part of embodiment of the method and illustrates.Further, exist
Herein, term " includes ", " comprising " or its any other variant are intended to the bag of nonexcludability
Contain, so that include that the process of a series of key element, method, article or equipment not only include that those are wanted
Element, but also include other key elements being not expressly set out, or also include for this process, method,
Article or the intrinsic key element of equipment.In the case of there is no more restriction, statement " include one
It is individual ... " key element that limits, it is not excluded that including the process of described key element, method, article or setting
Other identical element is there is also in Bei.It addition, one of ordinary skill in the art will appreciate that realize above-mentioned
All or part of step of embodiment can be completed by hardware, it is also possible to instructed relevant by program
Hardware complete, described program can be stored in a kind of computer-readable recording medium, mentioned above
Storage medium can be read only memory, disk or CD etc..
Described above illustrate and describes the preferred embodiments of the present invention, it should be understood that the present invention not limits to
In form disclosed herein, be not to be taken as the eliminating to other embodiments, and can be used for various other
Combination, amendment and environment, and can be in invention contemplated scope herein, by above-mentioned teaching or relevant neck
Technology or the knowledge in territory are modified.And the change that those skilled in the art are carried out and change are without departing from the present invention
Spirit and scope, the most all should be in the protection domain of claims of the present invention.
Claims (10)
1. the real-time joining method of panoramic picture based on mobile terminal, it is characterised in that include following
Step:
10. obtain the video flowing of preview in real time, and the present frame of the video flowing obtained is carried out feature spot check
Survey, obtain the characteristic point sequence of attributes of this present frame；
The characteristic point of the characteristic point sequence of attributes of described present frame with the former frame of described video flowing is belonged to by 20.
Property sequence carries out Feature Points Matching；
30. calculate the change between described present frame and described former frame according to the result of described Feature Points Matching
Change matrix；
Described present frame is spliced in real time with described former frame, at the beginning of obtaining by 40. according to described transformation matrix
Step panoramic picture, and this preliminary panoramic picture is carried out preview show；
50. repeat steps 10,20,30, and according to described transformation matrix by described present frame with described at the beginning of
Step panoramic picture splices in real time, obtains intermediate panoramic image, and carries out pre-to this intermediate panoramic image
Look at display；
60. repeat step 50, and according to described transformation matrix by described present frame and described intermediate panoramic figure
As splicing in real time, obtain final panoramic picture, and this final panoramic picture is carried out preview show.
A kind of real-time joining method of panoramic picture based on mobile terminal the most according to claim 1,
It is characterized in that: in described step 10, described feature point detection farther includes:
11. pairs of described present frames carry out gray processing process, generate gray-scale map；
12. pairs of gray level images carry out down-sampled, obtain the gray-scale map under one group of different scale, form an ash
Degree figure pyramid；
The gray-scale map of 13. pairs of different scales carries out FAST feature point detection, obtains described characteristic point attribute sequence
Row.
A kind of panoramic picture based on mobile terminal side of splicing in real time the most according to claim 1 and 2
Method, it is characterised in that: described characteristic point sequence of attributes includes: position attribution, direction attribute, yardstick belong to
Property and correspondence description son.
A kind of panoramic picture based on mobile terminal side of splicing in real time the most according to claim 1 and 2
Method, it is characterised in that: in described step 20, described Feature Points Matching farther includes:
21. utilize KNN algorithm, previous by the characteristic point sequence of attributes of described present frame and described video flowing
The characteristic point sequence of attributes of frame is mated, obtain between described present frame with described former frame mate right
Quantity；
The 22. pairs of described couplings to quantity carry out threshold calculations, if described quantity is more than or equal to presetting threshold
Value, then the match is successful；Otherwise it fails to match.
A kind of real-time joining method of panoramic picture based on mobile terminal the most according to claim 4,
It is characterized in that: in described step 30, calculate described present frame according to the result of described Feature Points Matching
And the transformation matrix between described former frame, its computational methods are:
31. if the result of described Feature Points Matching is that the match is successful, then calculate institute by RANSAC algorithm
State mate to optimum homography matrix, and the internal reference matrix combining demarcation carries out being calculated described change
Change matrix；
If the result of 32. described Feature Points Matching is that it fails to match, then calculate described according to gyro data
Transformation matrix between present frame and described former frame.
A kind of real-time joining method of panoramic picture based on mobile terminal the most according to claim 1,
It is characterized in that: in described step 40 or 50 or 60, according to described transformation matrix by described present frame
Splice in real time with described former frame or described preliminary panoramic picture or described intermediate panoramic image, enter one
Step includes:
41. carry out forward projection according to described transformation matrix to described present frame, and calculate described present frame
Characteristic point mapping coordinates on the face of cylinder that focal length is radius or sphere；
42. carry out back projection according to described transformation matrix to described mapping coordinates, and calculate described pinup picture
Coordinate 2D on the tangent perspective plane of the described face of cylinder or described sphere is projected in described present frame
Interpolated coordinates；
43. obtain, by the method for line sampling, the pixel that described interpolated coordinates is corresponding in described present frame
Value；
44. according to described interpolated coordinates and the pixel value of correspondence thereof, by described present frame and described former frame or
Described preliminary panoramic picture or described intermediate panoramic image splice.
A kind of real-time joining method of panoramic picture based on mobile terminal the most according to claim 6,
It is characterized in that: in described splicing, the most further to described present frame and described former frame or described
Preliminary panoramic picture or described intermediate panoramic image carry out emergence mixed processing, the side of this emergence mixed processing
Method includes:
71. calculate the side-play amount between described present frame and described former frame；
72. calculate the mixed weight-value between described present frame and described former frame according to described side-play amount；
73., according to described mixed weight-value and described interpolated coordinates and the pixel value of correspondence thereof, are calculated
The end value of mixing eventually.
8. the real-time splicing system of panoramic picture based on mobile terminal, it is characterised in that including:
Feature point detection module, for the video flowing of acquisition preview in real time, and working as the video flowing obtained
Front frame carries out feature point detection, obtains the characteristic point sequence of attributes of this present frame；
Feature Points Matching module, for by the characteristic point sequence of attributes of described present frame and described video flowing
The characteristic point sequence of attributes of former frame carries out Feature Points Matching；
Transformation matrix computing module, for according to the result of described Feature Points Matching calculate described present frame with
Transformation matrix between described former frame；
Panoramic mosaic module, for carrying out described present frame with described former frame according to described transformation matrix
Splicing, obtains preliminary panoramic picture in real time；Or, according to described transformation matrix by described present frame and institute
State preliminary panoramic picture to splice in real time, obtain intermediate panoramic image；Or according to described transformation matrix
Described present frame is spliced in real time with described intermediate panoramic image, obtains final panoramic picture；
Panorama preview module, for this preliminary panoramic picture or described intermediate panoramic image or described
Final panoramic picture carries out preview and shows.
A kind of real-time splicing system of panoramic picture based on mobile terminal the most according to claim 8,
It is characterized in that: described transformation matrix computing module farther includes the first transformation matrix computing module and
Two transformation matrix computing modules:
If the result of described Feature Points Matching is that the match is successful, then calls described first transformation matrix and calculate mould
Block, by RANSAC algorithm calculate described coupling to optimum homography matrix, and combine demarcation
Internal reference matrix carries out being calculated described transformation matrix；
If the result of described Feature Points Matching is that it fails to match, then calls described second transformation matrix and calculate mould
Block, calculates the transformation matrix between described present frame and described former frame according to gyro data.
A kind of real-time splicing system of panoramic picture based on mobile terminal the most according to claim 8,
It is characterized in that: also include emergence mixing module, in splicing to described present frame and described
Former frame or described preliminary panoramic picture or described intermediate panoramic image carry out emergence mixed processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610303650.2A CN105957008B (en) | 2016-05-10 | 2016-05-10 | The real-time joining method of panoramic picture and system based on mobile terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610303650.2A CN105957008B (en) | 2016-05-10 | 2016-05-10 | The real-time joining method of panoramic picture and system based on mobile terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105957008A true CN105957008A (en) | 2016-09-21 |
| CN105957008B CN105957008B (en) | 2019-08-16 |
Family
ID=56914332
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610303650.2A Active CN105957008B (en) | 2016-05-10 | 2016-05-10 | The real-time joining method of panoramic picture and system based on mobile terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105957008B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106504196A (en) * | 2016-11-29 | 2017-03-15 | 微鲸科技有限公司 | A kind of panoramic video joining method and equipment based on space sphere |
| CN106548446A (en) * | 2016-09-29 | 2017-03-29 | 北京奇艺世纪科技有限公司 | A kind of method and device of the pinup picture on Spherical Panorama Image |
| CN106846249A (en) * | 2017-01-22 | 2017-06-13 | 浙江得图网络有限公司 | A kind of panoramic video joining method |
| CN107071281A (en) * | 2017-04-19 | 2017-08-18 | 珠海市魅族科技有限公司 | Panorama shooting method and device |
| CN107659769A (en) * | 2017-09-07 | 2018-02-02 | 维沃移动通信有限公司 | A kind of image pickup method, first terminal and second terminal |
| CN108022204A (en) * | 2016-10-31 | 2018-05-11 | 央视国际网络无锡有限公司 | A kind of method that cylinder panorama video is converted to spherical panoramic video |
| CN110569927A (en) * | 2019-09-19 | 2019-12-13 | 浙江大搜车软件技术有限公司 | Method, terminal and computer equipment for scanning and extracting panoramic image of mobile terminal |
| CN110648281A (en) * | 2019-09-23 | 2020-01-03 | 华南农业大学 | Method, device and system for generating field panorama, server and storage medium |
| CN111445416A (en) * | 2020-03-30 | 2020-07-24 | 南京泓众电子科技有限公司 | Method and device for generating high-dynamic-range panoramic image |
| WO2021129669A1 (en) * | 2019-12-23 | 2021-07-01 | RealMe重庆移动通信有限公司 | Image processing method and system, electronic device, and computer-readable medium |
| CN113301274A (en) * | 2021-07-28 | 2021-08-24 | 北京海兰信数据科技股份有限公司 | Ship real-time video panoramic stitching method and system |
| CN113375555A (en) * | 2018-07-02 | 2021-09-10 | 广西电网有限责任公司北海供电局 | Power line clamp measuring method and system based on mobile phone image |
| WO2021238317A1 (en) * | 2020-05-29 | 2021-12-02 | 华为技术有限公司 | Panoramic image capture method and device |
| WO2022032538A1 (en) * | 2020-08-12 | 2022-02-17 | 深圳市大疆创新科技有限公司 | Panorama photographing method, apparatus and system, and computer-readable storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651268B (en) * | 2020-05-14 | 2023-01-31 | 武汉兰丁智能医学股份有限公司 | Quick processing system for microscopic image |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104519340A (en) * | 2014-12-30 | 2015-04-15 | 余俊池 | Panoramic video stitching method based on multi-depth image transformation matrix |
| CN105262949A (en) * | 2015-10-15 | 2016-01-20 | 浙江卓锐科技股份有限公司 | Multifunctional panorama video real-time splicing method |
| CN105376530A (en) * | 2015-10-27 | 2016-03-02 | 南京泓众电子科技有限公司 | Panorama video image generation device based on fixed position |
| CN105389787A (en) * | 2015-09-30 | 2016-03-09 | 华为技术有限公司 | Panorama image stitching method and device |
-
2016
- 2016-05-10 CN CN201610303650.2A patent/CN105957008B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104519340A (en) * | 2014-12-30 | 2015-04-15 | 余俊池 | Panoramic video stitching method based on multi-depth image transformation matrix |
| CN105389787A (en) * | 2015-09-30 | 2016-03-09 | 华为技术有限公司 | Panorama image stitching method and device |
| CN105262949A (en) * | 2015-10-15 | 2016-01-20 | 浙江卓锐科技股份有限公司 | Multifunctional panorama video real-time splicing method |
| CN105376530A (en) * | 2015-10-27 | 2016-03-02 | 南京泓众电子科技有限公司 | Panorama video image generation device based on fixed position |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10789672B2 (en) | 2016-09-29 | 2020-09-29 | Beijing Qiyi Century Science & Technology Co., Ltd. | Method and device for performing mapping on spherical panoramic image |
| CN106548446A (en) * | 2016-09-29 | 2017-03-29 | 北京奇艺世纪科技有限公司 | A kind of method and device of the pinup picture on Spherical Panorama Image |
| CN106548446B (en) * | 2016-09-29 | 2019-08-09 | 北京奇艺世纪科技有限公司 | A kind of method and device of the textures on Spherical Panorama Image |
| CN108022204A (en) * | 2016-10-31 | 2018-05-11 | 央视国际网络无锡有限公司 | A kind of method that cylinder panorama video is converted to spherical panoramic video |
| CN108022204B (en) * | 2016-10-31 | 2019-09-13 | 央视国际网络无锡有限公司 | A kind of method that cylinder panorama video is converted to spherical panoramic video |
| CN106504196A (en) * | 2016-11-29 | 2017-03-15 | 微鲸科技有限公司 | A kind of panoramic video joining method and equipment based on space sphere |
| CN106846249A (en) * | 2017-01-22 | 2017-06-13 | 浙江得图网络有限公司 | A kind of panoramic video joining method |
| CN107071281A (en) * | 2017-04-19 | 2017-08-18 | 珠海市魅族科技有限公司 | Panorama shooting method and device |
| CN107659769A (en) * | 2017-09-07 | 2018-02-02 | 维沃移动通信有限公司 | A kind of image pickup method, first terminal and second terminal |
| CN107659769B (en) * | 2017-09-07 | 2019-07-26 | 维沃移动通信有限公司 | A kind of image pickup method, first terminal and second terminal |
| CN113375555A (en) * | 2018-07-02 | 2021-09-10 | 广西电网有限责任公司北海供电局 | Power line clamp measuring method and system based on mobile phone image |
| CN110569927A (en) * | 2019-09-19 | 2019-12-13 | 浙江大搜车软件技术有限公司 | Method, terminal and computer equipment for scanning and extracting panoramic image of mobile terminal |
| CN110648281A (en) * | 2019-09-23 | 2020-01-03 | 华南农业大学 | Method, device and system for generating field panorama, server and storage medium |
| CN110648281B (en) * | 2019-09-23 | 2021-03-16 | 华南农业大学 | Method, device and system for generating field panorama, server and storage medium |
| WO2021129669A1 (en) * | 2019-12-23 | 2021-07-01 | RealMe重庆移动通信有限公司 | Image processing method and system, electronic device, and computer-readable medium |
| CN111445416A (en) * | 2020-03-30 | 2020-07-24 | 南京泓众电子科技有限公司 | Method and device for generating high-dynamic-range panoramic image |
| CN111445416B (en) * | 2020-03-30 | 2022-04-26 | 南京泓众电子科技有限公司 | Method and device for generating high-dynamic-range panoramic image |
| WO2021238317A1 (en) * | 2020-05-29 | 2021-12-02 | 华为技术有限公司 | Panoramic image capture method and device |
| WO2022032538A1 (en) * | 2020-08-12 | 2022-02-17 | 深圳市大疆创新科技有限公司 | Panorama photographing method, apparatus and system, and computer-readable storage medium |
| CN113301274A (en) * | 2021-07-28 | 2021-08-24 | 北京海兰信数据科技股份有限公司 | Ship real-time video panoramic stitching method and system |
| CN113301274B (en) * | 2021-07-28 | 2021-11-09 | 北京海兰信数据科技股份有限公司 | Ship real-time video panoramic stitching method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105957008B (en) | 2019-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105957008A (en) | Panoramic image real-time stitching method and panoramic image real-time stitching system based on mobile terminal | |
| CN104333675B (en) | A kind of panorama electronic image stabilization method based on spherical projection | |
| Saurer et al. | Rolling shutter stereo | |
| CN104699842B (en) | Picture display method and device | |
| CN101155267B (en) | Apparatus, method, and medium for generating panoramic image | |
| WO2020062434A1 (en) | Static calibration method for external parameters of camera | |
| US20060078214A1 (en) | Image processing based on direction of gravity | |
| US20060078215A1 (en) | Image processing based on direction of gravity | |
| CN107833179A (en) | The quick joining method and system of a kind of infrared image | |
| CN106060418A (en) | IMU information-based wide dynamic image fusion method | |
| CN109314752A (en) | Effective determination of light stream between image | |
| CN107154014A (en) | A kind of real-time color and depth Panorama Mosaic method | |
| CN104094318A (en) | System for filming a video movie | |
| CN107452016A (en) | For handling the method and apparatus of view data and driver assistance system for vehicle | |
| Mahdy et al. | Projector calibration using passive stereo and triangulation | |
| CN107730558A (en) | 360 ° of vehicle-running recording systems and method based on two-way fish eye camera | |
| CN109788270B (en) | 3D-360-degree panoramic image generation method and device | |
| CN107534731A (en) | Image processing apparatus and image processing method | |
| Fan et al. | RS-DPSNet: Deep plane sweep network for rolling shutter stereo images | |
| CN102081796B (en) | Image splicing method and device thereof | |
| US20090059018A1 (en) | Navigation assisted mosaic photography | |
| CN110278366A (en) | A kind of panoramic picture weakening method, terminal and computer readable storage medium | |
| US20220351650A1 (en) | Systems and methods for adding persistence to single photon avalanche diode imagery | |
| Gava et al. | Dense scene reconstruction from spherical light fields | |
| Hold-Geoffroy et al. | A perceptual measure for deep single image camera and lens calibration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |