1 采用针孔相机模型

1.1 畸变矫正

图像畸变矫正后，图像边缘处会有黑色区域，尤其对于视场角较大的相机，边缘的黑色区域会比较大，如图1所示${}^{[1]}$。

图1 图像畸变矫正后出现黑色边缘

图像畸变矫正前，需要生成新的内参数，生成新内参数的函数原型${}^{[2]}$如下:

/** @brief Returns the new camera intrinsic matrix based on the free scaling parameter.

@param cameraMatrix Input camera intrinsic matrix.
@param distCoeffs Input vector of distortion coefficients
\f$\distcoeffs\f$. If the vector is NULL/empty, the zero distortion coefficients are
assumed.
@param imageSize Original image size.
@param alpha Free scaling parameter between 0 (when all the pixels in the undistorted image are
valid) and 1 (when all the source image pixels are retained in the undistorted image). See
#stereoRectify for details.
@param newImgSize Image size after rectification. By default, it is set to imageSize .
@param validPixROI Optional output rectangle that outlines all-good-pixels region in the
undistorted image. See roi1, roi2 description in #stereoRectify .
@param centerPrincipalPoint Optional flag that indicates whether in the new camera intrinsic matrix the
principal point should be at the image center or not. By default, the principal point is chosen to
best fit a subset of the source image (determined by alpha) to the corrected image.
@return new_camera_matrix Output new camera intrinsic matrix.

The function computes and returns the optimal new camera intrinsic matrix based on the free scaling parameter.
By varying this parameter, you may retrieve only sensible pixels alpha=0 , keep all the original
image pixels if there is valuable information in the corners alpha=1 , or get something in between.
When alpha\>0 , the undistorted result is likely to have some black pixels corresponding to
"virtual" pixels outside of the captured distorted image. The original camera intrinsic matrix, distortion
coefficients, the computed new camera intrinsic matrix, and newImageSize should be passed to
#initUndistortRectifyMap to produce the maps for #remap .
 */
CV_EXPORTS_W Mat getOptimalNewCameraMatrix( InputArray cameraMatrix, InputArray distCoeffs,
                                            Size imageSize, double alpha, Size newImgSize = Size(),
                                            CV_OUT Rect* validPixROI = 0,
                                            bool centerPrincipalPoint = false);

改变cv::getOptimalNewCameraMatrix函数的参数double alpha，可调节保留的黑边或裁剪的图像范围。

• 如果alpha为0，则将所有黑边切除，保留最大内切矩形，损失最多的视场范围。
• 如果alpha大于0、小于1，则删除部分黑边，损失部分视场范围。
• 如果alpha为1，则保留原图像中的所有像素，包含所有黑边，不损失视场范围。

图像畸变矫正的过程如下:

cv::Size original_size;
cv::Size new_size;               
cv::Mat distor_img;
cv::Mat intrin_matrix;
cv::Mat distor_coeff;
// TODO: 为以上变量赋值

cv::Mat map_x, map_y;
const double alpha = 0.0;
// 生成新的内参数
cv::Mat new_intrinsic = cv::getOptimalNewCameraMatrix(intrin_matrix, distor_coeff,
                                                      original_size, alpha, new_size);
cv::initUndistortRectifyMap(intrin_matrix, distor_coeff, cv::Mat(),
                            new_intrinsic, new_size, CV_16SC2, map_x, map_y);

cv::Mat undistorted_img;   //畸变矫正后的图像
cv::remap(distor_img, undistorted_img, map_x, map_y, cv::INTER_LINEAR);
cv::imshow("undistorted_img", undistorted_img);
cv::waitKey(1000);

函数cv::getOptimalNewCameraMatrix和cv::initUndistortRectifyMap调用一次即可，函数cv::remap可重复多次调用。

1.2 三维点投影

关于3D点到图像的投影，例如3D激光雷达点云到图像的投影，可采用cv::projectPoints函数，其原型${}^{[2]}$如下:

CV_EXPORTS_W void projectPoints( InputArray objectPoints,
                                 InputArray rvec, InputArray tvec,
                                 InputArray cameraMatrix, InputArray distCoeffs,
                                 OutputArray imagePoints,
                                 OutputArray jacobian = noArray(),
                                 double aspectRatio = 0 );

需要特别注意的问题是，如果将3D点投影到畸变矫正后的图像中，输入cv::projectPoints函数的相机内参数必须是畸变矫正后的新内参数，即上述代码中的new_intrinsic。否则，将可能导致投影错误，尤其对于视场角较大（例如水平FOV大于110度）的相机，畸变矫正前后，内参变化较大。

如果投影过程需要包含畸变，直接将原始的相机内参数输入到上述函数即可，调用方式如下：

std::vector<cv::Point3f> object_points;
cv::Mat rvec, tvec;
cv::Mat original_intrin_matrix, original_dist_coeff;
// TODO: 对上述变量赋值
std::vector<cv::Point2f> proj_points;
cv::projectPoints(object_points, proj_points, rvec, tvec, original_intrin_matrix, original_dist_coeff);

如果对图像进行了畸变矫正，将第1.1节中的新内参数new_intrinsic输入cv::projectPoints，调用方式如下：

std::vector<cv::Point3f> object_points;
cv::Mat rvec, tvec;
cv::Mat new_intrinsic, dist_coeff_all_zero;  // dist_coeff_all_zero的值全是0
// TODO: 对上述变量赋值
std::vector<cv::Point2f> proj_points;
cv::projectPoints(object_points, proj_points, rvec, tvec, new_intrinsic, dist_coeff_all_zero);

需要特别注意的问题是，将点云投影到图像之前，要删除相机后方的点云，否则可能出现错误的图像投影现象。

2 采用鱼眼相机模型

2.1 畸变矫正

对于鱼眼相机的畸变矫正，需要使用下列函数生成新的内参数，其原型${}^{[2]}$如下:

CV_EXPORTS_W void estimateNewCameraMatrixForUndistortRectify(InputArray K, InputArray D, const Size &image_size, InputArray R,
        OutputArray P, double balance = 0.0, const Size& new_size = Size(), double fov_scale = 1.0);

在该函数中，

• 变量balance的值为0时，则切除所有黑边；
• balance的值为1时，则保留所有黑边。

畸变矫正的过程如下：

cv::Size original_size;
cv::Size new_size;
cv::Mat distor_img;
cv::Mat intrin_matrix;
cv::Mat distor_coeff;
// TODO: 为以上变量赋值

cv::Mat new_intr_mat_fisheye;
double balance = 0.0;
cv::Mat map1, map2;
// 生成新的内参数
cv::fisheye::estimateNewCameraMatrixForUndistortRectify(intrin_matrix, distor_coeff, original_size, cv::Matx33d::eye(), new_intr_mat_fisheye, balance, new_size);
cv::fisheye::initUndistortRectifyMap(intrin_matrix, distor_coeff, cv::Matx33d::eye(), new_intr_mat_fisheye,
                                     original_size, CV_16SC2, map1, map2);

cv::Mat undistorted_img;   //畸变矫正后的图像
cv::remap(distor_img, undistorted_img, map1, map1, cv::INTER_LINEAR);
cv::imshow("undistorted_img", undistorted_img);
cv::waitKey(1000);

同样，函数cv::fisheye::estimateNewCameraMatrixForUndistortRectify和cv::fisheye::initUndistortRectifyMap调用一次即可，函数cv::remap可重复多次调用。

2.2 三维点投影

关于3D点到图像的投影，例如3D激光雷达点云到图像的投影，可采用cv::fisheye::projectPoints函数，其原型${}^{[2]}$如下:

    /** @brief Projects points using fisheye model

    @param objectPoints Array of object points, 1xN/Nx1 3-channel (or vector\<Point3f\> ), where N is
    the number of points in the view.
    @param imagePoints Output array of image points, 2xN/Nx2 1-channel or 1xN/Nx1 2-channel, or
    vector\<Point2f\>.
    @param affine
    @param K Camera intrinsic matrix \f$cameramatrix{K}\f$.
    @param D Input vector of distortion coefficients \f$\distcoeffsfisheye\f$.
    @param alpha The skew coefficient.
    @param jacobian Optional output 2Nx15 jacobian matrix of derivatives of image points with respect
    to components of the focal lengths, coordinates of the principal point, distortion coefficients,
    rotation vector, translation vector, and the skew. In the old interface different components of
    the jacobian are returned via different output parameters.

    The function computes projections of 3D points to the image plane given intrinsic and extrinsic
    camera parameters. Optionally, the function computes Jacobians - matrices of partial derivatives of
    image points coordinates (as functions of all the input parameters) with respect to the particular
    parameters, intrinsic and/or extrinsic.
     */
    CV_EXPORTS void projectPoints(InputArray objectPoints, OutputArray imagePoints, const Affine3d& affine,
        InputArray K, InputArray D, double alpha = 0, OutputArray jacobian = noArray());

    /** @overload */
    CV_EXPORTS_W void projectPoints(InputArray objectPoints, OutputArray imagePoints, InputArray rvec, InputArray tvec,
        InputArray K, InputArray D, double alpha = 0, OutputArray jacobian = noArray());

如果投影过程需要包含畸变，直接将原始的相机内参数输入到上述函数即可，调用方式如下：

std::vector<cv::Point3f> object_points;
cv::Mat rvec, tvec;
cv::Mat original_intrin_matrix, original_dist_coeff;
// TODO: 对上述变量赋值
std::vector<cv::Point2f> proj_points;
cv::fisheye::projectPoints(object_points, proj_points, rvec, tvec, original_intrin_matrix, original_dist_coeff);

如果对图像进行了畸变矫正，鱼眼相机的图像就相当于“针孔相机图像”了，就不能直接用cv::fisheye::projectPoints函数进行投影了，而是将第2.1节中的新内参数new_intr_mat_fisheye输入cv::projectPoints，调用方式如下：

std::vector<cv::Point3f> object_points;
cv::Mat rvec, tvec;
cv::Mat new_intr_mat_fisheye, dist_coeff_all_zero;  // dist_coeff_all_zero的值全是0
// TODO: 对上述变量赋值
std::vector<cv::Point2f> proj_points;
cv::projectPoints(object_points, proj_points, rvec, tvec, new_intr_mat_fisheye, dist_coeff_all_zero);

参考文献

[1] https://github.com/Aaron20127/Camera-lidar-joint-calibration
[2] https://github.com/opencv/opencv