A review of pansharpening methods based on deep learning
-
摘要: 随着遥感技术的快速发展与广泛应用,对获取的遥感图像质量有了更高的要求。但是,难以直接获得高空间分辨率多光谱遥感图像。为了结合不同成像传感器的信息,获得高质量的图像,图像融合技术应运而生。空谱遥感图像融合是一种获取高空间分辨率多光谱图像的有效方法,目前已有许多学者针对空谱遥感图像融合展开研究,取得了较多成果。近年来,深度学习理论得到了快速发展,广泛应用于空谱遥感图像融合。为了让学者们能够更系统地了解空谱遥感图像融合的现状,推动空谱遥感图像融合的发展,首先对常用的遥感卫星作了介绍,并简单总结了传统的经典空谱图像融合算法; 其次,从监督学习、无监督学习和半监督学习3个方面,重点对基于深度学习的空谱图像融合算法进行了阐述,还对损失函数进行了描述与分析; 然后,为了证明基于深度学习方法的优越性以及分析损失函数的影响,开展了遥感图像融合实验; 最后,对基于深度学习的空谱图像融合方法进行了展望。Abstract: With the fast development and wide application of remote sensing technology, remote sensing images with higher quality are needed. However, it is difficult to directly acquire high-resolution, multispectral remote sensing images. To obtain high-quality images by integrating the information from different imaging sensors, pansharpening technology emerged. Pansharpening is an effective method used to obtain multispectral images with high spatial resolution. Many scholars have studied this method and obtained fruitful achievements. In recent years, deep learning theory has developed rapidly and has been widely applied in pansharpening. This study aims to systematically introduce the progress in pansharpening and promote its development. To this end, this study first introduced the traditional, classical pansharpening methods, followed by commonly used remote sensing satellites. Then, this study elaborated on the pansharpening methods based on deep learning from the perspective of supervised learning, unsupervised learning, and semi-supervised learning. After that, it described and analyzed loss functions. To demonstrate the superiority of the pansharpening methods based on deep learning and analyze the effects of loss functions, this study conducted remote sensing image fusion experiments. Finally, this study presented the future prospects of the pansharpening methods based on deep learning.
-
Key words:
- remote sensing image /
- pansharpening /
- deep learning /
- convolutional neural network
-
-
[1] 肖亮, 刘鹏飞, 李恒. 多源空—谱遥感图像融合方法进展与挑战[J]. 中国图象图形学, 2020, 25(5):851-863.
[2] Xiao L, Liu P F, Li H. Progress and challenges in the fusion of multisource spatial-spectral remote sensing images[J]. Journal of Image and Graphics, 2020, 25(5):851-863.
[3] Ghassemian H. A review of remote sensing image fusion methods[J]. Information Fusion, 2016, 32:75-89.
[4] 张丽霞, 曾广平, 宣兆成. 多源图像融合方法的研究综述[J]. 计算机工程与科学, 2021:1-14.
[5] Zhang L X, Zeng G P, Xuan Z C. A survey of fusion methods for multi-source image[J]. Computer Engineering and Science, 2021:1-14.
[6] 杨丽萍, 马孟, 谢巍, 等. 干旱区Landsat8全色与多光谱数据融合算法评价[J]. 国土资源遥感, 2019, 31(4):11-19.doi:10.6046/gtzyyg.2019.04.02.
[7] Yang L P, Ma M, Xie W, et al. Fusion algorithm evaluation of Landsat8 panchromatic and multispetral images in arid regions[J]. Remote Sensing for Land and Resources, 2019, 31(4):11-19.doi:10.6046/gtzyyg.2019.04.02.
[8] Weiss M, Jacob F, Duveiller G. Remote sensing for agricultural applications:A meta-review[J]. Remote Sensing of Environment, 2020, 236:111402-111420.
[9] 刘万军, 高健康, 曲海成, 等. 多尺度特征增强的遥感图像舰船目标检测[J]. 自然资源遥感, 2021, 33(3):97-106.doi:10.6046/zrzyyg.20200372.
[10] Liu W J, Gao J K, Qu H C, et al. Ship detection based on multi-scale feature enhancement of remote sensing images[J]. Remote Sensing for Natural Resources, 2021, 33(3):97-106.doi:10.6046/zrzyyg.20200372.
[11] 王琳, 李迅, 包云轩, 等. 遥感技术在交通气象灾害监测中的应用进展[J]. 国土资源遥感, 2018, 30(4):1-7.doi:10.6046/gtzyyg.2018.04.01.
[12] Wang L, Li X, Bao Y X, et al. Research progress of remote sensing application on transportation meteorological disasters[J]. Remote Sensing and for Land and Resources, 2018, 30(4):1-7.doi:10.6046/gtzyyg.2018.04.01.
[13] 王洪斌, 肖嵩, 曲家慧, 等. 基于多分支CNN高光谱与全色影像融合处理[J]. 光学学报, 2021, 41(7):55-63.
[14] Wang H B, Xiao S, Qu J H, et al. Pansharpening based on multi-branch CNN[J]. Acta Optica Sinica, 2021, 41(7):55-63.
[15] 张良培, 沈焕锋. 遥感数据融合的进展与前瞻[J]. 遥感学报, 2016, 20(5):1050-1061.
[16] Zhang L P, Shen H F. Progress and future of remote sensing data fusion[J]. Journal of Remote Sensing, 2016, 20(5):1050-1061.
[17] 李树涛, 李聪妤, 康旭东. 多源遥感图像融合发展现状与未来展望[J]. 遥感学报, 2021, 25(1):148-166.
[18] Li S T, Li C Y, Kang X D. Development status and future prospects of multi-source remote sensing image fusion[J]. National Remote Sensing Bulletin, 2021, 25(1):148-166.
[19] 张立福, 彭明媛, 孙雪剑, 等. 遥感数据融合研究进展与文献定量分析(1992—2018)[J]. 遥感学报, 2019, 23(4):603-619.
[20] Zhang L F, Peng M Y, Sun X J, et al. Progress and bibliometric analysis of remote sensing data fusion methods (1992—2018)[J]. Journal of Remote Sensing, 2019, 23(4):603-619.
[21] Meng X, Shen H, Li H, et al. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis:Practical discussion and challenges[J]. Information Fusion, 2019, 46:102-113.
[22] Javan F D, Samadzadegan F, Mehravar S, et al. A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171:101-117.
[23] Vivone G, Mura M D, Garzelli A, et al. A new benchmark based on recent advances in multispectral pansharpening:Revisiting pansharpening with classical and emerging pansharpening methods[J]. IEEE Geoscience and Remote Sensing Magazine, 2020, 9(1):53-81.
[24] Huang W, Xiao L, Wei Z, et al. A new pan-sharpening method with deep neural networks[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(5):1037-1041.
[25] Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7):594-615.
[26] Shahdoosti H R, Ghassemian H. Combining the spectral PCA and spatial PCA fusion methods by an optimal filter[J]. Information Fusion, 2016, 27:150-160.
[27] Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening:U.S.,US09/069232[P]. 2000-01-04.
[28] Ghahremani M, Ghassemian H. Nonlinear IHS:A promising method for pan-sharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(11):1606-1610.
[29] Garzelli A, Nencini F, Capobianco L. Optimal MMSE pan sharpening of very high resolution multispectral images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 46(1):228-236.
[30] Aiazzi B, Alparone L, Baronti S, et al. MTF-tailored multiscale fusion of high-resolution MS and pan imagery[J]. Photogrammetric Engineering and Remote Sensing, 2006, 72(5):591-596.
[31] Vivone G, Restaino R, Chanussot J. Full scale regression-based injection coefficients for panchromatic sharpening[J]. IEEE Transactions on Image Processing, 2018, 27(7):3418-3431.
[32] Khan M M, Chanussot J, Condat L, et al. Indusion:Fusion of multispectral and panchromatic images using the induction scaling technique[J]. IEEE Geoscience and Remote Sensing Letters, 2008, 5(1):98-102.
[33] Dong L, Yang Q, Wu H, et al. High quality multi-spectral and panchromatic image fusion technologies based on curvelet transform[J]. Neurocomputing, 2015, 159:268-274.
[34] El-Mezouar M C, Kpalma K, Taleb N, et al. A pan-sharpening based on the non-subsampled contourlet transform:Application to WorldView-2 imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(5):1806-1815.
[35] 曹义亲, 杨世超, 谢舒慧. 基于NSST的PCNN-SR卫星遥感图像融合方法[J]. 航天控制, 2020, 38(2):44-50.
[36] Cao Y Q, Yang S C, Xie S H. Satellite remote sensing image fusion method based on NSST and PCNN-SR[J]. Aerospce Control, 2020, 38(2):44-50.
[37] 吴一全, 王志来. 混沌蜂群优化的NSST域多光谱与全色图像融合[J]. 遥感学报, 2017, 21(4):549-557.
[38] Wu Y Q, Wang Z L. Multispectral and panchromatic image fusion using chaotic Bee Colony optimization in NSST domain[J]. Journal of Remote Sensing, 2017, 21(4):549-557.
[39] Li S, Yang B. A new pan-sharpening method using a compressed sensing technique[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 49(2):738-746.
[40] Yin H. PAN-guided cross-resolution projection for local adaptive sparse representation-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7):4938-4950.
[41] 李成轶, 田淑芳. 基于字典学习的遥感影像超分辨率融合方法[J]. 国土资源遥感, 2017, 29(1):50-56.doi:10.6046/gtzyyg.2017.01.08.
[42] Li C Y, Tian S F. Super-resolution fusion method for remote sensing image based on dictionary learning[J]. Remote Sensing and for Land and Resources, 2017, 29(1):50-56.doi:10.6046/gtzyyg.2017.01.08.
[43] Yin H. Sparse representation based pansharpening with details injection model[J]. Signal Processing, 2015, 113:218-227.
[44] Li S, Yin H, Fang L. Remote sensing image fusion via sparse representations over learned dictionaries[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(9):4779-4789.
[45] Scarpa G, Vitale S, Cozzolino D. Target-adaptive CNN-based pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9),5443-5457.
[46] Wei Y, Yuan Q, Shen H, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10),1795-1799.
[47] Yang J, Fu X, Hu Y, et al. PanNet:A deep network architecture for pan-sharpening[C]// Proceedings of the IEEE International Conference on Computer Vision.IEEE,Venice,Italy, 2017:5449-5457.
[48] Zhang H, Ma J. GTP-PNet:A residual learning network based on gradient transformation prior for pansharpening[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 172:223-239.
[49] Yang Y, Tu W, Huang S, et al. PCDRN:Progressive cascade deep residual network for pansharpening[J]. Remote Sensing, 2020, 12(4):676.
[50] Huang W, Feng J, Wang H, et al. A new architecture of densely connected convolutional networks for pan-sharpening[J]. ISPRS International Journal of Geo-Information, 2020, 9(4):242.
[51] Peng J, Liu L, Wang J, et al. PSMD-Net:A novel pan-sharpening method based on a multiscale dense network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(6):4957-4971.
[52] Luo S, Zhou S, Qi Y. CSAFNET:Channel similarity attention fusion network for multispectral pansharpening[J]. IEEE Geoscience and Remote Sensing Letters, 2020.
[53] 孔爱玲, 张承明, 李峰, 等. 基于知识引导的遥感影像融合方法[J]. 自然资源遥感, 2022, 34(2):47-55.doi:10.6046/zrzyyg.2021179.
[54] Kong A L, Zhang C M, Li F, et al. Knowledge-based remote sensing imagery fusion method[J]. Remote Sensing for Natural Resources, 2022, 34(2):47-55.doi:10.6046/zrzyyg.2021179.
[55] Jiang M, Shen H, Li J, et al. A differential information residual convolutional neural network for pansharpening[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 163:257-271.
[56] Lei D, Chen H, Zhang L, et al. NLRNet:An efficient nonlocal attention resnet for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60:5401113.
[57] Shao Z, Cai J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(5),1656-1669.
[58] Liu X, Liu Q, Wang Y. Remote sensing image fusion based on two-stream fusion network[J]. Information Fusion, 2020,55,1-15.
[59] Fu S, Meng W, Jeon G, et al. Two-path network with feedback connections for pan-sharpening in remote sensing[J]. Remote Sensing, 2020, 12(10),1674.
[60] He L, Xi D, Li J, et al. A spectral-aware convolutional neural network for pansharpening[J]. Applied Sciences, 2020, 10(17),5809.
[61] Zhang Y, Liu C, Sun M, et al. Pan-sharpening using an efficient bidirectional pyramid network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8),5549-5563.
[62] Fang S, Wang X, Zhang J, et al. Pan-sharpening based on parallel pyramid convolutional neural network[C]// IEEE International Conference on Image Processing (ICIP), 2020:453-457.
[63] 方帅, 方赛华, 姚宏亮. 基于深度金字塔网络的Pan-Sharpening算法[J]. 计算机辅助设计与图形学学报, 2019, 31(10):1831-1837.
[64] Fang S, Fang S H, Yao H L. Pan-sharpening based on a deep pyramid network[J]. Journal of Computer-Aided Design and Computer Graphics, 2019, 31(10):1831-1837.
[65] Ronneberger O, Fischer P, Brox T. U-Net:Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer,Cham, 2015:234-241.
[66] Yao W, Zeng Z, Lian C, et al. Pixel-wise regression using U-Net and its application on pansharpening[J]. Neurocomputing, 2018, 312:364-371.
[67] Wang W, Zhou Z, Liu H, et al. MSDRN:Pansharpening of multispectral images via multi-scale deep residual network[J]. Remote Sensing, 2021, 13(6):1200.
[68] Lai Z, Chen L, Jeon G, et al. Real-time and effective pan-sharpening for remote sensing using multi-scale fusion network[J]. Journal of Real-Time Image Processing, 2021:1-17.
[69] Benzenati T, Kallel A, Kessentini Y. Two stages pan-sharpening details injection approach based on very deep residual networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(6),4984-4992.
[70] He K, Zhang X, Ren S, et al. Deep residual learning for image reco-gnition[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016:770-778.
[71] Hu J, Du C, Fan S. Two-stage pansharpening based on multi-level detail injection network[J]. IEEE Access, 2020, 8:156442-156455.
[72] Zhang L, Zhang J, Ma J, et al. SC-PNN:Saliency cascade convolutional neural network for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021:1-19.
[73] Li W, Liang X, Dong M. MDECNN:A multiscale perception dense encoding convolutional neural network for multispectral pan-sharpening[J]. Remote Sensing, 2021, 13(3):535.
[74] Yuan Q, Wei Y, Meng X, et al. A multiscale and multidepth convolutional neural network for remote sensing imagery pan-sharpening[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2018, 11(3):978-989.
[75] Hu J, Hu P, Kang X, et al. Pan-sharpening via multiscale dynamic convolutional neural network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(3):2231-2244.
[76] Guo Y, Ye F, Gong H. Learning an efficient convolution neural network for pansharpening[J]. Algorithms, 2019, 12(1):16.
[77] Hu J, Hu P, Wang Z P, et al. Spatial dynamic selection network for remote-sensing image fusion[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19:8013205.
[78] Hu J, He Z, Wu J. Deep self-learning network for adaptive pansharpening[J]. Remote Sensing, 2019, 11(20):2395.
[79] Liu J, Feng Y, Zhou C, et al. PWnet:An adaptive weigh network for the fusion of panchromatic and multispectral images[J]. Remote Sensing, 2020, 12(17):2804.
[80] Deng L J, Vivone G, Jin C, et al. Detail injection-based deep convolutional neural networks for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020:1-16.
[81] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11):139-144.
[82] Ma J, Yu W, Chen C, et al. Pan-GAN:An unsupervised pan-sharpening method for remote sensing image fusion[J]. Information Fusion, 2020, 62:110-120.
[83] Zhou C, Zhang J, Liu J, et al. PercepPan:Towards unsupervised pan-sharpening based on perceptual loss[J]. Remote Sensing, 2020, 12(14):2318.
[84] Xiong Z, Guo Q, Liu M, et al. Pan-sharpening based on convolutional neural network by using the loss function with no-reference[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 14:897-906.
[85] Luo S, Zhou S, Feng Y, et al. Pansharpening via unsupervised convolutional neural networks[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13:4295-4310.
[86] 杜晨光, 胡建文, 胡佩. 半监督卷积神经网络遥感图像融合[J], 电子测量与仪器学报, 2021, 35(6):63-70.
[87] Du C G, Hu J W, Hu P. Semi-supervised convolutional neural network remote sensing image fusion[J]. Journal of Electronic Measurement and Instrumentation, 2021, 35(6):63-70.
[88] Vitale S, Scarpa G. A detail-preserving cross-scale learning strategy for CNN-based pansharpening[J]. Remote Sensing, 2020, 12(3):348.
[89] 黄珊珊, 江倩, 金鑫, 等. 结合双胞胎结构与生成对抗网络的半监督遥感图像融合[J]. 计算机辅助设计与图形学学报, 2021, 33(1):92-105.
[90] Huang S S, Jiang Q, Jin X, et al. Semi-supervised remote sensing image fusion method combining siamese structure with generative adversarial networks[J]. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(1):92-105.
[91] Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C]// International Conference on Machine Learning.PMLR, 2017:214-223.
[92] Mirza M, Osindero S. Conditional generative adversarial nets[J]. Computer Science, 2014:2672-2680.
[93] Liu X, Deng C, Zhao B, et al. Feature-level loss for multispectral pan-sharpening with machine learning[C]// IEEE International Geoscience and Remote Sensing Symposium(IGARSS).IEEE, 2018:8062-8065.
[94] Xu H, Ma J, Shao Z, et al. SDPNet:A deep network for pan-sharpening with enhanced information representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5):4120-4134.
[95] Choi J S, Kim Y, Kim M. S3:A spectral-spatial structure loss for pan-sharpening networks[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(5):829-833.
[96] Eghbalian S, Ghassemian H. Multi spectral image fusion by deep convolutional neural network and new spectral loss function[J]. International Journal of Remote Sensing, 2018, 39(12):3983-4002.
[97] Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution[C]// European Conference on Computer Vision.Springer,Cham, 2016:694-711.
[98] Bello J L G, Seo S, Kim M. Pan-sharpening with color-aware perceptual loss and guided re-colorization[C]// IEEE International Conference on Image Processing (ICIP).IEEE, 2020:908-912.
[99] Vivone G, Alparone L, Chanussot J, et al. A critical comparison among pansharpening algorithms[J]. IEEE Transactions on Geoscience and Remote Sensing, 2014, 53(5):2565-2586.
[100] Alparone L, Aiazzi B, Baronti S, et al. Multispectral and panchromatic data fusion assessment without reference[J]. Photogrammetric Engineering and Remote Sensing, 2008, 74(2):193-200.
-
计量
- 文章访问数: 2831
- PDF下载数: 1563
- 施引文献: 0
下载: