Transactions of Materials and Heat Treatment

Title：Grid image restoration technology for non-contact strain measurement based on deep learning

Authors： Dou Yuanhang1 Cui Xuexi2 Zhong Xinping3 Wu Xiangdong1 Wan Min1

Unit： 1.School of Mechanical Engineering and Automation Beihang University 2.Beijing Institute of Aerospace Testing Technology 3.Shanghai Aerospace Equipments Manufacturer Co. Ltd.

KeyWords： strain measurement image processing deep learning image restoration generative adversarial networks

ClassificationCode：TP391

year,vol(issue):pagenumber：2024,49(8):195-204

Abstract：

In non-contact grid strain measurement technology, for the problems of low measurement efficiency and accuracy caused by local defects in grid images, a two-stage grid image restoration model (SN PatchGAN Edge, SPE) based on deep convolutional generative adversarial networks with edge information assisted fusion spectrum normalization and partitioned small patches was proposed. In the first stage of the model, the Canny edge operator was used to extract edge images of grid images containing defects for restoration as a prior aid to the second stage of the model, while in the second stage, the defect areas of the real measured images were repaired. Furthermore, the model training was conducted by using masks with different proportions combined with real images to form a dataset. Finally, the restoration effect of the model on grid image defects was verified by test images. The results show that the SPE model algorithm performs better than the traditional PM (PatchMatch) and deep convolutional generative adversarial networks (DCGANs) algorithms. It can effectively restore the image defects caused by lighting and surface damage, effectively improving the measurement efficiency and accuracy of the results.

Funds：

国家自然科学基金资助项目（51875027）；宝钢汽车用钢开发与应用技术国家重点实验室基金资助项目（2021090602）

作者简介：豆远航（1998-），男，博士研究生 E-mail：douyuanhang@buaa.edu.cn 通信作者：吴向东（1970-），男，博士，副教授 E-mail：xdwu@buaa.edu.cn

Reference：

[1]徐海鹰.蒙皮拉形数值模拟系统的后置处理及软件 [D].北京：北京航空航天大学，2001.

Xu H Y. Application Development and Post-processing of NC Skin Stretch Forming Simulation [D]. Beijing: Beihang University, 2001.

[2]陈晓伟，万敏，王文平．金属板料预应力成形技术研究进展 [J]．锻压技术，2022,47(7):1-8.

Chen X W, Wan M, Wang W P. Research progress on pre-stress forming technology for sheet metal [J]. Forging & Stamping Technology, 2022,47(7):1-8.

[3]万敏，吴向东，李盛，等. 网格应变分析技术及系统 [J]. 锻造与冲压，2006(10): 34-37.

Wan M, Wu X D, Li S, et al. Grid strain analysis technology and system [J]. Foring & Metalforming, 2006(10): 34-37.

[4]吴向东,万敏,李盛.便携式板料应变测量系统GMAS[A]. 第九届全国塑性工程学术年会、第二届全球华人先进塑性加工技术研讨会论文集[C].太原: 2005.

Wu X D, Wan M, Li S, et al. Portable sheet metal strain measurement system GMAS [A]. Proceedings of the 9th National Academic Conference on Plastic Engineering and the 2nd Global Chinese Advanced Plastic Processing Technology Symposium[C].Taiyuan: 2005.

[5]项辉宇，钟约先，吴伯杰. 基于网格试验法的汽车覆盖件冲压成形分析 [J]. 清华大学学报，2004，44(5)：601-604.

Xiang H Y，Zhong Y X，Wu B J. Analysis of the forming characteristics of automobile stamped panel parts based on the experimental grid method [J]. Journal of Tsinghua University, 2004, 44(5):601-604.

[6]张福生, 景作军. 基于计算机视觉的辊弯成型应变测量系统研究[J].机械设计与制造,2010(5):96-97.

Zhang F S, Jing Z J. Research of strain measurement system for roll forming based on computer vision [J]. Machinery Design & Manufacture, 2010(5):96-97.

[7]孙永鹏, 钟佩思, 刘梅, 等. 基于YOLOv4算法的冲压件缺陷检测[J]. 锻压技术,2022,47(1):222-228.

Sun Y P, Zhong P S, Liu M, et al. Defect detection of stamping parts based on YOLOv4 algorithm[J]. Forging & Stamping Technology, 2022,47(1):222-228.

[8]王柯. 数字图像修复方法研究进展 [J]. 现代信息科技，2022，6(4)：38-40.

Wang K. Research progress of digital image inpainting methods [J]. Modern Information Technology, 2022, 6(4):38-40.

[9]Bertalmio M, Sapiro G, Caselles V,et a1.Image inpainting [A]. Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques [C]. Los Angeles:2000.

[10]Ballester C, Bertalmio M, Caselles V, et al. Filling-in by joint interpolation of vector fields and gray levels [A]. IEEE Transactions on Image Processing [C]. New York:2001.

[11]Rudin L I, Osher S, Fatemi E. Nonlinear total variation based noise removal algorithms [J]. Physica D Nonlinear Phenomena, 1992, 60(1-4):259-268.

[12]王璇. 基于GAN的唐卡图像修复算法研究与系统实现 [D]．银川：宁夏大学，2021.

Wang X. Research and System Implementation of Thangka Image Inpainting Algorithm Based on Deep GAN [D]. Yinchuan: Ningxia University, 2021.

[13]Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets [A]. Advances in Neural Information Processing Systems [C]. Montreal: 2014.

[14]梁加易. 基于深度学习的图像修复技术研究 [D]. 北京: 北京邮电大学, 2021.

Liang J Y. Research on Image Restoration Technology Based on Deep Learning [D]. Beijing: Beijing University of Post and Telecommunications, 2021.

[15]王奕超. 局部信息缺失的人脸图像修复与识别的研究与实现 [D]. 成都: 电子科技大学, 2020.

Wang Y C. Research and Implementation of Inpainting and Recognition of Partial Information Missed Face Images [D]. Chengdu: University of Electronic Science and Technology of China, 2020.

[16]Gao R, Grauman K. On-demand learning for deep image restoration [A]. IEEE Conference on Computer Vision and Pattern Recognition [C]. New York: 2016.

[17]Perez P, Gangnet M, Blake A. Poisson image editing [A]. ACM SIGGRAPH 2003 Papers [C]. New York ：2003.

[18]Lin Z, Xiong W, Barnes C, et al. Foreground-aware image inpainting[A]. IEEE Conference on Computer Vision and Pattern Recognition [C]. New York: 2019.

[19]Sahoo S K, Lu W. Image denoising using sparse approximation with adaptive window selection [A]. International conference on Information Communications Signal Processing[C]. New York: 2011.

[20]周林勇, 谢晓尧, 刘志杰, 等. 基于ACGAN的图像识别算法 [J]. 计算机工程, 2019, 45(10): 246-252,259.

Zhou L Y, Xie X Y, Liu Z J, et al. Image identification algorithm based on ACGAN [J]. Computer Engineering, 2019, 45(10): 246-252,259.

[21]Pathak D, Krahenbuhl P, Donahue J, et al. Context encoders: Feature learning by inpainting [A]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) [C]. IEEE,2016.

[22]Iizuka S, Simo-Serra E, Ishikawa H. Globally and locally consistent image completion [J]. ACM Transactions on Graphics (TOG), 2017,36(4):1-14.

[23]Yu J H, Lin Z, Yang J M, et al. Generative image inpainting with contextual attention [A]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition [C]. New York: 2018.

[24]Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Structure guided image inpainting using edge prediction [A]. Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops [C]. New York: 2019.

[25]Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning [A]. Computer Vision and Pattern Recognition [C]. Los Angeles：2019.

[26]郝跃军, 马泽, 安瑞中, 等. 基于改进的纹理性质的图像修复技术研究 [J]. 计算机与数字工程, 2023, 51(8):1844-1847.

Hao Y J, Ma Z, An R Z, et al. Research on image restoration technology based on improved texture properties [J]. Computer & Digital Engineering, 2023, 51(8):1844-1847.

[27]胥加洁.基于生成对抗网络的图像修复技术研究 [D].扬州: 扬州大学, 2020.

Xu J J. Research on Image Inpainting Technology Based on Generative Adversarial Networks [D]. Yangzhou: Yangzhou University, 2020.

[28]姜艺, 胥加洁, 柳絮, 等. 边缘指导图像修复算法研究 [J]. 计算机科学与探索, 2022, 16(3):669-682.

Jiang Y, Xu J J, Liu X, et al. Research on edge-guided image repair algorithm [J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3):669-682.

[29]朱德泉. 基于生成对抗网络的人脸图像修复的研究 [D]. 成都: 电子科技大学, 2020.

Zhu D Q. Research on Face Image Inpainting Based on Generative Adversarial Network [D]. Chengdu: University of Electronic Science and Technology of China, 2020.

[30]李悦城. 基于生成对抗网络的卫星图像修复方法研究 [D]. 哈尔滨: 哈尔滨工业大学,2021.

Li Y C. Research on Satellite Image Inpainting Based on Generative Adversarial Network [D]. Shenzhen: Harbin Institute of Technology, 2021.

[31]魏域林. 层间特征融合与多注意力的图像修复算法研究 [D]. 兰州: 兰州理工大学,2020.

Wei Y L. Image Inpainting with Interlayer Feature Fusion and Multi-attention [D]. Lanzhou: Lanzhou University of Technology, 2020.

[32]胡凯, 赵健, 刘昱, 等. 结构引导的图像修复[J]. 北京航空航天大学学报, 2022, 48(7):1269-1277.

Hu K, Zhao J, Liu Y, et al. Images inpainting via structure guidance [J]. Journal of Beijing University of Aeronautics and Astronautics, 2022, 48(7): 1269-1277.

[33]冯建周, 马祥聪, 刘亚坤, 等. 关于命名实体识别的生成式对抗网络的研究 [J]. 小型微型计算机系统, 2019, 40(6):1191-1196.

Feng J Z, Ma X C, Liu Y K, et al. Research on generative adversarial networks of named entity recognition [J]. Journal of Chinese Computer System, 2019, 40(6):1191-1196.

[34]杨忠鹏, 李启南. 改进SteGAN的嵌入式图像隐写方案 [J]. 兰州交通大学学报, 2022, 41(4):48-57.

Yang Z P, Li Q N. Improved SteGAN embedded image steganography scheme [J]. Journal of Lanzhou Jiaotong University, 2022, 41(4):48-57.

[35]余艳杰, 孙嘉琪, 葛思擘, 等. CycleGAN-SN:结合谱归一化和CycleGAN的图像风格化算法 [J]. 西安交通大学学报, 2020, 54(5):133-141.

Yu Y J, Sun J Q, Ge S B, et al. CycleGAN-SN：Image stylization algorithm combining spectral normalization and CycleGAN[J]. Journal of Xi′an Jiaotong University, 2020, 54(5):133-141.

[36]Redmon J, Divvala S, Girshick R, et al. You only look once: Unified, real-time object detection [A]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition [C]. New York: 2016.

[37]李亚男,万敏,吴向东. 基于应变非接触测量的裂纹拼接 [J]. 锻压技术,2013,38(6):111-115.

Li Y N, Wan M, Wu X D. Crack spliced based on non-contact strain measurement [J]. Forging & Stamping Technology, 2013, 38(6):111-115.

Service：

【This site has not yet opened Download Service】【Add Favorite】