Small targets detection in low-resolution remote sensing images based on super-resolution joint optimization

2.1.1. Design of generator

The GAN used in this study is derived from enhanced super-resolution generative adversarial network (ESRGAN) [18] and some processing details have been modified. The generator structure is shown in Fig. 2. As illustrated in Fig. 2, the generator network employs residual-in-residual dense blocks (RRDBs) as its fundamental operational units. Each RRDB module integrates three hierarchically interconnected dense blocks through cascaded residual pathways, and the batch normalization layer is removed from the dense block to reduce computational complexity. To mitigate training instability arising from network depth escalation, this study also scales down the residuals by multiplying a scaling factor (empirical constant of 0.2) before adding them to the main path. Each dense block strategically leverages densely connected convolutional layers to extract rich local feature representations, as illustrated in Fig. 2. This architectural optimization facilitates systematic integration of hierarchical features across network layers, thereby enhancing model capacity through multi-scale feature utilization while simultaneously elevating image reconstruction fidelity through improved feature preservation and propagation mechanisms.

Fig. 2Structure of generator network

2.1.2. Design of discriminator

As shown in Fig. 3, the discriminator employs VGG-19 as backbone network, augmented with relativistic adversarial learning through a relativistic average discriminator (RaD) [19]. In RaD, the task of discriminator is perform relative true and fake discrimination between two images. Here, $I_{LR}\in {\mathbb{R}}^{H\times W\times C}$ is considered the LR input of generator, $I_{HR}\in {\mathbb{R}}^{sH\times sW\times C}$ and $I_{Gen}\in {\mathbb{R}}^{sH\times sW\times C}$ are respectively considered the HR input and generated intermediate input of discriminator, where: $H$ and $W$ denote the height and width of the input LR image, respectively; $s$ denotes the upscaling factor; and $C$ denotes the number of channels. The mathematical relationship between $I_{LR}$ and $I_{Gen}$ is as follows:

where, $G_{{\theta }_G}$ denotes the network parameters of the generator. The mathematical expression of the RaD is expressed as follows:

where, $\mathrm{\sigma }(·)$ and $C(·)$ denote the Sigmoid function and the non-transformed discriminator output, respectively; $\mathbb{E}[·]$ denotes the expectation operation over the mini-batch of data, e.g., ${\mathbb{E}}_{I_{Gen}}[·]$ and ${\mathbb{E}}_{I_{HR}}[·]$ denote the operation of calculating mean for all generated intermediate images and original HR images in a mini-batch, respectively; $D_{Ra}$ denotes a relative average discriminant function. When the real image exhibits superior photorealism and perceptual fidelity compared to the generated image, the result of $D_{Ra}(I_{HR},I_{Gen})$ tends to be 1 (Eq. (2)); If the generated image demonstrates degraded quality relative to the real image, the result of $D_{Ra}(I_{Gen},I_{HR})$ tends to be 1 (Eq. (3)).

Building upon the theoretical framework of RaD, the adversarial loss for discriminator can be defined as follows:

Then, the adversarial loss for generator is expressed as follows:

Therefore, the generator in RaD benefits from the gradients from both generated data and real data in adversarial training, thus generating rich edge textures.

Fig. 3The structure of discriminator network

2.1.3. Design of perceptual loss

Grounding in human visual perception characteristics that emphasize hierarchical feature processing [20], the seminal work [17] established perceptual loss through feature representation from deep layers of pre-trained networks. While SRGAN extended this paradigm by computing loss on post-activation features, subsequently analyses reveal critical limitations: 1) Activation-induced feature sparsity in deep networks diminishes gradient signal integrity for supervision; 2) Nonlinear activation distortions introduce luminance inconsistency between reconstructed and HR references, destabilizing loss landscape optimization.

To overcome these constraints, this study uses the features before the activation layers. Specifically, given the strong generalization capability of the pre-trained VGG-19 features, this study compute the perceptual loss using the features extracted from VGG-19 network before the activation function layers to leverage pre-activation convolutional responses. This approach preserves richer gradient information and structural primitives-particularly enhancing edge acuity and texture fidelity through undistorted low-level feature preservation. The perceptual loss function is defined as follows:

where, $\phi (·)$ denotes the calculation process before the activation layers of a fine-tuned VGG-19 network, $\Vert ·{\Vert }_1$ denotes the L1 norm, and ${\mathbb{E}}_{I_{LR}}[·]$ denote the operation of calculating mean for all input LR images in a mini-batch.

In addition to the perceptual loss, this study introduces the content loss for measuring the 1-paradigm distance between $I_{Gen}$ and $I_{HR}$. The content loss function is defined as follows:

The total loss function for the generator is:

where, $\lambda$ and $\eta$ are denote the weight parameters, which are taken as 0.01 and 0.001, respectively.

2.2.1. Design of EEN structure

For the images reconstructed by the generator, the Laplacian operator [21] is initially utilized for detecting and extracting edges. In particular, this study employs the discrete convolution mask of the Laplacian defined as ([−1, −1, −1], [−1, 8, −1], [−1, −1, −1]). Using the convolution kernel, Laplacian procedure is outlined as follows:

where, $E_{Laplacian}(·)$ and $I_{Laplacian}$ denote the Laplacian edge detection operation and the extracted edge maps, respectively. After the edge information is extracted, this study firstly utilizes the strided convolution to extract edge maps while simultaneously transforming them into LR space. The downsampling operation consists of three strided convolutional layers (followed by a LeakyReLU activation function), each with a kernel size of 3×3. Specifically, the number of feature maps for each convolutional layer, in order, is 128, 256, and 64, with stride sizes of 2, 2, and 1, respectively. Compared to operation in HR space, the aforementioned strategy reduces computational complexity. Then, this study employs a DFESN containing multiple RRDB blocks to estimate and enhance edge information in EEN. The structural diagram of the DFESN is shown in Fig. 4.

To address the inherent edge noise in $I_{Laplacian}$, a dedicated denoising branch is constructed to dynamically learn the image mask for detecting and eliminating isolated noise artifacts introduced during the Laplacian edge extraction process. As shown in Fig. 4, this branch employs a hierarchical architecture comprising three stacked convolutional layers with a kernel size of 3×3 followed by a Sigmoid activation function to learn a spatially variant weight matrix. This operation produces a probabilistic edge map with normalized intensity values in the range (0, 1). This biologically-inspired mechanism mimics the selective attention mechanism in human visual perception, prioritizing salient edge features while suppressing spurious noise and erroneous edge responses. The refined edge maps are subsequently upsampled to HR space via a learnable upsampling module. The process of converting $I_{Laplacian}$ to enhanced edge maps $I_{Edge}$ in EEN can be formulated as follows:

where, $C_{DFESN}(·)$ denotes the operation of the DFESN to achieve hierarchical feature extraction and cross-layer feature fusion, $C_M(·)$ denotes the mask branch to learn the image mask for suppressing the noises and the false edges, $DS(·)$ denotes the downsampling operation with strided convolution to project Laplacian edge maps into LR spaces, and $US(·)$ denotes the upsampling operation with subpixel convolution to project refined edge maps into HR spaces.

In the terminal phase of EEN, a residual learning mechanism is employed to amplify discriminative feature variations between $I_{Edge}$ and $I_{Laplacian}$. Specifically, the pixel-wise discrepancy between $I_{Edge}$ and $I_{Laplacian}$ is computed, generating a residual map that encapsulates high-frequency edge details. This residual signal is then fused with $I_{Gen}$ through channel-wise summation, yielding the final SR reconstruction results $I_{SR}$:

2.2.2. Design of loss functions in EEN

To optimize the perceptual and structural fidelity of reconstructed imagery while mitigating artifacts, this study introduces a pixel-based Charbonnier penalty loss [22] to enhance the consistency of image contents between $I_{SR}$ and $I_{HR}$. The consistency loss function is formulated as follows:

where, $\rho (·)$ and ${\mathbb{E}}_{I_{SR}}[·]$ denote Charbonnier penalty function and the operation of calculating mean for all final SR images in a mini-batch, respectively. Considering the challenges of geometric distortion and artifactual noise propagation in edge representations, this study adopts a consistency loss calculation strategy that explicitly enforces geometric and photometric consistency between reconstructed and GT edge maps. The consistency loss function for the image edges is formulated as follows:

where, $I_{Edge\_HR}=E_{Laplacian}\left(I_{HR}\right)$, denotes the extracted edge maps of the original HR images; and ${\mathbb{E}}_{I_{Edge}}[·]$ denotes the operation of calculating mean for all Laplacian edge maps in a mini-batch. Finally, the supervised learning of EEN is performed with the total loss $L_{EEN}$ that aggregates task-specific objectives for both holistic image reconstruction and edge structure preservation:

3.4.1. Ablation study

To ensure comparative fairness, the experimental protocol employed original HR images and SR outputs from isolated SR models (SR1: adversarial framework [generator + discriminator]; SR2: SR1 + perceptual loss; SR3: SR2 + EEN) to train dedicated detectors, subsequently evaluating them across SR-restored test sets. As shown in Table 1, quantitative analysis reveals fundamental resolution-detection correlations: HR→LR detection achieved 40.50 % AP versus HR→HR upper bound of 78.10 % AP (+37.60 %), indicating the large impact of the resolution to the object detection quality and establishing theoretical performance limits for SR-based approaches. The adversarial framework was employed to preprocess LR images, with the detector subsequently trained on pristine HR data. This configuration yielded an 11.58 % AP improvement over direct LR detection, confirming the GAN’s capacity to recover spatially discriminative features for small vehicle recognition. Progressive architectural enhancements revealed systematic gains: introducing perceptual loss supervision elevated both reconstruction fidelity (+0.0672 dB PSNR) and detection precision (+1.24 % AP), empirically validating its role in aligning high-level semantics between SR and HR domains. Further integration of an EEN amplified these benefits (+0.0672 dB PSNR, +2.58 % AP), attributable to its edge-texture refinement critical for detection-oriented SR. Additionally, as evident from Table 1, detectors trained on outputs from any SR variant generalized robustly to the SR testing set, still achieving a better recognition result.

The SR networks were also jointly optimized with diverse architectural configurations and HR-MSRN in an end-to-end training paradigm to evaluate the efficacy of the proposed methodology. The detector’s loss gradient propagated backward through the SR network, thereby enabling synergistic optimization that enhanced the quality of LR imagery. The SR module was trained using LR-HR image pairs during the training phase. The generated SR outputs were subsequently fed into the detection network. During the testing phase, only LR inputs were processed through the integrated network architecture, which sequentially performed SR reconstruction followed by object detection. As quantified in Table 2, the end-to-end framework demonstrated statistically significant improvements across both SR reconstruction and detection tasks. Compared to separate training paradigm utilizing different SR networks, the progressive incorporation of adversarial learning (“generator + discriminator”), perceptual loss (“generator + discriminator + perceptual loss”), and EEN (“generator + discriminator + perceptual loss + EEN”) yields incremental quantitative enhancements, achieving PSNR improvements of 0.0679 dB, 0.1105 dB, and 0.1294 dB alongside corresponding AP metric gains of 1.25 %, 1.89 %, and 2.43 %, respectively. Fig. 6 visually demonstrates the enhancement of AP values achieved through end-to-end joint training of super-resolution and object detection networks across diverse architectural configurations. Furthermore, the complete JSRDN architecture (integrating generator, discriminator, perceptual loss, and EEN) demonstrates the optimal performance in both SR reconstruction fidelity and object detection accuracy for remote sensing imagery, confirming the synergistic benefits of joint optimization strategies.

Fig. 6Comparison of the effectiveness of end-to-end training; The left and right charts represent separate training and end-to-end training, respectively

Moreover, a visual analysis is performed to evaluate the effectiveness of EEN. As illustrated in Fig. 7, qualitative visual representations demonstrate that the EEN can effectively perceives high-frequency detail information within the input LR images. Subsequent super-resolved images display enhanced edge and textual granularity in subjective visual assessment, achieving sufficient discriminative clarity for vehicular object identification.

Fig. 7Visual effects of EEN

a) Input LR images

b) Enhanced edges

c) SR results

d) Detection results

3.4.2. Comparison with other SR approaches

Furthermore, this study conducted a comprehensive comparative analysis encompassing seven typical SR approaches, including Bicubic, super-resolution convolutional neural network (SRCNN) [26], very deep convolutional networks SR (VDSR) [27], local-global combined network (LGCNet) [28], SRGAN, enhanced deep super-resolution network (EDSR) [29], and deep residual dual-attention network (DRDAN) [30], to perform ×4 upscaling operations on LR input data. The SR outputs were evaluated using the detector based on the HR-MSRN. To ensure a fair comparison, the object detector was trained exclusively on original HR images, while the testing phase utilized SR images generated by each reconstruction model. The experimental outcomes are mainly compared from two aspects, one is the comparative analysis of the objective quantitative indicators of image reconstruction and object detection, the other is the subjective assessment of visual performance focusing on detection localization precision and structural preservation in reconstructed outputs.

Table 3 quantifies the PSNR and AP metrics for SR images generated by different algorithms on the COWC dataset. The data reveals a direct positive correlation between SR algorithm performance and vehicle recognition accuracy, indicating that enhanced spatial resolution from advanced SR methodologies directly improves object detection efficacy. Among CNN-based approaches (SRCNN, VDSR, LGCNet, SRGAN, EDSR, and DRDAN), all exhibit substantial AP improvements over the Bicubic algorithm, with gains of 2.66 %, 3.34 %, 3.06 %, 5.45 %, 5.59 %, and 6.74 % respectively. Notably, the proposed JSRDN framework achieves superior performance, surpassing DRDAN by 0.1819 dB in PSNR and 7.18 % in AP. The quantitative comparisons presented above demonstrate the effectiveness and superiority of the proposed method.

Fig. 8 visualizes the detection performance of SR images using different approaches. Other approaches struggle with leakage or misdetection due to challenges like blurring, small object sizes, complex lighting/shadow conditions, and inter-object interference. In contrast, by integrating GAN and EEN structures, JSRDN effectively preserves high-frequency details in remote sensing imagery. Furthermore, the joint optimization of a unified SR and object detection architecture substantially enhances detection robustness, particularly improving recognition reliability for small targets in complex environments. Significantly, the proposed JSRDN method generates detection results that exhibit enhanced spatial alignment with GT annotations, demonstrating superior consistency in localization precision.

Fig. 9 provides a magnified visual comparison of vehicle targets generated by various SR models. In comparison to other approaches, the proposed JSRDN excels in restoring the color, texture, and edge information of the vehicle targets, while achieving enhanced structural congruence with GT annotations. This advancement contributes to measurable improvements in detection efficacy of small objects with cluttered environments, particularly through optimized feature representation in complex scenarios.

Fig. 8Visual comparisons of the detection effects of different SR models

a) “Selwyn_BX22_Tile_RIGHT_15cm_0003.1.71” scene

b) “Selwyn_BX22_Tile_RIGHT_15cm_0003.2.72” scene

c) “Selwyn_BX22_Tile_RIGHT_15cm_0003.8.53” scene

Fig. 9Visual comparisons of reconstruction effects of different SR models

In summary, the proposed JSRDN achieves advanced performance through the synergistic integration of images SR reconstruction and small object detection tasks within a unified learning paradigm. The methodology demonstrate dual-domain superiority, excelling not only in reconstructing high-fidelity remote sensing imagery with preserved structural integrity but also in addressing the critical challenge of small object detection in LR scenarios under complex environmental conditions. The co-optimized architecture establishes JSRDN as a holistic framework capable of providing a comprehensive solution for challenging remote sensing image analysis tasks.