Self-supervised CNN for user behavior analysis on smart meter data

3.2.1. FFT

Applying the FFT to time series data allows us to obtain its frequency spectrum, which in turn enables the identification of dominant periods through frequency domain analysis. This approach is both common and critical in time series analysis, as it helps uncover periodic or recurrent patterns within the data. In the resulting frequency spectrum, peaks with higher amplitudes indicate that those frequencies play a more significant role in the original time series. Specifically, Considering a univariate smart meter time series $X_{in}\in {\mathbb{R}}^{1\times T}$, where $T$ represents the length, FFT is employed to analyze the series in the frequency domain. The process can be outlined as follows:

where, $FFT\left(·\right)$ and $Amp\left(·\right)$ denote the FFT and the calculation of amplitude values. $\mathbf{A}\in {\mathbb{R}}^T$ represents the calculated amplitude of each frequency. Considering the sparsity in the frequency domain and to avoid noise from irrelevant high-frequency components , only the top $D$ amplitudes are selected, enabling identification of the most significant frequencies $\left\{f_1,f_2,f_3...f_D\right\}$, where $D$ is equal to the dimensionality of the embedding vector obtained after the time series passes through the Patching and Embedding layer in the lower branch. The corresponding period lengths for these frequencies are $\left\{p_1,p_2,p_3...p_D\right\}$.

3.2.2. Patch embedding

For the input smart meter time series $X_{in}$, padding is first applied, followed by segmentation into $N$ overlapping patches, each of size $P$. The stride of the patching process is set to $S$. These patches are then embedded into $D$-dimensional embedding vectors:

where PE refers to the Patching and Embedding operation, $X_{emb}\in {\mathbb{R}}^{1\times D\times N}$ is the input embedding.

3.2.3. DWConv

As is shown in Fig. 2, DWConv is a combination of depthwise convolution and pointwise convolution that significantly reduces computational complexity and the number of parameters. Specifically, after the initial Patching and Embedding operations, the time series data is transformed into $X_{emb}$. In DWConv, the feature and variable channels are independent, with each channel learning the temporal dependencies separately. The number of convolution channels is set to 1×$D$ to extract features along the temporal dimension. Inspired by the approach in modernTCN [13], large kernels are used to expand the receptive field and enhance the model’s ability to capture temporal dependencies. To better leverage the periodic nature of time series data, the $D$ period lengths derived from FFT are proposed as kernel sizes for the $D$ convolutional filters in DWConv. This integration of periodicity into the convolution process is expected to improve model performance.

Fig. 2The structure of DW

3.2.4. ConvFFN

The ConvFFN comprises two modules, ConvFFN1 and ConvFFN2. As depicted in Fig. 3, each module consists of two pointwise convolution (PWConv) layers and adopts an inverted bottleneck structure, where the hidden channels in the ConvFFN block are expanded to be $r$ times wider than the input channels. Each ConvFFN module is designed to mix information from either the variable or feature dimensions exclusively, a decoupling design that reduces computational complexity and enhances model performance. Both ConvFFN modules utilize group convolution, with the number of groups set to the number of channels, meaning each group has only one channel. Specifically, in ConvFFN1, the number of variables is treated as the number of groups, allowing for the extraction of feature dimension information. Similarly, in ConvFFN2, the output from ConvFFN1 is transposed, treating the number of features as the number of groups to extract variable dimension information. To facilitate the training of deeper neural networks and improve the model’s generalization ability, residual connections are employed, linking the input embedding $X_{emb}$ with the output of ConvFFN2.

Fig. 3Structure of ConvFFN

3.3.1. Data augmentation

Data augmentation plays a critical role in contrastive learning [29, 44]. By generating different views of the same data sample, data augmentation enables the model to learn more robust feature representations. During contrastive learning, the model seeks to maximize the similarity between different views of the same sample while minimizing its similarity with other samples. This approach helps the model better capture the intrinsic characteristics of the data, thereby enhancing its generalization ability and robustness. Thus, designing effective data augmentations is essential. Specifically, for a given smart meter time series sample $X$, a strong augmentation module and a weak augmentation module generate a strongly augmented view $X_a$ and a weakly augmented view $X_b$, respectively. The strong augmentation operations include adding random noise and permutation. The weak augmentation strategy applies random mask and jitter. A comparison of the time series after both augmentations with the original sequence is presented in Fig. 5. These augmented views $X_a$ and $X_b$ are then fed into the Encoder module (which represents the proposed model), producing high-dimensional latent representations $z^a$ and $z^b$. These high-dimensional representations are subsequently input into the temporal contrastive module.

Fig. 4Overall architecture of Self-supervised learning. The self-supervised learning framework primarily consists of two components. In the temporal contrastive module, a rigorous cross-view prediction task is designed to learn robust representations from the time series data. Building on these robust representations, the contextual contrastive module further enables the model to learn discriminative features

Fig. 5The original time series and the four argument time series

3.3.2. Temporal contrasting

The input to the temporal contrast module consists of the high-dimensional latent representations $z^a$ and $z^b$ generated by the encoder, with the output being the corresponding context vectors $c_t^a$ and $c_t^b$. In this module, a cross-view prediction task is designed: the context vector $c_t^a$ is used to predict the future $k$ time steps ($1<k<K$) of $z^b$, and conversely, the context vector $c_t^b$ is used to predict the future $k$ time steps of $z^a$. To accomplish this prediction task, a log-bilinear model ${\mathcal{W}}_k$ is employed to map $c_t$ back to the same dimensionality as $z$, thereby preserving the mutual information between $c_t$ and $z$. Next, a contrastive loss is applied to maximize the dot product between the predicted representation and the true representation of the same sample, while minimizing the dot product with representations of other samples within the mini-batch ${\mathcal{N}}_{t,k}$. The temporal contrastive loss proposed in [19] is adopted, which includes the following two types of contrastive losses:

3.3.3. Contextual contrasting

The contextual contrastive module maps contexts into a space where contextual contrastive learning is applied, with the goal of encouraging the model to learn more discriminative representations. Specifically, given a batch of $N$ input samples, augmentation produces 2$N$ augmented views, resulting in 2$N$ contexts. For a context $c_t^i$, $c_t^{i^{+}}$ is denoted as the positive sample of $c_t^i$ , where the positive sample comes from the other augmented view of the same input. Hence, $\left(c_t^i,c_t^{i^{+}}\right)$ is considered a positive pair. Negative samples are drawn from the remaining 2$N$-2 contexts within the same batch from other inputs, meaning that $c_t^i$ can form 2$N$-2 negative pairs with its negative samples. The contextual contrastive loss is then designed to maximize the similarity between the sample and its positive pair while minimizing its similarity with the negative samples within the mini-batch. In this work, the contextual contrastive loss proposed in is adopted. The loss function is formulated as follows:

where $sim\left(\mathbf{u},\mathbf{v}\right)={\mathbf{u}}^T\mathbf{v}/‖\mathbf{u}‖‖\mathbf{v}‖$ denotes the dot product between $l_2$ normalized $\mathbf{u}$ and $\mathbf{v}$, ${\mathbb{l}}_{\left[m\ne i\right]}\in \left\{0,1\right\}$, $\tau$ represents the temperature coefficient.

The overall self-supervised loss is a weighted sum of two temporal contrastive losses and the contextual contrastive loss, as shown below:

where ${\lambda }_1$ and ${\lambda }_2$ are fixed scalar hyperparameters denoting the relative weight of each loss.

4.2.1. Self-supervised model

SimCLR: An unsupervised contrastive learning method based on data augmentation, originally developed in the field of computer vision.

CPC: a contrastive self-supervised learning method that focuses on predicting future time steps.

CLSTRAN: a multi-input unsupervised contrastive learning method.

TimeMAE: a representation learning method based on feature masking and reconstruction.

4.2.2. Universal time series representation learning model

ResNet1D-Wang: a classical one-dimensional fully convolutional residual network architecture.

Informer: a parallel multi-kernel feature extractor inspired by the Inception architecture, modified to process one-dimensional data effectively.

Autoformer: a long-term time series representation learning model based on a deep decomposition framework and auto-correlation mechanisms.

TimesNet: a model that adaptively transforms time series data into two-dimensional representations for comprehensive feature extraction.

ModernTCN: a pure convolutional architecture that employs extremely large convolutional kernels for enhanced temporal feature learning.

4.5.1. Compared with previous models

Comprehensive comparisons between the proposed model and previous approaches were conducted, with detailed quantitative results systematically presented in Table 2. The experimental evaluation demonstrates that the proposed model consistently maintains superior performance across all evaluation tasks. Specifically, compared to general time-series representation learning models, our proposed model achieves significant improvements over the suboptimal model, with accuracy increasing by 4.11 %, 3.25 %, and 2.96 % on Task300, Task310, and Task410 respectively; recall improving by 4.55 %, 3.6 %, and 3.01 %; and F1-score enhancing by 4.58 %, 3.59 %, and 3.05 %. These substantial improvements are primarily attributed to the proposed model’s enhanced representation learning capability for smart meter signals, achieved through an innovative pre-training strategy that effectively captures the unique characteristics of energy consumption patterns.

When compared with previously proposed self-supervised models, our architecture demonstrates distinct structural advantages in feature extraction. The key innovation lies in our adaptive convolutional kernel design, which dynamically adjusts its receptive field to better capture both short-term fluctuations and long-term periodic dependencies in smart meter data. This architectural superiority translates into consistent performance gains over the suboptimal self-supervised model TimeMAE, with accuracy improvements of 1.65 %, 1.05 %, and 0.32 %; recall enhancements of 1.66 %, 1.29 %, and 0.11 %; and F1-score increases of 1.74 %, 1.31 %, and 0.18 % across the three evaluation tasks respectively. These results not only validate the effectiveness of our model design but also highlight the critical importance of incorporating pre-training strategies when addressing the challenging task of demographic information identification from smart meter data, where signal patterns are typically subtle and highly variable.

Additionally, inference speed tests were conducted for various models, with the time values presented in the table corresponding to the inference duration per 100 samples. As can be observed, our proposed model demonstrates superior inference efficiency. Compared to Transformer-based architectures, the CNN-based structure exhibits fewer parameters and benefits from more optimized low-level implementations.

4.5.2. Ablation studies

In order to demonstrate the impact of adaptive convolution kernels on model performance and the performance improvement brought by super large convolution kernels at different locations, experiments were conducted on Task300, Task310, and Task410. The results are shown in Table 3.

Experimental results indicate that using adaptive-sized convolutional kernels in the first block significantly improves accuracy across all tasks. For example, on Task300, the model with adaptive kernels in the first block achieves an accuracy of 75.77 %, compared to 71.51 % for the baseline model (ModernTCN). Similar improvements are observed on Task310 (78.06 % vs. 74.92 %) and Task410 (65.80 % vs. 62.84 %). This suggests that the early layers of the model play a crucial role in capturing periodic features, which are essential for accurate predictions.

However, as the model depth increases, the effectiveness of larger convolutional kernels diminishes. For instance, applying adaptive kernels in the third block results in lower accuracy compared to the first block (72.01 % vs. 75.77 % on Task300). This is likely because features in deeper layers become more abstract, and their inherent periodic characteristics are less pronounced. Consequently, the use of extremely large convolutional kernels in later blocks becomes less meaningful.

Furthermore, due to the constraints of input sequence length, adaptive-sized convolutional kernels can only be applied simultaneously in the first and second blocks at most. While this configuration improves accuracy compared to the baseline (72.72 % vs. 71.51 % on Task300), it underperforms compared to using adaptive kernels solely in the first block (72.72 % vs. 75.77 %). This further underscores the importance of the first block in capturing critical periodic features.

To demonstrate the impact of different time series periodicity extraction methods on the guidance of subsequent representation learning, several non-learning techniques were employed and tested across three tasks. Specifically, the Autocorrelation Function (ACF), Hilbert-Huang Transform (HHT), Peak Detection (PD), Wavelet Transform (WT), and Fast Fourier Transform (FFT) were utilized. The results are presented in the accompanying Fig. 6 and Tabel 4.

Fig. 6The impact of different extraction methods on the final classification results.

FFT achieved the highest accuracy across all three tasks, followed by WT, while ACE and PD exhibited poor performance. By comparing the sizes of the convolutional kernels introduced by these methods, it appears that using larger kernels may be a crucial strategy for enhancing accuracy.

Contrastive self-supervised learning is a key step in improving the proposed model’s performance. To explore the impact of different data augmentation combinations on self-supervised learning performance in smart meter data, pairwise combinations of the four methods shown in Fig. 5 were evaluated across three datasets. The experimental results are shown in Fig. 7. It can be observed that “random noise + random mask” and “Permutation + Jitter” achieve the highest performance in Task300, with "Permutation + Jitter" yielding the best performance in Task310, and “random noise + Jitter” achieving the best performance in Task400. Analysis reveals that applying strong and weak augmentation on the two branches of contrastive learning is the best strategy. This is because different intensities of feature augmentation contain latent information at varying granularities, which requires the model to have stronger feature extraction capabilities to achieve the contrastive learning objective.

Fig. 7The impact of different data augmentation combinations on the final classification results.

a) Task300

b) Task310

c) Task410