Weather Recognition plays an important role in our daily lives and many computer vision applications. However, recognizing the weather conditions from a single image remains challenging and has not been studied thoroughly. Generally, most previous works treat weather recognition as a single-label classifica- tion task, namely, determining whether an image belongs to a specific weather class or not. This treat- ment is not always appropriate, since more than one weather conditions may appear simultaneously in a single image. To address this problem, we make the first attempt to view weather recognition as a multi- label classification task, i.e., assigning an image more than one labels according to the displayed weather conditions. Specifically, a CNN–RNN based multi-label classification approach is proposed in this paper. The convolutional neural network (CNN) is extended with a channel-wise attention model to extract the most correlated visual features. The Recurrent Neural Network (RNN) further processes the features and excavates the dependencies among weather classes. Finally, the weather labels are predicted step by step. Besides, we construct two datasets for the weather recognition task and explore the relationships among different weather conditions. Experimental results demonstrate the superiority and effectiveness of the proposed approach. The new constructed datasets will be available at

1. Introduction

 

The weather conditions influence our daily lives and production in many ways [1], such as wearing, traveling, solar technologies and so on. Therefore, acquiring weather conditions automatically   is important to a variety of human activities. A possible solution to weather recognition is utilizing various of hardwares. While these hardware equipments are usually expensive and need professionals to maintain. An alternative scheme is to recognize weather con- ditions from color images using computer vision techniques [2,3]. Nowadays, surveillance cameras are ubiquitous, which makes the computer vision solution feasible. Apart from the guiding signif- icance to our daily lives, weather recognition is also an impor-    tant function to many other computer vision applications [4–7], such as image retrieval [8], image restoration [9], and the relia- bility improvement of outdoor surveillance systems [3]. Robotic vi- sion [10,11] and vehicle assistant driving systems [12,13] can also benefit from the results of weather recognition. Thus, we can   draw

a simple conclusion that weather recognition from outdoor images has great research significance.

1.1. Motivation and overview

 

Although weather recognition is of remarkable value, only a few researches have been published to tackle this problem. Several pre- vious works [12,14–16] concentrated on recognizing weather con- ditions from images captured by in-vehicle cameras. Several other papers [1,17,18] exploited weather recognition from single outdoor images. All of these works referred to weather recognition as a single-label classification task (the weather label means weather category in this paper), namely, determining whether an image be- longs to a specific weather category or   not.

However, it is not always appropriate to view weather recogni- tion as a single-label classification problem. There are mainly two reasons to explain this inappropriateness. The first reason can be summarized as uncertainty, i.e., the class boundaries among some weather categories are ambiguous essentially. As can be seen from Fig. 1, the  changes from  Fig. 1(a)–(f) demonstrate that there are    a series of states between a pure sunny weather (like Fig. 1(a))   and an obvious cloudy weather (as illustrated in Fig. 1(f)). It  is hard to determine whether the category is sunny or cloudy whe

referring to an intermediate weather state like Fig. 1(c), (d) and (e) [2]. Thus, the uncertainty of such boundary samples causes the dif- ficulty to determine ground-truth labels even from the perspective of human beings, and few previous works present solutions to this problem. The second drawback of treating weather recognition as    a single-label classification task can be summarized as incomplete- ness, namely, a single weather label may not describe the weather conditions comprehensively for a given image. For example, the vi- sual effect of haze is obvious in Fig. 1(g), (h) and (i). Nevertheless, it can be seen from the comparisons among these three images    that Fig. 1(g) seems more sunny while Fig. 1(h) seems more over- cast, and Fig 1(i) seems snowy. Therefore, only a haze label cannot reveal the differences among these three   images.

Motivated by the aforementioned two reasons, we propose to view weather recognition as a multi-label classification problem, i.e., assigning multi-labels to an image according to the displayed weather conditions. Specifically, it is achieved by a CNN–RNN ar- chitecture. The intuition lies in two aspects. On one hand, most       of the previous works focused on exploiting hand-crafted weather features [1], [20], while these features did not achieve desired re- sults in the weather recognition task. Inspired by the great success of Convolutional Neural Network (CNN) in these years, we utilize CNN as the weather feature extractor. On the other hand, labels exhibit strong co-occurrence dependencies in weather domain. For example, snowy and cloudy usually occur together while rainy and sunny almost never co-occur. Inspired by the success of Recurrent Neural Network (RNN) in dependency modeling [21,22], we pro- pose to use RNN to model the dependencies among labels and pre- dict weather labels step by step. In such a way, when predicting subsequent labels, the network can refer to the previous hidden states that incorporate the historical information   implicitly.

For weather recognition, different image regions have different importances when predicting labels. As shown in Fig. 2, the blue sky is crucial for judging a sunny day, and snow on the ground is significant for estimating the snowy weather. Lu et al. [2] also em- phasized that such weather cues are critical. Therefore, it is nec- essary to make the weather cues discriminative and preserve the spatial information of the image. To achieve this goal, a channel- wise  attention  model  is  designed  to  exploit  more discriminative

features for the weather recognition task. Besides, we use convo- lutional Long Short-Term Memory (LSTM) [23] instead of vanilla RNN in our CNN–RNN architecture to preserve the spatial infor- mation. Convolutional LSTM uses convolution operations in both state-to-state and input-to-state transformations, which captures better spatio-temporal information than fully connected LSTM (FC- LSTM) [23].

In addition, considering that there  are  lacking  of  datasets  in the weather recognition task, two new datasets are constructed in this paper, where the first consists of about 8K images from seven weather categories, it is transformed from an existing transient at- tribute dataset [19].  The second is built from scratch containing 10K images from five weather categories.

1.1. Contributions

 

In summary, there are three main contributions of this work:

(1) We propose to treat weather recognition as a multi-label classification task by analyzing the drawbacks of classifying images with a single weather label and the co-occurrence relationships among different weather  conditions.

(2) We present a CNN–RNN architecture to tackle the multi- label weather classification task. It is composed of a CNN to extract features, a channel-wise attention model to recali- brate feature responses, and a convolutional LSTM to model the relationships among different weather   labels.

(3) We build a new multi-label weather  classification  dataset and transform an existing transient attribute dataset [19] for the weather recognition task. The datasets will be available on the project  website.

1.2. Organization

 

The remainder of this paper is in the following structure: In Section 2, some related works about weather recognition are re- viewed. In Section 3, we describe the proposed approach in detail. In Section 4, we first present the construction of the new multi- label weather image dataset and the modification of the transient attribute dataset [19]. Then, we analyze the experimental results on these two datasets. In Section 5, the conclusion of this paper is drawn.

2. Related work

 

We roughly classify the weather recognition works into two subcategories in this paper. One category focuses on designing hand-crafted weather features, and another  category  attempts  to use CNNs to solve the weather recognition  task.

2.1. Weather recognition with hand-crafted features

 

Many vehicle assistant driving systems use  weather  recogni- tion to improve the road safety. For example, they can set speed limit in extreme weather conditions, automatically open the wiper in a rainy day and so forth. Hand-crafted features are popular in these works. Kurihata et al. [12,24] proposed that rain drops are strong cues for the presence of  rainy  weather  and  developed  a rain feature to detect rain drops on the windshield. Roser et al.    [15] defined several regions of interest (ROI) and developed various types of histogram features for rainy weather recognition. Yan et al. [13] utilized gradient amplitude histogram, HSV color histogram as well as road information for the classification task among sunny, cloudy and rainy categories. Besides, several methods are proposed specially for fog detection, Hautiére et al. [14] used Koschmieder’s Law [25] to detect the presence of  fog and estimate  the visibil-   ity distance. Bronte et al. [26] utilized many techniques, includ-   ing a Sobel based sunny-foggy detector, edge binarization, hough line detection, vanishing point detection and road/sky segmenta- tion. Gallen et al. [27] focused on night fog detection by detecting backscattered veil caused by the vehicle ego lights or halos around the street lights. Pavli et al. [16,28] transformed images into fre- quency domain and detected the presence of fog through training different scaled and oriented Gabor filters in the power spectrum. Although the aforementioned approaches have shown good perfor- mance, they are usually limited to the in-vehicle perspective and cannot be applied to wider range of   applications.

There are also several researches devoted to weather recogni- tion from common outdoor images. Li et al. [29] proposed a pho- tometric stereo-based approach to estimate weather condition  of

a given site. Zhao et al. [9] pointed out that pixel-wise inten- sities of dynamic weather conditions (rainy, snowy, etc.) fluctu- ate over time while static weather conditions (sunny, foggy, etc.) almost stay unchanged. They proposed a two-stage classification scheme which first distinguishes between the two conditions then utilizes several spatio-temporal and chromatic features to further estimate the weather category. In [17], several global features were extracted for weather classification, such as inflection point in- formation, power spectral slope, edge gradient energy, saturation, contrast and image noise. Li et al. [18] also utilized several fea- tures in [17], and constructed a decision tree according to the dis- tance between features. Except for regular global features, [1] pro- posed multiple weather cues including reflection, shadow and sky descriptor for two-class weather recognition. They also exploited a collaborative learning strategy in which voters closer to the test image have more weights. Zhang et al. [20,30] proposed the sunny feature, rainy feature, snowy feature and haze feature individually for each weather class as well as two global features. Furthermore, a multiple kernel learning approach is proposed in [30] to fuse these features. In [31], both spatial appearance and temporal dy- namics were investigated on short video clips which can recognize several weather types.

Although researchers have elaborately designed many features for weather recognition, the features are usually limited to specific perspectives or weather classes, and cannot be applied to wider range of applications.

1.1. Weather recognition with CNNs

 

In recent years, convolutional neural networks have shown overwhelming performance in a variety of computer vision tasks, such as image classification [32], object detection [33], semantic segmentation [34], etc. Several excellent architectures of CNNs are proposed including AlexNet [32], VGGNet [35] and ResNet [36], which outperform the traditional approaches to a large extent. In- spired by the great success of CNNs, a few of works attempt to apply CNNs to weather recognition task. Elhoseiny et al. [3] di- rectly fine-tuned AlexNet [32] on a two-class weather classifica- tion dataset released by Lu et al. [1], and achieved a better result. Lu et al. [2] combined hand-crafted weather features with CNNs extracted features, and further improved the classification perfor- mance. While as discussed in [2], there is no closed boundaries among weather classes. Multiple weather conditions may appear simultaneously. Therefore, all the above approaches suffer from the information loss when they treat weather recognition as a single label classification problem. Li et al. [37] proposed to use auxil- iary semantic segmentation of weather cues to comprehensively describe the weather conditions. This strategy can alleviate the problem of information loss, while the segmentation mask is not intuitive for humans.

2. Our approach

 

In this paper, to comprehensively describe the weather condi- tions, we propose to treat weather recognition as a multi-label classification problem. Furthermore, a CNN–RNN model is devel- oped for this task, which formulates  the  multi-label  classifica-  tion as a step-wise prediction. Fig. 3 demonstrates the architec-   ture of the proposed approach. It mainly composes of three parts, i.e., the basic CNN, a channel-wise attention model and a con- volutional LSTM. The CNN extracts the preliminary features of a given outdoor image. Specifically, the first five groups of convolu- tional/pooling layers of VGGNet [35] are utilized in this paper. The channel-wise attention model adaptively calculates the channel- wise attention weights and recalibrates the feature responses. The convolutional LSTM uses visual features and the hidden state to

The Recurrent Neural Networks, especially LSTM, has recently achieved overwhelming success in sequence modeling tasks, such as image/video captioning [38] and neural  machine  translation [39]. Without loss of generality, the LSTM can be formulated as follows [40].

it = σ (Wiwxt + Uihht−1 + bi ), ft = σ (Wfwxt + Ufhht−1 + b f ), ot = σ (Wow xt + Uoh ht−1 + bo ),

gt = tanh(Wgw xt + Ugh ht−1 + bg ), ct  = ft ◦ ct−1 + it ◦ gt ,

ht = ot ◦ tanh ct ,(1)

where the subscript t indicates the tth step of LSTM, xt  denotes  the input data, ht stands for the hidden state, ct  is the cell state. it,  ft and ot are input gate, forget gate and output gate of the LSTM, respectively. Ws , Us and bs are weights and biases to be learned.   σ , tanh and ◦ represent the sigmoid function, hyperbolic tangent function and element-wise multiplication, respectively. As shown in Eq. (1), at each step, the data xt and the previous hidden state ht−1 is taken as the input of current LSTM unit, and the historical information are recorded in the hidden state ht, such that LSTM  can exploit the temporal  dependency.

Although the standard LSTM has demonstrated its powerful ca- pability in sequence modeling tasks, the spatial information is ig- nored when processing images [23]. As can be seen from Eq. (1),

fully connections are used in state-to-state and input-to-state transformations. Generally, if the input image data xt ∈ RW × H × C , it will be flattened to an 1D-vector before input to the LSTM. While

this process will suffer from the loss of spatial information. To overcome this drawback, the convolutional LSTM is employed in our approach [23], which can be formulated as  follows.

it  = σ (Wiw 0 xt + Uih 0 ht−1 + bi ), ft = σ (Wfw 0 xt + Ufh 0 ht−1 + b f ), ot = σ (Wow 0 xt + Uoh 0 ht−1 + bo ),

gt = tanh(Wgw 0 xt + Ugh 0 ht−1 + bg ),

ct  = ft ◦ ct−1 + it ◦ gt ,

ht = ot ◦ tanh ct ,(2)

where 0 denotes the convolution operator and other symbols are the same with Eq. (1). It should be noted that the input feature       xt, cell state ct, hidden state ht and gates it, ft, ot  of convolu- tional LSTM are all 3D tensors, and convolution operations are used in state-to-state and input-to-state transformations. Therefore, the spatial information of features are preserved in this way. Further- more, the convolution operation actually has implicit spatial at- tention mechanism, since regions corresponding to the target la-  bel usually have higher activation responses. In the experiment,   we also find that the convolutional LSTM pays attention to several critical regions for weather label prediction, and achieves better re- sults than common LSTM with or without spatial attention   model.

3.1. Channel-wise attention model in the CNN–RNN architecture

 

Usually, different regions will be activated in disparate chan- nels of the feature map, and different image regions have different importance when estimating various weather conditions. In our CNN–RNN architecture, each step of the convolutional LSTM will predict one weather label. Inspired by Hu et al. [41], we propose    a channel-wise attention model for the CNN–RNN architecture to adaptively recalibrate the feature responses when predicting differ- ent weather labels. The illustration of the proposed channel-wise attention model is shown in Fig.  4.

losst  = −   「 pi,t log ～pi,t  + (1 − pi,t ) log(1 − ～pi,t ),(8)

4.1. Dataset description

 

4.1.1. The transient attribute dataset

The first dataset is transformed from an existing transient attribute dataset [19] which was originally erected for outdoor scenes understanding and editing. Although the transient attribute dataset is not specially designed for weather recognition, this dataset presents many appealing properties. First, images are cap- tured across many outdoor scenes including mountains, cities, towns and urban sceneries. Images in this dataset are of different scales and views, which enhances the diversity across scenes. Sec- ond, images are selected elaborately to ensure they exhibit various appearances of the same scene. Moreover, the authors of [19] de- fined 40 transient attributes for this dataset including weather re- lated attributes (e.g., ‘sunny’, ‘rain’, ‘fog’, etc.). For each image, the weather related attributes  are  annotated  non-exclusively,  which  is important for our multi-label weather recognition experiments. Several examples of the  transient attribute  dataset are  illustrated  in Fig. 5.

For weather recognition, six weather related attributes among  all 40 transient attributes are selected, i.e., ‘sunny’, ‘cloudy’, ‘fog’, ‘snow’, ‘moist’ and ‘rain’, others are ignored in our experiments. Besides, we find that there exists a few examples in which all weather attribute strengths are very low. Some of them are cap- tured at dawn and dusk, others do not show obvious features cor- responding to any weather categories. Therefore, we add an ‘other’ class to represent those examples where every attribute strength    is  lower  than  0.5.  It  is  noteworthy  that  the  strength  lower than

0.5 indicates the annotation workers do not think the image ex- hibits the corresponding attribute. In this paper, for the weather recognition task, weather attributes greater and lower than 0.5 are set to 1 and 0, respectively. Finally, the dataset contains seven weather classes and 8571  images in total. The detailed statistics    of the dataset are displayed in Table   2.

4.1.2. The multi-label weather classification dataset

To further evaluate the proposed  approach,  we  construct  a new dataset from scratch, which contains 10,000 images from 5 weather classes, i.e., ‘sunny’, ‘cloudy’, ‘foggy’, ‘rainy’ and ‘snowy’. All images are elaborately selected from Internet. Compared  to other weather recognition datasets, our  dataset  has  the  follow-  ing advantages. First, most of the existing datasets focus on only two or three weather classes, while our dataset covers all common weather conditions in the daily life. Second, the new constructed dataset contains many different scenes including cities, villages, ur- ban areas and so on, as depicted in Fig. 6. In addition, this dataset also exhibits different scales and views. Third, in our dataset, the weather labels are not mutually exclusive, which can provide more weather information.

The annotation of multi-label weather classification dataset was completed by a crowd-sourced task. The annotation workers are asked to determine weather attribute strengths non-exclusively for a given image, and the range of strengths is from 0 to 1, in which

0.5 is a demarcation point. Weather attribute strength lower than

0.5 indicates that the image cannot be judged as the corresponding weather condition (even if the image contains corresponding at- tribute). In this dataset, an image is annotated by at least five workers, and the average value of each attribute strength is se- lected as the result. To ensure the effectiveness of the annotation task, we also calculate the variance of each attribute strength for      a given image. If the variance is bigger than a threshold, the re-  sult will be re-determined by discussion. Finally, to generate the weather labels, all attribute strengths greater than or equal to 0.5  are set to 1,  others are set to   0.

Fig. 7 shows the weather label distribution on the two experi- ment datasets. The detailed statistics can also be found at Table 2. In both datasets, cloudy is the class with large number of samples. This is because that cloudy usually co-occurs with other weather conditions. Apart from cloudy, the new constructed dataset is more

analysis of co-occurrence relationships among different weather conditions is also conducted according to the following equation,

}, conc(i, j)

R(i, j) Q,(10)

}, I(i)

Q

 

where both i and j denote a kind of weather condition, R(i, j) indi- cates the measurement of the co-occurrence relationship between

where Arr(i) denotes the attribute strength of weather condition i,

∧ represents the conjunction symbols. In summary, Eq. (10) indi- cates the ratio between co-occurrence number of the two weather conditions and the occurrence number of weather condition i   over

all images. Therefore, }, j R(i, j) and }, j R( j, i) represent the influ-

ence and dependence of label i to others, respectively. To exploit the dependencies when predicting the weather labels, it is natural for us to predict the most influential label first and the dependent label last. Based on this, the following equation is utilized to rank

OP =

,(14)

N · K

tion, and the remains are for training. The experimental result is

shown in Table 3, from which we can see that the  proposed  ap-  proach CNN-Att-ConvLSTM achieves the best results on OP, OR and OF1, and comparable results with the  state-of-the-arts  on  AP,  AR  and AF1. CNN–LSTM with spatial attention model (CLA) also gets good results. While without spatial attention model, CNN–LSTM suffers from serious performance degradation. This indicates the importance of some key regions in the weather recognition task. To evaluate the influences of LSTM in the CNN–RNN framework, we also test CNN–GRU with spatial attention model (CGA), and find   CGA achieves almost the same results with CLA. CNN-ConvLSTM also gets similar results with CLA, which denotes the effectiveness     of convolutional LSTM in information extraction of key regions. Overall, the proposed approach perform better than  multi-label  version of AlexNet, VGGNet, the general multi-label approaches ML-KNN, ML-ARAM, and other CNN–RNN methods, which proves the  superiority  of  our approach.

For per-class result, all these methods perform worse on ‘rainy’ and ‘other’ classes. This is because that most images in transient attribute dataset present distant views. It is difficult to recognize  the rainy weather from such distant views. In addition, samples of ‘other’ class are very rare, and can be easily misclassified as sunny or cloudy in this  dataset.

4.2. Results on the multi-label weather classification dataset

 

For multi-label weather classification dataset, 2000 images are randomly selected for testing, 1000 images for validation, and the remains for training. As presented in Table 4, CNN-Att-ConvLSTM performs the best on almost all the evaluation metrics, which demonstrates the effectiveness of the proposed approach   again.

To analyze the effectiveness of our approach, some weather recognition examples are presented in Fig. 9. It includes the clas- sification results, activation maps and attention weights from our approach. The results of VGGNet are utilized for comparison, since our approach also uses it as the deep feature extractor.

Specifically, our approach works well on the above three images. From the selected activation maps and their attention weights, we can see that our approach can attend to the most correlated weather cues when predicting different weather labels, while the results of VGGNet are not so satisfactory. For example, the first image is annotated as sunny and foggy,  correspondingly the blue sky, the bright area and the region of haze have stronger responses in our activation maps, and the attention weights of cor- responding activation maps are relatively high when predicting dif- ferent labels. However, the ground is activated by VGGNet mistak- enly, which leads to the wrong label, i.e., rainy. Besides, our ap- proach fails on the rest two images, where the fourth image is an- notated as sunny and cloudy, which means an intermediate state between sunny and cloudy. However, only the cloud regions are activated, and the sunny label is lost in our approach. It is mainly because the sunny label is a little ambiguous. The fifth image is annotated as cloudy and rainy.  However,  due to the wet ground    is not so obvious, it is mis-classified as cloudy and foggy in our approach. Overall, the results in Fig. 9 indicate that our approach performs well in most cases, but sometimes fails when the anno- tation is ambiguous and the weather cues are not obvious. It is reasonable since our approach is just based on the visual features, and maybe better performance can be achieved with other modal- ity information, such as humidity, which can be taken into consid- eration in our future  work.

4. Conclusion

 

Considering that more than one  weather  conditions  may  oc- cur simultaneously in one image, we firstly analyze the drawbacks of taking weather recognition as a single label classification task,

and propose a multi-label classification framework for the weather recognition task. It allows one image to belong to multiple weather categories, which can provide more comprehensive description of the weather conditions. Specifically, it is a CNN–RNN architecture, where CNN is extended with a channel-wise attention model to extract the most correlated visual features, and a convolutional LSTM is utilized to predict the weather labels step by step, mean- while, maintaining the spatial information of the visual feature. Be- sides, we build two datasets for the weather recognition task to  make up the problem of lacking training data. Practically, the ex- perimental results have verified the effectiveness of the proposed approach.

In the future work, we plan to introduce the distribution pre- diction task for weather recognition [47–50], which cannot only classify the image with multi-labels, but also predict the strengths of different weather class, so as to describe the weather   conditions

more comprehensively. Besides, other modality information, such as humidity and temperature, can also be utilized in  the  future work.