CycleGAN-based Approach for Masked Face Classification

Tomoya Matsubara and Ahmed Moustafa

Nagoya Institute of Technology, Nagoya, Japan

Keywords:

Machine Learning, Image, Pattern Recognition.

Abstract:

In this paper, we propose a learning model for not only distinguishing whether a person is wearing masks

but also classifying the position of the worn masks (mask on my chin, mask on my chin and mouth). First,

solutions to prevent the spread of COVID-19 and

keep oneself and others safe. In addition, a recent

study by researchers at the University of Edinburgh

to understand (Bandiera et al., 2020) the measures to

tackle the COVID-19 pandemic revealed the follow-

ing: Wearing a face mask or other cover that covers

the nose and mouth reduces the risk of coronavirus

infection. In this regard, an efficient system is much

needed that can recognize whether or not people’s

faces are masked in regulated areas and the position

FAces dataset (MAFA) (Ge et al., 2017), the Real-

World Masked Face Dataset (RMFD2) and a com-

prehensive masked face recognition dataset (Wang

et al., 2020) composed of Masked Face Detection

Dataset (MFDD), Real -world Masked Face Recogni-

tion Dataset (RMFRD) and Simulated Masked Face

Recognition Dataset (SMFRD).

Besides, many people have never worn masks or

are not wearing them properly due to bad habits or be-

havior. We then use the following dataset consisting

In addition, there are three types of incorrectly face

datasets (IMFD): nose and mouth masks, mouth and

chin only masks, and chin only masks:

However, this is a dataset created by synthesizing

fake masks. Therefore, we use CycleGAN (Zhu et al.,

2017) to take an approach that brings the mask of this

dataset closer to the mask of the real world. In ad-

dition, since the data set that can be used to detect a

human face mask is relatively small, transfer learning

is used to classify the presence or absence of a mask

concludes the paper and points out the future work.

2 PRELIMINARIES

2.1 GAN(Generative Adversarial

The Generative Adversarial Network (GAN) (Good-

fellow et al., 2014) consists of a generative model

G and a discriminative model D as shown in Figure

1, and learns the generation of data in which G is

indistinguishable from the original data x based on

the input noise z. do. The loss function of GAN

the second term teaches D to identify the data gener-

ated by G as not the original data. The relationship

between G and D here is often expressed by the re-

way, the counterfeiter and the police compete against

each other, and the counterfeiter learns how to make a

fake currency that is close to the real thing. Similarly,

in GAN, G and D are hostile to each other, allowing

G to generate data similar to the original data x. In

recent years, models such as CycleGAN and SRGAN

that perform super-resolution have been proposed by

applying GAN.

Pix2pix performs so-called Image-to-Image transfor-

mations that generate realistic objects from handwrit-

ten edges by acquiring transformation rules for each

image pair and unique loss functions between each

domain (Isola et al., 2017). Also propose the Cycle-

GAN to perform the Image-to- Image in the frame-

work of unsupervised learning by removing the pair

constraint of training data from this pix2pix (Zhu

et al., 2017).

CycleGAN (Zhu et al., 2017) is an advanced

version between X and Y, G

:Y → X, G

Since the mutual conversion here is performed by un-

supervised learning, there is no need for pairs between

datasets, and the constraints of datasets are relaxed.

The transformation obtained by solving this with the

following mini-max equation is G

, G

of Eq.(3) In

the work of Zhu et al., the loss function of his Cy-

of the Adversarial Loss learn to convert datasets to

each other. Eq.(5) is the ”Cycle Consistency Loss”

for making the transformations consistent, and Eq.(5)

teaches it to reduce the difference between the orig-

inal image and the reconstructed image when recon-

structed in two transformations. Fig.2 shows the data

transition when trying mutual conversion with a two-

dimensional image of apples and mikan as an exam-

ple of data transition in the CycleGAN, and the image

with a hat on the variable shows the converted image.

difficult. ResNet does not simply pass the conversion

F(x) by some processing block to the next layer like a

normal network, but shortcuts the input x to that pro-

cessing block, and H(x) = F(x) + x is passed to the

next layer. The processing unit including this short-

cut is called the residual module. In ResNet, the gra-

dient is directly transmitted to the lower layer during

backpropagation through the shortcut, and it has be-

come possible to learn efficiently even in a very deep

network.It is also used in Highway Networks (Srivas-

layers with 64 output channels. It is arranged. To be

precise, in addition to the convolutional layer, batch

normalization and ReLU, which will be described

later, are arranged, and ResNet uses a residual module

with the following structure

Here add shows the sum of F(x) and x. Fig.4(c)

shows the bottleneck version of the residual mod-

ule, which is a 1 × 1 convolution that reduces di-

mensions, then 3 × 3 convolutions, and then 1 × 1

module of Fig.4(b). Has been reported. Identity func-

tion f(x) = x is basically used as a shortcut for the

Residual module, but if the number of input chan-

nels and the number of output channels are different,

zero-padding is used to fill the missing channels with

0. Two patterns of shortcuts, projection, which ad-

just the number of channels by convolution of , 1 × 1,

are available as options. From these, the zero-padding

approach is better because it does not increase the pa-

learning. was there. Batchnormalization (Ioffe and

Szegedy, 2015) is a method to stabilize and speed up

learning by normalizing this internal covariate shift

and allowing each layer to learn independently as

much as possible. In ResNet, efficient learning of

deep networks is realized by incorporating this batch-

normalization in the residual module, and batchnor-

malization has come to be used as standard in the

models after ResNet.

3 PRPPOSED APPROACH

sible to identify a meaningful area, and the reason for

the identification has hardly been explained.

One way to know what shape or pattern the CNN

is looking at is to find out what features the CNN fil-

ter is looking at. However, since CNN learns to ex-

tract more complex features by combining simple fea-

tures extracted in the shallow layer in the deep layer,

it is necessary to express complex features in order to

know what CNN itself is looking at. It is necessary to

know the multiple simple features used in the above

the difference between images, it is necessary to know

not only the area but also the difference in shape and

pattern within the area, and it is not enough to specify

the area. Therefore, in order to explain fake masks

and real masks, there is a need for a method that can

identify the areas involved in their identification and

obtain differences in shapes and patterns within the

areas.

Therefore, in this study, we propose a method us-

ing the hostile generation network (GAN) as a method

there is CycleGAN, which is a model that applies

GAN. Since CycleGAN learns mutual conversion be-

tween datasets by unsupervised learning, there is no

need for data pairs between datasets, and it is pos-

sible to learn transformations that have no solution

in reality, such as mutual conversion between fake

masks and real masks.Specifically, the MaskedFace-

Net dataset is used as the ”fake mask” domain and

the MAsked FAces dataset (MAFA) (Ge et al., 2017)

is used as the ”real mask” domain to create a Cycle-

and G

each domain plays the role of a teacher domain with

each other, it should be considered that it exerts a

probabilistic learning effect on a data group such as

MaskedFace-Net that is not given clear teacher label

data.However, since CycleGAN converts the entire

image, the area related to the mask cannot be spec-

ified from the difference.

Therefore, in this study, we introduced the follow-

] (5)

In other words, the L1 norm is used to set the loss

function so that the distributions of the ”input” and the

”generated image” are close to each other. In other

words, by using this ”identity loss”, GAN works to

learn the conversion between each domain. Then, we

Figure 5: Schematic diagram of mask conversion learning between domains by CycleGAN Relationship between “fake

mask” domain X and “real mask” domain Y extracted from MaskedFace-Net G

: Learn Y → X using

an unsupervised method with no pair constraints.

Figure 6: Three types of training data sets. From left i)

MaskedFace-Net (original data), ii) Image converted by Cy-

cleGAN iii) Image converted by CycleGAN + Proposed

method.

• Changed Discriminator loss function to ”MSE”

instead of ”BCE”

stabilized by letting GAN learn the conversion be-

Table 1: Precision of each training data and classification.

Table 2: Recall of each training data and classification.

Figure 11: Mixed matrix classified using iii)Image con-

verted by CycleGAN + Proposed as training data.

tween domains using ”identity loss”. As a result, we

found that it is possible to classify the presence or ab-

sence of masks and their positions by CNN even with

composite images.

5 CONCLUSION

We suggested using CycleGAN and ”identity loss”

and three points to classify the presence and location

of masks. Experimental results have shown that the

having to meet people in person.

However, the classification accuracy of ”Mask

Chin” and ”Mask Mouth Chin” was not very good,

so it is necessary to work on improving the accuracy

by improving each model, verifying with more data,

and verifying the reliability of the result with differ-

ent data. We also want to develop a system that can

detect the presence and position of multiple people’s

masks at once by combining it with object detection

for multiple purposes.

Cabani, A., Hammoudi, K., Benhabiles, H., and Melkemi,

M. (2021). Maskedface-net–a dataset of cor-

rectly/incorrectly masked face images in the context

of covid-19. Smart Health, 19:100144.

Chowdary, G. J., Punn, N. S., Sonbhadra, S. K., and Agar-

wal, S. (2020). Face mask detection using transfer

learning of inceptionv3. In International Conference

on Big Data Analytics, pages 81–90. Springer.

Chu, C., Zhmoginov, A., and Sandler, M. (2017). Cy-

clegan, a master of steganography. arXiv preprint

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep resid-

Ioffe, S. and Szegedy, C. (2015). Batch normalization: Ac-

Jin, X., Qi, Y., and Wu, S. (2017). Cyclegan face-off. arXiv

Karras, T., Laine, S., and Aila, T. (2019). A style-based

Wang, Z., Wang, G., Huang, B., Xiong, Z., Hong, Q.,

Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. (2017).