Conant Pablos, Santiago EnriqueAmado Garfias, Alonso Javier2025-01-272024-12Amado Garfias, A. J. (2024), Real-time armed individual detection in video surveillance usingdeep learning and heuristic approaches [Tesis doctoral]. Instituto Tecnológico y de Estudios Superiores de Monterrey. Recuperado de: https://hdl.handle.net/11285/703118https://hdl.handle.net/11285/703118https://doi.org/10.60473/ritec.185https://orcid.org/0000-0001-6270-3164This researchaimstoenhancetheautomaticidentificationofarmedindividualsinvideo surveillanceinreal-time.Theproposedmethodologyinvolvesthedevelopmentofalgorithms specifically designedforthedetectionofindividualscarryinghandguns,whichincludepistols and revolvers.Toachievethis,theYOLOv4modelhasbeenselectedtodetectindividuals, handguns, andfaces.Subsequently,real-timeinformationisextractedfromtheYOLOmodel, including boundingboxcoordinates,distances,andintersectionareasbetweenhandgunsand individualswithineachvideoframe.Thisinformationfeedsourheuristicsanddifferentma- chine learning(ML)proposed,facilitatingtherecognitionofarmedindividuals.Severalchal- lenges mustbeaddressed,suchasocclusion,concealedguns,andproximityofindividualsto one another.Itencouragesthedevelopmentandcomparisonofdifferenttypesofsolutions. Theyaremadeupofthreeheuristics,seven-armedpeopledetectors(APD),and44APDto use ineachvideoframe(APD4F). The heuristicsaretheDeterministicMethodofCenters(DMC),theDeterministicMethod of Distances(DMD),andtheDeterministicMethodofIntersections(DMI).Furthermore, the APDmodelsareRandomForestClassifier(RFC-APD),MultilayerPerceptron(MLP- APD), k-Nearest-Neighbors(KNN-APD),SupportVectorMachine(SVM-APD),Logistic Regression(LR-APD),NaiveBayes(NB-APD),andGradientBoostingClassifier(GBC- APD). Thereby,IproposetocreateselectorsfordecidingwhichAPDtouseineachvideo frame (APD4F)toimprovethedetectionresults.Besides,weimplementedtwotypesof APD4Fs, onebasedonaRandomForestClassifier(RFC-APD4F)andanotherinaMultilayer Perceptron (MLP-APD4F).Wedeveloped44APD4FscombiningsubsetsofsixAPDs.The most ofAPD4FoutperformedoftheindependentuseofallAPDs.Amultilayerperceptron- based APD4F,whichcombinesanMLP-APD,aNB-APD,andaLR-APD,presentedthebest performance, achievinganaccuracyof95.84%,arecallof99.28%andanF1scoreof96.07%. This researchalsoproposesasolutiontooptimizetheproblemofdetectingarmedpeople when theweaponisnotvisible.Therefore,weapplyrecurrentneuralnetworks,suchasLong Short TermMemory(LSTM),topredictthecoordinatesoftheguns.Inthisway,itispossible to haveapredictionofarmedpeopleatalltimes.ThemeasurementbetweentheYOLO handgun detectionboundingboxesandtheLSTMpredictionresultedinanIoUof65.23%. When thefirearmdetectionbytheobjectdetectorisinterrupted,theweapon’spositionis generated bytheLSTMmodelsthat,togetherwiththeAPDs,identifythearmedpeople. When theLSTMsdeliveredtheirpredictionstotheAPDs,theNB-APDdemonstratedthe best performance,achievinganaccuracyof80.93%.TheLSTMsallowedtheanalysisof 5,288 recordsofthetestvideothatcouldnotbeanalyzedbeforeduetothelackofknowledge of thegun’sposition.TextoengopenAccesshttp://creativecommons.org/licenses/by-nc-sa/4.0INGENIERÍA Y TECNOLOGÍA::CIENCIAS TECNOLÓGICAS::TECNOLOGÍA DE LOS ORDENADORES::SISTEMAS EN TIEMPO REALTechnologyTesis de doctoradohttps://orcid.org/0000-0001-8447-3355Machine learningArmed people detectionComputer visiónObject detectionYOLOGuns563295