One of the major problems that we are facing is the increase in violent activities in society. There are many reasons for increased violent activities that need the immediate attention of society, but in this research article, we are discussing how we can detect violent behavior in surveillance videos. The surveillance cameras address the major concerns of our daily security needs and can record violent activities too. Today we are surrounded by surveillance cameras everywhere, such as in malls, streets, houses, and other public places. Surveillance cameras capture every movement of an object that comes under their range and produce huge visual data. This visual data can be analyzed to detect violent behavior manually, but it is almost impossible in this era where we are surrounded by so many cameras. This problem needs some technological intervention to detect the violence in surveillance videos as soon as it happens. Deep learning and computer vision have the potential to contribute a lot to this problem domain, specifically with pretrained models like vgg16 and vgg19. These pretrained models are already trained on a large dataset and capable of figuring out edges and shapes in scenes. We have used the improved visual geometry group 19 pretrained model to detect violence in video. The vgg19 consists of fixed sequence of convolutional layers, max pool layers and fully connected layers. Output is classified into 1000 different classes, but to fulfill the objective of this study, we have replaced the fully connected layers with our classification module to classify a video into two categories, namely violent and non-violent. The efficiency of the model is tested against three popular datasets, namely movie fights, hockey fights, and real-life violence situations. The accuracy of the proposed model is 96.25%, 98.75%, and 99.43% for the movie fight, hockey fight, and real-life violence situation datasets, respectively. The proposed model code is available on GitHub (https://github.com/profmahaveer/VGG19.git).