diff --git a/README.md b/README.md index 9fdae79..7580ab8 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,8 @@ Please see [High Resolution Object Detection Pipeline](./doc/pipeline.md) for mo - FFmpeg is an open source project licensed under LGPL and GPL. See https://www.ffmpeg.org/legal.html. You are solely responsible for determining if your use of FFmpeg requires any additional licenses. Intel is not responsible for obtaining any such licenses, nor liable for any licensing fees due, in connection with your use of FFmpeg. -### Datasets & Attributions -- This project utilizes third-party open datasets. Please see our [Data Attributions](docs/DATASETS.md) for full licensing, copyright details, and citation parameters. +### Attributions +- This project utilizes third-party open datasets and a sample fine-tuned model. Please see our [Attributions](./doc/ATTRIBUTES.md) for full licensing, copyright details, and citation parameters. ## Install Prerequisites: @@ -39,9 +39,14 @@ Please see [High Resolution Object Detection Pipeline](./doc/pipeline.md) for mo ## Deploy High Resolution Drone Detection using Smart Filtering -All components for this application are dockerize. +All components for this application are dockerized. Scripts are provided to make deployment easier. + +### Model Preparation +If using our sample fine-tuned model for Drone Detection, please download `drone_detection.pt` from [Video Curation Sample Models: Drone Detection](https://github.com/IntelLabs/video-curation-sample-models.git) and save it as `fastapi/resources/models/ultralytics/custom_models/drone_detection.pt`. + + ### Start [Optional] To make sure there aren't any running containers for this application, run the following which stops the application and prunes containers: ```bash diff --git a/doc/DATASETS.md b/doc/ATTRIBUTES.md similarity index 55% rename from doc/DATASETS.md rename to doc/ATTRIBUTES.md index 4c9b9a4..48bdcad 100644 --- a/doc/DATASETS.md +++ b/doc/ATTRIBUTES.md @@ -1,24 +1,32 @@ -# Dataset and Third-Party Media Attributions +# Media Attributions -This repository utilizes external datasets and media for training/demo purposes, without storing the actual files. +This repository utilizes external datasets and media for training/demo purposes. +
---- ## 1. SynDroneVision Dataset * **Source:** [Zenodo (Record 13360116)](https://zenodo.org/records/13360116) * **License:** [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/legalcode.en) * **Citation:** See DOI [10.1109/WACV61041.2025.00742](https://doi.org/10.1109/WACV61041.2025.00742) (Lenhard et al., WACV 2025). -* **Usage:** External data for fine-tuning model for drone detection. +* **Usage:** External data for fine-tuning sample model for drone detection. Model generated using this data is available at [Video Curation Sample Models: Drone Detection](https://github.com/IntelLabs/video-curation-sample-models/tree/main/Drone_Detection). +
---- -## 2. Sample Drone Video (`anduril_swarm.mp4`) +## 2. Sample Drone Detection Model +* **Source:** [Video Curation Sample Models: Drone Detection](https://github.com/IntelLabs/video-curation-sample-models/tree/main/Drone_Detection) +* **License:** [AGPL 3.0](https://github.com/IntelLabs/video-curation-sample-models/tree/main/Drone_Detection/LICENSE) +* **Citation:** N/A +* **Usage:** Model fine-tuned using Ultralytics Yolo11n model and SynDroneVision dataset. +
+ + +## 3. [Optional] Sample Drone Video (`anduril_swarm.mp4`) * **Source:** [droneforge/yolov11-UAV-finetune](https://github.com/droneforge/yolov11-UAV-finetune/blob/main/anduril_swarm.mp4) -* **Usage:** External sample video for pipeline demonstration. +* **Usage:** Open sourced sample video for pipeline demonstration. +
---- -## 3. [Optional] DUT Anti-UAV Dataset +## 4. [Optional] DUT Anti-UAV Dataset * **Source:** [wangdongdut/DUT-Anti-UAV](https://github.com/wangdongdut/DUT-Anti-UAV) * **License:** [Apache License 2.0](https://github.com/wangdongdut/DUT-Anti-UAV/blob/master/LICENSE) * **Citation:** See DOI [10.48550/arXiv.2205.10851](https://doi.org/10.48550/arXiv.2205.10851) (Zhao et al., IEEE T-ITS 2022). diff --git a/doc/finetune.md b/doc/finetune.md index a83faea..8ec999b 100644 --- a/doc/finetune.md +++ b/doc/finetune.md @@ -11,7 +11,7 @@ If detecting only one class, be sure the dataset contains negative images (witho For a well rounded dataset, be sure it contains train, validation, AND test sets to follow the expected [Ultralytics YOLO Format](https://docs.ultralytics.com/datasets/detect/). The provided script checks if the original dataset contains `train`, `validation`, AND `test` directories. If it does not, it proceeds with converting the dataset into this format assuming the original dataset has `images` and `labels` directory with sub-directories `train`, `validation`, AND `test`. -If your dataset does not follow this format, please modify `prepare_dataset` in `finetune.py`. +If your dataset does not follow this format, please modify `prepare_dataset` in [`finetune.py`](../finetune/app/finetune.py). In this guide, the [SynDroneVision dataset](https://zenodo.org/records/13360116) is used. Please see their [paper](https://ieeexplore.ieee.org/document/10943801) for more details. @@ -19,7 +19,7 @@ The original dataset is saved in `SynDroneVision` directory and since it is not ## Training Configurations -The configurations used for training on 2x NVIDIA A100 80GB PCIe are specified in `include/train_args.py`. +The configurations used for training on 2x NVIDIA A100 80GB PCIe are specified in [`include/train_args.py`](../finetune/app/include/train_args.py). Feel free to modify these parameters based on your hardware limitations such as VRAM of GPU. @@ -49,7 +49,7 @@ THe following arguments are available: ## Deployment Python 3.10.12 on Ubuntu 22 was used for testing. -To manually setup your environment, use the provided `requirements.txt` and `requirements.GPU.txt` files. +To manually setup your environment, use the provided [`requirements.txt`](../finetune/requirements.txt) file. Be sure your system contains the required NVIDIA packages to use GPUs for training. To avoid modifying your system for training, you can use the provided Dockerfile to deploy a container. @@ -82,6 +82,7 @@ Please see below for instructions for deploying container via `docker` and `dock ``` Once all stages are completed, stop and/or remove running container. - -Keep note of the latest model, as this model will be copied to different location for inclusion in full application. +Keep note of the latest model, as this model will be copied to a different location for the detection application. + +A sample model, generated using these instructions, is available in [Video Curation Sample Models: Drone Detection](https://github.com/IntelLabs/video-curation-sample-models/tree/main/Drone_Detection). diff --git a/doc/pipeline.md b/doc/pipeline.md index fbbbe08..cf00f6e 100644 --- a/doc/pipeline.md +++ b/doc/pipeline.md @@ -71,7 +71,7 @@ The following are parameters used to fine-tune the Smart Filtering pipeline. The current implementation of the Smart Filtering pipeline is optimized for the test use-case, drone detection. Drones are typically small in the video frames so if your use-case of interest has different objects, it may be beneficial to test the pipeline results on an existing test video for your use-case. -For this case, we provide [`test_detections.py`](/fastapi/tests/test_detections.py) which annotates ROIs identified by the Smart Filtering pipeline onto each frame of the video for visual inspection. +For this case, we provide [`test_detections.py`](../fastapi/tests/test_detections.py) which annotates ROIs identified by the Smart Filtering pipeline onto each frame of the video for visual inspection. For testing purposes, you can use VSCode DevContainer (easiest method) or manually deploy the fastapi dockerfile. Using VSCode is straight forward, so here, we will manually deploy the fastapi Dockerfile as it contains the same setup used in the application AND start the test script. Here we will build the container, if not available. If behind proxy, be sure to set them using `--build-arg`. @@ -186,6 +186,8 @@ Place your model in the appropriate directory for the application. | `fastapi/resources/models/ultralytics/${MODEL_NAME}/FP16` | Ultralytics YOLO models are typically placed in this directory where MODEL_NAME is the short name for the model (i.e. `yolo11n`). The PT model (`${MODEL_NAME}.pt`) is exported to OpenVINO (`${MODEL_NAME}_openvino_model/`) or TensorRT (`${MODEL_NAME}.engine`), dependent on device used. | Please note the model labels are retrieved from the model directly, so the model must contain these details. + +A sample model is available in [Video Curation Sample Models: Drone Detection](https://github.com/IntelLabs/video-curation-sample-models/tree/main/Drone_Detection). If using this model, download and save model as `fastapi/resources/models/ultralytics/custom_models/drone_detection.pt`.
@@ -232,12 +234,12 @@ Here we provide details on each available test. | Component | Test File | Description | | --------- | --------- | ----------- | -| Model | [test_model.py](/fastapi/tests/test_model.py) | Test the model for GPU on provided RTSP URL or video file | -| Stream Readers | [test_readers.py](/fastapi/tests/test_readers.py) | Independently test the stream readers for GPU on provided RTSP URL or video file | -| Smart Filtering | [test_detections.py](/fastapi/tests/test_detections.py) | Independently test the detection pipeline (with and without Smart Filtering) only. Test does not include video clip generation or sending metadata to database for querying. | -| Stream Readers | [test_pipeline.py](/fastapi/tests/test_pipeline.py) | Scenario 1 tests the behavior of Readers when provided an invalid RTSP url.
Scenario 2 reads the RTSP url or video file for a specified duration or until it ends. | -| Video Clip Generation | [test_pipeline.py](/fastapi/tests/test_pipeline.py) | Scenario 3 mimics the clip generation within the pipeline. | -| Smart Filtering | [test_pipeline.py](/fastapi/tests/test_pipeline.py) | Scenario 4 tests the entire pipeline and saves output video of results. | +| Model | [test_model.py](../fastapi/tests/test_model.py) | Test the model for GPU on provided RTSP URL or video file | +| Stream Readers | [test_readers.py](../fastapi/tests/test_readers.py) | Independently test the stream readers for GPU on provided RTSP URL or video file | +| Smart Filtering | [test_detections.py](../fastapi/tests/test_detections.py) | Independently test the detection pipeline (with and without Smart Filtering) only. Test does not include video clip generation or sending metadata to database for querying. | +| Stream Readers | [test_pipeline.py](../fastapi/tests/test_pipeline.py) | Scenario 1 tests the behavior of Readers when provided an invalid RTSP url.
Scenario 2 reads the RTSP url or video file for a specified duration or until it ends. | +| Video Clip Generation | [test_pipeline.py](../fastapi/tests/test_pipeline.py) | Scenario 3 mimics the clip generation within the pipeline. | +| Smart Filtering | [test_pipeline.py](../fastapi/tests/test_pipeline.py) | Scenario 4 tests the entire pipeline and saves output video of results. | ### Test Model