Deep Learning Detector with Trained Model

[Cong-Lee]

Dear all，
I have been using it recently, and I am truly impressed with its capabilities. However, I believe there is room for improvement, especially in terms of detection accuracy. I would like to suggest incorporating deep learning models to enhance the detection performance.
Suggestion:
Adopting deep learning models can significantly improve the detection accuracy. These models have shown remarkable results in various detection tasks, and I think they could bring substantial benefits to your project. You can refer to the following link for some excellent deep learning models that might be suitable for your use case:

https://github.com/wentaozhu/AutoShot
https://github.com/soCzech/TransNetV2

Best regards,

[Breakthrough]

I'm open to having the ability to plug-and-play deep learning models into PySceneDetect for detection purposes, but I'm not sure the best way to integrate that. Will definitely have a look at the links you provided. One thing I've noticed is that a lot of folks are using PySceneDetect as part of pre-processing for data ingestion pipelines, so I'm weary about introducing some kind of feedback loop there.

I'm open to extending PySceneDetect to include capabilities to use pre-trained models, but this first requires coming up with a standardized interface for them. If anyone could help drive that aspect forward, please reach out!

[Breakthrough]

A recent blog post also shows how important this is for the future. It looks like TransNetV2 might be a good candidate to start with for integrating this. I haven't had time yet to dig into how the code functions, but it looks like the model itself is reasonable in size (~30 MiB or so).

Definitely worth exploring some more! I'll be doing so when I get some time, but as I mentioned already, if folks would like to help out with this, please reach out ☄️

[l1ebl]

@Breakthrough Since I've ported TransnetV2 to various private frameworks already, and I'm personally quite interested in bringing TransnetV2 to PySceneDetect, here is a basic proof of concept:

https://github.com/l1ebl/PySceneDetect/tree/transnetv2

An example of how to use it:

from scenedetect import detect, ContentDetector, TransnetV2Detector, split_video_ffmpeg
from pathlib import Path

path = Path("/Users/liebl/Desktop/A New Approach to Lighting - Shaping Atmospheres [w-LZhKYhGjM].mp4")

# https://www.dropbox.com/scl/fi/c7wrg2ulmg1moqnm456yc/transnetv2.onnx?rlkey=9kh1xpw9mx1zd7q4xxq1ptfq1&st=uupz3vmx&dl=0
model_path = Path.home() / "transnetv2.onnx"

det = TransnetV2Detector(model_path=model_path)
scenes = detect(str(path), det, show_progress=True)

split_video_ffmpeg(path, scenes)

A few remarks on this:

• The new detector relies on providing the pretrained TransnetV2 model as ONNX (I did the conversion manually, the converted model is available under the Dropbox link above). Since the goal here is model inference and longterm maintenance, ONNX should be better choice than a deep learning framework such as pytorch.
• If this is of interest I could add some more documentation on the actual sliding window logic which is kind of obscure both in the original TransnetV2 implementation as well as in my numpy-based implementation here

Further remarks on the state of the code:

• CLI commands are not yet supported
• Image resizing happens twice, once due to downscale the main loop, then in the detector itself; this is not ideal - some setting that would allow a detector implementation to tell the main loop to not apply any scaling would be helpful here I guess
• I tested the code on one video and it seems to work flawlessly

[Breakthrough]

I'm personally quite interested in bringing TransnetV2 to PySceneDetect

Very cool! Would love to get a PR going for this if you're keen. Everything you wrote sounds reasonable to me.

The new detector relies on providing the pretrained TransnetV2 model as ONNX (I did the conversion manually, the converted model is available under the Dropbox link above). Since the goal here is model inference and longterm maintenance, ONNX should be better choice than a deep learning framework such as pytorch.

ONNX sounds like a logical choice here, agreed. Could you share how you converted the model to ONNX? Even if that was done with a one-off script, we probably will want to check it in as a tool somewhere.

If this is of interest I could add some more documentation on the actual sliding window logic which is kind of obscure both in the original TransnetV2 implementation as well as in my numpy-based implementation here

That would definitely be appreciated (and useful for other folks in the future!). I think that would be very valuable.

CLI commands are not yet supported

No problem with this, we can certainly get a PR going to add the new detector and tackle the CLI as a follow up. Apologies for the current state of the codebase, it's going through a huge timecode API overhaul. It looks like your changes are already using those new APIs though, which is awesome! I don't think any changes will be required on your end for that, so this is mostly just a heads up.

Image resizing happens twice, once due to downscale the main loop, then in the detector itself; this is not ideal - some setting that would allow a detector implementation to tell the main loop to not apply any scaling would be helpful here I guess

The same issue happens right now for the perceptual hash detector. I would also like to have this fixed for the next release - filed #530 to track this.

[romg93]

Small note on transnetv2_detector.py:

process_frame() passes frame_img (BGR, as decoded by OpenCV) straight into cv2.resize(), with no conversion before inference.
The official TransNetV2 implementation explicitly expects RGB input (see the inference-pytorch README: "frame width x RGB (not BGR) channels").

Shouldn't a cv2.cvtColor(pixels, cv2.COLOR_BGR2RGB) be added after the resize ?

[bliebl]

@romg93 Good catch, I think you're right. PySceneDetect frame importers in general seem to output BGR, I should change and re-test this. I'm quite busy at the moment, hopefully soon.