data-preprocessing-pipelines

Open source project for data preparation for GenAI applications

Python package for Customizable Data Preprocessing Pipelines

A lightweight framework for collecting and processing data from HTTP POST requests

Video quality assessment and filtering pipeline for ML training data. Automatically handles format conversion, scene segmentation, face detection, text detection, and audio-video sync checking. Supports 127 concurrent processes with checkpoint recovery

This repository containing code for preprocessing text data from PDF and DOCX files for use with GPT-3. It includes steps such as tokenization, removal of stop words and punctuation, and formatting for GPT-3 input.

The data process library to help better industrial data understanding.

Understand and Implement decision tree

This repository contains a sample text data-preparation code using Nemo Curator for pre-training or synthetic data generation

Pymimic3 is a scalable experimentation platform for MIMIC-III, featuring ready-to-run models, fully tested utilities for concept drift research, and a parallelized, configurable data pipeline.

Project for Machine Learning Data Mining course

A from-scratch OpenCV pipeline that quality-checks, normalizes, and skull-strips brain MRI scans structured after the intake pipelines real medical-imaging teams run before any scan reaches a model or a radiologist.

This work highlights my contribution as a "ML Engineer" at "adorsho praniSheb"(an ML based agro farming company of Bangladesh) where I was assigned the task of designing the preprocessing pipeline.

Comparative study of CNN and SVM models for facial emotion recognition on CK+ (CNN: 96%, SVM: 97%) and RAF-DB (CNN: 85%, SVM: 77%) datasets. Full data preprocessing pipeline in Python. Published in Springer 2024.

Core computational engine for the sabby ecosystem. Provides the low-level C++, Fortran, and OpenMP infrastructure required for heavy analytical workloads.

Machine learning models cannot be directly applied to raw data. This desktop application consists of a central server and two client servers. The main servers send raw data to clients, where the data is preprocessed and prepared to be fed to the machine learning model.