Rishab K Pattnaik

Working in the Avatar team on interactive talking‑head avatars for B2B products that enhance user experience.

Key Contributions

• Data creation & compilation: Curated and processed SOTA large‑scale multimodal datasets (video, audio, facial landmarks) to train expressive avatars.
• Model research & development: Investigated state‑of‑the‑art generative architectures (GANs, diffusion models, normalizing flows) and loss functions to improve lip‑sync, facial dynamics, expression disentaglement and identity preservation.
• Real‑time deployment: Integrated the final model into a WebRTC‑based streaming pipeline for low‑latency, interactive avatar responses.
• Collaborated closely with product and engineering teams to iterate on model performance and user feedback.

Technologies

• PyTorch Lightning, TensorBoard
• GANs, Diffusion Models, Normalizing Flows
• WebRTC, FastAPI, Docker, Aiortc, Git, Micromamba

Currently engaged in Emergency Research within the Department of Surgery at Hamad Medical Corporation, one of the leading academic medical centers in the Middle East.

Key Contributions

• Developing multimodal AI frameworks for emergency triage prioritization using chest X-rays, clinical notes, and lab results.
• Building novel architectures combining Vision Transformers, Mamba state-space models, and LLMs for clinical decision support.
• Working on chronic kidney disease (CKD) diagnosis using vision-language models.
• Collaborating across BITS Pilani and Hamad Medical Corporation research teams.

Technologies

• PyTorch, HuggingFace Transformers, Mamba (state-space models)
• ViT, DINOv2, ClinicalBERT for multimodal fusion
• Docker, FastAPI, Streamlit for deployment pipelines

As a Research Assistant in the Department of ECE under Prof Rajesh Kumar Tripathy at BITS Pilani, I contributed to pioneering research in medical imaging with a focus on deep learning architectures for clinical diagnostics.

Key Contributions

• Developed a novel multiband-frequency aware network (wavelet-CNN) for bone fracture detection achieving 92.22% accuracy.
• Utilised Novel Fourier Block Transformer for detection of Osteopenia and Osteoporosis with high accuracy and less parameters for Edge Deployment.
• Co-authored research published in Healthcare Technology Letters (IET/Wiley), and IEEE Sensor Letters.
• Implemented multi-scale feature extraction combining wavelet decomposition with CNN backbone networks and leveraging frequency domains instead of Spatial for Medical Imaging.

Technologies

• PyTorch, TensorFlow, OpenCV, Android Studio
• Wavelet transforms (PyWavelets), CNN, Vision Transformers, Mamba
• Medical imaging datasets (MURA, custom X-ray collections)

Research internship at IGCAR, focused on advancing computer vision methods for industrial and scientific inspection applications.

Key Contributions

• Advanced camouflaged object detection by fine-tuning Meta's Segment Anything Model (SAM) for domain-specific use cases.
• Built custom data pipelines for annotating and augmenting camouflage detection datasets.
• Experimented with adapter-based fine-tuning strategies to improve SAM on low-contrast edge cases.
• Benchmarked performance against SOTA methods on CAMO and COD10K datasets.

Technologies

• PyTorch, Segment Anything Model (SAM), LoRA fine-tuning
• OpenCV, Albumentations for data augmentation
• CAMO, COD10K camouflage object datasets

Disease Overview

Pneumonia (PN)

• Type of respiratory disease affecting millions annually
• Caused by bacterial and viral infections
• Requires accurate and early detection for effective treatment

Tuberculosis (TB)

• Bacterial infection primarily affecting lungs and other body parts
• Global health concern with significant impact
• Early diagnosis crucial for preventing complications

Diagnostic Approach

Chest X-Ray (CXR) Imaging

• Widely used diagnostic modality for TB and PN detection
• Non-invasive and accessible imaging technique
• Standard practice in respiratory disease diagnosis

Computer-Aided Diagnosis (CAD) Systems

• Automated analysis of CXR images
• Assists radiologists in accurate disease diagnosis
• Reduces human error and improves diagnostic efficiency

Deep Representation Learning (DRL) Framework

Network Architecture

• Transfer learning blocks utilized as feature extractors
• Multiple dense layers for feature processing
• Final classification layer for disease categorization

Performance Evaluation

• Tested on CXR images from public databases
• 87.08% overall accuracy achieved
• Capable of classifying viral PN, bacterial PN, and TB diseases

Implementation and Deployment

Cloud-Based Framework

• DRL network deployed on cloud infrastructure
• Internet of Things (IoT)-enabled detection system
• Scalable and accessible diagnostic solution

Future Directions

• Development of CAD systems in federated learning environments
• Integration with IoT-based healthcare networks
• Enhanced privacy-preserving diagnostic capabilities

• Published: Healthcare Technology Letters, 2025
• DOI: 10.1049/htl2.70021
• Authors: Rishab Kumar Pattnaik, Dr. Rajesh Kumar Tripathy (BITS Pilani), Dr. Haipeng Liu (Coventry University)
• Received: July 14, 2025 | Revised: September 12, 2025 | Accepted: October 1, 2025

Abstract & Objective

• Developed an AI-driven automated approach for detecting bone fractures in muscle X-ray (MXR) images using the Multiband-Frequency Aware Deep Representation Learning Network (MFADRLN)
• Addresses critical challenges: subtle fracture detection, class imbalance, and limited annotated datasets
• Designed to improve patient outcomes through rapid, accurate diagnostic support

Novel Methodology

• Multiresolution Analysis: Utilized Discrete Wavelet Transform (2DDWT) to decompose MXR images into four frequency subbands (LL, LH, HL, HH)
• Architecture: Applied frozen EfficientNetV2B2-based Deep Representation Learning (DRL) blocks to each subband independently
• Feature Fusion: Concatenated subband-specific features and processed through dense layers with batch normalization and dropout
• Key Innovation: Captures frequency-specific and spatial information simultaneously for detecting subtle fracture patterns

Dataset & Training

• Datasets: Mendeley dataset (2048 fracture, 127 non-fracture) + FracAtlas dataset (717 fracture, 3337 non-fracture)
• Total Images: 6,229 MXR images with 72% training, 8% validation, 20% test split
• Hyperparameters: Adam optimizer, binary cross-entropy loss, batch size 32, 50 epochs, learning rate 0.001
• Regularization: Batch normalization, dropout layers, and early stopping to prevent overfitting

Results & Performance

• Test Accuracy: 92.22% with precision 93.40%, recall 88.87%, F1-score 0.841
• Cohen's Kappa: 0.8418 (excellent agreement beyond chance)
• AUC Metrics: AUC-ROC 0.959, AUC-PR 0.875
• Cross-Validation: 90.11 ± 0.42% average accuracy across 5 folds
• Generalization: Only 2.5% gap between training and validation accuracy

Comparative Analysis

• Transfer Learning Models: Outperformed ResNet50 (85.22%), DenseNet201 (84.57%), MobileNetV2 (84.57%), InceptionV3 (88.79%), XceptionNet (84.57%), EfficientNetV2B2 (88.14%)
• Transformer Models: Surpassed Vision Transformer ViT B/16 (84.42%) and Swin Transformer Base (87.84%)
• Literature Comparison: Superior to modified InceptionV3 (81%), VGG16 (82%), DenseNet169 (84%)
• Ablation Studies: All four wavelet subbands yielded optimal performance vs. individual or subset combinations

Model Interpretability

• Grad-CAM Visualization: Implemented for each subband to reveal model decision-making process
• LL-Subband: Captured broader structural information around fracture regions
• LH, HL, HH Subbands: Highlighted directional discontinuities and fine texture details
• Clinical Value: Enhanced trust by revealing frequency-specific features driving fracture detection

Real-World Deployment

• WebApp Platform: Deployed using Streamlit cloud-based framework for IoT-enabled detection
• Real-Time Predictions: Provides confidence scores (achieved 0.97 for fracture detection)
• Performance: Average prediction time of 12.01 seconds from upload to diagnosis
• Accessibility: Remote access, platform independence, enabling rapid clinical decision support

Technical Contributions

• First Application: Multiband-frequency domain DRL for bone fracture detection in medical imaging
• Novel Integration: Wavelet-based signal processing with modern deep learning architectures
• Efficiency: Reduced trainable parameters via frozen pre-trained blocks while maintaining superior performance
• Proven Impact: Frequency-domain decomposition significantly improved detection vs. direct image analysis

Clinical Impact & Future Directions

• Diagnostic Support: Reduces human error, minimizes treatment delays, improves patient outcomes in orthopedic care
• Key Strength: Identifies subtle fracture regions often missed in manual visual inspection
• Future Development: Lightweight models for embedded devices (FPGA, microcontrollers), federated learning for multi-hospital collaboration
• Multimodal Extension: Integration with EHRs and BERT-based language models for comprehensive diagnostic systems

• Published in: IEEE Sensors Letters, Volume 10, Issue 3, March 2026
• Article Sequence Number: 6003004
• Date of Publication: 03 February 2026
• Electronic ISSN: 2475-1472
• Authors: Rishab Kumar Pattnaik, Dr. Rajesh Kumar Tripathy (BITS Pilani), Dr. Ram Bilas Pachori (IIT Indore)

Abstract & Motivation

• Early detection of osteoporosis (OPRS) and osteopenia (OPNA) is critical for preventing fractures in the aging population.
• Existing deep learning methods rely on cloud processing, limiting real‑time, point‑of‑care deployment.
• This work proposes an Android‑based edge‑enabled lightweight Fourier block‑based transformer (LFBBT) for real‑time OPRS/OPNA detection directly on mobile devices using knee X‑ray sensor images.

Methodology – LFBBT Model

• Architecture: Patch embedding layer + discrete Fourier transform (DFT) block + dense layers + dropout + output layer.
• The DFT block captures frequency‑domain features, reducing computational complexity while retaining diagnostic information.
• Designed specifically for resource‑constrained mobile environments (Android) with TensorFlow Lite deployment.

Dataset & Evaluation

• Evaluated on a publicly available knee X‑ray database containing normal, osteopenia, and osteoporosis cases.
• Metrics: accuracy, precision, recall, F1‑score, and inference speed on Android devices.

Results & Performance

• Overall accuracy: 88.41% – surpassing various transfer learning models and pre‑trained transformers.
• Real‑time throughput: 60 images per minute on an Android smartphone.
• Enables instant, on‑device diagnosis without internet connectivity, ideal for rural and point‑of‑care settings.

Significance & Impact

• First lightweight transformer optimized for bone health screening directly on mobile hardware.
• Brings AI‑assisted diagnosis to the edge, reducing dependency on cloud infrastructure and improving accessibility.
• Demonstrates the feasibility of combining Fourier‑based processing with transformer architectures for medical imaging on low‑power devices.

Future Directions

• Extension to multi‑joint analysis (hip, spine) and integration with electronic health records.
• Further model compression for even faster inference on entry‑level smartphones.

U-Tube AI is a revolutionary multimodal AI agent that transforms YouTube videos into comprehensive knowledge assets by analyzing both audio and visual streams. Unlike mainstream tools (NoteGPT, Notta, MyMap.AI) that rely solely on transcripts, U-Tube AI employs adaptive frame sampling and OCR to capture slides, diagrams, and code shown on screen. This research-backed approach achieves 70-90% cost reduction compared to direct VLM processing while maintaining complete visual context that transcript-only tools miss entirely.

----------------------------------------------------------------------------------------------------------------

GitHub Repository: https://github.com/Rishab27279/U-Tube-AI

----------------------------------------------------------------------------------------------------------------

Core Innovation

U-Tube AI addresses the critical limitation of existing AI note-taking tools that are completely blind to visual content. The framework implements a sophisticated multimodal architecture combining:

• Adaptive Frame Sampling: Intelligent extraction of key visual moments (slides, diagrams, code snippets) rather than uniform processing
• Dual-Model Synthesis: Gemini Flash for rapid structure mapping, Gemini Pro for targeted deep synthesis
• Confidence-Based OCR Routing: Smart decision engine that triggers visual extraction only when visual content enhances understanding
• Cost Efficiency: Reduces processing costs from $139/hour (direct VLM) to $15-40 for hour-long videos while maintaining 90%+ information capture

Technical Architecture

The system operates through an intelligent dual-stage pipeline optimized for both quality and production-ready efficiency:

• Stage 1 - Structure Mapping: Gemini Flash analyzes full transcripts to extract topic hierarchies and identify visually dense sections requiring OCR
• Stage 2 - Deep Synthesis: Gemini Pro performs targeted deep dives on each section, fusing transcript and extracted visual data into coherent narratives
• RAG Pipeline: ChromaDB-powered vector database enables semantic search with timestamp retrieval and context-aware Q&A across video content
• Iterative Refinement: Low-confidence outputs trigger up to 3 enhancement loops, progressively improving note quality

Key Technical Advantages

• Research-Backed Sampling: Built on CVPR 2025 (AKS, BOLT) and AdaFrame principles showing intelligent frame selection outperforms uniform sampling by 5-6% while processing 70-85% fewer frames
• Multimodal Fusion: Hierarchical synthesis where structural analysis guides targeted visual extraction to information-dense segments
• Production-Ready Design: Practical cost efficiency for real-world use cases, avoiding prohibitively expensive full-video analysis
• Export Flexibility: Notes available in Markdown, PDF, and DOCX formats for seamless workflow integration

Performance Benchmarks

U-Tube AI demonstrates significant advantages over existing solutions while maintaining comprehensive understanding:

• Cost per Hour: $15-40 vs $139 (direct VLM processing) - 70-90% reduction
• Visual Content Capture: 100% of slides, diagrams, and code vs 0% for transcript-only tools
• Information Retention: Maintains >90% information capture despite selective frame processing
• Processing Efficiency: Moderate speed balancing thoroughness with resource optimization

Real-World Applications

The framework excels across diverse educational and professional content types, addressing critical gaps in existing tools:

• Technical Tutorials: Programming lectures with code editors and terminal outputs (Strivers, Programming with Mosh)
• Academic Content: Slide-heavy presentations and architectural diagrams requiring visual understanding (CampusX, MIT OpenCourseWare)
• Professional Development: Conference talks and technical presentations with visual aids and data visualizations
• Interactive Learning: RAG-powered chat enables Q&A with timestamp retrieval for targeted video review

Intelligent Decision System

Not every video requires frame-by-frame analysis. U-Tube AI implements a confidence-based routing engine that optimizes resource allocation:

• Visual Density Evaluation: Assesses whether visual content would enhance note quality before triggering OCR
• Talking-Head Optimization: Avoids unnecessary frame processing when transcripts alone suffice
• Iterative Refinement Loops: Low-confidence outputs trigger enhancement cycles up to 3 times
• Cost-Quality Balance: Ensures efficient resource usage without compromising comprehensive understanding

Open Source Commitment

• Complete Implementation: Full codebase with Streamlit UI, dual-model orchestration, and RAG pipeline

U-Tube AI my try of a paradigm shift in YouTube content analysis by treating visual information as first-class data rather than an afterthought. The framework combines cutting-edge computer vision research with practical cost efficiency, enabling students and professionals to efficiently digest complex technical content without losing critical visual context that drives true understanding.

EdgeSeg-AI is a revolutionary framework that makes advanced image segmentation accessible to everyone by introducing a novel, resource-efficient approach to prompt-based image segmentation. The framework addresses the computational limitations of cutting-edge segmentation methods by sequentially orchestrating three specialized models: Large Language Model (LLM), Fine-tuned VLM and Segment Anything Model (SAM). This innovative architecture achieves a 60-70% reduction in peak memory usage while maintaining high segmentation quality.

----------------------------------------------------------------------------------------------------------------

GitHub Repository: https://github.com/Rishab27279/EdgeSeg-AI

----------------------------------------------------------------------------------------------------------------

Core Innovation

EdgeSeg-AI presents a unique architectural approach that sequentially orchestrates three specialized models, representing a fundamental shift from conventional approaches that load all models simultaneously:

• Large Language Model (LLM): Complex prompt interpretation and simplification
• Fine-tuned Florence-2 (VLM): Precise object detection from natural language descriptions
• Segment Anything Model (SAM): High-quality mask generation
• Memory Efficiency: 60-70% reduction in peak memory usage while maintaining segmentation quality

Technical Architecture

The framework operates through a carefully designed three-stage pipeline that maximizes efficiency and accessibility:

• Prompt Simplification: Complex natural language queries are processed and refined for optimal object detection
• Object Detection: Florence-2 generates precise bounding boxes from simplified prompts
• Segmentation: SAM produces detailed masks using the detected bounding boxes as guidance

Key Technical Advantages

• Interactive Prompt Refinement: User control and transparency in the segmentation process
• Memory-Efficient Sequential Loading: Revolutionary approach to model orchestration
• Maintained Accuracy: High segmentation quality despite resource constraints
• Consumer Hardware Compatible: Works with standard 4GB+ GPU memory

Performance Benchmarks

EdgeSeg-AI demonstrates significant improvements in resource efficiency while maintaining segmentation quality:

• Peak Memory Usage: Reduced from 12-16GB to 4-6GB (60-70% reduction)
• Minimum GPU Memory: Reduced from 8GB to 4GB (50% reduction)
• Processing Time: 3-5 minutes vs 1 minute (trade-off for accessibility)
• Segmentation Quality: High quality maintained across all metrics

Real-World Applications

The framework demonstrates robust performance across diverse contexts, showcasing the potential for democratizing AI-powered image analysis:

• Waste Management: Environmental applications ('Where should I throw the wrapper?')
• Automotive Interfaces: Vehicle dashboard analysis ('Where should I look to know the speed?')
• Multi-Domain Versatility: Adaptable across various industrial and consumer applications
• Accessibility Focus: Designed for widespread adoption and implementation

Research Community Engagement

This work builds upon the foundational contributions of the LLM-Seg paper by Junchi Wang and Lei Ke from ETH Zurich, adapting their approach to prioritize computational efficiency and accessibility. The research community's insights are invaluable for advancing this work further in the following areas:

• Benchmarking: Against additional state-of-the-art methods
• Dataset Evaluation: Testing on diverse datasets and use cases
• Memory Optimization: Strategies for further memory reduction
• Novel Applications: Implementation in resource-constrained environments

Open Source Commitment

• Full Open Source: Complete codebase available for exploration and contribution
• Community Driven: Open for critical evaluation and collaborative improvement
• Accessibility Focus: Democratizing advanced AI capabilities for everyone
• Research Transparency: Supporting reproducible research and innovation

EdgeSeg-AI represents a significant advancement in making sophisticated image segmentation technology accessible to a broader audience, combining cutting-edge AI research with practical resource efficiency. The framework's innovative sequential model loading approach opens new possibilities for deploying advanced computer vision capabilities on standard consumer hardware, democratizing access to powerful AI tools across various domains and applications.

Moody.AI is a cutting-edge multimodal AI system that analyzes emotions from video content using computer vision, audio processing, and natural language processing. Powered by state-of-the-art deep learning models including DINOv2, Wav2Vec2, DistilBERT, and Whisper, the system provides comprehensive sentiment analysis with an intuitive web interface and achieves 61% accuracy on the challenging MELD dataset.

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Docker Hub: https://hub.docker.com/r/rishab27279/moody-ai | GitHub: https://github.com/Rishab27279/MoodyAI

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

System Features

• Multimodal analysis with simultaneous processing of vision, audio, and text
• Real-time emotion classification with interactive confidence scores
• Production-ready containerized deployment with Docker
• Modern glass morphism UI with animated visualizations
• Supports video files (MP4, MOV, MKV) up to 200MB
• Fast processing: 10-30 seconds per video on GPU

Model Architecture and Performance

Moody.AI employs a sophisticated trimodal fusion architecture combining multiple AI models:

• Vision Processing: DINOv2 ViT-B/14 for facial expression analysis with 768-dimensional embeddings compressed to 256 dimensions
• Audio Processing: Fine-Tuned-Wav2Vec2-base Model for emotional audio representations achieving 32% accuracy on MELD
• Text Processing: Fine-tuned DistilBERT-base-uncased achieving 50% accuracy on MELD dataset
• Speech-to-Text: OpenAI Whisper for multilingual transcription with automatic language detection

Trimodal Fusion Innovation

The breakthrough architecture combines all three modalities through advanced fusion techniques:

• Vision Feature Compression: Novel noise reduction technique that improved accuracy from 45-49% to 61%
• Cross-Modal Attention: Advanced fusion mechanisms enabling interaction between text, audio, and vision features
• Progressive Architecture: Systematic evolution from bimodal (56-57% accuracy) to trimodal fusion
• Dropout Regularization: 0.3 and 0.2 dropout rates to prevent overfitting in fusion layers

Video Processing Pipeline

• Frame Extraction: ~1 frame per second sampling with CVLib face detection
• Audio Extraction: MoviePy conversion to 16kHz WAV with Whisper transcription
• Feature Extraction: Parallel processing of vision (518×518), audio (768-dim), and text (512 tokens) features
• Multimodal Fusion: Cross-attention mechanism with concatenated feature fusion
• Emotion Classification: Softmax output across 5 emotion classes

Technical Stack

• Deep Learning: PyTorch 1.13.0+ with Transformers 4.21.0+
• Web Interface: Streamlit 1.24.0+ with custom CSS styling
• Computer Vision: OpenCV for face detection and image processing
• Audio Processing: Librosa 0.9.2+ and MoviePy 1.0.3+ for audio extraction
• Model Optimization: TensorFlow Lite compatibility with TIMM 0.6.7+

Emotion Categories Detected

• Happy 😄
• Surprise 😢
• Angry 😠
• Melancholy 😨
• Neutral 😐

Performance Benchmarks

• Overall Accuracy: 61% on MELD dataset (competitive with SOTA DialogueRNN at 60.25%)
• Processing Speed: GPU (RTX 3050) processes videos in 10-20 seconds
• Memory Usage: ~4GB model loading, 6-7GB peak usage for large videos
• F1-Score: 0.59-0.63 macro average across emotion classes

User Interface and Deployment

• Web Application: Streamlit-based interface with glass morphism design
• Processing Modes: Lightweight, Balanced, and High Fidelity options
• Docker Deployment: One-command deployment with pre-built container
• Cloud Ready: Compatible with Streamlit Cloud, AWS, Azure, and Google Cloud Run
• Interactive Results: Real-time emotion visualization with confidence scores and probability distributions

Hardware Requirements

• Minimum: 8GB RAM, CPU-only processing (30-60 seconds per video)
• Recommended: 16GB RAM, NVIDIA GPU with 4GB+ VRAM
• Optimal: 32GB RAM, Modern GPU like RTX 3060+ (5-10 seconds per video)

This project demonstrates the advancement of multimodal AI on edge devices, combining computer vision, natural language processing, and audio analysis into a comprehensive emotion recognition system with state-of-the-art performance.

OsteoDiagnosis.AI is an innovative Android application that leverages advanced signal processing and deep learning techniques for automated bone health assessment. The app utilizes a novel, lightweight neural network architecture combining Signal Processing (Fourier) and Deep Learning to classify bone density conditions into three categories: Osteoporosis, Osteopenia, and Normal bone density. This research-driven project represents a significant advancement in mobile healthcare AI, combining our cutting-edge computer vision with clinical diagnostic applications.

----------------------------------------------------------------------------------------------------------------

APK: https://github.com/Rishab27279/OS_Detection_Binary_And_3_Class_DWT/releases/download/v1.0/app-debug.apk

----------------------------------------------------------------------------------------------------------------

Research Context

This project was developed under academic supervision as part of ongoing research in medical AI diagnostics in BITS Pilani Hyderabad. While the complete technical methodology and results are currently confidential as the Research Paper is in it's last stages, the application demonstrates the successful integration of signal processing techniques with Modern deep learning architectures for bone health assessment.

Application Features

• Three-class bone density classification (Osteoporosis, Osteopenia, Normal)
• Real-time inference capabilities on mobile devices
• Offline processing with no internet connectivity required
• Intuitive user interface designed for clinical workflow integration
• Optimized model architecture for mobile deployment efficiency (Model Size < 100Mb)
• High accuracy performance validated through rigorous testing protocols and Benchmarks (Accuracy ~ 90% ; Kappa Score ~ 0.82)

Technical Innovation

The core innovation lies in the fusion of traditional signal processing methodologies with state-of-the-art deep learning approaches:

• Novel Architecture: Custom-designed neural network optimized for bone density classification tasks
• Signal Processing Integration: Powerful Signal Processing techniques enhance feature extraction and model performance
• Mobile Optimization (< 100 Mb): Lightweight model design ensures efficient on-device inference
• High Accuracy (~90%): Achieved superior classification performance through innovative architectural design

Research Methodology

Due to the ongoing nature of this research and pending publication, specific technical details regarding the model architecture, training protocols, and validation datasets remain confidential. The methodology represents a novel contribution to the field of medical AI, particularly in bone health diagnostics.

Technical Implementation

• Android Studio: Native Android application development environment
• Deep Learning Framework: Advanced neural network implementation for mobile deployment[4]
• Signal Processing Libraries: Integrated preprocessing pipeline for enhanced feature extraction
• Mobile Optimization: Efficient model compression and inference optimization techniques

Classification Categories

• Normal Bone Density: Healthy bone structure classification
• Osteopenia: Mild bone density reduction detection
• Osteoporosis: Severe bone density loss identification

Clinical Significance

• Early Detection: Enables timely identification of bone health deterioration
• Accessibility: Mobile-first approach increases diagnostic accessibility
• Efficiency: Rapid classification reduces diagnostic time and healthcare costs
• Accuracy: High-performance model ensures reliable clinical decision support

This project demonstrates the successful application of advanced AI techniques to critical healthcare challenges, showcasing the potential for mobile-deployed deep learning solutions in clinical diagnostics. The combination of academic rigor with practical implementation highlights the intersection of research innovation and real-world healthcare applications.

Expression.AI is a real-time facial expression recognition Android application that uses deep learning to detect and classify human emotions. Powered by a custom made TensorFlow Lite model named ResInceptionCNN, the app is designed for fast, on-device inference and intuitive user experience. It combines the power of computer vision with emotion AI for mobile devices.

App APK -> https://github.com/Riiishaab/Expression.AI/releases/download/v1.0/ExpressionAI.apk

Application Features

• Real-time facial emotion detection using the device's camera
• Supports classification from uploaded images via gallery
• Runs fully offline with no internet required for inference
• Emotion results displayed with matching emojis and labels
• Fast and efficient performance optimized for mobile usage

Model and Dataset

Expression.AI is built on a custom deep learning model trained for facial expression recognition:

• Model Name: ResInceptionCNN — a hybrid architecture combining strengths of Residual and Inception networks
• Dataset: Trained on the FER2013 dataset containing 48x48 grayscale facial images labeled with 7 emotion classes
• Conversion: Exported to TensorFlow Lite format for mobile deployment

ResInceptionCNN Architecture

The core of the project is the ResInceptionCNN model:

• Inception Blocks: Capture multi-scale spatial features for rich local and global context
• Residual Connections: Enable deeper networks by addressing vanishing gradients and improving feature flow
• Batch Normalization: Applied after each convolution to stabilize and speed up training
• Dropout Layers: Reduce overfitting and improve generalization
• Final Dense Layer: Outputs emotion class probabilities through a softmax activation

Technical Stack

• Android Studio: Used for developing the UI and app logic
• TensorFlow Lite: Facilitates fast, on-device model inference
• OpenCV: Handles face detection from camera and gallery inputs
• CameraX: Provides real-time camera feed with face capture

Emotion Categories Detected

• Angry 😠
• Disgust 😖
• Fear 😨
• Happy 😄
• Sad 😢
• Surprise 😲
• Neutral 😐

User Interface

• Live Camera View: Real-time emotion recognition with facial overlay
• Image Upload: Choose from gallery for static image analysis
• Emoji and Label Output: Emotion results shown with emojis and text
• Minimal UI: Clean design with clear interaction flow and responsive layout

This project demonstrates the potential of deep learning on edge devices, blending AI, mobile development, and human emotion understanding into a seamless Android experience.

This project implements real-time hand gesture recognition using Apple's FastViT architecture, leveraging transfer learning on the HaGRID dataset to achieve 97.5% accuracy while maintaining efficiency for real-time applications.

🔑 Key Features

• Real-time gesture recognition through webcam integration
• 19 different hand gestures recognized with high accuracy
• Efficient model architecture using FastViT for low-latency predictions
• Simple deployment through Google Colab (no local setup required)

🧠 Model & Architecture

The core intelligence is provided by a hybrid vision transformer:

• Backbone: fastvit_t8.apple_in1k pretrained model
• Training approach: Transfer learning with a frozen backbone
• Input size: 256×256 px
• Classes: 19 hand gestures

FastViT was chosen for its efficiency advantages over ConvNeXT, offering high accuracy in resource-constrained environments.

📊 Dataset

Trained on the Hand Gesture Recognition Image Dataset (HaGRID) 150k subset:

• 19 gesture classes including 'thumbs up', 'peace sign', and 'stop'
• Used the manageable 150k version for Colab training
• Properly split between training and validation sets

🚀 Usage

Option 1: Google Colab

• Open the training notebook
• Run all cells to train or load pretrained weights
• Follow webcam integration instructions

Option 2: Inference with Pretrained Model

• Open the inference notebook
• Upload the pretrained model file (sign_lang_model.pkl)
• Run the webcam inference cell

📈 Results

• 97.5% accuracy on the validation set
• Robust across different lighting conditions
• Real-time inference (>30 FPS on modern hardware)

🔮 Future Work

• Expanded gesture vocabulary: Full sign language alphabet and phrases
• Standalone deployment: Integration with video conferencing
• Sequence modeling: Temporal information for dynamic gestures
• Model optimization: Quantization and pruning for edge devices

Developed an AI-driven document processing system combining DeepSeek R1-1.5B for structured data extraction and Llama-7B for summarization. Achieved 92% accuracy in entity extraction and 88% ROUGE-L summarization scores, enabling efficient processing of legal and technical documents.

This code implements a comprehensive Checkers (Draughts) game featuring two AI players with contrasting strategies: a sophisticated Smart AI using the Minimax algorithm with alpha-beta pruning and a baseline Random AI that makes random legal moves.

Game Structure

• The game utilizes an 8×8 checkerboard grid with traditional alternating dark and light squares
• Two AI players compete: Red (Smart AI) and Blue (Random AI)
• Turn-based gameplay with visual representation of each move
• King promotion mechanics when pieces reach the opponent's back row
• Standard checkers rules including mandatory captures and multiple jumps

AI Implementation

The Minimax algorithm implementation represents the core intelligence of the Smart AI:

• Depth-Limited Search: The Smart AI (Red player) looks ahead 3 moves using the Minimax algorithm
• Alpha-Beta Pruning: Optimizes the search by eliminating branches that won't affect the final decision
• Recursive Evaluation: Each potential move is evaluated by simulating future game states
• Heuristic Evaluation: Board positions are scored based on piece count, king status, and positioning
• Maximizing Strategy: The AI selects moves that maximize its advantage while assuming optimal opponent play

The Random AI provides a contrasting approach:

• Identifies all pieces with legal moves
• Randomly selects one of these pieces
• Randomly selects one of the legal moves for that piece
• Executes without strategic planning

Game Mechanics

• Move Validation: Comprehensive checking of legal moves including diagonal movement and jumps
• Capture Logic: Implementation of mandatory jump rules and multiple capture sequences
• Visual Representation: Board state displayed using HTML/CSS with color-coded pieces
• Game Termination: Ends after 100 moves or when a player has no remaining pieces/moves
• Winner Determination: Based on remaining piece count or inability of opponent to move

User Interface

• Interactive Display: Color-coded pieces with clear visual distinction between regular pieces and kings
• Move Animation: Step-by-step visualization with appropriate delays for user comprehension
• Notifications: Popup alerts for game start, end, and important events
• Status Updates: Ongoing information about game progress, turn count, and current player
• Board Rendering: Clear representation of the current game state after each move

This implementation demonstrates advanced concepts in game AI development.

Implemented a gender detection system using hybrid CNN-InceptionV3 architecture, achieving 94.35% accuracy on the CelebA dataset. Features include dynamic augmentation, adaptive learning rate scheduling, and quantized TensorFlow Lite deployment.