Deep Learning vs Machine Learning

Machine Learning: systems that learn from data without being explicitly programmed

Includes:
  Linear / Logistic Regression
  Decision Trees and Random Forests
  Support Vector Machines (SVM)
  k-Nearest Neighbours
  Naive Bayes
  Gradient Boosted Trees (XGBoost, LightGBM)
  Deep Learning (neural networks with many layers)

All these methods learn patterns from training data.
Deep learning is a subset of ML.

Traditional ML:
  Feature engineering: human expert creates features from raw data
  Model: learns a function from these hand-crafted features to the label
  
  Example (clinical risk score):
    Human decides: age, BP, cholesterol, BMI are relevant features
    Model: logistic regression on these 4 features

Deep Learning:
  End-to-end learning: model learns BOTH features AND the prediction function
  Feature learning happens automatically in early layers
  
  Example (ECG classification):
    Raw input: 12-lead ECG signal (thousands of time points)
    Model: CNN learns which waveform patterns matter → classification
    No human feature engineering needed

Shallow ML (1-2 layers):
  Logistic regression, SVM, 1-hidden-layer neural net
  Limited representational capacity
  Requires good feature engineering

Deep learning (3+ layers, often dozens):
  Each layer learns a more abstract representation
  
  Computer vision example:
    Layer 1: edges and corners
    Layer 2: shapes and textures
    Layer 3: object parts (eyes, wheels, fins)
    Layer 4+: complete objects
  
  NLP example:
    Layer 1: character/token patterns
    Layer 2: morphemes, word forms
    Layer 3: syntactic structures
    Layer 4+: semantic meaning
  
The "depth" enables hierarchical feature learning without human intervention.

Aspect          | Traditional ML          | Deep Learning
----------------|-------------------------|----------------------------------
Feature eng.    | Manual (domain expert)  | Automatic (learned)
Data needed     | Moderate (100s–10Ks)    | Large (10Ks–billions)
Compute         | Low–moderate (CPU)      | High (GPU/TPU required)
Interpretability| High (linear models,    | Low (black box)
                | decision trees)         |
Training speed  | Fast (seconds–minutes)  | Slow (hours–weeks)
Inference speed | Fast                    | Moderate (depends on model)
Works on        | Tabular, structured     | Images, text, audio, time series,
                | data                    | unstructured data
Handles raw     | No (needs features)     | Yes
input           |                         |

# Decision framework

def choose_approach(
    data_size: int,
    data_type: str,    # "tabular", "image", "text", "audio", "time_series"
    compute_budget: str,  # "low", "medium", "high"
    interpretability_required: bool,
) -> str:
    
    if data_type in ("image", "text", "audio") and data_size > 10_000:
        return "deep_learning"  # DL dominates on unstructured data
    
    if data_type == "tabular":
        if data_size < 10_000:
            return "gradient_boosting"  # XGBoost/LightGBM often wins small tabular
        if interpretability_required:
            return "logistic_regression_or_decision_tree"
        if data_size > 1_000_000:
            return "deep_learning_or_gradient_boosting"  # test both
        return "gradient_boosting"  # XGBoost usually wins mid-size tabular
    
    if compute_budget == "low":
        return "traditional_ml"
    
    return "deep_learning"

In practice, for tabular data:
  Gradient boosted trees (XGBoost, LightGBM, CatBoost) often beat deep learning
  They require less data, less compute, and less tuning
  
  Kaggle competitions on tabular data: trees win most of the time

For unstructured data (images, text, audio):
  Deep learning dominates completely
  Pre-trained models (BERT, ResNet, Whisper) enable transfer learning
  with moderate data sizes

Clinical AI example:
  Structured EHR features (lab values, vitals, demographics):
    → XGBoost often wins over neural networks
  ECG signals, chest X-rays, clinical notes:
    → Deep learning (CNN, Transformer) is the right choice