PyTorch vs TensorFlow

PyTorch (define-by-run, dynamic graph):
  The computation graph is built on-the-fly during the forward pass.
  Python control flow (if, for) works naturally inside model code.
  Easy to debug: print and inspect tensors anywhere.
  
TensorFlow 1.x (define-then-run, static graph):
  The computation graph is defined first, then compiled, then run.
  Fast and optimised, but hard to debug — graphs are opaque.
  Required special tf.cond / tf.while_loop for control flow.

TensorFlow 2.x with Keras:
  Added eager execution by default (dynamic, like PyTorch).
  tf.function decorator compiles to a static graph for performance.
  Bridged the gap — now more similar to PyTorch in usage.

# ───────────────── PyTorch ─────────────────
import torch
import torch.nn as nn
import torch.optim as optim

class MLP_PyTorch(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 64)
        self.fc2 = nn.Linear(64, 1)
    
    def forward(self, x):
        return self.fc2(torch.relu(self.fc1(x)))

model_pt = MLP_PyTorch()
optimizer = optim.Adam(model_pt.parameters(), lr=1e-3)
criterion = nn.BCEWithLogitsLoss()

# Training loop
for X, y in dataloader:
    optimizer.zero_grad()
    output = model_pt(X)
    loss = criterion(output.squeeze(), y.float())
    loss.backward()
    optimizer.step()


# ───────────────── TensorFlow/Keras ─────────────────
import tensorflow as tf
from tensorflow import keras

model_tf = keras.Sequential([
    keras.layers.Dense(64, activation="relu", input_shape=(10,)),
    keras.layers.Dense(1, activation="sigmoid"),
])
model_tf.compile(optimizer="adam", loss="binary_crossentropy")

# Training — managed by fit()
model_tf.fit(X_train, y_train, epochs=10, batch_size=32, validation_split=0.2)

Aspect             | PyTorch                      | TensorFlow/Keras
-------------------|------------------------------|---------------------------
Popularity         | #1 in research               | #1 in production (legacy)
Graph type         | Dynamic (eager by default)   | Dynamic (eager since TF2)
Debugging          | Easy — Python debugger works | Moderate — tf.function can be opaque
Custom training    | Full control via Python      | fit() or custom train_step
Deployment         | TorchScript, ONNX, TorchServe| TF Serving, TFLite, TFJS
Mobile/edge        | PyTorch Mobile               | TFLite (more mature)
Research papers    | 80%+ implemented in PyTorch  | Some exclusive TF papers
Hugging Face       | Native PyTorch (also supports TF)
JAX compatibility  | Limited                      | Limited (separate ecosystem)
LLM ecosystem      | vLLM, Transformers, DeepSpeed| Less common

2024 landscape:
  Research: PyTorch dominates (~80%)
  New production: PyTorch gaining
  Legacy production: TensorFlow holds
  New entrant: JAX (Google research, functional paradigm)

# 1. Debugging — just print or use pdb
def forward(self, x):
    h = self.fc1(x)
    print(f"h shape: {h.shape}, mean: {h.mean():.4f}")  # easy inspection
    return self.fc2(torch.relu(h))

# 2. Dynamic control flow
def forward(self, x, use_skip: bool = True):
    h = self.encoder(x)
    if use_skip:                # real Python if — works perfectly
        h = h + self.skip(x)
    return self.head(h)

# 3. Gradient inspection
loss.backward()
for name, param in model.named_parameters():
    if param.grad is not None:
        print(f"{name}: grad norm = {param.grad.norm():.4f}")

# 4. Custom training loops without ceremony
for epoch in range(n_epochs):
    for batch in dataloader:
        # ... anything you want here ...

import tensorflow as tf

# 1. Built-in high-level training
model.fit(
    X_train, y_train,
    epochs=10,
    batch_size=32,
    validation_data=(X_val, y_val),
    callbacks=[
        tf.keras.callbacks.EarlyStopping(patience=5),
        tf.keras.callbacks.ModelCheckpoint("best.h5", save_best_only=True),
        tf.keras.callbacks.TensorBoard(log_dir="./logs"),
    ]
)

# 2. TFLite for mobile deployment
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
with open("model.tflite", "wb") as f:
    f.write(tflite_model)

# 3. TF Serving for production APIs
# model.save("./saved_model/")
# docker run tensorflow/serving --model_base_path=/models/my_model

Choose PyTorch when:
  ✓ Research or academic setting
  ✓ Implementing a paper
  ✓ Using Hugging Face Transformers
  ✓ Training LLMs (vLLM, DeepSpeed, FSDP ecosystem)
  ✓ Flexibility and debugging are priorities
  ✓ Team has PyTorch experience

Choose TensorFlow/Keras when:
  ✓ Deploying to mobile (TFLite) or browser (TensorFlow.js)
  ✓ Existing TF codebase to maintain
  ✓ Need TF Serving for production ML serving
  ✓ Team has Keras experience (quickest to get started)

Neutral (both work equally well):
  Standard image/text classification
  Transfer learning from pre-trained models
  Tabular data neural networks