How AI & LLMs Actually Work: A Developer's Guide

Input:  "The capital of France is"
Output: "Paris"   ← the most probable next token

"Hello, world!"      → ["Hello", ",", " world", "!"]           — 4 tokens
"Unbelievable!"      → ["Un", "bel", "iev", "able", "!"]       — 5 tokens
"def calculate():"   → ["def", " calculate", "():", ]          — 3 tokens

"King"  → [0.23, -0.11, 0.87, ...]
"Queen" → [0.24, -0.10, 0.88, ...]   ← very similar vector
"Apple" → [-0.45, 0.67, -0.12, ...]  ← different direction entirely

Temperature = 0.0  → always picks the most probable token — deterministic, repetitive
Temperature = 0.7  → some randomness — good for most tasks  
Temperature = 1.0  → higher creativity — more varied, may drift
Temperature = 2.0  → very random — often incoherent

// Low temp for factual/code tasks
var response = await client.Chat.Completions.CreateAsync(new()
{
    Model       = "gpt-4o-mini",
    Temperature = 0.1f,    // deterministic — good for code generation
    Messages    = [ ... ],
});

// Higher temp for creative writing
var story = await client.Chat.Completions.CreateAsync(new()
{
    Model       = "gpt-4o",
    Temperature = 0.9f,
    Messages    = [ ... ],
});

OpenAI         → GPT-4o, GPT-4o-mini, o1, o3 — market leader, best tooling
Anthropic      → Claude 3.7 Sonnet, Claude Opus — strongest for reasoning/code
Google         → Gemini 1.5 Pro, Gemini Flash — 2M context window
Meta           → Llama 3.x — open weights, run locally or on your own infra
Mistral        → Mistral Large, Mixtral — European, efficient models
Microsoft      → Azure OpenAI — OpenAI models hosted on Azure, compliance friendly

dotnet add package Azure.AI.OpenAI
# or for standard OpenAI:
dotnet add package OpenAI

// appsettings.json
{
  "OpenAI": {
    "ApiKey": "sk-...",
    "Model":  "gpt-4o-mini"
  }
}

// Program.cs — register the client
builder.Services.AddSingleton(sp =>
{
    var config = sp.GetRequiredService<IConfiguration>();
    return new OpenAIClient(config["OpenAI:ApiKey"]);
});

// Simple completion
public class AiService
{
    private readonly OpenAIClient _client;
    private readonly string _model = "gpt-4o-mini";

    public AiService(OpenAIClient client) => _client = client;

    public async Task<string> AskAsync(string question, CancellationToken ct = default)
    {
        var options = new ChatCompletionOptions
        {
            Temperature = 0.3f,
            MaxOutputTokenCount = 1024,
        };

        var messages = new List<ChatMessage>
        {
            ChatMessage.CreateSystemMessage("You are a helpful assistant for developers."),
            ChatMessage.CreateUserMessage(question),
        };

        var response = await _client
            .GetChatClient(_model)
            .CompleteChatAsync(messages, options, ct);

        return response.Value.Content[0].Text;
    }
}

// Controller
[ApiController]
[Route("api/ai")]
public class AiController : ControllerBase
{
    private readonly AiService _ai;
    public AiController(AiService ai) => _ai = ai;

    [HttpPost("ask")]
    public async Task<IActionResult> Ask([FromBody] AskRequest req, CancellationToken ct)
    {
        var answer = await _ai.AskAsync(req.Question, ct);
        return Ok(new { answer });
    }
}

public async IAsyncEnumerable<string> StreamAsync(
    string question,
    [EnumeratorCancellation] CancellationToken ct = default)
{
    var messages = new List<ChatMessage>
    {
        ChatMessage.CreateSystemMessage("You are a helpful assistant."),
        ChatMessage.CreateUserMessage(question),
    };

    await foreach (var chunk in _client
        .GetChatClient(_model)
        .CompleteChatStreamingAsync(messages, cancellationToken: ct))
    {
        foreach (var part in chunk.ContentUpdate)
            yield return part.Text;
    }
}

// Stream from a Minimal API endpoint
app.MapGet("/api/ai/stream", async (string q, AiService ai, HttpResponse response, CancellationToken ct) =>
{
    response.ContentType = "text/event-stream";
    await foreach (var token in ai.StreamAsync(q, ct))
    {
        await response.WriteAsync($"data: {token}\n\n", ct);
        await response.Body.FlushAsync(ct);
    }
});

from openai import OpenAI

client = OpenAI(api_key="sk-...")

response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.3,
    messages=[
        {"role": "system", "content": "You are a helpful developer assistant."},
        {"role": "user",   "content": "Explain async/await in one paragraph."},
    ]
)

print(response.choices[0].message.content)

# Streaming
for chunk in client.chat.completions.create(
    model="gpt-4o-mini",
    stream=True,
    messages=[{"role": "user", "content": "Write a haiku about Python."}]
):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

var response = await chatClient.CompleteChatAsync(messages, options, ct);

var usage = response.Value.Usage;
_logger.LogInformation(
    "OpenAI call — model: {Model}, input: {Input} tokens, output: {Output} tokens, total: {Total}",
    _model,
    usage.InputTokenCount,
    usage.OutputTokenCount,
    usage.TotalTokenCount);

// Store in metrics
_metrics.RecordTokenUsage(usage.InputTokenCount, usage.OutputTokenCount);

Task                                → Best Model
─────────────────────────────────────────────────
Simple Q&A, summarisation           → gpt-4o-mini (cheap, fast)
Complex reasoning, multi-step       → gpt-4o or o1
Code generation                     → gpt-4o or claude-3.7-sonnet
Long document analysis (100k+ words)→ Gemini 1.5 Pro (2M context)
Running locally / no API cost       → Llama 3 or Mistral via Ollama
Production on Azure (compliance)    → Azure OpenAI (gpt-4o-mini)