Course (Part 717)

Course: Introduction to GPTs

Lesson

Topic: Introduction to GPTs

Conceptual Explanation

Generative Pre-trained Transformers, commonly called GPTs, are advanced AI models designed to generate coherent and contextually relevant text by learning statistical patterns from large text corpora. At their core, GPTs are built on the Transformer architecture introduced in 2017, which uses self-attention to model relationships among tokens without relying on strictly sequential processing. The "pre-trained" stage exposes a GPT to diverse text data so that it learns grammar, typical facts, and many language patterns through next-token prediction. After pre-training, fine-tuning adapts the model to specific tasks or domains by supervised learning on smaller labeled datasets. A distinguishing capability of GPTs is few-shot and zero-shot performance: they can often perform new tasks with little or no task-specific training when provided an appropriate prompt. As model size increases—measured in parameters—GPTs generally exhibit improved fluency and broader knowledge, though computational costs also rise. Practical applications include content generation, conversational agents, tutoring aids, code assistance, translation, and research summarization. Despite their utility, GPTs inherit limitations from training data, such as biases, stereotypes, and occasional factual errors, since they generate outputs from learned patterns rather than genuine understanding. Ethical challenges include misinformation risks, privacy concerns, and questions about authorship and accountability when AI-produced content is used publicly. Responsible deployment requires careful dataset curation, evaluation metrics, human oversight, and mitigation strategies for bias and misuse. As the field progresses, multimodal and more efficient GPT variants promise broader capabilities, but they also demand continued attention to governance and societal impact.

Learning Outcomes

• Define what a GPT is and identify its core components.
• Explain the Transformer architecture and the role of self-attention.
• Differentiate pre-training from fine-tuning and describe few-shot/zero-shot behavior.
• Recognize key applications and limitations of GPTs.
• Articulate major ethical considerations when deploying GPTs.

Bilingual (EN → বাংলা)

• Generative Pre-trained Transformer (GPT) — জেনারেটিভ প্রি-ট্রেইনড ট্রান্সফর্মার
• Self-attention — স্ব-মনোযোগ (কোনও টোকেন অন্য টোকেনের সাথে সম্পর্ক মাপা)
• Pre-training — পূর্ব-প্রশিক্ষণ (বৃহৎ টেক্সট কর্পাসে অ-পরিচিত লেবেলযুক্ত শিক্ষা)
• Fine-tuning — সূক্ষ্ম-অভিযোজন (নির্দিষ্ট কাজে পরিচালিত তত্ত্বাবধান শিখন)
• Bias — পক্ষপাত (প্রশিক্ষণ ডেটার কারণে পক্ষপাতপূর্ণ ফল)

সংক্ষিপ্ত: GPT একধরনের ভাষাগত AI যা বড় ডেটা থেকে শেখে এবং প্রম্পট অনুযায়ী মানবসদৃশ টেক্সট উৎপন্ন করে।

Important Notes

• GPTs predict tokens probabilistically; outputs are not proofs of fact.
• Training data quality directly affects fairness and factuality of outputs.
• Large models require substantial compute and energy—consider efficiency trade-offs.
• Human-in-the-loop verification is essential for sensitive or high-impact uses.
• Record and disclose model provenance, dataset sources, and evaluation metrics where possible.

Exercises

True / False

Statement	Answer
1. GPTs use recurrent neural networks to model long-range dependencies.	Answer False. GPTs are built on Transformer architectures that use self-attention rather than recurrent connections to capture long-range dependencies.
2. Pre-training a GPT typically uses supervised labeled datasets.	Answer False. Pre-training is usually unsupervised (or self-supervised), e.g., next-token prediction on unlabeled text corpora.
3. Few-shot learning means a model can perform a task after seeing only a handful of examples.	Answer True. Few-shot learning refers to achieving reasonable performance with only a few examples provided in the prompt or during fine-tuning.
4. Increasing the number of parameters in a GPT always reduces bias.	Answer False. Larger models may amplify patterns from training data; parameter count alone does not eliminate bias and can sometimes magnify it.

Fill in the Blanks

1. Transformer models rely on the __________ mechanism to compare tokens across a sequence.
2. Predicting the next word during pre-training is an example of __________ learning.
3. When a model generates biased output due to training data, this is called __________.
4. Fine-tuning typically uses a smaller, __________ dataset tailored to a task.
5. GPT-3 contains approximately __________ billion parameters.

Answer

1. self-attention
2. self-supervised (or unsupervised)
3. bias
4. supervised (labeled)
5. 175

Multiple Choice Questions (MCQ)

1. Which paper introduced the Transformer architecture?
   • A. "Deep Learning" by Goodfellow et al.
   • B. "Attention is All You Need" by Vaswani et al.
   • C. "ImageNet Classification" by Krizhevsky et al.
   • D. "Sequence to Sequence Learning" by Sutskever et al.
   Answer

   B. "Attention is All You Need" by Vaswani et al.
2. Few-shot capability in GPT means:
   • A. The model must be retrained from scratch for each new task.
   • B. The model ignores examples in the prompt.
   • C. The model can adapt to tasks with only a few prompt examples.
   • D. The model uses reinforcement learning exclusively.
   Answer

   C. The model can adapt to tasks with only a few prompt examples.
3. Which is NOT a common mitigation for bias in GPT outputs?
   • A. Curating training data
   • B. Post-generation filtering
   • C. Ignoring evaluation metrics
   • D. Human-in-the-loop review
   Answer

   C. Ignoring evaluation metrics (this would worsen bias handling).
4. The output layer of a GPT most commonly uses:
   • A. Sigmoid activation over one token
   • B. Softmax over the vocabulary
   • C. ReLU to choose the next token
   • D. Linear regression to predict token frequency
   Answer

   B. Softmax over the vocabulary.
5. Which application is a GPT especially well-suited for?
   • A. Deterministic arithmetic operations requiring exact proofs
   • B. Generating coherent natural-language text
   • C. Replacing relational databases
   • D. Real-time control of physical actuators without simulation
   Answer

   B. Generating coherent natural-language text.

Matching

Match the term (A–E) with the correct description (1–5).

Term	Description
A. Pre-training	1. Adjusting a pre-trained model to a specific small dataset.
B. Self-attention	2. A metric for how well a model predicts next tokens (lower is better).
C. Fine-tuning	3. Mechanism that lets tokens attend to every other token.
D. Perplexity	4. Initial unsupervised training on large corpora.
E. Bias	5. Systematic skew in outputs caused by training data.

Answer

1. A → 4 (Pre-training = initial unsupervised training on large corpora)
2. B → 3 (Self-attention = tokens attending to others)
3. C → 1 (Fine-tuning = adjusting to a specific small dataset)
4. D → 2 (Perplexity = metric for next-token prediction)
5. E → 5 (Bias = systematic skew from training data)

Descriptive

1. Explain in 3–5 sentences how self-attention enables better long-range context modeling than RNNs.
Answer
Self-attention computes pairwise interactions between all tokens in a sequence, allowing the model to directly weigh the relevance of distant tokens when forming representations. Unlike RNNs, which propagate information sequentially and can suffer from gradient decay across long steps, self-attention accesses global context in a single layer through learned attention weights. This parallel, direct mechanism makes it easier for transformers to model long-range dependencies and capture relationships regardless of token distance.
2. Discuss two ethical risks associated with deploying GPTs in public information services and propose one mitigation per risk.
Answer
Risk 1: Misinformation—GPTs may generate plausible but false statements. Mitigation: enforce human review for high-stakes outputs, use citation-aware prompting, and integrate fact-checking modules. Risk 2: Bias and stereotyping—models can reproduce societal biases from training data. Mitigation: curate and diversify training data, test outputs for disparate impacts, and apply debiasing/post-filtering strategies.

Creative

Prompt: Write a short (4–5 sentence) chat reply from a GPT-based tutor explaining the difference between pre-training and fine-tuning to a beginner.

Answer

Pre-training is like teaching the model general language skills by showing it lots of books and articles so it learns grammar, facts, and patterns. Fine-tuning is a second, focused lesson where the model practices a specific job—like answering math questions or grading essays—using examples for that task. Think of pre-training as building broad language knowledge and fine-tuning as giving task-specific training wheels. Together they let the model generalize well and perform particular tasks accurately.

Activities, Vocabulary, Homework, Practice Task

Activities

1. Pair discussion: list five real-world tasks in your field that could use a GPT and justify the choice.
2. Hands-on: run a short prompt experiment with a pre-trained model (or demo) and record variations with different prompt phrasings.

Vocabulary

• Token — smallest text unit processed by the model (word piece or subword).
• Self-attention — mechanism to compute context-aware weights between tokens.
• Perplexity — a measure of a language model's uncertainty; lower is better.
• Few-shot learning — performing tasks with very few examples.

Homework

1. Read "Attention is All You Need" (Vaswani et al., 2017) and write a one-page summary highlighting self-attention.
2. Find one article describing GPT misuse in the wild; summarize the incident and propose at least two safeguards that could have reduced harm.

Practice Task

Build a simple prompt template that converts a user request into a structured prompt for a GPT-based summarizer: include an instruction, a context window, and a desired length/style. Test it on two different types of text (news article and scientific abstract) and record the outputs.

End of lecture sheet — Instructor notes: verify model demos respect privacy and include source citations where available.