NLP with Transformers
Understand transformer architecture properly, then apply it: tokenization, attention, embeddings, and fine-tuning encoder models for classification and extraction.
Course Overview
Transformers underpin every current language model, and working with them productively requires understanding what the architecture actually does rather than treating it as an opaque API. This course builds that understanding concretely: how tokenization determines what the model can represent, how self-attention computes relationships between tokens, why positional encoding is necessary, and what distinguishes encoder, decoder, and encoder-decoder designs.
Much practical NLP work does not need a generative model at all. Classification, named entity recognition, and semantic similarity are handled faster, cheaper, and more reliably by a fine-tuned encoder model like BERT or one of its successors than by prompting a large generative model. A substantial portion of the course is hands-on work with the Hugging Face ecosystem building exactly these systems.
The course closes on the practical concerns that decide whether a model ships: evaluating beyond accuracy on imbalanced data, handling long documents that exceed the context limit, distillation and quantization for inference cost, and choosing sensibly between a small fine-tuned encoder and an API call to a large model.
Duration: 3 days|Delivery: onsite, virtual, or hybrid
Prerequisites
- Solid Python programming
- Basic machine learning concepts: training, validation, overfitting
- Some exposure to PyTorch or TensorFlow is helpful but not required
- No prior NLP experience needed
Who Should Attend
- Engineers building text classification, extraction, or search features
- Data scientists moving from classical NLP to transformer-based approaches
- ML engineers who need to reduce inference cost by replacing API calls with a small model
- Anyone who wants to understand what happens inside an LLM before working with one
Course Outline
- 1From bag-of-words to transformers: what earlier approaches could not represent
- 2Tokenization in practice: BPE, WordPiece, SentencePiece, and how vocabulary shapes behavior
- 3Self-attention mechanics worked through by hand on a small example
- 4Multi-head attention, positional encoding, and the full transformer block
- 5Encoder, decoder, and encoder-decoder architectures and what each is suited to
- 6The Hugging Face ecosystem: transformers, datasets, tokenizers, and the Hub
- 7Fine-tuning encoders for text classification with imbalanced real-world data
- 8Named entity recognition and token-level tasks
- 9Sentence embeddings and semantic similarity with sentence-transformers
- 10Long documents: truncation, chunking, sliding windows, and long-context architectures
- 11Evaluation beyond accuracy: precision, recall, macro-F1, and calibration
- 12Inference efficiency: distillation, quantization, ONNX export, and batching
Learning Outcomes
- Explain how attention works and why transformers replaced earlier sequence models
- Diagnose tokenization problems that silently degrade model performance
- Fine-tune encoder models for classification and entity extraction on your own data
- Build semantic similarity and search features using sentence embeddings
- Evaluate NLP models appropriately for imbalanced and multi-class problems
- Choose between a small fine-tuned model and a large model API on cost and quality grounds
- Cut inference cost through distillation and quantization without losing task accuracy
What You Will Build
- A fine-tuned text classifier evaluated on a realistic imbalanced dataset
- A named entity recognition model for a domain-specific entity type
- A semantic search prototype built on sentence embeddings
- A cost and latency comparison between a fine-tuned encoder and a hosted LLM API
Frequently Asked Questions
- Why learn encoder models when LLMs can do classification?
- Cost, latency, and reliability. A fine-tuned encoder with a few hundred million parameters typically classifies more accurately than a prompted general-purpose LLM on a narrow task, runs in milliseconds on modest hardware, costs a fraction as much per prediction, and produces stable outputs that do not drift when a vendor updates their model. For high-volume classification and extraction, it remains the right default.
- Is this course about building LLMs from scratch?
- No. It teaches the architecture well enough to work with transformers effectively and to fine-tune encoder models on your own data. Pretraining a large model from scratch is out of scope and is not something most organizations should do.
- How much labeled data do we need for fine-tuning?
- For text classification, a few hundred examples per class is often enough to beat a zero-shot LLM baseline, and a few thousand gets close to the ceiling. The course covers using an LLM to bootstrap initial labels and then correcting them, which is usually the fastest path to a usable dataset.
- Does the course cover multilingual text?
- Yes. Multilingual models, the tokenization problems that appear in non-Latin scripts, and cross-lingual transfer are covered, including the cases where a language-specific model beats a multilingual one.
- What background do participants usually have?
- Most are software or data engineers with solid Python and little or no NLP experience. The attention material is taught from first principles with a worked numeric example rather than assuming familiarity with the original paper.
Related Courses
Deep Learning with PyTorch
Build, train, and debug neural networks in PyTorch: autograd, training loops, regularization, transfer learning, and the practices that make results reproducible.
LLM Fine-Tuning
Decide when fine-tuning is warranted, then do it properly: dataset construction, LoRA and QLoRA, hyperparameters, evaluation, and serving the result.
Introduction to RAG
Learn Retrieval-Augmented Generation from first principles: chunking, embeddings, vector search, and grounding LLM answers in your own documents.
ML Model Deployment
Serve models reliably: inference APIs, batching, autoscaling, GPU utilization, model optimization, and hitting a latency target without overspending.
Ready to Get Started?
Contact us to schedule training for your team or inquire about upcoming sessions.