Machine Learning Fundamentals

The "switch" inside a neural network that decides whether a neuron should fire, allowing the AI to learn complex non-linear patterns.

Mathematical functions applied to the output of a neuron in a neural network to introduce non-linearity. Common examples include ReLU, Sigmoid, and Tanh. Without them, neural networks would behave like simple linear regression models.

Why it matters: Essential for deep learning models to handle real-world data like images and language that aren't linearly separable.

Full definition: Activation Functions →

A technique where the AI asks humans to label only the most confusing examples, saving time and money on data labeling.

A machine learning approach where the model actively selects the most informative data points for labeling, reducing the need for large labeled datasets. It iteratively queries a human expert to label uncertain examples.

Why it matters: Drastically reduces data annotation costs (often by 50-80%) while maintaining model performance.

Full definition: Active Learning →

Finding the "weird" stuff in a dataset, like a credit card charge in a foreign country or a broken machine part.

The identification of data points, patterns, or behaviors that deviate significantly from expected norms. Algorithms flag unusual observations that could indicate fraud, failures, or errors.

Why it matters: The core technology behind fraud prevention and predictive maintenance, saving billions annually.

Full definition: Anomaly Detection →

A hypothetical "super-AI" that can learn and do any intellectual task a human can do, not just one specific thing.

A hypothetical AI system with the ability to understand, learn, and apply knowledge across any intellectual task at a level equal to or surpassing human intelligence. Unlike narrow AI, AGI would generalize across domains.

Why it matters: The "holy grail" of AI research that would fundamentally transform society and economics.

Full definition: Artificial General Intelligence (AGI) →

Finding "what goes with what" patterns in data, like people who buy beer often buy diapers too.

A rule-based machine learning technique for discovering co-occurrence patterns between items in large datasets, expressed as 'if A, then B' rules. Three metrics judge each rule: support (how often the items appear together), confidence (how often B follows A), and lift (how much more likely B is given A than by chance). The Apriori and FP-Growth algorithms find these rules efficiently.

Why it matters: Association rules power the 'customers who bought this also bought that' recommendations that drive retail and e-commerce revenue. Beyond retail, they surface hidden patterns in transactions, web behavior, and medical records that guide cross-sell, layout, and inventory decisions.

Full definition: Association Rules →

Letting an AI focus on the important words in a sentence while ignoring the rest, just like humans pay attention to key details.

A component of neural networks that allows the model to dynamically focus on the most relevant parts of the input when generating each output token. The core innovation powering Transformer models (like GPT).

Why it matters: The breakthrough that enabled modern LLMs to understand context and long-range dependencies in text.

Full definition: Attention Mechanism →

Tools that automatically pick the best AI model and settings for your data, so you don't have to do it manually.

The automation of the end-to-end machine learning workflow, including data preprocessing, feature engineering and selection, model selection, and hyperparameter tuning. Instead of a data scientist manually testing combinations, AutoML systems search the space of options and return the best-performing pipeline. Offerings range from open-source libraries to managed cloud platforms that also handle deployment.

Why it matters: AutoML speeds time-to-value for experienced data science teams by handling the repetitive search work, and it lets analysts without deep ML expertise build usable models. The caution is that automation can hide assumptions, so results still need human review before they drive decisions.

Full definition: Automated Machine Learning (AutoML) →

A diagram that maps out cause-and-effect relationships and probabilities (e.g., "If it rains, grass is 90% likely wet").

Probabilistic graphical models that represent variables as nodes and their conditional dependencies as directed edges in an acyclic graph. Each node carries a probability table describing how it depends on its parents. Using Bayes' theorem, the network updates its beliefs as new evidence arrives, letting you reason from causes to effects or from observed effects back to likely causes.

Why it matters: Bayesian networks shine at reasoning under uncertainty with limited data, where pure deep learning struggles. They are widely used in medical diagnosis, fault detection, and risk modeling because their structure is interpretable and shows exactly which factors drive a prediction.

Full definition: Bayesian Networks →

The balancing act between a model that's too simple (misses patterns) and one that's too complex (memorizes noise).

A fundamental ML concept: bias is error from oversimplified assumptions (underfitting); variance is error from sensitivity to training data fluctuations (overfitting). Optimal models balance both.

Why it matters: Understanding this tradeoff is essential for diagnosing why a model underperforms and choosing the right fix.

Full definition: Bias-Variance Tradeoff →

Teaching an AI to sort things into categories, like "spam" or "not spam," "high-risk deal" or "likely to close."

A supervised learning task where the model predicts which category or class an input belongs to. Binary classification (two classes) and multi-class classification (many classes) are common variants.

Why it matters: One of the most widely deployed ML tasks, powers spam filters, medical diagnosis, deal scoring, and fraud detection.

Full definition: Classification →

An AI model like GPT-4 that you can use but not see inside, the recipe and ingredients are secret.

An AI model whose weights, training data, and architecture are proprietary and not publicly accessible. Users interact only via an API. Examples include GPT-4, Claude, and Gemini.

Why it matters: Often offers highest performance but poses risks regarding data privacy and vendor lock-in.

Full definition: Closed Model →

Automatically grouping similar data points together without being told the categories, the AI discovers the structure.

An unsupervised learning technique that partitions data into groups (clusters) where items within a group are more similar to each other than to items in other groups. Includes K-means, DBSCAN, and hierarchical methods.

Why it matters: Powers customer segmentation, anomaly detection, and pattern discovery when you don't know what groups exist in your data.

Full definition: Clustering →

Blueprints for AI that try to mimic the structure of the human mind, including memory, goals, and perception.

Computational frameworks that model the structure of human cognition to guide AI design. Rather than solving one narrow task, they integrate distinct modules for perception, working and long-term memory, goal management, reasoning, and action into a unified system. Long-standing examples include SOAR and ACT-R, and the ideas increasingly inform how autonomous agents combine memory, planning, and tool use.

Why it matters: Cognitive architectures represent the effort to move AI beyond pattern matching toward systems that reason, remember, and pursue goals over time. Their concepts are resurfacing in agent design, where durable memory and structured planning separate a capable agent from a stateless chatbot.

Full definition: Cognitive Architectures →

Recommending stuff by saying, "You're like this other user, and they liked X, so you'll probably like X too."

A recommendation technique that predicts user preferences by identifying patterns among many users. Assumes that if users agreed in the past, they will agree in the future.

Why it matters: The core algorithm behind "Users who bought this also bought..." on Amazon and Netflix.

Full definition: Collaborative Filtering →

An AI that makes predictions by asking a series of yes/no questions, like a flowchart.

A supervised learning algorithm that splits data into branches based on feature values, creating a tree-like structure of decisions. Highly interpretable and the basis for ensemble methods like Random Forests.

Why it matters: One of the most intuitive ML algorithms, easy to explain to non-technical stakeholders.

Full definition: Decision Trees →

AI powered by neural networks with many layers, capable of learning incredibly complex patterns from massive amounts of data.

A subset of machine learning that uses neural networks with multiple layers (hence "deep") to automatically learn hierarchical representations from data. Powers modern AI breakthroughs in vision, language, and speech.

Why it matters: The engine behind virtually every major AI breakthrough since 2012, from AlexNet to GPT-4.

Full definition: Deep Learning →

Simplifying complex data by keeping only the most important features, like summarizing a 50-page report into key bullet points.

Techniques for reducing the number of input variables in a dataset while retaining the most important information. Methods include PCA, t-SNE, and UMAP. Used for visualization and preprocessing.

Why it matters: Makes complex datasets manageable and helps models train faster by removing noise and redundancy.

Full definition: Dimensionality Reduction →

Surprising abilities that appear in large AI models that were never explicitly trained for, they just emerge at scale.

Capabilities that arise unexpectedly in large models trained at sufficient scale, which were not present in smaller versions. Examples include in-context learning, chain-of-thought reasoning, and tool use.

Why it matters: One of the most fascinating phenomena in modern AI, and a key reason why scaling continues to produce breakthroughs.

Full definition: Emergent Capabilities →

Combining predictions from multiple AI models to get a better answer, like asking three doctors instead of one.

Techniques that combine multiple models to produce a prediction that is more accurate and robust than any single model. Includes bagging (Random Forests), boosting (XGBoost), and stacking.

Why it matters: Consistently top leaderboards in ML competitions; most production ML systems use ensembles for reliability.

Full definition: Ensemble Methods →

Training an AI model across many devices without ever collecting the raw data in one place, privacy by design.

A machine learning technique where a model is trained across multiple decentralized devices or servers holding local data samples, without exchanging raw data. Only model updates (gradients) are shared.

Why it matters: Enables AI training on sensitive data (medical records, financial data) without compromising privacy.

Full definition: Federated Learning →

AI designed to understand data that comes in networks and connections, like social networks, molecules, or supply chains.

Neural networks built to operate on graph-structured data, where entities are nodes and their relationships are edges. Instead of assuming inputs are independent, a GNN learns by repeatedly aggregating information from each node's neighbors, so a node's representation reflects its position and connections in the wider network. This message-passing approach captures relational structure that grid- or sequence-based networks miss.

Why it matters: Much real-world data is relational: social networks, molecules, supply chains, and transaction graphs. GNNs are a leading approach for fraud detection, drug discovery, recommendation systems, and logistics, where the connections between entities carry as much signal as the entities themselves.

Full definition: Graph Neural Networks (GNNs) →

Grouping similar things together automatically, like sorting customers into segments based on their behavior.

An unsupervised learning algorithm that partitions data into K distinct clusters based on similarity. Iteratively assigns points to the nearest cluster center and updates centers until convergence.

Why it matters: One of the most widely used algorithms for customer segmentation, anomaly detection, and data exploration.

Full definition: K-Means Clustering →

The hidden "map" inside an AI where similar concepts are grouped close together, like an internal library organized by meaning.

The compressed, abstract representation space learned by a model where input data is encoded into meaningful dimensions. Similar items cluster together. Used in autoencoders, VAEs, and embedding models.

Why it matters: Understanding latent spaces is key to debugging generative models and improving search/recommendation quality.

Full definition: Latent Space →

Teaching computers to learn patterns from data and make predictions, without being explicitly programmed for every scenario.

A subset of AI where algorithms learn patterns from data to make predictions or decisions without being explicitly programmed. Includes supervised, unsupervised, and reinforcement learning paradigms.

Why it matters: The foundation of modern AI, every intelligent system, from spam filters to self-driving cars, uses ML.

Full definition: Machine Learning →

Teaching an AI how to learn faster, so it can pick up new tasks with minimal examples or training.

Machine learning techniques that improve a model's ability to learn new tasks quickly by leveraging experience from previous tasks. Often called "learning to learn."

Why it matters: Enables AI systems to adapt to new domains rapidly without starting from scratch every time.

Full definition: Meta-Learning →

An AI architecture where different "expert" sub-networks specialize in different types of inputs, and a router picks the right expert.

A neural network architecture that routes different inputs to different specialized sub-networks (experts). Only a subset of experts activate for each input, improving efficiency. Used in Mixtral and GPT-4.

Why it matters: Enables building larger, more capable models without proportionally increasing compute costs.

Full definition: Mixture of Experts (MoE) →

An AI system loosely inspired by the human brain, layers of connected "neurons" that learn patterns from data.

A computational model inspired by biological neural networks, consisting of layers of interconnected nodes (neurons). Input passes through layers, getting transformed by learned weights at each step.

Why it matters: The building block of all modern deep learning, from image recognition to language generation.

Full definition: Neural Network →

Combining neural networks (pattern recognition) with symbolic logic (rules and reasoning), getting the best of both worlds.

AI systems that combine neural networks (learning from data) with symbolic reasoning (logic, rules, knowledge graphs). Aims to achieve both the learning capability of neural nets and the reasoning of symbolic AI.

Why it matters: A promising path toward more reliable, explainable AI that can reason about cause and effect.

Full definition: Neuro-Symbolic AI →

An AI that can recognize a new concept from just a single example, like seeing one photo of a new face and remembering it.

A machine learning approach where a model can generalize from a single training example per class. Often uses metric learning or siamese networks to compare new inputs against stored examples.

Why it matters: Critical for applications where data is inherently scarce, face recognition, rare disease detection, security.

Full definition: One-Shot Learning →

AI models whose code and weights are publicly available, anyone can download, modify, and use them freely.

AI models released with publicly accessible weights, code, and often training details. Examples include Llama, Mistral, and Stable Diffusion. Enable customization, transparency, and on-premises deployment.

Why it matters: Democratizes AI access and enables companies to own their AI infrastructure without vendor lock-in.

Full definition: Open Source Models →

The simplest possible neural network, a single neuron that makes binary yes/no decisions based on weighted inputs.

The fundamental unit of neural networks: a linear classifier that computes a weighted sum of inputs and applies an activation function. First introduced by Rosenblatt in 1958. Multi-layer perceptrons form the basis of deep learning.

Why it matters: Understanding the perceptron is understanding the atom of deep learning, everything else builds from here.

Full definition: Perceptron →

An AI that builds hundreds of decision trees and lets them vote on the answer, wisdom of the (tree) crowd.

An ensemble learning method that constructs many decision trees during training and outputs the mode (classification) or mean (regression) of the individual trees. Resistant to overfitting.

Why it matters: Consistently one of the best "off-the-shelf" algorithms for tabular data, often beats deep learning on structured datasets.

Full definition: Random Forest →

A neural network designed for sequences, it has a "memory" that processes data one step at a time, remembering what came before.

A class of neural networks that maintain hidden state across sequence steps, making them suitable for sequential data like text and time series. Largely superseded by Transformers for NLP.

Why it matters: Historically important for language and time-series tasks, understanding RNNs explains why Transformers were a breakthrough.

Full definition: Recurrent Neural Network (RNN) →

Teaching an AI to predict a number, like a home's price, a deal's close probability, or next quarter's revenue.

A supervised learning task where the model predicts a continuous numerical output. Linear regression, polynomial regression, and neural network regression are common approaches.

Why it matters: Powers revenue forecasting, pricing optimization, demand prediction, and virtually every quantitative business model.

Full definition: Regression →

An AI that learns by trial and error, getting rewards for good actions and penalties for bad ones, like training a dog.

A learning paradigm where an agent learns optimal behavior by interacting with an environment and receiving feedback (rewards or penalties). The agent learns a policy that maximizes cumulative reward.

Why it matters: Powers game-playing AI (AlphaGo), robotics, and recommendation systems that learn from user behavior.

Full definition: Reinforcement Learning (RL) →

Training an AI with a small amount of labeled data and a large amount of unlabeled data, getting more from less.

A learning paradigm that combines a small amount of labeled data with a large amount of unlabeled data during training. Techniques include self-training, co-training, and label propagation.

Why it matters: A practical middle ground when labeling data is expensive, gets most of the benefit of supervised learning at a fraction of the labeling cost.

Full definition: Semi-Supervised Learning →

Training an AI with labeled examples, showing it the right answers so it can learn to predict them on its own.

A machine learning paradigm where the model learns from labeled training data (input-output pairs). The model learns a mapping function from inputs to outputs. Includes classification and regression.

Why it matters: The most widely used ML paradigm, powers most business applications from churn prediction to deal scoring.

Full definition: Supervised Learning →

The AI architecture behind GPT, Claude, and every major language model, processes all words in parallel using attention.

A neural network architecture introduced in the 2017 paper "Attention Is All You Need" that processes sequences using self-attention mechanisms. Processes all tokens in parallel rather than sequentially, enabling efficient training on massive datasets.

Why it matters: The single most important architecture in modern AI, powers virtually every major language model and many vision models.

Full definition: Transformer →

A test where a human chats with an AI and tries to tell if it's a machine, if they can't tell, the AI "passes."

A test of machine intelligence proposed by Alan Turing in 1950. A machine passes if a human evaluator cannot reliably distinguish it from a human in natural language conversation.

Why it matters: While no longer the primary benchmark for AI capability, it remains a cultural touchstone for measuring AI progress.

Full definition: Turing Test →

Training an AI on unlabeled data to find hidden patterns and groupings, no right answers provided, just data.

A machine learning paradigm where the model learns patterns from unlabeled data without explicit target variables. Includes clustering, dimensionality reduction, and anomaly detection.

Why it matters: Essential when labeled data is unavailable, powers customer segmentation, anomaly detection, and data exploration.

Full definition: Unsupervised Learning →

The numerical values inside a neural network that get adjusted during training, collectively, they encode everything the model has learned.

The learnable parameters in a neural network that determine how inputs are transformed through each layer. Training is the process of finding optimal weights. GPT-4 has hundreds of billions of weights.

Why it matters: When people say a model is "175 billion parameters," they mean weights. The weights ARE the model's knowledge.

Full definition: Weight →

A powerful, fast machine learning algorithm that wins most competitions on tabular data, the workhorse of structured data ML.

An optimized gradient boosting library that builds an ensemble of decision trees sequentially, with each tree correcting the errors of the previous ones. Known for speed, performance, and handling of missing data.

Why it matters: The most successful algorithm for tabular/structured data in production, often outperforms deep learning on spreadsheet-style data.

Full definition: XGBoost →