Research topics in ML, AI and Deep Learning
neocognitron
LeNet-5
Graph Neural Network, Chemical Precursors, Weisfeiler-Lehman (WL) test
DeepSets
CNN > Symmetry translation, GNN > Symmetry Permutation
CNNs are translation-equivariant; GNN layers are permutation-equivariant, with permutation-invariant graph readouts. https://arxiv.org/abs/2104.">
neocognitron
LeNet-5
Graph Neural Network, Chemical Precursors, Weisfeiler-Lehman (WL) test
DeepSets
CNN > Symmetry translation, GNN > Symmetry Permutation
CNNs are translation-equivariant; GNN layers are permutation-equivariant, with permutation-invariant graph readouts. https://arxiv.org/abs/2104.">
Subscribe
Geometric Deep Learning<br>Euclidean Geometry<br>Group Invariance Theorem<br>Universal Approximation<br>Player Perceptron, Rosenblatt perceptron<br>Curse of Dimensionality > neocognitron<br>LeNet-5<br>Software 1.0 and 2.0Graph Neural Network, Chemical Precursors, Weisfeiler-Lehman (WL) test<br>DeepSets<br>CNN > Symmetry translation, GNN > Symmetry Permutation<br>CNNs are translation-equivariant; GNN layers are permutation-equivariant, with permutation-invariant graph readouts. https://arxiv.org/abs/2104.13478<br>5G of GDL - https://arxiv.org/abs/2104.13478<br>Grids<br>Groups<br>Graphs<br>Geodesics<br>Gauges<br>Supervised ML sources of errors > approximation, estimation, optimization<br>minimization of empirical risk<br>Error Decomposition<br>error decompositionTotal excess error = approximation error + estimation error + optimization error<br>error vs model complexityDouble Descent<br>double descentNearest Neighbor Classifier<br>Approximation rates<br>Soboler Class dimensionality cursed<br>curse in optimization<br>benign landscapes
Lipschitz estimation error > too large<br>Sobolev - approx erro > too small<br>Graph Foundation Models<br>knowledge graphs TransE<br>Message Passing Neural Network<br>Link Prediction with GNN on Knowledge Graphs<br>Transferability on Knowledge Graphs<br>Transformers Research Gaps > Feasible with Less Compute<br>Quadratic Complexity > LoRA, KV Cache Optimization<br>Quadratic attention cost > sparse/local/linear attention or FlashAttention; LoRA is for parameter-efficient fine-tuning, while KV-cache optimization targets inference memory. https://arxiv.org/abs/2209.04881<br>Fixed Context window > Flash Attention, Sliding Window, RAG<br>Context limits > long-context training, positional scaling and sliding-window attention; RAG supplies external context, while FlashAttention mainly improves efficiency. https://arxiv.org/abs/2205.14135<br>Lack of Inductive bias > Annotations<br>Lack of inductive bias > architectural priors such as locality, equivariance, recurrence or structured positional encoding; annotations add supervision. https://arxiv.org/abs/2104.13478<br>Struggles with Maths<br>Stochastic Diffusion Models, Deterministic Flow Matching<br>Diffusion often uses stochastic dynamics, while flow matching usually learns a deterministic ODE vector field; the frameworks can overlap. https://arxiv.org/abs/2210.02747<br>Applied<br>Stochastic<br>Differential<br>Equations<br>Recipes for ML success<br>Large and diverse dataset<br>clear performance criterion<br>degenerate solution spaceSuitable hypothesis class and tractable optimization
Dataset void training areas:<br>simple arithmetic<br>sort listing<br>thermostat regulation<br>moral reasoning<br>Elmo RNN of Language Modeling<br>ELMo: contextual embeddings derived from a deep bidirectional LSTM language model. https://arxiv.org/abs/1802.05365<br>Limitations of RNN<br>RNN trained with back propagation through time<br>unroll computation graph to standard back prop for each<br>Slow and difficult to parallelize over long sequences.<br>Transformers<br>Self-attention uses queries, keys and values: Q, K and V.<br>Residual Connection, LayerNorm<br>Positional embeddings in addition to word embeddings<br>Casual LM forward probability:<br>p(xt | x), predicting token t from preceding tokens.<br>Transformer decoder<br>Data: trillions of tokens<br>Compute: massive parallel training<br>Outlines Guidance, n grammar:<br>Outlines and Guidance support constrained generation using JSON Schema, regular expressions and context-free grammars. https://dottxt-ai.github.io/outlines/1.0.0/features/core/output_types/<br>Repetition penalty, grammar matching,<br>instruction tuning SFT, ROUGE-Lsum for summarization<br>Encoded in params, tokenization meets instruction tuning<br>Knowledge is partly encoded in model parameters; tokenization and instruction tuning are separate components.<br>Reward model training, R (prompt, response option) > score
triplet of prompt, winning, losing (prompt, Ow, Oc) train R such that R (prompt, Ow) > R (prompt, Oc)<br>Reinforcement learning from Human Feedback<br>good to maximize reward with staying close to original model<br>KL-regularized, PPO and DPO lossPPO updates - Proximal Policy Optimization, DPO - Direct preference optimization<br>RL via PPO, RL with Verifiable Rewards (RLVR)<br>Efficiency Techniques, memory and compute<br>KV Cache Optimization > Inference<br>Memory: quantization, 4-bit quantization > 4x memory reduction<br>4-bit weights are theoretically 4x smaller than FP16 weights, but total runtime savings...