Prof. Thomas G. Dietterich discusses the current state of large language models like ChatGPT. He explains their capabilities and limitations, emphasizing their statistical nature and tendency to hallucinate. Dietterich explores the challenges in uncertainty quantification for these models and proposes integrating them with formal reasoning systems. He advocates for more robust knowledge representation methods, such as knowledge graphs, and discusses the importance of safety in AI development. Dietterich also touches on the changing landscape of academic AI research and the potential of open-source models to accelerate innovation in the field.
MLST is sponsored by Tufa Labs:
Are you interested in working on ARC and cutting-edge AI research with the MindsAI team (current ARC winners)?
Focus: ARC, LLMs, test-time-compute, active inference, system2 reasoning, and more.
Future plans: Expanding to complex environments like Warcraft 2 and Starcraft 2.
Interested? Apply for an ML research position: benjamin@tufa.ai
Thomas G. Dietterich
Distinguished Professor (Emeritus), Computer Science, Oregon State University
https://scholar.google.com/citations?user=09kJn28AAAAJ&hl=en
https://medium.com/@tdietterich
https://x.com/tdietterich
TOC:
00:00:00 LLMs: Capabilities, Limitations, and Statistical Nature
00:09:25 Uncertainty Quantification in Large Language Models
00:15:49 Integrating Reasoning and Formal Systems with LLMs
00:19:39 Knowledge Structures and Future AI Architectures
00:20:11 Knowledge Extraction and Representation in AI
00:23:51 Challenges in LLM-based Knowledge Systems
00:33:12 Scientific Publishing and Knowledge Graphs
00:37:14 AI Safety and Truth in Knowledge Representation
00:39:56 AI Regulation and Safety Engineering
00:41:57 Challenges in AI Perception and Novelty Detection
00:50:04 Verification and Safety in Complex AI Systems
00:59:23 Open-Source Models and Academic AI Research
01:03:16 Limitations of Transformers and Future AI Architectures
REFS:
00:00:16 Meta releases the biggest and best open-source AI model yet, https://www.theverge.com/2024/7/23/24204055/meta-ai-llama-3-1-open-source-assistant-openai-chatgpt
00:00:27 Tamper-Resistant Safeguards for Open-Weight LLMs, https://arxiv.org/pdf/2408.00761
00:00:41 Identifying latent disease factors, https://arxiv.org/pdf/2410.07890
00:00:53 EY position paper on AI, https://www.ey.com/en_ch/news/2023/12/new-ey-position-paper-on-artificial-intelligence
00:00:53 RAG for Knowledge-Intensive NLP Tasks, https://arxiv.org/pdf/2005.11401
00:00:55 AI’s international research networks mapped, https://www.nature.com/articles/d41586-024-02986-2
00:01:15 A Survey of Large Language Models, https://arxiv.org/pdf/2303.18223
00:03:30 Embers of Autoregression, https://arxiv.org/pdf/2309.13638
00:07:33 GPT-4 Technical Report, https://cdn.openai.com/papers/gpt-4.pdf
00:10:52 LM-Polygraph, https://arxiv.org/pdf/2311.07383
00:14:16 Shifting Attention to Relevance, https://arxiv.org/pdf/2307.01379
00:16:00 Retrieval-Augmented Generation, https://arxiv.org/pdf/2005.11401
00:20:34 Baldur: Whole-Proof Generation, https://arxiv.org/pdf/2303.04910
00:22:11 Never-Ending Learning, https://www.cs.cmu.edu/~tom/pubs/NELL_aaai15.pdf
00:25:36 Machine Unlearning, https://arxiv.org/pdf/2404.01206
00:36:09 Papers with code or without code, https://www.sciencedirect.com/science/article/abs/pii/S0306457323002145
00:39:12 CYC: a large-scale investment in knowledge infrastructure, https://dl.acm.org/doi/10.1145/219717.219745
00:40:23 Community Notes, https://communitynotes.x.com/guide/en/about/introduction
00:41:39 REGULATION (EU), https://eur-lex.europa.eu/eli/reg/2024/1689/oj
00:43:44 Open Category Problem, https://futureoflife.org/ai-researcher-profile/ai-researcher-thomas-dietterich/
00:50:38 BO as Assisstant, https://dl.acm.org/doi/pdf/10.1145/3526113.3545664
00:50:59 Testing of Autonomous Vehicles, https://research-assets.waabi.ai/GUARD/paper.pdf
00:54:02 Engineering a Safer World, https://mitpress.mit.edu/9780262533690/engineering-a-safer-world/
00:54:50 The nature of the Internet, https://www.pnas.org/doi/epdf/10.1073/pnas.0501426102
00:58:43 Anomaly and Novelty Detection, https://dl.acm.org/doi/10.1145/3447548.3469453
00:59:06 Causality 2nd Edition, https://www.amazon.ca/Causality-Judea-Pearl/dp/052189560X
01:03:38 LLaMA: Open and Efficient Foundation Language Models, https://arxiv.org/pdf/2302.13971
