NSSpain 2023: Running LLMs and diffusion models on Mac & iPhone with Swift and Core ML

Link to recording

Detailed image description of the sketchnote

LLM in CoreML

• scales more, efficently as custom solutions

Tools

• Hugging Face Hub to choose models or to train vour own model
• StarCoder but Swift was accidentally removed. Oops!
• Falcon
• LLaMA 2 / Code Llama
• High-level tools available
• steep learning curve

Convert to Core ML

• open source bug fix

Tokenization

• Byte Pair Encoding (BPE)
• swift-transformers package with most common logic

Use the Model

• Generating Text one token at a time!

Optimization

• Stable Diffusion with Core ML on Apple Silicon
• Quantization & Caching