The Ruby AI Podcast

• Evaluating LLMs with Leva

  In this episode of the Ruby AI Podcast, host Valentino Stoll talks with special guest Kieran, a prominent figure in the Ruby AI space. Kieran recently gave a talk at the San Francisco Ruby Meetup about his new gem, Leva, which focuses on LLM evaluations in Ruby. Kieran discusses his background, his passion for AI and Ruby, as well as his journey in building AI products, including his tool Cora, which helps manage email inboxes by categorizing and summarizing emails using AI. Together, Valentino and Kieran explore the process, challenges, and best practices of creating AI-driven gems and tools in Ruby, the importance of evaluations, and the fun and creative aspects of integrating AI into Ruby on Rails projects.Mentioned in the show:Kieran Klaassen – Ruby developer, creator of Cora and Leva.Leva gem – Kieran's LLM evaluation framework for Rails.Jumpstart Pro – “is the best Ruby on Rails SaaS template out there”.Stepper / Stepper Motor (workflow engine) – a “journey” with steps for background jobs.Jaccard Index – A metric for set similarity (|A∩B|/|A∪B|).LangSmith – a platform for building production-grade LLM applications.Morph LLM – The Fastest Way to Apply AI Edits (4500+ tokens/sec).Friday AI Agent – An AI-powered coding agent that handles PRs from start to finish.DSPy.rb – Framework for building AI agents and optimizing prompts.Highlights:00:00 Introduction and Guest Welcome00:53 Kieran's Background and AI Journey01:20 Building AI Tools and the Leva Gem03:47 Challenges and Best Practices in AI Development07:16 Evaluations and Real-World Applications07:36 Community Recognition and Adoption12:37 Prompt Engineering and Model Testing22:06 Leveraging AI for Workflow Optimization28:35 Visualizing Workflows and Tools31:44 Exploring Hybrid Orchestration Layers33:15 Debating Deterministic Workflows vs. Agent Flows34:28 The Fun of Experimenting with AI and Ruby34:55 Building Gems and Learning Through Creation40:03 The Value of Rails in AI Development46:28 Evaluating AI Outputs and Metrics50:40 Annotation and Continuous Improvement53:50 Future of AI and Rails Integration54:54 Closing Thoughts and Recommendations

  --------  

  1:00:00

  --------

  1:00:00
• Roasting Ruby AI Workflows with Obie Fernandez

  Ruby legend Obie Fernandez joins hosts Valentino Stoll and Joe Leo to unveil Roast—the new open-source Ruby framework for declaring reliable AI workflows—and celebrate the 1.0 release of its engine library, Raix. The trio dig into agent swarms, prompt-engineering best practices, code-base refactors, and why unleashing creativity matters more than ever in an AI-driven future."Show NotesObie’s book — https://leanpub.com/patterns-of-application-development-using-aiRoast (GitHub) — https://github.com/Shopify/roastRoast (intro post) — https://shopify.engineering/introducing-roastRaix (core library) — https://github.com/OlympiaAI/raixRaix for Rails — https://github.com/OlympiaAI/raix-railsClaude Swarm (multi-agent YAML swarms) — https://github.com/parruda/claude-swarmClaude Squad https://github.com/smtg-ai/claude-squadClaude Code (agentic coding tool) — https://www.anthropic.com/claude-codeClaude Opus (model family) — https://www.anthropic.com/claude“Software 3.0” (Karpathy talk) — https://www.youtube.com/watch?v=LCEmiRjPEtQSuno (AI music) — https://suno.com/Olympia (AI team platform) — https://olympia.chat/“The Bitter Lesson” (R. Sutton) — https://www.incompleteideas.net/IncIdeas/BitterLesson.htmlPOODR (Sandi Metz) — https://www.poodr.com/Refactoring (Martin Fowler) — https://martinfowler.com/books/refactoring.htmlClean Code (R.C. Martin) — https://www.informit.com/store/clean-code-a-handbook-of-agile-software-craftsmanship-9780135398579Hosts & Guest on Social @thecodenamev @jleo3 @obie

  --------  

  1:16:55

  --------

  1:16:55
• Active Agent with Justin Bowen

  Seventeen-year Ruby veteran Justin Bowen joins hosts Valentino Stoll and Joe Leo to unveil Active Agent—a Rails-native framework that treats every agent like a controller and every prompt like a view, letting you weave LLMs, vector search, and business logic straight into MVC.The crew also digs into the real-world mechanics of shipping AI: defining ground-truth datasets, replay-ready evaluation harnesses, and tight retry logic that keeps hallucinations out of production. You’ll hear a candid take on the current hype cycle (and its parallels to crypto), the challenges of long-term gem maintenance, and fresh ways to keep open-source sustainable—think GitHub Sponsors, corporate grants, and pro-tier gems.What you’ll hearActive Agent 101 – agents as abstract controllers, templated prompts as viewsTesting in the wild – fingerprints, VCR cassettes & CI pipelines for non-deterministic codeContext is king – why ground truth matters when counting cows or parsing legal docsOSS meets ROI – balancing passion projects with sustainable monetisationRails vs. Python/Next.js – reclaiming the one-person startup stackCommunity fuel – Discords, hackathons, and the push for academic & corporate sponsorshipMentioned In The Show:Active Agent (GitHub)  – Justin’s Rails-native, agent-oriented framework for building AI features. Vercel AI SDK  – TypeScript toolkit whose generative-UI ideas helped inspire Active Agent. Maestra.ai  – YC W24 startup offering AI transcription, dubbing, and hosted agent runtimes.Matz's 2025 Ruby Kaigi AI Keynote ONNX Runtime Ruby  – Gem that runs ONNX models (CPU/GPU) from Ruby. PGVector gem  – Ruby bindings for PostgreSQL’s pgvector extension (embeddings storage). Neighbor gem  – k-NN / ANN vector search for Rails & Postgres—pairs nicely with PGVector.Hugging Face JS – Run models in the browser with WebGPU and ONNX  Hugging Face Spaces  – No-config platform for hosting ML demos; handy for sharing agent prototypes. LangSmith (LangChain)  – Evaluation & observability service discussed as a monetization model. CrewAI (GitHub)  – Python framework for orchestrating multi-agent “crews”; Joe’s current go-to. Honeybadger  – Rails-first error-monitoring SaaS—an inspiration for future Active Agent services.Rising Impact – A Netflix anime special about a third-grader's journey to be the world's best golfer. Osmo AI  – Google-born startup using AI to digitise smell—cited in the show’s “AI hype” chat. Ruby AI Builders Discord  – Public Discord community for Rubyists building AI apps.

  --------  

  1:22:25

  --------

  1:22:25
• Sublayer and Artificial Ruby with Scott Werner

  Scott Werner—author of the Works on My Machine newsletter and creator of the Sublayer AI-agent framework—joins Valentino and Joe for a fast-moving conversation on how Rubyists are bending large-language models to their will. We unpack Sublayer’s “generators + actions” architecture, the delightfully chaotic Monkey’s Paw prompt-driven web framework, and Phoenix’s AI-generated test suites, all while debating what remains uniquely human in an age of code that writes itself. If you care about Ruby, rapid prototyping, and staying sane as models ship weekly, this one’s for you.Show NotesMeet Scott Werner – from early Rails days to Works on My Machine and the Artificial Ruby meetup scene. Inside Sublayer – why “string-in → string-out” thinking led to Generators, Actions, and the idea of promptable architecture for code that assembles itself.Monkey’s Paw – a Ruby gem where Markdown “wishes” become full web pages via an LLM—hallucinations welcome.Blueprints & Semantic Linting – templated agent blueprints now built into Sublayer and text-based rules that keep AI code reviews on-message.Phoenix.love – Joe’s Rails-centric tool that churns out thousands of AI-generated tests and the ops pain (alerts, idle “vibe-waiting”) that follows.Feedback Loops & Human Taste – why Paul McCartney’s Get Back jam session is the right metaphor for iterating with an LLM collaborator. When the Model Eats Your Product – surviving weekly model upgrades, function-calling APIs, and the temptation to rebuild everything (again).Ruby’s Next Act – AI-inspired namespacing proposals, Ractors explained, and why dynamic languages still win the “unknown unknowns.”Show-and-Tell PicksScott: TLDraw for visual AI pipelines. Valentino: “AI Software Architect” markdown blue-prints. Joe: “Demystifying Ruby” blog series on threads, fibers & ractors. Referenced URLsSublayer – https://sublayer.comSublayer (GitHub) – https://github.com/sublayerapp/sublayerMonkey’s Paw (GitHub) – https://github.com/sublayerapp/monkeyspawPhoenix – https://phoenix.loveWorks on My Machine newsletter – https://worksonmymachine.substack.comTLDraw – https://tldraw.com---00:00 Introduction to Ruby and AI02:04 Scott's Journey with Ruby and AI04:41 The Evolution of Programming Languages06:38 The Ruby Community's Impact on Software Engineering08:43 Monkey's Paw: A New Approach to Web Development10:35 AI's Role in Creative Processes11:30 Collaboration with AI in Software Development14:50 The Future of Software Development17:24 The Impact of AI on Customer Feedback20:24 Navigating the Rapid Changes in Software Products22:51 Understanding User Feedback in AI Development24:53 The Human Element in AI Collaboration28:20 Prototyping with AI Tools30:18 The Evolving Roles in Teams31:43 Sublayer Tech: Innovations and Frameworks34:36 Blueprints and Code Generation37:14 Navigating Existential Dread in AI Development40:15 The Future of AI and Product Development44:12 Community and Collaboration in Tech47:08 Monitoring AI Processes50:19 The Importance of Orchestration52:03 Final Thoughts and Recommendations

  --------  

  1:11:06

  --------

  1:11:06
• Beyond Chat: Phoenix Tests, Ruby Agents & the AI Tipping Point

  Valentino Stoll and co-host Joe Leo kick off The Ruby AI Podcast with a candid deep-dive into what it really takes to ship AI-powered products in Ruby today. From the origin story of Joe’s test-writing automation platform Phoenix to the surge of new Ruby-first agent libraries, the duo explore why the community is approaching a tipping point, how to escape “chat-bot-only” thinking, and where reactive, evaluation-driven tooling is headed next. Along the way they trade war stories about semver mishaps, code-review “LLM tells,” and the projects, meet-ups, and conferences that keep the Ruby-AI scene buzzing.TakeawaysThe Ruby AI community is growing and offers valuable networking opportunities.Ruby's syntax is well-suited for AI applications, making it a fun choice for developers.Generative AI tools can increase productivity but also add cognitive burden to developers.The integration of AI tools in Ruby applications presents unique challenges.Developers are relearning how to program with the advent of generative AI.AI frameworks are evolving, and Ruby developers need to stay updated.The importance of evaluating AI tools and their effectiveness in real-world applications.Ruby's flexibility allows for creative solutions in AI development.The future of AI in software development will require continuous adaptation.Emerging AI frameworks in Ruby are promising but require careful evaluation. Referenced In The ShowPhoenix by DefMethod – https://www.phoenix.love/OpenAI Ruby SDK – https://github.com/openai/openai-rubySublayer – https://github.com/sublayerapp/sublayerCrewAI – https://github.com/crewAIInc/crewAIActive Agent – https://github.com/activeagents/activeagentRaix – https://github.com/OlympiaAI/raixShopify Roast – https://github.com/Shopify/roastLangChain.rb – https://github.com/patterns-ai-core/langchainrbHugging Face smolagents – https://huggingface.co/docs/smolagents/indexBuilding Code Agents with Hugging Face smolagents – https://www.deeplearning.ai/short-courses/building-code-agents-with-hugging-face-smolagents/V's side project, NowReading.dev – https://nowreading.dev

  --------  

  53:05

  --------

  53:05

Más podcasts de Economía y empresa

Podcasts a la moda de Economía y empresa

Acerca de The Ruby AI Podcast