Building LLM Powered Applications: Create intelligent apps and agents with large language models

Building LLM Powered Applications

Create intelligent apps and agents with large language models

Valentina Alto

Building LLM Powered Applications

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Senior Publishing Product Manager: Tushar Gupta

Acquisition Editors – Peer Reviews: Tejas Mhasvekar and Jane D'Souza

Project Editor: Namrata Katare

Content Development Editors: Shruti Menon and Bhavesh Amin

Copy Editor: Safis Editing

Technical Editor: Anirudh Singh

Proofreader: Safis Editing

Indexer: Subalakshmi Govindhan

Presentation Designer: Ajay Patule

Developer Relations Marketing Executive: Monika Sangwan

First published: May 2024

Production reference: 1140524

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK.

ISBN 978-1-83546-231-7

www.packt.com

Contributors

About the author

Valentina Alto is an AI enthusiast, tech author, and runner. After completing her master's in data science, she joined Microsoft in 2020, where she currently works as an AI specialist. Passionate about machine learning and AI since the outset of her academic journey, Valentina has deepened her knowledge in the field, authoring hundreds of articles on tech blogs. She also authored her first book with Packt, titled Modern Generative AI with ChatGPT and OpenAI Models. In her current role, she collaborates with large enterprises, aiming to integrate AI into their processes and create innovative solutions using large foundation models.

Beyond her professional pursuits, Valentina loves hiking in the beautiful Italian mountains, running, traveling, and enjoying a good book with a cup of coffee.

About the reviewers

Alexandru Vesa has over a decade of expertise as an AI engineer and is currently serving as the CEO at Cube Digital, an AI software development firm he leads with a vision inspired by the transformative potential of AI algorithms. He has a wealth of experience in navigating diverse business environments and shaping AI products in both multinational corporations and dynamic startups. Drawing inspiration from various disciplines, he has built a versatile skill set and seamlessly integrates state-of-the-art technologies with proven engineering methods. He is proficient in guiding projects from inception to scalable success.

Alex is a key figure in the DecodingML publication, collaborating with Paul Iusztin to curate the groundbreaking hands-on course LLM Twin: Building Your Production-Ready AI Replica, hosted on the Substack platform. His problem-solving and communication skills make him an indispensable force in utilizing AI to foster innovation and achieve tangible results.

Louis Owen is a data scientist/AI engineer hailing from Indonesia. Currently contributing to NLP solutions at Yellow.ai, a leading CX automation platform, he thrives on delivering innovative solutions. Louis’s diverse career spans various sectors, including NGO work with The World Bank, e-commerce with Bukalapak and Tokopedia, conversational AI with Yellow.ai, online travel with Traveloka, smart city initiatives with Qlue, and FinTech with Do-it. Louis has also written a book with Packt, titled Hyperparameter Tuning with Python, and published several papers in the AI field.

Outside of work, Louis loves to spend time mentoring aspiring data scientists, sharing insights through articles, and indulging in his hobbies of watching movies and working on side projects.

Join our community on Discord

Join our community’s Discord space for discussions with the author and other readers:

https://packt.link/llm

Contents

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Introduction to Large Language Models

What are large foundation models and LLMs?

AI paradigm shift – an introduction to foundation models

Under the hood of an LLM

Most popular LLM transformers-based architectures

Early experiments

Introducing the transformer architecture

Training and evaluating LLMs

Training an LLM

Model evaluation

Base models versus customized models

How to customize your model

Summary

References

LLMs for AI-Powered Applications

How LLMs are changing software development

The copilot system

Introducing AI orchestrators to embed LLMs into applications

The main components of AI orchestrators

LangChain

Haystack

Semantic Kernel

How to choose a framework

Summary

References

Choosing an LLM for Your Application

The most promising LLMs in the market

Proprietary models

GPT-4

Gemini 1.5

Claude 2

Open-source models

LLaMA-2

Falcon LLM

Mistral

Beyond language models

A decision framework to pick the right LLM

Considerations

Case study

Summary

References

Prompt Engineering

Technical requirements

What is prompt engineering?

Principles of prompt engineering

Clear instructions

Split complex tasks into subtasks

Ask for justification

Generate many outputs, then use the model to pick the best one

Repeat instructions at the end

Use delimiters

Advanced techniques

Few-shot approach

Chain of thought

ReAct

Summary

References

Embedding LLMs within Your Applications

Technical requirements

A brief note about LangChain

Getting started with LangChain

Models and prompts

Data connections

Memory

Chains

Agents

Working with LLMs via the Hugging Face Hub

Create a Hugging Face user access token

Storing your secrets in an .env file

Start using open-source LLMs

Summary

References

Building Conversational Applications

Technical requirements

Getting started with conversational applications

Creating a plain vanilla bot

Adding memory

Adding non-parametric knowledge

Adding external tools

Developing the front-end with Streamlit

Summary

References

Search and Recommendation Engines with LLMs

Technical requirements

Introduction to recommendation systems

Existing recommendation systems

K-nearest neighbors

Matrix factorization

Neural networks

How LLMs are changing recommendation systems

Implementing an LLM-powered recommendation system

Data preprocessing

Building a QA recommendation chatbot in a cold-start scenario

Building a content-based system

Developing the front-end with Streamlit

Summary

References

Using LLMs with Structured Data

Technical requirements

What is structured data?

Getting started with relational databases

Introduction to relational databases

Overview of the Chinook database

How to work with relational databases in Python

Implementing the DBCopilot with LangChain

LangChain agents and SQL Agent

Prompt engineering

Adding further tools

Developing the front-end with Streamlit

Summary

References

Working with Code

Technical requirements

Choosing the right LLM for code

Code understanding and generation

Falcon LLM

CodeLlama

StarCoder

Act as an algorithm

Leveraging Code Interpreter

Summary

References

Building Multimodal Applications with LLMs

Technical requirements

Why multimodality?

Building a multimodal agent with LangChain

Option 1: Using an out-of-the-box toolkit for Azure AI Services

Getting Started with AzureCognitiveServicesToolkit

Setting up the toolkit

Leveraging a single tool

Leveraging multiple tools

Building an end-to-end application for invoice analysis

Option 2: Combining single tools into one agent

YouTube tools and Whisper

DALL·E and text generation

Putting it all together

Option 3: Hard-coded approach with a sequential chain

Comparing the three options

Developing the front-end with Streamlit

Summary

References

Fine-Tuning Large Language Models

Technical requirements

What is fine-tuning?

When is fine-tuning necessary?

Getting started with fine-tuning

Obtaining the dataset

Tokenizing the data

Fine-tuning the model

Using evaluation metrics

Training and saving

Summary

References

Responsible AI

What is Responsible AI and why do we need it?

Responsible AI architecture

Model level

Metaprompt level

User interface level

Regulations surrounding Responsible AI

Summary

References

Emerging Trends and Innovations

The latest trends in language models and generative AI

GPT-4V(ision)

DALL-E 3

AutoGen

Small language models

Companies embracing generative AI

Coca-Cola

Notion

Malbek

Microsoft

Summary

References

Other Books You May Enjoy

Index

Landmarks

Cover

Index

Preface

With this book, we embark upon an exploration of large language models (LLMs) and the transformative paradigm they represent within the realm of artificial intelligence (AI). This comprehensive guide helps you delve into the fundamental concepts, from solid theoretical foundations of these cutting-edge technologies to practical applications that LLMs offer, ultimately converging on the ethical and responsible considerations while using generative AI solutions. This book aims to provide you with a firm understanding of how the emerging LLMs in the market can impact individuals, large enterprises, and society. It focuses on how to build powerful applications powered by LLMs, leveraging new AI orchestrators such as LangChain and uncovering new trends in modern application development.

By the end of this book, you will be able to navigate the rapidly evolving ecosystem of generative AI solutions more easily; plus, you will have the tools to get the most out of LLMs in both your daily tasks and your businesses. Let’s get started!

Who this book is for

The book is designed to mainly appeal to a technical audience with some basic Python code foundations. However, the theoretical chapters and the hands-on exercises are based on generative AI foundations and industry-led use cases, which might be of interest to non-technical audiences as well.

Overall, the book caters to individuals interested in gaining a comprehensive understanding of the transformative power of LLMs and define, enabling them to navigate the rapidly evolving AI landscape with confidence and foresight. All kinds of readers are welcome, but readers who can benefit the most from this book include:

Software developers and engineers: This book provides practical guidance for developers looking to build applications leveraging LLMs. It covers integrating LLMs into app backends, APIs, architectures, and so on.

Data scientists: For data scientists interested in deploying LLMs for real-world usage, this book shows how to take models from research to production. It covers model serving, monitoring, and optimization.

AI/ML engineers: Engineers focused on AI/ML applications can leverage this book to understand how to architect and deploy LLMs as part of intelligent systems and agents.

Technical founders/CTOs: Startup founders and CTOs can use this book to evaluate if and how LLMs could be used within their apps and products. It provides a technical overview alongside business considerations.

Students: Graduate students and advanced undergraduates studying AI, ML, natural language processing (NLP), or computer science can learn how LLMs are applied in practice from this book.

LLM researchers: Researchers working on novel LLM architectures, training techniques, and so on will gain insight into real-world model usage and the associated challenges.

What this book covers

Chapter 1, Introduction to Large Language Models, provides an introduction to and deep dive into LLMs, a powerful set of deep learning neural networks in the domain of generative AI. It introduces the concept of LLMs, their differentiators from classical machine learning models, and the relevant jargon. It also discusses the architecture of the most popular LLMs, moving on to explore how LLMs are trained and consumed and compare base LLMs with fine-tuned LLMs. By the end of this chapter, you will have the foundations of what LLMs are and their positioning in the landscape of AI, creating the basis for the subsequent chapters.

Chapter 2, LLMs for AI-Powered Applications, explores how LLMs are revolutionizing the world of software development, leading to a new era of AI-powered applications. By the end of this chapter, you will have a clearer picture of how LLMs can be embedded in different application scenarios, with the help of new AI orchestrator frameworks that are currently available in the AI development market.

Chapter 3, Choosing an LLM for Your Application, highlights how different LLMs may have different architectures, sizes, training data, capabilities, and limitations. Choosing the right LLM for your application is not a trivial decision as it can significantly impact the performance, quality, and cost of your solution. In this chapter, we will navigate the process of choosing the right LLM for your application. We will discuss the most promising LLMs in the market, the main criteria and tools to use when comparing LLMs, and the various trade-offs between size and performance. By the end of this chapter, you should have a clear understanding of how to choose the right LLM for your application and how to use it effectively and responsibly.

Chapter 4, Prompt Engineering, explains how prompt engineering is a crucial activity while designing LLM-powered applications since prompts have a massive impact on the performance of LLMs. In fact, there are several techniques that can be implemented to not only to refine your LLM’s responses but also reduce risks associated with hallucination and biases. In this chapter, we will cover the emerging techniques in the field of prompt engineering, from basic approaches up to advanced frameworks. By the end of this chapter, you will have the foundations to build functional and solid prompts for your LLM-powered applications, which will also be relevant in the upcoming chapters.

Chapter 5, Embedding LLMs within Your Applications, discusses a new set of components introduced into the landscape of software development with the advent of developing applications with LLMs. To make it easier to orchestrate LLMs and their related components in an application flow, several AI frameworks have emerged, of which LangChain is one of the most widely used. In this chapter, we will take a deep dive into LangChain and how to use it, and learn how to call open-source LLM APIs into code via Hugging Face Hub and manage prompt engineering. By the end of this chapter, you will have the technical foundations to start developing your LLM-powered applications using LangChain and open-source Hugging Face models.

Chapter 6, Building Conversational Applications, allows us to embark on the hands-on section of this book with your first concrete implementation of LLM-powered applications. Throughout this chapter, we will cover a step-by-step implementation of a conversational application, using LangChain and its components. We will configure the schema of a simple chatbot, adding a memory component, non-parametric knowledge, and tools to make the chatbot agentic. By the end of this chapter, you will be able to set up your own conversational application project with just a few lines of code.

Chapter 7, Search and Recommendation Engines with LLMs, explores how LLMs can enhance recommendation systems, using both embeddings and generative models. We will discuss the definition and evolution of recommendation systems, learn how generative AI is impacting this field of research, and understand how to build recommendation systems with LangChain. By the end of this chapter, you will be able to create your own recommendation application and leverage state-of-the-art LLMs using LangChain as the framework.

Chapter 8, Using LLMs with Structured Data, covers a great capability of LLMs: the ability to handle structured, tabular data. We will see how, with plug-ins and an agentic approach, we can use LLMs as a natural language interface between us and our structured data, reducing the gap between the business user and the structured information. To demonstrate this, we will build a database copilot with LangChain. By the end of this chapter, you will be able to build your own natural language interface for your data estate, combining unstructured with structured sources.

Chapter 9, Working with Code, covers another great capability of LLMs: working with programming languages. In the previous chapter, we’ve already seen a glimpse of this capability, when we asked our LLM to generate SQL queries against a SQL Database. In this chapter, we are going to examine in which other ways LLMs can be used with code, from simple code understanding and generation to the building of applications that behave as if they were an algorithm. By the end of this chapter, you will be able to build LLM-powered applications for your coding projects, as well as build LLM-powered applications with natural language interfaces to work with code.

Chapter 10, Building Multimodal Applications with LLMs, goes beyond LLMs, introducing the concept of multi-modality while building agents. We will see the logic behind the combination of foundation models in different AI domains – language, images, audio – into one single agent that can adapt to a variety of tasks. You will learn how to build a multi-modal agent with single-modal LLMs using LangChain. By the end of this chapter, you will be able to build your own multi-modal agent, providing it with the tools and LLMs needed to perform various AI tasks.

Chapter 11, Fine-Tuning Large Language Models, covers the technical details of fine-tuning LLMs, from the theory behind it to hands-on implementation with Python and Hugging Face. We will delve into how you can prepare your data to fine-tune a base model on your data, as well as discuss hosting strategies for your fine-tuned model. By the end of this chapter, you will be able to fine-tune an LLM on your own data so that you can build domain-specific applications powered by that LLM.

Chapter 12, Responsible AI, introduces the fundamentals of the discipline behind the mitigation of the potential harms of LLMs – and AI models in general – that is, responsible AI. This is important because LLMs open the doors to a new set of risks and biases to be taken into account while developing LLM-powered applications.

We will then move on to the risks associated with LLMs and how to prevent or, at the very least, mitigate them using proper techniques. By the end of this chapter, you will have a deeper understanding of how to prevent LLMs from making your application potentially harmful.

Chapter 13, Emerging Trends and Innovations, explores the latest advancements and future trends in the field of generative AI.

To get the most out of this book

This book aims to provide a solid theoretical foundation of what LLMs are, their architecture, and why they are revolutionizing the field of AI. It adopts a hands-on approach, providing you with a step-by-step guide to implementing LLMs-powered apps for specific tasks and using powerful frameworks like LangChain. Furthermore, each example will showcase the usage of a different LLM, so that you can appreciate their differentiators and when to use the proper model for a given task.

Overall, the book combines theoretical concepts with practical applications, making it an ideal resource for anyone who wants to gain a solid foundation in LLMs and their applications in NLP. The following pre-requisites will help you to get the most out of this book:

A basic understanding of the math behind neural networks (linear algebra, neurons and parameters, and loss functions)

A basic understanding of ML concepts, such as training and test sets, evaluation metrics, and NLP

A basic understanding of Python

Download the example code files

The code bundle for the book is hosted on GitHub at https://github.com/PacktPublishing/Building-LLM-Powered-Applications. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781835462317.

Conventions used

There are a number of text conventions used throughout this book.

CodeInText:

Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. For example: "I set the two variables

system_message

and

instructions

."

A block of code is set as follows:

[default] $pip install openai ==

0.28

import

os

import

openai openai.api_key = os.environment.get('OPENAI_API_KEY') response = openai.ChatCompletion.create(     model=

gpt-35-turbo

,

# engine = deployment_name.

    messages=[         {

role

:

system

,

content

: system_message},         {

role

:

user

,

content

: instructions},     ] )

Any command-line input or output is written as follows:

{'text': "Terrible movie. Nuff Said.[…] 'label': 0}

Bold: Indicates a new term, an important word, or words that you see on the screen. For instance, words in menus or dialog boxes appear in the text like this. For example: "[…] he found that repeating the main instruction at the end of the prompt can help the model to overcome its inner recency bias."

Warnings or important notes appear like this.

Tips and tricks appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: Email

feedback@packtpub.com

and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at

questions@packtpub.com

.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you reported this to us. Please visit http://www.packtpub.com/submit-errata, click Submit Errata, and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at

copyright@packtpub.com

with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.

Share your thoughts

Once you’ve read Building LLM Powered Application, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily.

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below:

https://packt.link/free-ebook/9781835462317

Submit your proof of purchase.

That’s it! We’ll send your free PDF and other benefits to your email directly.

1

Introduction to Large Language Models

Dear reader, welcome to Building Large Language Model Applications! In this book, we will explore the fascinating world of a new era of application developments, where large language models (LLMs) are the main protagonists.

During the last year, we all learned the power of generative artificial intelligence (AI) tools such as ChatGPT, Bing Chat, Bard, and Dall-E. What impressed us the most was their stunning capabilities of generating human-like content based on user requests made in natural language. It is, in fact, their conversational capabilities that made them so easily consumable and, therefore, popular as soon as they entered the market. Thanks to this phase, we learned to acknowledge the power of generative AI and its core models: LLMs. However, LLMs are more than language generators. They can be also seen as reasoning engines that can become the brains of our intelligent applications.

In this book, we will see the theory and practice of how to build LLM-powered applications, addressing a variety of scenarios and showing new components and frameworks that are entering the domain of software development in this new era of AI. The book will start with Part 1, where we will introduce the theory behind LLMs, the most promising LLMs in the market right now, and the emerging frameworks for LLMs-powered applications. Afterward, we will move to a hands-on part where we will implement many applications using various LLMs, addressing different scenarios and real-world problems. Finally, we will conclude the book with a third part, covering the emerging trends in the field of LLMs, alongside the risk of AI tools and how to mitigate them with responsible AI practices.

So, let’s dive in and start with some definitions of the context we are moving in. This chapter provides an introduction and deep dive into LLMs, a powerful set of deep learning neural networks that feature the domain of generative AI.

In this chapter, we will cover the following topics:

Understanding LLMs, their differentiators from classical machine learning models, and their relevant jargon

Overview of the most popular LLM architectures

How LLMs are trained and consumed

Base LLMs versus fine-tuned LLMs

By the end of this chapter, you will have the fundamental knowledge of what LLMs are, how they work, and how you can make them more tailored to your applications. This will also pave the way for the concrete usage of LLMs in the hands-on part of this book, where we will see in practice how to embed LLMs within your applications.

What are large foundation models and LLMs?

LLMs are deep-learning-based models that use many parameters to learn from vast amounts of unlabeled texts. They can perform various natural language processing tasks such as recognizing, summarizing, translating, predicting, and generating text.

Definition

Deep learning is a branch of machine learning that is characterized by neural networks with multiple layers, hence the term deep. These deep neural networks can automatically learn hierarchical data representations, with each layer extracting increasingly abstract features from the input data. The depth of these networks refers to the number of layers they possess, enabling them to effectively model intricate relationships and patterns in complex datasets.

LLMs belong to a wider set of models that feature the AI subfield of generative AI: large foundation models (LFMs). Hence, in the following sections, we will explore the rise and development of LFMs and LLMs, as well as their technical architecture, which is a crucial task to understand their functioning and properly adopt those technologies within your applications.

We will start by understanding why LFMs and LLMs differ from traditional AI models and how they represent a paradigm shift in this field. We will then explore the technical functioning of LLMs, how they work, and the mechanisms behind their outcomes.

AI paradigm shift – an introduction to foundation models

A foundation model refers to a type of pre-trained generative AI model that offers immense versatility by being adaptable for various specific tasks. These models undergo extensive training on vast and diverse datasets, enabling them to