Category: gen-ai

Evolution of Generative AI Many years ago, our smartphone’s predictive text feature made us wonder. Then Gmail made our lives easy with the Smart Reply feature which utilizes machine learning algorithms to provide us one one-sentence response. These basic innovations are examples of initial forms of generative AI. But, since then, with the advancements in generative AI we have come a long way. The evolution of generative AI is a fascinating story. Let’s see how GenAI has evolved and transformed the landscape of AI − Early AI Exploration (1950s – 1980) 1950s was the time when early ideas of AI emerged. In 1950, Alan Turing in his paper “Computer Machinery and Intelligence” explored the mathematics possibilities of AI and raised a question why machines can’t think like a human being. Then in 1952, Christopher Strachey, a British computer scientist, created a program for the Manchester Mark 1 computer that generated a simulated love letter. This program was the first text-generated software. In 1966, Joseph Weizenbaum, an MIT professor, creates the first chatbot named ELIZA. It was an early natural language processing program which simulates conversations with a psychotherapist. In 1968, a Stanford university student Terry Winograd at MIT creates a natural language processing computer program named SHRDLU. It was actually a demonstration of a system capable of understanding and responding to commands in a restricted block world environment. Michael Toy and Glenn Wichman developed a Unix-based video game called Rogue in 1980. It was one of the first games to implement procedural generation for dynamically generating new game levels. Neural Network Resurgence (1980s – 2010) In 1985, Judea Pearl, a renowned computer scientist and philosopher introduced Bayesian networks which are also known as belief networks of casual networks. Bayesian networks establish the modeling concept in Generative AI. Michael Irwin Jordan, in 1986, with his publication “Serial order: A parallel distributed processing approach” lay the foundation for use of RNNs (Recurrent Neural Networks). In 1989, Yann LeCun and Yosua Bengio demonstrate the use of CNN (Convolutional Neural Networks) for image recognition. In the year 2003, Researchers from University of Montreal published a paper, “A Neural Probabilistic Language Model”. This paper suggests a technique for language modeling using feed-forward neural networks. In 2006, Fei-Fei Li, a professor at Stanford University creates ImageNet database that provides the foundation for visual object recognition. Deep Learning Dominance & Transformer Revolution (2010s – 2020) In 2011, Apple released Siri, a text-to-speech voice based on deep learning technology. In 2012, the AlexNet CNN architecture was introduced by Alex Krizhevsky. It was indeed an innovative approach to automatically train neural networks that take advantage of recent GPU advances. Ian Goodfellow and his colleagues develops Generative Adversarial Networks (GANs) in the year 2014. In the same year, Max Welling and Diederick Kingma developed Variational Encoders (VAEs) to generate text, images, and videos. In 2015, a group of researchers from Stanford University published a paper “Deep Unsupervised Learning using Nonequilibrium Thermodynamics”. They introduced a technique on diffusion model that provides a way to reverse engineer the process of adding noise to an image. Google researchers, in the year 2017, introduce the concept of transformers. This technique automatically parses unlabeled text into large language models (LLMs). In 2018, Google implemented transformers into BERT (Bidirectional Encoder Representations from Transformers). In the same year, GPT-1, a transformer-based language model was introduced by OpenAI. Specialized Generative Models (2020s – Present) In 2020, OpenAI released the third iteration of their Generative Pre-trained Transformer, i.e., GPT-3. It was one of the largest language models capable of generating human-like text. Next year, in 2021, OpenAI launched Dall-E that can generate images from text prompts. On November 30, 2022, OpenAI unveiled the web preview of ChatGPT. Open AI Unveiled GPT-4 in 2023. The AI company claims that “GPT-4 can solve challenging problems with greater accuracy, thanks to its broader general knowledge and advanced reasoning capabilities.” On August 20, 2023, OpenAI launched DALL-E3. In March 2023, Google released the Bard chat service based on its LaMDA engine. But, on February 8, 2024, Google rebranded the Bard chatbot as Gemini.

Types of Generative Models Generative models have gained significant popularity in recent times. These innovative algorithms, mainly used for unsupervised learning, are proficient in dealing with the underlying distribution of data and generating complex output, such as images, music, and natural language, comparable to the original training data. Read this chapter to explore three prominent and most widely used types of generative models: Generative Adversarial Networks (GANs), Autoencoders, and Variational Autoencoders (VAEs). Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) were introduced by Ian Goodfellow and his teammates in 2014. GANs, an approach to generative modeling, are based on deep neural network architecture that generates a new complex output that looks like the original training data. GAN framework has two neural networks- ‘Generator’ and ‘Discriminator’. Working of GANs Let’s understand the working of GAN model with the help of below given diagram − As depicted in the diagram, the GANs has two main components: a generator network and a discriminative network. The process starts by providing the generator with a random seed/noise vector. Now, the generator uses this input to creates new, synthetic samples. Then, these generated samples along with the real data sample provided to the discriminative network. The discriminative network then evaluates the realism of these samples, i.e., if the sample if real or fake. Finally, the discriminator provides feedback on the generator”s output by adjusting the generator”s parameters through backpropagation. The generator and discriminator then continue to learn and adapt to each other, until the generator is producing highly realistic samples that can fool the discriminator. Application of GANs Generative Adversarial Networks (GANs) find their applications in various domains. In fact, DALL-E, a specific model developed by OpenAI, combines ideas from GANs and transformers to generate images from textual descriptions. Some other applications of GANs include the following − Image Generation Data Augmentation Text-to-Image Synthesis Video Generation and Prediction Anomaly Detection Face Aging and Rejuvenation Style Transfer and Image Editing Autoencoders Another widely used generative model that has revolutionized various domains, from computer vision to natural language processing is autoencoders. An autoencoder is an designed to learn data encodings in an unsupervised manner. Traditional neural networks, used for supervised learning tasks such as classification and regression, map input data to corresponding output labels. On the other hand, autoencoders learn to reconstruct input data by decoding high-dimensional input data into lower-dimensional representation. The Architecture of Autoencoders The architecture of autoencoders consists of three main parts − Encoder −It compresses the information into a dense encoding by mapping the input data to a lower-dimensional representation. Bottleneck Layer (Latent Space) −In this layer the latent space representation captures the essential features of the input data in a compressed form. Decoder −It decompresses the compressed representation back to the original input space by reconstructing it. The main aim of this module is to minimize reconstruction errors. Application of Autoencoders Listed below are some of the applications of Autoencoders − Image Compression and Reconstruction Feature Learning and Representation Anomaly Detection Dimensionality Reduction Natural Language Processing Variational Autoencoders Variational autoencoders (VAEs) are a class of generative models that are based on the concept of autoencoders we have studied above. Traditional autoencoders learn deterministic mappings between input and latent space representations. VAEs, on the other hand, generate parameters for probability distribution in the latent space. This feature enables VAEs to capture the underlying probability distribution of the input data samples. Architecture and Components of VAEs Like autoencoders, the architecture of VAEs consists of two main components: an encoder and a decoder. The encoder, in VAEs, rather than using deterministic mappings as in autoencoders, proposes probability modeling into the latent space. Given below are the key components of VAEs − Encoder − It maps the input data samples to the parameters of a probability distribution in the latent space. After mapping the encoder gives mean and variance vectors of each data point. Latent Space − This component represents the learned probabilities of the input sample data by the encoder. Decoder − It reconstructs the data samples by using the samples from latent space. The aim of the decoder is to match the input data distribution. Application of Variational Autoencoders (VAEs) Variational autoencoders (VAEs) find their application across various domains like autoencoders. Some of these are listed below − Image Generation Data Visualization Feature Learning Anomaly Detection Natural Language Processing In the subsequent chapters, we will discuss these prominent and most widely used types of generative models in detail. Conclusion In this chapter, we presented an overview of the three most widely used generative models namely, Generative Adversarial Networks (GANs), Autoencoders, and Variational Autoencoders (VAEs). Their unique capabilities contribute to the advancements of generative modeling. GANs, with their adversarial training framework, can generate a new complex output that looks like the original training data. We discussed GANs working using its framework that consists of two neural networks: Generator and Discriminator. Autoencoders, on the other hand, aim to learn data encodings in an unsupervised manner. They reconstruct input data by decoding high-dimensional input data into lower-dimensional representation. Variational Autoencoders (VAEs) introduced the probabilistic latent space representations. They bridge the gap between autoencoders and probabilistic modeling by capturing the underlying probability distribution of sample input data. Whether it’s generating realistic images, learning meaningful representations of data, or exploring the probabilistic latent space representations, GANs, autoencoders, and VAEs shaping the future of AI-driven generative technologies.

Generative Adversarial Network – Architecture and Types A Generative Adversarial Network (GAN) typically utilizes architectures such as convolutional neural networks (CNN). GAN framework is composed of two neural networks: Generator and Discriminator. These networks play an important role where the generator focuses on creating new data and the discriminator evaluates it. Read this chapter to learn the architecture of GANs, their components, types and mechanism that make them so powerful. The Role of Generator in GAN Architecture The first primary part of GAN architecture is the Generator. Let’s see its function and structure − Generator: Function and Structure The primary goal of the generator is to generate new data samples that are intended to resemble real data from the dataset. It begins with a random noise vector and transforms it through fully connected layers like Dense or Convolutional layers to generate synthetic data sample. Generator: Layers and Components Listed below are the layers and components of the generator neural network − Input Layer − The generator receives a low dimensionality random noise vector or input data as input. Fully Connected Layers − The FLC is used to increase the input noise vector dimensionality. Transposed Convolutional Layers − These layers are also known as deconvolutional layers. It is used for upsampling i.e., to generate an output feature map having greater spatial dimension than the input feature map. Activation Functions − Two commonly used activations functions are: Leaky ReLU and Tanh. The Leaky ReLU activation function helps in decreasing the dying ReLU problem, while the Tanh activation function makes sure that the output is within a specific range. Output Layer − It produces the final data output like an image of a certain resolution. Generator: Objective Function The goal of generator neural network is to create data that the discriminator cannot distinguish from real data. This is achieved by minimizing the generator’s loss function − $$\mathrm{L_{G} \: = \: \log(1 \: – \: D(G(Z)))}$$ Here, G(z) is the generated data and D(⋅) represents the discriminator’s output. The Role of Discriminator in GAN Architecture The second part of GAN architecture is the Discriminator. Let’s see its function and structure − Discriminator: Function and Structure The primary goal of the discriminator is to classify the input data as real or generated by the generator. It takes a data sample as input and gives a probability as output that indicates whether the sample is real or fake. Discriminator: Layers and Components Listed below are the layers and components of the discriminator neural network − Input Layer − The discriminator receives a data sample from either the real dataset or the generator as input. Convolutional Layers − It is used for downsampling the input data to extract relevant features. Fully Connected Layers − The FLC is used to process the extracted features and make a final classification. Activation Functions − It uses Leaky ReLU activation function to address the vanishing gradient problem. It also introduces non-linearity. Output Layer − As name implies, it gives a single probability value between 0 and 1 as output that indicates whether the sample is real or fake. Discriminator: Objective Function The goal of discriminator neural network is to maximize its ability to correctly distinguish real data from generated data. This is achieved by minimizing the discriminator’s loss function − $$\mathrm{L_{D} \: = \: -(\log D(X) \: + \: \log(1 \: – \: D(G(Z))))}$$ Here, “x” is a real data sample. Types of Generative Adversarial Networks We can have different types of GAN models based on the way the generator and the discriminator networks interact with each other. Here are some notable variations − Vanilla GAN Vanilla GAN represents the simplest form of generative adversarial networks (GANs). It provides a fundamental understanding of how GANs work. The term “Vanilla” implies that this is the simplest form without any advanced modifications or enhancements. Deep Convolutional GANs (DCGANs) DCGANs is one of the most popular implementations of GANs. It is composed of ConvNets in the place of multi-layer perceptron to stabilize GAN training. These guidelines have significantly stabilized GAN training particularly for image generation tasks. Some of the key features of DCGANs include the use of: Strided Convolutions Batch Normalization The removal of fully connected hidden layers Conditional GANs (cGANs) Conditional GAN (cGAN) includes additional condition information like class labels, attributes, or even other data samples, into both generator and discriminator. With the help of these conditioning information, Conditional GANs provide us the control over the characteristic of the generated output. CycleGANs CycleGANs are designed for unpaired image-to-image translation tasks where there is no relation between the input and output images. A cycle consistency loss is used to ensure that translating from one domain to another and back again produces consistent results. Progressive GANs (ProGANs) ProGANs generate high-resolution images by progressively increasing the resolution of both the generator and discriminator during training. With this approach, you can create more detailed and higher-quality images. StyleGANs StyleGANs, developed by NVIDIA, is specifically designed for generating photo-realistic high-quality images. They introduced some innovative techniques for improved image synthesis and have some better control over specific attributes. Laplacian Pyramid GAN (LAPGAN) Laplacian Pyramid GAN (LAPGAN) is a type of generative adversarial network that uses a multi-resolution approach to generate high-quality images. It uses a Laplacian pyramid framework where images are generated at multiple scales. LAPGANs are mainly effective in creating detailed and realistic images as compared to standard GANs. Conclusion GANs enable us to create realistic data across various domains. In this chapter, we explained the architecture and mechanisms of GANs.

Discriminative vs Generative Models Human minds inspire Machine Learning (ML) and Deep Learning (DL) technologies, how we learn from our experience to make better choices in the present and future. These technologies are the most dynamic and ever-changing fields of study and while we are already using them in many ways, the possibilities are endless. These advancements empower machines to learn from past data and predict even from unseen data inputs. To extract meaningful insights from raw data, machines depend upon mathematics, models/algorithms, and data processing methods. There are two ways we can enhance machine efficiency; one is to increase the volume of data, and another is to develop new and more robust algorithms. Getting fresh data is very easy as quintillions of data are generated daily. But to work with such huge data, we need to either build new or scale up the existing models/algorithms. Mathematics serves as the backbone of these models/algorithms which can be broadly categorized into two groups namely, Discriminative Models and Generative Models. In this chapter we will be looking at discriminative and generative ML models along with the core differences between them. What are Discriminative Models? Discriminative models are ML models and, as the name suggests, concentrate on modeling the decision boundary between several classes of data using probability estimates and maximum likelihood. These types of models, mainly used for supervised learning, are also known as conditional models. Discriminative models are not much affected by the outliers. Although this makes them a better choice than generative models, it also leads to misclassification problem which can be a big drawback. From a mathematical perspective, the process of training a classifier involves estimating either, A function represented as f : X → Y, or The probability P(Y│X). However, discriminative classifiers − Assume a particular functional form of probability P(Y|X), and Directly estimate the parameters of probability P(Y|X) from the training dataset. Popular Discriminative Models Discussed below are some examples of widely used discriminative models − Logistic Regression Logistic Regression is a statistical technique used for binary classification tasks. It models the relationship between the dependent variable and one or more independent variables using the logistic function. It produces an output between 0 and 1. Logistic regression can be used for various classifications of problems like cancer detection, diabetes prediction, spam detection, etc. Support Vector Machines A support vector machine (SVM) is a powerful yet flexible supervised ML algorithm with applications in regression as well as classification scenarios. Support vectors divide an n-dimensional data space into numerous classes in a hyperplane using decision boundaries. K-nearest Neighbor (KNN) KNN is a supervised ML algorithm that uses feature similarity to predict new data points value. The values assigned to new data points depend on how closely they match the points in the training set. Decision trees, neural nets, conditional random field (CRF), random forest are few other examples of the commonly used discriminative models. What are Generative Models? Generative models are ML models and, as the name suggests, aim to capture the underlying distribution of data, and generate new data comparable to the original training data. These types of models, mainly used for unsupervised learning, are categorized as a class of statistical models capable of generating new data instances. The only drawback of generative models, when compared to discriminative models, is that they are prone to outliers. As discussed above, from a mathematical perspective, the process of training a classifier involves estimating either, A function represented as f : X → Y, or The probability P(Y│X). However, generative classifiers − Assume a particular functional form for the probabilities such as P(Y), P(X|Y) Directly estimate the parameters of probability such as P(X│Y), P(Y) from the training dataset. Calculates the posterior probability P(Y|X) using the Bayes’ Theorem. Popular Generative Models Highlighted below are some examples of widely used generative models − Bayesian Network A Bayesian Network, also known as Bayes’ network, is a probabilistic graphical model that represents relationships between variables using a directed acyclic graph (DAG). It has many applications in various fields such as healthcare, finance, and natural language processing for tasks like decision-making, risk assessment, and prediction. Generative Adversarial Network (GAN) These are based on deep neural network architecture consisting of two main components namely a generator and a discriminator. The generator trains and creates new data instances and the discriminator evaluates these generated data into real or fake instances. Variational Autoencoders (VAEs) These models are a type of autoencoder, trained to learn a probabilistic latent representation of the input data. It learns to generate new samples like the input data by sampling from a learned probability distribution. VAEs are useful for tasks like generating images from text descriptions, as seen in DALL-E-3, or crafting human-like text responses like ChatGPT. Autoregressive model, Naïve Bayes, Markov random field, Hidden Markov model (HMM), Latent Dirichlet Allocation (LDA) are few other examples of the commonly used generative models. Difference Between Discriminative and Generative Models Data scientists and machine learning experts need to understand the differences between these two types of models to choose the most suitable one for a particular task. The table below depicts the core differences between discriminative and generative models − Characteristic Discriminative Models Generative Models Objective Focus on learning the boundary between different classes directly from the data. Their primary objective is to classify input data accurately based on the learned decision boundary. Aim to understand the underlying data distribution and generate new data points that resemble the training data. They focus on modeling the process of data generation, allowing them to create synthetic data instances. Probability Distribution Estimates the parameters of probability P(Y|X) from the training dataset. Calculates the posterior probability P(Y|X) using the Bayes’ Theorem. Handling Outliers Relatively robust to outliers Prone to outliers Property They do not possess generative properties. They possess discriminative properties. Applications Commonly used in classification tasks, such as image recognition and sentiment analysis. Commonly used in tasks like data generation, anomaly detection, and data augmentation, beyond traditional classification tasks.

Generative Adversarial Networks Applications Generative Adversarial Networks (GANs) can generate highly realistic data samples across various domains, so they gained significant attention in generative modeling. GANs are a type of deep learning architecture that can generate highly realistic data which is difficult to distinguish from real data. That’s the reason GANs are used in various applications including image generation, cartoon characters generation, generation of 3D objects, and video prediction. Applications of Generative Adversarial Networks Read this chapter to get a clear understanding of the types of problems that GANs can address and the domains where they are widely used. Generating Example for Image Dataset GANs can be used to generate synthetic images which are difficult to distinguish from real data. This application is useful in cases where it is either expensive or difficult to get a large amount of real data. Researchers can augment datasets with synthetic data and train ML models. It enhances the performance of ML models in various tasks such as classification, segmentation, and object detection. Generating Photographs of Human Faces GANs can generate highly realistic photographs of human faces, including those that don’t even exist in the real world, with various characteristics like different ages, backgrounds, and expressions. We can use these generated faces in creating avatars for social media, generating training data for facial recognition system, etc. For example, we have a tool that uses real-life and synthetic datasets for generating human photos from scratch by AI. Below is an example of photo generated by this tool based on the description provided in left side. Generating Realistic Images GANs can generate highly realistic images of objects, scenes, etc. We can use these generated images in various domains like virtual reality (VR), gaming, and content creation. For example, in architecture and interior design, architects can use GANs to generate realistic visualizations of buildings and interiors. One such tool is which is developed by OpenAI. It is an AI-powered image creator that changes the way architects generate visuals and scale their designs. Generating Cartoon Characters Artists and animators can use GANs to generate cartoon style images with different styes and characteristics. These generated cartoon images can be used in animation, comics, and character design. For example, we have the tool for generating cartoon characters. Image-to-Image Translation GANs, specifically Conditional GANs, are best suited for tasks like translating images from one domain to another. Translating images includes converting satellite images to maps, transforming sketches into realistic images, or converting day-time scenes to night-time scenes etc. For example, provides us an AI image translator tool. Semantic-Image-to-Realistic Image Translation GANs can generate realistic images based on textual descriptions or semantic layouts. For example, if you provide a semantic layout of a room, GANs can generate a photorealistic image of that room. This technique is useful in the field of architectural visualization and interior design. Generating Face Frontal View GANs can generate frontal views of faces from non-frontal images. This application is useful in face recognition systems. For example, is a popular tool that you can use for generating AI faces. Generating New Poses GANs, specifically StyelGANs, can generate new human poses. This technique finds its best application in animation, sports analytics, and virtual try-on applications. For example, in sports analytics, GANs can be used to generate realistic poses for athletes to analyze their movements and techniques. Generating Emojis from Photos GANs can create personalized emojis-style images from the photographs you provided to it. This helps enhance the communication platforms with custom visual expressions. For example, you can use “magickimg.com” to generate emojis from photos. Image Super-Resolution GANs can be used for image super-resolution tasks in which low-resolution images are transformed into similar high-resolution images. Image Inpainting using GAN Based on contextual information, GANs can fill in missing parts of the images. For example, image inpainting techniques using GANs can reconstruct damaged areas in photographs that helps in the recovery of valuable visual information in forensic investigations or in historical document preservation. provides us an AI tool for image inpainting. Video Prediction using GAN Based on a given sequence of past frames in a video, GANs can generate realistic future frames in a video sequence. This helps in reducing the amount of data required to store and transmit videos. This technique is very useful for entertainment and virtual reality applications. Generating 3D Objects using GAN GANs can generate realistic, high-quality 3D models of objects like buildings, cars, and people from their 2D images. We can use this in virtual reality, video games, and computer-aided design (CAD). We have a tool which can generate 3D objects. Clothing Translation GANs have been used to develop systems for clothing translation which converts an image of clothing from one design to another. The tool named can be used for this application. It also gives you a free trial. Face Morphing GANs provide us with smooth morphing between different facial attributes. This feature makes GAN useful for applications like age progression, gender transformation, and facial expression transfer. For example, can also be used for face morphing. Conclusion The applications of Generative Adversarial Networks (GANs) explored in this chapter demonstrate the remarkable potential of this cutting-edge technology across various domains. GANs provide innovative approaches to image-to-image translation, generating new poses, and semantic-image-to-realistic photo translation. From generating realistic images of human faces to enhancing the resolution of photographs and enabling virtual try-on experiences in the fashion industry, GANs have changed the way we create, manipulate, and interact with visual content.

How do Generative Adversarial Networks (GAN) Work? Generative Adversarial Networks (GAN) is a powerful approach to generative modeling. GANs are based on deep neural network architecture that generates a new complex output that looks like the original training data. GANs typically utilize architectures such as convolutional neural networks (CNN). In fact, ChatGPT, like other LLMs (Large Language Models) based on deep learning, is a remarkable application of GAN. This chapter covers all you need to know about GAN and its working. What is a Generative Adversarial Network? A Generative Adversarial Network (GAN) is a type of artificial intelligence framework that is used for unsupervised learning. GANs are made up of two neural networks: a Generator and a Discriminator. GANs use adversarial training to produce artificial data that resembles the actual data. GANs can be divided to have three components − Generative − This component focuses on learning how to generate new data by understanding the underlying patterns in the dataset. Adversarial − In simple terms, “adversarial” means setting two things in opposition. In GANs, the generated data is compared to real data from the dataset. This is done using a model trained to distinguish between real and fake data. This model is known as discriminator. Networks − To enable the learning process, GANs uses deep neural networks. Before getting into how GANs work, let”s first discuss its two primary parts: the Generator model and the Discriminator model. The Generator Model The goal of the generator model is to generate new data samples that are intended to resemble real data from the dataset. It takes random input data as input and transforms it into synthetic data samples. Once transformed, the other goal of the generator is to produce data that is identical to real data when presented to the discriminator. The generator is implemented as a neural network model. Depending on the type of data being generated, it uses fully connected layers like Dense or Convolutional layers. The Discriminator Model The goal of the discriminator model is to evaluate the input data and tries to distinguish between real data samples from the dataset and fake data samples generated by the generator model. It takes input data and predicts whether it is real or fake. Another goal of the discriminator model is to correctly classify the source of the input data as real or fake. Like the generator model, the discriminator model is also implemented as a neural network model. It also uses Dense or Convolutional layers. During the training of a GAN, both the generator and the discriminator are trained simultaneously but in adverse ways, i.e., in competition with each other. How does a GAN Work? To understand how a GAN works, first take a look at this diagram that shows how the different components of a GAN function to generate new data samples that closely resemble with real data − GANs have two main components: a generator network and a discriminative network. Given below are the steps involved in the working of a GAN − Initialization The GAN consists of two neural networks: the Generator (say G) and the Discriminator (say D). The goal of the generator is to generate new data samples like images or text that closely resemble the real data from the dataset. The discriminator, playing the role of a critic, has the goal to distinguish between the real data and the data generated by the generator. Training Loop The training loop involves alternating between training the generator and the discriminator. Training the Discriminator While training the discriminator, for each iteration − First, select a batch of real data samples from the dataset. Next, enerate a batch of fake data samples using the current generator. Once generated, train the discriminator on both the real and fake data samples. Finally, the discriminator learns to distinguish between real and fake data by adjusting its weights to minimize its classification error. Training the Generator While training the generator, for each iteration − First, generate a batch of fake data samples using the generator. Next, train the generator to produce fake data that the discriminator classifies as the real data. To do this, we need to pass the fake data through the discriminator and update the generator”s weights based on the discriminator”s classification error. Finally, the generator will learn to produce more realistic fake data by adjusting its weights to maximize the discriminator”s error when classifying its generated samples. Adversarial Training As the training progresses, both the generator and discriminator improve their performance in an adversarial manner, i.e., in opposition. The generator gets better at creating fake data that resembles real data, while the discriminator gets better at distinguishing between real and fake data. With the help of this adversarial relationship between the generator and discriminator, both the networks try to improve continuously until the generator generates data that is identical to the real data. Evaluation Once the training is over, the generator can be used to generate new data samples that resemble the real data from the dataset. We can evaluate the quality of the generated data either by inspecting samples visually or using quantitative measures like similarity scores or classifier accuracy. Fine-Tuning and Optimization Depending on the application, you can fine-tune the trained GAN model to improve its performance or adapt it to specific tasks or datasets. Conclusion Generative Adversarial Networks (GANs) is one of the most prominent and widely used generative models. In this chapter, we explained the basics of a GAN and how it works using neural networks to produce artificial data that resembles actual data. The steps that are involved in the working of a GAN include: Initialization, Training Loop, Training the Discriminator, Training the Generator, Adversarial Training, Evaluation, and Fine tuning and Optimization.

ML and Generative AI After the initial release of on November 30, 2022, the interest in artificial intelligence has become widespread. ChatGPT (GPT stands for Generative Pre-Trained Transformer) is a conversational AI system developed by OpenAI that anyone can experiment with and use as it facilitates natural conversations between humans and the bot. Within a very short period of time, ChatGPT has made us think about how AI affects our society and economy. But one thing is sure, AI is becoming a crucial part of our lives and will shape our future in the coming years. Just like Generative AI is the brain behind tools like ChatGPT and Dall-E3, one can regard “Machine Learning” and “Deep Learning” as the primary components that shape Generative AI. Read this chapter to get an overview of ML and DL and how these two concepts play a critical role in shaping Generative AI in its present form. AI is not an isolated discipline; it is an umbrella of every technology that helps transcend human capabilities. With the help of below given diagram, let’s understand the relationships of various disciplines to each other and to AI. Generative AI is the latest subtype of AI that is reshaping the landscape of creativity and innovation. Other subtypes of AI namely Machine Learning and Deep Learning lay the foundations of generative AI. In this chapter, we will briefly overview the foundations of generative AI, including machine learning, its subtypes, and deep learning. Machine Learning – A Brief Overview is a subset of that enables computer systems or machines to extract patterns from raw data by using an algorithm or method. It builds a model themselves by learning from experience and available data, without being explicitly programmed. Based on training methods as well as data availability, machine learning has following three basic learning categories − Supervised Learning In this category of machine learning, the algorithm is trained using a labeled dataset. Basically, the algorithm or model in supervised learning is provided with input-output pairs where every input is matched with a corresponding output or label. The main goal is to make a model to learn the relationship between input and outputs, enable it to accurately predict or classify new, unseen data. Unsupervised Learning In this category, as opposed to supervised learning, the model is trained without labeled dataset. It learns to analyze and derive insights from data autonomously. The main goal is to make a model to learn the relationship within unlabeled data. Reinforcement Learning In this paradigm of machine learning, rather than using labeled or unlabeled data, the model is trained with the help of agent and environment. The agent learns to make decisions by interacting with an environment. First it takes actions in the environment and then receives feedback in the form of rewards or punishments. Finally, the agent uses the feedback to improve its decision-making. Contribution of ML in Generative AI Let’s understand how machine learning contributes to the foundation of generative AI − Learning From Data In the early stages of development, generative AI models use supervised learning to train models so that they can generate content based on the learned relationship between input and outputs. Understanding Patterns and Relationships Generative AI utilizes unsupervised learning to uncover patterns and relationships. It helps generative AI models to generate new content from unlabeled data. Adaptability and Improvements In generative AI, adaptability is very important, especially for the tasks that need continuous improvements. Generative AI models use reinforcement learning to refine their output based on feedback and rewards. ChatGPT, in fact, uses Reinforcement Learning with Human Feedback (RLHF) that involves a small increment of human feedback to refine the agent’s learning process. Optimizing Model Parameters Generative AI models use ML optimization techniques to fine-tune parameters. It enhances their performance, and they can generate more accurate content. Transfer Learning Generative AI uses another ML paradigm called transfer learning, to pre-train their model. It helps the model to accelerate the learning for specific content generation process. Deep Learning – A Brief Overview is a subset of ML inspired by the structure and function of the human brain. It uses a multi-layered structure of algorithms called artificial neural network (ANN) to extract complex features from input data. Deep Learning algorithms, in contrast to algorithms, once set up requires less human intervention. It also requires less time for testing and hence can generate results instantaneously. Let’s understand how deep learning contribute to the foundation of generative AI − Hierarchical Representations To generate diverse content, generative AI needs to learn hierarchical representations of data. Deep neural networks (neural networks with multiple layers), a kind of deep learning model, helps generative AI models to do so. Convolutional Neural Network (CNNs) It is a type of ANN (artificial neural network) used for analyzing images. They automatically learn spatial hierarchies of features from input images using convolutional layers. Generative AI models use CNN to extract features from visual data and to facilitate cross-modal tasks like text-to-image generation. This makes CNN a powerful tool in advancing generative AI capabilities. Recurrent Neural Network (RNN) They are feedforward neural networks with closed loops i.e., all the nodes are connected to all the other nodes. Each node in RNN works as both input and output. Generative AI models use RNNs to learn from examples and create new sequences of data that follow patterns they have learned. Large-Scale Data Handling To train generative AI models we need to access large-scale datasets. Deep learning models help to handle such datasets. Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) are a type of deep neural network architecture used for generative modelling. GANs have proven highly effective for their innovative approach to generate realistic images, videos, and other kinds of content generation. Conclusion In this chapter, we explained how the various disciplines of AI are related to each other. We also seen an overview of machine learning and deep learning and how they play an important role in laying the foundation of generative

The Role of Probability Distribution in Generative Models Machine Learning (ML) and Deep Learning (DL) advancements empower machines to learn from past data and predict even from unseen data. One such advancement is Generative Models that capture the underlying distribution of data and generate new data comparable to the original training data. But how do they do it? It’s Probability Distribution with the help of which generative models can manage the uncertainties and variations in the data. Read this chapter to understand probability distribution, its types, its uses in Generative Modeling, and its applications. What is Probability Distribution? Probability Distribution is a mathematical function that represents the probability of different possible values of a random variable within a given range. We can use either graphs or probability tables to depict probability distribution. For example, imagine flipping a coin, there is a probability distribution that tells us the chances of getting heads or tails. The following probability table describes it − Outcomes Probability Heads 0.5 Tails 0.5 A probability distribution is a theoretical representation of frequency distribution (FD). In statistics, FD describes the number of occurrences of a variable in a dataset. On the other hand, probability distribution, along with the frequencies of number of occurrences, also assigns probabilities to them. As we know probability, that says how likely something is to occur is a number, is between 0 (means impossible) and 1 (means certain). That’s why the higher probability of a value represents its higher frequency in a sample. Types of Probability Distributions There are two types of probability distributions − Discrete Probability Distributions Continuous Probability Distributions Let”s take a closer look at these two types of probability distributions. Discrete Probability Distributions Discrete probability distributions are mathematical functions that describe the probabilities of different occurrences from a discrete or categorial random variables. Discrete probability distribution includes only those values with a possible probability. In simple words, it does not include any value with zero probability. For example, 5.5 is not a possible outcome of dice rolls, hence it does not include as a probability distribution of dice rolls. The total of the probabilities of all possible values in a discrete probability distribution is always one. Let’s see some common discrete probability distributions − Discrete Probability Distribution Explanation Example Bernoulli Distribution It describes the probability of success (1) or failure (0) in a single experiment. The outcome of a single coin flip. Binomial Distribution It models the number of successes in a fixed number of trials n with p probability. The number of times it comes heads when you toss a coin 10 times. Poisson Distribution It predicts the k number of events occurring in a fixed interval of time or space. The number of emails messages received per day. Geometric Distribution It represents the number of trials needed to achieve the first success in a sequence of trials. The number of times a coin is flipped until it lands on heads. Hypergeometric Distribution It calculates the probability of drawing a specific number of successes from a finite population. The number of red balls drawn from a bag of mixed colored balls. Continuous Probability Distributions As the name implies, continuous probability distributions are mathematical functions that describe the probabilities of different occurrences within a continuous range of values. Continuous probability distribution includes an infinite number of possible values. For example, in the interval [4, 5] there are infinite values between 4 and 5. Let’s see some common continuous probability distributions − Continuous Probability Distribution Explanation Example Continuous Uniform Distribution It assigns equal probability to all values within equal-sized interval. The height of a person between 5 to 6 feet. Normal (Gaussian) Distribution It forms a bell-shaped curve and describes the data clustered around the mean and symmetrical tails. IQ scores Exponential Distribution It models the time between events in a Poisson process, where events occur at a constant rate. The time until the next customer arrives. Log-normal Distribution It represents the right-skewed data when plotted on a logarithmic scale. Stock prices, income distributions, etc. Beta Distribution It describes the random variables constrained to a finite interval. It is often used in Bayesian statistics. The probability of success in a binomial trial. Use of Probability Distributions in Generative Modeling Probability distributions play a crucial role in generative modeling. Let’s check out some of the important ways in which probability distributions are used in generative modeling − Data Distribution − Generative Models aim to capture the underlying probability distribution of data from which the samples are taken. Generating New Samples − Once understanding the data distribution is done, generative models can generate new data comparable to the original dataset. Evaluation and Training − Probability distributions are used to evaluate and train generative models. Evaluation metrics such as likelihood, perplexity, and Wasserstein distance are used to evaluate the quality of generated samples compared to the original dataset. Variability and Uncertainty − Probability distributions are used to find the variability and uncertainty present in the data. Generative models can use this information to generate distinct and realistic samples. Applications of Probability Distribution There is a wide range of generative modeling tasks across various domains that use probability distributions, some of which are listed below − Image Generation − Generative models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) use probability distributions to generate realistic images from scratch. This has applications in computer graphics, creative design, and content generation. Text Synthesis − Language models, such as OpenAI”s ChatGPT, use probability distributions to generate relevant text output based on a given prompt or input. This has applications in chatbots, virtual assistants, and automated content generation systems. Anomaly Detection − Generative model, by learning the underlying probability distribution of normal data, can be used for anomaly detection and outlier identification in datasets. This has applications in fraud detection, network security, and medical diagnostics. Conclusion In this chapter, we explained the critical role of probability distribution in generative modeling. We first covered what probability distributions are

Generative AI Models – Maximum Likelihood Estimation Maximum Likelihood Estimation (MLE) is a statistical method providing a principal approach to estimate the parameters of a probability distribution that best describes a given dataset. MLE assumes that the specified distribution generates the data. In simple terms, MLE is a method used to find out the most likely values for the unknown parameters of a model, such as the average or spread of a set of data points. It is something like we guess the missing numbers in a sequence so that it fits the pattern of numbers we already know. In the field of generative AI, especially in generative models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), MLE finds extensive applications. For example, while generating images of handwritten digits (0-9), we want our model to generate images that resemble the ones in our dataset (like MNIST). We can achieve this by maximizing the likelihood of observing our training data given the parameters of the model. maximize Σ log P(x | θ) We will cover this later in detail when we create our first GAN model using python programming language. Read this chapter to understand the concept of Maximum Likelihood Estimation, its significant role in generative modeling, applications of MLE in generative modeling, and its Python implementation. Understanding Maximum Likelihood Estimation (MLE) Maximum Likelihood Estimation (MLE) is a powerful statistical method used to estimate the parameters of probability distributions based on observed data. Let’s understand it in more detail with the help of its mathematical foundations − Mathematical Foundation of MLE At the heart of MLE lies the likelihood function: $\mathrm{L(\theta | x)}$. Here, $\mathrm{\theta}$ represents the parameters of the distribution, and x denotes the observed data. The likelihood function quantifies the probability of observing the data given specific parameter values. Mathematically, it is expressed as the joint probability density function (PDF) or probability mass function (PMF) of the observed data. $$\mathrm{L(\theta | x) \: = \: f(x | \theta)}$$ To keep the computation simple, we usually work with the log-likelihood function $\mathrm{l(\theta | x)}$, which is the natural logarithm of the likelihood function − $$\mathrm{l(\theta | x) \: = \: \log L(\theta | x)}$$ Actually, the goal of MLE is to find the parameter values $\mathrm{\hat{\theta}}$ that maximize the likelihood function $\mathrm{L(\theta | x)}$ or equivalently, the log- likelihood function $\mathrm{l(\theta | x)}$ − $$\mathrm{\hat{\theta} \: = \: argmax_{\theta} L(\theta | x)}$$ Or, $$\mathrm{\hat{\theta} \: = \: argmax_{\theta} l(\theta | x)}$$ Now, to obtain the maximum likelihood estimates $\mathrm{\hat{\theta}}$, we differentiate the log- likelihood function $\mathrm{l(\theta | x)}$ with respect to the parameters $\mathrm{\theta}$ and set the derivatives equal to zero − $$\mathrm{\frac{\partial \: l(\theta | x)}{\partial \: \theta} \: = \: 0}$$ Solving the above equation gives the MLE $\mathrm{\hat{\theta}}$. MLE in Generative Modeling Generative modeling, as we discussed earlier, involves capturing the underlying distribution of data and generates new data comparable to the original training data. In training generative models, MLE plays a crucial role by estimating the parameters of the underlying probability distribution. Let’s see how MLE is applied in generative modeling − Model Selection We first need to choose a probabilistic model that captures the underlying data distribution. Some of the common models are Gaussian distributions, mixture models, neural networks, etc. Likelihood Function Next we need to define the likelihood function. This likelihood function measures the probability of observing the given data. For example, for a given dataset $\mathrm{D \: = \: \lbrace x_{1},x_{2},x_{3},\: \dots \: x_{n} \rbrace}$, the likelihood function $\mathrm{L(\theta | D)}$ depends on the model parameter $\mathrm{\theta}$ and is given by the product of the probabilities of observing each data point − $$\mathrm{L(\theta | D) \: = \: \prod_{i=1}^N p(x_{i} | \theta)}$$ Maximization Now we need to maximize the likelihood function with respect to the model parameters $\mathrm{\theta}$. Maximization involves finding the values of $\mathrm{\theta}$ that make the observed data most likely under the model. Parameter Estimation Finally, when the likelihood function is maximized, the resulting parameter values are used as the estimates for the parameters of the generative model. These estimated parameters define the learned distribution, which can then be used to generate new data points comparable to the observed data. Applications of MLE in Generative Modeling MLE is having wide-range applications across various domains of generative modeling. Given below are some of the significant applications − Gaussian Mixture Models (GMMs) − MLE is used to estimate the parameters of Gaussian components in GMMs. These parameters enable the modeling of complex data distributions with multiple modes. Variational Autoencoders (VAEs) − In VAEs, MLE is used to learn the parameters of the latent variable distribution. It allows the model to generate new data samples by sampling from this learned distribution. Generative Adversarial Networks (GANs) − The GANs do not directly optimize the likelihood function but MLE are used in the training of GANs to guide the learning process and improve sample quality. Implementing Maximum Likelihood Estimation using Python We can implement MLE using Python and visualize it using libraries like Matplotlib. Given below is a simple example to perform MLE to estimate the parameters of a Gaussian distribution from a given dataset − Example import numpy as np import matplotlib.pyplot as plt from scipy.stats import norm # Sample dataset (you can replace this with your own data) data = np.random.normal(loc=2, scale=1, size=2000) # Maximum Likelihood Estimation for a Gaussian distribution def maximum_likelihood_estimation(data): # Calculate the mean and standard deviation of the data mu = np.mean(data) sigma = np.std(data) return mu, sigma # Perform Maximum Likelihood Estimation estimated_mu, estimated_sigma = maximum_likelihood_estimation(data) # Generate x values for plotting x = np.linspace(min(data), max(data), 1000) # Plot histogram of the data plt.figure(figsize=(7.2, 5.5)) plt.hist(data, bins=30, density=True, alpha=0.6, color=”blue”, label=”Data Histogram”) # Plot the true Gaussian distribution plt.plot(x, norm.pdf(x, loc=2, scale=1), color=”red”, linestyle=”–”, label=”True Gaussian Distribution”) # Plot the estimated Gaussian distribution using MLE plt.plot(x, norm.pdf(x, loc=estimated_mu, scale=estimated_sigma), color=”green”, linestyle=”-”, label=”Estimated Gaussian Distribution (MLE)”) plt.xlabel(”Value”) plt.ylabel(”Probability Density”) plt.title(”Maximum Likelihood Estimation for Gaussian Distribution”) plt.legend() plt.grid(True) plt.show()

Basics of Generative AI Generative AI (GenAI) is the latest subtype of AI that broadly describes models or algorithms. GenAI is reshaping the landscape of creativity and innovation. You all know about , but have you ever wondered how it chats just like humans do. All thanks to Generative AI. This smart technology serves as the brain of ChatGPT and enables it to generate responses like a real person. So, when you chat with ChatGPT, you are basically observing the power of Generative AI. In this chapter, we will have an overview of generative AI, how it has evolved, its use cases, benefits, and limitations. Difference Between Traditional AI and Generative AI Generative AI, as name suggests, generates new text, audio, video, or any other type of content. It learns patterns from existing training data and produces new and unique output that resembles real-world data. Although there is hierarchical relationship between AI and GenAI, they both have different objectives and approaches. The below given table describes the differences between traditional AI and generative AI − Traditional AI Generative AI AI is used to create intelligent systems that can perform those tasks which generally require human intelligence. It generates new text, audio, video, or any other type of content by learning patterns from existing training data. The purpose of AI algorithms or models are to mimic human intelligence across wide range of applications. The purpose of generative AI algorithms or models is to generate new data having similar characteristics as data from the original dataset. Use Cases of Generative AI Content Generation − It can create articles, blogs, and other kinds of content. Chatbots and Virtual Assistants − You can implement chatbots for customer support and service with generative AI. Deepfake Creation − Generative AI can be used for mimicking individuals by applying deepfakes. Text Summarization − You can use it to create concise and readable summaries of complex reports or any other kind of communications. Code Generation − Its capabilities can be used for writing, understanding, and debugging of any code. Unique Art Creation − You can use generative AI to create photorealistic art in specific styles. Music Composition − You can compose music with specific styles or tones using generative AI. Benefits of Generative AI We have seen above that generative AI can be used in many areas of business. Let’s see some of the benefits of implementing generative AI − It automates the process of writing content. It reduces the effort of replying to emails. It improves the response to technical queries. It simplifies the content creation process. It creates a realistic representation of individuals. Limitations of Generative AI Below are some limitations to consider during the use of generative AI applications − Lack of Source Verification − It does not always verify the source of content. Ethical Concerns − There can be misuse of generated AI content such as deepfakes. Coherence Overemphasis − It may generate biased outputs because of overemphasis on coherence. Creativity Issues − Sometimes it is difficult for generative AI to generate truly creative outputs. Fine-tuning Complexity − It may be challenging to fine-tune the models for specific tasks. Dependency on Training Data − The output is dependent on the quality of training data.