Category: saltstack

SaltStack – Salt for Cloud Infrastructure ”; Previous Next Salt provides a separate module, Salt Cloud to provide virtual machines of various cloud providers like Amazon AWS, Google Compute, etc. Salt Cloud is a generic interface to configure and manage VMs of various cloud providers. Cloud Config − The main configuration file for Salt Cloud is /etc/salt/cloud and it applies to all the VMs. The main configuration file is called Cloud Config. Cloud Providers − To target specific cloud providers, we can use specific configuration file and they are located in /etc/salt/cloud.providers.d/*.conf, these are called as Cloud Providers. Cloud Profiles − To target a specific VM, we can also use a special configuration file, which applies to that particular VM only and they are located at /etc/salt/cloud.profiles.d/*.conf, these are called as Cloud Profiles. For a virtual machine, settings in the Cloud Config is applied first and then in the Cloud Providers and finally override is done by the Cloud Profiles. Installation of Salt Cloud By default, Salt Cloud is built into the Salt and is readily available. If it is not available, we can install it by using the command below. pip install salt-cloud Since Salt Cloud is a separate module and runs in its own process, it can be installed in the Salt Minion system as well instead of the Salt Master. Provisioning a Virtual Machine To provision a virtual machine, we need to define a cloud provider and a cloud profile. Once both are created, we can provision a new virtual machine for using it. Cloud Provider Cloud host information are configured in the Cloud provider configuration file. Normally, the basic information, which needs to be configured are cloud driver, username, password, private key, etc. Let us create a new cloud provider named as my-amazon-cloud. Create a file, my-amazon-cloud.conf under /etc/salt/cloud.providers.d/ Add a new provider using the ec2 driver. my-amazon-cloud: driver: ec2 id: ”<AWS_ID>” key: ”<AWS_KEY>” private_key: /path/to/privatekey.pem keyname: <name of the key> securitygroup: default minion: master: <master server> Salt provides drivers for various cloud host out of the box, such as GoGrid, HP Cloud, Google Compute Engine (GCE), Amazon AWS, Joyent, Linode, OpenNebula, ProfitBricks, Proxmox, Saltify, VexxHost, VMWare, etc. Once the cloud providers are configured, we can query the available location of the provider, available machine images, and its various sizes. salt-cloud –list-location my-amazon-cloud salt-cloud –list-images my-amazon-cloud salt-cloud –list-sizes my-amazon-cloud Cloud Profile A Cloud profile specifies the virtual machine image and size. It can be configured under – /etc/salt/cloud.profiles.d/. Let us create a simple profile, simple.conf. aws_micro: provider: my-amazon-cloud image: <image_id> size: <machine_id e.g. t1.micro> Virtual Machine Once the provider and profiles are configured, we can easily provide a virtual machine using the salt-cloud as shown below. salt-cloud -p aws_micro master minion1 minion2 Where, p – Profile name master, minion1 and minion2 are the new virtual machines. The details of the newly created virtual machine can be obtained by using the following command. salt-cloud –query The virtual machines can be destroyed using the following command − slat-cloud -d master minion1 Cloud Map A Cloud Map is a special format to create multiple virtual machines at once. The format of the map file is to specify the profile and then add a list of the virtual machines under it. A sample map file is as follows − micro: – web1 – web2 large: – db1 – db2 The map file can be passed as an argument to the salt-cloud command to create the virtual machine as follows − salt-cloud -m /path/to/mapfile Print Page Previous Next Advertisements ”;

SaltStack – Overview ”; Previous Next In this chapter, we will learn the basics of SaltStack. SaltStack’s remote execution capabilities allow administrators to run commands on various machines in parallel with a flexible targeting system. Salt configuration management establishes a master-minion model to quickly, very easily, flexibly and securely bringing infrastructure components in line with a given policy. What is SaltStack? Salt is a very powerful automation framework. Salt architecture is based on the idea of executing commands remotely. All networking is designed around some aspect of remote execution. This could be as simple as asking a Remote Web Server to display a static Web page, or as complex as using a shell session to interactively issue commands against a remote server. Salt is an example of one of the more complex types of remote execution. Salt is designed to allow users to explicitly target and issue commands to multiple machines directly. Salt is based around the idea of a Master, which controls one or more Minions. Commands are normally issued from the Master to a target group of Minions, which then execute the tasks specified in the commands and then return the resulting data back to the Master. Communications between a master and minions occur over the ZeroMQ message bus. SaltStack modules communicate with the supported minion operating systems. The Salt Master runs on Linux by default, but any operating system can be a minion, and currently Windows, VMware vSphere and BSD Unix variants are well supported. The Salt Master and the minions use keys to communicate. When a minion connects to a master for the first time, it automatically stores keys on the master. SaltStack also offers Salt SSH, which provides an “agent less” systems management. Need for SaltStack SaltStack is built for speed and scale. This is why it is used to manage large infrastructures with tens of thousands of servers at LinkedIn, WikiMedia and Google. Imagine that you have multiple servers and want to do things to those servers. You would need to login to each one and do those things one at a time on each one and then you might want to do complicated things like installing software and then configuring that software based on some specific criteria. Let us assume you have ten or maybe even 100 servers. Imagine logging in one at a time to each server individually, issuing the same commands on those 100 machines and then editing the configuration files on all 100 machines becomes very tedious task. To overcome those issues, you would love to update all your servers at once, just by typing one single command. SaltStack provides you exactly the solution for all such problems. Features of SaltStack SaltStack is an open-source configuration management software and remote execution engine. Salt is a command-line tool. While written in Python, SaltStack configuration management is language agnostic and simple. Salt platform uses the push model for executing commands via the SSH protocol. The default configuration system is YAML and Jinja templates. Salt is primarily competing with Puppet, Chef and Ansible. Salt provides many features when compared to other competing tools. Some of these important features are listed below. Fault tolerance − Salt minions can connect to multiple masters at one time by configuring the master configuration parameter as a YAML list of all the available masters. Any master can direct commands to the Salt infrastructure. Flexible − The entire management approach of Salt is very flexible. It can be implemented to follow the most popular systems management models such as Agent and Server, Agent-only, Server-only or all of the above in the same environment. Scalable Configuration Management − SaltStack is designed to handle ten thousand minions per master. Parallel Execution model − Salt can enable commands to execute remote systems in a parallel manner. Python API − Salt provides a simple programming interface and it was designed to be modular and easily extensible, to make it easy to mold to diverse applications. Easy to Setup − Salt is easy to setup and provides a single remote execution architecture that can manage the diverse requirements of any number of servers. Language Agnostic − Salt state configuration files, templating engine or file type supports any type of language. Benefits of SaltStack Being simple as well as a feature-rich system, Salt provides many benefits and they can be summarized as below − Robust − Salt is powerful and robust configuration management framework and works around tens of thousands of systems. Authentication − Salt manages simple SSH key pairs for authentication. Secure − Salt manages secure data using an encrypted protocol. Fast − Salt is very fast, lightweight communication bus to provide the foundation for a remote execution engine. Virtual Machine Automation − The Salt Virt Cloud Controller capability is used for automation. Infrastructure as data, not code − Salt provides a simple deployment, model driven configuration management and command execution framework. Introduction to ZeroMQ Salt is based on the ZeroMQ library and it is an embeddable networking library. It is lightweight and a fast messaging library. The basic implementation is in C/C++ and native implementations for several languages including Java and .Net is available. ZeroMQ is a broker-less peer-peer message processing. ZeroMQ allows you to design a complex communication system easily. ZeroMQ comes with the following five basic patterns − Synchronous Request/Response − Used for sending a request and receiving subsequent replies for each one sent. Asynchronous Request/Response − Requestor initiates the conversation by sending a Request message and waits for a Response message. Provider waits for the incoming Request messages and replies with the Response messages. Publish/Subscribe − Used for distributing data from a single process (e.g. publisher) to multiple recipients (e.g. subscribers). Push/Pull − Used for distributing data to connected nodes. Exclusive Pair − Used for connecting two peers together, forming a pair. ZeroMQ is a highly flexible networking tool for exchanging messages among clusters, cloud and other multi system environments. ZeroMQ is the default transport library

SaltStack – Git as a File Server ”; Previous Next Git is an open-source distributed version control system. It can be used to keep a track of changes in any files. Salt sends files from Git repositories using the Git file server. You can configure Git to the fileserver_backend list option and if you need to configure one or more repositories, you can do so by using the gitfs_remotes option. This chapter explains about how to install and configure the Git file server. Before moving towards the installation, you must have the following prerequisites. Salt Server Prerequisites for Using Git The minimal requirement for the salt server to use Git as the file server is as follows − pygit2 Dulwich Pygit2 and Dulwich are configured using the gitfs_provider parameter in the master config file. If gitfs_provider is not configured in the master file then Salt will prefer pygit2, if the suitable version is available, followed by GitPython and Dulwich. Install pygit2 The following commands are used to install pygit2 in the Fedora and Ubuntu based systems − Fedora-based system yum install python-pygit2 Ubuntu-based system apt-get install python-pygit2 Here, the minimum supported version of pygit2 is 0.20.3. Install GitPYTHON GitPython can be easily installed on the master using the yum / apt command as shown below. Fedora-based system yum install GitPython Ubuntu-based system apt-get install python-git Install DULWICH Dulwich can be easily installed on the master using the yum command. Fedora-based system yum install python-dulwich Ubuntu-based system apt-get install python-dulwich Now, we have installed all the dependencies for the Git file server. Let us now configure this Git file server using the fileserver_backend section in the master config file. Backend Configuration In order to use the Git file server, you need to add Git in the fileserver_backend list in the master config file. It is described as follows − fileserver_backend: – git Let us further understand how to configure the Git file server in a remote configuration. gitfs_remotes Configuration You can specify any one or all of the URLs such as git://, https://, file://, or ssh:// for the gitfs_remotes configuration in the master file. This is used to search for the requested files. The simple https URL specification is defined below. gitfs_remotes: – https://github.com The ssh configuration can be done as shown below. gitfs_remotes: – [email protected]:user1/sample.git – ssh://[email protected]/path/to/sample.git Now, we have configured the Git file server using the two options fileserver_backend and gitfs_remotes. Restart Master After making all the changes in the master file, now restart the master to load all the configurations in the Git file server. Multiple Remotes Configuration The following command is used for multiple configuration in gitfs_remotes in the master file. gitfs_remotes: – git://github.com/sample/sample1.git – https://github.com/sample/sample2.git – file:///root/user/sample Here, the repositories sample1.git, sample2.git, and sample.doc may have the following files. sample1.git: top.sls sample2.git edit/vimrc sample.doc edit/vimrc Print Page Previous Next Advertisements ”;

SaltStack – Architecture ”; Previous Next The architecture of SaltStack is designed to work with any number of servers, from local network systems to other deployments across different data centers. Architecture is a simple server/client model with the needed functionality built into a single set of daemons. Take a look at the following illustration. It shows the different components of SaltStack architecture. SaltMaster − SaltMaster is the master daemon. A SaltMaster is used to send commands and configurations to the Salt slaves. A single master can manage multiple masters. SaltMinions − SaltMinion is the slave daemon. A Salt minion receives commands and configuration from the SaltMaster. Execution − Modules and Adhoc commands executed from the command line against one or more minions. It performs Real-time Monitoring. Formulas − Formulas are pre-written Salt States. They are as open-ended as Salt States themselves and can be used for tasks such as installing a package, configuring and starting a service, setting up users or permissions and many other common tasks. Grains − Grains is an interface that provides information specific to a minion. The information available through the grains interface is static. Grains get loaded when the Salt minion starts. This means that the information in grains is unchanging. Therefore, grains information could be about the running kernel or the operating system. It is case insensitive. Pillar − A pillar is an interface that generates and stores highly sensitive data specific to a particular minion, such as cryptographic keys and passwords. It stores data in a key/value pair and the data is managed in a similar way as the Salt State Tree. Top File − Matches Salt states and pillar data to Salt minions. Runners − It is a module located inside the SaltMaster and performs tasks such as job status, connection status, read data from external APIs, query connected salt minions and more. Returners − Returns data from Salt minions to another system. Reactor − It is responsible for triggering reactions when events occur in your SaltStack environment. SaltCloud − Salt Cloud provides a powerful interface to interact with cloud hosts. SaltSSH − Run Salt commands over SSH on systems without using Salt minion. In the next chapter, we will learn in detail about the various competitors of SaltStack and their features. Print Page Previous Next Advertisements ”;

SaltStack – Competitors ”; Previous Next Salt, Puppet, Chef, and Ansible are the leading configuration management and orchestration tools, each of which takes a different path to server automation. They were built to make it easier to configure and maintain dozens, hundreds or even thousands of servers. Let us understand how SaltStack competes primarily with Puppet, Chef, and Ansible. Platforms and Support Following is a list of all the platforms that support SaltStack and its competitors. SaltStack − SaltStack software runs on and manages many versions of Linux, Windows, Mac OS X and UNIX. Puppet − Red Hat Enterprise Linux, CentOS, Oracle Linux, Scientific Linux, SUSE Linux Enterprise Server and Ubuntu. Chef − Chef is supported on multiple platforms such as AIX, RHEL/CentOS, FreeBSD, OS X, Solaris, Microsoft Windows and Ubuntu. Ansible − Fedora distribution of Linux, CentOS, and Scientific Linux via Extra Packages for Enterprise Linux (EPEL) as well as for other operating systems. Origin Language SaltStack − Python Puppet − Ruby Chef − Ruby and its CLI uses ruby-based DSL Ansible − Python Supported Language SaltStack − Any language Puppet − Ruby Chef − Ruby and its CLI uses ruby-based DSL Ansible − Any language Web UI SaltStack − Web UI offers views of running jobs, minion status and event logs. Puppet − Web UI handles reporting, inventorying and real-time node management. Chef − Web UI lets you search and inventory nodes, view node activity and assign Cookbooks, roles and nodes. Ansible − Web UI lets you configure users, teams and inventories and apply Playbooks to inventories. Management Tools SaltStack − SaltStack Enterprise is positioned as the main tool for managing the orchestration of cloud and IT operations, as well as DevOps. Puppet − Puppet comes in two flavors, Puppet Enterprise and Open Source Puppet. In addition to providing functionalities of the Open Source Puppet, Puppet Enterprise also provides GUI, API and command line tools for node management. Chef − CFEngine is the configuration management tool. Ansible − Ansible 1.3 is the main tool for management. Performance SaltStack − Salt is designed for high-performance and scalability. Salt’s communication system establishes a persistent data pipe between the Salt master and minions using ZeroMQ. Puppet − Secure as well as high-performance and no agents required. Chef − The most apparent struggle for Chef Server is search; Search is slow and is not requested concurrently from clients. Ansible − Secure, high-performance and no agents required. Price and Value SaltStack − Free open source version. SaltStack Enterprise costs $150 per machine per year. Puppet − Free open source version. Puppet Enterprise costs $100 per machine per year. Chef − Free open source version; Enterprise Chef free for 5 machines, $120 per month for 20 machines, $300 per month for 50 machines. Ansible − Free open source version; Ansible free for 10 machines, then $100 or $250 per machine per year depending on the support you needed. Usage SaltStack − SaltStack is used by Cisco and Rackspace. It can integrate with any cloud-based platform. Puppet − Puppet is used by Zynga, Twitter, the New York Stock Exchange, PayPal, Disney, Google and so on. Chef − Chef can integrate with cloud-based platforms such as Internap, Amazon EC2, Google Cloud Platform, OpenStack, Microsoft Azure and Rackspace. Ansible − Ansible can deploy to virtualization environments, cloud environments including Amazon Web Services, Cloud Stack, DigitalOcean, and Google Cloud Platform and so on. Print Page Previous Next Advertisements ”;

SaltStack – Salt Package Manager ”; Previous Next Salt formulas are packaged and distributed to Salt masters using the package manager. This concept was influenced from RPM, Yum and Pacman packaging systems. Salt state, pillar, file templates and other files are used by the formula and then packaged into a single file. After a formula package is created, it is copied to the Repository System to make it to use for Salt masters. Before moving to the package manager, let us have a look at how to install an “nginx” package using the Salt basic command. The following syntax is used to install a “nginx” package. root@saltmaster:/home/vagrant# salt ”*” pkg.install nginx Here, the pkg.install nginx command is used to install a package. After execution, you will see the following response. It will produce the following output − minion2: ———- httpd: ———- new: 1 old: httpd-cgi: ———- new: 1 old: libxslt1.1: ———- new: 1.1.28-2build1 old: nginx: ———- new: 1.4.6-1ubuntu3.7 old: nginx-common: ———- new: 1.4.6-1ubuntu3.7 old: nginx-core: ———- new: 1.4.6-1ubuntu3.7 old: minion1: ———- httpd: ———- new: 1 old: httpd-cgi: ———- new: 1 old: libxslt1.1: ———- new: 1.1.28-2build1 old: nginx: ———- new: 1.4.6-1ubuntu3.7 old: nginx-common: ———- new: 1.4.6-1ubuntu3.7 old: nginx-core: ———- new: 1.4.6-1ubuntu3.7 old: Now, you have installed a package. To start the services for that package, use the command given below. root@saltmaster:/home/vagrant# salt ”*” service.start nginx After running this command, the result looks as shown in the code block below. minion1: True minion2: True Therefore, we have installed and started the services for the “nginx” package using the basic command. Let us now discuss how to build and install packages in the Salt package manager. Building Packages Packages can be built on any system where you can install Salt. There are three type of packages and they are follows. Formula Reactor Conf Let us now understand how to build packages using the Fomula File. Formula File Most files from the package are located at the – /srv/spm/salt/ directory by default, but the pillar file can be placed in the – /srv/spm/pillar/ directory. The Formula file describes the package. Example name: apache os: RedHat os_family: RedHat version: 201607 release: 2 summary: Formula for installing Apache description: Formula for installing Apache Here, Name − The name of the package. Here, the package name is apache. os − It is used to know which operating systems can support this package. os_family − It is used to know which operating system families can support this package. Version − The version of the package. It is specified in an YYYYMM format. Release − This field refers primarily to a release of a version. Summary − Short lines description of the package. Description − A more detailed description of the package. REACTOR The reactor files resides in the /srv/spm/reactor/ directory. CONF The files in this type of a package are configuration files for Salt, which normally live in the /etc/salt/ directory. Configuration files for packages other than Salt can and should be handled with a Salt State (using a formula type of package). Let us continue with the following steps to build a package. Create a FORMULA file and place it in the root of the package folder. Assemble the formula files in a folder on the build system. Run spm build. The package is built and placed in the /srv/spm_build folder. The following command is used to build a package. spm build /path/to/salt-packages-source/formula Now, copy the .spm file to a folder on the repository system. You can share the srv/spm_build folder on the network, or copy the files to your FTP or the Webserver. Generate repo metadata using the following command. spm create_repo /srv/spm_build Installing Packages This section explains about installing Salt package manager packages. Configure Remote Repositories To configure remote repositories, the Salt Master needs to know where the repository is through a configuration process. Files are in the /etc/salt/spm.repos.d/spm.repo directory. Example file_repository: url: https://spm.example.com/ Here, the file contains the name of the repository and the link to the repository. You can also use http, https, ftp, or the file path. To use file path, you can access it using the URL: file:///srv/spm_build. Update Metadata After the repository is configured on the Salt master, repository metadata is downloaded using the command below. spm update_repo Update File Roots The SPM packages are located in the srv/spm/salt directory. Add the following path to the file roots on the Salt master, file_roots: base: 1. /srv/salt 2. /srv/spm/salt Now, restart the salt master. Install Apache Package To install the package apache, use the following command. spm install apache You can also install directly from the SPM file using the command below. spm local install /srv/spm/apache-201607-1.spm Removing a Package To remove a package, for example – apache, use the following command, spm remove apache Note that, if any files have been modified, they will not be removed. Print Page Previous Next Advertisements ”;