Category: requests

Requests – Handling GET Requests ”; Previous Next This chapter will concentrate more on the GET requests, which is the most common and used very often. The working of GET in the requests module is very easy. Here is a simple example about working with the URL using the GET method. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.content) getdata.content, will print all the data available in the response. Output E:prequests>python makeRequest.py b”[n {n “id”: 1,n “name”: “Leanne Graham”,n “username”: “Bret”,n “email”: “[email protected]”,n “address”: {n “street”: “Kulas Light “,n “suite”: “Apt. 556”,n “city”: “Gwenborough”,n “zipcode”: “ 92998-3874”,n “geo”: {n “lat”: “-37.3159”,n “lng”: “81.149 6″n }n },n “phone”: “1-770-736-8031 x56442”,n “website”: “hildegard.org”,n “company”: {n “name”: “Romaguera-Crona”,n “catchPhrase”: “Multi-layered client-server neural-net”,n “bs”: “harness real-time e-markets”n }n } You can also pass parameters to the get method using the param attribute as shown below − import requests payload = {”id”: 9, ”username”: ”Delphine”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, params=payload) print(getdata.content) The details are stored in the object payload in the key/value pair and passed to params, inside get() method. Output E:prequests>python makeRequest.py b”[n {n “id”: 9,n “name”: “Glenna Reichert”,n “username”: “Delphine”, n “email”: “[email protected]”,n “address”: {n “street”: “Dayna Park”,n “suite”: “Suite 449”,n “city”: “Bartholomebury”,n “zipcode”: “76495-3109”,n “geo”: {n “lat”: “24.6463”,n “lng”: “-168.8889″n }n },n “phone”: “(775)976-6794 x41206”,n “ website”: “conrad.com”,n “company”: {n “name”: “Yost and Sons”,n “catchPhrase”: “Switchable contextually-based project”,n “bs”: “aggregate real-time technologies”n }n }n]” Print Page Previous Next Advertisements ”;

Requests – HTTP Requests Headers ”; Previous Next In the previous chapter, we have seen how to make the request and get the response. This chapter will explore a little more on the header section of the URL. So, we are going to look into the following − Understanding Request Headers Custom Headers Response Headers Understanding Request Headers Hit any URL in the browser, inspect it and check in developer tool network tab. You will get response headers, request headers, payload, etc. For example, consider the following URL − https://jsonplaceholder.typicode.com/users You can get the header details as follows − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, stream = True) print(getdata.headers) Output E:prequests>python makeRequest.py {”Date”: ”Sat, 30 Nov 2019 05:15:00 GMT”, ”Content-Type”: ”application/json; charset=utf-8”, ”Transfer-Encoding”: ”chunked”, ”Connection”: ”keep-alive”, ”Set-Cookie”: ”__cfduid=d2b84ccf43c40e18b95122b0b49f5cf091575090900; expires=Mon, 30-De c-19 05:15:00 GMT; path=/; domain=.typicode.com; HttpOnly”, ”X-Powered-By”: ”Express”, ”Vary”: ”Origin, Accept-Encoding”, ”Access-Control-Allow-Credentials”: ”t rue”, ”Cache-Control”: ”max-age=14400”, ”Pragma”: ”no-cache”, ”Expires”: ”-1”, ” X-Content-Type-Options”: ”nosniff”, ”Etag”: ”W/”160d-1eMSsxeJRfnVLRBmYJSbCiJZ1qQ “”, ”Content-Encoding”: ”gzip”, ”Via”: ”1.1 vegur”, ”CF-Cache-Status”: ”HIT”, ”Age”: ”2271”, ”Expect-CT”: ”max-age=604800, report-uri=”https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct””, ”Server”: ”cloudflare”, ”CF-RAY”: ”53da574f f99fc331-SIN”} To read any http header you can do so as follows − getdata.headers[“Content-Encoding”] // gzip Custom Headers You can also send headers to the URL being called as shown below. Example import requests headers = {”x-user”: ”test123”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, headers=headers) The headers passed has to be string, bytestring, or Unicode format. The behavior of the request will not change as per the custom headers passed. Response Headers The response headers look like below when you check the URL in the browser developer tool, network tab − To get the details of the headers from the requests module use. Response.headers are as shown below − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.headers) Output E:prequests>python makeRequest.py {”Date”: ”Sat, 30 Nov 2019 06:08:10 GMT”, ”Content-Type”: ”application/json; charset=utf-8”, ”Transfer-Encoding”: ”chunked”, ”Connection”: ”keep-alive”, ”Set-Cookie”: ”__cfduid=de1158f1a5116f3754c2c353055694e0d1575094090; expires=Mon, 30-Dec-19 06:08:10 GMT; path=/; domain=.typicode.com; HttpOnly”, ”X-Powered-By”: ”Express”, ”Vary”: ”Origin, Accept-Encoding”, ”Access-Control-Allow-Credentials”: ”t rue”, ”Cache-Control”: ”max-age=14400”, ”Pragma”: ”no-cache”, ”Expires”: ”-1”, ” X-Content-Type-Options”: ”nosniff”, ”Etag”: ”W/”160d-1eMSsxeJRfnVLRBmYJSbCiJZ1qQ “”, ”Content-Encoding”: ”gzip”, ”Via”: ”1.1 vegur”, ”CF-Cache-Status”: ”HIT”, ”Age”: ”5461”, ”Expect-CT”: ”max-age=604800, report-uri=”https://report-uri.cloudf lare.com/cdn-cgi/beacon/expect-ct””, ”Server”: ”cloudflare”, ”CF-RAY”: ”53daa52f 3b7ec395-SIN”} You can get any specific header you want as follows − print(getdata.headers[“Expect-CT”]) Output max-age=604800, report-uri=”https://report-uri.cloudflare.com/cdn-cgi/beacon/exp ect-ct You can also get the header details by using the get() method. print(getdata.headers.get(“Expect-CT”)) Output max-age=604800, report-uri=”https://report-uri.cloudflare.com/cdn-cgi/beacon/exp ect-ct Print Page Previous Next Advertisements ”;

Requests – Working with Requests ”; Previous Next In this chapter, we will understand how to work with the requests module. We will look into the following − Making HTTP Requests. Passing Parameters to HTTP Requests. Making HTTP Requests To make a Http request, we need to first import the request module as shown below − import requests Let us now see, how to make a call to URL using the requests module. Let us use the URL − https://jsonplaceholder.typicode.com/users in the code, to test Requests Module. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.status_code) The url − https://jsonplaceholder.typicode.com/users is called using requests.get() method. The response object of the URL is stored in the getdata variable. When we print the variable, it gives the 200 response code, which means that we have got the response successfully. Output E:prequests>python makeRequest.py <Response [200]> To get the content from the response, we can do so using getdata.content as shown below − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.content) getdata.content, will print all the data available in the response. Output E:prequests>python makeRequest.py b”[n {n “id”: 1,n “name”: “Leanne Graham”,n “username”: “Bret”,n “email”: “[email protected]”,n “address”: {n “street”: “Kulas Light “,n “suite”: “Apt. 556”,n “city”: “Gwenborough”,n “zipcode”: “ 92998-3874”,n “geo”: {n “lat”: “-37.3159”,n “lng”: “81.149 6″n }n },n “phone”: “1-770-736-8031 x56442”,n “website”: “hild egard.org”,n “company”: {n “name”: “Romaguera-Crona”,n “catchPhr ase”: “Multi-layered client-server neural-net”,n “bs”: “harness real-time e-markets”n }n } Passing Parameters to HTTP Requests Just requesting the URL is not sufficient, we also need to pass parameters to the URL. The params are mostly passed as key/value pair, for example − https://jsonplaceholder.typicode.com/users?id=9&username=Delphine So, we have id = 9 and username = Delphine. Now, will see how to pass such data to requests Http module. Example import requests payload = {”id”: 9, ”username”: ”Delphine”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, params = payload) print(getdata.content) The details are stored in the object payload in the key/value pair and passed to params, inside get() method. Output E:prequests>python makeRequest.py b”[n {n “id”: 9,n “name”: “Glenna Reichert”,n “username”: “Delphin e”,n “email”: “[email protected]”,n “address”: {n “street”: “Dayna Park”,n “suite”: “Suite 449”,n “city”: “Bartholomebury”,n “zipcode”: “76495-3109”,n “geo”: {n “lat”: “24.6463”,n “lng”: “-168.8889″n }n },n “phone”: “(775)976-6794 x41206”,n “ website”: “conrad.com”,n “company”: {n “name”: “Yost and Sons”,n “catchPhrase”: “Switchable contextually-based project”,n “bs”: “aggregate real-time technologies”n }n }n]” We are now getting the details of the id = 9 and username = Delphine details in the response. If you want to see, how the URL looks after passing the parameters, making use of the response object to the URL. Example import requests payload = {”id”: 9, ”username”: ”Delphine”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, params = payload) print(getdata.url) Output E:prequests>python makeRequest.py https://jsonplaceholder.typicode.com/users?id=9&username=Delphine Print Page Previous Next Advertisements ”;

Requests – Working with Cookies ”; Previous Next This chapter will discuss how to deal with cookies. You can get the cookies as well as send your cookies while calling the URL using the requests library. The url, https://jsonplaceholder.typicode.com/users when hits in the browser we can get the details of the cookies as shown below − You can read the cookies as shown below − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.cookies[“__cfduid”]) Output E:prequests>python makeRequest.py d1733467caa1e3431fb7f768fa79ed3741575094848 You can also send cookies when we make a request. Example import requests cookies = dict(test=”test123”) getdata = requests.get(”https://httpbin.org/cookies”,cookies=cookies) print(getdata.text) Output E:prequests>python makeRequest.py { “cookies”: { “test”: “test123″ } } Print Page Previous Next Advertisements ”;

Requests – Working with Errors ”; Previous Next This chapter will discuss how to deal with errors coming down when working with the Http request library. It is always a good practice to have errors managed for all possible cases. Error Exception The requests module gives the following types of error exception − ConnectionError − This will be raised, if there is any connection error. For example, the network failed, DNS error so the Request library will raise ConnectionError exception. Response.raise_for_status() − Based on status code i.e. 401, 404 it will raise HTTPError for the url requested. HTTPError − This error will be raised for an invalid response coming down for the request made. Timeout − Errors raised for a timeout for the URL requested. TooManyRedirects − If the limit is crossed for maximum redirections than it will raise TooManyRedirects error. Example Here is an example of errors shown for timeout − import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”,timeout=0.001) print(getdata.text) Output raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=”jsonplaceholder.ty picode.com”, port=443): Max retries exceeded with url: /users (Caused by Connect TimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000000B02AD E76A0>, ”Connection to jsonplaceholder.typicode.com timed out. (connect timeout = 0.001)”)) Print Page Previous Next Advertisements ”;

Requests – Event Hooks ”; Previous Next We can add events to the URL requested using event hooks. In the example below, we are going to add a callback function that will get called when the response is available. Example To add the callback, we need to make use of hooks param as shown in the example below − import requests def printData(r, *args, **kwargs): print(r.url) print(r.text) getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, hooks={”response”: printData}) Output E:prequests>python makeRequest.py https://jsonplaceholder.typicode.com/users [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] You can also call multiple callback functions as shown below − Example import requests def printRequestedUrl(r, *args, **kwargs): print(r.url) def printData(r, *args, **kwargs): print(r.text) getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, hooks = {”response”: [printRequestedUrl, printData]}) Output E:prequests>python makeRequest.py https://jsonplaceholder.typicode.com/users [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] You can also add the hook to the Session created as shown below − Example import requests def printData(r, *args, **kwargs): print(r.text) s = requests.Session() s.hooks[”response”].append(printData) s.get(”https://jsonplaceholder.typicode.com/users”) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] Print Page Previous Next Advertisements ”;

Requests – Quick Guide ”; Previous Next Requests – Overview Requests is a HTTP library that provides easy functionality to deal with http request/response in your web application. The library is developed in python. The official website of Python Requests which is available at https://2.python-requests.org/en/master/ defines Requests as follows − Requests is an elegant and simple HTTP library for Python, built for human beings. Features of Requests The features of Requests are discussed below − Request The python requests library has easy to use methods available to handle Http request. Passing of parameters and handling the request type like GET, POST, PUT, DELETE, etc. is very easy. Response You can get the response in the format you need and the supported ones are text format, binary response, json response, and raw response. Headers The library allows you to read, update or send new headers as per your requirements. Timeouts Timeouts can be easily added to the URL you are requesting using python requests library. It so happens that you are using a third-party URL and waiting for a response. It is always a good practice to give a timeout on the URL as we might want the URL to respond within that timeout with a response or an error that is coming because of timeout. Not doing so can cause either to wait on that request indefinitely. Error handling The requests module gives support for error handling and some of which are Connection Error, Timeout errors, TooManyRedirects, Response.raise_for_status errors, etc. Cookies The library allows you to read, write and update for the requested URL. Sessions To maintain the data, you require between requests you need sessions. So, if the same host is called again and again you can re-use the TCP connection which in turn will improve the performance. SSL certificates SSL certificate is a security feature that comes with secure urls. When you use Requests, it also verifies SSL certificates for the https URL given. SSL Verification is enabled by default in the requests library and will throw an error if the certificate is not present. Authentication HTTP authentication is on the server-side asking for some authentication information like username, password when the client requests a URL. This is an additional security for the request and the response being exchanged between the client and the server. Advantages of using Python Requests Library Following are the advantages of using Python Requests Library − Easy to use and fetch the data from the URL given. Requests library can be used to scrape the data from the website. Using requests, you can get, post, delete, update the data for the URL given. The handling of cookies and session is very easy. The security is also taken care of the help of authentication module support. Requests – Environment Setup In this chapter, we will work on the installation of Requests. To start working with the Requests module, we need to install Python first. So we are going to work on following− Install Python Install Requests Installing Python Go to the Python official site: https://www.python.org/downloads/ as shown below and click on the latest version available for Windows, Linux/Unix, and Mac OS. Download Python as per your 64 or 32 bit OS available with you. Once you have downloaded, click on the .exe file and follow the steps to install python on your system. The python package manager, i.e., pip will also get installed by default with the above installation. To make it work globally on your system, directly add the location of python to the PATH variable. The same is shown at the start of the installation to remember to check the checkbox which says ADD to PATH. In case you forget to check it, please follow the below-given steps to add to PATH. To add to PATH follow the steps− Right-click on your Computer icon and click on properties > Advanced System Settings. It will display the screen as shown below − Click on Environment Variables as shown above. It will display the screen as shown below − Select Path and click on Edit button, add the location path of your python at the end. Now, let us check the python version. Checking the python version E:prequests>python –version Python 3.7.3 Install Requests Now that we have python installed, we are going to install Requests. Once python is installed, python package manager i.e. pip will also get installed. Following is the command to check pip version. E:prequests>pip –version pip 19.1.1 from c:usersxxxxxappdatalocalprogramspythonpython37libsite-p ackagespip (python 3.7) We have pip installed and the version is 19.1.1. Now, will use pip to install Requests module. The command is given below− pip install requests E:prequests>pip install requests Requirement already satisfied: requests in c:usersxxxxappdatalocalprograms pythonpython37libsite-packages (2.22.0) Requirement already satisfied: certifi>=2017.4.17 in c:userskamatappdataloca lprogramspythonpython37libsite-packages (from requests) (2019.3.9) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxxxxxappdatalocalprogramspythonpython37libsite-packages (from requests ) (1.25.3) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxxxxxxxappdatalocalpr ogramspythonpython37libsite-packages (from requests) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxxxxxappdatal ocalprogramspythonpython37libsite-packages (from requests) (3.0.4) We already have the module installed, so in the command prompt it says Requirement already satisfied; if not installed it would have downloaded the required packages for installation. To check the details of the requests module installed, you can use the following command− pip show requests E:prequests>pip show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: [email protected] License: Apache 2.0 Location: c:usersxxxxxappdatalocalprogramspythonpython37libsite-package S Requires: certifi, idna, urllib3, chardet Required-by: The version of Requests module is 2.22.0. Requests – How Http Requests Work? Python’s Requests is a HTTP library that will help us exchange data between the client and the server. Consider you have a UI with a form, wherein you need to enter the user details, so once you enter it, you have to submit the data which is nothing but a Http POST or PUT request from the client to server to save the data. When you want the data, you need to fetch it from the server, which

Requests – Environment Setup ”; Previous Next In this chapter, we will work on the installation of Requests. To start working with the Requests module, we need to install Python first. So we are going to work on following − Install Python Install Requests Installing Python Go to the Python official site: https://www.python.org/downloads/ as shown below and click on the latest version available for Windows, Linux/Unix, and Mac OS. Download Python as per your 64 or 32 bit OS available with you. Once you have downloaded, click on the .exe file and follow the steps to install python on your system. The python package manager, i.e., pip will also get installed by default with the above installation. To make it work globally on your system, directly add the location of python to the PATH variable. The same is shown at the start of the installation to remember to check the checkbox which says ADD to PATH. In case you forget to check it, please follow the below-given steps to add to PATH. To add to PATH follow the steps − Right-click on your Computer icon and click on properties > Advanced System Settings. It will display the screen as shown below − Click on Environment Variables as shown above. It will display the screen as shown below − Select Path and click on Edit button, add the location path of your python at the end. Now, let us check the python version. Checking the python version E:prequests>python –version Python 3.7.3 Install Requests Now that we have python installed, we are going to install Requests. Once python is installed, python package manager i.e. pip will also get installed. Following is the command to check pip version. E:prequests>pip –version pip 19.1.1 from c:usersxxxxxappdatalocalprogramspythonpython37libsite- packagespip (python 3.7) We have pip installed and the version is 19.1.1. Now, will use pip to install Requests module. The command is given below − pip install requests E:prequests>pip install requests Requirement already satisfied: requests in c:usersxxxxappdatalocalprograms pythonpython37libsite-packages (2.22.0) Requirement already satisfied: certifi>=2017.4.17 in c:userskamatappdatalocal programspythonpython37libsite-packages (from requests) (2019.3.9) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxxxxxappdatalocalprogramspythonpython37libsite-packages (from requests ) (1.25.3) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxxxxxxxappdatalocal programspythonpython37libsite-packages (from requests) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxxxxxappdata localprogramspythonpython37libsite-packages (from requests) (3.0.4) We already have the module installed, so in the command prompt it says Requirement already satisfied; if not installed it would have downloaded the required packages for installation. To check the details of the requests module installed, you can use the following command − pip show requests E:prequests>pip show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: [email protected] License: Apache 2.0 Location: c:usersxxxxxappdatalocalprogramspythonpython37libsite-package S Requires: certifi, idna, urllib3, chardet Required-by: The version of Requests module is 2.22.0. Print Page Previous Next Advertisements ”;

Requests – Handling Sessions ”; Previous Next To maintain the data between requests you need sessions. So, if the same host is called again and again, you can reuse the TCP connection which in turn will improve the performance. Let us now see, how to maintain cookies across requests made using sessions. Adding cookies using session import requests req = requests.Session() cookies = dict(test=”test123”) getdata = req.get(”https://httpbin.org/cookies”,cookies=cookies) print(getdata.text) Output E:prequests>python makeRequest.py { “cookies”: { “test”: “test123″ } } Using session, you can preserve the cookies data across requests. It is also possible to pass headers data using the session as shown below − Example import requests req = requests.Session() req.headers.update({”x-user1”: ”ABC”}) headers = {”x-user2”: ”XYZ”} getdata = req.get(”https://httpbin.org/headers”, headers=headers) print(getdata.headers) Print Page Previous Next Advertisements ”;