Category: requests

Requests – Working with Requests ”; Previous Next In this chapter, we will understand how to work with the requests module. We will look into the following − Making HTTP Requests. Passing Parameters to HTTP Requests. Making HTTP Requests To make a Http request, we need to first import the request module as shown below − import requests Let us now see, how to make a call to URL using the requests module. Let us use the URL − https://jsonplaceholder.typicode.com/users in the code, to test Requests Module. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.status_code) The url − https://jsonplaceholder.typicode.com/users is called using requests.get() method. The response object of the URL is stored in the getdata variable. When we print the variable, it gives the 200 response code, which means that we have got the response successfully. Output E:prequests>python makeRequest.py <Response [200]> To get the content from the response, we can do so using getdata.content as shown below − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.content) getdata.content, will print all the data available in the response. Output E:prequests>python makeRequest.py b”[n {n “id”: 1,n “name”: “Leanne Graham”,n “username”: “Bret”,n “email”: “[email protected]”,n “address”: {n “street”: “Kulas Light “,n “suite”: “Apt. 556”,n “city”: “Gwenborough”,n “zipcode”: “ 92998-3874”,n “geo”: {n “lat”: “-37.3159”,n “lng”: “81.149 6″n }n },n “phone”: “1-770-736-8031 x56442”,n “website”: “hild egard.org”,n “company”: {n “name”: “Romaguera-Crona”,n “catchPhr ase”: “Multi-layered client-server neural-net”,n “bs”: “harness real-time e-markets”n }n } Passing Parameters to HTTP Requests Just requesting the URL is not sufficient, we also need to pass parameters to the URL. The params are mostly passed as key/value pair, for example − https://jsonplaceholder.typicode.com/users?id=9&username=Delphine So, we have id = 9 and username = Delphine. Now, will see how to pass such data to requests Http module. Example import requests payload = {”id”: 9, ”username”: ”Delphine”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, params = payload) print(getdata.content) The details are stored in the object payload in the key/value pair and passed to params, inside get() method. Output E:prequests>python makeRequest.py b”[n {n “id”: 9,n “name”: “Glenna Reichert”,n “username”: “Delphin e”,n “email”: “[email protected]”,n “address”: {n “street”: “Dayna Park”,n “suite”: “Suite 449”,n “city”: “Bartholomebury”,n “zipcode”: “76495-3109”,n “geo”: {n “lat”: “24.6463”,n “lng”: “-168.8889″n }n },n “phone”: “(775)976-6794 x41206”,n “ website”: “conrad.com”,n “company”: {n “name”: “Yost and Sons”,n “catchPhrase”: “Switchable contextually-based project”,n “bs”: “aggregate real-time technologies”n }n }n]” We are now getting the details of the id = 9 and username = Delphine details in the response. If you want to see, how the URL looks after passing the parameters, making use of the response object to the URL. Example import requests payload = {”id”: 9, ”username”: ”Delphine”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, params = payload) print(getdata.url) Output E:prequests>python makeRequest.py https://jsonplaceholder.typicode.com/users?id=9&username=Delphine Print Page Previous Next Advertisements ”;

Handling POST, PUT, PATCH and DELETE Requests ”; Previous Next In this chapter, we will understand how to use the POST method using requests library and also pass parameters to the URL. Using POST For PUT request, the Requests library has requests.post() method, the example of it is shown below − import requests myurl = ”https://postman-echo.com/post” myparams = {”name”: ”ABC”, ”email”:”[email protected]”} res = requests.post(myurl, data=myparams) print(res.text) Output E:prequests>python makeRequest.py {“args”:{},”data”:””,”files”:{},”form”:{“name”:”ABC”,”email”:”[email protected]”}, “headers”:{“x-forwarded-proto”:”https”,”host”:”postman-echo.com”,”content- length”:”30″,”accept”:”*/*”,”accept-encoding”:”gzip,deflate”,”content- type”:”application/x-www-form-urlencoded”,”user-agent”:”python- requests/2.22.0″,”x-forwarded- port”:”443″},”json”:{“name”:”ABC”,”email”:”[email protected]”}, “url”:”https://postman-echo.com/post”} In the example shown above, you can pass the form data as key-value pair to the data param inside requests.post(). We will also see how to work with PUT, PATCH and DELETE in requests module. Using PUT For PUT request, the Requests library has requests.put() method, the example of it is shown below. import requests myurl = ”https://postman-echo.com/put” myparams = {”name”: ”ABC”, ”email”:”[email protected]”} res = requests.put(myurl, data=myparams) print(res.text) Output E:prequests>python makeRequest.py {“args”:{},”data”:””,”files”:{},”form”:{“name”:”ABC”,”email”:”[email protected]”}, “headers”:{“x-forwarded-proto”:”https”,”host”:”postman-echo.com”,”content- length”: “30”,”accept”:”*/*”,”accept-encoding”:”gzip, deflate”,”content- type”:”applicatio n/x-www-form-urlencoded”,”user-agent”:”python-requests/2.22.0″,”x-forwarded- port “:”443″},”json”:{“name”:”ABC”,”email”:”[email protected]”}, “url”:”https://postman-echo.com/put”} Using PATCH For the PATCH request, the Requests library has requests.patch() method, the example of it is shown below. import requests myurl = https://postman-echo.com/patch” res = requests.patch(myurl, data=”testing patch”) print(res.text) Output E:prequests>python makeRequest.py {“args”:{},”data”:{},”files”:{},”form”:{},”headers”:{“x-forwarded- proto”:”https” ,”host”:”postman-echo.com”,”content-length”:”13″,”accept”:”*/*”,”accept- encoding “:”gzip, deflate”,”user-agent”:”python-requests/2.22.0″,”x-forwarded- port”:”443″ },”json”:null,”url”:”https://postman-echo.com/patch”} Using DELETE For the DELETE request, the Requests library has requests.delete() method, the example of it is shown below. import requests myurl = ”https://postman-echo.com/delete” res = requests.delete(myurl, data=”testing delete”) print(res.text) Output E:prequests>python makeRequest.py {“args”:{},”data”:{},”files”:{},”form”:{},”headers”:{“x-forwarded- proto”:”https” ,”host”:”postman-echo.com”,”content-length”:”14″,”accept”:”*/*”,”accept- encoding “:”gzip, deflate”,”user-agent”:”python-requests/2.22.0″,”x-forwarded- port”:”443″ },”json”:null,”url”:”https://postman-echo.com/delete”} Print Page Previous Next Advertisements ”;

Requests – SSL Certification ”; Previous Next SSL certificate is a security feature that comes with secure urls. When you use Requests library, it also verifies SSL certificates for the https URL given. SSL verification is enabled by default in the requests module and will throw an error if the certificate is not present. Working with secure URL Following is the example of working with secure URL − import requests getdata = requests.get(https://jsonplaceholder.typicode.com/users) print(getdata.text) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] We are easily getting a response from the above https URL, and it is because the request module can verify the SSL certificate. You can disable the SSL verification by simply adding verify=False as shown in the example below. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, verify=False) print(getdata.text) You will get the output, but it will also give a warning message that, the SSL certificate is not verified and adding certificate verification is advised. Output E:prequests>python makeRequest.py connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3 .readthedocs.io/en/latest/advanced-usage.htm l#ssl-warnings InsecureRequestWarning) [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] You can also verify SSL certificate by hosting it at your end, and giving the path using verify param as shown below. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, verify=”C:UsersAppDataLocalcertificate.txt”) print(getdata.text) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] Print Page Previous Next Advertisements ”;

Requests – Working with Cookies ”; Previous Next This chapter will discuss how to deal with cookies. You can get the cookies as well as send your cookies while calling the URL using the requests library. The url, https://jsonplaceholder.typicode.com/users when hits in the browser we can get the details of the cookies as shown below − You can read the cookies as shown below − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.cookies[“__cfduid”]) Output E:prequests>python makeRequest.py d1733467caa1e3431fb7f768fa79ed3741575094848 You can also send cookies when we make a request. Example import requests cookies = dict(test=”test123”) getdata = requests.get(”https://httpbin.org/cookies”,cookies=cookies) print(getdata.text) Output E:prequests>python makeRequest.py { “cookies”: { “test”: “test123″ } } Print Page Previous Next Advertisements ”;

Requests – Working with Errors ”; Previous Next This chapter will discuss how to deal with errors coming down when working with the Http request library. It is always a good practice to have errors managed for all possible cases. Error Exception The requests module gives the following types of error exception − ConnectionError − This will be raised, if there is any connection error. For example, the network failed, DNS error so the Request library will raise ConnectionError exception. Response.raise_for_status() − Based on status code i.e. 401, 404 it will raise HTTPError for the url requested. HTTPError − This error will be raised for an invalid response coming down for the request made. Timeout − Errors raised for a timeout for the URL requested. TooManyRedirects − If the limit is crossed for maximum redirections than it will raise TooManyRedirects error. Example Here is an example of errors shown for timeout − import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”,timeout=0.001) print(getdata.text) Output raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=”jsonplaceholder.ty picode.com”, port=443): Max retries exceeded with url: /users (Caused by Connect TimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000000B02AD E76A0>, ”Connection to jsonplaceholder.typicode.com timed out. (connect timeout = 0.001)”)) Print Page Previous Next Advertisements ”;

Requests – Event Hooks ”; Previous Next We can add events to the URL requested using event hooks. In the example below, we are going to add a callback function that will get called when the response is available. Example To add the callback, we need to make use of hooks param as shown in the example below − import requests def printData(r, *args, **kwargs): print(r.url) print(r.text) getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, hooks={”response”: printData}) Output E:prequests>python makeRequest.py https://jsonplaceholder.typicode.com/users [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] You can also call multiple callback functions as shown below − Example import requests def printRequestedUrl(r, *args, **kwargs): print(r.url) def printData(r, *args, **kwargs): print(r.text) getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, hooks = {”response”: [printRequestedUrl, printData]}) Output E:prequests>python makeRequest.py https://jsonplaceholder.typicode.com/users [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] You can also add the hook to the Session created as shown below − Example import requests def printData(r, *args, **kwargs): print(r.text) s = requests.Session() s.hooks[”response”].append(printData) s.get(”https://jsonplaceholder.typicode.com/users”) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] Print Page Previous Next Advertisements ”;

Requests – Quick Guide ”; Previous Next Requests – Overview Requests is a HTTP library that provides easy functionality to deal with http request/response in your web application. The library is developed in python. The official website of Python Requests which is available at https://2.python-requests.org/en/master/ defines Requests as follows − Requests is an elegant and simple HTTP library for Python, built for human beings. Features of Requests The features of Requests are discussed below − Request The python requests library has easy to use methods available to handle Http request. Passing of parameters and handling the request type like GET, POST, PUT, DELETE, etc. is very easy. Response You can get the response in the format you need and the supported ones are text format, binary response, json response, and raw response. Headers The library allows you to read, update or send new headers as per your requirements. Timeouts Timeouts can be easily added to the URL you are requesting using python requests library. It so happens that you are using a third-party URL and waiting for a response. It is always a good practice to give a timeout on the URL as we might want the URL to respond within that timeout with a response or an error that is coming because of timeout. Not doing so can cause either to wait on that request indefinitely. Error handling The requests module gives support for error handling and some of which are Connection Error, Timeout errors, TooManyRedirects, Response.raise_for_status errors, etc. Cookies The library allows you to read, write and update for the requested URL. Sessions To maintain the data, you require between requests you need sessions. So, if the same host is called again and again you can re-use the TCP connection which in turn will improve the performance. SSL certificates SSL certificate is a security feature that comes with secure urls. When you use Requests, it also verifies SSL certificates for the https URL given. SSL Verification is enabled by default in the requests library and will throw an error if the certificate is not present. Authentication HTTP authentication is on the server-side asking for some authentication information like username, password when the client requests a URL. This is an additional security for the request and the response being exchanged between the client and the server. Advantages of using Python Requests Library Following are the advantages of using Python Requests Library − Easy to use and fetch the data from the URL given. Requests library can be used to scrape the data from the website. Using requests, you can get, post, delete, update the data for the URL given. The handling of cookies and session is very easy. The security is also taken care of the help of authentication module support. Requests – Environment Setup In this chapter, we will work on the installation of Requests. To start working with the Requests module, we need to install Python first. So we are going to work on following− Install Python Install Requests Installing Python Go to the Python official site: https://www.python.org/downloads/ as shown below and click on the latest version available for Windows, Linux/Unix, and Mac OS. Download Python as per your 64 or 32 bit OS available with you. Once you have downloaded, click on the .exe file and follow the steps to install python on your system. The python package manager, i.e., pip will also get installed by default with the above installation. To make it work globally on your system, directly add the location of python to the PATH variable. The same is shown at the start of the installation to remember to check the checkbox which says ADD to PATH. In case you forget to check it, please follow the below-given steps to add to PATH. To add to PATH follow the steps− Right-click on your Computer icon and click on properties > Advanced System Settings. It will display the screen as shown below − Click on Environment Variables as shown above. It will display the screen as shown below − Select Path and click on Edit button, add the location path of your python at the end. Now, let us check the python version. Checking the python version E:prequests>python –version Python 3.7.3 Install Requests Now that we have python installed, we are going to install Requests. Once python is installed, python package manager i.e. pip will also get installed. Following is the command to check pip version. E:prequests>pip –version pip 19.1.1 from c:usersxxxxxappdatalocalprogramspythonpython37libsite-p ackagespip (python 3.7) We have pip installed and the version is 19.1.1. Now, will use pip to install Requests module. The command is given below− pip install requests E:prequests>pip install requests Requirement already satisfied: requests in c:usersxxxxappdatalocalprograms pythonpython37libsite-packages (2.22.0) Requirement already satisfied: certifi>=2017.4.17 in c:userskamatappdataloca lprogramspythonpython37libsite-packages (from requests) (2019.3.9) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxxxxxappdatalocalprogramspythonpython37libsite-packages (from requests ) (1.25.3) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxxxxxxxappdatalocalpr ogramspythonpython37libsite-packages (from requests) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxxxxxappdatal ocalprogramspythonpython37libsite-packages (from requests) (3.0.4) We already have the module installed, so in the command prompt it says Requirement already satisfied; if not installed it would have downloaded the required packages for installation. To check the details of the requests module installed, you can use the following command− pip show requests E:prequests>pip show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: [email protected] License: Apache 2.0 Location: c:usersxxxxxappdatalocalprogramspythonpython37libsite-package S Requires: certifi, idna, urllib3, chardet Required-by: The version of Requests module is 2.22.0. Requests – How Http Requests Work? Python’s Requests is a HTTP library that will help us exchange data between the client and the server. Consider you have a UI with a form, wherein you need to enter the user details, so once you enter it, you have to submit the data which is nothing but a Http POST or PUT request from the client to server to save the data. When you want the data, you need to fetch it from the server, which

Requests – Environment Setup ”; Previous Next In this chapter, we will work on the installation of Requests. To start working with the Requests module, we need to install Python first. So we are going to work on following − Install Python Install Requests Installing Python Go to the Python official site: https://www.python.org/downloads/ as shown below and click on the latest version available for Windows, Linux/Unix, and Mac OS. Download Python as per your 64 or 32 bit OS available with you. Once you have downloaded, click on the .exe file and follow the steps to install python on your system. The python package manager, i.e., pip will also get installed by default with the above installation. To make it work globally on your system, directly add the location of python to the PATH variable. The same is shown at the start of the installation to remember to check the checkbox which says ADD to PATH. In case you forget to check it, please follow the below-given steps to add to PATH. To add to PATH follow the steps − Right-click on your Computer icon and click on properties > Advanced System Settings. It will display the screen as shown below − Click on Environment Variables as shown above. It will display the screen as shown below − Select Path and click on Edit button, add the location path of your python at the end. Now, let us check the python version. Checking the python version E:prequests>python –version Python 3.7.3 Install Requests Now that we have python installed, we are going to install Requests. Once python is installed, python package manager i.e. pip will also get installed. Following is the command to check pip version. E:prequests>pip –version pip 19.1.1 from c:usersxxxxxappdatalocalprogramspythonpython37libsite- packagespip (python 3.7) We have pip installed and the version is 19.1.1. Now, will use pip to install Requests module. The command is given below − pip install requests E:prequests>pip install requests Requirement already satisfied: requests in c:usersxxxxappdatalocalprograms pythonpython37libsite-packages (2.22.0) Requirement already satisfied: certifi>=2017.4.17 in c:userskamatappdatalocal programspythonpython37libsite-packages (from requests) (2019.3.9) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxxxxxappdatalocalprogramspythonpython37libsite-packages (from requests ) (1.25.3) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxxxxxxxappdatalocal programspythonpython37libsite-packages (from requests) (2.8) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxxxxxappdata localprogramspythonpython37libsite-packages (from requests) (3.0.4) We already have the module installed, so in the command prompt it says Requirement already satisfied; if not installed it would have downloaded the required packages for installation. To check the details of the requests module installed, you can use the following command − pip show requests E:prequests>pip show requests Name: requests Version: 2.22.0 Summary: Python HTTP for Humans. Home-page: http://python-requests.org Author: Kenneth Reitz Author-email: [email protected] License: Apache 2.0 Location: c:usersxxxxxappdatalocalprogramspythonpython37libsite-package S Requires: certifi, idna, urllib3, chardet Required-by: The version of Requests module is 2.22.0. Print Page Previous Next Advertisements ”;

Requests – Handling Sessions ”; Previous Next To maintain the data between requests you need sessions. So, if the same host is called again and again, you can reuse the TCP connection which in turn will improve the performance. Let us now see, how to maintain cookies across requests made using sessions. Adding cookies using session import requests req = requests.Session() cookies = dict(test=”test123”) getdata = req.get(”https://httpbin.org/cookies”,cookies=cookies) print(getdata.text) Output E:prequests>python makeRequest.py { “cookies”: { “test”: “test123″ } } Using session, you can preserve the cookies data across requests. It is also possible to pass headers data using the session as shown below − Example import requests req = requests.Session() req.headers.update({”x-user1”: ”ABC”}) headers = {”x-user2”: ”XYZ”} getdata = req.get(”https://httpbin.org/headers”, headers=headers) print(getdata.headers) Print Page Previous Next Advertisements ”;