Discuss Requests ”; Previous Next Requests in an open source HTTP library that provides easy functionality to deal with Http request/response in your web application. The library is developed in python. Print Page Previous Next Advertisements ”;
Requests – Proxy ”; Previous Next So far, we have seen clients directly connecting and talking to the server. Using proxy, the interaction happens as follows − The client sends a request to the proxy. The proxy sends the request to the server. The server sends back the response to the proxy. The proxy will send a response back to the client. Using Http-proxy is additional security assigned to manage the data exchange between client and server. The requests libraries also have provision to handle proxy, by using the proxies param as shown below − Example import requests proxies = { ”http”: ”http://localhost:8080” } res = requests.get(”http://httpbin.org/”, proxies=proxies) print(res.status_code) The request will route to http://localhost:8080 URL. Output 200 Print Page Previous Next Advertisements ”;
Requests – Web Scraping using Requests ”; Previous Next We have already seen how we can get data from a given URL using python requests library. We will try to scrap the data from the site of Tutorialspoint which is available at https://www.tutorialspoint.com/tutorialslibrary.htm using the following − Requests Library Beautiful soup library from python We have already installed the Requests library, let us now install Beautiful soup package. Here is the official website for beautiful soup available at https://www.crummy.com/software/BeautifulSoup/bs4/doc/ in case you want to explore some more functionalities of beautiful soup. Installing Beautifulsoup We shall see how to install Beautiful Soup below − E:prequests>pip install beautifulsoup4 Collecting beautifulsoup4 Downloading https://files.pythonhosted.org/packages/3b/c8/a55eb6ea11cd7e5ac4ba cdf92bac4693b90d3ba79268be16527555e186f0/beautifulsoup4-4.8.1-py3-none-any.whl ( 101kB) |████████████████████████████████| 102kB 22kB/s Collecting soupsieve>=1.2 (from beautifulsoup4) Downloading https://files.pythonhosted.org/packages/81/94/03c0f04471fc245d08d0 a99f7946ac228ca98da4fa75796c507f61e688c2/soupsieve-1.9.5-py2.py3-none-any.whl Installing collected packages: soupsieve, beautifulsoup4 Successfully installed beautifulsoup4-4.8.1 soupsieve-1.9.5 We now have python requests library and beautiful soup installed. Let us now write the code, that will scrap the data from the URL given. Web scraping import requests from bs4 import BeautifulSoup res = requests.get(”https://www.tutorialspoint.com/tutorialslibrary.htm”) print(“The status code is “, res.status_code) print(“n”) soup_data = BeautifulSoup(res.text, ”html.parser”) print(soup_data.title) print(“n”) print(soup_data.find_all(”h4”)) Using requests library, we can fetch the content from the URL given and beautiful soup library helps to parse it and fetch the details the way we want. You can use a beautiful soup library to fetch data using Html tag, class, id, css selector and many more ways. Following is the output we get wherein we have printed the title of the page and also all the h4 tags on the page. Output E:prequests>python makeRequest.py The status code is 200 <title>Free Online Tutorials and Courses</title> [<h4>Academic</h4>, <h4>Computer Science</h4>, <h4>Digital Marketing</h4>, <h4>Monuments</h4>,<h4>Machine Learning</h4>, <h4>Mathematics</h4>, <h4>Mobile Development</h4>,<h4>SAP</h4>, <h4>Software Quality</h4>, <h4>Big Data & Analytics</h4>, <h4>Databases</h4>, <h4>Engineering Tutorials</h4>, <h4>Mainframe Development</h4>, <h4>Microsoft Technologies</h4>, <h4>Java Technologies</h4>, <h4>XML Technologies</h4>, <h4>Python Technologies</h4>, <h4>Sports</h4>, <h4>Computer Programming</h4>,<h4>DevOps</h4>, <h4>Latest Technologies</h4>, <h4>Telecom</h4>, <h4>Exams Syllabus</h4>, <h4>UPSC IAS Exams</h4>, <h4>Web Development</h4>, <h4>Scripts</h4>, <h4>Management</h4>,<h4>Soft Skills</h4>, <h4>Selected Reading</h4>, <h4>Misc</h4>] Print Page Previous Next Advertisements ”;
Requests – Handling Timeouts ”; Previous Next Timeouts can be easily added to the URL you are requesting. It so happens that, you are using a third-party URL and waiting for a response. It is always a good practice to give a timeout on the URL, as we might want the URL to respond within a timespan with a response or an error. Not doing so, can cause to wait on that request indefinitely. We can give timeout to the URL by using the timeout param and value is passed in seconds as shown in the example below − Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”,timeout=0.001) print(getdata.text) Output raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host=”jsonplaceholder.typicode.com”, port=443): Max retries exceeded with url: /users (Caused by Connect TimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x000000B02AD E76A0>, ”Connection to jsonplaceholder.typicode.com timed out. (connect timeout = 0.001)”)) The timeout given is as follows − getdata = requests.get(”https://jsonplaceholder.typicode.com/users”,timeout=0.001) The execution throws connection timeout error as shown in the output. The timeout given is 0.001, which is not possible for the request to get back the response and throws an error. Now, we will increase the timeout and check. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”,timeout=1.000) print(getdata.text) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] With a timeout of 1 second, we can get the response for the URL requested. Print Page Previous Next Advertisements ”;
Requests – Authentication ”; Previous Next This chapter will discuss the types of authentication available in the Requests module. We are going to discuss the following − Working of Authentication in HTTP Requests Basic Authentication Digest Authentication OAuth2 Authentication Working of Authentication in HTTP Requests HTTP authentication is on the server-side asking for some authentication information like username, password when the client requests a URL. This is additional security for the request and the response being exchanged between the client and the server. From the client-side these additional authentication information i.e. username and password can be sent in the headers, which later on the server side will be validated. The response will be delivered from the server-side only when the authentication is valid. Requests library has most commonly used authentication in requests.auth, which are Basic Authentication (HTTPBasicAuth) and Digest Authentication (HTTPDigestAuth). Basic Authentication This is the simplest form of providing authentication to the server. To work with basic authentication, we are going to use HTTPBasicAuth class available with requests library. Example Here is a working example of how to use it. import requests from requests.auth import HTTPBasicAuth response_data = requests.get(”httpbin.org/basic-auth/admin/admin123”, auth = HTTPDigestAuth(”admin”, ”admin123”)) print(response_data.text) We are calling the url, https://httpbin.org/basic-auth/admin/admin123 with user as admin and password as admin123. So, this URL will not work without authentication, i.e. user and password. Once you give the authentication using the auth param, then only the server will give back the response. Output E:prequests>python makeRequest.py { “authenticated”: true, “user”: “admin” } Digest Authentication This is another form of authentication available with requests. We are going to make use of HTTPDigestAuth class from requests. Example import requests from requests.auth import HTTPDigestAuth response_data = requests.get(”https://httpbin.org/digest-auth/auth/admin/admin123”, auth = HTTPDigestAuth(”admin”, ”admin123”)) print(response_data.text) Output E:prequests>python makeRequest.py { “authenticated”: true, “user”: “admin” } OAuth2 Authentication To use OAuth2 Authentication, we need “requests_oauth2” library. To install “requests_oauth2” do the following − pip install requests_oauth2 The display in your terminal while installing will be something as shown below − E:prequests>pip install requests_oauth2 Collecting requests_oauth2 Downloading https://files.pythonhosted.org/packages/52/dc/01c3c75e6e7341a2c7a9 71d111d7105df230ddb74b5d4e10a3dabb61750c/requests-oauth2-0.3.0.tar.gz Requirement already satisfied: requests in c:usersxyzappdatalocalprograms pythonpython37libsite-packages (from requests_oauth2) (2.22.0) Requirement already satisfied: six in c:usersxyzappdatalocalprogramspyth onpython37libsite-packages (from requests_oauth2) (1.12.0) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:use rsxyzappdatalocalprogramspythonpython37libsite-packages (from requests ->requests_oauth2) (1.25.3) Requirement already satisfied: certifi>=2017.4.17 in c:usersxyzappdataloca lprogramspythonpython37libsite-packages (from requests->requests_oauth2) (2 019.3.9) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:usersxyzappdatal ocalprogramspythonpython37libsite-packages (from requests->requests_oauth2) (3.0.4) Requirement already satisfied: idna<2.9,>=2.5 in c:usersxyzappdatalocalpr ogramspythonpython37libsite-packages (from requests->requests_oauth2) (2.8) Building wheels for collected packages: requests-oauth2 Building wheel for requests-oauth2 (setup.py) … done Stored in directory: C:UsersxyzAppDataLocalpipCachewheels90efb443 3743cbbc488463491da7df510d41c4e5aa28213caeedd586 Successfully built requests-oauth2 We are done installing “requests-oauth2”. To use the API’s of Google, Twitter we need its consent and the same is done using OAuth2 authentication. For OAuth2 authentication we will need Client ID and a Secret Key. The details of how to get it, is mentioned on https://developers.google.com/identity/protocols/OAuth2. Later on, login to Google API Console which is available at https://console.developers.google.com/and get the client id and secret key. Example Here is an example of how to use “requests-oauth2”. import requests from requests_oauth2.services import GoogleClient google_auth = GoogleClient( client_id=”xxxxxxxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com”, redirect_uri=”http://localhost/auth/success.html”, ) a = google_auth.authorize_url( scope=[“profile”, “email”], response_type=”code”, ) res = requests.get(a) print(res.url) We will not be able to redirect to the URL given, as it needs to login to the Gmail account, but here, you will see from the example, that google_auth works and the authorized URL is given. Output E:prequests>python oauthRequest.py https://accounts.google.com/o/oauth2/auth?redirect_uri= http%3A%2F%2Flocalhost%2Fauth%2Fsuccess.html& client_id=xxxxxxxxxxxxxxxxxxxxx.apps.googleusercontent.com& scope=profile+email&response_type=code Print Page Previous Next Advertisements ”;
Requests – Useful Resources ”; Previous Next The following resources contain additional information on Requests. Please use them to get more in-depth knowledge on this. Useful Video Courses Laravel RESTful APIs – Admin App, Docker, Open API(Swagger) 36 Lectures 3.5 hours Antonio Papa More Detail Ruby on Rails Course for Beginners 98 Lectures 7.5 hours Skillbakery More Detail Postman Course for Beginners 13 Lectures 1 hours Taurius Litvinavicius More Detail Learn Web Development with HTML, CSS , Javascript ,Typescript, and Angular 30 Lectures 58 mins James Coonce More Detail Learn Basic HTML, CSS & JavaScript (Typescript) with Svelte 31 Lectures 1 hours James Coonce More Detail Vue 3 and Laravel: Breaking a Monolith to Microservices 156 Lectures 14 hours Packt Publishing More Detail Print Page Previous Next Advertisements ”;
Requests – SSL Certification ”; Previous Next SSL certificate is a security feature that comes with secure urls. When you use Requests library, it also verifies SSL certificates for the https URL given. SSL verification is enabled by default in the requests module and will throw an error if the certificate is not present. Working with secure URL Following is the example of working with secure URL − import requests getdata = requests.get(https://jsonplaceholder.typicode.com/users) print(getdata.text) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] We are easily getting a response from the above https URL, and it is because the request module can verify the SSL certificate. You can disable the SSL verification by simply adding verify=False as shown in the example below. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, verify=False) print(getdata.text) You will get the output, but it will also give a warning message that, the SSL certificate is not verified and adding certificate verification is advised. Output E:prequests>python makeRequest.py connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3 .readthedocs.io/en/latest/advanced-usage.htm l#ssl-warnings InsecureRequestWarning) [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] You can also verify SSL certificate by hosting it at your end, and giving the path using verify param as shown below. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, verify=”C:UsersAppDataLocalcertificate.txt”) print(getdata.text) Output E:prequests>python makeRequest.py [ { “id”: 1, “name”: “Leanne Graham”, “username”: “Bret”, “email”: “[email protected]”, “address”: { “street”: “Kulas Light”, “suite”: “Apt. 556”, “city”: “Gwenborough”, “zipcode”: “92998-3874”, “geo”: { “lat”: “-37.3159”, “lng”: “81.1496” } }, “phone”: “1-770-736-8031 x56442”, “website”: “hildegard.org”, “company”: { “name”: “Romaguera-Crona”, “catchPhrase”: “Multi-layered client-server neural-net”, “bs”: “harness real-time e-markets” } } ] Print Page Previous Next Advertisements ”;
Requests – How Http Requests Work? ”; Previous Next Python’s Requests is a HTTP library that will help us exchange data between the client and the server. Consider you have a UI with a form, wherein you need to enter the user details, so once you enter it, you have to submit the data which is nothing but a Http POST or PUT request from the client to server to save the data. When you want the data, you need to fetch it from the server, which is again a Http GET request. The exchange of data between the client when it requests the data and the server responding with the required data, this relationship between the client and the server is very important. The request is made to the URL given and it could be a secure or non-secure URL. The request to the URL can be done using GET, POST, PUT, DELETE. The most commonly used is the GET method, mainly used when you want to fetch data from the server. You can also send data to the URL as a query string for example − https://jsonplaceholder.typicode.com/users?id=9&username=Delphine So here, we are passing id = 9 and username = Delphine to the URL. All the values are sent in key/value pair after the question mark(?) and multiple params are passed to the URL separated by &. Using the request library, the URL is called as follows using a string dictionary. Wherein the data to the URL is sent as a dictionary of strings. If you want to pass id = 9 and username = Delphine, you can do as follows − payload = {”id”: ”9”, ”username”: ”Delphine”} The requests library is called as follows − res = requests.get(”https://jsonplaceholder.typicode.com/users”, params = payload”) Using POST, we can do as follows − res = requests.post(”https://jsonplaceholder.typicode.com/users”, data = {”id”:”9”, ”username”:”Delphine”}) Using PUT res = requests.put(”https://jsonplaceholder.typicode.com/users”, data = {”id”:”9”, ”username”:”Delphine”}) Using DELETE res = requests.delete(”https://jsonplaceholder.typicode.com/users”) The response from the Http request can be in text encoded form, binary encoded, json format or raw response. The details of the request and response are explained in detail in the next chapters. Print Page Previous Next Advertisements ”;
Requests – Handling Redirection ”; Previous Next This chapter will take a look at how the Request library handles the url redirection case. Example import requests getdata = requests.get(”http://google.com/”) print(getdata.status_code) print(getdata.history) The url: http://google.com will be redirected using status code 301(Moved Permanently) to https://www.google.com/. The redirection will be saved in the history. Output When the above code is executed, we get the following result − E:prequests>python makeRequest.py 200 [<Response [301]>] You can stop redirection of a URL using allow_redirects = False. It can be done on GET, POST, OPTIONS, PUT, DELETE, PATCH methods used. Example Here is an example on the same. import requests getdata = requests.get(”http://google.com/”, allow_redirects=False) print(getdata.status_code) print(getdata.history) print(getdata.text) Now if you check the output, the redirection will not be allowed and will get a status code of 301. Output E:prequests>python makeRequest.py 301 [] <HTML><HEAD><meta http-equiv=”content-type” content=”text/html;charset=utf-8″> <TITLE>301 Moved</TITLE></HEAD><BODY> <H1>301 Moved</H1> The document has moved <A HREF=”http://www.google.com/”>here</A>. </BODY></HTML> Print Page Previous Next Advertisements ”;
Requests – Handling GET Requests ”; Previous Next This chapter will concentrate more on the GET requests, which is the most common and used very often. The working of GET in the requests module is very easy. Here is a simple example about working with the URL using the GET method. Example import requests getdata = requests.get(”https://jsonplaceholder.typicode.com/users”) print(getdata.content) getdata.content, will print all the data available in the response. Output E:prequests>python makeRequest.py b”[n {n “id”: 1,n “name”: “Leanne Graham”,n “username”: “Bret”,n “email”: “[email protected]”,n “address”: {n “street”: “Kulas Light “,n “suite”: “Apt. 556”,n “city”: “Gwenborough”,n “zipcode”: “ 92998-3874”,n “geo”: {n “lat”: “-37.3159”,n “lng”: “81.149 6″n }n },n “phone”: “1-770-736-8031 x56442”,n “website”: “hildegard.org”,n “company”: {n “name”: “Romaguera-Crona”,n “catchPhrase”: “Multi-layered client-server neural-net”,n “bs”: “harness real-time e-markets”n }n } You can also pass parameters to the get method using the param attribute as shown below − import requests payload = {”id”: 9, ”username”: ”Delphine”} getdata = requests.get(”https://jsonplaceholder.typicode.com/users”, params=payload) print(getdata.content) The details are stored in the object payload in the key/value pair and passed to params, inside get() method. Output E:prequests>python makeRequest.py b”[n {n “id”: 9,n “name”: “Glenna Reichert”,n “username”: “Delphine”, n “email”: “[email protected]”,n “address”: {n “street”: “Dayna Park”,n “suite”: “Suite 449”,n “city”: “Bartholomebury”,n “zipcode”: “76495-3109”,n “geo”: {n “lat”: “24.6463”,n “lng”: “-168.8889″n }n },n “phone”: “(775)976-6794 x41206”,n “ website”: “conrad.com”,n “company”: {n “name”: “Yost and Sons”,n “catchPhrase”: “Switchable contextually-based project”,n “bs”: “aggregate real-time technologies”n }n }n]” Print Page Previous Next Advertisements ”;