Scrapy – Item Loaders ”; Previous Next Description Item loaders provide a convenient way to fill the items that are scraped from the websites. Declaring Item Loaders The declaration of Item Loaders is like Items. For example − from scrapy.loader import ItemLoader from scrapy.loader.processors import TakeFirst, MapCompose, Join class DemoLoader(ItemLoader): default_output_processor = TakeFirst() title_in = MapCompose(unicode.title) title_out = Join() size_in = MapCompose(unicode.strip) # you can continue scraping here In the above code, you can see that input processors are declared using _in suffix and output processors are declared using _out suffix. The ItemLoader.default_input_processor and ItemLoader.default_output_processor attributes are used to declare default input/output processors. Using Item Loaders to Populate Items To use Item Loader, first instantiate with dict-like object or without one where the item uses Item class specified in ItemLoader.default_item_class attribute. You can use selectors to collect values into the Item Loader. You can add more values in the same item field, where Item Loader will use an appropriate handler to add these values. The following code demonstrates how items are populated using Item Loaders − from scrapy.loader import ItemLoader from demoproject.items import Demo def parse(self, response): l = ItemLoader(item = Product(), response = response) l.add_xpath(“title”, “//div[@class = ”product_title”]”) l.add_xpath(“title”, “//div[@class = ”product_name”]”) l.add_xpath(“desc”, “//div[@class = ”desc”]”) l.add_css(“size”, “div#size]”) l.add_value(“last_updated”, “yesterday”) return l.load_item() As shown above, there are two different XPaths from which the title field is extracted using add_xpath() method − 1. //div[@class = “product_title”] 2. //div[@class = “product_name”] Thereafter, a similar request is used for desc field. The size data is extracted using add_css() method and last_updated is filled with a value “yesterday” using add_value() method. Once all the data is collected, call ItemLoader.load_item() method which returns the items filled with data extracted using add_xpath(), add_css() and add_value() methods. Input and Output Processors Each field of an Item Loader contains one input processor and one output processor. When data is extracted, input processor processes it and its result is stored in ItemLoader. Next, after collecting the data, call ItemLoader.load_item() method to get the populated Item object. Finally, you can assign the result of the output processor to the item. The following code demonstrates how to call input and output processors for a specific field − l = ItemLoader(Product(), some_selector) l.add_xpath(“title”, xpath1) # [1] l.add_xpath(“title”, xpath2) # [2] l.add_css(“title”, css) # [3] l.add_value(“title”, “demo”) # [4] return l.load_item() # [5] Line 1 − The data of title is extracted from xpath1 and passed through the input processor and its result is collected and stored in ItemLoader. Line 2 − Similarly, the title is extracted from xpath2 and passed through the same input processor and its result is added to the data collected for [1]. Line 3 − The title is extracted from css selector and passed through the same input processor and the result is added to the data collected for [1] and [2]. Line 4 − Next, the value “demo” is assigned and passed through the input processors. Line 5 − Finally, the data is collected internally from all the fields and passed to the output processor and the final value is assigned to the Item. Declaring Input and Output Processors The input and output processors are declared in the ItemLoader definition. Apart from this, they can also be specified in the Item Field metadata. For example − import scrapy from scrapy.loader.processors import Join, MapCompose, TakeFirst from w3lib.html import remove_tags def filter_size(value): if value.isdigit(): return value class Item(scrapy.Item): name = scrapy.Field( input_processor = MapCompose(remove_tags), output_processor = Join(), ) size = scrapy.Field( input_processor = MapCompose(remove_tags, filter_price), output_processor = TakeFirst(), ) >>> from scrapy.loader import ItemLoader >>> il = ItemLoader(item = Product()) >>> il.add_value(”title”, [u”Hello”, u”<strong>world</strong>”]) >>> il.add_value(”size”, [u”<span>100 kg</span>”]) >>> il.load_item() It displays an output as − {”title”: u”Hello world”, ”size”: u”100 kg”} Item Loader Context The Item Loader Context is a dict of arbitrary key values shared among input and output processors. For example, assume you have a function parse_length − def parse_length(text, loader_context): unit = loader_context.get(”unit”, ”cm”) # You can write parsing code of length here return parsed_length By receiving loader_context arguements, it tells the Item Loader it can receive Item Loader context. There are several ways to change the value of Item Loader context − Modify current active Item Loader context − loader = ItemLoader (product) loader.context [“unit”] = “mm” On Item Loader instantiation − loader = ItemLoader(product, unit = “mm”) On Item Loader declaration for input/output processors that instantiates with Item Loader context − class ProductLoader(ItemLoader): length_out = MapCompose(parse_length, unit = “mm”) ItemLoader Objects It is an object which returns a new item loader to populate the given item. It has the following class − class scrapy.loader.ItemLoader([item, selector, response, ]**kwargs) The following table shows the parameters of ItemLoader objects − Sr.No Parameter & Description 1 item It is the item to populate by calling add_xpath(), add_css() or add_value(). 2 selector It is used to extract data from websites. 3 response It is used to construct selector using default_selector_class. Following table shows the methods of ItemLoader objects − Sr.No Method & Description Example 1 get_value(value, *processors, **kwargs) By a given processor and keyword arguments, the value is processed by get_value() method. >>> from scrapy.loader.processors import TakeFirst >>> loader.get_value(u”title: demoweb”, TakeFirst(), unicode.upper, re = ”title: (.+)”) ”DEMOWEB` 2 add_value(field_name, value, *processors, **kwargs) It processes the value and adds to the field where it is first passed through get_value by giving processors and keyword arguments before passing through field input processor. loader.add_value(”title”, u”DVD”) loader.add_value(”colors”, [u”black”, u”white”]) loader.add_value(”length”, u”80”) loader.add_value(”price”, u”2500”) 3 replace_value(field_name, value, *processors, **kwargs) It replaces the collected data with a new value. loader.replace_value(”title”, u”DVD”) loader.replace_value(”colors”, [u”black”, u”white”]) loader.replace_value(”length”, u”80”) loader.replace_value(”price”, u”2500”) 4 get_xpath(xpath, *processors, **kwargs) It is used to extract unicode strings by giving processors and keyword arguments by receiving XPath. # HTML code: <div class = “item-name”>DVD</div> loader.get_xpath(“//div[@class = ”item-name”]”) # HTML code: <div id = “length”>the length is 45cm</div> loader.get_xpath(“//div[@id = ”length”]”, TakeFirst(), re = “the length is (.*)”) 5 add_xpath(field_name, xpath, *processors, **kwargs) It receives XPath to the field