Category: mongoengine

MongoEngine – Document Inheritance ”; Previous Next It is possible to define an inherited class of any user defined Document class. The inherited class may add extra fields if required. However, since such as a class is not a direct subclass of Document class, it will not create a new collection, instead its objects are stored in a collection used by its parent class. In the parent class, meta attribute ‘allow_inheritance the following example, we first define employee as a document class and set allow_inheritance to true. The salary class is derived from employee, adding two more fields dept and sal. Objects of Employee as well as salary classes are stored in employee collection. In the following example, we first define employee as a document class and set allow_inheritance to true. The salary class is derived from employee, adding two more fields dept and sal. Objects of Employee as well as salary classes are stored in employee collection. from mongoengine import * con=connect(”newdb”) class employee (Document): name=StringField(required=True) branch=StringField() meta={”allow_inheritance”:True} class salary(employee): dept=StringField() sal=IntField() e1=employee(name=”Bharat”, branch=”Chennai”).save() s1=salary(name=”Deep”, branch=”Hyderabad”, dept=”Accounts”, sal=25000).save() We can verify that two documents are stored in employee collection as follows − { “_id”:{“$oid”:”5ebc34f44baa3752530b278a”}, “_cls”:”employee”, “name”:”Bharat”, “branch”:”Chennai” } { “_id”:{“$oid”:”5ebc34f44baa3752530b278b”}, “_cls”:”employee.salary”, “name”:”Deep”, “branch”:”Hyderabad”, “dept”:”Accounts”, “sal”:{“$numberInt”:”25000″} } Note that, in order to identify the respective Document class, MongoEngine adds a “_cls” field and sets its value as “employee” and “employee.salary”. If you want to provide extra functionality to a group of Document classes, but without overhead of inheritance, you can first create an abstract class and then derive one or more classes from the same. To make a class abstract, meta attribute ‘abstract’ is set to True. from mongoengine import * con=connect(”newdb”) class shape (Document): meta={”abstract”:True} def area(self): pass class rectangle(shape): width=IntField() height=IntField() def area(self): return self.width*self.height r1=rectangle(width=20, height=30).save() Print Page Previous Next Advertisements ”;

MongoEngine – Custom Query Sets ”; Previous Next By default, the objects attribute on a document class returns a QuerySet without applying any filter. However, you can define a classmethod on a document that modifies a queryset. Such a method should accept two arguments – doc_cls and queryset and it needs to be decorated with queryset_manager() in order for it to be recognized. @queryset_manager def qry_method(docs_cls,queryset): …. —- In the following example, the document class called products has an expensive_prods() method which is decorated by @queryset_manager. The method itself applies a filter to queryset such that only objects with price >20000 are returned. This method is now the default document query and objects attribute of products class returns filtered documents. from mongoengine import * con=connect(”newdb”) class products (Document): ProductID=IntField(required=True) company=StringField() Name=StringField() price=IntField() @queryset_manager def expensive_prods(docs_cls,queryset): return queryset.filter(price__gt=20000) for product in products.expensive_prods(): print (“Name:{} company:{} price:{}”.format(product.Name, product.company, product.price)) Output Name:Laptop company:Acer price:25000 Name:TV company:Samsung price:50000 Name:TV company:Philips price:31000 Name:Laptop company:Dell price:45000 If you wish to customize methods for filtering documents, first declare a subclass of QuerySet class, and use it as value of queryset_class property in meta dictionary. The example below uses MyQuerySet class as definition of custom queryset. The myqrymethod() in this class filters the documents whose name field ends with ‘er’. In products class, meta attribute refers to this queryset subclass is used as value of queryset_class property. from mongoengine import * con=connect(”newdb”) class MyQuerySet(QuerySet): def myqrymethod(self): return self.filter(Name__endswith=”er”) class products (Document): meta = {”queryset_class”: MyQuerySet} ProductID=IntField(required=True) company=StringField() Name=StringField() price=IntField() for product in products.objects.myqrymethod(): print (“Name:{} company:{} price:{}”.format(product.Name, product.company, product.price)) Output Name:Router company:Iball price:2000 Name:Scanner company:Cannon price:5000 Name:Printer company:Cannon price:12500 Print Page Previous Next Advertisements ”;

MongoEngine – Signals ”; Previous Next Signals are events dispatched by a sender object, any number of receiver objects can subscribe to such events. A signal receiver can subscribe to a specific sender or may receive signals from many senders. In MongoEngine, signal handling is supported by blinker library, which means you need to install it using pip utility. The mongoengine.signals module has the definitions of following signals − pre_init Called during the creation of a new Document or EmbeddedDocument instance and executed after the constructor arguments have been collected but before any additional processing has been done to them. post_init Called after all processing of a new Document or EmbeddedDocument instance has been completed. pre_save Called within save() prior to performing any actions. pre_save_post_validation Called within save() after validation has taken place but before saving. post_save Called within save() after most actions (validation, insert/update) have completed successfully. An additional Boolean keyword argument is passed to indicate if the save was an insert or an update. pre_delete Called within delete() prior to attempting the delete operation. post_delete Called within delete() upon successful deletion of the record. pre_bulk_insert Called after validation of the documents to insert, but prior to any data being written. post_bulk_insert Called after a successful bulk insert operation. An additional Boolean argument, loaded, identifies the contents of documents as either Document instances when True or a list of primary key values for the inserted records if False. An event handler function is then attached to Document class. Note that EmbeddedDocument only supports pre/post_init signals. pre/post_save, etc., should be attached to Document’s class only. You can also use a decorator to quickly create a number of signals and attach them to your Document or EmbeddedDocument subclasses as class decorators. In the following example, used as demonstration of signal handlers, we also use Python’s standard library module – logging and set the logging level to debug. from mongoengine import * from mongoengine import signals import logging logging.basicConfig(level=logging.DEBUG) We then write a document class so that corresponding collection is created in newdb database. Inside the class, two class mehods pre_save() and post_save() methods are defined which are intended to be invoked before and after a document is saved in Author collection. class Author(Document): name = StringField() def __unicode__(self): return self.name @classmethod def pre_save(cls, sender, document, **kwargs): logging.debug(“Pre Save: %s” % document.name) @classmethod def post_save(cls, sender, document, **kwargs): logging.debug(“Post Save: %s” % document.name) if ”created” in kwargs: if kwargs[”created”]: logging.debug(“Created”) else: logging.debug(“Updated”) Both the class methods are defined with arguments for classname, sender object and document with optional list of keyword arguments. Finally, we register the signal handlers. signals.pre_save.connect(Author.pre_save, sender=Author) signals.post_save.connect(Author.post_save, sender=Author) As we create an instance of Document subclass, the console log will show the pre and post save signals being processed by respective event handlers. Author(name=”Lathkar”).save() Python console reports the log as shown below − DEBUG:root:Pre Save: Lathkar DEBUG:root:Post Save: Lathkar DEBUG:root:Created Print Page Previous Next Advertisements ”;

MongoEngine – Quick Guide ”; Previous Next MongoEngine – MongoDB NoSQL databases have seen rise in popularity in the last decade. In today’s world of real time web applications, huge amount of data is being generated with mobile and embedded devices. Traditional relational databases (like Oracle, MySQL, etc.) are not suitable for strings. The processing of such data is also difficult as they have fixed and predefined schema, and are not scalable. NOSQL databases have flexible schema and are stored in distributed manner on a large number of community servers. NOSQL databases are classified on the basis of organization of data. MongoDB is a popular Document Store NOSQL database. Fundamental constituent of a MongoDB database is called a document. A document is a collection of key-value pairs stored in JSON format. More than one documents are stored in a collection. A collection can be considered as analogous to a table in any relational database, and a Document as row in a table. However, it should be noted that since MongoDB is schema less, number of key-value pairs in each document of a Collection need not be the same. MongoDB is developed by MongoDB Inc. It is a general-purpose, distributed document based database. It is available in enterprise as well as community edition. Latest version of Community version for Windows operating system can be downloaded from https://fastdl.mongodb.org/win32/mongodb-win32-x86_64-2012plus-4.2.6-signed.msi. Install MongoDB in a folder of your choice and start the server with the following command− D:mongodbbin>mongod Server is now ready for incoming connection requests at port 27017. MongoDB databases are stored in bin/data directory. This location can be changed by –dbpath option in above command. In another command terminal, start MongoDB console with the following command − D:mongodbbin>mongo MongoDB prompt is similar to what we normally see in MySQL or SQLite terminal. All database operations such as creating database, inserting a document, updating and deleting as well as retrieval of documents can be done from within the console. E:mongodbbin>mongo MongoDB shell version v4.0.6 connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb Implicit session: session { “id” : UUID(“0d848b11-acf7-4d30-83df-242d1d7fa693″) } MongoDB server version: 4.0.6 — > Default database in use is test. > db Test With ”use” command any other database is set as current. If the named database does not exist, new one is created. > use mydb switched to db mydb Please refer to our detailed tutorial on MongoDB at https://www.tutorialspoint.com/mongodb/index.htm. MongoEngine – MongoDB Compass MongoDB has also developed a GUI tool for handling MongoDB databases. It is called MongoDB Compass. It is a convenient tool for performing all CRUD operations without manually writing queries. It helps in many activities such as indexing, document validation, etc. Download community edition of MongoDB Compass from https://www.mongodb.com/download-center/compass and start MongoDBCompassCommunity.exe (Ensure that MongoDB server is running before starting Compass). Connect to the local server by giving correct host and port number. All the databases currently available will be listed as below − Click on + button (shown at the bottom of left panel) to create new database. Choose name of database from list and select a Collection as shown below − You can add document directly or import from CSV or JSON file. Choose Insert Document from Add data drop down. Documents added will be displayed in JSON, list or tabular form − Note that, just as a table in relational database has a primary key, document in MongoDB database has a special key called “_id” that is automatically generated. MongoDB Inc. provides a Python driver for connection with MongoDB databases. It is called PyMongo whose usage is similar to standard SQL queries. After installing PyMongo module, we need object of MongoClient class for interacting with MongoDB server. <<< from pymongo import MongoClient <<< client=MongoClient() New database is created with the following statement − db=client.mydatabase CRUD operations on this database are performed with methods such as insert_one() (or insert_many()), find(), update() and delete() methods. Detailed discussion of PyMongo library is available at https://www.tutorialspoint.com/python_data_access/python_mongodb_introduction.htm. However, Python’s user defined objects cannot be stored in database unless it is converted in MongoDB’s data types. This is where we need MongoEngine library. MongoEngine – Object Document Mapper MongoDB is a document based database. Each document is a JSON like representation of fields and values. A document in MongoDB is roughly equivalent to a row in RDBMS table (MongoDB equivalent of table is Collection). Even though MongoDB does not enforce any predefined schema, the field objects in a document have certain data type. MongoDB data types are very much similar to Python’s primary data types. If one has to store object of Python’s user defined class, its attributes have to be manually parsed to equivalent MongoDB data types. MongoEngine provides a convenient abstraction layer over PyMongo and maps each object of Document class to a document in MongoDB database. MongoEngine API has been developed by Hary Marr in August 2013. Latest version of MongoEngine is 0.19.1. MongoEngine is to MongoDB what SQLAlchemy is to RDBMS databases. MongoEngine library provides a Document class that is used as base for defining custom class. Attributes of this class form the fields of MongoDB document. The Document class defines methods to perform CRUD operations. In subsequent topics, we shall learn how to use them. MongoEngine – Installation To use MongoEngine, you need to have already installed MongoDB and MongoDB server should be running as described earlier. Easiest way to install MongoEngine is by using PIP installer. pip install mongoengine If your Python installation does not have Setuptools installed, you will have to download MongoEngine from https://github.com/MongoEngine/mongoengine and run the following command − python setup.py install MongoEngine has the following dependencies −

MongoEngine – GridFS ”; Previous Next In MongoDB, the files with size larger than 16 MB are stored using GridFS specifications. A file is divided into multiple chunks each with a default size of 255KB. Large chunk may be as large as necessary. GridFS uses two collections, one for chunks and other for metadata. GridFS may be used to store any file if you want to access it without having to load it entirely in the memory. MongoEngine API supports GridFS through FileField object. Using this object, it is possible to insert and retrieve data. The FileField object’s put() method helps writing the file as a part of Document. from mongoengine import * con=connect(”newdb”) class lang (Document): name=StringField() developer=StringField() logo=FileField() l1=lang() l1.name=”Python” l1.developer=”Van Rossum” f=open(”pylogo.png”,”rb”) l1.logo.put(f,content_type=”image/png”) l1.save() Contents of FileField can be retrieved by read() method of Python’s File object. logo = l1.logo.read() There is also delete() method to delete the stored file. l1 = lang.objects(name=”Python”).first() l1.logo.delete() l1.save() Note that the FileField stores only the ID of file in a separate GridFS collection. Hence delete() method does not delete the file physically. The replace() method helps in replacing reference of file with another file. l1 = lang.objects(name=”Python”).first() f=open(”newlogo.png”,”rb”) l1.logo.replace(f,content_type=”image/png”) l1.save() Print Page Previous Next Advertisements ”;

MongoEngine – Text search ”; Previous Next MongoDB supports use of query operators that can perform text search on a string content. As described earlier, to set a text index prefix name of index with $ symbol. For a text index, the weight of an indexed field denotes the significance of the field relative to the other indexed fields in terms of the text search score. You can also specify default language in meta dictionary of the class. List of supported languages can be found at https://docs.mongodb.com/manual/reference/text-search-languages/ MongoEngine API consists of search_text() method for QuerySet object. The string to be searched in indexed fields is given as argument. In the following example, we first define a Document class called lang with two string fields, name of language and its features. We also create indexes on both fields with respective weights. from mongoengine import * con=connect(”newdb”) class lang (Document): name=StringField() features=StringField() meta = {”indexes”: [ {”fields”: [”$name”, “$features”], ”default_language”: ”english”, ”weights”: {”name”: 2, ”features”: 10} }] } l1=lang() l1.name=”C++” l1.features=”Object oriented language for OS development” l1.save() l2=lang() l2.name=”Python” l2.features=”dynamically typed and object oriented for data science, AI and ML” l2.save() l3=lang() l3.name=”HTML” l3.features=”scripting language for web page development” l3.save() In order to perform search for word ‘oriented’, we employ search_text() method as follows − docs=lang.objects.search_text(”oriented”) for doc in docs: print (doc.name) Output of the above code will be names of languages in whose description the word ‘oriented’ occurs (‘Python and ‘C++’ in this case). Print Page Previous Next Advertisements ”;

MongoEngine – Advanced Queries ”; Previous Next In order to get more efficiency in retrieving a subset of fields in a document, use only() method of Objects attribute. This will significantly improve performance especially for fields with extremely large length such as ListField. Pass the required field to only() function. If other fields are accessed after executing only() query, default value is returned. from mongoengine import * con=connect(”newdb”) class person (Document): name=StringField(required=True) city=StringField(default=”Mumbai”) pin=IntField() p1=person(name=”Himanshu”, city=”Delhi”, pin=110012).save() doc=person.objects.only(”name”).first() print (”name:”,doc.name) print (”city:”, doc.city) print (”PIN:”, doc.pin) Output name: Himanshu city: Mumbai PIN: None Note − The value of city attribute is used as default. As default is not specified for PIN, it prints None. You may call reload() function if you need missing fields. When a document class has a ListField or DictField, while iterating through it, any DBREf objects are automatically dereferenced. To increase the efficiency further, especially if the document has ReferenceField, number of queries can be limited by using select_related() function which converts QuerySet in a list and effects dereferencing. MongoEngine API contains Q class which is useful for constructing advanced queries consisting of number of constraints. Q represents a part of query which can be initialized by keyword argument syntax and binary & and | operators. person.objects(Q(name__startswith=’H’) &Q(city=’Mumbai’)) Print Page Previous Next Advertisements ”;

MongoEngine – Atomic Updates ”; Previous Next Atomicity is one of the ACID transaction properties. A database transaction has to be indivisible and irreducible so that it either occurs completely or doesn’t occur at all. This property is called Atomicity. MongoDB supports Atomicity only on single documents and not on multi-document transactions. MongoEngine provides the following methods for atomic updates on a queryset. update_one() − Overwrites or adds first document matched by query. update() − Performs atomic update on fields matched by query. modify() − Update a document and return it. Following modifiers may be used with these methods. (These modifiers come before the field, not after). set set a particular value unset delete a particular value inc increment a value by a given amount dec decrement a value by a given amount push append a value to a list push_all append several values to a list pop remove the first or last element of a list depending on the value pull remove a value from a list pull_all remove several values from a list add_to_set add value to a list only if its not in the list already The following is an example of atomic update, we first create a Document class called tests and add a document in it. from mongoengine import * con=connect(”newdb”) class tests (Document): name=StringField() attempts=IntField() scores=ListField(IntField()) t1=tests() t1.name=”XYZ” t1.attempts=0 t1.scores=[] t1.save() Let us use update_one() method to update name field from XYZ to MongoDB. tests.objects(name=”XYZ”).update_one(set__name=”MongoDB”) The push modifier is used to add data in ListField (scores). tests.objects(name=”MongoDB”).update_one(push__scores=50) To increment attempts field by one, we can use inc modifier. tests.objects(name=”MongoDB”).update_one(inc__attempts=1) The updated document looks as follows − { “_id”:{“$oid”:”5ebcf8d353a48858e01ced04″}, “name”:”MongoDB”, “attempts”:{“$numberInt”:”1″}, “scores”:[{“$numberInt”:”50″}] } Print Page Previous Next Advertisements ”;

MongoEngine – Javascript ”; Previous Next QuerySet object of MongoEngine has exec_js() method that allows execution of a Javascript function on MongoDB server. This function processes the following arguments − exec_js(code, *field_names, **options) Where, code − a string containing Javascript code to execute fields − to be used in your function, which will be passed as arguments options − options that you want available to the function (accessed in Javascript through the options object) In addition, some more variables are also made available to the function’s scope as given below − collection − name of the collection corresponding to the Document class. This should be used to get the Collection object from db in Javascript code. query − the query that has been generated by the QuerySet object; passed into the find() method on a Collection object in the Javascript function. options − an object containing the keyword arguments passed into exec_js(). Note that attributes in MongoEngine document class may use different names in the database (set using the db_field keyword argument to a Field constructor). class BlogPost(Document): title = StringField(db_field=”doctitle”) For this purpose, a mechanism exists for replacing MongoEngine field attribute with the database field names in Javascript code. When accessing a field on a collection object, use square-bracket notation, and prefix the MongoEngine field name with a tilde (~) symbol. The field name that follows the tilde will be translated to the name used in the database. document”: doc[~title]; Note that when Javascript code refers to fields on embedded documents, the name of the EmbeddedDocumentField, followed by a dot, should be used before the name of the field on the embedded document. Print Page Previous Next Advertisements ”;

MongoEngine – QuerySet Methods ”; Previous Next The QuerySet object possesses following useful methods for querying the database. first() First document satisfying the query is returned. Following code will return first document in products collection, that has price < 20000. qset=products.objects(price__lt=20000) doc=qset.first() print (”Name:”,doc.Name, ”Price:”,doc.price) Output Name: Router Price: 2000 exclude() This will cause mentioned fields to be excluded from Query Set. Here, to_json() mehod of Document class is used to obtain JSONified version of Document. ProductID field will not appear in the result. for product in products.objects.exclude(”ProductID”): print (product.to_json()) Output {“_id”: {“$oid”: “5c8dec275405c12e3402423c”}, “Name”: “Laptop”, “price”: 25000} {“_id”: {“$oid”: “5c8dec275405c12e3402423d”}, “Name”: “TV”, “price”: 50000} {“_id”: {“$oid”: “5c8dec275405c12e3402423e”}, “Name”: “Router”, “price”: 2000} {“_id”: {“$oid”: “5c8dec275405c12e3402423f”}, “Name”: “Scanner”, “price”: 5000} {“_id”: {“$oid”: “5c8dec275405c12e34024240”}, “Name”: “Printer”, “price”: 12500} fields() Use this method to manipulate which fields to load in the query set. Use field names as keyword arguments and set to 1 to include, 0 to exclude. for product in products.objects.fields(ProductID=1,price=1): print (product.to_json()) Output {“_id”: {“$oid”: “5c8dec275405c12e3402423c”}, “ProductID”: 1, “price”: 25000} {“_id”: {“$oid”: “5c8dec275405c12e3402423d”}, “ProductID”: 2, “price”: 50000} {“_id”: {“$oid”: “5c8dec275405c12e3402423e”}, “ProductID”: 3, “price”: 2000} {“_id”: {“$oid”: “5c8dec275405c12e3402423f”}, “ProductID”: 4, “price”: 5000} {“_id”: {“$oid”: “5c8dec275405c12e34024240”}, “ProductID”: 5, “price”: 12500} Setting field keyword argument to 0 in fields() method works similar to exclude() method. for product in products.objects.fields(price=0): print (product.to_json()) Output {“_id”: {“$oid”: “5c8dec275405c12e3402423c”}, “ProductID”: 1, “Name”: “Laptop”} {“_id”: {“$oid”: “5c8dec275405c12e3402423d”}, “ProductID”: 2, “Name”: “TV”} {“_id”: {“$oid”: “5c8dec275405c12e3402423e”}, “ProductID”: 3, “Name”: “Router”} {“_id”: {“$oid”: “5c8dec275405c12e3402423f”}, “ProductID”: 4, “Name”: “Scanner”} {“_id”: {“$oid”: “5c8dec275405c12e34024240”}, “ProductID”: 5, “Name”: “Printer”} only() Effect of this method is similar to fields() method. Fields corresponding to keyword arguments only will appear in the query set. for product in products.objects.only(”Name”): print (product.to_json()) Output {“_id”: {“$oid”: “5c8dec275405c12e3402423c”}, “Name”: “Laptop”} {“_id”: {“$oid”: “5c8dec275405c12e3402423d”}, “Name”: “TV”} {“_id”: {“$oid”: “5c8dec275405c12e3402423e”}, “Name”: “Router”} {“_id”: {“$oid”: “5c8dec275405c12e3402423f”}, “Name”: “Scanner”} {“_id”: {“$oid”: “5c8dec275405c12e34024240”}, “Name”: “Printer”} sum() This method computes sum of given field in the query set. average() This method calculates average of given field in the query set. avg=products.objects.average(”price”) ttl=products.objects.sum(”price”) print (”sum of price field”,ttl) print (”average of price field”,avg) Output sum of price field 94500 average of price field 18900.0 Print Page Previous Next Advertisements ”;