MongoEngine – Aggregation
”;
The term ‘aggregation’ is used for the operation that processes data and returns computed result. Finding sum, count and average on one or more fields of documents in a collection can be called as aggregation functions.
MongoEngine provides aggregate() function that encapsulates PyMongo’s aggregation framework. Aggregation operation uses a collection as input and returns one or more documents as a result.
MongoDB uses concept of data processing pipelines. A pipeline can have multiple stages. Basic stage provides that provide filter and operate like queries. Others provide tools for grouping and/or sorting by one or more fields, string concatenation tasks, array aggregation tools, etc.
Following stages are defined in MongoDB pipeline creation −
| Name | Description |
|---|---|
| $project | Reshapes each document in the stream, by adding new fields or removing existing fields. |
| $match | Filters the document stream to allow only matching documents to pass unmodified into the next stage. $match uses standard MongoDB queries. |
| $redact | Reshapes each document by restricting the content for each document based on information stored in the documents themselves. |
| $limit | Limits documents to be passed unmodified to the pipeline |
| $skip | Skips the first n documents and passes the remaining documents unmodified to the pipeline. |
| $group | Groups input documents by a given identifier expression and applies the accumulator expressions to each group. The output documents only contain the identifier field and accumulated fields. |
| $sort | Reorders the document stream by a specified sort key. |
| $out | Writes the resulting documents of the aggregation pipeline to a collection. |
Aggregation expressions use field path to access fields in the input documents. To specify a field path, use a string that prefixes with a dollar sign $$$ the field name. Expression can use one or more Boolean operators ($and, $or, $not) and comparison operators ($eq, $gt, $lt, $gte, $lte and $ne).
Following arithmetic expressions are also used for aggregation −
| $add | Adds numbers to return the sum. Accepts any number of argument expressions |
| $subtract | Returns the result of subtracting the second value from the first |
| $multiply | Multiplies numbers to return the product. Accepts any number of argument expressions |
| $divide | Returns the result of dividing the first number by the second. Accepts two argument expressions |
| $mod | Returns the remainder of the first number divided by the second. Accepts two argument expressions |
Following string expression can also be used in aggregation −
| $concat | Concatenates any number of strings |
| $substr | Returns a substring of a string, starting at a specified index position up to a specified length |
| $toLower | Converts a string to lowercase. Accepts a single argument expression |
| $toUpper | Converts a string to uppercase. Accepts a single argument expression |
| $strcasecmp | Performs string comparison and returns 0 if two strings are equivalent, 1 if first is greater than second, and -1 if first string is less than second |
To demonstrate how aggregate() function works in MongoEngine, let us first define a Document class called orders.
from mongoengine import *
con=connect(''mydata'')
class orders(Document):
custID = StringField()
amount= IntField()
status = StringField()
We then add following documents in orders collection −
| _id | custID | amount | status |
|---|---|---|---|
| ObjectId(“5eba52d975fa1e26d4ec01d0”) | A123 | 500 | A |
| ObjectId(“5eba536775fa1e26d4ec01d1”) | A123 | 250 | A |
| ObjectId(“5eba53b575fa1e26d4ec01d2”) | B212 | 200 | D |
| ObjectId(“5eba540e75fa1e26d4ec01d3”) | B212 | 400 | A |
The aggregate() function is to be used to find sum of amount field for each custID only when status equals ‘A’. Accordingly, the pipeline is constructed as follows.
First stage in pipeline uses $match to filter documents with status=’A’. Second stage uses $group identifier to group documents on CustID and performs sum of amount.
pipeline = [
{"$match" : {"status" : "A"}},
{"$group": {"_id": "$custID", "total": {"$sum": "$amount"}}}
]
This pipeline is now used as argument to aggregate() function.
docs = orders.objects().aggregate(pipeline)
We can iterate over the document cursor with a for loop. The complete code is given below −
from mongoengine import *
con=connect(''mydata'')
class orders(Document):
custID = StringField()
amount= IntField()
status = StringField()
pipeline = [
{"$match" : {"status" : "A"}},
{"$group": {"_id": "$custID", "total": {"$sum": "$amount"}}}
]
docs = orders.objects().aggregate(pipeline)
for doc in docs:
print (x)
For the given data, the following output is generated −
{''_id'': ''B212'', ''total'': 400}
{''_id'': ''A123'', ''total'': 750}
”;