DynamoDB Filter Expressions (Ultimate Guide w/ Examples)

What is Filter Expression in DynamoDB?

Filter Expression is a technique used in DynamoDB to filter data during scan or query operations. They are highly similar to WHERE clauses in SQL. However, they behave slightly differently because of the nature of NoSQL.

Let's see how it works with scan or query operations. There are three steps in this scenario:

1. Retrieve the requested data.
2. If there is a filter expression, it will run and remove the items that don't match.
3. Finally, return data to the client.

It is important to set the read limits in step 1 before applying a filter expression. For example, if you obtain 100KB of data in step 1 and filter it down to 10KB in step 2, you'll use the read capacity units for 100KB of data instead of the 10KB that was filtered down. Further, a 1MB limit is applied to all operations, regardless of the read capacity units on a table.

When to Use DynamoDB Filter Expression

When Excluding only a Small Set of Items

Since using a filter does not reduce the read capacity of the query or scan operations, filters are most efficient when only a small set of items are excluded.

To Reduce Payload

In some situations, developers perform filtering and search operations on the client-side after retrieving all the data from the backend. However, this can affect application performance as the payload size increases since it consumes many resources. You can use filter expression with query and scan operations to reduce the payload in such situations.

For Easier Application Filter Processing

Readability and simplicity are essential things in the application development process. For example, when you need to deal with filtering, you can filter data from the application side instead of the DB query. But it could increase the complexity and reduce the readability of the code. So instead, you can use a filter with scan or query operation for the filtering process by using an access pattern.

When Not to Use DynamoDB Filter Expression

When You Don't Have a Proper Data Model

Data modeling is one of the key concepts we have to follow in DynamoDB. Using a proper data model with suitable primary keys or secondary indexes can reduce the read item capacity when running query or scan operations. If not, you have to read all table items in the first step before using the filter expression in the query. Therefore, it may lead to a performance issue in the application.

When Operating only with Primary Key

You cannot use filter expressions on primary keys in a query operation. However, scan operations do not have this limitation. You can use filter expressions to primary keys with scan operations.

When Removing a Large Portion of Data

Usually, scan and query operations read all the table or index data before running the filter expression. If you filter a small portion of data from a large data set, it will remove a large portion of data. However, it will read all the data matching your query or scan from the database in the first step. Therefore, you need to manage these situations to avoid the use of unnecessary read capacity.

How to Use Filter Expression in DynamoDB Using Node.js

Filter expressions are similar to the key condition expressions in queries. They need you to provide an attribute to function and a condition to apply. Let's see a few examples with filter expression in DynamoDB.

Not Equal Filter Expression

Let's assume you have a table named Projects, and you need to get all the projects where the name is not equal to Project X.

If you are using the query operation, you have to use filter expression with attribute name (#name) and an expression attribute value (:name) like below:

const AWS = require("aws-sdk");
AWS.config.update({ region: "us-east-1" });
const dynamodb = new AWS.DynamoDB.DocumentClient();

var params = {
    TableName: 'projects-manager',
    KeyConditionExpression: '#PK = :PK',
    FilterExpression : "#name <> :name",
    ExpressionAttributeNames: { "#PK": "PK","#name": "name" },
    ExpressionAttributeValues: {
        ':PK': 'ORG#0182eec0-1d2a-4c4b-a7c6-059d5f717157',
        ':name':"Project X"
    }
};

dynamodb.query(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});
const AWS = require("aws-sdk");
AWS.config.update({ region: "us-east-1" });
const dynamodb = new AWS.DynamoDB.DocumentClient();

var params = {
    TableName: 'projects-manager',
    KeyConditionExpression: '#PK = :PK',
    FilterExpression : "#name <> :name",
    ExpressionAttributeNames: { "#PK": "PK","#name": "name" },
    ExpressionAttributeValues: {
        ':PK': 'ORG#0182eec0-1d2a-4c4b-a7c6-059d5f717157',
        ':name':"Project X"
    }
};

dynamodb.query(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});

Here, we have used the operator symbol <> to get all projects data that is not equal to the project name called Project X.

However, if you are using the scan operator, you have to use key condition expressions with the filter expression. You cannot filter data without using the key condition expression in the query operator.

var params = {
    TableName: 'projects-manager',
    FilterExpression : "PK = :PK and #name <> :name",
    ExpressionAttributeNames: { "#name": "name" },
    ExpressionAttributeValues: {
        ':PK': 'ORG#0182eec0-1d2a-4c4b-a7c6-059d5f717157',
        ':name':"Project X"
}
};

dynamodb.scan(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});
var params = {
    TableName: 'projects-manager',
    FilterExpression : "PK = :PK and #name <> :name",
    ExpressionAttributeNames: { "#name": "name" },
    ExpressionAttributeValues: {
        ':PK': 'ORG#0182eec0-1d2a-4c4b-a7c6-059d5f717157',
        ':name':"Project X"
}
};

dynamodb.scan(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});

Contains Filter Expression

The below example shows how to use the filter expression to get all projects that contain the word "Project" in their name.

var params = {
    TableName: 'projects-manager',
    FilterExpression : "contains(#name, :name)",
    ExpressionAttributeNames: { "#name": "name" },
    ExpressionAttributeValues: {
        ':name':"Project"
    }
};

dynamodb.scan(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});
var params = {
    TableName: 'projects-manager',
    FilterExpression : "contains(#name, :name)",
    ExpressionAttributeNames: { "#name": "name" },
    ExpressionAttributeValues: {
        ':name':"Project"
    }
};

dynamodb.scan(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});

Multiple Filtering with a Filter Expression

Filter expressions are not limited to a single option. You can use the AND operator to combine multiple filtering options. The below example shows how we can combine a contains the condition and an equal condition to write a multi-condition filter.

var params = {
    TableName: 'projects-manager',
    FilterExpression : "contains(#name, :name) AND #projectId = :projectId ",
    ExpressionAttributeNames: {"#name": "name", "#projectId":"projectId" },
    ExpressionAttributeValues: {
        ':projectId': 'ffed08f6-dc4e-4fba-a56f-ad6c80af8d01',
        ':name':"Project"
    }
};

dynamodb.scan(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});
var params = {
    TableName: 'projects-manager',
    FilterExpression : "contains(#name, :name) AND #projectId = :projectId ",
    ExpressionAttributeNames: {"#name": "name", "#projectId":"projectId" },
    ExpressionAttributeValues: {
        ':projectId': 'ffed08f6-dc4e-4fba-a56f-ad6c80af8d01',
        ':name':"Project"
    }
};

dynamodb.scan(params, function (err, data) {
    if (err) console.log(err);
    else console.log(data);
});

Count and Boolean Filter Expression

Filter options can be used to filter data based on counts and boolean values. In the below example, results are filtered based on the employee count and status. Employee count should be greater than 10, and status should be true.

This is another example of a multi conditional filter, and we have combined a comparison and an equal condition.

var params = {
    TableName: 'happy-projects',
    FilterExpression : "#employee_count >= :employee_count AND #status = :status",
    ExpressionAttributeNames: { "#employee_count": "employee_count" , "#status": "status"},
    ExpressionAttributeValues: {
        ':employee_count':10,
        ':status':true
    }
};
var params = {
    TableName: 'happy-projects',
    FilterExpression : "#employee_count >= :employee_count AND #status = :status",
    ExpressionAttributeNames: { "#employee_count": "employee_count" , "#status": "status"},
    ExpressionAttributeValues: {
        ':employee_count':10,
        ':status':true
    }
};

Best Practices for Using Filter Expressions

When using filter expressions, it's crucial to understand their impact on performance and cost. Always aim to minimize the amount of data read by DynamoDB by designing your table with appropriate primary keys and secondary indexes. This can significantly reduce the need for extensive filtering and improve query efficiency. Additionally, consider using projection expressions to limit the attributes returned by your query or scan, further reducing the payload size and improving performance. Lastly, always monitor your DynamoDB usage and performance metrics to identify and optimize any inefficient queries or scans.