Time to 10x your DynamoDB productivity with Dynobase [learn more]
DynamoDB Go Query Examples
This cheat sheet should help you understand how to perform a variety of operations starting from simple queries ending with complex transactions using AWS DynamoDB and Go programming language.
The examples were written using Go 1.16 and the "github.com/aws/aws-sdk-go-v2" package.
Dynobase's codegen figures out queries and writes code for you!
Setup
Setting up your Go application to work with DynamoDB is fairly easy. First, make sure that you have "github.com/aws/aws-sdk-go-v2/config" and "github.com/aws/aws-sdk-go-v2/service/dynamodb" dependencies installed. Then, paste the following piece of code:
Keep in mind that using access and secret keys is against best security practices, and you should instead use IAM roles/policies to interact with DynamoDB. This code, if ran on Lambda function or EC2 instance, will automatically use IAM Role attached to it.
Create table
DynamoDB structures data in tables, so if you want to save some data to DynamoDB, first you need to create a table. You can do that using AWS Console, AWS CLI or using AWS-SDK for Go, like this:
After this call resolves, it does not necessarily mean that table status is ACTIVE and it's is ready for read and write operations. Before we start manipulating items in it, we should check if it's in ACTIVE state. We can leverage waiters to do so.
package main
import("context""time""github.com/aws/aws-sdk-go-v2/aws""github.com/aws/aws-sdk-go-v2/service/dynamodb""github.com/pkg/errors")funcwaitForTable(ctx context.Context, db *dynamodb.Client, tn string)error{
w := dynamodb.NewTableExistsWaiter(db)
err := w.Wait(ctx,&dynamodb.DescribeTableInput{
TableName: aws.String(tn),},2*time.Minute,func(o *dynamodb.TableExistsWaiterOptions){
o.MaxDelay =5* time.Second
o.MinDelay =5* time.Second
})if err !=nil{return errors.Wrap(err,"timed out while waiting for table to become active")}return err
}
Delete table
If you changed your mind and need to remove DynamoDB table, don't worry, it's simple:
If you want to check what tables are available at your disposal in current region, use listTables call. Keep in mind that if selected region has more than 100 tables you'll have to paginate through them to fetch a complete list. We can leverage paginators to do most of the hard work for us.
After our table is provisioned and it's in ACTIVE state, first thing that we probably would like to do is get all items in it aka use (DynamoDB Scan operation):
If you want to narrow your search results, use FilterExpressions combined with ExpressionAttributeNames object. With Go SDK, this can be done in two ways.
The first way of performing the operation, would be to manually populate the FilterExpression and ExpressionNames properties.
The second would be to leverage the "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" package to compute the FilterExpression and ExpressionAttributeNames properties.
The snippets above will in fact return all the items in the table under one condition - you have less than 1MB of data inside it. If your table is bigger than that, you'll have to run Scan command a few times in a loop using pagination.
For simplicity sake, rest of the examples, whenever required, will be using the explicit version of providing DynamoDB expression related attributes. Please note though, that in every such situation it's completely possible to use the "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" package.
If you find writing DynamoDB expressions to be cumbersome, you can always use the Codegen feature inside Dynobase. It helps you generate Golang code using WYSWYG query editor:
Get Item
If you know the exact Partition Key (and Sort Key if using composite key) of the item that you want to retrieve from the DynamoDB table, you can use get operation.
Just like in the case of performing a scan operation, the get operation can be
in two ways.
The first way would be to specify the Key properly manually, without any kind of marshalling.
Note the dynamodbav tag. This is a special tag used by "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" package.
For simplicity sake, rest of the examples, whenever required, will be specifying the Key property manually. Please note though, that in every such situation it's completely possible to marshal a value into the structure expected by the DynamoDB SDK.
Batch Get Item
If you want to retrieve multiple items identified by a key(s) in one call, use BatchGetItem call with the following syntax:
Please note, that as previously stated, as the Item property, we could have provided a marshalled value.
Batch Write / Put Item
If you need to insert, update or delete multiple items in a single API call, use BatchWrite operation. It bundles multiple database requests against multiple tables into a single SDK call. It decreases amount of network calls needed to be made, reduces the overall latency and makes your application faster.
An example request deleting one item with key id = 123, inserting an item with id = 234 in the context of one table and deleting an item with key id = 456 in the context of another table:
If your table has composite key (which is the best practice), in order to get a collection of items sharing the same Parition Key, use Query method. It also allows to use multiple operators for SortKey such as begins_with or mathematical ones like >, =, >= and so on.
Take note of the ExpressionAttributeNames property. Some keywords are reserved for use by DynamoDB. Since date is one of them, we use ExpressionAttributeNames to map the plaintext value of date to #date. Appending an # to the underlying value is an convention.
Keep in mind that Query can return up to 1MB of data and you can also use FilterExpressions here to narrow the results on non-key attributes.
DynamoDB is hard. We get it.
Dynobase's codegen figures out queries and writes code for you!
Specify the index you want to query using IndexName parameter
Provide correct KeyConditionExpression with corresponding ExpressionAttributeValues and ExpressionAttributeNames
So, to give you an example. Imagine your table is having a Global Secondary Index called GSI1 with attributes gsi1pk and gsi1sk. If you'd like to query that index in Node.js, it will look like this:
DynamoDB also support transactions - they allow to run multiple write operations atomically meaning that either all of operations are executed successfully or none of them. It is especially useful when dealing with applications where data integrity is essential, e.g. in e-commerce - adding an item to a cart and decrementing count of items still available to buy.
Such flow should:
Should happen atomically - these two operations should be treated as one, we don't want to have a single moment in time where there's a discrepancy in items count
Should succeed only if count of items available to buy is greater than zero
Unfortunately, DynamoDB offers only one way of sorting the results on the database side - using the sort key. If your table does not have one, your sorting capabilities are limited to sorting items in application code after fetching the results. However, if you need to sort DynamoDB results on sort key descending or ascending, you can use following syntax:
package main
import("context""fmt""github.com/aws/aws-sdk-go-v2/aws""github.com/aws/aws-sdk-go-v2/config""github.com/aws/aws-sdk-go-v2/service/dynamodb""github.com/aws/aws-sdk-go-v2/service/dynamodb/types")funcmain(){
cfg, err := config.LoadDefaultConfig(context.TODO(),func(o *config.LoadOptions)error{
o.Region ="us-east-1"returnnil})if err !=nil{panic(err)}
svc := dynamodb.NewFromConfig(cfg)
out, err := svc.Query(context.TODO(),&dynamodb.QueryInput{
TableName: aws.String("my-table"),
IndexName: aws.String("Index"),
KeyConditionExpression: aws.String("id = :hashKey and #date > :rangeKey"),
ExpressionAttributeValues:map[string]types.AttributeValue{":hashKey":&types.AttributeValueMemberS{Value:"123"},":rangeKey":&types.AttributeValueMemberN{Value:"20150101"},},
ExpressionAttributeNames:map[string]string{"#date":"date",},
ScanIndexForward: aws.Bool(true),// true or false to sort by "date" Sort/Range key ascending or descending})if err !=nil{panic(err)}
fmt.Println(out.Items)}
Query (and Scan) DynamoDB Pagination
Both Query and Scan operations return results with up to 1MB of items. If you need to fetch more records, you need to invoke a second call to fetch the next page of results. We can levrage paginators to do most of the hard work for us.
package main
import("context""fmt""github.com/aws/aws-sdk-go-v2/aws""github.com/aws/aws-sdk-go-v2/config""github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue""github.com/aws/aws-sdk-go-v2/service/dynamodb""github.com/aws/aws-sdk-go-v2/service/dynamodb/types")funcmain(){
cfg, err := config.LoadDefaultConfig(context.TODO(),func(o *config.LoadOptions)error{
o.Region ="us-east-1"returnnil})if err !=nil{panic(err)}
svc := dynamodb.NewFromConfig(cfg)
p := dynamodb.NewQueryPaginator(svc,&dynamodb.QueryInput{
TableName: aws.String("my-table"),
Limit: aws.Int32(1),
KeyConditionExpression: aws.String("id = :hashKey and #date > :rangeKey"),
ExpressionAttributeValues:map[string]types.AttributeValue{":hashKey":&types.AttributeValueMemberS{Value:"123"},":rangeKey":&types.AttributeValueMemberN{Value:"20150101"},},
ExpressionAttributeNames:map[string]string{"#date":"date",},})type Item struct{
ID string`dynamodbav:"id"`
Date int`dynamodbav:"date"`}var items []Item
for p.HasMorePages(){
out, err := p.NextPage(context.TODO())if err !=nil{panic(err)}var pItems []Item
err = attributevalue.UnmarshalListOfMaps(out.Items,&pItems)if err !=nil{panic(err)}
items =append(items, pItems...)}
fmt.Println(items)}
Update Item
DynamoDB update operation in Go consists of two main parts:
Sometimes we want to update our record only if some condition is met, e.g. item is not soft-deleted (does not have deletedAt attribute set). To do that, use ConditionExpression which has similar syntax to the FilterExpression:
In the above examples the name attribute of the record with partition key id = 123 in table my-table will be only updated if this item does not have attribute deletedAt and its attribute company has value Apple.
Increment Item Attribute
Incrementing a Number value in DynamoDB item can be achieved in two ways:
Get item, update the value in the application code and send a put request back to DDB overwriting item
Using update operation
While it might be tempting to use first method because Update syntax is unfriendly, I strongly recommend using second one because of the fact it's much faster (requires only one request) and atomic:
In this example the score attribute of the record with partition key id = 123 in table my-table will incremented by one. Of course, you can use other mathematical operators too.
Delete Item
Removing single item from table is very similar to Get Item operation. The parameters of the call are actually exactly the same, the only difference is that we call delete instead of get:
Unfortunately, there's no easy way to delete all items from DynamoDB just like in SQL-based databases by using DELETE FROM my-table;. To achieve the same result in DynamoDB, you need to query/scan to get all the items in a table using pagination until all items are scanned and then perform delete operation one-by-one on each record.
If you need to use DynamoDB offline locally, you can use DynamoDB local distributed by AWS or DynamoDB from Localstack. Connecting to it is as easy as changing the EndpointResolver parameter of the client.
