Amazon DynamoDB | Global secondary indexes Flashcards
How does throughput capacity managed by Auto Scaling work with my Reserved Capacity?
Global secondary indexes
Amazon DynamoDB | Database
Auto Scaling works with reserved capacity in the same manner as manually provisioned throughput capacity does today. Reserved Capacity is applied to the total provisioned capacity for the region you purchased it in. Capacity provisioned by Auto Scaling will consume the reserved capacity first, billed at discounted prices, and any excess capacity will be charged at standard rates. To limit total consumption to the reserved capacity you purchased, distribute maximum capacity limit across all tables with Auto Scaling enabled, to be cumulatively less than total reserved capacity amount you have purchased.
What are global secondary indexes?
Global secondary indexes
Amazon DynamoDB | Database
Global secondary indexes are indexes that contain a partition or partition-and-sort keys that can be different from the table’s primary key.
For efficient access to data in a table, Amazon DynamoDB creates and maintains indexes for the primary key attributes. This allows applications to quickly retrieve data by specifying primary key values. However, many applications might benefit from having one or more secondary (or alternate) keys available to allow efficient access to data with attributes other than the primary key. To address this, you can create one or more secondary indexes on a table, and issue Query requests against these indexes.
Amazon DynamoDB supports two types of secondary indexes:
Local secondary index — an index that has the same partition key as the table, but a different sort key. A local secondary index is “local” in the sense that every partition of a local secondary index is scoped to a table partition that has the same partition key.
Global secondary index — an index with a partition or a partition-and-sort key that can be different from those on the table. A global secondary index is considered “global” because queries on the index can span all items in a table, across all partitions.
Secondary indexes are automatically maintained by Amazon DynamoDB as sparse objects. Items will only appear in an index if they exist in the table on which the index is defined. This makes queries against an index very efficient, because the number of items in the index will often be significantly less than the number of items in the table.
Global secondary indexes support non-unique attributes, which increases query flexibility by enabling queries against any non-key attribute in the table.
Consider a gaming application that stores the information of its players in a DynamoDB table whose primary key consists of UserId (partition) and GameTitle (sort). Items have attributes named TopScore, Timestamp, ZipCode, and others. Upon table creation, DynamoDB provides an implicit index (primary index) on the primary key that can support efficient queries that return a specific user’s top scores for all games.
However, if the application requires top scores of users for a particular game, using this primary index would be inefficient, and would require scanning through the entire table. Instead, a global secondary index with GameTitle as the partition key element and TopScore as the sort key element would enable the application to rapidly retrieve top scores for a game.
A GSI does not need to have a sort key element. For instance, you could have a GSI with a key that only has a partition element GameTitle. In the example below, the GSI has no projected attributes, so it will just return all items (identified by primary key) that have an attribute matching the GameTitle you are querying on.
When should I use global secondary indexes?
Global secondary indexes
Amazon DynamoDB | Database
Global secondary indexes are particularly useful for tracking relationships between attributes that have a lot of different values. For example, you could create a DynamoDB table with CustomerID as the primary partition key for the table and ZipCode as the partition key for a global secondary index, since there are a lot of zip codes and since you will probably have a lot of customers. Using the primary key, you could quickly get the record for any customer. Using the global secondary index, you could efficiently query for all customers that live in a given zip code.
To ensure that you get the most out of your global secondary index’s capacity, please review our best practices documentation on uniform workloads.
How do I create a global secondary index for a DynamoDB table?
Global secondary indexes
Amazon DynamoDB | Database
GSIs associated with a table can be specified at any time. For detailed steps on creating a Table and its indexes, see here. You can create a maximum of 5 global secondary indexes per table.
Does the local version of DynamoDB support global secondary indexes?
Global secondary indexes
Amazon DynamoDB | Database
Yes. The local version of DynamoDB is useful for developing and testing DynamoDB-backed applications. You can download the local version of DynamoDB here.
What are projected attributes?
Global secondary indexes
Amazon DynamoDB | Database
The data in a secondary index consists of attributes that are projected, or copied, from the table into the index. When you create a secondary index, you define the alternate key for the index, along with any other attributes that you want to be projected in the index. Amazon DynamoDB copies these attributes into the index, along with the primary key attributes from the table. You can then query the index just as you would query a table.
Can a global secondary index key be defined on non-unique attributes?
Global secondary indexes
Amazon DynamoDB | Database
Yes. Unlike the primary key on a table, a GSI index does not require the indexed attributes to be unique. For instance, a GSI on GameTitle could index all items that track scores of users for every game. In this example, this GSI can be queried to return all users that have played the game “TicTacToe.”
How do global secondary indexes differ from local secondary indexes?
Global secondary indexes
Amazon DynamoDB | Database
Both global and local secondary indexes enhance query flexibility. An LSI is attached to a specific partition key value, whereas a GSI spans all partition key values. Since items having the same partition key value share the same partition in DynamoDB, the “Local” Secondary Index only covers items that are stored together (on the same partition). Thus, the purpose of the LSI is to query items that have the same partition key value but different sort key values. For example, consider a DynamoDB table that tracks Orders for customers, where CustomerId is the partition key.
An LSI on OrderTime allows for efficient queries to retrieve the most recently ordered items for a particular customer.
In contrast, a GSI is not restricted to items with a common partition key value. Instead, a GSI spans all items of the table just like the primary key. For the table above, a GSI on ProductId can be used to efficiently find all orders of a particular product. Note that in this case, no GSI sort key is specified, and even though there might be many orders with the same ProductId, they will be stored as separate items in the GSI.
In order to ensure that data in the table and the index are co-located on the same partition, LSIs limit the total size of all elements (tables and indexes) to 10 GB per partition key value. GSIs do not enforce data co-location, and have no such restriction.
When you write to a table, DynamoDB atomically updates all the LSIs affected. In contrast, updates to any GSIs defined on the table are eventually consistent.
LSIs allow the Query API to retrieve attributes that are not part of the projection list. This is not supported behavior for GSIs.
How do global secondary indexes work?
Global secondary indexes
Amazon DynamoDB | Database
In many ways, GSI behavior is similar to that of a DynamoDB table. You can query a GSI using its partition key element, with conditional filters on the GSI sort key element. However, unlike a primary key of a DynamoDB table, which must be unique, a GSI key can be the same for multiple items. If multiple items with the same GSI key exist, they are tracked as separate GSI items, and a GSI query will retrieve all of them as individual items. Internally, DynamoDB will ensure that the contents of the GSI are updated appropriately as items are added, removed or updated.
DynamoDB stores a GSI’s projected attributes in the GSI data structure, along with the GSI key and the matching items’ primary keys. GSI’s consume storage for projected items that exist in the source table. This enables queries to be issued against the GSI rather than the table, increasing query flexibility and improving workload distribution. Attributes that are part of an item in a table, but not part of the GSI key, primary key of the table, or projected attributes are thus not returned on querying the GSI index. Applications that need additional data from the table after querying the GSI, can retrieve the primary key from the GSI and then use either the GetItem or BatchGetItem APIs to retrieve the desired attributes from the table. As GSI’s are eventually consistent, applications that use this pattern have to accommodate item deletion (from the table) in between the calls to the GSI and GetItem/BatchItem.
DynamoDB automatically handles item additions, updates and deletes in a GSI when corresponding changes are made to the table. When an item (with GSI key attributes) is added to the table, DynamoDB updates the GSI asynchronously to add the new item. Similarly, when an item is deleted from the table, DynamoDB removes the item from the impacted GSI.
Can I create global secondary indexes for partition-based tables and partition-sort schema tables?
Global secondary indexes
Amazon DynamoDB | Database
Yes, you can create a global secondary index regardless of the type of primary key the DynamoDB table has. The table’s primary key can include just a partition key, or it may include both a partition key and a sort key.
What is the consistency model for global secondary indexes?
Global secondary indexes
Amazon DynamoDB | Database
GSIs support eventual consistency. When items are inserted or updated in a table, the GSIs are not updated synchronously. Under normal operating conditions, a write to a global secondary index will propagate in a fraction of a second. In unlikely failure scenarios, longer delays may occur. Because of this, your application logic should be capable of handling GSI query results that are potentially out-of-date. Note that this is the same behavior exhibited by other DynamoDB APIs that support eventually consistent reads.
Consider a table tracking top scores where each item has attributes UserId, GameTitle and TopScore. The partition key is UserId, and the primary sort key is GameTitle. If the application adds an item denoting a new top score for GameTitle “TicTacToe” and UserId “GAMER123,” and then subsequently queries the GSI, it is possible that the new score will not be in the result of the query. However, once the GSI propagation has completed, the new item will start appearing in such queries on the GSI.
Can I provision throughput separately for the table and for each global secondary index?
Global secondary indexes
Amazon DynamoDB | Database
Yes. GSIs manage throughput independently of the table they are based on. When you enable Auto Scaling for a new or existing table from the console, you can optionally choose to apply the same settings to GSIs. You can also provision different throughput for tables and global secondary indexes manually.
Depending upon on your application, the request workload on a GSI can vary significantly from that of the table or other GSIs. Some scenarios that show this are given below:
A GSI that contains a small fraction of the table items needs a much lower write throughput compared to the table.
A GSI that is used for infrequent item lookups needs a much lower read throughput, compared to the table.
A GSI used by a read-heavy background task may need high read throughput for a few hours per day.
As your needs evolve, you can change the provisioned throughput of the GSI, independently of the provisioned throughput of the table.
Consider a DynamoDB table with a GSI that projects all attributes, and has the GSI key present in 50% of the items. In this case, the GSI’s provisioned write capacity units should be set at 50% of the table’s provisioned write capacity units. Using a similar approach, the read throughput of the GSI can be estimated. Please see DynamoDB GSI Documentation for more details.
How does adding a global secondary index impact provisioned throughput and storage for a table?
Global secondary indexes
Amazon DynamoDB | Database
Similar to a DynamoDB table, a GSI consumes provisioned throughput when reads or writes are performed to it. A write that adds or updates a GSI item will consume write capacity units based on the size of the update. The capacity consumed by the GSI write is in addition to that needed for updating the item in the table.
Note that if you add, delete, or update an item in a DynamoDB table, and if this does not result in a change to a GSI, then the GSI will not consume any write capacity units. This happens when an item without any GSI key attributes is added to the DynamoDB table, or an item is updated without changing any GSI key or projected attributes.
A query to a GSI consumes read capacity units, based on the size of the items examined by the query.
Storage costs for a GSI are based on the total number of bytes stored in that GSI. This includes the GSI key and projected attributes and values, and an overhead of 100 bytes for indexing purposes.
Can DynamoDB throttle my application writes to a table because of a GSI’s provisioned throughput?
Global secondary indexes
Amazon DynamoDB | Database
Because some or all writes to a DynamoDB table result in writes to related GSIs, it is possible that a GSI’s provisioned throughput can be exhausted. In such a scenario, subsequent writes to the table will be throttled. This can occur even if the table has available write capacity units.
How often can I change provisioned throughput at the index level?
Global secondary indexes
Amazon DynamoDB | Database
Tables with GSIs have the same daily limits on the number of throughput change operations as normal tables.
How am I charged for DynamoDB global secondary index?
Global secondary indexes
Amazon DynamoDB | Database
You are charged for the aggregate provisioned throughput for a table and its GSIs by the hour. When you provision manually, while not required, You are charged for the aggregate provisioned throughput for a table and its GSIs by the hour. In addition, you are charged for the data storage taken up by the GSI as well as standard data transfer (external) fees. If you would like to change your GSI’s provisioned throughput capacity, you can do so using the DynamoDB Console or the UpdateTable API or the PutScalingPolicy API for updating Auto Scaling policy settings.
Can I specify which global secondary index should be used for a query?
Global secondary indexes
Amazon DynamoDB | Database
Yes. In addition to the common query parameters, a GSI Query command explicitly includes the name of the GSI to operate against. Note that a query can use only one GSI.
What API calls are supported by a global secondary index?
Global secondary indexes
Amazon DynamoDB | Database
The API calls supported by a GSI are Query and Scan. A Query operation only searches index key attribute values and supports a subset of comparison operators. Because GSIs are updated asynchronously, you cannot use the ConsistentRead parameter with the query. Please see here for details on using GSIs with queries and scans.
What is the order of the results in scan on a global secondary index?
Global secondary indexes
Amazon DynamoDB | Database
For a global secondary index, with a partition-only key schema there is no ordering. For global secondary index with partition-sort key schema the ordering of the results for the same partition key is based on the sort key attribute.
