Amazon DynamoDB | Storage backend for Titan

Question 1

How do I download and install the Amazon DynamoDB Logstash Plugin?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 1

The Amazon DynamoDB Logstash Plugin is available on GitHub. Read our documentation page to learn more about installing and running the plugin.

Question 2

What is the DynamoDB Storage Backend for Titan?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 2

The DynamoDB Storage Backend for Titan is a plug-in that allows you to use DynamoDB as the underlying storage layer for Titan graph database. It is a client side solution that implements index free adjacency for fast graph traversals on top of DynamoDB.

Question 3

What is a graph database?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 3

A graph database is a store of vertices and directed edges that connect those vertices. Both vertices and edges can have properties stored as key-value pairs.

A graph database uses adjacency lists for storing edges to allow simple traversal. A graph in a graph database can be traversed along specific edge types, or across the entire graph. Graph databases can represent how entities relate by using actions, ownership, parentage, and so on.

Question 4

What applications are well suited to graph databases?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 4

Whenever connections or relationships between entities are at the core of the data you are trying to model, a graph database is a natural choice. Therefore, graph databases are useful for modeling and querying social networks, business relationships, dependencies, shipping movements, and more.

Question 5

How do I get started using the DynamoDB Storage Backend for Titan?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 5

The easiest way to get started is to launch an EC2 instance running Gremlin Server with the DynamoDB Storage Backend for Titan, using the CloudFormation templates referred to in this documentation page. You can also clone the project from the GitHub repository and start by following the Marvel and Graph-Of-The-Gods tutorials on your own computer by following the instructions in the documentation here. When you’re ready to expand your testing or run in production, you can switch the backend to use the DynamoDB service. Please see the AWS documentation for further guidance.

Question 6

How does the DynamoDB Storage Backend differ from other Titan storage backends?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 6

DynamoDB is a managed service, thus using it as the storage backend for Titan enables you to run graph workloads without having to manage your own cluster for graph storage.

Question 7

Is the DynamoDB Storage Backend for Titan a fully managed service?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 7

No. The DynamoDB storage backend for Titan manages the storage layer for your Titan workload. However, the plugin does not do provisioning and managing of the client side. For simple provisioning of Titan we have developed a CloudFormation template that sets up DynamoDB Storage Backend for Titan with Gremlin Server; see the instructions available here.

Question 8

How much does using the DynamoDB Storage Backend for Titan cost?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 8

You are charged the regular DynamoDB throughput and storage costs. There is no additional cost for using DynamoDB as the storage backend for a Titan graph workload.

Question 9

Does DynamoDB backend provide full compatibility with the Titan feature set on other backends?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 9

A table comparing feature sets of different Titan storage backends is available in the documentation.

Question 10

Which versions of Titan does the plugin support?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 10

We have released DynamoDB storage backend plugins for Titan versions 0.5.4 and 1.0.0.

Question 11

I use Titan with a different backend today. Can I migrate to DynamoDB?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 11

Absolutely. The DynamoDB Storage Backend for Titan implements the Titan KCV Store interface so you can switch from a different storage backend to DynamoDB with minimal changes to your application. For full comparison of storage backends for Titan please see our documentation.

Question 12

I use Titan with a different backend today. How do I migrate to DynamoDB?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 12

You can use bulk loading to copy your graph from one storage backend to the DynamoDB Storage Backend for Titan.

Question 13

How do I connect my Titan instance to DynamoDB via the plugin?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 13

If you create a graph and Gremlin server instance with the DynamoDB Storage Backend for Titan installed, all you need to do to connect to DynamoDB is provide a principal/credential set to the default AWS credential provider chain. This can be done with an EC2 instance profile, environment variables, or the credentials file in your home folder. Finally, you need to choose a DynamoDB endpoint to connect to.

Question 14

How durable is my data when using the DynamoDB Storage Backend for Titan?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 14

When using the DynamoDB Storage Backend for Titan, your data enjoys the strong protection of DynamoDB, which runs across Amazon’s proven, high-availability data centers. The service replicates data across three facilities in an AWS Region to provide fault tolerance in the event of a server failure or Availability Zone outage.

Question 15

How secure is the DynamoDB Storage Backend for Titan?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 15

The DynamoDB Storage Backend for Titan stores graph data in multiple DynamoDB tables, thus is enjoys the same high security available on all DynamoDB workloads. Fine-Grained Access Control, IAM roles, and AWS principal/credential sets control access to DynamoDB tables and items in DynamoDB tables.

Question 16

How does the DynamoDB Storage Backend for Titan scale?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 16

The DynamoDB Storage Backend for Titan scales just like any other workload of DynamoDB. You can choose to increase or decrease the required throughput at any time.

Question 17

How many vertices and edges can my graph contain?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 17

You are limited by Titan’s limits for (2^60) for the maximum number of edges and half as many vertices in a graph, as long as you use the multiple-item model for edgestore. If you use the single-item model, the number of edges that you can store at a particular out-vertex key is limited by DynamoDB’s maximum item size, currently 400kb.

Question 18

How large can my vertex and edge properties get?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 18

The sum of all edge properties in the multiple-item model cannot exceed 400kb, the maximum item size. In the multiple item model, each vertex property can be up to 400kb. In the single-item model, the total item size (including vertex properties, edges and edge properties) can’t exceed 400kb.

Question 19

How many data models are there? What are the differences?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 19

There are two different storage models for the DynamoDB Storage Backend for Titan – single item model and multiple item model. In the single item storage model, vertices, vertex properties, and edges are stored in one item. In the multiple item data model, vertices, vertex properties and edges are stored in different items. In both cases, edge properties are stored in the same items as the edges they correspond to.

Question 20

Which data model should I use?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 20

In general, we recommend you use the multiple-item data model for the edgestore and graphindex tables. Otherwise, you either limit the number of edges/vertex-properties you can store for one out-vertex, or you limit the number of entities that can be indexed at a particular property name-value pair in graph index. In general, you can use the single-item data model for the other 4 KCV stores in Titan versions 0.5.4 and 1.0.0 because the items stored in them are usually less than 400KB each. For full list of tables that the Titan plugin creates on DynamoDB please see here.

Question 21

Do I have to create a schema for Titan graph databases?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 21

Titan supports automatic type creation, so new edge/vertex properties and labels will get registered on the fly (see here for details) with the first use. The Gremlin Structure (Edge labels=MULTI, Vertex properties=SINGLE) is used by default.

Question 22

Can I change the schema of a Titan graph database?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 22

Yes, however, you cannot change the schema of existing vertex/edge properties and labels. For details please see here.

Question 23

How does the DynamoDB Storage Backend for Titan deal with supernodes?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 23

DynamoDB deals with supernodes via vertex label partitioning. If you define a vertex label as partitioned in the management system upon creation, you can key different subsets of the edges and vertex properties going out of a vertex at different partition keys of the partition-sort key space in the edgestore table. This usually results in the virtual vertex label partitions being stored in different physical DynamoDB partitions, as long as your edgestore has more than one physical partition. To estimate the number of physical partitions backing your edgestore table, please see guidance in the documentation.

Question 24

Does the DynamoDB Storage Backend for Titan support batch graph operations?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 24

Yes, the DynamoDB Storage Backend for Titan supports batch graph with the Blueprints BatchGraph implementation and through Titan’s bulk loading configuration options.

Question 25

Does the DynamoDB Storage Backend for Titan support transactions?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 25

The DynamoDB Storage Backend for Titan supports optimistic locking. That means that the DynamoDB Storage Backend for Titan can condition writes of individual Key-Column pairs (in the multiple item model) or individual Keys (in the single item model) on the existing value of said Key-Column pair or Key.

Question 26

Can I have a Titan instance in one region and access DynamoDB in another?

Storage backend for Titan

Amazon DynamoDB | Database

Answer 26

Accessing a DynamoDB endpoint in another region than the EC2 Titan instance is possible but not recommended. When running a Gremlin Server out of EC2, we recommend connecting to the DynamoDB endpoint in your EC2 instance’s region, to reduce the latency impact of cross-region requests. We also recommend running the EC2 instance in a VPC to improve network performance. The CloudFormation template performs this entire configuration for you.