How to start with DynamoDB in your Ktor application (Part 1)

After working mostly with relation databases in my career, in the past nearly 8 months I started heavily using DynamoDB in one of my projects that run on production. Starting with DynamoDB was very easy but in order to profit from it advantages it is necessary to learn some different concepts comparing to relation databases. In this post, I explore the basics of DynamoDB and demonstrate how to integrate it in a Ktor application. I walk through setting up a DynamoDB connection, create a table and performing basic CRUD (Create, Read, Update, Delete) operations using AWS SDK. To write more idiomatic Kotlin code, I extend the usage of the AWS SDK by an additional library.

This post only covers the absolut basics to start with DynamoDB. The things you will learn will be the foundation to start with more advanced topics like modeling the tables (also using composite keys), using converters for custom data types, creating GSI (Global Secondary Index) for querying large amount of data in a performant way and using batch processing for fast reading/writing data.

Before diving into code let’s first answer the question of what DynamoDB is.

What is DynamoDB?

Amazon DynamoDB is a fully managed NoSQL database service provided by AWS that offers fast and predictable performance with seamless scalability. It stores data in key-value and document formats, making it ideal for handling large amounts of structured or semi-structured data. DynamoDB automatically manages data replication, backups, and recovery across multiple AWS regions, providing high availability and fault tolerance.

Why using DynamoDB?

There are a lot of benefits for using a DynamoDB database.

• Fully Managed and Scalable: No server management, AWS handles all infrastructure tasks. Automatic scaling adjusts capacity based on demand.

• Flexible Data Model NoSQL database with key-value and document data models. Query data via primary keys or secondary indexes.

• Backup, Restore, and TTL On-demand backups, point-in-time recovery, and automatic expiration with Time-to-Live (TTL).

• High Performance and Low Latency Single-digit millisecond response times. Horizontally scales to maintain performance at any scale.

• ACID Transactions Supports atomic multi-item transactions ensuring data consistency.

To see how DynamoDB can be used, let’s have a look on how it can be integrated.

Prerequisites

To start with DynamoDB there are 3 things necessary:

1. A running DynamoDB database to access.

2. A Ktor application that wants to access the database for writing and reading data.

3. The AWS SDK to connect to the database.

This is the time for diving into code…​

DynamoDB

In production DynamoDB is used as an AWS service, but to not make it necessary to create an AWS account and to spend money for using AWS services, I decide to use a locally running DynamoDB database.

In the official DynamoDB documentation of AWS you can find multiple options for running DynamoDB as a local service. For this article I choose the Docker variant and create a docker-compose.yml file that contains the configuration:

version: '3.8'
services:
  dynamodb-local:
    command: "-jar DynamoDBLocal.jar -sharedDb -dbPath ./data"
    image: "amazon/dynamodb-local:latest"
    container_name: dynamodb-local
    ports:
      - "8000:8000"
    volumes:
      - "./docker/dynamodb:/home/dynamodblocal/data"
    working_dir: /home/dynamodblocal

Before proceeding, I check if the access to the DynamoDB local instance is possible. For this I can use the AWS CLI. If you don’t have it already installed, you can find the instructions in the official documentation.

First start the local DynamoDB:

docker-compose up

Then check for available tables in the database:

aws dynamodb list-tables --endpoint-url http://localhost:8000

The parameter endpoint-url is important so that it is not tried to access AWS remotely.

The command should print out

{
    "TableNames": []
}

because I have not created a table yet, but the connection is successful. That is enough for now.

Ktor

The next step is to set up the Ktor application, which can be done by using the Ktor starter or the project wizard in IntelliJ.

The resulting build.gradle.kts file should look like below:

val kotlin_version: String by project
val logback_version: String by project

plugins {
    kotlin("jvm") version "2.0.20"
    id("io.ktor.plugin") version "3.0.0-rc-1"
    kotlin("plugin.serialization") version "2.0.20"
}

group = "com.poisonedyouth"
version = "0.0.1"

application {
    mainClass.set("com.poisonedyouth.ApplicationKt")

    val isDevelopment: Boolean = project.ext.has("development")
    applicationDefaultJvmArgs = listOf("-Dio.ktor.development=$isDevelopment")
}

repositories {
    mavenCentral()
    maven { setUrl("https://jitpack.io" )}
}

dependencies {
    implementation("io.ktor:ktor-server-core-jvm")
    implementation("io.ktor:ktor-server-cio-jvm")
    implementation("ch.qos.logback:logback-classic:$logback_version")

    implementation("io.ktor:ktor-server-content-negotiation")
    implementation("io.ktor:ktor-serialization-kotlinx-json")

    testImplementation("io.ktor:ktor-server-test-host-jvm")
    testImplementation("org.jetbrains.kotlin:kotlin-test-junit:$kotlin_version")
}

There is nothing special. I choose the CIO server engine and kotlinx.serialization for the handling of the serialization and deserialization from and to JSON. This is necessary for the HTTP endpoints that I use for testing the usage of the DynamoDB.

AWS SDK

The last step of the prerequisites is the connection of the Ktor application with the running local DynamoDB database. For this it is necessary to include additional entries to the dependencies section:

dependencies {
    implementation("software.amazon.awssdk:dynamodb-enhanced:2.28.1")
    implementation("software.amazon.awssdk:dynamodb:2.28.1")
    implementation("dev.andrewohara:dynamokt:1.0.0")
}

The first two are provided by AWS to set up a client for the connection and the third one is an extension to be able to communicate with DynamoDB in a more Kotlin idiomatic way by allowing to use data classes with immutable properties.

Implementation

Now that the prerequisites are finished I can start with the implementation of the sample application, that I use to demonstrate the usage of the DynamoDB together with Ktor.

Step 1: Define the Domain Model

I define a simple Kotlin data class to represent the product model. The properties of the Product are modelled as value classes to include some validation logic.

@Serializable
data class Product(
    val productId: ProductId,
    val productName: ProductName,
    val price: Price
)

@JvmInline
@Serializable
value class ProductId(val value: String){
    init {
        require(value.isNotBlank()) { "Product Id cannot be blank" }
        require(value.length == 16) { "Product Id must be 16 characters" }
    }
}

@JvmInline
@Serializable
value class ProductName(val value: String){
    init {
        require(value.isNotBlank()) { "Product Name cannot be blank" }
        require(value.length <= 32) { "Product Name cannot be longer than 32 characters" }
    }
}

@JvmInline
@Serializable
value class Price(val value: Double) {
    init {
        require(value >= 0.0) { "Price must be positive." }
    }
}

To separate the persistence from the domain model I create a separate data class that represents an entry in the DynamoDb table using data types that are supported.

data class ProductEntity(
    @DynamoKtPartitionKey
    val productId: String,
    val productName: String,
    val price: Double
)

There is 1 annotation necessary for using this class as an entity.

• @DynamoKtPartitionKey: This identifies the partition key of the table. This is necessary for every DynamoDB table. It is also possible to provide a composite key but that is out of scope of this post.

Step 2: Create the DynamoDB Client

For the connection to the database I create a low-level client instance. This instance provides basic functionality like creation of tables and operations for adding, updating, deleting and retrieving data. There are 2 client variants available:

• DynamoDbClient

• DynamoDbAsyncClient

The first one is a synchronous variant that blocks until the request is finished, the second one is an asynchronous version that is perfectly working together with Coroutines. Because I used the Ktor CIO engine for this example application I create a DynamoDbAsyncClient.

fun Application.createDynamoDbClient(): DynamoDbAsyncClient {
    val url = environment.config.property("ktor.database.dynamodbUrl").getString()

    return DynamoDbAsyncClient.builder()
        .endpointOverride(URI(url)) // Local DynamoDB
        .build()
}

You may wonder where the credentials for the access to the database is configured. I use a local running DynamoDB that does not need any credentials. In a productive environment a credentials provider need to be configured that contains the credentials for accessing AWS.

DynamoDbAsyncClient.builder()
    .credentialsProvider {
        TODO()
    }
    .endpointOverride(URI(url))
    .build()

Using the DynamoDbClient directly is very inconvenient and makes it necessary to write a lot of boilerplate code, so to reduce this, the AWS SDK also provides an enhanced variant of the client.

fun createEnhancedDynamoDbClient(dynamoDbClient: DynamoDbAsyncClient): DynamoDbEnhancedAsyncClient {
    return DynamoDbEnhancedAsyncClient.builder()
        .dynamoDbClient(dynamoDbClient)
        .build()
}

The enhanced client wraps the DynamoDbAsyncClient.

Step 3: Creating a DynamoDB Table

Next, I need to create a table in the DynamoDB to store the products.I use the createTable() function for this. Because an exception is thrown if I try to create an already existing table again, I need to check first for all available tables.This is one of the API calls that is not available for the DynamoDbEnhancedAsyncClient so I need the low-level variant for this.

suspend fun createNecessaryTables(dynamoDbClient: DynamoDbAsyncClient, dynamoDbEnhancedClient: DynamoDbEnhancedAsyncClient) {
    val logger = LoggerFactory.getLogger(Application::class.java)

    val existingTables = dynamoDbClient.listTables().await().tableNames()

    val productEntity = ProductEntity::class
    val tableSchema = DataClassTableSchema(productEntity)
    if (existingTables.contains(productEntity.simpleName)) {
        logger.info("Table '${productEntity.simpleName}' already exists.")
    } else {
        dynamoDbEnhancedClient.table(productEntity.simpleName, tableSchema).createTable().await()
        logger.info("Table '${productEntity.simpleName}' created successfully.")
    }
}

The above code creates a table named ProductEntity with productId as the partition key.

In a productive environment it is not recommended to create the tables by the application but use i.e. Terraform for this task.

Step 4: Implement CRUD operations for product entity

Now that I have a table, let’s implement the operations to create, update, find and delete products.

Using the enhanced client makes this very convenient. I create a DataClassTableSchema using the ProductEntity type and can call the required CRUD operations on this instance. The calls with the client are asynchronous, so I need to call await() on every operation. This is an extension function that is provided by the kotlinx.coroutines library.

private val tableName = ProductEntity::class.simpleName
private val tableSchema = DataClassTableSchema(ProductEntity::class)

suspend fun add(product: Product): Unit = coroutineScope {
        table.putItem(product.toProductEntity()).await()
    }

    suspend fun findById(productId: String): Product? {
        return table.getItem(
            Key.builder().partitionValue(productId).build()
        ).await()?.toProduct()
    }

     suspend fun findAll(): List<Product> {
        return buildList {
            table.scan().subscribe { page ->
                page.items().stream().forEach { item -> add(item.toProduct()) }
            }.await()
        }
    }

    suspend fun deleteById(productId: String) {
        table.deleteItem(Key.builder().partitionValue(productId).build()).await()
    }

    suspend fun update(product: Product) {
        table.updateItem(product.toProductEntity()).await()
    }

That’s it. This ist very simple and by wrapping the client in a repository class I can separate the access to the DynamoDB database from the rest of the application code.

Step 5: Provide REST endpoints.

In the last step I need to provide some REST endpoints to test if the repository implementation is working as expected.

Below, you can find the functionality for a POST endpoint that allows to create new products.

Service:

class ProductService(
    private val productRepository: ProductRepository
) {

    suspend fun addProduct(product: Product) {
        val existingProduct = productRepository.findById(product.productId.value)
        if (existingProduct != null) {
            error("Product with id ${product.productId} already exists.")
        }
        productRepository.add(product)
    }
    //...
}

Routing:

fun Application.configureRouting(productService: ProductService) {
    routing {
         post("/product") {
            productService.addProduct(call.receive())
            call.respond(HttpStatusCode.Created)
        }
    //...
    }
}

To keep things simple I omit to introduce Koin for the dependency injection, but manually inject the dependencies.

fun Application.module() = runBlocking {
    val dynamoDbClient = createDynamoDbClient()
    val dynamoDbEnhancedClient = createEnhancedDynamoDbClient(dynamoDbClient)
    createNecessaryTables(dynamoDbClient, dynamoDbEnhancedClient)

    configureSerialization()

    val productRepository = ProductRepository(dynamoDbEnhancedClient)
    val productService = ProductService(productRepository)
    configureRouting(productService)
}

With this the application is complete and I can create some requests to check if everything works as expected. For this an IntelliJ scratch file is an easy way.

POST http://localhost:8080/product
Content-Type: application/json

{
 "productId" : "1111111111111111",
  "productName" : "Testproduct",
  "price": 2.32
}

###
POST http://localhost:8080/product
Content-Type: application/json

{
  "productId" : "1111111111111112",
  "productName" : "Testproduct2",
  "price": 5.12
}

###
GET http://localhost:8080/product/1111111111111111
Accept: application/json

###

GET http://localhost:8080/product
Accept: application/json

###
PUT localhost:8080/product
Content-Type: application/json

{
  "productId" : "1111111111111112",
  "productName" : "Testproduct",
  "price": 12.12
}

###
DELETE localhost:8080/product/1234

Executing the POST request returns the expected result:

POST http://localhost:8080/product

HTTP/1.1 201 Created
Content-Length: 0

<Response body is empty>

Response code: 201 (Created); Time: 473ms (473 ms); Content length: 0 bytes (0 B)

That’s it for today.

Conclusion

Today I showed how easy DynamoDB can be used as persistence storage for Kotlin applications, using Ktor as an example. AWS provides an SDK that allows to connect to the database in either synchronous or asynchronous way. The available client provides all necessary functionality for the classical CRUD operations. Comparing to use a relational database using the classical JDBC connection the DynamoDB can be used in a similar way. The dynamodb-kotlin-module makes it very convenient to use the AWS SDK, that is written in Java, in a Kotlin idiomatic way.

This post only covers the absolut surface of working with DynamoDB. As already mentioned at the beginning there are a lot of more advanced topics available, that are necessary to know when using the database in a productive environment.

In the next post I will continue with the following topics:

• Composite keys for tables.

• Using GSI (global secondary index).

• Use converters for custom data types.

• Use batch processing for read and write operations.

• Filter expressions and query conditionals.

• Using TTL.

---

You can find the full code that is used for this article on Github.