In the first part of this series, I explained physical partitions, their theoretical maximum, reasons for creating multiple partitions, RU distribution, and key metrics.
In this post, the focus will be on Logical Partitions, addressing common questions such as:
- What is a Logical Partition?
- What is a Partition Key?
- What is the Max Size?
- What is the impact on Physical Partitions?
Let’s explore these concepts with a real-time streaming scenario.
Scenario
Suppose an energy company (Oil and Gas) has millions of sensors streaming data to local data loggers. These loggers, located in different geographical areas, stream data to a centralized Kafka or IoT Hub. A service listens to this data, processes it, and stores it in CosmosDB.
Each message in Kafka or IoT Hub is in JSON format, as shown below:
| Name | Description |
|---|---|
deviceId | Sensor or Device ID |
deviceName | Sensor or Device Name |
dataLoggerId | Data Logger transmitting the data |
plantId | Plant where the Data Logger is connected |
plantName | Friendly name of the Plant |
time | UTC time of the telemetry in ISO 8601 format |
ticks | Ticks when the telemetry was generated |
epoch | Epoch time (seconds) of the telemetry |
location | Latitude, Longitude, and Accuracy |
device-data | Type, Measure Unit, Measure Value |
JSON Example:
{
"deviceId": "a7d96e7e-16ae-400e-a616-e84dba0d8633",
"deviceName": "TEMP-ALBA-01",
"dataLoggerId": "0f472328-7214-4bbd-9789-50519f44bb23",
"plantId": "PLANT-1",
"plantName": "North-Alba-01",
"time": "2022-07-19T10:10:11.5091350Z",
"ticks": 637097550115091350,
"epoch": 1658251025,
"deviceData": {
"type": "temp-sensor",
"measureUnitDescription": "degrees Celsius",
"measureUnit": "C",
"measureValue": 21
},
"location": {
"latitude": "35.063375",
"accuracy": "0.0",
"longitude": "-89.963738"
}
}
Scenario Details
Messages per second: ~10,000
Message size: 1–2 KB
Data retention: 30 days (TTL: 30 days)
Estimated storage:
- ~70 GB/day
- ~2 TB/month
What is a Logical Partition?
A logical partition consists of items sharing the same partition key. The maximum size for a logical partition is 20 GB (subject to change).
What is a Partition Key?
A partition key has two components:
Partition Key Path: The property in the JSON document used as the key.
Partition Key Value: The value associated with the key path.
Partition keys can be strings or numeric types and support alphanumeric and underscore characters (_). Nested objects can use standard path notation (/).
How Can We Decide the Partition Key in This Scenario?
Given the write-intensive nature of this scenario, careful partition key selection is crucial. Assume the company operates 15 plants, with one plant generating 25% of the total traffic.
Additionally, an application consuming CosmosDB data allows users to filter by plant, data logger, or sensor.
Alternative 1: Using plantId as the Partition Key
"plantId": "PLANT-1"
Advantages:
- Groups all data for a plant within the same logical partition, avoiding cross-partition queries.
Disadvantages:
- Limited logical partitions (only 15, one for each plant).
- Maximum logical partition size is 20 GB. With 15 plants, the total capacity is 300 GB, far below the 2 TB needed.
Conclusion: plantId is not a suitable partition key.
Alternative 2: Using plantId + CurrentDate
"partitionKey": "PLANT-1-20220819"
In this approach, data is still grouped by Plant ID, allowing us to query all records for the current day or prior days. This structure is advantageous as it helps avoid cross-partition queries, improving query efficiency.
Now, let’s analyze the partition size. Based on the scenario, the storage requirement is approximately 70 GB per day, with one plant contributing 25% of the total traffic. This means a single plant will generate around 17.5 GB of data per day.
Since 17.5 GB < 20 GB, the size of a logical partition stays within the allowed limit. However, let’s examine the impact this design has on physical partitioning behavior.
The image highlights two physical partitions that are fully utilized (marked in red) due to the presence of a large logical partition. If additional data continues to be written to this logical partition, the Cosmos DB engine will need to redistribute it by moving some logical partitions to other physical partitions with available space. This rebalancing process is necessary to free up space in the over-utilized physical partition where new data is being added.
This situation is called Hot partition, and it can be observed on insights metrics for Cosmos DB
Summary of this scenario
Advantages:
- Groups data by plant and day, reducing the size of each logical partition.
- Avoids cross-partition queries for daily queries.
Disadvantages:
- Potential for hot partitions: If one plant generates significant traffic, logical partitions may fill up, triggering repartitioning.
Conclusion: This is better than plantId but not ideal.
What is the Right Partition Key?
For heavy-write scenarios, prioritize small logical partitions for faster insertion. A UUID or a more granular composite key (e.g., plantId + epoch) is preferable.
Recommendations:
Custom Indexing: Define a custom indexing policy instead of indexing all properties.
Monitor Metrics: Use Log Analytics to track partition size.
Analyzing Partition Key Size
Microsoft provides a Log Analytics query to analyze partition key sizes:
AzureDiagnostics
| where Category == "PartitionKeyStatistics"
| summarize arg_max(TimeGenerated, *) by databaseName_s, collectionName_s, partitionKey_s, _ResourceId
| extend utilizationOf20GBLogicalPartition = sizeKb_d / 20000000
| project TimeGenerated, databaseName_s, collectionName_s, partitionKey_s, sizeKb_d, utilizationOf20GBLogicalPartition, _ResourceId
Sample Output:
sizeKb_d: Logical partition size (e.g., 6.26 GB).
utilizationOf20GBLogicalPartition: Utilization percentage (e.g., 30%).
Conclusion
Choosing the right partition key is critical for optimal performance in Azure CosmosDB. For write-intensive scenarios like this, selecting a key that evenly distributes data across logical partitions is essential to prevent hot partitions and ensure scalability. Proper monitoring and indexing strategies can further enhance performance and reduce costs.