Efficiently querying vast amounts of data in real-time is essential for gaining actionable insights and making informed decisions. Amazon OpenSearch Service combined with Amazon S3 provides a powerful solution for organizations looking to leverage their data effectively. This comprehensive guide is the first-ever detailed walkthrough that covers every step of integrating these powerful AWS services, ensuring you can implement and optimize direct queries with ease.
Table of Contents
- High-Level Architecture
- Prerequisites
Step-by-Step Implementation
- 1. Creating an Amazon S3 Bucket
- 2. Setting Up AWS Glue Data Catalog
- 3. Configuring IAM Roles and Policies
- 4. Creating an Amazon OpenSearch Service Domain
- 5. Creating Spark Tables Using Query Workbench
- 6. Implementing Accelerations
- Best Practices
- Testing and Validation
- Troubleshooting
- Conclusion
- References
- Glossary
- Appendices
- Custom Open Graph (OG) Image
High-Level Architecture
Step 2: Create a New Bucket
- Click on "Create bucket".
Configure Bucket Settings:
Bucket name:opensearch-s3-poc-json-bucket(must be unique across all AWS accounts).
Region: Select the same AWS Region as your OpenSearch domain (e.g.,us-east-1).
Set Object Ownership:
Recommended: Set to Bucket owner preferred.
Block Public Access:
Enable: Ensure all options are checked to block public access.
Additional Settings:
- Configure Versioning, Tags, Default encryption as per requirements.
Review and Create:
- Click Create bucket after reviewing all settings.
Step 3: Upload Sample Data to S3
- Open the newly created bucket
opensearch-s3-poc-json-bucket.
Create a Folder (Optional):
- Example:
transactions/for better organization.
- Example:
Upload Files:
- Click Upload.
Add Files: Select and upload yourtransactions.json.- Click Upload.
Best Practice: Organize data using prefixes (folders) like transactions/ and ensure data consistency.
2. Setting Up AWS Glue Data Catalog
Step 1: Navigate to the AWS Glue Console
URL:
Step 2: Create an IAM Policy
- In the IAM console, click on "Policies" in the left navigation pane.
- Click "Create policy".
- Select the JSON tab.
Paste the following policy, replacing
<account-id>and<opensearch-domain-name>with your actual AWS Account ID and desired OpenSearch domain name.
CODE{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "OpenSearchDomainPermissions",
"Effect": "Allow",
"Action": "es:ESHttp*",
"Resource": "arn:aws:es:us-east-1:<account-id>:domain/<opensearch-domain-name>/*"
},
{
"Sid": "S3ReadAccess",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:GetObjectVersion",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::opensearch-s3-poc-json-bucket",
"arn:aws:s3:::opensearch-s3-poc-json-bucket/*"
]
},
{
"Sid": "GluePermissions",
"Effect": "Allow",
"Action": [
"glue:GetDatabase",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions"
],
"Resource": "*"
},
{
"Sid": "S3CheckpointAccess",
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource": [
"arn:aws:s3:::opensearch-s3-poc-json-bucket/checkpoint/transactions/",
"arn:aws:s3:::opensearch-s3-poc-json-bucket/checkpoint/transactions/*"
]
}
]
}
Click "Next: Tags" (optional).
Click "Next: Review".
Name:
OpenSearchS3AccessPolicy.Description:
Policy granting OpenSearch access to S3 and Glue.Click "Create policy".
Step 3: Create an IAM Role
- In the IAM console, click on "Roles" in the left navigation pane.
- Click "Create role".
Trusted Entity: Select "Custom trust policy".- Click "Switch to trusted entity".
Paste the following trust policy:
CODE{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"opensearchservice.amazonaws.com",
"directquery.opensearchservice.amazonaws.com",
"glue.amazonaws.com"
]
},
"Action": "sts:AssumeRole"
}
]
}
Click "Next: Permissions".
Search for
OpenSearchS3AccessPolicy.Select the policy and click "Next: Tags".
Click "Next: Review".
Role name:
OpenSearchS3AccessRole.Description:
Role for OpenSearch to access S3 and Glue.Click "Create role".
Best Practice: Use clear and descriptive names for roles and policies for easier management.
4. Creating an Amazon OpenSearch Service Domain
Step 1: Navigate to the Amazon OpenSearch Service Console
URL:
Conclusion
Integrating Amazon OpenSearch Service with Amazon S3 for direct queries offers a robust solution for high-performance data analytics and real-time search capabilities. This integration leverages the scalability and durability of Amazon S3 for data storage, the schema management prowess of AWS Glue, and the advanced search and analytics features of OpenSearch Service.
Key Takeaways:
Streamlined Data Access: Direct queries enable efficient data retrieval from S3 without the need for data ingestion into OpenSearch indices.
Enhanced Performance: Implementing accelerations such as Skipping Indexes, Covering Indexes, and Materialized Views significantly improves query performance and reduces latency.
Scalability and Flexibility: The architecture supports scalability, allowing you to handle large volumes of data and complex queries seamlessly.
Security and Compliance: Adhering to best practices in IAM configurations and data encryption ensures your data remains secure and compliant with industry standards.
Cost Efficiency: Optimizing resource allocation and implementing data lifecycle policies help manage costs effectively without sacrificing performance.
By following this guide and adhering to the outlined best practices, you can establish an efficient, scalable, and secure data querying environment using Amazon OpenSearch Service and Amazon S3. This setup empowers your organization to perform real-time analytics, derive actionable insights, and drive informed decision-making with confidence.
References
Amazon S3 Documentation:
Amazon OpenSearch Service Documentation:
OpenSearch Query Workbench:
Monitoring and Observability:
Apache Spark SQL Documentation: , sharing it with your network, or leaving a comment below. Your feedback helps improve the content and supports the community!↗ Original-Artikel auf dev.to lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR