Welcome to issue #205 of the AWS open source newsletter, the newsletter where we try and provide you the best open source on AWS content. This will be the last edition of 2024, but looking forward to coming back in 2025 with more great open source project and content. As always, we have more great new projects to check out, which include a project that simplifies how you manage complex Kubernetes resource configurations, a library to accelerate open source analytics using Amazon S3, a look at a GraphRAG project, a project that helps NodeJS developers make it easier to make use of some advanced Amazon Aurora capabilities, an ESLinting tool for AWS CDK, and more.
Also featured in this edition is content on some of your (and mine) favourite open source projects, which include AWS CDK, Kubernetes, Apache Iceberg, Dapr, Lustre, kro, Projen, ComfyUI, Finch, Open Cybersecurity Schema Framework, PostgreSQL, Amazon EMR, Apache Cassandra, MySQL, HBase, AWS Amplify, Apache XTable, Mountpoint for Amazon S3, Valkey, Apache Airflow, Bottlerocket, Apache Kafka, and Apache Flink. That lot should keep you busy until the new year!
As always, get in touch if you want me to feature your projects in this open source newsletter. Until the next time, I will leave you to dive into the good stuff!
Latest open source projects
The great thing about open source projects is that you can review the source code. If you like the look of these projects, make sure you that take a look at the code, and if it is useful to you, get in touch with the maintainer to provide feedback, suggestions or even submit a contribution. The projects mentioned here do not represent any formal recommendation or endorsement, I am just sharing for greater awareness as I think they look useful and interesting!
Tools
kro
the Analytics Accelerator Library for Amazon S3 is an open source library that accelerates data access from client applications to Amazon S3. With this tool you can 1/ lower processing times and compute costs for data analytics workloads, 2/ implement S3 best practices for performance, 3/utilise optimisations specific to Apache Parquet files, such as pre-fetching metadata located in the footer of the object and predictive column pre-fetching, and 4/improve the price performance for your data analytics applications, such as workloads based on Apache Spark.
Project is currently in Alpha, so bear that in mind. More examples and details in the README.
graphrag-toolkit
is complementary to an existing NodeJS driver and aims to extend the functionality of the driver to enable applications to take full advantage of the features of clustered databases such as Amazon Aurora. In other words, the AWS Advanced NodeJS Wrapper does not connect directly to any database, but enables support of AWS and Aurora functionalities on top of an underlying NodeJS driver of the user's choice.
Hosting a database cluster in the cloud via Aurora is able to provide users with sets of features and configurations to obtain maximum performance and availability, such as database failover. However, at the moment, most existing drivers do not currently support those functionalities or are not able to entirely take advantage of it. The main idea behind the AWS Advanced NodeJS Wrapper is to add a software layer on top of an existing NodeJS driver that would enable all the enhancements brought by Aurora, without requiring users to change their workflow with their databases and existing NodeJS drivers.
Check out the README for additional details on how this simplifies life for developers wanting to access those advanced features.
amazon-q-developer-cli
provides an ESLint plugin for AWS CDK. ESLint is a static code analysis tool for identifying problematic patterns found in JavaScript code. This provides rules for the AWS CDK to help you write readable, maintainable, and reusable code. There are extensive rules provided (.
Drag-and-Drop-Email-Designer
this repo is a tool designed to automate the deployment and management of FMBench for benchmarking on Amazon EC2, Amazon SageMaker and Amazon Bedrock. In case of benchmarking on EC2, we could benchmark on multiple instances simultaneously, and these instances can be of different instance types (so you could run g6e, p4de and a trn1 instances via the same config file), in different AWS regions and also test multiple FMBench config files. This orchestrator automates the creation of Security Groups, Key Pairs, EC2 instances, runs FMBench for a specific config, retrieves the results, and shuts down the instances after completion. Thus it simplifies the benchmarking process (no more manual creation of SageMaker Notebooks, EC2 instances and cleanup, downloading results folder) and ensures a streamlined and scalable workflow. Very detailed README that provides much more details on how this works.
terraform-f1-telemetry-infra
which provides a good primer if you want to know about this very hot open source technology. If you work with data, you will need to know about Apache Iceberg so I recommend folk read this post. Following that are two posts from AWS Community Builder Omid Eidivandi covering AWS CDK. In where he shares both his experiences as well as some practical source code on how he sets up CDK for different use cases.
In . Remember folks, if you don't tag your Cloud resources, do they even exist? Following that he takes a look at Dapr on Kubernetes in his post, where he shares how you can handle failure and maintain service availability even when an entire region or data center goes down, by replicating data, services, and infrastructure across multiple regions.
This weeks essential reading
(there were more than usual, as it has been a few weeks since the last newsletter!)
builds on from some earlier posts and sample code, and shows how you can deploy the ComfyUI project on Amazon EKS [hands on]
is a follow up from a previous post (also featured in an earlier version of this newsletter) that looks at some of the more advanced capabilities that help improve developers experience and productivity
looks at a new serverless horizontal scaling (sharding) capability of Amazon Aurora called Aurora PostgreSQL Limitless Database, where you can scale beyond the existing Aurora limits for write throughput and storage by distributing a database workload over multiple Aurora writer instances while maintaining the ability to use it as a single database [hands on]
Cloud Native
demonstrates how you can use the ARC zonal shift and zonal autoshift capabilities to prepare for and recover from AWS Region or Availability Zone (AZ) impairments [hands on]
helps you learn how to launch a high availability instance fleet cluster using the newly redesigned Amazon EMR console [hands on]
guide you through the process of benchmarking the performance of Amazon RDS for PostgreSQL using the Dedicated Log Volume (DLV) feature that is available in Amazon RDS - if you are a heavy user of PostgreSQL, this is a must read post [hands on]
is a great post that show how to improve query performance of a new replica by warming up the cache using pg_prewarm [hands on]
covers how Cassandra interacts with the OS’s file system and page cache when reading from disk, using various Linux tools like iostat, xfsdist, xfsslower, cachestat, and biolatency, to get insights into different layers of disk I/O performance [hands on]
provides some real-world migration cases to introduce the process of migrating HBase to Amazon EMR HBase using HBase snapshot and replication and the deployment mode of HBase on Amazon S3 [hands on]
Other posts to check out
looks at how this emerging open source project helps facilitates seamless conversions between open table formats (OTFs), providing hands on examples and code [hands on]
provides some nice details on how Amazon FSx for Lustre now supports 12 times higher (up to 1200 Gbps) per-client throughput compared to the previous FSx for Lustre version
Quick updates
Mountpoint for Amazon S3
You can now use Amazon S3 Express One Zone as a high performance read cache with Mountpoint for Amazon S3. The cache can be shared by multiple compute instances and can elastically scale to any dataset size. Mountpoint for S3 is a file client that translates local file system API calls to REST API calls on S3 objects. With this launch, Mountpoint for S3 can cache data in S3 Express One Zone after it’s read, making the subsequent read requests up to 7x faster compared to reading data from S3 Standard.
Previously, Mountpoint for S3 could cache recently accessed data in Amazon EC2 instance storage, EC2 instance memory, or an Amazon EBS volume. This improved performance for repeated read access from the same compute instance for dataset sizes up to the size of the available local storage. Starting today, you can also opt in to caching data in S3 Express One Zone, benefiting applications that repeatedly read a shared dataset across multiple compute instances, without any limits on the total dataset size. Once you opt in, Mountpoint for S3 retains objects with sizes up to one megabyte in S3 Express One Zone. This is ideal for compute-intensive use cases such as machine learning training for computer vision models where applications repeatedly read millions of small images from multiple instances.
PostgreSQL
There have been a few PostgreSQL related updates since the last newsletter. Here are some of the key ones
Amazon Relational Database Service (RDS) for PostgreSQL now supports pgvector 0.8.0, an open-source extension for PostgreSQL for storing and efficiently querying vector embeddings in your database, letting you use retrieval-augmented generation (RAG) when building your generative AI applications. pgvector 0.8.0 release includes improvements on PostgreSQL query planner’s selection of index when filters are present, which can deliver better query performance and improve search result quality. pgvector 0.8.0 release includes a variety of improvements to how pgvector filters data using conditions in WHERE clauses and joins that can improve query performance and usability. Additionally, the iterative index scans help prevent ‘overfiltering’, ensuring generation of sufficient results to satisfy the conditions of a query. If an initial index scan doesn't satisfy the query conditions, pgvector will continue to search the index until it hits a configurable threshold. This release also has performance improvements for searching and building HNSW indexes. pgvector 0.8.0 is available on database instances in Amazon RDS running PostgreSQL 17.1 and higher, 16.5 and higher, 15.9 and higher, 14.14 and higher, and 13.17 and higher in all applicable AWS Regions.
Amazon Relational Database Service (RDS) for PostgreSQL now supports the latest minor versions 17.2, 16.6, 15.10, 14.15, 13.18, and 12.22. We recommend that you upgrade to the latest minor versions to fix known security vulnerabilities in prior versions of PostgreSQL, and to benefit from the bug fixes added by the PostgreSQL community. You are able to leverage automatic minor version upgrades to automatically upgrade your databases to more recent minor versions during scheduled maintenance window. Additionally, starting with PostgreSQL major version 18, Amazon RDS for PostgreSQL will deprecate plcoffee and plls PostgreSQL extensions. We recommend that you stop using Coffee scripts and LiveScript in your applications, ensuring you have an upgrade path for future.
MySQL
Amazon RDS for MySQL now supports MySQL major version 8.4, the latest long-term support (LTS) release from the MySQL community. RDS for MySQL 8.4 is integrated with AWS Libcrypto (AWS-LC) FIPS module (Certificate #4816), and includes support for multi-source replication plugin for analytics, Group Replication plugin for continuous availability, as well as several performance and feature improvements added by the MySQL community. Learn more about the community enhancements in the MySQL 8.4 release notes. You can leverage Amazon RDS Managed Blue/Green deployments to upgrade your databases from MySQL 8.0 to MySQL 8.4.
Mershad Irani and Vijay Karumajji have collaborated on the post,
In more Valkey news, ElastiCache for Valkey self-designed (node-based) clusters now support server-side write request latency and read request latency metrics. With this launch, you can now measure the server-side response time for Valkey commands and troubleshoot latency spikes in your ElastiCache for Valkey cluster. Monitoring response time is critical to improving end user experience by tracking application trends and adjusting cluster configurations, as needed. The server-side latency metrics provides a complete picture of the server-side response time that includes command pre-processing, command execution, and command post-processing. SuccessfulWriteRequestLatency and SuccessfulReadRequestLatency measure the time that the Valkey engine takes to respond to a successfully executed request.
The final update which came in as I was writing this, is news that the Valkey Glide client 1.2 was released. In this update AWS added support for Availability Zone (AZ) awareness. Valkey GLIDE is a reliable, high-performance, and highly available client, and it’s pre-configured with best practices from over a decade of operating Amazon ElastiCache. Valkey GLIDE is compatible with versions 7.2 and 8.0 of Valkey, as well as versions 6.2, 7.0, and 7.2 of Redis OSS. With this update, Valkey GLIDE will direct requests to Valkey nodes within the same Availability Zone, minimising cross-zone traffic and reducing response time. Java, Python, and Node.js are the currently supported languages for Valkey GLIDE, with further languages in development. With this update, Valkey GLIDE 1.2 also supports Amazon ElastiCache and Amazon MemoryDB’s JavaScript Object Notation (JSON) data type, allowing customers to store and access JSON data within their clusters. In addition, it supports MemoryDB’s Vector Similarity Search, empowering customers to store, index, and search vectors for AI applications at single-digit millisecond speed.
Apache Airflow
A couple of updates that I know Apache Airflows are going to love.
First up is news that Amazon Managed Workflows for Apache Airflow (MWAA) now offers a micro environment size, giving customers of the managed service the ability to create multiple, independent environments for development and data isolation at a lower cost. Amazon MWAA is a managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud. With Amazon MWAA micro environments, customers can now create smaller, cost-effective environments that are more efficient for development use, as well as for teams that require data isolation with lightweight workflow requirements. Read [hands on] from Chandan Rupakheti and Hernan Garcia.
Bottlerocket
You can now use AMIs for Bottlerocket that are preconfigured to use FIPS 140-3 validated cryptographic modules, including the Amazon Linux 2023 Kernel Crypto API and AWS-LC. Bottlerocket is a Linux-based operating system purpose-built for running containers, with a focus on security, minimal footprint, and safe updates. With these FIPS-enabled Bottlerocket AMIs, the host software uses only FIPS-approved cryptographic algorithms for TLS connections. This includes connectivity to AWS services such as EC2 and Amazon Elastic Container Registry (ECR). Additionally, in regions where FIPS endpoints are available, the AMIs automatically use FIPS-compliant endpoints for these services by default, streamlining secure configurations for containerised workloads.
Amazon EMR
Amazon EMR 7.4 is now available, which supports Apache Spark 3.5.2, Apache Hadoop 3.4.0, Trino 446, Apache HBase 2.5.5, Apache Phoenix 5.2.0, Apache Flink 1.19.0, Presto 0.287 and Apache Zookeeper 3.9.2. Amazon EMR 7.4 enables in-transit encryption for 7 additional endpoints used with distributed applications like Apache Livy, Apache Hue, JupyterEnterpriseGateway, Apache Ranger and Apache Zookeeper. This update builds on the previous release Amazon EMR 7.3, which enabled in-transit encryption for 22 endpoints. In-Transit Encryption enables you to run workloads that meet strict regulatory or compliance requirements by protecting the confidentiality and integrity of your data.
Apache Kafka
A few weeks ago saw the general availability of Express brokers for Amazon Managed Streaming for Apache Kafka (Amazon MSK). Express brokers are a new broker type for Amazon MSK Provisioned designed to deliver up to 3x more throughput per broker, scale up to 20x faster, and reduce recovery time by 90% as compared to standard Apache Kafka brokers. Express brokers come preconfigured with Kafka best practices by default, support all Kafka APIs, and provide the same low-latency performance that Amazon MSK customers expect, so they can continue using existing client applications without any changes. With Express brokers, customers can provision, scale up, and scale down Kafka cluster capacity in minutes, offload storage management with virtually unlimited pay-as-you-go storage, and build highly resilient applications. Customers can also continue using all of the Amazon MSK key features, including security, connectivity, and observability options, as well as popular integrations, including Amazon MSK Connect, Amazon Simple Storage Service (Amazon S3), AWS Glue Schema Registry, and more. Express brokers are currently available on Kafka version 3.6 and come in three different sizes of Graviton3-based M7g instances: large, 4xlarge, and 16xlarge. Each broker is charged an hourly rate with storage and data ingested charged separately on a pay-as-you-go basis.
A nice post covers this in more details, so go read it -
Videos of the week
Future-Proof Your Workflow Orchestration: Automate Migration to Airflow
If you are looking to move your current workflow orchestration solutions to Apache Airflow, check out this video. Impetus' LeapLogic offer a solution that simplifies the migration process, and might be just want you need to get started.
Building with Open Source at AWS & What's Next for Developers
Check out Adam Seligman and Alex Williams as they talk about open source, and explore some of AWS' current investments in open source.
Building an open source data strategy on AWS
Recorded at the AWS Summit London, check out this session for a glimpse on how to take full advantage of open source technologies while leveraging the scalability, flexibility, and cost efficiency of AWS Cloud. The talk looks at data architecture patterns that leverage open source on AWS.
Events for your diary
If you are planning any events in 2024, either virtual, in person, or hybrid, get in touch as I would love to share details of your event with readers.
re:Invent
December 2nd - 6th, Las Vegas USA
AWS's flagship event is only days away. You can
Celebrating open source contributors
The articles and projects shared in this newsletter are only possible thanks to the many contributors in open source. I would like to shout out and thank those folks who really do power open source and enable us all to learn and build on top of what they have created.
So thank you to the following open source heroes: Abhay Saxena, Rashim Gupta, Mershad Irani, Vijay Karumajji, Hernan Garcia, Sriharsh Adari, Chandan Rupakheti, Julian Wood, Leandro Cavalcante Damascena, Amine Hilaly, Christina Andonov, Lukonde Mwila, Wang Rui, Hemanth AVS, Ashok Srirama, Sai Charan Teja Gopaluni, Danny Banks, Dipankar Mazumdar, Matthias Rudolph, Stephen Said, Michael Tran, Alain Krok, Dinesh Sajwan, Sanket Nasre, Ashish Ameta, Seshanth Kannappan, Mahalingam Sivaprakasam, Frank Fan, Sheetal Joshi, Vikram Venkataraman, Ashok Srirama, George John, Leo Li, Prateek Gogia, Mark Terenzoni, Garima Arora, Ravi Kumar Singh, Tarun Chanana, Shawn McCoy, Divya Sharma, Sid Vantair, Naga Appani, Belma Canik, Juhi Patil, Steve Dille, Mark Greenhalgh, Lei Zeng, Ravi Mathur, Jon Haddad, Diego Garcia Garcia, Francisco Morillo, Phaneendra Vuliyaragoli, Dalei Xu, Zhiyong Su, Shijian Tang, Jan Michael Go Tan, JS Labonte, Timur Galeev, Saifeddine Rajhi, Claudio Taverna, and Omid Eidivandi.
Feedback
Please please please take 1 minute to for more open source goodness.
One of the pieces of feedback I received in 2023 was to create a repo where all the projects featured in this newsletter are listed. Where I can hear you all ask? Well as you ask so nicely, you can meander over to newsletter-oss-projects.
Made with ♥ from DevRel
↗ Original-Artikel auf dev.to lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR