Lädt...


🔧 Glue Iceberg Rest Api and PyIceberg


Nachrichtenbereich: 🔧 Programmierung
🔗 Quelle: dev.to

Access Glue Iceberg tables via the Iceberg Rest Api

AWS Released silenty Iceberg REST-API support. This is a standard API to access iceberg tables on different platforms. More information can be found here https://iceberg.apache.org/concepts/catalog/

PyIceberg is a python library with generic iceberg support. I also supports the rest api.

from pyiceberg.catalog import load_catalog
import logging

# Set up logging to show debug messages
logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Specifically for PyIceberg logging
logger = logging.getLogger('pyiceberg')
logger.setLevel(logging.DEBUG)



def main():
    rest_catalog = load_catalog(
    "ibtest1",
    **{
        "type": "rest",
        "uri": "https://glue.eu-central-1.amazonaws.com/iceberg",
             "rest.sigv4-enabled": "true",
               "rest.signing-name": "glue",
               "rest.signing-region": "eu-central-1"
    }
    )
    print(rest_catalog.list_namespaces())
    print(rest_catalog.list_tables("ibtest"))
    print(rest_catalog.load_table("ibtest.ibtest1").scan().to_pandas())



if __name__ == "__main__":
    main()

Glue Catalog version

For comparison this is the native glue version in pyiceberg. This uses the boto api.

def main():
    glue_catalog = load_catalog("glue", **{"type": "glue"})


    print(glue_catalog.list_namespaces())
    print(glue_catalog.list_tables("ibtest"))
    print(glue_catalog.load_table("ibtest.ibtest1").scan().to_pandas())

Output

ListNameSpace

2024-12-22 15:13:41,061 - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-12-22 15:13:41,062 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-12-22 15:13:41,062 - botocore.auth - DEBUG - CanonicalRequest:
GET
/iceberg/v1/config

accept:*/*
accept-encoding:gzip, deflate
content-type:application/json
host:glue.eu-central-1.amazonaws.com
x-amz-date:20241222T141341Z
x-client-version:0.14.1
x-iceberg-access-delegation:vended-credentials

accept;accept-encoding;content-type;host;x-amz-date;x-client-version;x-iceberg-access-delegation
xxxxxxxx
2024-12-22 15:13:41,062 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241222T141341Z
20241222/eu-central-1/glue/aws4_request
xxxxxxx
2024-12-22 15:13:41,062 - botocore.auth - DEBUG - Signature:
xxxxxx
2024-12-22 15:13:41,062 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): glue.eu-central-1.amazonaws.com:443
2024-12-22 15:13:41,213 - urllib3.connectionpool - DEBUG - https://glue.eu-central-1.amazonaws.com:443 "GET /iceberg/v1/config HTTP/1.1" 200 327
2024-12-22 15:13:41,237 - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-12-22 15:13:41,237 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-12-22 15:13:41,237 - botocore.auth - DEBUG - CanonicalRequest:
GET
/iceberg/v1/catalogs/311141556126/namespaces

accept:*/*
accept-encoding:gzip, deflate
content-type:application/json
host:glue.eu-central-1.amazonaws.com
x-amz-date:20241222T141341Z
x-client-version:0.14.1
x-iceberg-access-delegation:vended-credentials

accept;accept-encoding;content-type;host;x-amz-date;x-client-version;x-iceberg-access-delegation
xxxxxx
2024-12-22 15:13:41,237 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241222T141341Z
20241222/eu-central-1/glue/aws4_request
xxxxxxx
2024-12-22 15:13:41,237 - botocore.auth - DEBUG - Signature:
xxxxxx
2024-12-22 15:13:41,237 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): glue.eu-central-1.amazonaws.com:443
2024-12-22 15:13:41,435 - urllib3.connectionpool - DEBUG - https://glue.eu-central-1.amazonaws.com:443 "GET /iceberg/v1/catalogs/311141556126/namespaces HTTP/1.1" 200 48
[('ibtest',), ('sourcedata_sales',)]

ListTable

2024-12-22 15:13:41,462 - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-12-22 15:13:41,463 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-12-22 15:13:41,463 - botocore.auth - DEBUG - CanonicalRequest:
GET
/iceberg/v1/catalogs/311141556126/namespaces/ibtest/tables

accept:*/*
accept-encoding:gzip, deflate
content-type:application/json
host:glue.eu-central-1.amazonaws.com
x-amz-date:20241222T141341Z
x-client-version:0.14.1
x-iceberg-access-delegation:vended-credentials

accept;accept-encoding;content-type;host;x-amz-date;x-client-version;x-iceberg-access-delegation
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-12-22 15:13:41,463 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241222T141341Z
20241222/eu-central-1/glue/aws4_request
xxxxxxx
2024-12-22 15:13:41,463 - botocore.auth - DEBUG - Signature:
xxxx
2024-12-22 15:13:41,541 - urllib3.connectionpool - DEBUG - https://glue.eu-central-1.amazonaws.com:443 "GET /iceberg/v1/catalogs/311141556126/namespaces/ibtest/tables HTTP/1.1" 200 59

[('ibtest', 'ibtest1')]

Scan Tables

2024-12-22 15:13:41,567 - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2024-12-22 15:13:41,567 - botocore.auth - DEBUG - Calculating signature using v4 auth.
2024-12-22 15:13:41,567 - botocore.auth - DEBUG - CanonicalRequest:
GET
/iceberg/v1/catalogs/311141556126/namespaces/ibtest/tables/ibtest1

accept:*/*
accept-encoding:gzip, deflate
content-type:application/json
host:glue.eu-central-1.amazonaws.com
x-amz-date:20241222T141341Z
x-client-version:0.14.1
x-iceberg-access-delegation:vended-credentials

accept;accept-encoding;content-type;host;x-amz-date;x-client-version;x-iceberg-access-delegation
xxxxxx
2024-12-22 15:13:41,567 - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20241222T141341Z
20241222/eu-central-1/glue/aws4_request
xxxxx
2024-12-22 15:13:41,567 - botocore.auth - DEBUG - Signature: xxxxxx
2024-12-22 15:13:41,712 - urllib3.connectionpool - DEBUG - https://glue.eu-central-1.amazonaws.com:443 "GET /iceberg/v1/catalogs/311141556126/namespaces/ibtest/tables/ibtest1 HTTP/1.1" 200 2123

    id  name                 created
0  001  test 2024-12-22 13:48:31.381

Conclusion

With the latest release of Glue you can access Iceberg tables on AWS using the standard iceberg REST_API opening the infrastructure to multiple tools.
The only AWS specific call is the signing with sigv4. https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html. This is already used by many tools for the S3 access.

This decouples the code from AWS Specific access and allows you to use more generic tools

...

🔧 Glue Iceberg Rest Api and PyIceberg


📈 81.72 Punkte
🔧 Programmierung

🔧 Glue Iceberg Rest Api and PyIceberg


📈 81.72 Punkte
🔧 Programmierung

🔧 Unity Catalog Iceberg Rest Api and PyIceberg


📈 63.19 Punkte
🔧 Programmierung

🔧 Unity Catalog Iceberg Rest Api and PyIceberg


📈 63.19 Punkte
🔧 Programmierung

🔧 Reflecting on “Technical Leadership and Glue Work” / “Being Glue”


📈 38.28 Punkte
🔧 Programmierung

🔧 Integration of Apache Iceberg in S3, Glue, Athena, Matillion, and Snowflake – Part 1


📈 37.74 Punkte
🔧 Programmierung

🔧 Integration of Apache Iceberg in S3, Glue, Athena, Matillion, and Snowflake – Part 2


📈 37.74 Punkte
🔧 Programmierung

🔧 Replicate data from DynamoDB to Apache Iceberg tables using Glue Zero-ETL integration


📈 36.52 Punkte
🔧 Programmierung

🔧 Quick tip: Using SingleStore with PyIceberg


📈 30.16 Punkte
🔧 Programmierung

🔧 What Apache Iceberg REST Catalog is and isn't


📈 28.14 Punkte
🔧 Programmierung

🕵️ WP Live Chat Support up to 8.0.32 on WordPress REST API REST API Call privilege escalation


📈 27.63 Punkte
🕵️ Sicherheitslücken

🎥 The Chatty API vs the Reserved One: GraphQL vs REST #graphql #rest #api #data


📈 27.63 Punkte
🎥 IT Security Video

📰 Boost Your Cloud Data Applications with DuckDB and Iceberg API


📈 24.11 Punkte
🔧 AI Nachrichten

🔧 The REST API Handbook – How to Build, Test, Consume and Document REST APIs


📈 23.96 Punkte
🔧 Programmierung

🔧 Rest API v/s Web API v/s SOAP API


📈 23.61 Punkte
🔧 Programmierung

🔧 Rest API Vs HTTP API Vs WebSocket API


📈 23.61 Punkte
🔧 Programmierung

🕵️ WordPress bis 4.7.0 REST API class-wp-rest-users-controller.php Information Disclosure


📈 22.74 Punkte
🕵️ Sicherheitslücken

🕵️ WordPress bis 4.7.0 REST API class-wp-rest-users-controller.php Information Disclosure


📈 22.74 Punkte
🕵️ Sicherheitslücken

🕵️ WordPress 4.7.0 REST API class-wp-rest-users-controller.php information disclosure


📈 22.74 Punkte
🕵️ Sicherheitslücken

📰 Rest API Testing: So testest du Rest APIs richtig!


📈 22.74 Punkte
Web Tipps

🔧 Rest API Testing: How to test Rest APIs properly!


📈 22.74 Punkte
🔧 Programmierung

🔧 Master API Development with Django REST Framework – Learn Django REST Framework for Free!


📈 22.74 Punkte
🔧 Programmierung

🔧 Building REST API Endpoints with Django REST Framework: A Step-by-Step Guide


📈 22.74 Punkte
🔧 Programmierung

🔧 Deploying a REST API and Angular Frontend Using AWS CDK, S3, and API Gateway


📈 21.16 Punkte
🔧 Programmierung

🪟 How to get the new Like Glue and Thorn named shotguns in The Division 2 and instantly make them proficient


📈 20.98 Punkte
🪟 Windows Tipps

🔧 Why Data Analysts, Engineers, Architects and Scientists Should Care about Dremio and Apache Iceberg


📈 20.44 Punkte
🔧 Programmierung

🔧 Automate REST API Management with Terraform: AWS API Gateway Proxy Integration and Lambda


📈 19.94 Punkte
🔧 Programmierung

🔧 Basic CRUD API using Django and Django REST Framework (DRF) || Simple API for managing a list of books.


📈 19.94 Punkte
🔧 Programmierung

matomo