How to build a new Harlequin adapter with Poetry

Welcome to the first post in LETSQL's tutorial series!

In this blog post, we take a detour from our usual theme of data pipelines to demonstrate how to create and publish a Python package with Poetry, using DataFusion as an example.

Introduction

adapter before. Thankfully, it was really easy to add one.

In this post, We'll demonstrate these concepts by building a Harlequin adapter for DataFusion. And, by way of doing so, we will also cover Poetry's essential features, project setup, and the steps to publish your package on PyPI.

To get the most out of this guide, you should have a basic understanding of , and and available in is a SQL IDE that runs in the terminal. It provides a powerful and feature-rich alternative to traditional command-line database tools, making it versatile for data exploration and analysis workflows.

Some key things to know about Harlequin:

• Harlequin supports multiple .
  
  

  
  
  
  Poetry
  

  
  
  

  Poetry is a modern, feature-rich tool that streamlines dependency management and packaging for Python projects, making development more deterministic and efficient.
  
  From the .

  
  

  
  
  
  The Harlequin Adapter Template
  

  
  
  

  The first step for developing a Harlequin adapter is to generate a new repo from the existing are repositories that serve as starting points for new projects. They provide pre-configured files, structures, and settings that are copied to new repositories, allowing for quick project setup without the overhead of forking.
  
  This feature streamlines the process of creating consistent, well-structured projects based on established patterns.

  
  
  

  The harlequin-adapter-template comes with a poetry.lock file and a pyproject.toml file, in addition to some boilerplate code for defining the required classes.

  
  

  
  
  
  Coding the Adapter
  

  
  
  

  Let's explore the essential files needed for package distribution before we get into the specifics of coding.

  
  

  
  
  
  Package configuration
  

  
  
  

  The pyproject.toml file is now the standard for configuring Python packages for publication and other tools. Introduced in , this

  
  
  

  ::: {.warning}
  
  Poetry should always be installed in a dedicated virtual environment to isolate it from the rest of your system. It should in no case be installed in the environment of the project that is to be managed by Poetry.
  
  :::

  
  
  

  Here we will presume you have access to Poetry by running pipx install poetry

  
  

  
  
  
  Developing in the virtual environment
  

  
  
  

  With our file structure clarified, let's begin the development process by setting up our environment. Since our project already includes pyproject.toml and poetry.lock files, we can initiate our environment using the poetry shell command.

  
  
  

  This command activates the virtual environment linked to the current Poetry project, ensuring all subsequent operations occur within the project's dependency context. If no virtual environment exists, poetry shell automatically creates and activates one.

  
  
  

  poetry shell detects your current shell and launches a new instance within the virtual environment. As Poetry centralizes virtual environments by default, this command eliminates the need to locate or recall the specific path to the activate script.

  
  
  

  To verify which Python environment is currently in use with Poetry, you can use the following commands:
  
  

  
  
  
  
  

  CODE Kopieren

  poetry env list --full-path
  
  

  
  
  
  
  
  
  
  

  This will show all the virtual environments associated with your project and indicate which one is currently active.
  
  As an alternative, you can get the full path of only the current environment:
  
  

  
  
  
  
  

  CODE Kopieren

  poetry env info -p
  
  

  
  
  
  
  
  
  
  

  With the environment activated, use poetry install to install the required dependencies. The command works as follows

  
  
  
  
  
  1. If a poetry.lock file is present, poetry install will use the exact versions specified in that file rather than resolving the dependencies dynamically. This ensures consistent, repeatable installations across different environments.
     i. If you run poetry install and it doesn't seem to be progressing, you may need to run export PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring in the shell you're installing in
  
  
  2. Otherwise, it reads the pyproject.toml file in the current project, resolves the dependencies listed there, and installs them.
  
  
  3. If no poetry.lock file exists, poetry install will create one after resolving the dependencies, otherwise it will update the existing one.
  
  
  
  
  

  To complete the environment setup, we need to add the datafusion library to our dependencies. Execute the following command:
  
  

  
  
  
  
  

  CODE Kopieren

  poetry add datafusion
  
  

  
  
  
  
  
  
  
  

  This command updates your pyproject.toml file with the datafusion package and installs it. If you don't specify a version, Poetry will automatically select an appropriate one based on available package versions.

  
  
  

  
  
  
  Implementing the Interfaces
  

  
  
  

  To create a Harlequin Adapter, you need to implement three interfaces defined as abstract classes in the harlequin.adapter module.

  
  
  

  The first one is the HarlequinAdapter.
  
  

  
  
  
  
  

  CODE Kopieren

  #| eval: false
  
  #| code-fold: false
  
  #| code-summary: implementation of HarlequinAdapter
  
  
  
  class DataFusionAdapter(HarlequinAdapter):
  
      def __init__(self, conn_str: Sequence[str], **options: Any) -> None:
  
          self.conn_str = conn_str
  
          self.options = options
  
  
  
      def connect(self) -> DataFusionConnection:
  
          conn = DataFusionConnection(self.conn_str, self.options)
  
          return conn
  
  

  
  
  
  
  
  
  
  

  The second one is the HarlequinConnection, particularly the methods execute and get_catalog.
  
  

  
  
  
  
  

  CODE Kopieren

  #| eval: false
  
  #| code-fold: false
  
  #| code-summary: implementation of execution of HarlequinConnection
  
  
  
   def execute(self, query: str) -> HarlequinCursor | None:
  
       try:
  
           cur = self.conn.sql(query)  # type: ignore
  
           if str(cur.logical_plan()) == "EmptyRelation":
  
               return None
  
       except Exception as e:
  
           raise HarlequinQueryError(
  
               msg=str(e),
  
               title="Harlequin encountered an error while executing your query.",
  
           ) from e
  
       else:
  
           if cur is not None:
  
               return DataFusionCursor(cur)
  
           else:
  
               return None
  
  

  
  
  
  
  
  
  
  

  For brevity, we've omitted the implementation of the get_catalog function. You can find the full code in the is a mechanism for code to advertise components it provides to be discovered and used by other code.

  
  
  

  Notice that registering a plugin with Poetry is equivalent to the following pyproject.toml executes the given command inside the project’s virtualenv.

  
  
  

  
  
  
  Building and Publishing to PyPI
  

  
  
  

  With the tests passing, we're nearly ready to publish our project. Let's enhance our pyproject.toml file to make our package more discoverable and appealing on PyPI. We'll add key metadata including:

  
  
  
  
  
  1. A link to the GitHub repository
  
  
  2. A path to the README file
  
  
  3. A list of relevant classifiers
  
  
  
  
  

  These additions will help potential users find and understand our package more easily.
  
  

  
  
  
  
  

  CODE Kopieren

  classifiers = [
  
      "Development Status :: 3 - Alpha",
  
      "Intended Audience :: Developers",
  
      "Topic :: Software Development :: User Interfaces",
  
      "Topic :: Database :: Database Engines/Servers",
  
      "License :: OSI Approved :: MIT License",
  
      "Programming Language :: Python :: Implementation :: CPython"
  
  ]
  
  readme = "README.md"
  
  repository = "https://github.com/mesejo/datafusion-adapter"
  
  

  
  
  
  
  
  
  
  

  For reference:

  
  
  
  
  
  • The complete list of classifiers is available on .
    
  
  
  • The formal, technical specification for pyproject.toml can be found on :
    
    
    
    
    
    • 
      harlequin_datafusion is the package (or distribution) name
    
    
    • 
      0.1.1 is the version number
    
    
    • 
      py3 indicates it's compatible with Python 3
    
    
    • 
      none compatible with any CPU architecture
    
    
    • 
      any with no if you haven't already.

    
    

    • Generate an .

      
      
      

      To publish to the actual Python Package Index (PyPI) instead:

      
      
      
      
      

      1. Create an account at when publishing your project.

         
         
         

         
         
         
         Automated Publishing on GitHub release
         

         
         
         

         Manually publishing each time is repetitive and error-prone, so to fix this problem, let us create a GitHub Action to
         
         publish each time we create a release.

         
         
         

         Here are the key steps to publish a Python package to PyPI using GitHub Actions and Poetry:

         
         
         
         
         

         1. Set up PyPI authentication: You must provide your PyPI credentials (the API token) as GitHub secrets so the GitHub Actions workflow can access them. Name these secrets something like PYPI_TOKEN.

         
         

         2. Create a GitHub Actions workflow file: In your project's .github/workflows directory, create a new file like publish.yml with the following content:
            
            

         
         
         
         
         
         
         

         CODE Kopieren

            name: Build and publish python package
         
         
         
            on:
         
              release:
         
                types: [ published ]
         
         
         
            jobs:
         
              publish-package:
         
                runs-on: ubuntu-latest
         
                permissions:
         
                  contents: write
         
                steps:
         
                  - uses: actions/checkout@v3
         
                  - uses: actions/setup-python@v4
         
                    with:
         
                      python-version: '3.10'
         
         
         
                  - name: Install Poetry
         
                    uses: snok/install-poetry@v1
         
         
         
                  - run: poetry config pypi-token.pypi "${{ secrets.PYPI_TOKEN }}"
         
         
         
                  - name: Publish package
         
                    run: poetry publish --build --username __token__
         
         

         
         
         
         
         
         
         
         

         The key is to leverage GitHub Actions to automate the publishing process and use Poetry to manage your package's dependencies and metadata.

         
         
         

         
         
         
         Conclusion
         

         
         
         

         Poetry is a user-friendly Python package management tool that simplifies project setup and publication. Its intuitive command-line interface streamlines environment management and dependency installation. It supports plugin development, integrates with other tools, and emphasizes testing for robust code. With straightforward commands for building and publishing packages, Poetry makes it easier for developers to share their work with the Python community.

         
         
         

         At LETSQL, we're committed to contributing to the developer community. We hope this blog post serves as a straightforward guide to developing and publishing Python packages, emphasizing best practices and providing valuable resources.
         
         To subscribe to our newsletter, visit letsql.com.

         
         
         

         
         
         
         Future Work
         

         
         
         

         As we continue to refine the adapter, we would like to provide better autocompletion and direct reading from files (parquet, csv) as in the DataFusion-cli. This requires a tighter integration with the Rust library without going through the Python bindings.

         
         
         

         Your thoughts and feedback are invaluable as we navigate this journey. Share your experiences, questions, or suggestions in the comments below or on our community forum. Let's redefine the boundaries of data science and machine learning integration.

         
         
         

         
         
         
         Acknowledgements
         

         
         
         

         Thanks to Dan Lovell and Hussain Sultan for the comments and the thorough review.

         Vollständiger Original-Bericht

         Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

         ↗ Original-Artikel auf dev.to lesen