Pushing Python Packages to Artifact Registry Using Cloud Build

Google Artifact Registry is a powerful solution for managing and hosting Python package artifacts in a private, secure, and scalable way. This guide provides a step-by-step walkthrough to push Python package .whl files to the Artifact Registry using Google Cloud Build and a secret (creds) from Google Secret Manager for authentication.

Prerequisites

1. 
   

   Artifact Registry Setup:

   
   
   
   
   
   • Create a Python repository in your Artifact Registry:
     
   
   
   
   
   CODE Kopieren

    gcloud artifacts repositories create python-packages \
   
      --repository-format=python \
   
      --location=us-central1 \
   
      --description="Python packages repository"
   
   

   
   
   


2. 
   

   Secret Setup:

   
   
   
   
   
   • Store your key as a secret in Google Secret Manager:
     
   
   
   
   
   CODE Kopieren

    gcloud secrets create creds --data-file=path/to/key.json
   
   

   
   
   

• 
  

  Grant Cloud Build access to the secret:(Optional, can also be done using IAM)
  
  

  
  
  CODE Kopieren

   gcloud secrets add-iam-policy-binding creds \
  
     --member="serviceAccount:$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')@cloudbuild.gserviceaccount.com" \
  
     --role="roles/secretmanager.secretAccessor"
  
  

  
  
  

1. 
   Cloud Build Permissions:
   Ensure your Cloud Build service account has the necessary permissions to access the Artifact Registry and Secret Manager.

Cloud Build YAML Configuration

Here's the full working cloudbuild.yaml file:

options:

  machineType: E2_HIGHCPU_8

  substitutionOption: ALLOW_LOOSE

  logging: CLOUD_LOGGING_ONLY



steps:

  # Step 1: Access the secret `creds` and save it as `key.json`

  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'

    entrypoint: bash

    args:

      - '-c'

      - |

        gcloud secrets versions access latest --secret=creds > /workspace/key.json



  # Step 2: Configure `.pypirc` with the Artifact Registry credentials

  - name: 'python'

    entrypoint: bash

    args:

      - '-c'

      - |

        cat > ~/.pypirc << EOL

        [distutils]

        index-servers = tower-common-repo



        [tower-common-repo]

        repository: https://us-central1-python.pkg.dev/$PROJECT_ID/python-packages/

        username: _json_key_base64

        password: $(base64 -w0 /workspace/key.json)

        EOL



        # Step 3: Build and upload the Python package

        pip install twine build && \

        python -m build && \

        twine upload --repository tower-common-repo dist/* --verbose

Step-by-Step Explanation

1. 
   

   Define Build Options:

   
   
   
   
   
   • Set the machine type, substitution behavior, and logging options.
   
   
   • These configurations ensure efficient builds and manageable logs.
   
   
   
   


2. 
   

   Retrieve key.json Secret:

   
   
   
   
   
   • Use gcloud secrets versions access to fetch the key.json file securely from Secret Manager.
   
   
   • Save the file to a known location (/workspace/key.json).
   
   
   
   


3. 
   

   Configure .pypirc:

   
   
   
   
   
   • Generate a .pypirc file dynamically. This file is required for twine to authenticate with the Artifact Registry.
   
   
   • The password is base64-encoded content of key.json.
   
   
   
   


4. 
   

   Build and Push Package:

   
   
   
   
   
   • Install necessary tools (twine, build).
   
   
   • Build the Python package (python -m build).
   
   
   • Use twine upload to push the .whl file to the Artifact Registry.
   
   
   
   

Triggering the Build

Save the cloudbuild.yaml file and trigger the build or can connect to github repository:

gcloud builds submit --config=cloudbuild.yaml .

Key Points

• 
  Secure Secrets Management: The secret (key.json) is accessed securely using Google Secret Manager.


• 
  Dynamic Configuration: .pypirc is generated during the build, ensuring no sensitive data is stored in the repository.


• 
  Automated Upload: The process automates package building and pushing, reducing manual intervention.

Validation

After the build completes:

1. Verify the uploaded package in the Artifact Registry:
   

   gcloud artifacts packages list --repository=python-packages --location=us-central1

1. Check for errors or warnings in the build logs.