Code Archeologist: AI-Powered Git Repository Analysis with PostgreSQL

How the Project Works

Code Archeologist utilizes a combination of PostgreSQL extensions and open-source AI models to deliver a seamless analysis experience:

Data Ingestion

• The application connects to a GitHub repository and fetches commit history, contributors, file changes, issues, and dependencies using the GitHub API.

Embedding Generation

• Commit messages and relevant text data are processed using Ollama to generate 768-dimensional vector embeddings.


• These embeddings are stored in PostgreSQL using the pgvector extension, which allows efficient storage and retrieval of high-dimensional vectors.

Indexing and Similarity Search

• The pgvectorscale extension is employed to create a diskann index on the embedding column, enabling fast approximate nearest neighbor searches.


• This setup allows the application to perform rapid similarity searches, facilitating features like semantic commit search and question answering.

AI-Driven Features

• 
  Refactoring Suggestions: Analyzes code evolution to recommend potential improvements.


• 
  Summarization: Generates concise summaries of commit messages using AI models.


• 
  Question Answering: Allows users to ask natural language questions about the codebase and receive AI-generated answers based on the commit history.

Visualization

The frontend built with Vue.js displays various visualizations such as genetic trees, heatmaps, timelines, and dependency graphs, providing users with insightful views of their codebase.

Performance and Scalability

By leveraging PostgreSQL with pgvector and pgvectorscale, the application ensures efficient storage, rapid querying, and scalability to handle large datasets without relying on external vector databases.

How the Backend Works

The backend of Code Archeologist is built using Node.js and Express.js, interfacing with a PostgreSQL database enhanced with AI-specific extensions. Here’s an overview of its functionality:

Session Management

• Utilizes express-session to handle user sessions, ensuring secure and persistent interactions.

Database Initialization

• Connects to PostgreSQL using the pg library.


• Initializes the database schema, creating tables like code_analysis and commit_embeddings.


• Ensures necessary extensions (pgvector, pgvectorscale) are installed for vector operations and AI functionalities.

Data Processing

• 
  Fetching Data: Retrieves commits, contributors, issues, and dependencies from GitHub repositories.


• 
  Embedding Generation: Uses Ollama to generate vector embeddings for commit messages, storing them with pgvector for efficient similarity searches.


• 
  Indexing: Implements pgvectorscale with diskann indexing to optimize search performance.

AI Integration

• 
  OpenAI & Ollama: Integrates OpenAI for generating completions and Ollama for creating text embeddings, facilitating features like refactoring suggestions and question answering.

Error Handling

• Implements robust error handling across all endpoints, ensuring meaningful responses and logging errors for debugging.

Technologies and Tools Used

Programming Languages & Frameworks

• 
  Node.js: Server-side JavaScript runtime.


• 
  Express.js: Web framework for building API endpoints.

Databases

• 
  PostgreSQL: Relational database system.


• 
  pgvector: PostgreSQL extension for storing vector embeddings.


• 
  pgvectorscale: Extension for optimized vector similarity searches.

AI

• 
  Ollama: Generates text embeddings using open-source models (I used nomic-embed-text model).


• 
  OpenAI SDK: Facilitates AI-driven features like summary and question answering.

APIs & Libraries

• 
  Octokit: GitHub API client for fetching repository data.


• 
  dotenv: Loads environment variables from a .env file.


• 
  express-session: Manages user sessions.


• 
  Winston: Logging library for capturing application logs.


• 
  Cors: Enables Cross-Origin Resource Sharing.


• 
  GitHub API: Accesses repository information.

Final Thoughts

Building Code Archeologist was a great experience that showed me just how powerful combining PostgreSQL extensions with AI tools can be. Using pgvector, pgvectorscale, and Ollama together made it possible to create a strong, scalable app that can handle complex searches and give useful insights. This project really boosted my appreciation for using open-source tools to build smart AI applications.

Thanks for considering my submission!