Building a RAG-Based AWS VPC Flow Log Analyzer

If you’ve ever opened a raw VPC Flow Log file, you know the feeling with thousands of lines of space-delimited fields, IPs, ports, packet counts, and timestamps. Somewhere in there is the answer to your question. You just have to find it.

Was that SSH connection rejected?

Which IP keeps hitting port 443?

Is this traffic normal or a problem?

Manually digging through VPC Flow Logs is slow, reactive, and honestly painful. It usually means grepping through files, exporting to spreadsheets, or writing one-off scripts just to answer simple questions.

What if you could just ask your logs?

In this article, we’ll build a Retrieval-Augmented Generation (RAG) powered VPC Flow Log Analyzer that turns static network telemetry into an interactive security assistant

The Challenge of Manual Log Analysis

AWS VPC Flow Logs capture essential information about network traffic. Yet, analysing these raw logs to detect threats like SQL injection attempts or unauthorised access presents significant challenges:

• Information Overload: The sheer volume of logs is overwhelming. Finding specific patterns or anomalies is like searching for a needle in a haystack.




• Context Fragmentation: Raw logs lack context. Identifying related packets across different components and time frames is labour-intensive and error-prone.

The RAG-based VPC Flow Log Analyser uses:

• Streamlit (interactive UI)


• LangChain (RAG orchestration)


• Chroma (vector database)


• OpenAI GPT-4o (reasoning engine)

At the end, you'll have a conversational security assistant capable of answering questions like:

• “Which IPs were rejected?”


• “Was there unusual traffic to port 22?”


• “Which destinations received the most packets?”

Step 1: Creating Virtual Environment and Installing Dependencies

git clone https://github.com/Damdev-95/rag_aws_flow_logs



python -m venv venv



source venv/bin/activate



cd rag_aws_flow_logs



pip install -r requirements.txt

• Once you click on 'Browse files', you will be able to upload log files on the application; ensure the log file format is in txt.

• Successfully created index events after the embedding process
  
  

  
  
  

  
  
  
  Additional examples of queries with interaction
  

  
  
  
  
  

  Stay tuned for additional RAG and GenerativeAI projects in cloud networking by reading my articles.

  
  
  

  I look forward to your comments.

  Vollständiger Original-Bericht

  Ausführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.

  ↗ Original-Artikel auf dev.to lesen