Crafting a Scalable Node.js API: Insights from My RealWorld Project with Express, Knex, and AWS CDK

and the infrastructure code

By adopting this serverless architecture, we benefit from scalability, reduced operational overhead, and efficient resource management, allowing us to focus on developing features and improving the application.

Infrastructure Management

The infrastructure for this project is fully managed using AWS CDK, and I have created a dedicated repository specifically for managing it. This repository includes all necessary resources such as IAM roles, SQS queues, and CodePipeline configurations that support the application.

To streamline the deployment process, this setup includes two CI/CD pipelines:

• 
  Infrastructure CI/CD Pipeline: This pipeline is triggered by GitHub Actions and handles the deployment of the entire infrastructure stack. If the pipeline code is well-structured (e.g., with no circular dependencies between stacks) and has been thoroughly tested in lower environments, it can deploy to production seamlessly without requiring manual adjustments on local. This pipeline updates the infrastructure components, ensuring consistency across all environments.


• 
  Lambda Deployment CI/CD Pipeline: This pipeline manages the deployment of the application code. It pulls the latest code from the application repository, builds the application, and uploads the resulting package to the Lambda function. This deployment pipeline is also triggered by GitHub Actions, allowing for a smooth and automated code deployment process.

Both CodePipeline processes are manually triggered through GitHub Actions. Each pipeline runs quality checks and test cases in GitHub Actions before triggering the corresponding AWS CodePipeline deployment for infrastructure or application code. This approach ensures that only tested and verified changes make it to production, providing a robust and automated deployment pipeline for the entire application stack.

Testing Strategies

This repository employs a comprehensive testing strategy, including unit tests, integration tests, and end-to-end tests, to ensure stability and reliability across all layers of the application.

• 
  Unit Tests: Unit tests are written for functions and core modules, targeting the smallest components in the application. By mocking dependencies within each layer - such as database, repository, and service layers - we can effectively test both happy paths and edge cases. This isolation allows us to focus on individual functions and modules, ensuring each component works as expected without dependencies interfering.


• 
  Integration Tests: Integration tests focus on validating the APIs in the apps layer. Using supertest in conjunction with an in-memory SQLite database, we test each API endpoint in a realistic environment. This approach enables us to verify that modules interact correctly and that data flows as expected from one layer to another.


• 
  End-to-End Tests: For end-to-end testing, we set up a real MySQL database in GitHub Actions and launch the server to simulate actual deployment conditions. We use Newman (a tool for running Postman collections) to test the full user experience, from API requests to final responses. This allows us to validate that the application behaves as intended from a user's perspective.

Testing is a crucial part of our development workflow, supporting rapid iteration and refactoring. Well-written test cases allow us to confidently add features or make modifications without the risk of breaking other parts of the system. Good test coverage also provides a foundation for future refactoring, ensuring that as long as tests pass, the refactored code performs correctly.

In many cases, tight timelines may require us to prioritize feature rollouts, which can lead to sacrifices in code quality. However, I prioritize writing robust test cases first, as they serve as a safety net for future improvements. This practice enables us to refactor and enhance the codebase with confidence, knowing that as long as tests pass, the quality and functionality of the code remain intact.

Conclusion

Refactoring this repository has been a valuable learning experience, allowing me to apply new technologies and best practices along the way. I continuously explore and integrate improvements to keep the project up-to-date and efficient. If you find this repository or article helpful, consider giving it a star or clap - your support is appreciated! Thank you.