1. Code Organization and Structure
tornadobest-practicescode-organizationperformancesecurity
Description
Comprehensive best practices and coding standards for the Tornado framework. This rule provides guidelines on code organization, performance, security, testing, and common pitfalls when developing Tornado applications.
Globs
**/*.py---
description: Comprehensive best practices and coding standards for the Tornado framework. This rule provides guidelines on code organization, performance, security, testing, and common pitfalls when developing Tornado applications.
globs: **/*.py
---
## 1. Code Organization and Structure
### 1.1. Directory Structure Best Practices
A well-structured directory layout enhances code maintainability and scalability. Consider the following structure:
project_name/
├── app/
│ ├── __init__.py
│ ├── handlers/
│ │ ├── __init__.py
│ │ ├── base_handler.py # Base class for handlers
│ │ ├── home_handler.py # Specific request handlers
│ │ ├── auth_handler.py
│ ├── models/
│ │ ├── __init__.py
│ │ ├── user.py # Data models
│ │ ├── packet.py
│ ├── services/
│ │ ├── __init__.py
│ │ ├── authentication_service.py # Business Logic
│ │ ├── packet_service.py
│ ├── utils/
│ │ ├── __init__.py
│ │ ├── database.py # Database connection and setup
│ │ ├── helpers.py # Utility functions
│ ├── config.py # Application configuration
│ ├── application.py # Tornado Application setup
│ └── main.py # Entry point for the application
├── tests/
│ ├── __init__.py
│ ├── test_handlers.py # Unit tests for handlers
│ ├── test_models.py # Unit tests for models
│ └── test_services.py # Unit tests for services
├── static/ # Static files (CSS, JavaScript, images)
├── templates/ # HTML templates
├── docs/ # Documentation
├── requirements.txt # Project dependencies
├── .env # Environment variables (development)
└── README.md
### 1.2. File Naming Conventions
* Use descriptive and consistent file names.
* Handlers: `home_handler.py`, `user_handler.py`
* Models: `user.py`, `product.py`
* Utilities: `database.py`, `string_utils.py`
* Tests: `test_handlers.py`, `test_models.py`
### 1.3. Module Organization
* Group related functionalities into modules.
* Keep modules focused and avoid creating overly large modules.
* Use clear and concise names for modules.
* Use packages (directories with `__init__.py`) to further organize modules.
### 1.4. Component Architecture
* **MVC (Model-View-Controller):** A common architectural pattern.
* **Models:** Represent data and business logic.
* **Views:** (Templates) Display data to the user.
* **Controllers (Handlers):** Handle user requests and interact with models.
* **Microservices:** Decompose the application into smaller, independent services.
* **Hexagonal Architecture (Ports and Adapters):** Isolates the core application logic from external dependencies.
### 1.5. Code Splitting
* **Split large handler files:** Decompose complex handlers into smaller, reusable functions or classes.
* **Modularize functionality:** Extract common logic into separate modules or services.
* **Lazy loading:** Import modules only when needed to reduce startup time. Avoid doing heavy imports at the top of the file. Consider `import tornado.ioloop` inside of a method if it's only used there.
## 2. Common Patterns and Anti-patterns
### 2.1. Design Patterns
* **Singleton:** For managing global resources like database connections.
* **Factory:** For creating instances of objects without specifying their concrete classes.
* **Observer:** For implementing event-driven architectures.
* **Template Method:** For defining a skeleton algorithm in a base class and letting subclasses implement specific steps.
* **Middleware:** Implement custom request processing logic (authentication, logging).
### 2.2. Recommended Approaches
* **Asynchronous Operations:** Leverage Tornado's asynchronous capabilities (using `async` and `await`) for non-blocking I/O operations. This includes database queries, API calls, and file system operations.
* **Use `IOLoop.add_callback` for thread safety:** When interacting with Tornado objects from other threads.
* **Configuration Management:** Utilize environment variables for storing configuration settings, adhering to the Twelve-Factor App methodology.
* **Logging:** Implement robust logging using Tornado's built-in logging capabilities.
* **Routing:** Utilize Tornado's routing system effectively by defining URL patterns and corresponding request handlers.
* **Authentication/Authorization:** Employ decorators like `@tornado.web.authenticated` for securing routes.
* **Use a BaseHandler class:** Encapsulate common logic, such as authentication checks, error handling, and template rendering.
### 2.3. Anti-patterns and Code Smells
* **Blocking I/O in Handlers:** Performing synchronous operations within request handlers will block the I/O loop, severely impacting performance. Always use asynchronous alternatives.
* **Ignoring Exceptions:** Catching exceptions without proper handling or logging can mask critical errors.
* **Overly Complex Handlers:** Large, monolithic handlers are difficult to maintain and test. Break them down into smaller, focused functions or classes.
* **Hardcoding Configuration:** Embedding configuration values directly in the code makes it difficult to manage and deploy the application across different environments. Use environment variables or configuration files.
* **Global State:** Over-reliance on global variables can lead to unpredictable behavior and make the code harder to reason about. Use dependency injection or other techniques to manage state explicitly.
* **Ignoring Thread Safety:** Tornado applications are single-threaded, but interactions with external resources might not be. Using `IOLoop.run_in_executor` is key to thread safety. If using WSGI containers, this becomes extremely important.
### 2.4. State Management
* **Stateless Handlers:** Design handlers to be stateless to improve scalability and fault tolerance.
* **Session Management:** Use signed cookies or external session stores (Redis, Memcached) for managing user sessions.
* **Caching:** Implement caching mechanisms (in-memory, Redis, Memcached) to reduce database load and improve response times.
* **Avoid Global Variables:** Minimize the use of global variables to prevent unintended side effects.
### 2.5. Error Handling
* **Centralized Error Handling:** Create a custom error handler (e.g., a base handler with error handling logic) to handle exceptions consistently across the application.
* **Logging:** Log all exceptions and errors with sufficient context (request details, user information).
* **Custom Error Pages:** Provide user-friendly error pages for common HTTP errors (404, 500).
* **Exception Propagation:** Allow exceptions to propagate to the Tornado error handler to ensure proper logging and handling.
* **Use `try...except` Blocks:** Wrap potentially failing code blocks in `try...except` blocks to gracefully handle errors.
## 3. Performance Considerations
### 3.1. Optimization Techniques
* **Asynchronous I/O:** Use asynchronous I/O operations to avoid blocking the I/O loop.
* **Connection Pooling:** Use connection pooling for database connections to reduce connection overhead. Libraries like SQLAlchemy provide this.
* **Caching:** Implement caching strategies to reduce database load and improve response times. Consider using `functools.lru_cache` for frequently called, pure functions.
* **Gzip Compression:** Enable Gzip compression for responses to reduce bandwidth usage.
* **Optimize Database Queries:** Ensure that database queries are efficient and properly indexed.
* **Minimize External Dependencies:** Reduce the number of external dependencies to minimize startup time and improve performance.
* **WebSockets**: Implement websockets using Tornado's native support for persistent connections to offload work from HTTP handlers.
### 3.2. Memory Management
* **Avoid Memory Leaks:** Be mindful of memory leaks, especially when dealing with long-lived connections or background tasks. Use tools like `memory_profiler` to identify and fix memory leaks.
* **Use Generators:** Use generators for processing large datasets to avoid loading the entire dataset into memory at once.
* **Object Pooling:** Use object pooling for frequently created objects to reduce object creation overhead.
* **Limit Object Sizes:** Avoid creating excessively large objects that can consume significant memory.
### 3.3. Rendering Optimization (If applicable)
* **Template Caching:** Cache compiled templates to reduce rendering overhead. Tornado automatically caches templates by default.
* **Minimize Template Logic:** Keep template logic simple and move complex calculations to the handler.
* **Use Template Inheritance:** Use template inheritance to reuse common template elements and reduce duplication.
* **Optimize Static Files:** Minimize, compress, and cache static files (CSS, JavaScript, images).
### 3.4. Bundle Size Optimization (If applicable)
* **Minify CSS and JavaScript:** Use tools like UglifyJS or CSSNano to minify CSS and JavaScript files.
* **Combine CSS and JavaScript:** Combine multiple CSS and JavaScript files into fewer files to reduce HTTP requests.
* **Use a CDN:** Serve static files from a Content Delivery Network (CDN) to improve loading times.
### 3.5. Lazy Loading
* **Lazy-load Modules:** Import modules only when needed to reduce startup time.
* **Lazy-load Images:** Load images only when they are visible in the viewport.
* **Lazy-load Data:** Fetch data only when it is needed.
## 4. Security Best Practices
### 4.1. Common Vulnerabilities
* **Cross-Site Scripting (XSS):** Sanitize user input to prevent XSS attacks.
* **SQL Injection:** Use parameterized queries or an ORM to prevent SQL injection attacks.
* **Cross-Site Request Forgery (CSRF):** Implement CSRF protection to prevent malicious websites from performing unauthorized actions on behalf of logged-in users. Tornado includes CSRF protection features.
* **Authentication and Authorization Flaws:** Implement robust authentication and authorization mechanisms to prevent unauthorized access to resources.
* **Session Hijacking:** Use secure cookies (HTTPOnly, Secure) and session management techniques to prevent session hijacking.
* **Denial of Service (DoS):** Implement rate limiting and other measures to prevent DoS attacks.
### 4.2. Input Validation
* **Validate All Input:** Validate all user input to prevent malicious data from entering the application.
* **Use Whitelisting:** Use whitelisting to define the allowed characters and formats for input fields.
* **Sanitize Input:** Sanitize input to remove or escape potentially dangerous characters.
* **Escape Output:** Escape output to prevent XSS attacks.
### 4.3. Authentication and Authorization
* **Use Strong Passwords:** Enforce strong password policies and use a secure hashing algorithm (bcrypt, Argon2) to store passwords.
* **Implement Two-Factor Authentication (2FA):** Use 2FA to add an extra layer of security to user accounts.
* **Use Role-Based Access Control (RBAC):** Implement RBAC to control access to resources based on user roles.
* **Principle of Least Privilege:** Grant users only the minimum privileges required to perform their tasks.
* **JSON Web Tokens (JWT):** Use JWT for stateless authentication.
### 4.4. Data Protection
* **Encrypt Sensitive Data:** Encrypt sensitive data at rest and in transit.
* **Use HTTPS:** Use HTTPS to encrypt communication between the client and server.
* **Store Secrets Securely:** Store secrets (API keys, database passwords) securely using environment variables or a secrets management system.
* **Regularly Rotate Secrets:** Rotate secrets regularly to reduce the risk of compromise.
### 4.5. Secure API Communication
* **Use HTTPS:** Enforce HTTPS for all API endpoints.
* **Implement API Authentication:** Use API keys, JWT, or OAuth 2.0 to authenticate API clients.
* **Rate Limiting:** Implement rate limiting to prevent abuse and DoS attacks.
* **Input Validation:** Validate all API input to prevent malicious data from entering the application.
* **Output Sanitization:** Sanitize API output to prevent XSS attacks.
## 5. Testing Approaches
### 5.1. Unit Testing
* **Test Individual Components:** Unit tests should focus on testing individual components (handlers, models, services) in isolation.
* **Use Mocking and Stubbing:** Use mocking and stubbing to isolate the component under test from its dependencies.
* **Test Edge Cases:** Test edge cases and boundary conditions to ensure that the component handles unexpected inputs correctly.
* **Follow Arrange-Act-Assert Pattern:** Structure unit tests using the Arrange-Act-Assert pattern.
* **Use `unittest` or `pytest`:** Python's built-in `unittest` module or the popular `pytest` framework are commonly used.
### 5.2. Integration Testing
* **Test Interactions Between Components:** Integration tests should focus on testing the interactions between different components of the application.
* **Use a Test Database:** Use a separate test database to avoid polluting the production database.
* **Test API Endpoints:** Test API endpoints to ensure that they function correctly and return the expected results.
* **Test Database Interactions:** Test database interactions to ensure that data is being read and written correctly.
### 5.3. End-to-End Testing
* **Test the Entire Application:** End-to-end tests should test the entire application from the user's perspective.
* **Use a Testing Framework:** Use a testing framework like Selenium or Cypress to automate end-to-end tests.
* **Test User Flows:** Test common user flows to ensure that the application functions correctly.
### 5.4. Test Organization
* **Organize Tests by Module:** Organize tests into separate files or directories for each module.
* **Use Descriptive Test Names:** Use descriptive test names to make it clear what each test is testing.
* **Keep Tests Independent:** Ensure that tests are independent of each other and can be run in any order.
### 5.5. Mocking and Stubbing
* **Use Mocking Libraries:** Use mocking libraries like `unittest.mock` or `pytest-mock` to create mock objects and stubs.
* **Mock External Dependencies:** Mock external dependencies (databases, APIs) to isolate the component under test.
* **Stub Complex Logic:** Stub complex logic to simplify the test and focus on the specific behavior being tested.
## 6. Common Pitfalls and Gotchas
### 6.1. Frequent Mistakes
* **Blocking the I/O Loop:** Performing synchronous operations in handlers.
* **Not Handling Exceptions Properly:** Ignoring or mishandling exceptions.
* **Incorrectly Using `async` and `await`:** Forgetting to `await` asynchronous calls.
* **Not Understanding Thread Safety:** Improperly accessing Tornado objects from threads.
* **Over-complicating handlers:** Making handlers too long and hard to understand.
### 6.2. Edge Cases
* **Handling Long-lived Connections:** Managing WebSocket connections and preventing memory leaks.
* **Dealing with Slow Clients:** Implementing timeouts and connection limits.
* **Handling Unexpected Input:** Validating and sanitizing user input.
* **Dealing with Network Errors:** Implementing retry logic and error handling.
### 6.3. Version-Specific Issues
* **Compatibility with Python Versions:** Ensuring compatibility with supported Python versions.
* **API Changes:** Being aware of API changes between Tornado versions.
* **Deprecated Features:** Avoiding the use of deprecated features.
### 6.4. Compatibility Concerns
* **WSGI Compatibility:** Understanding the limitations of WSGI integration.
* **Asynchronous Libraries:** Ensuring compatibility with other asynchronous libraries.
* **Database Drivers:** Using asynchronous database drivers compatible with Tornado.
### 6.5. Debugging Strategies
* **Logging:** Use extensive logging to track the flow of execution and identify errors.
* **Debugging Tools:** Use debugging tools like `pdb` or IDE debuggers to step through the code and inspect variables.
* **Profiling:** Use profiling tools to identify performance bottlenecks.
* **Monitoring:** Use monitoring tools to track the application's health and performance in production.
## 7. Tooling and Environment
### 7.1. Recommended Development Tools
* **IDE:** VS Code, PyCharm
* **Virtual Environment:** `venv`, `virtualenvwrapper`, `conda`
* **Package Manager:** `pip`, `uv`
* **Debugging Tools:** `pdb`, IDE debuggers
* **Profiling Tools:** `cProfile`, `memory_profiler`
* **Linting and Formatting:** `flake8`, `pylint`, `black`
* **Testing Frameworks:** `unittest`, `pytest`
### 7.2. Build Configuration
* **Use `requirements.txt`:** Specify project dependencies in a `requirements.txt` file.
* **Use `setup.py` or `pyproject.toml`:** For packaging and distributing the application.
* **Use a Build System:** Consider using a build system like `make` or `tox` to automate build tasks.
### 7.3. Linting and Formatting
* **Use a Linter:** Use a linter like `flake8` or `pylint` to enforce coding standards and identify potential errors.
* **Use a Formatter:** Use a formatter like `black` to automatically format the code according to PEP 8.
* **Configure Editor:** Configure the editor to automatically run the linter and formatter on save.
### 7.4. Deployment
* **Use a Production-Ready Server:** Deploy the application on a production-ready server like Gunicorn or uWSGI.
* **Use a Reverse Proxy:** Use a reverse proxy like Nginx or Apache to handle SSL termination, load balancing, and caching.
* **Use a Process Manager:** Use a process manager like Supervisor or systemd to manage the application process.
* **Use a Containerization Technology:** Consider using containerization technologies like Docker to package and deploy the application.
### 7.5. CI/CD Integration
* **Use a CI/CD System:** Use a CI/CD system like Jenkins, GitLab CI, GitHub Actions, or CircleCI to automate the build, test, and deployment processes.
* **Automate Testing:** Automate unit, integration, and end-to-end tests as part of the CI/CD pipeline.
* **Automate Deployment:** Automate the deployment process to reduce the risk of human error.
* **Implement Rollback Strategy:** Implement a rollback strategy to quickly revert to a previous version in case of errors.