1. Code Organization and Structure

---
description: This rule file provides comprehensive best practices for Django's Object-Relational Mapper (ORM), covering code organization, performance, security, testing, and common pitfalls. It aims to guide developers in building efficient, maintainable, and secure Django applications.
globs: **/*.py
---

## 1. Code Organization and Structure

### 1.1. Directory Structure Best Practices

- **Application-Centric Structure:** Organize your Django project around applications (apps), where each app represents a specific feature or module.
- **Dedicated 'models' Module:** Keep all model definitions within the `models.py` file of each app.
- **Separate Queries:** For complex queries, consider creating a `queries.py` or `managers.py` file within your app to house custom querysets and managers.
- **Consistent Naming:** Use consistent naming conventions for app and model names.


myproject/
 ├── myproject/
 │   ├── __init__.py
 │   ├── asgi.py
 │   ├── settings.py
 │   ├── urls.py
 │   ├── wsgi.py
 ├── app1/
 │   ├── __init__.py
 │   ├── admin.py
 │   ├── apps.py
 │   ├── models.py
 │   ├── managers.py  # Custom managers and querysets
 │   ├── views.py
 │   ├── urls.py
 │   ├── forms.py
 │   ├── tests.py
 │   └── migrations/
 ├── app2/
 │   └── ...
 └── manage.py


### 1.2. File Naming Conventions

- **Models:** Use singular nouns and CamelCase (e.g., `UserProfile`, `BlogPost`).
- **Fields:** Use lowercase with underscores (snake_case, e.g., `first_name`, `date_joined`).
- **Managers/Querysets:** Use descriptive names related to their functionality (e.g., `ActiveUsersManager`, `PublishedPostsQuerySet`).
- **Files:** Use lowercase with underscores (snake_case, e.g., `models.py`, `forms.py`, `urls.py`).

### 1.3. Module Organization

- **Keep Models Concise:** Limit the number of models in each app to a manageable amount (ideally under 10). If an app becomes too large, refactor it into smaller, more focused apps.
- **Avoid Circular Imports:** Be mindful of circular dependencies between models and other modules within the same app or across apps.  Use string references in `ForeignKey` and `ManyToManyField` definitions to avoid immediate import requirements.
- **Utilize Abstract Base Classes:** For common fields across multiple models, define an abstract base class and inherit from it. This promotes code reuse and reduces redundancy. Don't forget to set `abstract = True` in the `Meta` class.
- **Consider Proxy Models:** Use proxy models (`proxy = True` in `Meta`) to change a model's behavior without altering the database schema. This is useful for adding methods or changing the default ordering.

### 1.4. Component Architecture

- **DRY (Don't Repeat Yourself):** Create reusable components such as custom model fields, managers, and querysets.
- **Model Mixins:** Use model mixins to add common functionality to multiple models (e.g., a mixin for soft deletion or timestamping).
- **Form Abstraction:** Create reusable form classes and widgets for data input and validation.
- **Template Tags and Filters:** Develop custom template tags and filters to simplify common tasks in templates related to model data.

### 1.5. Code Splitting Strategies

- **App-Based Splitting:** Organize models and related code within distinct Django apps.
- **Manager-Based Splitting:** Divide complex queries into custom managers and querysets.
- **Mixin-Based Splitting:** Utilize mixins to compartmentalize functionality across models.
- **Signal-Based Splitting:** Implement signals for decoupling operations, such as triggering tasks when a model is saved or deleted.

## 2. Common Patterns and Anti-patterns

### 2.1. Design Patterns

- **Repository Pattern:** Abstract data access logic behind a repository interface, allowing you to switch between different data sources without modifying the rest of your application.
- **Unit of Work Pattern:** Group multiple database operations into a single transaction to ensure atomicity and consistency.
- **Specification Pattern:** Encapsulate query logic into reusable specification objects, allowing you to combine and reuse queries across different parts of your application.
- **Model Decorators:** Use decorators to add functionality to models without modifying the original model class. Useful for things like caching calculated properties.

### 2.2. Recommended Approaches for Common Tasks

- **Filtering Data:** Use `filter()` and `exclude()` for efficient data retrieval.
- **Ordering Data:** Utilize `order_by()` to specify the order of results.
- **Aggregating Data:** Employ `aggregate()` to calculate summary statistics.
- **Creating and Updating Objects:** Use `create()` and `save()` for object manipulation.
- **Deleting Objects:** Use `delete()` for removing objects.
- **Prefetch Related:** Use `select_related` for ForeignKey and OneToOneField relationships, and `prefetch_related` for ManyToManyField and reverse ForeignKey relationships.

### 2.3. Anti-patterns and Code Smells

- **Overusing `raw()` Queries:** Prefer Django ORM methods over raw SQL queries to maintain database abstraction and security.
- **Ignoring Database Indexes:** Neglecting to add indexes to frequently queried columns can lead to significant performance bottlenecks. Use `db_index = True` and `Index` in Meta to specify indices.
- **Using `null=True` for String-Based Fields:** Avoid `null=True` for `CharField` and `TextField` fields. Use `blank=True` and an empty string (`''`) instead.
- **Creating Large Querysets without Pagination:** Avoid retrieving large datasets without pagination. Use Django's `Paginator` to break results into manageable chunks.
- **Performing Logic in Templates:** Keep templates focused on presentation. Move complex logic to views or custom template tags.
- **N+1 Query Problem:** Accessing related objects in a loop can cause the N+1 query problem. Use `select_related()` and `prefetch_related()` to mitigate this.
- **Inefficient Use of Exists/Count:** Prefer `exists()` over `count()` when simply checking for the existence of objects.
- **Unnecessary Database Hits:** Optimize data access patterns to minimize the number of database queries.

### 2.4. State Management

- **Database as Primary State:** Rely on the database as the single source of truth for application state.
- **Session Management:** Utilize Django's session framework for storing user-specific state.
- **Caching:** Implement caching strategies to reduce database load and improve response times.
- **Avoid Global Mutable State:**  Minimize the use of global mutable state, as it can lead to unexpected behavior and concurrency issues.

### 2.5. Error Handling

- **Catch Database Exceptions:** Handle database exceptions (e.g., `IntegrityError`, `OperationalError`) gracefully.
- **Use Transactions:** Wrap multiple database operations in a transaction to ensure atomicity.
- **Log Errors:** Log errors and exceptions for debugging and monitoring purposes.  Use `logger.exception()` to capture the traceback.
- **Custom Exception Handling Middleware:** Consider writing custom middleware to catch and handle database errors at a global level, providing user-friendly error messages.
- **Validate Data Before Saving:** Always validate data before saving it to the database to prevent errors and data inconsistencies. Use Django's built-in validators or create custom validators.

## 3. Performance Considerations

### 3.1. Optimization Techniques

- **Query Optimization:** Use `select_related` and `prefetch_related` to reduce the number of database queries.
- **Indexing:** Add indexes to frequently queried columns.
- **Caching:** Implement caching strategies for frequently accessed data.
- **Pagination:** Use pagination for large datasets.
- **Batch Operations:** Use `bulk_create`, `bulk_update` and `bulk_delete` for bulk operations.
- **Defer/Only:** use `defer` to exclude fields from the query or `only` to only include specific fields in the query.
- **Conditional Expressions:** Use `Case`, `When`, and `Value` within queries to perform conditional logic directly in the database.

### 3.2. Memory Management

- **Limit Queryset Size:** Avoid retrieving large datasets into memory. Use pagination or generators.
- **Use Generators:** For processing large datasets, use generators to process data in chunks.
- **Garbage Collection:** Understand Python's garbage collection mechanisms and how they affect memory usage.
- **Profiling:** Use profiling tools to identify memory leaks and optimize memory usage.

### 3.3. Rendering Optimization

- **Template Caching:** Cache frequently used templates to reduce rendering time.
- **Context Processors:** Minimize the data passed to templates via context processors.
- **Lazy Loading:** Use lazy loading for images and other resources to improve initial page load time.  Consider using template tags like `{% static %}` and `{% get_static_prefix %}` effectively.

### 3.4. Bundle Size Optimization

- **Remove Unused Code:** Eliminate unused models, fields, and queries.
- **Optimize Static Files:** Minify and compress static files (CSS, JavaScript, images).
- **Code Splitting:**  Split JavaScript and CSS into smaller chunks for efficient loading.

### 3.5. Lazy Loading

- **Load Related Data on Demand:** Use lazy loading for related data to improve initial performance. Consider using Django Rest Framework serializers with `depth=0` and selectively increasing the depth when needed.
- **Database Views:** Create database views for complex queries to improve performance.
- **QuerySet Iterators:** Instead of converting large querysets into lists, iterate over them using `.iterator()` for memory efficiency.

## 4. Security Best Practices

### 4.1. Common Vulnerabilities and Prevention

- **SQL Injection:** Prevent SQL injection by using Django ORM methods instead of raw SQL queries. Always parameterize your queries.
- **Cross-Site Scripting (XSS):** Sanitize user input and escape output in templates.
- **Cross-Site Request Forgery (CSRF):** Use Django's CSRF protection mechanisms.
- **Authentication and Authorization:** Implement robust authentication and authorization mechanisms.
- **Session Hijacking:** Secure sessions using HTTPS and appropriate session settings.

### 4.2. Input Validation

- **Use Django Forms:** Employ Django Forms for validating user input.
- **Sanitize User Input:** Sanitize user input to prevent XSS attacks.
- **Validate Data Types:** Validate data types to prevent database errors.
- **Limit Input Length:** Restrict input length to prevent buffer overflows.
- **Custom Validators:** Create custom validators for complex validation logic.
- **Consider using DRF Serializers:** Use Django Rest Framework Serializers to add an extra layer of validation to your models and data.

### 4.3. Authentication and Authorization

- **Use Django's Built-in Authentication:** Leverage Django's built-in authentication framework.
- **Implement Role-Based Access Control (RBAC):** Define roles and permissions for different user types.
- **Enforce Password Policies:** Enforce strong password policies.
- **Use Multi-Factor Authentication (MFA):** Implement MFA for sensitive accounts.
- **Rate Limiting:** Implement rate limiting to prevent brute-force attacks.

### 4.4. Data Protection

- **Use HTTPS:** Use HTTPS to encrypt data in transit.
- **Encrypt Sensitive Data:** Encrypt sensitive data at rest.
- **Store Passwords Securely:** Store passwords using a strong hashing algorithm (e.g., bcrypt, Argon2).
- **Regularly Back Up Data:** Regularly back up your database.
- **Audit Logging:** Implement audit logging to track user activity and data changes.

### 4.5. Secure API Communication

- **Use API Keys:** Use API keys for authenticating API requests.
- **Implement OAuth 2.0:** Implement OAuth 2.0 for secure API authorization.
- **Validate API Input:** Validate API input to prevent injection attacks.
- **Rate Limiting:** Implement rate limiting to prevent API abuse.
- **API Versioning:** Use API versioning to manage changes to your API.

## 5. Testing Approaches

### 5.1. Unit Testing

- **Test Model Methods:** Unit test model methods to ensure they behave as expected.
- **Test Custom Managers:** Unit test custom managers to verify their query logic.
- **Test Validators:** Unit test validators to ensure they correctly validate data.
- **Mock Database Dependencies:** Use mocking to isolate model tests from the database.

### 5.2. Integration Testing

- **Test Views and Models:** Integrate test views and models to ensure they work together correctly.
- **Test Forms and Models:** Integrate test forms and models to verify data validation and persistence.
- **Use Test Fixtures:** Use test fixtures to create consistent test data.
- **Test Database Interactions:** Test database interactions to ensure they are efficient and correct.

### 5.3. End-to-End Testing

- **Test User Workflows:** End-to-end test user workflows to ensure they are functional and user-friendly.
- **Test API Endpoints:** End-to-end test API endpoints to ensure they are secure and functional.
- **Use Selenium or Similar Tools:** Use Selenium or similar tools for automating end-to-end tests.
- **Test Across Different Browsers:** Test across different browsers to ensure compatibility.

### 5.4. Test Organization

- **Organize Tests by App:** Organize tests by app to improve maintainability.
- **Use Descriptive Test Names:** Use descriptive test names to clearly indicate what is being tested.
- **Follow a Consistent Test Structure:** Follow a consistent test structure to improve readability.
- **Use Test Runners:** Use test runners to automate test execution.

### 5.5. Mocking and Stubbing

- **Mock Database Queries:** Mock database queries to isolate tests from the database.
- **Stub External Dependencies:** Stub external dependencies to control test environments.
- **Use Django's `override_settings` Decorator:** Use Django's `override_settings` decorator to modify settings during tests.
- **Use `patch` for Mocking:** Use `unittest.mock.patch` for mocking objects and functions.

## 6. Common Pitfalls and Gotchas

### 6.1. Frequent Mistakes

- **Not Using Indexes:** Forgetting to add indexes to frequently queried columns.
- **Overusing `null=True`:** Using `null=True` for string-based fields.
- **Ignoring Database Transactions:** Not using database transactions for critical operations.
- **Creating Large Querysets:** Creating large querysets without pagination.
- **Performing Logic in Templates:** Performing logic in templates instead of views.
- **N+1 Query Problem:** Not addressing the N+1 query problem.

### 6.2. Edge Cases

- **Unicode Handling:** Handling Unicode characters correctly.
- **Time Zones:** Dealing with time zones accurately.
- **Concurrency Issues:** Managing concurrency issues in multi-threaded environments.
- **Database-Specific Differences:** Handling database-specific differences.

### 6.3. Version-Specific Issues

- **Deprecated Features:** Being aware of deprecated features and migrating to newer alternatives.
- **Compatibility Changes:** Handling compatibility changes between Django versions.

### 6.4. Compatibility Concerns

- **Python Version:** Ensuring compatibility with the supported Python versions.
- **Database Compatibility:** Ensuring compatibility with the chosen database.
- **Third-Party Libraries:** Ensuring compatibility with third-party libraries.

### 6.5. Debugging

- **Use Django's Debug Toolbar:** Use Django's Debug Toolbar for profiling and debugging.
- **Inspect Database Queries:** Inspect database queries to identify performance bottlenecks.
- **Log Errors and Exceptions:** Log errors and exceptions for debugging purposes.
- **Use a Debugger:** Use a debugger to step through code and inspect variables.
- **Read Stack Traces Carefully:** Read stack traces carefully to understand the source of errors.

## 7. Tooling and Environment

### 7.1. Recommended Development Tools

- **IDE:** PyCharm, VS Code with Python extension.
- **Database Client:** pgAdmin, MySQL Workbench, DB Browser for SQLite.
- **Virtual Environment Manager:** virtualenv, pipenv, conda.
- **Debug Toolbar:** Django Debug Toolbar.
- **Profiling Tools:** cProfile, line_profiler.

### 7.2. Build Configuration

- **Use a Virtual Environment:** Use a virtual environment to isolate project dependencies.
- **Specify Dependencies:** Specify dependencies in a `requirements.txt` or `Pipfile`.
- **Use a Build System:** Use a build system (e.g., Make, Fabric) to automate build tasks.
- **Configure Static File Handling:** Configure static file handling correctly for production.

### 7.3. Linting and Formatting

- **Use a Linter:** Use a linter (e.g., pylint, flake8) to enforce code style.
- **Use a Formatter:** Use a formatter (e.g., black, autopep8) to automatically format code.
- **Configure Editor Integration:** Configure editor integration to automatically lint and format code on save.
- **Follow PEP 8:** Adhere to PEP 8 style guidelines.

### 7.4. Deployment

- **Use a Production-Ready Web Server:** Use a production-ready web server (e.g., Gunicorn, uWSGI).
- **Use a Reverse Proxy:** Use a reverse proxy (e.g., Nginx, Apache) to handle static files and load balancing.
- **Use a Database Server:** Use a dedicated database server (e.g., PostgreSQL, MySQL).
- **Configure Caching:** Configure caching to improve performance.
- **Secure Deployment:** Secure the deployment environment with appropriate firewalls and security settings.
- **Use Environment Variables:** Store sensitive configuration data (e.g., database passwords, API keys) in environment variables.

### 7.5. CI/CD Integration

- **Use a CI/CD System:** Use a CI/CD system (e.g., Jenkins, Travis CI, GitLab CI) to automate testing and deployment.
- **Run Tests Automatically:** Run tests automatically on every commit.
- **Deploy Automatically:** Deploy automatically on successful builds.
- **Use Infrastructure as Code:** Use infrastructure as code (e.g., Terraform, Ansible) to automate infrastructure provisioning.

---

By consistently applying these best practices, you'll establish a robust foundation for building high-quality Django applications that are performant, maintainable, and secure.