feat: Rebase#29
Merged
Merged
Conversation
Fix/bugfixes
Add weighted OHEM loss; Add torch.amp.GradScaler to train
Add cityscapes class weights
Add more data augmentations; Update evaluate
Add focal loss weighted
Fix/bugfixes
Add 4 patch splitting inside uavid; Add data augmentations to uavid
Add community health files, update to Apache 2.0
Move license outisde
feat: Add correct email
This commit addresses all high-priority refactoring issues identified in the code review: 1. Fix code duplication: - Extract duplicate _DWConv and _DSConv classes to shared module - Create src/models/layers/common.py with reusable layer components - Update cab.py and cabinet.py to import from shared module 2. Improve error handling patterns: - Create src/utils/exceptions.py with custom exception classes - Replace generic Exception catches with specific exception types - Add ModelLoadError, ConfigurationError, TrainingError, DatasetError - Use logging instead of print statements for errors - Include better error context and traceback information 3. Add comprehensive type annotations: - Add type hints to all forward() methods in model classes - Add return type annotations (-> torch.Tensor, -> Tuple, etc.) - Add parameter type hints across cabinet.py, cab.py - Add docstrings with Args and Returns sections - Improve code documentation and IDE support 4. Extract magic numbers to named constants: - Create src/models/constants.py for all configuration constants - Replace hardcoded 960/576 with MOBILENET_*_FEATURES - Replace hardcoded 0.7 with DEFAULT_SCORE_THRESHOLD - Replace hardcoded //16 with OHEM_DIVISOR - Replace hardcoded 5/6.0 with EVAL_STRIDE_RATE - Replace hardcoded eval scales with DEFAULT_EVAL_SCALES - Replace hardcoded 19 classes with CITYSCAPES_NUM_CLASSES - Replace hardcoded 50 limit with VISUALIZATION_SAMPLE_LIMIT - Add MODEL_CONFIG dictionary for mode-specific settings Files changed: - New: src/models/layers/ (shared layer components) - New: src/models/constants.py (configuration constants) - New: src/utils/exceptions.py (custom exceptions) - Modified: src/models/cabinet.py (use shared layers, constants, types) - Modified: src/models/cab.py (use shared layers, add type hints) - Modified: src/scripts/train.py (use constants, better error handling) - Modified: src/scripts/evaluate.py (use constants) - Modified: src/scripts/visualize.py (use constants, add type hints) Impact: - Reduces code duplication by ~30 lines - Improves code maintainability and readability - Enables better IDE autocomplete and type checking - Makes configuration changes centralized and easier - Provides more informative error messages
This commit addresses medium-priority refactoring issues: 1. Remove commented-out dead code: - Removed unused `# device = x.device` in cabinet.py - Cleaned up code comments that served no purpose 2. Fix inconsistent naming conventions: - Renamed `h_sigmoid` → `HardSigmoid` (PascalCase) - Renamed `h_swish` → `HardSwish` (PascalCase) - Updated all references throughout mobilenetv3.py - Added type annotations to activation classes - Added docstrings explaining each activation function 3. Add comprehensive docstrings: - Added module-level docstring to transform.py - Added detailed docstring to CABiNet forward() method - Added docstrings to HardSigmoid and HardSwish classes - Improved documentation across the codebase 4. Improve code documentation: - Added typing imports where needed - Enhanced existing docstrings with better descriptions - Added parameter and return type documentation Files changed: - Modified: src/models/cabinet.py (remove dead code, add docstrings) - Modified: src/models/mobilenetv3.py (rename classes, add types) - Modified: src/datasets/transform.py (add module docstring, imports) Impact: - Improved code readability and maintainability - Better IDE support with proper class names - Enhanced documentation for developers - Consistent naming conventions throughout
This commit addresses low-priority enhancements to improve development workflow and project infrastructure: 1. Add requirements.txt: - Created comprehensive requirements.txt with pinned versions - Organized dependencies by category (framework, data, config, dev) - Included development tools (pytest, black, isort, flake8, mypy) - Added version constraints for reproducibility 2. Add development documentation: - Created CONTRIBUTING.md with comprehensive guidelines - Added setup instructions and development workflow - Documented code style guidelines and naming conventions - Included PR process and commit message format - Added testing and documentation standards - Provided code quality check commands 3. Infrastructure improvements: - Established consistent development practices - Documented code organization structure - Added performance considerations - Included examples for common tasks Files changed: - New: requirements.txt (project dependencies) - New: CONTRIBUTING.md (development guidelines) Impact: - Easier onboarding for new contributors - Consistent development environment setup - Clear guidelines for code quality and style - Better project documentation - Reproducible dependency management Next Steps: - Contributors can now follow standardized workflow - Easy to set up development environment - Clear expectations for code quality
…iority-YYeqX refactor: Phase 1 - High priority fixes
…priority-YYeqX refactor: Phase 2 - Medium priority fixes
…ority-YYeqX refactor: Phase 3 - Low priority enhancements
This commit adds comprehensive testing infrastructure, CI/CD pipelines,
and performance profiling utilities.
## 1. Comprehensive Test Suite
### Unit Tests
- Created `tests/unit/test_models.py`:
* Tests for all model components (DepthwiseConv, CAB, CABiNet)
* Parametrized tests for different model modes
* Tests for forward/backward passes
* CUDA compatibility tests
* Model configuration validation tests
- Created `tests/unit/test_loss.py`:
* OHEM Cross Entropy Loss tests
* Softmax Focal Loss tests
* Tests with class weights and ignore labels
* Backward pass validation
* Edge case handling (empty valid pixels)
- Created `tests/unit/test_transforms.py`:
* Tests for all data augmentation transforms
* Geometric transforms (scale, flip, crop, rotate)
* Photometric transforms (color jitter, gamma, noise)
* Regularization (cutout)
* Compose pipeline tests
### Integration Tests
- Created `tests/integration/test_training_pipeline.py`:
* End-to-end training step tests
* Loss reduction validation
* DataLoader integration tests
* CUDA training tests
* Evaluation mode consistency tests
### Test Infrastructure
- `tests/conftest.py`: Shared fixtures (devices, sample data, configs)
- `tests/README.md`: Comprehensive testing documentation
- Proper test structure with unit/integration/fixtures separation
## 2. GitHub Actions CI/CD Pipeline
### Main CI Workflow (`.github/workflows/ci.yml`)
- **Code Quality Checks**:
* Black formatting validation
* isort import sorting
* flake8 linting
* mypy type checking
- **Test Matrix**:
* Multiple Python versions (3.8, 3.9, 3.10, 3.11)
* Multiple OS (Ubuntu, macOS)
* Parallel test execution with pytest-xdist
* Coverage reporting with Codecov integration
- **Security Scanning**:
* Bandit security checks
* Safety vulnerability checks
- **Package Build**:
* Build validation
* Package checking with twine
* Artifact upload for distribution
### Pre-commit Workflow (`.github/workflows/pre-commit.yml`)
- Automated pre-commit hook execution on PRs
- Caching for faster runs
- Shows diffs on failures
## 3. Performance Profiling Utilities
Created `src/utils/profiler.py`:
### PerformanceProfiler Class
- **Inference Time Measurement**:
* Average, min, max, median latency
* FPS calculation
* Warmup iterations for accurate timing
* Statistical analysis with std deviation
- **Memory Usage Profiling**:
* Peak GPU memory tracking
* Baseline vs inference memory
* Current memory allocation
- **FLOPs Profiling**:
* torch.profiler integration
* CPU and CUDA time tracking
* Detailed operation breakdown
- **Full Benchmark Suite**:
* Combined timing, memory, and FLOPs analysis
* Comprehensive reporting
* Easy to use API
### Utility Functions
- `count_parameters()`: Model parameter counting
- Context manager for timing operations
- Proper CUDA synchronization
## 4. Enhanced Pre-commit Hooks
Improved `.pre-commit-config.yaml`:
### Added Hooks
- **mypy**: Type checking with proper configuration
- **bandit**: Security vulnerability scanning
- **pyupgrade**: Automatic Python syntax upgrades
- **nbstripout**: Jupyter notebook output stripping
- **pylint**: TODO/FIXME comment tracking (manual)
### Improved Configuration
- Consistent line length (100)
- Better exclusion patterns (legacy/, notebooks)
- Additional flake8 plugins (docstrings, bugbear, comprehensions)
- Standard hooks improvements:
* check-ast, check-builtin-literals
* debug-statements, end-of-file-fixer
* mixed-line-ending, trailing-whitespace
### CI Integration
- Auto-fix commit messages
- Weekly auto-updates
- Proper skip configuration for slow hooks
## Impact
### Testing
- **Coverage**: Comprehensive test suite for all components
- **Confidence**: Automated validation of changes
- **Quality**: Catch bugs before they reach production
- **Documentation**: Tests serve as usage examples
### CI/CD
- **Automation**: Automated testing on every PR
- **Multi-platform**: Tests on multiple Python versions and OS
- **Fast Feedback**: Parallel testing and caching
- **Security**: Automated vulnerability scanning
### Performance
- **Benchmarking**: Easy performance measurement
- **Optimization**: Identify bottlenecks
- **Monitoring**: Track performance across changes
- **Profiling**: Detailed analysis tools
### Developer Experience
- **Pre-commit**: Catch issues before commit
- **Consistency**: Automated code formatting
- **Quality**: Multiple quality checks
- **Documentation**: Clear testing guidelines
## Files Added/Modified
**New Files:**
- `.github/workflows/ci.yml` - Main CI pipeline
- `.github/workflows/pre-commit.yml` - Pre-commit workflow
- `src/utils/profiler.py` - Performance profiling
- `tests/conftest.py` - Test fixtures
- `tests/README.md` - Testing documentation
- `tests/__init__.py` - Test package init
- `tests/unit/test_models.py` - Model tests
- `tests/unit/test_loss.py` - Loss function tests
- `tests/unit/test_transforms.py` - Transform tests
- `tests/integration/test_training_pipeline.py` - Integration tests
- `tests/{unit,integration,fixtures}/__init__.py` - Package inits
**Modified Files:**
- `.pre-commit-config.yaml` - Enhanced with more hooks
## Usage
### Running Tests
```bash
# All tests
pytest tests/
# With coverage
pytest tests/ --cov=src --cov-report=html
# Parallel execution
pytest tests/ -n auto
```
### Performance Profiling
```python
from src.utils.profiler import PerformanceProfiler
profiler = PerformanceProfiler(model)
results = profiler.run_full_benchmark()
```
### Pre-commit
```bash
# Install hooks
pre-commit install
# Run manually
pre-commit run --all-files
```
## Next Steps
- CI/CD pipelines will run automatically on PRs
- Tests provide regression protection
- Performance profiling enables optimization
- Pre-commit ensures code quality
…-cicd-YYeqX feat: Phase 4 - Testing, CI/CD, and Performance
…oject structure - Add badges for license, Python version, and PyTorch - Include performance metrics table - Add detailed project structure with file descriptions - Add links to key files and directories - Improve installation instructions with multiple options - Expand usage examples for training, evaluation, and profiling - Add testing and development sections - Maintain all original content while improving organization - Remove excessive formatting, keep professional tone
fix: Fix tests
…-cicd-YYeqX fix: Branch names in CI
- Format code with black for consistent styling - Fix import ordering with isort - Remove unused imports (F401 flake8 errors) - Remove unused typing imports from transform.py - Remove unused layer imports from cabinet.py - Remove unused Optional from common.py - Remove unused TrainingError from train.py - Remove unused Callable from profiler.py - Remove .github/README.md (duplicate of main README) All critical CI checks now pass: - black --check: ✓ - isort --check-only: ✓ - flake8 critical errors: ✓
- Update actions/upload-artifact from v3 to v4 - Update actions/cache from v3 to v4 Fixes deprecation warnings in CI pipeline.
Replace types-all with specific type stubs (types-PyYAML, numpy) to avoid dependency resolution errors. The types-all meta-package has issues with types-pkg-resources which no longer exists. Since we use --ignore-missing-imports, we only need type stubs for core dependencies that we actually use in type annotations.
Add version suffix to cache key to force invalidation of old cached pre-commit environments that still reference types-all.
- Replace flake8 + isort with ruff for faster linting - Update all hook versions to latest (pre-commit-hooks v6.0.0, black 25.12.0, mypy v1.19.1) - Simplify mypy dependencies (types-PyYAML, types-requests) - Add conventional-pre-commit for commit message validation - Remove redundant hooks (docformatter, pyupgrade, nbstripout, pylint) - Bust cache to v3 to force new environment - Fix unused imports detected by ruff in test files
fix: CI code quality checks and remove duplicate README
Signed-off-by: Saumya Saksena <saumya.saksena@gitlab.navya.tech>
chore: Add citation files
Signed-off-by: dronefreak <kumaar324@gmail.com>
chore: Add citation files
Signed-off-by: dronefreak <kumaar324@gmail.com>
chore: Add citation files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebasing