Python Boilerplate
python_boilerplate is a boilerplate project for Python. Based on template sourcery-ai/python-best-practices-cookiecutter.
Features
Here are the highlights of python_boilerplate:
-
Inherited from modern and the latest Python technologies:
Pipenv
is to Python virtualenv management tool for the project. -
Data validation using Python type hints with Pydantic.
-
Highly customizable data analysis with pandas, enhanced array operation with NumPy. Supports CSV, Excel, JSON, and so on.
-
Simple and flexible retry with Tenacity.
-
Environment variable and configuration with pyhocon. Read
${ENVIRONMENT_VARIABLE}
when startup. -
Sensible and human-friendly approach to creating, manipulating, formatting and converting dates, times, and timestamps with Arrow.
-
Generate fake data with Faker.
-
Customized function decorator
@async_function
to enable function to run asynchronously;@peewee_table
class decorator to register ORM tables;@elapsed_time(level="INFO")
to profile a function elapsed time. -
Testing with pytest, integrating pytest-mock for mocking, pytest-cov for code coverage analysis and pyinstrument for Python stack profiler.
-
Formatting with black.
-
Import sorting with isort.
-
Static typing with mypy.
-
Linting with flake8.
-
Git hooks that run all the above with pre-commit.
-
Deployment ready with Docker.
-
Continuous Integration with GitHub Actions.
-
Loguru logging configuration. The log sample,
2022-09-17 14:13:52.385 | ⚠️ WARNING | 6860 | MainThread | python_boilerplate.repository.model.base_model.<module>:24 - SQLite database created. Path: [/Users/johnny/Projects/PyCharmProjects/python_boilerplate/data/python_boilerplate.db], <peewee.SqliteDatabase object at 0x1191e1390>
2022-09-17 14:13:52.386 | ℹ️ INFO | 6860 | MainThread | python_boilerplate.common.orm.peewee_table:16 - Registering peewee table: StartupLog
2022-09-17 14:13:52.387 | 🐞 DEBUG | 6860 | MainThread | peewee.execute_sql:3185 - ('CREATE TABLE IF NOT EXISTS "startup_log" ("id" INTEGER NOT NULL PRIMARY KEY, "current_user" VARCHAR(50) NOT NULL, "host" VARCHAR(50) NOT NULL, "command_line" TEXT NOT NULL, "current_working_directory" TEXT NOT NULL, "startup_time" DATETIME NOT NULL, "created_by" VARCHAR(50) NOT NULL, "created_time" DATETIME NOT NULL, "modified_by" VARCHAR(50) NOT NULL, "modified_time" DATETIME NOT NULL)', [])
2022-09-17 14:13:52.530 | ℹ️ INFO | 6860 | MainThread | python_boilerplate.<module>:53 - Started python_boilerplate in 0.117 seconds (117.26 ms)
Usage
-
Clone or download this project.
$ git clone https://github.com/johnnymillergh/python_boilerplater.git
-
Build with the newest PyCharm.
-
Click the green triangle to Run.
Setup
-
Setup the development environment
# Install pipx if pipenv and cookiecutter are not installed $ python3 -m pip install pipx $ python3 -m pipx ensurepath # Install pipenv using pipx $ pipx install pipenv
-
Install dependencies
$ pipenv install --quiet --dev
-
Install mypy types
$ pipenv run mypy --install-types
-
Setup pre-commit and pre-push hooks
$ pipenv run pre-commit install -t pre-commit $ pipenv run pre-commit install -t pre-push
Useful Commands
Run Python Module
$ python3 -m python_boilerplate
Run Python Script
Append your project’s root directory to PYTHONPATH
— In any environment you wish to run your Python application such as Docker, vagrant or your virtual environment i.e. in bin/activate, run the below command:
For macOS or Linux,
# Ensure `pwd` is the root directory of the project
$ PYTHONPATH=`pwd` pipenv run python3 python_boilerplate/demo/pandas_usage.py
$ PYTHONPATH=`pwd` pipenv run python3 python_boilerplate/demo/multithread_and_thread_pool_usage.py
# Run the main module
$ PYTHONPATH=`pwd` pipenv run python3 python_boilerplate/__main__.py
# Run a pytest script
$ pytest --log-cli-level=DEBUG --capture=no tests/common/test_debounce_throttle.py
# Run a pytest script with `-k` EXPRESSION
$ pytest --log-cli-level=DEBUG --capture=no tests/common/test_debounce_throttle.py -k 'test_debounce'
# For more details of pytest command
$ pipenv run pytest --help
For Windows Terminal,
# Ensure `$PWD.Path` is the root directory of the project
$ $env:PYTHONPATH=$PWD.Path; pipenv run python .\python_boilerplate\demo\pandas_usage.py
$ $env:PYTHONPATH=$PWD.Path; pipenv run python .\python_boilerplate\demo\multithread_and_thread_pool_usage.py
# Run the main module
$ $env:PYTHONPATH=$PWD.Path; pipenv run python .\python_boilerplate\__main__.py
Package with PyInstaller
⚠️ WANRING
Need to call
freeze_support()
immediately when startup inpython_boilerplate/__init__.py
from multiprocessing import freeze_support freeze_support()
Build artifact with macOS or Linux,
$ pipenv run pyinstaller --console \
--add-data "setup.cfg:." \
--add-data "python_boilerplate/resources/*:python_boilerplate/resources" \
--name pandas_usage \
--clean --noconfirm python_boilerplate/demo/pandas_usage.py
On Windows,
$ pipenv run pyinstaller --console `
--add-data "setup.cfg;." `
--add-data "python_boilerplate/resources/*;python_boilerplate/resources" `
--name multithread_and_thread_pool_usage `
--clean --noconfirm python_boilerplate/demo/multithread_and_thread_pool_usage.py
Run Unit Tests
Run with pytest, analyze code coverage, generate HTML code coverage reports, fail the test if coverage percentage is under 90%,
$ pipenv run pytest --cov --cov-report html --cov-fail-under=85 --capture=no --log-cli-level=INFO
Benchmark with pytest,
$ pipenv run pytest --capture=no --log-cli-level=ERROR -n 0 --benchmark-only
Conventional Changelog CLI
-
Install global dependencies (optional if installed):
$ npm install -g conventional-changelog-cli
-
This will not overwrite any previous changelogs. The above generates a changelog based on commits since the last semver tag that matches the pattern of “Feature”, “Fix”, “Performance Improvement” or “Breaking Changes”.
$ conventional-changelog -p angular -i CHANGELOG.md -s
-
If this is your first time using this tool and you want to generate all previous changelogs, you could do:
$ conventional-changelog -p angular -i CHANGELOG.md -s -r 0
Check Versions of Python Packages
$ pipenv run pip list --outdated
Output be like,
Package Version Latest Type
---------------- --------- ---------- -----
black 23.10.1 23.11.0 wheel
certifi 2023.7.22 2023.11.17 wheel
Faker 19.12.0 20.1.0 wheel
identify 2.5.31 2.5.32 wheel
idna 3.4 3.6 wheel
matplotlib 3.8.0 3.8.2 wheel
mypy 1.6.1 1.7.1 wheel
numpy 1.26.1 1.26.2 wheel
pandas 2.1.2 2.1.3 wheel
pip 23.1.2 23.3.1 wheel
platformdirs 3.11.0 4.0.0 wheel
pydantic 2.4.2 2.5.2 wheel
pydantic_core 2.10.1 2.14.5 wheel
pyinstaller 6.1.0 6.2.0 wheel
pyinstrument 4.6.0 4.6.1 wheel
pytest-html 4.0.2 4.1.1 wheel
pytest-xdist 3.3.1 3.5.0 wheel
setuptools 68.2.2 69.0.2 wheel
types-setuptools 68.2.0.0 69.0.0.0 wheel
urllib3 2.0.7 2.1.0 wheel
virtualenv 20.24.6 20.24.7 wheel
wheel 0.41.3 0.42.0 wheel
CI (Continuous Integration)
- GitHub Actions are for building projects and running tests.
Travis CI is for publishing Docker Hub images of SNAPSHOT and RELEASE.
FAQ
-
ExecutableNotFoundError: Executable
/bin/bash
not found when using pre-commitSolution: delete
.git/hooks/pre-commit.legacy
and then reinstall, also deleted.git/hooks/pre-commit
just to be sure before installing again. -
Installing all stub type packages,
$ mypy --install-types
Maintainers
Contributing
Feel free to dive in! Open an issue.
Contributors
This project exists thanks to all the people who contribute.
- Johnny Miller [@johnnymillergh]
- …
Sponsors
Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [Become a sponsor]
Credits
This package was created with Cookiecutter and the sourcery-ai/python-best-practices-cookiecutter project template.
Inspired by How to set up a perfect Python project.
License
Apache License © Johnny Miller
2021—Present