datatracker/requirements.txt
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00

86 lines
3.3 KiB
Plaintext

# -*- conf-mode -*-
setuptools>=51.1.0 # Require this first, to prevent later errors
#
argon2-cffi>=21.3.0 # For the Argon2 password hasher option
beautifulsoup4>=4.11.1 # Only used in tests
bibtexparser>=1.2.0 # Only used in tests
bleach>=6
types-bleach>=6
boto3>=1.35,<1.36
boto3-stubs[s3]>=1.35,<1.36
botocore>=1.35,<1.36
celery>=5.2.6
coverage>=4.5.4,<5.0 # Coverage 5.x moves from a json database to SQLite. Moving to 5.x will require substantial rewrites in ietf.utils.test_runner and ietf.release.views
defusedxml>=0.7.1 # for TastyPie when using xml; not a declared dependency
Django>4.2,<5
django-analytical>=3.1.0
django-bootstrap5>=21.3
django-celery-beat>=2.3.0
django-celery-results>=2.5.1
django-csp>=3.7
django-cors-headers>=3.11.0
django-debug-toolbar>=3.2.4
django-markup>=1.5 # Limited use - need to reconcile against direct use of markdown
django-oidc-provider==0.8.2 # 0.8.3 changes logout flow and claim return
django-referrer-policy>=1.0
django-simple-history>=3.0.0
django-storages>=1.14.4
django-stubs>=4.2.7,<5 # The django-stubs version used determines the the mypy version indicated below
django-tastypie>=0.14.7,<0.15.0 # Version must be locked in sync with version of Django
django-vite>=2.0.2,<3
django-widget-tweaks>=1.4.12
djangorestframework>=3.15,<4
djlint>=1.0.0 # To auto-indent templates via "djlint --profile django --reformat"
docutils>=0.18.1 # Used only by dbtemplates for RestructuredText
drf-spectacular>=0.27
drf-standardized-errors[openapi] >= 0.14
types-docutils>=0.18.1
factory-boy>=3.3
github3.py>=3.2.0
gunicorn>=20.1.0
hashids>=1.3.1
html2text>=2020.1.16 # Used only to clean comment field of secr/sreq
html5lib>=1.1 # Only used in tests
importlib-metadata<8.5.0 # indirect req of Markdown/inflect; https://github.com/ietf-tools/datatracker/issues/7924
inflect>= 6.0.2
jsonfield>=3.1.0 # for SubmissionCheck. This is https://github.com/bradjasper/django-jsonfield/.
jsonschema[format]>=4.2.1
jwcrypto>=1.2 # for signed notifications - this is aspirational, and is not really used.
logging_tree>=1.9 # Used only by the showloggers management command
lxml>=5.3.0
markdown>=3.3.6
types-markdown>=3.3.6
mock>=4.0.3 # Used only by tests, of course
types-mock>=4.0.3
mypy~=1.7.0 # Version requirements determined by django-stubs.
oic>=1.3 # Used only by tests
Pillow>=9.1.0
psycopg2>=2.9.6
pyang>=2.5.3
pydyf>0.8.0,<0.10.0 # until weasyprint adjusts for 0.10.0 and later
pyflakes>=2.4.0
pyopenssl>=22.0.0 # Used by urllib3.contrib, which is used by PyQuery but not marked as a dependency
pyquery>=1.4.3
python-dateutil>=2.8.2
types-python-dateutil>=2.8.2
python-json-logger>=3.1.0
python-magic==0.4.18 # Versions beyond the yanked .19 and .20 introduce form failures
pymemcache>=4.0.0 # for django.core.cache.backends.memcached.PyMemcacheCache
python-mimeparse>=1.6 # from TastyPie
pytz==2022.2.1 # Pinned as changes need to be vetted for their effect on Meeting fields
types-pytz==2022.2.1 # match pytz version
requests>=2.31.0
types-requests>=2.27.1
requests-mock>=1.9.3
rfc2html>=2.0.3
scout-apm>=2.24.2
selenium>=4.0
tblib>=1.7.0 # So that the django test runner provides tracebacks
tlds>=2022042700 # Used to teach bleach about which TLDs currently exist
tqdm>=4.64.0
Unidecode>=1.3.4
urllib3>=1.26,<2
weasyprint>=59
xml2rfc[pdf]>=3.23.0
xym>=0.6,<1.0