datatracker/ietf
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
..
admin feat: dev mode admin + refactor api init (#7628) 2024-08-07 09:16:40 -05:00
api feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
bin chore: remove unused activate_this code branch (#8230) 2024-11-18 13:57:33 -06:00
community refactor: separate concerns / rename notify_events (#8328) 2024-12-12 16:48:44 -06:00
cookies chore: remove unnecessary .gitignore files (#6262) 2023-08-31 17:12:01 -05:00
database-notes chore: remove unnecessary .gitignore files (#6262) 2023-08-31 17:12:01 -05:00
dbtemplate chore: set gitattributes for normalizing line endings (#8245) 2024-11-20 16:48:00 -06:00
doc feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
group feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
help refactor: Move document state help to /doc (#7206) 2024-03-22 15:03:10 -05:00
idindex feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
iesg fix: guard against KeyError (#8485) 2025-01-30 10:21:15 -06:00
ietfauth feat: django-rest-framework + Person/Email API (#8256) 2024-11-27 14:54:28 -06:00
ipr chore: typing fixes for factory-boy 3.3.3 (#8501) 2025-02-05 14:31:35 -06:00
liaisons feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
mailinglists fix: mailman3 links for nonwg lists (#7507) 2024-06-06 13:11:54 -05:00
mailtrigger fix: rectify mixed types in gathering mailtrigger recipients (#7932) 2024-09-12 17:04:12 -05:00
meeting feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
message chore: add task tests; move message task to message app (#6964) 2024-01-24 08:53:42 -06:00
name feat: remove "AD is watching" state (#7960) 2024-09-24 12:13:12 -05:00
nomcom feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
person feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
redirects chore: remove unnecessary .gitignore files (#6262) 2023-08-31 17:12:01 -05:00
release feat: reflect breaking change in internal release description (#3826) 2022-04-14 15:04:09 -05:00
review Make review settings history usable. (#7205) 2024-03-20 19:04:37 -05:00
secr fix: lock session requests for new meetings (#8251) 2024-11-22 08:13:53 -06:00
static feat: CSS other-session-selected. Fixes #8525 (#8529) 2025-02-14 17:30:24 -06:00
stats fix: remove unreliable statistics (#8307) 2024-12-09 10:33:03 -06:00
status chore: remove dead code (#8207) 2024-11-13 09:51:43 -06:00
submit feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
sync fix: map "Editorial" -> StreamName (#8442) 2025-01-20 11:59:18 -04:00
templates fix: deal with "GMT" time zone in iCalendar files (#8506) 2025-02-06 12:35:07 -06:00
utils feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
.gitignore chore: update .gitignore 2023-06-05 18:10:50 -04:00
.yarnrc Preliminary work on converting bower management of web resources to yarn. New management command, replacing bower_install will be needed. 2019-07-25 20:28:18 +00:00
__init__.py chore: Bump version to 12 in a couple places (#6789) 2024-03-11 15:54:38 -05:00
celeryapp.py ci: fix celery scout env var names (#7373) 2024-05-13 21:41:36 -04:00
checks.py refactor: don't use filesystem for group aliases (#7556) 2024-06-18 08:28:09 -05:00
context_processors.py feat: Improve links to meeting tools from group pages (#5130) 2023-02-11 10:00:30 -06:00
feed_urls.py Changed urlpattern order to list the more specific of overlapping patterns first, and made the rfc feed per year pattern more permissive. 2017-12-04 16:32:58 +00:00
LICENSE
manage.py Moved warnings settings from manage.py and test_runner.py to one single place, settings.py. Decreased the verbosity level when invoking 'loaddata' from the test runner. 2016-12-15 13:25:20 +00:00
middleware.py feat: is_authenticated request logging + cleanup (#7893) 2024-09-03 17:24:26 -05:00
settings.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
settings_test.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
settings_testcrawl.py chore: improve ALLOWED_HOSTS in settings (ports are not useful there) (#4232) 2022-07-20 15:16:14 -05:00
urls.py feat: Site status message (#7659) 2024-08-07 13:36:21 -05:00
virtualenv-manage.py More Py2/3 adaptations fetched from production. 2020-02-23 16:27:30 +00:00
wsgi.py feat: improve / clean up logging (#7591) 2024-06-26 14:53:05 -05:00