datatracker/ietf/submit
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
..
management chore: remove unused submit by email functionality (#7249) 2024-03-26 09:44:49 -05:00
migrations refactor: Purge old migrations and update ci to build on a clean schema (#5431) 2023-03-28 18:06:07 +09:00
parsers refactor: Drop submission validation via libmagic (#6500) 2023-10-23 10:00:04 -05:00
templatetags chore: remove unnecessary .gitignore files (#6262) 2023-08-31 17:12:01 -05:00
__init__.py Remove backtrace email code from submit __init__.py, after the submit revamp I think I am responsible for all remaining bugs and not esanchez@yaco.es, and the code sometimes annoyingly interferes (through a bug in admin) with loading the submit module in the shell 2013-11-19 16:14:27 +00:00
admin.py Allow external resources to be set/suggested during submission process. Fixes #3068. Commit ready for merge. 2021-04-19 20:21:19 +00:00
checkers.py fix: keep the ftp dir in sync when the datatracker creates artifacts (#8401) 2025-01-09 11:23:20 -06:00
factories.py Bring the factory-boy and Faker dependencies forward through some minor breaking changes. Stop using the deprecated fr_QC locale in Faker. Commit ready for merge. 2021-08-03 22:22:35 +00:00
forms.py fix: clarify replace wording (#8244) 2024-12-05 08:50:07 -06:00
mail.py chore: remove unused submit by email functionality (#7249) 2024-03-26 09:44:49 -05:00
models.py chore: merge main and fix conflicts 2023-08-30 16:35:20 -05:00
resources.py fix: dont serialize Submission.first_two_pages through /api/v1. (#4145) 2022-07-06 14:45:27 -05:00
tasks.py feat: re-run yang checks via celery (#7558) 2024-06-18 10:42:13 -05:00
test_submission.bad Improved the verification of submitted file extensions and mimetype. 2015-08-07 12:10:26 +00:00
test_submission.nonascii.txt feat: Process uploaded submissions asynchronously (#5580) 2023-05-09 15:21:46 -05:00
test_submission.pdf Added support of xml-source only drafts submissions. 2015-07-22 21:35:44 +00:00
test_submission.txt fix: Fix spurious author extraction errors (#4799) 2022-12-02 15:41:21 -06:00
test_submission.xml Added asciiFullname and asciiSurname attributes to the XML submission test template. 2019-09-26 14:41:48 +00:00
test_submission_invalid_yang.txt feat: Process uploaded submissions asynchronously (#5580) 2023-05-09 15:21:46 -05:00
test_submission_no_org_or_address.txt Accept 'None' when updating DocumentAuthor affiliation / country. Commit ready for merge. 2021-05-21 15:54:03 +00:00
test_submission_no_org_or_address.xml Accept 'None' when updating DocumentAuthor affiliation / country. Commit ready for merge. 2021-05-21 15:54:03 +00:00
test_submission_v2_country_only.xml Run v2v3 converter internally when v2 XML is submitted. Fixes #3305. Commit ready for merge. 2021-07-20 18:13:41 +00:00
tests.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
urls.py chore: remove unused submit by email functionality (#7249) 2024-03-26 09:44:49 -05:00
utils.py feat: write objects to blob storage (#8557) 2025-02-19 17:41:10 -06:00
views.py chore: clarify transaction-related comments (#8180) 2024-11-08 11:55:47 +00:00