Commit graph

88 commits

Author SHA1 Message Date
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
dependabot[bot] e0b1fd29ee
chore(deps): bump the /dev/deploy-to-container npm dependencies (#8440)
Bumps the npm group in /dev/deploy-to-container with 2 updates: [dockerode](https://github.com/apocas/dockerode) and [fs-extra](https://github.com/jprichardson/node-fs-extra).


Updates `dockerode` from 4.0.3 to 4.0.4
- [Release notes](https://github.com/apocas/dockerode/releases)
- [Commits](https://github.com/apocas/dockerode/compare/v4.0.3...v4.0.4)

Updates `fs-extra` from 11.2.0 to 11.3.0
- [Changelog](https://github.com/jprichardson/node-fs-extra/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jprichardson/node-fs-extra/compare/11.2.0...11.3.0)

---
updated-dependencies:
- dependency-name: dockerode
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
- dependency-name: fs-extra
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-02 18:04:24 -05:00
dependabot[bot] 135f759b30
chore(deps): bump dockerode in /dev/deploy-to-container in the npm group (#8393)
Bumps the npm group in /dev/deploy-to-container with 1 update: [dockerode](https://github.com/apocas/dockerode).


Updates `dockerode` from 4.0.2 to 4.0.3
- [Release notes](https://github.com/apocas/dockerode/releases)
- [Commits](https://github.com/apocas/dockerode/compare/v4.0.2...v4.0.3)

---
updated-dependencies:
- dependency-name: dockerode
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-01-13 15:02:45 -05:00
Robert Sparks e5c4a9f298
feat: additional filesystem monitoring (#8405)
* feat: additional filesystem monitoring

* chore: rename setting for tmp directory

* fix: restructure path to new endpoint

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-01-09 13:07:51 -06:00
dependabot[bot] 98b251487d
chore(deps): bump nanoid in /dev/deploy-to-container in the npm group (#8293)
Bumps the npm group in /dev/deploy-to-container with 1 update: [nanoid](https://github.com/ai/nanoid).


Updates `nanoid` from 5.0.8 to 5.0.9
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/5.0.8...5.0.9)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-13 19:15:44 -05:00
Robert Sparks 2ec7a71edf
chore: remove unused setting from various settings_local templates (#8311) 2024-12-09 10:34:53 -06:00
dependabot[bot] 7c025c9f2b
chore(deps): bump nanoid in /dev/deploy-to-container in the npm group (#8098)
Bumps the npm group in /dev/deploy-to-container with 1 update: [nanoid](https://github.com/ai/nanoid).


Updates `nanoid` from 5.0.7 to 5.0.8
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/5.0.7...5.0.8)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-20 12:59:39 -05:00
dependabot[bot] 51f96eb952
chore(deps): bump tar in /dev/deploy-to-container in the npm group (#7793)
Bumps the npm group in /dev/deploy-to-container with 1 update: [tar](https://github.com/isaacs/node-tar).


Updates `tar` from 7.4.0 to 7.4.3
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-tar/compare/v7.4.0...v7.4.3)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-10-15 16:01:15 -04:00
Jennifer Richards d8d52eedbf
feat: email ingestor api test endpoint (#7915)
* feat: email ingestor api test endpoint

* ci: add ingestion test token for sandbox

* chore: fix comments
2024-09-09 10:00:54 -05:00
dependabot[bot] b0e2e1877f
chore(deps): bump tar (#7586)
Bumps the npm group with 1 update in the /dev/deploy-to-container directory: [tar](https://github.com/isaacs/node-tar).


Updates `tar` from 7.1.0 to 7.4.0
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-tar/compare/v7.1.0...v7.4.0)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-26 17:19:49 -04:00
Nicolas Giard d344f5fd3c
ci: fix deploy-to-container script 2024-05-16 16:34:38 -04:00
dependabot[bot] 5fbeb8cd90
chore(deps): bump tar in /dev/deploy-to-container in the npm group (#7397)
Bumps the npm group in /dev/deploy-to-container with 1 update: [tar](https://github.com/isaacs/node-tar).


Updates `tar` from 7.0.1 to 7.1.0
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-tar/compare/v7.0.1...v7.1.0)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-05-06 10:04:47 -04:00
dependabot[bot] 000e6b7dca
chore(deps): bump tar in /dev/deploy-to-container in the npm group (#7323)
Bumps the npm group in /dev/deploy-to-container with 1 update: [tar](https://github.com/isaacs/node-tar).


Updates `tar` from 6.2.1 to 7.0.1
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-tar/compare/v6.2.1...v7.0.1)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-29 14:12:30 -04:00
Robert Sparks cedd58f950
feat: obviate ghostlinkd (#7336)
* wip: identify whats needed to obviate ghostlinkd

* fix: hardlink new charter files to ftp directory

* fix: hardlink new charter files to ftp directory (continued)

* chore: bring settings comment up to date

* chore: add archive and ftp dirs to setup of various environments

* fix: test charter submits write to ftp dir

* chore: remove debug

* fix: test charter approval writes to ftp dir

* fix: link review revisions into ftp dir

* fix: link to all archive and ftp on submission post

* chore: clean comments, move action to github issue

* fix: link idindex files to all archive and ftp

* chore: deflake

* chore: remove TODO comment

* fix: use settings

* chore: rename new setting
2024-04-19 16:18:52 -05:00
dependabot[bot] 280481b924
chore(deps): bump nanoid in /dev/deploy-to-container in the npm group (#7299)
Bumps the npm group in /dev/deploy-to-container with 1 update: [nanoid](https://github.com/ai/nanoid).


Updates `nanoid` from 5.0.6 to 5.0.7
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/5.0.6...5.0.7)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-10 17:05:32 -04:00
dependabot[bot] ca98af431a
chore(deps): bump the npm group (#7256)
Bumps the npm group in /dev/deploy-to-container with 1 update: [tar](https://github.com/isaacs/node-tar).


Updates `tar` from 6.2.0 to 6.2.1
- [Release notes](https://github.com/isaacs/node-tar/releases)
- [Changelog](https://github.com/isaacs/node-tar/blob/main/CHANGELOG.md)
- [Commits](https://github.com/isaacs/node-tar/compare/v6.2.0...v6.2.1)

---
updated-dependencies:
- dependency-name: tar
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-02 13:32:21 -04:00
Jennifer Richards 80cc6a3802
ci: Add SITE_URL to settings_local for sandbox (#7112) 2024-02-28 08:30:51 -06:00
dependabot[bot] 58209b62db
chore(deps): bump the npm group (#7100)
Bumps the npm group in /dev/deploy-to-container with 1 update: [nanoid](https://github.com/ai/nanoid).


Updates `nanoid` from 5.0.5 to 5.0.6
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/5.0.5...5.0.6)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-26 15:29:55 -05:00
dependabot[bot] 6f03c6da6d
chore(deps): bump the npm group (#7022)
Bumps the npm group in /dev/deploy-to-container with 1 update: [nanoid](https://github.com/ai/nanoid).


Updates `nanoid` from 5.0.4 to 5.0.5
- [Release notes](https://github.com/ai/nanoid/releases)
- [Changelog](https://github.com/ai/nanoid/blob/main/CHANGELOG.md)
- [Commits](https://github.com/ai/nanoid/compare/5.0.4...5.0.5)

---
updated-dependencies:
- dependency-name: nanoid
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: npm
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-12 15:12:48 -05:00
Nicolas Giard f071c8c365
ci: fix sandboxNoDbRefresh flag 2024-01-03 21:51:40 -05:00
Nicolas Giard b3c0b30838
ci: add disable daily DB refresh flag 2024-01-03 21:24:44 -05:00
dependabot[bot] 3e84424260
chore(deps): bump /dev/deploy-to-container npm dependencies (#6869)
Bumps the all group in /dev/deploy-to-container with 1 update: [dockerode](https://github.com/apocas/dockerode).


Updates `dockerode` from 4.0.0 to 4.0.2
- [Release notes](https://github.com/apocas/dockerode/releases)
- [Commits](https://github.com/apocas/dockerode/compare/v4.0.0...v4.0.2)

---
updated-dependencies:
- dependency-name: dockerode
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: all
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-03 18:14:20 -05:00
depfu[bot] e3309242ab
chore(deps): update all npm dependencies for dev/deploy-to-container (#6795)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
2024-01-03 17:46:48 -05:00
Robert Sparks ec90725f12
chore: remove rfc editor sync from sandbox deploy startup 2023-12-22 08:18:43 -06:00
Nicolas Giard f3f3f51ce2
ci: disable LETS_ENCRYPT for deploy-to-container CLI 2023-12-21 23:54:00 -05:00
Robert Sparks 79cac6e565
Merge branch 'main' into feat/rfc 2023-12-05 13:58:47 -06:00
Nicolas Giard ca60be1299
ci: fix deploy-to-container/cli.js 2023-12-05 14:57:10 -05:00
Robert Sparks 03e161bae9
chore: merge branch 'main' into feat/rfc 2023-12-05 09:12:11 -06:00
Nicolas Giard 0fa51e9016
ci: Update deploy-to-container/cli.js 2023-11-28 12:25:49 -05:00
Nicolas Giard 6dcb998a59
ci: update dev.ietf.org settings_local.py 2023-11-21 12:34:24 -06:00
Nicolas Giard 377db9d091
ci: update dev.ietf.org settings_local.py 2023-11-21 13:25:37 -05:00
depfu[bot] efcbe018a1
chore(deps): update all npm dependencies for dev/deploy-to-container 2023-11-20 06:14:28 +00:00
Robert Sparks 28c3e1bd5d
chore: merge branch 'main' into feat/rfc 2023-11-01 10:54:29 -05:00
depfu[bot] 7f7f4391fd
chore(deps): update all npm dependencies for dev/deploy-to-container (#6516)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
2023-10-23 15:10:17 -05:00
Robert Sparks ab931f0510
chore: merge main into feat/rfc 2023-10-12 15:11:02 -05:00
Robert Sparks 4f4f6b8bc1
ci: yet another typo correction 2023-09-26 22:03:03 -05:00
Robert Sparks 72ae962424
ci: correct typo 2023-09-26 21:31:59 -05:00
Robert Sparks 1c9494d43a
ci: rfc-index sync on deploy 2023-09-26 17:28:55 -05:00
depfu[bot] 1dc06f0a15
chore(deps): update all npm dependencies for dev/deploy-to-container (#6366)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
2023-09-26 12:19:36 -05:00
Robert Sparks f9c9644263
chore: merge main and fix conflicts 2023-08-30 16:35:20 -05:00
Nicolas Giard 5de4c0b45b
ci: fix sandbox start.sh memcached user 2023-08-08 16:08:11 -04:00
Nicolas Giard b327a27736
ci: update dev sandbox init script to start memcached 2023-08-08 11:44:05 -04:00
depfu[bot] d565c349ce
chore(deps): update all npm dependencies for dev/deploy-to-container (#5753)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
2023-06-08 13:48:30 -05:00
Jennifer Richards 7475aa01b6
Merge pull request #5634 from ietf-tools/main
chore: Merge main into feat/django4
2023-05-16 13:06:49 -03:00
NGPixel c777f6f67d
ci: fix sandbox refresh script 2023-05-15 13:45:05 -04:00
NGPixel ad992b96ec
ci: add refresh sandbox DB containers script 2023-05-15 13:42:23 -04:00
depfu[bot] 675de0ac73
chore(deps): update all npm dependencies for dev/deploy-to-container (#5617)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
2023-05-15 10:32:35 -05:00
Jennifer Richards 264ff60cd2
Merge pull request #5601 from jennifer-richards/django4
chore: Upgrade to Django 3.0
2023-05-11 11:04:36 -04:00
Jennifer Richards ed571ae50b
chore: Rename DB engine to drop the deprecated "_psycopg2" suffix 2023-05-10 12:40:37 -03:00
depfu[bot] c53d7836ed
chore(deps): update all npm dependencies for dev/deploy-to-container (#5587)
Co-authored-by: depfu[bot] <23717796+depfu[bot]@users.noreply.github.com>
2023-05-09 15:23:33 -05:00