Commit graph

42 commits

Author SHA1 Message Date
Robert Sparks 997239a2ea
feat: write objects to blob storage (#8557)
* feat: basic blobstore infrastructure for dev

* refactor: (broken) attempt to put minio console behind nginx

* feat: initialize blobstore with boto3

* fix: abandon attempt to proxy minio. Use docker compose instead.

* feat: beginning of blob writes

* feat: storage utilities

* feat: test buckets

* chore: black

* chore: remove unused import

* chore: avoid f string when not needed

* fix: inform all settings files about blobstores

* fix: declare types for some settings

* ci: point to new target base

* ci: adjust test workflow

* fix: give the tests debug environment a blobstore

* fix: "better" name declarations

* ci: use devblobstore container

* chore: identify places to write to blobstorage

* chore: remove unreachable code

* feat: store materials

* feat: store statements

* feat: store status changes

* feat: store liaison attachments

* feat: store agendas provided with Interim session requests

* chore: capture TODOs

* feat: store polls and chatlogs

* chore: remove unneeded TODO

* feat: store drafts on submit and post

* fix: handle storage during doc expiration and resurrection

* fix: mirror an unlink

* chore: add/refine TODOs

* feat: store slide submissions

* fix: structure slide test correctly

* fix: correct sense of existence check

* feat: store some indexes

* feat: BlobShadowFileSystemStorage

* feat: shadow floorplans / host logos to the blob

* chore: remove unused import

* feat: strip path from blob shadow names

* feat: shadow photos / thumbs

* refactor: combine photo and photothumb blob kinds

The photos / thumbs were already dropped in the same
directory, so let's not add a distinction at this point.

* style: whitespace

* refactor: use kwargs consistently

* chore: migrations

* refactor: better deconstruct(); rebuild migrations

* fix: use new class in mack patch

* chore: add TODO

* feat: store group index documents

* chore: identify more TODO

* feat: store reviews

* fix: repair merge

* chore: remove unnecessary TODO

* feat: StoredObject metadata

* fix: deburr some debugging code

* fix: only set the deleted timestamp once

* chore: correct typo

* fix: get_or_create vs get and test

* fix: avoid the questionable is_seekable helper

* chore: capture future design consideration

* chore: blob store cfg for k8s

* chore: black

* chore: copyright

* ci: bucket name prefix option + run Black

Adds/uses DATATRACKER_BLOB_STORE_BUCKET_PREFIX option. Other changes
are just Black styling.

* ci: fix typo in bucket name expression

* chore: parameters in app-configure-blobstore

Allows use with other blob stores.

* ci: remove verify=False option

* fix: don't return value from __init__

* feat: option to log timing of S3Storage calls

* chore: units

* fix: deleted->null when storing a file

* style: Black

* feat: log as JSON; refactor to share code; handle exceptions

* ci: add ietf_log_blob_timing option for k8s

* test: --no-manage-blobstore option for running tests

* test: use blob store settings from env, if set

* test: actually set a couple more storage opts

* feat: offswitch (#8541)

* feat: offswitch

* fix: apply ENABLE_BLOBSTORAGE to BlobShadowFileSystemStorage behavior

* chore: log timing of blob reads

* chore: import Config from botocore.config

* chore(deps): import boto3-stubs / botocore

botocore is implicitly imported, but make it explicit
since we refer to it directly

* chore: drop type annotation that mypy loudly ignores

* refactor: add storage methods via mixin

Shares code between Document and DocHistory without
putting it in the base DocumentInfo class, which
lacks the name field. Also makes mypy happy.

* feat: add timeout / retry limit to boto client

* ci: let k8s config the timeouts via env

* chore: repair merge resolution typo

* chore: tweak settings imports

* chore: simplify k8s/settings_local.py imports

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-02-19 17:41:10 -06:00
Robert Sparks e5c4a9f298
feat: additional filesystem monitoring (#8405)
* feat: additional filesystem monitoring

* chore: rename setting for tmp directory

* fix: restructure path to new endpoint

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2025-01-09 13:07:51 -06:00
Robert Sparks 2ec7a71edf
chore: remove unused setting from various settings_local templates (#8311) 2024-12-09 10:34:53 -06:00
Nicolas Giard bdf753e052
chore: set gitattributes for normalizing line endings (#8245)
* chore: add proper .gitattributes

* chore: normalize file endings

* chore: fix .gitattributes

* chore: normalize file endings (2)
2024-11-20 16:48:00 -06:00
Lars Eggert 35074660dc
chore: Prevent test suite artifact creation in work directory (#6438)
* chore: Prevent test suite artifact creation in work directory

Also remove a few other stale test assets while I'm here.

* Try and fix CI

* Change IDSUBMIT_REPOSITORY_PATH

* Make the dir

* test: clean up media/nomcom directories

* test: de-dup settings_temp_path_overrides

* chore: remove debug

* refactor: avoid premature import of test_utils

* refactor: drop useless lambda wrapper

---------

Co-authored-by: Jennifer Richards <jennifer@staff.ietf.org>
2024-09-18 14:08:01 -05:00
Robert Sparks cedd58f950
feat: obviate ghostlinkd (#7336)
* wip: identify whats needed to obviate ghostlinkd

* fix: hardlink new charter files to ftp directory

* fix: hardlink new charter files to ftp directory (continued)

* chore: bring settings comment up to date

* chore: add archive and ftp dirs to setup of various environments

* fix: test charter submits write to ftp dir

* chore: remove debug

* fix: test charter approval writes to ftp dir

* fix: link review revisions into ftp dir

* fix: link to all archive and ftp on submission post

* chore: clean comments, move action to github issue

* fix: link idindex files to all archive and ftp

* chore: deflake

* chore: remove TODO comment

* fix: use settings

* chore: rename new setting
2024-04-19 16:18:52 -05:00
Jennifer Richards 85927c083d
chore: Move IDSUBMIT_REPOSITORY_PATH for dev (#6218) 2023-08-26 16:14:02 -05:00
Robert Sparks 73137364cc
chore: rename test settings and shorten runserver instructions (#6131) 2023-08-11 11:21:38 -05:00
Robert Sparks dbe1749438
feat: IAB statements (#5940)
* feat: support iab and iesg statements. Import iab statements. (#5895)

* feat: infrastructure for statements doctype

* chore: basic test framework

* feat: basic statement document view

* feat: show replaced statements

* chore: black

* fix: state help for statements

* fix: cleanout non-relevant email expansions

* feat: import iab statements, provide group statements tab

* fix: guard against running import twice

* feat: build redirect csv for iab statements

* fix: set document state on import

* feat: show published date on main doc view

* feat: handle pdf statements

* feat: create new and update statements

* chore: copyright block updates

* chore: remove flakes

* chore: black

* feat: add edit/new buttons for the secretariat

* fix: address PR #5895 review comments

* fix: pin pydantic until inflect catches up (#5901) (#5902)

* chore: re-un-pin pydantic
2023-07-23 11:00:24 -07:00
Nicolas Giard added8603a
chore(dev): increase dev nginx timeout to 1d 2023-06-19 16:58:22 -04:00
Nicolas Giard f3831de3ef
chore(dev): fix internal static proxy + add /pgadmin to reverse-proxy (#5809) 2023-06-13 10:56:11 -05:00
Nicolas Giard fd2303044c
chore: rewrite host header in dev nginx proxy config (#5772) 2023-06-06 16:11:32 -05:00
Nicolas Giard cfaf966492
fix: add internal static.ietf.org var for pdfized render (#5765) 2023-06-05 20:18:35 -05:00
Nicolas Giard 03a192b54c
chore: add nginx reverse proxy to dev environment (#5745)
* chore: add nginx reverse proxy to dev environment

* chore: replace settings_local in docker and backup old
2023-06-05 16:47:06 -05:00
Jennifer Richards 1922a6066e
Merge pull request #5700 from ietf-tools/main
chore: main -> django4
2023-05-31 10:59:09 -03:00
Nicolas Giard 0a11892414
chore: use variable for static.ietf.org references (#5692) 2023-05-25 17:23:18 -05:00
Jennifer Richards ed571ae50b
chore: Rename DB engine to drop the deprecated "_psycopg2" suffix 2023-05-10 12:40:37 -03:00
Robert Sparks 400a830661
chore: add bibxml-ids dir to container build. (#5590) 2023-05-08 15:16:31 -05:00
Nicolas Giard 82d0d42b28
chore: add pgadmin to dev environment 2023-04-25 18:02:34 -04:00
Nicolas Giard ee2ed27e9f
chore: change db name to datatracker 2023-04-18 19:19:58 -04:00
Nicolas Giard 9c40b9300a
chore: update remaining config files to point to new db image 2023-04-18 17:16:13 -04:00
Jim Fenton 0f1a6c960f
chore: Update test docs to postgresql; remove extraneous sqlite stuff (#5400)
* chore: Update test docs to postgresql; remove extraneous sqlite stuff

* Reverted addition of settings_postgrestest.py to .gitignore
2023-03-29 10:45:03 +09:00
Robert Sparks 736382a13e
fix: default settings_local needs a pyflakes ignore (#5381) 2023-03-20 17:19:49 -05:00
Robert Sparks f010766654
Merge remote-tracking branch 'ietf-tools/main' into feat/postgres 2023-02-27 12:55:47 -06:00
Lars Eggert 1fb5bd7429
fix: Use the location app-rsync-extras.sh uses for the I-D archive (#5161) 2023-02-16 09:04:20 -06:00
Nicolas Giard a60d52253e
chore: use prebuilt pg db image in dev (#4890) 2022-12-15 14:00:51 -06:00
Robert Sparks 912c7b5ee3
fix: simplify migrations when building postgres db image (#4889)
* chore: remove the effectively unused DATABASE_TEST_OPTIONS setting

* fix: simplify default settings_local so that db-include-fix.py is not needed in CI

* fix: simplify migrations in db-pg-migrate.sh
2022-12-13 17:37:24 -05:00
Robert Sparks 15569771ff
Merge remote-tracking branch 'upstream/main' into feat/postgres 2022-12-12 09:54:49 -06:00
Robert Sparks e257ddbfea
fix: restore sql_debug functionality from the container (#4835) 2022-12-06 08:48:23 -06:00
Robert Sparks 27fccc6ba0
feat: move to postgresql (#4744)
* feat: move to postgresql

* fix: repair fractional replace statement

* fix: use pathlib to manipulate settings_local

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>

* fix: do two string replacements, not one followed by another that throws away the first.

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>

* fix: use pathlib again to manipulate settings_local

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>

* fix: properly use assert (1/2)

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>

* fix: properly use assert (2/2)

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>

Co-authored-by: Jennifer Richards <jennifer@painless-security.com>
2022-11-11 11:01:01 +00:00
Nicolas Giard dde5ed4650
chore: create separate run config for vite dev (#4221) 2022-07-18 08:40:32 -05:00
Nicolas Giard aa9490faf6
feat(ui): new dynamic agenda view (#4086)
* feat: agenda page in vue (wip)

* feat: scroll to agenda day

* fix: vue 3 composition api + eslint settings

* fix: agenda day scroll match indicator

* fix: convert vite deps to yarn

* fix: missing lodash + legacy build step

* fix: agenda - move calendar into drawer

* fix: improve agenda filter UI

* fix: download ics + move agenda into own component

* feat: use fullcalendar for agenda calendar view (wip)

* feat: add events to agenda calendar

* feat: agenda filter UI improvements

* feat: agenda add to calendar dropdown

* feat: agenda calendar filter + timezone + event coloring

* feat: agenda calendar color improvements

* chore: exclude dist-neue from git

* feat: agenda calendar event modal

* fix: rebuild yarn deps

* chore: add run migration task to vscode

* fix: agenda buttons display flag

* feat: agenda event modal component

* feat: show calendar event quick info on hover

* fix: clear calendar quick info on timezone change

* feat: agenda list view improvements

* feat: agenda list row coloring

* feat: agenda list note

* feat: agenda list icons for office hours + hackathon

* fix: agenda top links

* refactor: use pinia as store for agenda components

* feat: agenda jump to now

* fix: agenda mobile improvements

* feat: agenda search

* feat: agenda search improvements

* feat: agenda event recordings buttons for post-meeting

* fix: agenda switch to meeting timezone on load

* feat: agenda pre & live session buttons

* fix: remove agenda utc + personalize links in top menu

* feat: add pre-vue loading state on page load

* feat: filter from agenda picker mode

* fix: agenda UI improvements

* fix: django-vite non-dev mode

* chore: update yarn dependencies for vue + vite

* feat: agenda settings panel + UI improvements

* feat: agenda settings colors + import/export feature

* feat: agenda color assignments + responsive UI improvements

* feat: agenda realtime red line + debug datetime offset

* feat: agenda add aria labels for settings

* feat: add new agenda path + pages/menu

* fix: bring base/menu.html up to main

* fix: agenda various fixes

* test: add new agenda item to meetings menu for item count

* chore: restore devcontainer extensions list

* fix: agenda UI improvements + montserrat default font

* feat: agenda bolder text + hide event icons options

* feat: agenda warning badge

* fix: agenda various UI improvements + intersectionObserver fix

* feat: agenda floorplan page + various UI improvements

* feat: agenda floor plan pin

* feat: view floor plan room from agenda

* feat: agenda floor plan mobile optimization

* feat: adjust calendar options + default calendar view in settings

* feat: agenda persist picked events + change base font only on new agenda page

* feat: agenda mobile view optimizations

* fix: add .vite to cached volumes

* fix: mobile view for filters, calendar, settings panels

* test: upgrade cypress existing tests to work on bs5 + update dependencies

* fix: use named url patterns to avoid hardcoded URLs. Add rudimentary test coverage for the neue views.

Co-authored-by: Robert Sparks <rjsparks@nostrum.com>
2022-07-13 16:20:23 -05:00
Nicolas Giard e02ce05612
chore: make /assets a named volume for dev container (#4018)
* chore: make /assets a static volume for dev container

* chore: fix create-dirs script to point to /assets
2022-05-31 16:43:29 -05:00
Nicolas Giard 9a4505dc74
chore: remove svn/trac infrastructure (#3649) (#3739)
* chore: Remove svn/trac related infrastructure

* chore: Remove more svn/trac infrastructure

* fix: remove commented out Trac requirement
2022-03-24 12:24:35 -05:00
Jennifer Richards 6c260a5b7e Merge 7.45.1.dev0 into Bootstrap 5 update branch. Made a first pass at reconciling differences.
- Legacy-Id: 19945
2022-02-17 20:09:49 +00:00
Lars Eggert 67ebadeadd A bunch more cosmetic fixes.
- Legacy-Id: 19913
2022-02-09 10:11:03 +00:00
Lars Eggert bbf088e18b Hopefully the final check-ins.
- Legacy-Id: 19909
2022-02-08 17:29:15 +00:00
Robert Sparks 877d8800ce Add de-gfm to the docker setup. Fixes #3494. Commit ready for merge.
- Legacy-Id: 19863
2022-01-19 19:44:28 +00:00
Robert Sparks 2a0d54afcd Merged in [19811] from lars@eggert.org:
When using Docker, the runserver isn't being accessed over loopback, so we need
to initialize INTERNAL_IPS based on the current interface configuration.
 - Legacy-Id: 19814
Note: SVN reference [19811] has been migrated to Git commit 46ca676736
2022-01-06 20:54:53 +00:00
Lars Eggert 46ca676736 When using Docker, the runserver isn't being accessed over loopback, so we need
to initialize INTERNAL_IPS based on the current interface configuration.

Commit ready for merge.
 - Legacy-Id: 19811
2022-01-06 11:20:49 +00:00
Robert Sparks 13d4e636bc Docker related changes: Mark cleandb executable. Clean settings_local and settings_local_sqlitetest of flakes and types issues.
- Legacy-Id: 19637
2021-11-11 16:35:30 +00:00
nick d52ad4fbc2 misc: import docker improvements from 7.39.1.dev2
- Legacy-Id: 19629
2021-11-10 21:51:55 +00:00