ietfdb (6.103.0) ietf; urgency=medium **Draft submission tool changes** This release contains two changes to the submission tool: During draft submission, the email verification is skipped for logged-in users, and author email addresses are now required for all authors, in order to avoid author notification problems during IETF and RFC-Editor processing. Commit log details: * Changed draft submission to not require an email round-trip when the submitter is logged in. Added corresponding test cases. * Added code to detect conflicting information for draft name, revision, and title when both XML and text versions of a document are submitted. * Changed the draft submission form and views so that an email address is required for each author in order to complete self-service draft submission. It is possible to add these as metadata updates, but this will lead to submission through the secretariat (and the email addresses will still be required). Also updated related templates to make the handling of form section headings more consistent. * Added some caching to /release/ pages. With now more than 350 releases, the page response times had increased to typically 4s. Caching in view and template brought this down to ~160ms. -- Henrik Levkowetz 26 Sep 2019 18:42:18 +0000 ietfdb (6.102.0) ietf; urgency=medium **Links to RFC text with in-lined errata. HTML for v3 XML draft submissions** * Added conversion of version 3 XML draft submissions to v3 HTML, in addition to text. Added a link to HTML versions (when present) to the document status page, and renamed the link to the htmlized version to 'htmlized'. Converted the test-file for XML submission tests to v3. Added test code to check that HTML is generated for v3 XML submissions. * Added class 'bg-danger' to submission error messages on the submission upload result page. The big 'Meta-Data errors found' pane with red background is so noticeable that class 'text-danger' (which gives bold text, but no change of background) is insufficient to stand out clearly. Adding 'bg-danger' gives field error messages red background, too. * Added a link from the datatracker RFC pages to the new RFC renderings with inline errata on the RFC-Editor web page (where available). * Updated rfc-editor sync code to recognise the new Tooling Issue queue state. * Updated ietf.sync.rfceditor.parse_index() to use the new rfc-index schema (changed placement of the page-count element). * Added code to catch a form error regularly caused by form spam and return a failure instead of triggering a server 500. -- Henrik Levkowetz 20 Sep 2019 16:18:30 +0000 ietfdb (6.101.0) ietf; urgency=medium **IANA expert review tracking, new constraints labelling** This release adds IANA expert review tracking for drafts, changes the session request constraints labels, improves the handling of session slide proposal uploads, and adds a number of smaller enhancements and issue fixes. From the commit log: * Merged in [16656] from rjsparks@nostrum.com: Added IANA expert review tracking. * Merged in [16650], [16655] from rjsparks@nostrum.com: Removed the bootstrap project todo page. #2778 points to the old content so we can make new tickets for anything that is still relevant. * Merged in [16649] from rjsparks@nostrum.com: Guard against attempts to use the review request view with a ReviewRequest id that does not match the document for the ReviewRequest. This would have exposed #2776 much earlier. * Merged in [16647] from rjsparks@nostrum.com: Corrected a view that was looking at review assignment ids when it should have been looking at request review ids. Fixes #2776. * Added a 'domain' variable to context supplied to the agenda_package.txt template. Fixes issue #2793. * Merged in [16646] from rjsparks@nostrum.com: Don't manually linewrap when looking at feedback - let the pasted styling do that. Fixes #2761. * Merged in [16630] from rjsparks@nostrum.com: Rework how we are labeling constraints to provide a quick-fix for session requests at IETF106. Turned off several javascript functions that added/enforced old (possibly no longer correct) semantics for the session request forms. Fixes #2754. * Merged in [16629] from rjsparks@nostrum.com: Removing unused template. I think it was an artifact of the Django 1.2 to 1.6 conversion from 6 years ago. * Merged in [16615] from peter@akayla.com: Spelling correction in a comment. No open issue. * Merged in [16614] from peter@akayla.com: Improved so that new draft diffs (from the working group drafts page) work correctly for both -00 drafts and later revisions. Fixes issue #2697. * Merged in [16604] from rjsparks@nostrum.com: Look at all scheduled sessions for a group when deciding to display a cancelled banner. Fixes #2764. * The logic for displaying 'Email requested to be sent for earlier discuss' on IESG positions was wrong; it would count later discusses and discusses from other ADs as an 'earlier discuss'. Moved any_email_sent() from ballot to position, and refined the criteria. * Made sure Person.active_drafts() and .expired_drafts() don't return duplicate entries. Fixes issue #2784. * Changed requirements to permit pyang 2.0 and greater. * Added a migration to fix some m2m table columns incorrectly left as varchar(8) when we changed the size of name slugs to varchar(32). * Merged in [16601] from rjsparks@nostrum.com: A management command to correct the state of several slide presentations uploaded during IETF105, and perform some missed powerpoint to pdf conversions. * Merged in [16600] from rjsparks@nostrum.com: Fixed a subtle bug in the code that converts slides to PDF that was writing back into settings. * Merged in [16599] from rjsparks@nostrum.com: Correct the uploaded_filename for slides that go through the approve-proposed-slides workflow. * Merged in [16598] from rjsparks@nostrum.com: Add the ppt* to pdf conversion to the approve-suggested-slides workflow. Fixes #2762. * Added textual information about patent application formats according to WIPO's Handbook on Industrial Property Information and Documentation, as input to future extension of the regex set. * Added some new formats to the regex validator for patent numbers and patent application numbers. This is still not comprehensive. * Adjusted migrations for Python2/3 compatibility. * Updated database options settings to match our production database and dump files. * Fixed an issue where the text file charset information for an upload would be overwritten on upload of multiple files for a draft. Fixes issue #2768. * Added 30 minutes caching to agenda.ics -- Henrik Levkowetz 08 Sep 2019 15:28:29 +0000 ietfdb (6.100.0) ietf; urgency=medium **IETF 105 code sprint** This release contains datatracker bug fixes and enhancements from the IETF-105 Code Sprint. This sprint saw several participants diving deeper into the datatracker than before, accomplishing substantial tasks. Thanks to everyone who contributed! Notable improvements: The document list pages produced by searches, group document pages, and similar has been speeded up by a factor of more than 2 by doing detailed timing measurements and one by one addressing the biggest time sinks. A number of places where links to session slides were versioned have been changed so as to always point to the latest version. A calendar display component has been added, making it straightforward to display for instance calendar pages for upcoming meetings. Ability to handle markdown uploads and on-the-fly conversion for agendas and minutes has been added, as well as the ability to provide WG closing notes, sent out in announcement and saved for posterity on the group history page. The recently introduced Slide proposal feature has been made more solid, as has the Python 2 / Python 3 compatibility. Additional information, excerpted from the commit log: * Merged in [16593] from rjsparks@nostrum.com: Added sorting of the documents in the agenda, minutes, bluesheets section of the session_details view. Fixes #2759. * Merged in [16592] from rjsparks@nostrum.com: Changed how the session details page links to documents to not link to a specific version. Fixes #2758. See also #2756. * Merged in [16590] from rjsparks@nostrum.com: Example of using webtest. The full testcase was not converted because the add_rule form is modified in-flight by javascript when a control value is changed. * Merged in [16583] from rjsparks@nostrum.com: Restored the ability to remove documents while managing a personal list. Required updating code to match the recent change to Document's primary key. Fixes #2757. * Merged in [16569] from rjsparks@nostrum.com: Added quotes around document name in the mailarchive search link on a document's main page. Fixes #2659. * Merged in [16589] from rjsparks@nostrum.com: Added a calendar to the upcoming meetings page. Fixes #2719. * Prevent login name case issues by generating apache permissions file entries for both uppercase and lowercase logins. * Added a Py3 version of requirements.txt, excluding Trac which is still only available under Py2.7 * Did some preliminary work on converting bower management of web resources to yarn. New management command, replacing bower_install will be needed. * Merged in [16562] from rcross@amsl.com: Added encoding='utf-8' parameter to io.open operations in tests that starting failing since adding unicode_literals. * Merged in [16561] from rjsparks@nostrum.com: Added a prompt for login for approving slide suggestions instead of just presenting an error. Fixes #2750. * Merged in [16559] from fenton@bluepopcorn.net: Added Closing Notes to history and about pages for groups. Fixes issue #2725. * Merged in [16558] from rjsparks@nostrum.com: Made removed IPR disclosures visually distinctive on search results. Partially addresses #2687. * Merged in [16557] from pusateri@bangj.com: Added the ability to upload agenda and minutes in markdown format. Partially addresses issue #1926. We're still displaying markdown as text. Fixes #2704. * Fixed a number of additional Py2/3 compatibility issues. * Added LookupError to the exceptions caught during text upload decoding, leading to upload form validation error. * Merged in [16554] from rjsparks@nostrum.com: Rework how SlideSuggestion objects name their files. Improve the UI when there are multiple pending suggestions for updating the same presentation. * Moved djangobwr into the local repository, due to consistent difficulties pulling it from github as an svn:external * Enabled display of group pages also for groups of type adhoc and ise. * Updated the PLAN to adjust priority on modeling BCPs correctly * Added another check before building submitter email for mailtrigger entries, to ensure valid email addresses. -- Henrik Levkowetz 05 Aug 2019 16:43:03 +0000 ietfdb (6.99.1) ietf; urgency=medium This bugfix release contains a number of fixes for issues with the big Python 3 conversion release, many of which has already been patched in production. Additionally, there are some fixes related to new slide upload functionality and the test suite. From the commit log: * Fixed some issues with the Py2/3 changes to the populate_yang_model_dirs management command. * Added a check for submitter found before trying to get submitter name and email when generating email notification address lists. * Shortened generated document titles a tad, to avoid line wrapping in the IESG agenda package (which sometimes breaks the relevant tests). * Merged in [16515] from rjsparks@nostrum.com: Save files correctly when updates to an existing set of slides are proposed. Fixes #2748. * Added information about versions (Datatracker, Python, and Django) at the beginning of test suite runs, to be able to clearly see which Python and Django versions were used during a test run. * Replaced six.ensure_str with django's force_str, in order to be able to use earlier versions of six which more frequently are available on the development and production target platforms. * Changed PDF page-counting code to read PDF files as binary. * Replaced the email-sending logger.error() on bad Submission.submitter email address with syslog logging. * Added an email gather handler for submission_submitter that tries an Alias name lookuup if the submitter doesn't look as if it contains an email address. This deals better with submission records that only has a name in the submitter field. * Changed import path for settings_local.py to work with absolute_import under Py2, and also under Py3. * Changed agenda file reading to read files as binary when looking for draft names listed in the agenda, in order to avoid issues with older uploads using latin-1 or other encodings where the lower 127 character match ASCII. -- Henrik Levkowetz 18 Jul 2019 20:34:35 +0000 ietfdb (6.99.0) ietf; urgency=medium **Python 3 compatible code** This is a major release which contains conversion of the datatracker Python code to run under Python-3.7 where it previously was specific to Python-2.7. The code was first converted to Python-3.7, and then additional work was done to make it compatible with both Python 2 and 3. The changes updated 458 files, and touched close to 4000 lines. Making the code compatible with both Python 2 and 3 lets us continue to use the converted codebase until we're ready to move the primary webserver's mod_wsgi completely to Python 3. Otherwise, we would have to merge all new bugfixes and sprint contributions to both trunk (Python 2.7) and the new Python 3 branch, with doubled merge work until we were ready to move the server to Python 3. (Testing, though, will still need to be done under both 2 and 3 to ensure we don't loose compatibility.) -- Henrik Levkowetz 16 Jul 2019 22:21:56 +0000 ietfdb (6.98.4) ietf; urgency=medium This is a small bugfix release, to clear the slate before merging in the Python 2/3 conversion. From the commit log: * Added a validation step for SearchablePersonField, to avoid later server 500 errors on bad input. * Merged in [16404] from rjsparks@nostrum.com: Only look for ietf/datatracker-env images when extracting the most recent build to tag as latest. * Merged in [16359] from rcross@amsl.com: Fix registration import, use user.person if it exists. * Merged in [16096] from rcross@amsl.com: Update admin permissions. Grant secretariat change permissions on dbtemplate so they can update proceedings pages. * Turned off html autoescape in IPR email templates. This fixes inappropriate html escapes that occurred in various IPR-related emails. * Added missing code to skip coverage measurement for skippable tests. -- Henrik Levkowetz 16 Jul 2019 14:14:12 +0000 ietfdb (6.98.3) ietf; urgency=medium * Increased the length of the list Subscribed email field from 64 to 128, updated the import_mailman_listinfo management command, and added a migration for the model change. -- Henrik Levkowetz 04 Jul 2019 16:07:14 +0000 ietfdb (6.98.2) ietf; urgency=medium This is a minor release that fixes some bugs and tweaks some settings. From the commit log: * Changed some tests to match production group type changes for some special groups. * Made the bin/daily cron script quiteter. * Various changes to make yang-related management commands quieter * Added a fix for an issue in get_meeting_registration_data() which could happen if we tried to create a Person record for with a user of an existing Person. * Changed the permitted length of patent_title from 127 to 255. * Added additional guards against duplicate m2m entries. * Added a utility to check copyright statements in specified files. * Updated some functions and views in secr/srec/ to use GroupFeatures instead of hardcoded lists of group types. * Made session requests also work for ad-hoc groups, which may not have a parent area. -- Henrik Levkowetz 03 Jul 2019 20:27:54 +0000 ietfdb (6.98.1) ietf; urgency=medium This is a bugfix release that cleans up some remaining issues from the Document/DocAlias primary key refactoring, and also fixes some other unrelated issues. From the commit log: * Fixed bad logic for the display of previous related IPR information in the IPR details template. * Tweaked some test failure messages for better debugging. * Merged in [16287] from rjsparks@nostrum.com: Send mail to a list when a new IETF WG -00 is submitted. Fixes #2730. * Fixed some name versus pk issues lingering after the Document/DocAlias primary key refactoring. Fixes issue #2729. * Added the option to order API resource lists by primary key, for stable ordering when paging through a list. * Added generation of Meta.ordering for API ModelResources in the * mkresources management command. * The Document and DocAlias primary key refactoring caused the Tastypie API under /api/v1/ to change -- reverting that to use the old form, with name ask API key. * Added a better error message for too long draft names during submission. * Adjusted the test_runner to generate migration coverage data also when permitting mixed migrations. * Added a guard against duplicate insertions in SearchRule.name_contains_index * Fixed a docalias problem with generate-draft-aliases * Merged in [16248] from rjsparks@nostrum.com: Removed the IETF chair as a recipient of new IPR disclosures if there is no group or active document to send them to. Fixes #2727. * Added a management command to update the index table used by some community list search rules. Running this fixes the missing listings of drafts related to a group on group document pages and in searches. * Improved the support for the mixed-release switch in the bin/mkrelease script. -- Henrik Levkowetz 20 Jun 2019 13:16:54 +0000 ietfdb (6.98.0) ietf; urgency=medium **Refactoring of the primary index type for Document and DocAlias objects** This release contains a series of 216 migration steps that changes the Document and DocAlias primary keys from character strings to integers, and makes corresponding code changes. This was prompted by database limitations discovered when trying to make DocAlias use a m2m document field; with 255 long strings as primary keys for Document and DocAlias this violated the MySQL database limitations. Changing the primary keys to integers should also improve efficiency. Additionally, in order to make it possible to have DocAlias objects referring to multiple documents, in order to reflect STD and BCP names that refer to multiple RFCs, DocAlias has been refactored to use many-to-many tables instead of foreign keys do toduments. Due to the data migrations which create the new integer primary keys and adds corresponding integer foreign keys matching the previous string foreign keys in all tables having foreign keys to Document and DocAlias, some of these migrations take a long time. The total set of migrations are expected to have a runtime on the order of 2 hours. This release is scheduled to be deployed on 16 Jun 2019 at 16:00 UTC, with an expected datatracker downtime, for safety backup and application of the migrations to run, of about 3 hours. -- Henrik Levkowetz 16 Jun 2019 14:16:12 +0000 ietfdb (6.97.1) ietf; urgency=medium This is a bugfix release which clears the way for the upcoming major release. From the commit log: * Added validation of draft names extracted from XML submission files. * Added static resources for favicon.ico and robots.txt at the top of the ietf static tree. * Updated references to RFC 5226, to RFC 8126, which obsoletes 5226. This fixes issue #2726. * Fixed an issue where the 'Request closing group' button incorrectly was showing for group chairs. Fixes issue #2724. * Moved some explicit URLs from code and templates to settings (related to the production server www6/ directory cleanup). * Refined the handling of draft xml version 3 versus version 2. This fixes a problem where v3 documents were rendered with no section titles in the Table of Contents. -- Henrik Levkowetz 14 Jun 2019 13:30:43 +0000 ietfdb (6.97.0) ietf; urgency=medium **Additional meeting-related links** This is a small feature release which provides a new email notification to the RFC Editor, new meeting-related menu links and iCalendar links, and restores some review management page information inadvertently lost in release 6.96.0. From the commit log: * Added a link to the meeting registration page to the meetings menu. * Added a link to the registration page from the meeting 'Important Dates' page. * Removed additional instances of links to unposted IPR disclosures that would result in 404 pages. * After feedback from the RFC-Editor staff, changed the URL POST notification of Group name changes to an Email notification. * Added .ics links for future meetings to the group meetings tab. * Added a signal receiver for notifications to the RFC editor about changed Group names. * Added sample notification JWT signature verification code to /api/index.html. * Updated settings with EC/P-256 api keys (JWT does not support RS2048, which is what we had). Added a variable for RFC Editor group name notifications, with a placeholder value. * Fixed template and view to restore previous review display on the Manage Unassigned Reviews page. Fixes issue #2718. * Tweaked the review assignment presentation in review/request_info.html. * Added review assignment information to the review-document view. Addresses issue #2717. -- Henrik Levkowetz 23 May 2019 15:09:45 +0000 ietfdb (6.96.0) ietf; urgency=medium **Refactored review request handling, introducing review assingnments** This release splits out document review assignments as individual objects, instead of being part of review requests. This makes handling of review assignments more flexible and intuitive. The datatracker menu URL for past meeting proceedings has been corrected, as well as the URL for BOF requests, and old, now invalid URLs to MHonArc mailing list repositories have been removed. The Wiki maintenance scripts now support a generic NomCom wiki, managed by the chairs and adivisors of currently active NomComs. Additional details from the commit log: * Merged in a refactoring of document review code from rjsparks@nostrum.com. This introduces ReviewAssignments, letting us do multiple assingments for a given RevieRequest, without conflating the assignments and their results with the request itself. * Updated the ReviewAssignment api resource to match the current model. * Added a no-follow option to the test crawler, in order to be able to easily test a specific list of URLs. * Modified a template to handle old reviews with incomplete information appropriately, instead of giving 500 errors. * Fixed a problem in 3 views that were accessible through URLs without meeting numbers (e.g., .../meeting/proceedings/overview/) but gave 500 errors when accessed. * Corrected an old URL for past meeting proceedings. * Fixed some migrations issues that prevented the review assignment refactoring migrations from being reversed and re-applied. * Changed a ReviewAssignmentDocEvent type from 'closed_review_request' to 'closed_review_assignment', for consistency and readability. * After having test_duplicate_person_name() fail for a non-ascii name, changed the test to always supply a non-ascii name. * Removed links to the old MHonArc mail archives, and related view. * Changed the BOF request procedures page URL. * Tweaked the greate_group_wikis management command to accept trailing fileglob wildcards on adhoc wiki group acronyms. Added a generic NomCom wiki to the list of adhoc wikis to be created and maintained, with admins from active nomcom* groups. * Fixed bad html in a proceedings template. -- Henrik Levkowetz 06 May 2019 15:34:13 +0000 ietfdb (6.95.1) ietf; urgency=medium This is a bugfix release, addressing the following issues: * Modified the display of updates to a rejected IPR to avoid links that result in 404 pages. * Combined [16154] from rjsparks@nostrum.com with previous work on ticket #2633 in [16124]. This adds bluesheet links to the meeting materials page. Fixes issue #2633. * Merged in [16152] from rjsparks@nostrum.com: Show minutes on the popup for the meeting materials button on the agenda page when those minutes are available. Fixes #2711. * Updated the role handling to use GroupFeatures.groupman_roles consistently for group management access. Fixes a IRTF RG delegate permissions issue. * Fixed some program group feature settings. * Added a missing test directory cleanup action. * Merged in [16145] from housley@vigilsec.com: Updated the out-of-date links in the footer of each datatracker page. Fixes #2710. * Merged in [16144] from housley@vigilsec.com: Removed some clutter from I-D upload page. The information was moved to the Instructions tab, but a higligh to the Note Well and BCP 78 was added. Fixes #2057. * Added some HTTP header settings for better security. Brings results at https://securityheaders.com/ up to an 'A' grade. -- Henrik Levkowetz 26 Apr 2019 11:23:57 +0000 ietfdb (6.95.0) ietf; urgency=medium **IETF 104 code sprint** This release contains datatracker bug fixes and enhancements from the IETF-104 Code Sprint. We have several first-time sprint contributors, and a substantial set of improvements. Thanks to everyone who contributed! Notable improvements: The datatracker now supports reminders of and direct addition of downref entries at the time of document approval. Document review management has been improved with table sort options. There are a number of improvements to the agenda and materials pages, and there is now support for meeting session presenters to upload their own slides, for later approval by chairs. Additional comments, excerpted from the commit log: * Merged in [16126] from housley@vigilsec.com: Allow Secretariat to handle downrefs when they approve a document * Merged in [16124] from rcross@amsl.com: Add No Bluesheets indicator to materials page. * Merged in [16121] from rjsparks@nostrum.com: Provides menubar offset for wg anchors on the /wg page. Fixes #1481. * Merged in [16110] from rjsparks@nostrum.com: Include the note itself in email sent when the RFC editor note changes after document approval. Also changed the flow to redirect back to the writeups pages after editing the RFC editor note. Fixes #2672. * Merged in [16109] from rjsparks@nostrum.com: Protect drafts in certain IRTF states from expiring. Fixes #2669. * Merged in [16108] from fenton@bluepopcorn.net: Moved old nomcom announcements to database and removed references to legacy website. Fixes #2686. * Merged in [16107] from rjsparks@nostrum.com: Show the agenda note, if it exits, on a sessions SUMMARY line in the ical view of an agenda. Fixes #2676. * Merged in [16106] from rjsparks@nostrum.com: Don't allow users to edit the 'active' value for unknown-email-* addresses. Related to #2696. * Merged in [16103] from rjsparks@nostrum.com: Modified the admin model for SlideSubmission to use a couple of raw_id_field widgets. * Merged in [16102] from rjsparks@nostrum.com: Allow logged in users to propose slides for meeting sessions. Fixes #2547 and #2403. * Merged in [16095] from peter@akayla.com: Fixed bad calls to rfcdiff for -00 drafts and replaced drafts. Fixes issue #2697. * Merged in [16094] from magnus.westerlund@ericsson.com: Added sortable columns to Review Team Reviewers page. Partially addresses issue #2656 * Merged in [16092] from rcross@amsl.com: Remove ability to cancel a break session. Fixes #2703. * Changed the etherpad url schema to https. * Temorary workaround to upload xml validation. The tree.validate() of xml2rfc should be updated to check vocabulary version and validate against the appropriate schema, to fix this properly. * Added dbtemplate validation to the DBTemplate class, to make sure one cannot enter invalid DBtemplates through the admin. * Merged in [16089] from mahoney@nostrum.com: added instructions for entering the draft name. Fixes #1375. * Added creation of a group with the user's GID inside the docker container, if it's missing. * Merged in [16088] from rcross@amsl.com: Fixed multiple issues with secretariat non-session view, including an indicator of cancelled sessions. Fixes #2701. * Merged in [16087] from housley@vigilsec.com: Add downward references to telechat moderator package and Secretariat telechat pages. * Merged in [16085] from rcross@amsl.com: Fixed an issue where cancelling one break session would cause other breaks to get cancelled. Fixes #2699. * Added filtering by meeting for the UrlResource admin. * Changed docker/updatedb to use https://, as cloudflare doesn't do the right thing for our http:// database dump URL any more. * Added a skipIf() decorator for a test which should not be run unless the ghostscript binary is available in the test environment. -- Henrik Levkowetz 09 Apr 2019 11:55:21 +0000 ietfdb (6.94.1) ietf; urgency=medium This is a minor bugfix release, to clear the slate for the IETF 104 sprint. * Put a length limit to generated cache keys in stats/views.py, to avoid tickling memcached's limit of 250. * Changed sending of PubReq messages to explicitly use on-behalf-of, without Reply-To. Reply-To is picked up as requestor by RT. Fixes issue #2691 anew. * Changed etherpad server address from etherpad.tools.ietf.org to etherpad.ietf.org. -- Henrik Levkowetz 19 Mar 2019 18:01:07 +0000 ietfdb (6.94.0) ietf; urgency=medium **Added review and AD emails 'Reply-To:' fields** This release contains another change to email sending, adding a Reply-To header field for mail sent on behalf of a user, and also contains a number of other changes and fixes, as follows: * Merged in [16022] from rcross@amsl.com: Modified IPR Patent number regex to allow a new Italian patent number format, IT[15 digits]. Fixes #2694. * Reverted earlier application of on_behalf_of() at mail sending call points, and instead did all on_behalf_of processing in condition_message(). Added insertion of Reply-To header fields in this case. Changed all use of the mail-sending 'extra' parameters to use value lists. Updated a bunch of tests accordingly. * Removed a statement about IETF being an activity of the Internet Society from a template file, and tweaked the remaining text. * Converted a test to use .assertIn() instead of .assertTrue(... in ...) for better failure reports. * Tweaked the group_features_group_filter() function. * Refactored some templates to reduce materials page permission lookups from ~2*#sessions to just a few. * Rewrote a function to be more readable and actually do the right thing. This moved some filtering from database into code, but it seems acceptable here. * Transitioned group/views.py to unicode strings * Improved a somewhat misleading UI for adding account email addresses. Fixes issue #2692. * Changed the on-behalf-of name template from 'On behalf of ...' to '... via Datatracker' to make the originator's name come first. * Changed ietf/utils/mail.py to use unicode strings, and changed a call in ietf/review/utils.py to use unicode for mail Subject, to fix a server 500 issue. -- Henrik Levkowetz 13 Mar 2019 15:09:42 +0000 ietfdb (6.93.0) ietf; urgency=medium **Changed review and AD emails to use 'On behalf of ...' sender** This release changes the email sender for emails sent through the datatracker by ADs and reviewers to say "On behalf of" and the username, with a datatracker email address, instead of using the user's from address, since that has increasingly (and rightly) been running into problems with SPF and DMARC processing. The release also provides session short names in the meeting/agenda.json file, for use by Meetecho scripts. From the commit log: * Added on_behalf_of() calls for from addresses needing it throughout the code. * Added html escaping of initial text-area content in a view function to avoid mangling. * Updated the tests to always use @ietf.org addresses where appropriate, and not do so when not appropriate. All test user login addresses have been changed away from @ietf.org. * Added two new functions to replace email From: of users with On-behalf-of addresses, with associated values in settings. Added a catchall stanza in the mail sending pipeline to catch anything not already changed to On-behalf-of addresses, which should have been changed. * Merged in [15982] from rjsparks@nostrum.com: Add session.short to the output from meeting/agenda.json when it is available. -- Henrik Levkowetz 05 Mar 2019 17:52:55 +0000 ietfdb (6.92.1) ietf; urgency=medium This release contains one database schema migration delayed from the previous release in order to deploy new code before a fields used by old code was removed, and one migration to extend the width of a field used to hold URLs. From the commit log: * Increased the DocumentURL.url field length to the legal URL maximum of 2083. * Removed the GroupFeatures.has_dependencies field. * Added a note to the draft/RFC stats pages about missing data. -- Henrik Levkowetz 26 Feb 2019 12:27:22 +0000 ietfdb (6.92.0) ietf; urgency=medium **Generalized the submission approval protection for special groups** Before this rewrite, code explicitly looked for draft names like draf-iab-, draft-iana=, etc., but did not derive the list of draft name prefixes needing approval from the group types in the database. The new code extends this protection to any new group types in the database (and a number of old ones that were not protected before such as draft-iesg- ). There are also some bugfixes. From the commit log: * Increased the document count for group document lists from the default(200) to 500, as MPLS now exceeds 200. * Rewrote the submission deduce_group() to consider all groups with acronyms that are also GroupTypeNames. Added GroupTypeName for iesg,ise,adhoc groups, and appropriate GroupFeatures for each. * Fixed a problem with the secretariat IESG agenda handling, where removal of a document from the telechat agenda could lead to a server 500 in a different view. Insetead, redirect to the removed document (which will show as not on the agenda any more). * Tweaked mkdevbranch to take an optional extra message paragraph. * Modified a view function to fill in missing default CommunityLists for groups with features.has_documents == True, to avoid a possible server error. * Removed the has_dependencies GroupFeature, and used has_documents instead in the one place it was used. * Fixed a submission failure for xml files missing abstract sections. -- Henrik Levkowetz 25 Feb 2019 13:08:34 +0000 ietfdb (6.91.0) ietf; urgency=low **Expanded group property table entries** This release contains a refactoring of a large number of instances in the code where group types (wg, area, rg, etc.) have been hardcoded in the past in order to reflect different properties for different groups. Instead, group types have now aquired a list of new 'feature' settings, for instance is_schedulable, show_on_agenda, has_session_materials, and more. This will make it easier to add new group types without having to change the code, by indicating for each new group type which properties it should have. Additionally, some minor unrelated fixes has been applied. From the commit log: * Merged in GroupFeatures refactoring and expansion from ^/personal/henrik/6.89.4.dev0@15911. * Reverted r15071 with respects to showing the submit for publication button to shepherds. Fixes issue #2673. * Updated code and templates to use the new docman and groupman group features. * Updated fixture with new group features and more. * Added a tiny management command to list group types per feature. * Removed origin information from the liaison_title.html template, as it's used in a context where html comments will be rendered. Fixes issue #2674. -- Henrik Levkowetz 04 Feb 2019 11:57:59 +0000 ietfdb (6.90.0) ietf; urgency=low * Modified the rolodex code to make sure we capture information about the origin of newly added email addresses. This requires the secretariat to be clear about where new email addresses come from, in order to be able to remove them if we later receive a GDPR personal data deletion request. * In the draft metadata extraction code, moved the __future__ imports down so as not to obscure the module docstring. Fixes the earlier inability to run '$ ietf/utils/draft.py -h'. * Applied a series of patches for python 3 compatibility from dkg@fifthhorseman.net to the draft metadata extraction code: - Use a list of dictionary keys. In python3, dict.keys() produces a dict_keys object, not a list. Since this code treats it as a list, we'll just be explicit about that. - Fix regex manipulation for word characters. In python 3.7, re.sub() started treating unknown escape sequences in as errors. Fix this by sending an escaped \ where we mean to pass it through raw. - Use // for explicit integer division. Without this fix, in modern versions of python, the changed line produces: 'TypeError: 'float' object cannot be interpreted as an integer'. - Use modern form of exception handling. - Use the print function instead of print statement. * Added more name normalization for test factory person names. -- Henrik Levkowetz 21 Jan 2019 11:49:20 +0000 ietfdb (6.89.4) ietf; urgency=low * Moved __future__ imports down in ietf/utils/draft.py so as not to obscure the module docstring. Fixes inability to run '$ ietf/utils/draft.py -h'. * Applied 5 patches from dkg@fifthhorseman.net, for py3 compatibility in the text draft metainfo extractor ietf/utils/draft.py: - Use a list of dictionary keys. In python3, dict.keys() produces a dict_keys object, not a list. Since this code treats it as a list, we'll just be explicit about that. - Fix regex manipulation for word characters. In python 3.7, re.sub() started treating unknown escape sequences in as errors. Fix this by sending an escaped \ where we mean to pass it through raw. See https://docs.python.org/3/library/re.html#re.sub - Use // for explicit integer division. Without this fix, in modern versions of python, the changed line produces: TypeError: 'float' object cannot be interpreted as an integer - Use modern form of exception handling - Use print() function instead of print statement * Added a bit of name normalization for test factory person names. -- Henrik Levkowetz 14 Jan 2019 22:29:06 +0000 ietfdb (6.89.3) ietf; urgency=low * Updated the timezone information to match the latest timezone database release (2018i) * Merged in [15865] from rjsparks@nostrum.com: Cleanup: removed the infrastructure for importing reviews from the old review tool. * Merged in [15864] from rjsparks@nostrum.com: Refactored DocumentInfo to address overloading the external_url field with strings that are not URLs. * Merged in [15846] from rjsparks@nostrum.com: Simple tests of some log utilities. * Merged in [15845] from rjsparks@nostrum.com: simple test of BowerStorageFinder. * Merged in [15833] from rjsparks@nostrum.com: Incremental improvements to the submit_existing tests. * Merged in [15829] from rjsparks@nostrum.com: Make state/substate changes on new submissions sensitive to stream. Fixes #2642. * Merged in [15828] from rjsparks@nostrum.com: Show more group types on the concluded groups page. Fixes #2643. * Additional tweaks to submission yang testing and the test runner * Updated PLAN -- Henrik Levkowetz 14 Jan 2019 15:25:32 +0000 ietfdb (6.89.2) ietf; urgency=low * Added a migration to capture changes to HistoricalRecords triggered by changes in django-simple-history * Bumped the minimum version requirement on django-simple-history; HistoricalChanges which we now use was introduced with 2.3.0. * Changed the construction of element IDs that could trigger ad-blockers in a NomCom page to avoid the problem. * Removed an assertion for iesg state for documents that can be in any stream. * Merged in [15824] from rjsparks@nostrum.com: Make it more obvious that a group has been replaced when looking at its pages. Fixes #2661. * Merged in [15818] from rjsparks@nostrum.com: Whitespace only change to make the conditional structure followable. * Merged in [15822] and [15809] from rjsparks@nostrum.com: Added an explicit ID-Exists state for the IESG state machine. Reworked code so that the IESG state machine always has a state. Added the ability to release a document from a working group, research group, or the independent stream. Releasing a document removes all stream state, and sets the document to have no stream. * Fixed a bug in the RFC Index parsing that caused extraction of only one paragraph of an abstract, even if there were multiple. * Merged in [15816] from rjsparks@nostrum.com: Provide a correct link to jabber logs for plenaries in past agendas. Fixes #2641. * Merged in [15810] from rcross@amsl.com: Grant secretariat admin permissions to mailtrigger models. Fixes #2637. * Merged in [15808] from rjsparks@nostrum.com: Allow notify_ad_when to be blank. * Merged in [15807] from rjsparks@nostrum.com: Repaired another instance of a literal path with a reverse lookup. Related to #2657. * Added a description of the /doc//doc.json API to the /api/ index page, and also added a table-of-contents column. * Merged in [15806] from rjsparks@nostrum.com: Changed an inappropriate literal path to a reverse url lookup. Fixes #2657. * Merged in [15804] from rjsparks@nostrum.com: Add history to ReviewerSettings. Fixes #2147. -- Henrik Levkowetz 19 Dec 2018 08:27:48 +0000 ietfdb (6.89.1) ietf; urgency=low * Merged in [15787] from rcross@amsl.com: Rework page flow and forms of secr/meetings app. Fixes #2609. * Added code to catch a possible attribute error on event notification. * Added two new management commands to make it easier to load back selected objects that have been removed by mistake (providing they are available in a full database dump or backup that can be loaded and worked with): dumprelated and loadrelated. * Increased the document limit from 200 to 500 for the AD documents page. -- Henrik Levkowetz 03 Dec 2018 18:09:35 +0000 ietfdb (6.89.0) ietf; urgency=low This release provides improved export of statistics charts and data, improved and more consistent test coverage, and fixes a few bugs: * Fixed an issue with the To: and Cc: addresses of extra review email to responsible AD. * Changed the liaison tool From: address to . * Always show the liaison submitted date on liaison pages * Added testing of unidecode_name() and name_parts(), in order to not rely on random name generation to excercise all paths. * Updated highcharts from v4.2.7 to v6.2.0, and added support for CSV export from charts. Also tweaked the look of the highcharts context menu separators. * Added handling for an EOF exception that can occur when using the django file-based cache. -- Henrik Levkowetz 20 Nov 2018 16:30:42 +0000 ietfdb (6.88.0) ietf; urgency=low **IETF 103 code sprint** This release contains datatracker bug fixes and enhancements from the IETF-103 Code Sprint. Again, this looks to be a sprint with an extremely large number of contributions. Thanks to everyone who contributed! Notable improvements: Reviewer statistics are now more readable. Reasons for reviewer unavailability is now only shown to authorised persons. It is now possible to set a mail alias for review group secretaries, instead of having individual mails sent on updates. AD notifications about reviews can now be controlled better and the default settings are better tuned. Import of review text from mailing lists should now handle zero search results better. Logging in is now possible using all active email addresses registered for an account, not only the primary login name. Inaccurate, misspelled or misleading text has been fixed for a number of web pages and email messages. Additionally, a large number of individual bugs has been squashed: * Merged in [15722] from rcross@amsl.com: Added a link to meeting/../agendas/list. Fixes #2627. * Merged in [15720] from rjsparks@nostrum.com: Invalidate the get_state_slug cache when the get_state cache is invalidated. Fixes #2630. * Merged in [15719] from rcross@amsl.com: Removed resources field from secr/meetings/../rooms. Tweaked CSS for this page to eliminate horizontal scrolling. Includes new base templates to facilitate transition to bootstrap. Fixes #2628. * Merged in [15717] from rcross@amsl.com: Reworked the buttons on secr/meetings. Fixes #2629. * Merged in [15715] from rcross@amsl.com: Fixed the issue where new non-session location field value wasn't saved. Fixes #2625. * Merged in [15709] from rjsparks@nostrum.com: Show counts on the active groups page and on the active WGs page. Fixes #2060. * Updated the script to fix ambiguous time stamps so that it will fix ambiguous time fields for additional types of objects. * Added a test for various name part combinations, and fixed a couple of issues with utils.name.name_parts(), making the test coverage percentages more stable. * Added exclusion of cancelled session when doing audio file import. This fixes an server 500 error when a cancelled session is shown in the same room and time slot as an actual session. * Corrected the rendering of non-ASCII email names in nomcom templates. * Merged in [15708] and [15730] from housley@vigilsec.com: Added a mail alias for all review team secretaries. Fixes #2139. Use the alias if there is one for email about review request changes. Added a test for the review team secretary alias functionality. * Added a test that checks the mailarch search failure return, and tweaks the no-result return handling. * Merged in [15706] from kivinen@iki.fi: Added easier to detect error message when search returns 0 results. This currently parses HTML result from the mail archive system, but when we get proper API to mail archive system it should be changed to do that instead. Fixes #2126. * Merged in [15705] from johnl@taugh.com: Trivial fix for trivial issue #1907, warn user that added addresses will be confirmed. * Merged in [15702] from johnl@taugh.com: Allow email as well as username when logging in. * Merged in [15701] from kivinen@iki.fi: Changed format of the reviewer statistics in lists. Fixes #2131. * Merged in [15683] and [15699] from housley@vigilsec.com: Do not show reasons for review team member unavailability outside the team. Fixes #2614. * Added normalisation of the email addresses returned by ietf.nomcom.utils.parse_email, in order to not treat feedback from the same person as coming from different sources due to email name+address variations. * Merged in [15698] from rjsparks@nostrum.com: Notify the RFC Editor when an RFC Editor note is changed after a draft has been approved. Fixes #2440. * Merged in [15696] from peter@akayla.com: Minor text fixes. Fixes issue #2621. * Merged in [15694] from rjsparks@nostrum.com: Clearly label cancelled sessions on the week-view on the agenda page. Fixes #2619. * Merged in [15693] from mahoney@nostrum.com: Fixed link to webex request page. Fixes issue #2564. * Added a guard against getting the attribute of None in api_set_session_video_url(). Fixes a server 500 for some hackathon sessions. * Provided a tool-tip with full time information for the document history dates. * Merged in [15689] from mahoney@nostrum.com: It need not be an error to find duplicate person records, such as for instance for 'Matt Larson'. Took out the statement in the profile template that this is an error. Fixes issue #2616. * Factored out some parts of the meeting/agenda.html into separate files, and added session buttons for 'other' session (e.g., hotrfc) * Merged in [15688] from mahoney@nostrum.com: Fixed bad variable in title of requests.html. Fixes issue #2608. * Don't show slides in main IETF meeting agenda for sessions with their own session agenda. * Merged in [15684] from rjsparks@nostrum.com: Removed the field and widget that made editing document states easier on the document admin page since they do not survive validation failure of other fields on the form. Excluded states from the model form since the default states widget is almost impossible to use without producing bad data. If states need to be edited through the admin, it will, for now, need to be done gruelingly through the admin form for State. Fixes #2524. * Merged in [15681] from rjsparks@nostrum.com: Only do state-parent cross-validation on the group edit form when both fields are used on the form. (Many invocations of the form suppress all but one field.) Fixes #2597. * Merged in [15679] from mahoney@nostrum.com: Verb tense used in Ballot Issued emails now matches whether the LC end date is in the past or future. Fixes issue #2482. * Merged in [15677] from rjsparks@nostrum.com: Tune the recipients when a teams settings or a reviewer indicate ADs should be notified about a review. * Merged in [15676] from rjsparks@nostrum.com: Let a reviewer indicate that an AD should give this review early attention. Fixes #2553. * Merged in [15675] from rjsparks@nostrum.com: Let review teams opt in to poking a responsible AD when unhappy reviews are submitted. Fixes #2544. -- Henrik Levkowetz 13 Nov 2018 09:47:39 +0000 ietfdb (6.87.3) ietf; urgency=low This release contains one single fix, to give stable test results for the test_edit_timeslot_type() test. * Merged in [15648] from rjsparks@nostrum.com: Use only ietf meetings for testing editing timeslots. -- Henrik Levkowetz 02 Nov 2018 02:51:32 +0000 ietfdb (6.87.2) ietf; urgency=medium This is a small bugfix release to clean the slate before the code sprint. From the commit log: * Merged in [15613] from rjsparks@nostrum.com: Make having a trailing slash optional for doc/html/docname. Fixes #2247. * Merged in [15603] from rjsparks@nostrum.com: Changed some text and button labels to make the workflow for commiting group schedules more self-documenting. * Clarified that the document search author field accepts also author affiliation as a search term. * Removed microscopic photos from nomcom feedback page buttons. Few browsers render them correctly with badges, and they are too small to provide meaningful information. * Changed the generation of the 1id-index.txt file to not list individual drafts in states Candidate for WG Adoption and Call for WG Adoption Issued as group drafts. -- Henrik Levkowetz 31 Oct 2018 16:13:17 +0000 ietfdb (6.87.1) ietf; urgency=medium This is a bugfix release, whith the following changes: * Merged in [15601] from rjsparks@nostrum.com: Corrected some DocTypeName entries' prefix value. Made the search code more robust against DocTypeName objects that have an empty prefix value. Fixes #2599. * Merged in [15596] from rjsparks@nostrum.com: Improved coverage for several tests. * Merged in [15595] from rjsparks@nostrum.com: Converted stats, submit, sync. and utils/test to use factories. Excluded utils/test_data from coverage checks. * Fixed some incorrect topic audience id slugs, to make the NomCom topic availability filtering work correctly. * Updated the nomcom reference from 3777 to 7437 on the Person profile edit page. * Merged in [15580] from rjsparks@nostrum.com: Better constrain a liaison test object to match the preconditions for the test. * Tweaked the mail sending routines' handling of the copy flag. * Changed the mail sending routines so it's possible to not copy the debug archive on some messages. * Added 'os.chdir()' to virtualenv_activate to place us correctly to pick up patch files. * Added Message-ID to log lines for sent email messages. -- Henrik Levkowetz 22 Oct 2018 16:17:25 +0000 ietfdb (6.87.0) ietf; urgency=medium **Nomcom, agenda, and test improvements, new API for secretariat use** This is a small feature release, with the following changes: * Updated the nomcom admin pages for Feedback to make it possible to see and sort on the Nominees field. * Merged in [15551] from rjsparks@nostrum.com: Used factories instead of make_test_data for malinglists, mailtrigger, message, person, secr/announcement, and secr/areas tests. * Merged in [15550] from rjsparks@nostrum.com: Added meeting.agenda_info_note. * Merged in [15549] from rjsparks@nostrum.com: Renamed meeting.agenda_note to meeting.agenda_warning_note. * Changed the 'View' button on the NomCom email classification page to render the email body within 
, and to not offer classification 
    choices, as this encourages a slow workflow, and also seems to have been 
    broken for some time.

  * Bleach 3.0.0 has a bug that can result in exceptions for some inputs.  
    Bug report filed.  Disallowed that version in requirements.txt

  * Turned the nomcom chair dropdown menu item 'Classify pending feedback' 
    into a tab with a mail count badge.

  * Merged in [15263], [15264], [15265], [15287] from rcross@amsl.com:
    Added new custom API endpoints for export of person data, restricted
    to secretariat use.  Added a test for the new custom API.

 -- Henrik Levkowetz   15 Oct 2018 05:46:07 +0000


ietfdb (6.86.0) ietf; urgency=medium

  **NomCom tool improvements**

  This is a feature release with only small tweaks, mostly related to the
  NomCom tools, but there are also some IESG page, test suite, account, and
  utility script improvements, 

  * Added pagination to the nomcom pending-feedback page, to reduce load 
    time.

  * Changed the default sort for nomcom.models.Nominees to be 
    'person__name'.

  * Merged in [15523] from rjsparks@nostrum.com:
    Converted the liaisons tests to use factories.  

  * Added a '--reminder' switch to the GDPR request mailer, for repeat 
    notifications.

  * Merged in [15522] from rcross@amsl.com:
    Made email address read-only in rolodex edit form.  Fixes #2602.   

  * Prevent issues for logins without Person records by not letting 
    Person-less users through login.

  * Added an send_email field to BallotPostitionDocEvent to reflect whether
    the 'Send email' button was used to save the ballot position.  Added icons
    on documents' IESG evaluation record to reflect this.

  * Adjusted the test_submit_nonascii_name() to match the output from 
    idnits 2.16.0, which emits warnings rather than errors for nonascii content.

  * Built out test_edit_nomcom_view() to also test adding and removing 
    reminder dates.

  * Added a form clean() for the reminder dates of the nomcom edit form, to 
    make something appropriate out of edits that remove reminder dates, rather 
    than clicking 'delete'.  This fixes a potential exception, when a blank 
    date cannot be saved, and also does what the user expected (deletes a 
    blanked date).

  * The nomcom send_reminders command was not being run daily.  Fixed that.

  * Fixed a typo in the GDPR consent request email template.

  * Merged in [15492] from rjsparks@nostrum.com:
    Use factories for ipr tests.  

 -- Henrik Levkowetz   08 Oct 2018 13:23:53 +0000


ietfdb (6.85.1) ietf; urgency=medium

  **Email address managment and NomCom tool improvements**

  This release brings no new user features, but provides a script that
  will read email delivery-status messages and disable addresses with
  permanent delivery failures.  There are also some other tweaks:

  * Added a management command to deactivate email addresses found in 
    delivery-status emails or given on the command-line.

  * Added column sorting to the private nomcom nominee table.

  * Tweaked the management command to send GDPR consent request emails, to 
    correctly perform exclusion based on the time of previous consent request 
    emails.

  * Added a PersonEvent type for email address deactivation.

  * Tweaked PersonAdmin to show name_from_draft after the name field.

  * Added a clarification to the GDPR consent request email text after
    receiving about 100 reply emails in response to the first email round that
    contained what seems to be automatic responses indicating 'Yes, I
    confirm'.  All those were from gmail addresses.  The consent request
    now clearly says that people need to log in to their account in order
    to indicate consent.

 -- Henrik Levkowetz   30 Sep 2018 14:47:48 +0000


ietfdb (6.85.0) ietf; urgency=medium

  **Improvements in NomCom pages and personal information handling**

  This is a combined feature and bugfix release, with fixes related
  to User and Person record handling and merging, discovered when people
  responded to the GDPR consent request emails, and enhancements in
  the NomCom tools:

  * Added username validation to the change username form, to avoid 
    database key collision exceptions.  Instead give a message to contact the 
    secretariat to resolve the situation.

  * Fixed a problem where an exception could be raised when reporting an 
    exception during draft XML processing to the user.

  * Added 'consent' to the fields handled when merging person records.

  * Merged in [15486] from rjsparks@nostrum.com:
    Allow a nomcom to show nominees that have accepted nomination before 
    feedback is open. Fixes #2598.  

  * Tweaked the consent request email text to point out an alternative to 
    follow email links.

  * Merged in [15450] from rcross@amsl.com:
    Change content-type of ajax_get_liaison_info to application/json.   

  * Tweaked the text of the personal information help page to remove some 
    ambiguity.

  * A string.Template with a unicode template must be given unicode 
    dictionary values if the values contain unicode codepoints.  Providing 
    values that are objects with __str__() methods, or utf-8 encoded strings 
    doesn't work.  Fixes an issue with nomcom template interpolation for 
    nominees with non-ascii names.

  * Tweaked the GDPR reminder command to avoid bounces to RT and copies to 
    gmail.

  * Changed the sort order of nominees on the private nomcom index page to be
    according to name (instead of email address).

  * Fixed an issue where a user record without matching person record will 
    cause an exception.

  * Removed the 2.5 hour session length choice for IETF 103, since there 
    will be no Friday sessions that meeting.

  * Updated the personal information notice email text based on advice from 
    councel (Tom Zych).

 -- Henrik Levkowetz   28 Sep 2018 23:50:01 +0000


ietfdb (6.84.0) ietf; urgency=medium

  **GDPR updates part 4**

  This is the last in a series of GDPR-related releases.  It introduces two
  management commands; one to email out reminders to all users that still have
  not provided consent for storage of the personal information for which the
  GDPR requires consent, and the other to delete all information for which
  we don't have required consent.

  The two new management commands are:

    - ietf/manage.py send_gdpr_consent_request
    - ietf/manage.py delete_data_lacking_consent

  Both of these provide descriptions of the commands and switches when given
  a --help switch, to assist the secretariat in using these.

  There are also a few bugfixes and features not related to the GDPR, see
  below:

  * Merged in the latest GDPR changes.  This refines the handling of the 
    consent checkbox on the account page; refines the Consent Needed warning 
    given on login if consent is needed; tweaks several models to set the 
    on_deletion fields for FK to User and Person appropriately; adds a 
    Person.needs_consent() method to capture the logic of which fields require 
    consent; refines the Person.plain_name() method and the user.log.log() 
    function; and adds 2 management commands to send out consent requests and 
    delete non-consent information, respectively.

  * Refined utils.log.log() to flush stdout before logging if we're writing 
    log messages to stderr.

  * Moved document methods rfc_number() and rfcnum() to the abstract parent 
    class to have them available also for historical document records.  Should 
    fix the recent server 500 on /doc/draft-ietf-aaa-diameter/17/.

  * Merged in [15448] from rjsparks@nostrum.com:
    Add CORS headers for the api and views that serve json. Fixes #2546, 
    #2551, and #2552.  

  * Added a bit of javascript and styling to make it possible to use URL 
    fragment identifiers to link to position-specific tabs on the nomocom 
    requirements and questionnaire pages.

  * Merged in [15446] from rjsparks@nostrum.com:
    Use unversioned links in ics agendas. Partially fixes #2542.  

  * Merged in [15444] from rjsparks@nostrum.com:
    Review assignment email includes a link to the review request that 
    ensures the reviewer is logged in. Fixes #2543.  

  * Added a 'Mailing list discussion' link to the 'Additional URLs' section 
    of the status page for drafts with doc.group and doc.group.list_archive set.

  * Merged in [15441] from peter@akayla.com:
    Fixed 2 trivial spelling errors in NomCom tools.  Fixes issue #2569.   

  * Fixed a problem with fetching the mysql release signing key from a gpg 
    keyserver.

  * Added more hint information when patch actions in ietf/checks.py fail.

  * Rewrote docker/run to work around a missing switch in BSD 'xargs'.

  * Merged in [15435] from rjsparks@nostrum.com:
    Include area groups in the request for minutes. Do not include sessions 
    that have been canceled.  Fixes #2568. 

 -- Henrik Levkowetz   18 Sep 2018 14:07:27 +0000


ietfdb (6.83.1) ietf; urgency=medium

  This is a small bugfix and cleanup release:

  * Replaced some cases of using person.plain_name when linking to a person's
    profile with person.name, as the link using plain_name can fail.

  * Generalized the log.assertion() function a bit to be able to provide 
    more information on assertion failure.

  * Merged in [15423] from rcross@amsl.com:
    Removed WG milestone management from secretariat groups app.   

  * Removed youtube video url import, which is not needed any more given the
    /api/meeting/session/video/url API endpoint.  This also prevents duplicate
    video URLs.

  * Changed the menu listing of review dirs and their display on the 
    Directorates page back to what it was before the 'reviews' group type was 
    split out.  Added a 'Reviews' column to the Directoratges page.

  * Added a guard against comparing email origin with username of None.

  * Added a None guard for the person merge last_login date comparison.  
    Fixes issue #2562

 -- Henrik Levkowetz   14 Aug 2018 12:31:32 +0000


ietfdb (6.83.0) ietf; urgency=medium

  **IETF 102 code sprint**

  This release contains datatracker bugfixes and enhancements from the
  IETF-102 Code Sprint.  While we've had many more code sprint participants
  before, I don't believe we've ever had this number of contributions from an
  IETF week :-)

  The release contains a number of different enhancements to the document
  review pages (thanks Tero, Peter); refactoring of a number of tests to use
  test object factories instead of indiscriminate creation of a large number
  of often unused test objects, thus speeding up the tests (thanks to Robert);
  fixes to template language and anchors (thanks to Jean); a fix for annoying
  issue with a liaison form (thanks, Ben); a number of updates to secretariat
  pages and proceedings (thanks, Ryan); a fix for a long-standing issue with
  the document search forms behaviour when no search string has been given
  (thanks, Jim); and a test to expose bugs in downref listing in last call
  announcements (thanks, Russ).

  From the commit log:

  * Show links to personal profile pages from group photo pages also when no 
    photo is available -- the profile pages now contains more than just the 
    photo.

  * Merged in [15399] from rjsparks@nostrum.com:
    Use factories for ietf.ietfauth.tests.  

  * Merged in [15395] from rjsparks@nostrum.com:
    Allow the IRTF stream to set shepherds and shepherd writeups. Fixes #1993.  

  * Merged in [15387] from rjsparks@nostrum.com:
    Use factories for ietf.iesg.tests.

  * Merged in [15380] from rjsparks@nostrum.com:
    Use factories for ietf.idindex.tests.

  * Merged in [15370] from rjsparks@nostrum.com:
    Use factories for ietf.group.tests_review.

  * Merged in [15358] from kivinen@iki.fi:
    Fixed the review tablesorter issue by adding display:none character in the
    front of the date. Tablesorter looks first characters and picks the parser
    based on that, meaning that if the text starts with number, it will try to
    parse everything in column as numbers, and if there is anything that is
    not numebr it will fail, and only sorts the column once. This is bit ugly
    but I do not think we want to add yet another jquery meta tool. Fixes
    #2089, and #2151.  

  * Added a missing migration (for review unavailable-period reason)

  * Merged in [15357] from mahoney@nostrum.com:
    Added anchors for working group rows. Still needs some work since the 
    anchor ends up being under the menu bar. Partially addresses issue #1481.  

  * Merged in [15356] from ben@nostrum.com:
    Changed newlines to commas in to contacts for new liaison email 
    notification. Fixes issue 2525.  

  * Merged in [15355] from rjsparks@nostrum.com:
    Used factories for ietf.group.tests_info.  

  * Tweaked a test to prevent failures when looking for a specific string 
    in line-wrapped text.

  * Added a tiny utility function unwrap() to unwrap wrapped text for 
    matching expected strings in tests.

  * Merged in [15354] from kivinen@iki.fi:
    Added default to review completion draft version number. If the reviewer
    is filling in the review form himself the default will be the latest
    version of the draft. If it is someone else filling in the review
    then the version number is the latest version at the point when review
    was assigned (i.e., it assumes reviewer did review on version he was
    requested to and then document authors might have updated the document).
    Also mark the document versions that are earlier than the review request
    date with red background and those which are newer with green background.
    Fixes #2125.  

  * Merged in [15353] from peter@akayla.com:
    Add a sidebar to display feedback section names and simplify navigation.
    Could use some visual improvements but otherwise works.  Fixes #2431.  

  * Merged in [15351] from rcross@amsl.com:
    Update secr/drafts/tests_views to align with view logic.  Fixes #2480.  

  * Merged in [15349] from kivinen@iki.fi:
    Removed the requirement for review_content field to be mandatory when
    using \'link\' method. This partially addresses #2520. 

  * Merged in [15347] from kivinen@iki.fi:
    Add review request date to the top. Add tooltips for drafts versions
    providing the date they were sumbmitted. Fixes #2218. 

  * Limited the document name for uploaded slides to use 128 slugified 
    characters from the slideset title as part of the name.  Fixes a possible 
    server 500 issue.

  * Merged in [15345] from rcross@amsl.com:
    Remove duplicate functionality in secr/drafts.  Fixes #2540.   

  * Merged in [15344] from mahoney@nostrum.com:
    email confirming interim meeting request now says 'the Area Director of 
    the ' instead of ' Area Director'. Fixes issue #2522.  

  * Added additional checks to the meeting_session_video_url api test.

  * Fixed api_set_session_video_url() to update the external video url if 
    given a new value.

  * Merged in [15342] from peter@akayla.com:
    Added optional reviewer unavailability reason support.  Fixes issue #2146.   

  * Merged in [15335] from rcross@amsl.com:
    Fixed the ordering of meeting select in secr/meetings.  Fixes #2539.

  * Adjusted some timezones in test code to use permitted time zones after 
    the previous changes in listed timezones.  Added a migration for the 
    changed timezone list.

  * Fixed display of xml2rfc exceptions when processing draft submission in 
    xml form.

  * Modified the list of timezones available when setting a meeting 
    timezone so as to not show timezones that are links.  (Also edited the 102 
    meeting timezone in the database to avoid this).  Fixes issues #2531 and 
    #2534.

  * Merged in [15338] from fenton@bluepopcorn.net:
    Adds _required_ parameter to search widget. Fixes #1832.

  * Merged in [15331] from housley@vigilsec.com:
    Add test for downref in Last Call announcement 

  * Merged in [15329] from rcross@amsl.com:
    Add important dates link to proceedings.  Fixes #2538.   

  * Merged in [15328] from rcross@amsl.com:
    Give secretariat the ability to cancel non-working group sessions. 
    Fixes #2537.   

  * Updated a test to give predictable results (no future meeting dates) 
    for commit [15361].

  * Fixed an issue where /meeting/ showed the materials page of an earlier 
    meeting, rather than the current.

  * Merged in [15327] from mahoney@nostrum.com:
    Clarified description in the search form, removing the etc since you 
    can't search on author names. Fixes issue #1159.  

  * Merged in [15321] from rjsparks@nostrum.com:
    Converted ietf.group.tests to factories.  

  * Merged in [15320] from rjsparks@nostrum.com:
    The cookie tests were all calling make_test_data, but not using anything
    it created. Removed the calls.  

  * Merged in [15314] from rjsparks@nostrum.com:
    Converted the community app tests to use factories.  

 -- Henrik Levkowetz   25 Jul 2018 19:08:41 +0000


ietfdb (6.82.0) ietf; urgency=medium

  **Group features moved from code to database**

  This release replaces the code defined group features with features held
  in a database table:

  - Added a GroupFeatures model to the group models, and removed the old
    features.py
  - Added a agenda type for future use in showing different group types on
    different agendas.
  - Renamed the group feature has_materials to has_nonsession_materials.
  - Added API resources and admin support for the new tables.
  - Added a Directorate (with reviews) group type as complement to
    Directorate, to distinguish between directorates with and without
    reviews.
  - Adjusted tests as needed.
  - Updated the fixtures, and fixed the generate_fixtures script to include
    the new AgendaTypeName objects.

  There still exists about 70 instances of code comparing the group type
  with a list of types; most of these should probably be replaced with new
  features, instead, to make it possible to add new group types through the
  database table, rather than having to edit the code.  That was the purpose
  of this refactoring from the start, but the presence of this large number
  of comparisons of group type against lists of types defeats the goal until
  we add appropriate features and replace the group type list comparisons.

  Additional changes:

  * Added exception logging for ietf/bin/rfc-editor-queue-updates.  Changed 
    the logging in the ietf/bin/rfc-editor-* scripts to use 
    ietf.utils.log.log().  Reordered some imports.

  * Replaced the use of a deprecated Django field attribute with the 
    sanctioned replacement.

  * Merged in [15282] from housley@vigilsec.com:
    Set reply-to for I-D Announcements that include a CC to a mail list.
    Fixes #2530. 

  * Added support for patent 'kind codes' as part of patent numbers.

  * Tweaked the rsync arguments for IANA YANG models.

  * Upped the permitted number of search results for the /stream/ise/ page 
    (and friends) from 200 to 1000.  Fixes issue #2527.

 -- Henrik Levkowetz   14 Jul 2018 16:47:58 +0000


ietfdb (6.81.3) ietf; urgency=medium

  This is a bugfix release to clear the board for the final GDPR release.
  From the commit log:

  * Merged in [15266] from rcross@amsl.com:
    Updated the blue sheet template.  Replaced start time / end time with date.  

  * Merged in [15267] from rcross@amsl.com:
    Fixed an issue with a decorator on utils.html.remove_tags().   

  * Tweaked the 'Related documents' part of the Document admin page to more 
    intuitively show the relationships.

  * Tweaked unidecode_name() to avoid double-quotes in the ascii string.

  * Merged in [15248] from rjsparks@nostrum.com:
    Improvements to tests after converting to factories suggested by studying
    coverage_changes.  

  * Merged in [15247] from rjsparks@nostrum.com:
    Tweaked the color palette for the agenda editor.  

  * Removed two instances of log.assertion(); we have enough data on these
    now.

  * Changed an attribute guard to work with a reverse relationship attribute
    (using hasattr()).

  * Fixed a broken mailto link on the account edit page.

  * Merged in [15238] from rjsparks@nostrum.com:
    Use factories for ietf.doc.tests_status_change. Repair previous commit
    errors in ietf.doc.tests_material.  

  * Merged in [15228] from rjsparks@nostrum.com:
    Use factories for ietf.doc.tests_downref.  

  * Merged in [15227] from rjsparks@nostrum.com:
    Use factories for ietf.doc.tests_conflict_review.  

  * Merged in [15226] from rjsparks@nostrum.com:
    Use factories for ietf.doc.tests_charter.  

  * Tweaked unidecode_name() to not produce single-letter ascii surnames 
    from non-ascii codepoints.  The unidecode transliteration is in any case 
    somewhat arbitrary, and in most cases a real person will tweak the ascii 
    name of his account.  When running tests, however, this tweak avoids some 
    false test failures.  And no, it's not simple to fix the draft 
    author-extraction heuristics to deal well with single-letter surnames.

  * Added a guard against trying to access attributes of a None object.

  * Merged in [15225] from rjsparks@nostrum.com:
    Use factories for ietf.doc.tests_draft.  

 -- Henrik Levkowetz   20 Jun 2018 21:47:42 +0000


ietfdb (6.81.2) ietf; urgency=medium

  **GDPR updates part 3**

  This is the third in a small series of releases to address compliance with
  the European GDPR (General Data Protection Regulation).  It provides
  serialized (machine-readable) access to information associated with a user's
  account, adds photo and some links to the user account page, and improves
  the annotation of which information is consent-based on the account page.
  It adds a warning on login if consent is missing for some personal
  information, and requests the user to give consent.  It also brings in
  refactoring of a number of test cases to use test object factories, giving
  improved test performance.

  * Merged in [15219] from rjsparks@nostrum.com:
    Use factories for ietf.doc.tests_ballots. Fixed a long-hidden bug in
    test_edit_verify_permissions where non-existing usernames were used and
    passed the tests accidently. Added list_email to the GroupFactory.  

  * Merged in [15217], [15205], [15204], [15203], [15202], and [15201] from
    rjsparks@nostrum.com:
    Converted ietf.doc.tests to use object Factories. Created draft type
    specific factories.  

  * Fixed the logic of gathering which personal information needs consent 
    confirmation in the code that requests consent on login.

  * Added a request to give consent for non-LI information on user login.  
    Addresses issue #2505.

  * Fixed a bug in a form clean method.

  * Added a missing related object to the person information export.  
    Related to issue #2501.

  * Tweaked the language on the user account page.  Related to issue #2501.

  * Added a recursive object to JSON serializer and a view which will let 
    any logged-in user download a JSON serialized copy of the datatracker 
    information related to his person record.  Added information about this, 
    and a link, to the account page.  Related to issue #2501.

  * Added a requirement of oauth2client, needed by google-api-python-client 
    but not always automatically pulled in.

  * Added a link to the overall IETF Statement concerning personal data to 
    the datatracker personal-information page.

  * Added some text about BOF requests to the important-dates page for the 
    session request opens dates, on request from the IESG.

 -- Henrik Levkowetz   08 Jun 2018 16:05:15 +0000


ietfdb (6.81.1) ietf; urgency=medium

  **GDPR updates part 2**

  This is the second in a small series of releases to address compliance
  with the European GDPR (General Data Protection Regulation).

  This release provides two data migrations to polulate email address origin
  and other trace information, to make it possible to differentiate between
  that personal information which is to be treated as Consent-based, that
  which falls under Legitimate Interest.

 -- Henrik Levkowetz   29 May 2018 14:38:09 +0000


ietfdb (6.81.0) ietf; urgency=medium

  **GDPR updates part 1**

  This is the first in a small series of releases to address compliance with
  with the European GDPR (General Data Protection Regulation).  

  This release provides necessary GUI and internal changes, but lacks some
  admin utilities, and also data migration routines which takes some time to
  run, and therefore will be packaged in a separate release and applied
  separately.

  From the commit log:

  * Updated the personal information page with reviewed text from legal 
    counsel.  Fixes issue #2503.

  * Removed the Person.address field, which is not being used.  This was a 
    legacy from the 2001 perl-based datatracker tables.  Fixes issue #2504.

  * Added django-simple-history and replaced the old (and unused) 
    PersonHistory class with a history=HistoricalRecords() field on Person.  
    Added the needed migrations and changes to admin, resources, and settings.  
    Related to issues #2505 and #2507.

  * Added a new field name_from_draft to Person, to hold the name field 
    equivalent as captured from drafts, in case name has been modified by the 
    user and we're asked to remove that info under GDPR.  Added history for 
    Email, and also an origin field to capture from where we got an email 
    address (draft name, username, meeting registration, etc.)  Added a 
    log.assertion() to Email.save() in order to ensure we don't create any 
    email without setting origin.

  * Added origin information to all places where we create email address 
    entries.

  * Removed all references to the removed Person.affiliation field.

  * Added email origin information to some function calls that needed it.

  * Overwrite earlier email origin when we've picked up the address from a 
    submission.

  * Added a consent field to the Person model.

  * Disallow profile changes without consent given.  Together with previous 
    commits this fixes issues #2505 and #2507.

  * Added another category of personal information to the 
    personal-information page, after review of personal information in the 
    code.  Completes issue #2501.

  * Added a dagger at the end of some fields in the account data forms to 
    signify consent-based fields, and made the consent field required.

  * Simplified the email.origin assignment code for outgoing liaisons.

  * Fixed a long-standing bug in the liaison.name() code.

  * Added assingment of the person.name_from_draft field on draft 
    submission.


  * Updated the edit_profile template with information about consent-based 
    fields.  Fixes issue #2502.

 -- Henrik Levkowetz   29 May 2018 11:10:28 +0000

ietfdb (6.80.1) ietf; urgency=medium

  This is a bugfix release which also clears the slate for the upcoming
  GDPR-related release.  From the commit log:

  * Added a meeting helper function update_important_dates(), and a management
    command update_important_dates to update the important dates for a meeting
    if the offsets in the MeetingDateName table has been changed.

  * Added some text to the important dates page for 'opensched', on request
    from the IESG via Stephanie.

  * Merged in [15153] from rjsparks@nostrum.com:
    Changed how the mail utilities handle lists in their override and extra
    arguments. Added a test for send_mail_preformatted exercising override and
    extra.  

  * Merged in [15152] from rjsparks@nostrum.com:
    Send email when the responsible AD for a conflict review is changed.
    Include the stream manager and stream steering group when the responsible
    AD changes or when the telechat date changes. Fixes #2497.  

  * Merged in [15151] from rjsparks@nostrum.com:
    Provide an email group button for WG and RG documents. Fixes #2512.  

  * Merged in [15145] from rjsparks@nostrum.com:
    Secretariat requested tweak to the request for minutes template.  

  * Updated the non-WG mailing list page to include advertised lists for
    closed WGs, on request from the secretariat.

 -- Henrik Levkowetz   24 May 2018 15:53:08 +0000


ietfdb (6.80.0) ietf; urgency=medium

  This release fixes a number of bugs, and adds a view that makes it easier
  for the secretariat to generate the meeting minutes request emails.

  * Merged in [15125] from rjsparks@nostrum.com:
    Added a view that generates the 'send minutes' email for the secretariat.
    Fixes #2493.  

  * Merged in [15124] from rjsparks@nostrum.com:
    Provide a __unicode__ member for ImportantDate.  

  * Merged in [15123] from rjsparks@nostrum.com:
    Use importantdates for material submisssion cutoff and revision if they
    exist. Fixes #2514.  

  * Merged in [15119] and [15120] from rjsparks@nostrum.com:
    Change stream state to replaced when an ISE, IAB, or IRTF document is
    replaced. Fixes #1963.

  * Merged in [15114] from rjsparks@nostrum.com:
    Restored a test that was unintentionally removed. Removed unused
    templates.  Related to #2513.

  * Merged in [15113] from rjsparks@nostrum.com:
    Removed the already disabled 'Make RFC' functionality from /secr/drafts.
    Fixes #2513.
    
    Merged in [15112] from rjsparks@nostrum.com:
    Removed the 'announce' functionality from /secr/drafts. Addresses part of
    #2513.

  * Merged in [15111] from rjsparks@nostrum.com:
    Removed the \'replaces\' functionality from /secr/drafts. Addresses 
    part of #2513.  

  * Merged in [15110] from rjsparks@nostrum.com:
    Made settings.URL_REGEXPS.name non-greedy. Added a test to exercise the 
    bug at #2429. Fixes #2429.  

  * Fixed an incorrect file name in a log message.

 -- Henrik Levkowetz   08 May 2018 06:37:38 +0000


ietfdb (6.79.1) ietf; urgency=medium

  This bugfix release fixes problems with IESG ballot clearing, marks dead
  code as unreachable for later removal, adds new tests to increase test
  coverage, and fixes some issues with the test suite.  It also tweaks the
  roles which are permitted to request document publication, loosens the
  need for trailing slashes for some URLs, and adds a 'Recent drafts' page.

  * Merged in [15084] from rjsparks@nostrum.com:
    Corrected the implementation of clear_ballot. Added a test for it.
    Restored functionality to ballot popups that was missing because a
    template variable was not passed in.  

  * Added a 'recent drafts' page, showing drafs submitted the last 7 days.

  * Merged in [15083] from rjsparks@nostrum.com:
    Removed memoize from latest_event.  

  * Merged in [15080] from rjsparks@nostrum.com:
    Marked a couple of functions as unreachable.  

  * Reverted a change to UserFactory to avoid nepalese names.  The issue is 
    bigger, and needs a different resulution.  Changed the url pattern that 
    used \w to match names to instead use a more inclusive regex.

  * Fixed a PyQuery property access that should have been a method call.

  * Merged in [15076] from rjsparks@nostrum.com:
    Added a test for meeting/requests.

  * Tweaked the IPR factory to be less likely to produce invalid patent 
    info.

  * Merged in [15075] from rjsparks@nostrum.com:
    Improved testing for docs_for_ad. 

  * Merged in [15074] from rjsparks@nostrum.com:
    Simplified docs_in_iesg_process. Added a test for it. 

  * Merged in [15073] from rjsparks@nostrum.com:
    Rearranged ietf/doc/urls.py to allow matching some special pages under 
    /doc without a trailing slash. Adjusted several of those special URLs to 
    work without the trailing slash. 

  * Added permission for a document shepherd to request publication.  Fixes 
    issue #2509.

 -- Henrik Levkowetz   02 May 2018 10:42:26 +0000


ietfdb (6.79.0) ietf; urgency=medium

  **IPR patent info field fixes, IESG position API enhancement, and more.**

  This release enhances the IESG position API to send notification emails on
  changes, widens some IPR disclosure form fields to accept more text, and
  fixes a patent application number validation issue for IPR forms.  It adds
  some useful links and a new filtering option to the past meetings page,
  updates document stream status indications so as to be less prone to
  misinterpretation, addresses an issue with session request forms, and fixes
  an issue with I-D expirations and another related to ordering of documents
  on IESG agendas.

  From the commit log:

  * Set file upload permissions to 0644 to avoid problems with photo access.

  * Increased the length of the patent title in IPR forms

  * Fixed a typo.  Fixes issue #2500 which caused different document ordering
    on the IESG agenda compared with the IESG agenda documents page.

  * Increased the size of the patent_notes form field, and tweaked the
    validate_patent_number validator to accept US application numbers
    containing a series number and forward slash.

  * Merged in [15055] from rcross@amsl.com:
    Fixed an issue where user can make two session request submissions,
    corrupting data.  Also, ignore deleted session requests in edit view.  

  * Merged in [15047] from rcross@amsl.com:
    Reduced the get_meeting buffer to handle earlier shceduling requests date.   

  * Added origin indication to a group of stats templates to make it easier 
    to find the origin of stats configurations.

  * Added group links to the past meetings page, and also important dates 
    links to IETF meetings with important-date pages.

  * Added IETF to past meetings filter pane, in order to be able to filter 
    out interims.

  * Fixed a problem with the expire-ids code, and added logging to catch 
    future problems.

  * Modified the IESG position setting API to send email messages about 
    positions set.

  * Tweaked the wording of coverage_changes output.

  * Fixed an issue where simplified sorting of DocEvents resulted in some 
    ballot icons not being shown on the IESG agenda documents page.

  * Changed incorrect use of 404 status code in api to 400.  Fixes issue 
    #2498.

  * Removed what could be a misleading 'RFC' from  the stream indication in 
    document search results.

 -- Henrik Levkowetz   23 Apr 2018 12:28:57 +0000


ietfdb (6.78.0) ietf; urgency=medium

  **Document search on IRTF state, performance improvements**

  This release brings document search support for filtering on IRTF states,
  courtesy of Mat Ford, and further performance improvements to the IESG
  agenda document page and document search pages.  There are also improvements
  to the testing infrastructure, HTML cleanup, some added API documentation,
  and a number of bugfixes, as follows:

  * Merged in [15012] from ford@isoc.org:
    Added IRTF State option to Additional Search Criteria pulldown.  

  * Included django.utils.html.escape() in the linkify() code, and removed 
    it from various templates, to make sure escape is consistently applied 
    before linkify.  Fixes issue #2492.

  * Tweaked the error return from Document.get_document_content() to be 
    more helpful during development.

  * Fixed various html issues.

  * Added more information about personal API keys to the API help page.

  * Tweaked the test runner to more consistently force the test coverage 
    tests to be run last.

  * Changed calls to .related_that*() and .relations_that*() to use tuples 
    rather than lists (a slight optimization).

  * Silenced another deprecation warning from a library.

  * Reduced the max number of results returned by document searches from 500
    to 200, for a better user experience.  A faster return and a search term
    tweak to pinpoint the desired search result is better than a long wait and
    then maybe having to tweak the search terms anyway.

  * Use .assertIn(...) in tests instead of .assertTrue(x in y) as the 
    failure messages are much more informative

  * Changed the signature of some document methods to not accept lists (but
    still accept tuples) in order to make caching possible.  Added @memoize to
    some methods.

  * Added another prefetch to search doc query results.

  * Imposed sort order (most recent docs first) on document search results, 
    in order to retain the most recent docs in case we have to chop the result
    list due too many hits.

  * Tweaked the django_cprofile_middleware patch to use a query arg of 
    psort instead of sort, to avoid conflict with app query args 'sort'.

  * Added back the on-agenda calendar links to document lists, now in an 
    implementation with better performance characteristics.

  * Updated minimum requirements for bootstrap3, to work with django 1.11

  * Fixed an issue where the change of slide naming between IETF 96 and 97 
    caused the datatracker to not provide correct link to pre-IETF-97 
    materials.  Thanks to tjw.ietf@gmail.com for the heads-up.

  * Merged in [14970] from rjsparks@nostrum.com:
    Factories for the ipr app. Coverage of ietf.ipr.views.ajax_search, and 
    a factory-driven test of ietf.ipr.views.edit.  

  * Enhanced debug.py: made the mark() function indicate file and line 
    number it was called from.  Added if debug: conditions around some other 
    function actions.

  * Another round of iesg page optimizations.  Reduced additional instances 
    of individual lookups to a single lookup followed by annotation of document 
    objects.  Reduced several instances of repeating the same work multiple 
    times (queries for telechat documents, queries for telechat dates).

  * Fixed a number of tests to deal correctly with model method caching.

  * Changed the ballot_icon template tag to use an already found ballot if 
    available, rather than doing lookups to find it (twice!).

  * Added a patch for debug mode, to add a filtering option to the 
    django-cprofile-middleware.

  * Be more specific with bootstrap's layout classes for small devices in 
    the base template, to avoid the bottom links and footer overlaying the 
    bottom of the content area.  Fixes a problem with entering login 
    information on small devices.

 -- Henrik Levkowetz   10 Apr 2018 16:33:46 +0000


ietfdb (6.77.1) ietf; urgency=medium

  **Performance improvements**

  This release brings performance improvements, focused in particular on the
  IESG agenda documents page.  Some of the changes will also provide
  performance benefits for other pages, in particular pages that lists
  documents, such as search result pages and WG/RG document lists.  All in
  all, the changes speed up the rendering of /iesg/agenda/documents with a
  factor of about 2.  The page generation time is still roughly proportional
  to the number of documents on the page, so a large number of documents on
  future IESG telechats will still give a slow page.  More substantial
  refactoring will probably be needed for further speedups.

  From the commit log:

  * Added an index for (doc, type) on DocEvent to speed up latest_event()
    queries.

  * Changed the reviewed_by_teams list to provide acronyms directly, to avoid
    group lookups during template rendering.

  * Changed telechat_page_count() to accept a list of documents if such a list
    has already been generated, to avoid double work.

  * Provided telechat page counts to the template from the view, instead of
    repeated (costly) filtering through the telechat_page_count template
    filter in the template, which caused new, redundant document lookups every
    time.  Removed the telechat_page_count template filter altogether, due to
    its cost. 

  * Tweaked some lookups in fill_in_document_table_attributes().

  * Added terms to the select_related() list for documents in IESG
    agenda_documents().

  * Added a get_active_ads() utility function that uses the cache to reduce
    database access, and replaced database queries in various places with
    calls to get_active_ads().

  * Moved some utility functions so as to reduce the likelihood of import
  * loops.

  * Changed some instances of .slug to _id, to avoid unnecessary database
    lookups.

  * Added a profiling middleware to development settings, if installed.

 -- Henrik Levkowetz   02 Apr 2018 11:55:07 +0000


ietfdb (6.77.0) ietf; urgency=medium

  **Automation API endpoint: /api/meeting/session/video/url**

  This release provides provides a new API endpoint to let video urls be
  associated with sessions, intended for use by Meetecho to update the
  datatracker with sesssion recording links as soon as a recording has been
  uploaded to the YouTube (or equivalent).  It also introduces a new
  'Recording Manager' role, used to authorize the API access.  A short
  description of the API is available at
  https://datatracker.ietf.org/api#session-video-url-api

 -- Henrik Levkowetz   30 Mar 2018 19:11:56 +0000


ietfdb (6.76.2) ietf; urgency=medium

  This release fixes a number of blemishes and cleans the slate for merging
  in a meetecho API branch.

  * Fixed an issue which could result in server 500 errors for paths like 
    /meeting/99/session/plenary.

  * Removed superfluous stats pages headers which would be repeated with 
    different layout on the next line, as part of the graphs.

  * Added a test for IPR edit.

  * Filled in the IPR edit form, section V, with patent information derived 
    from the patent_info field, the same way we do for the IPR update form.

  * Added a pip version restriction to requirements.txt, as there seems to be
    issues with pip 9.0.2 and 9.0.3; they seem to leave dross when down- and
    upgrading packages.

  * Don't show links to individual meeting stats pages when there's no 
    content (meeting 71 and earlier currently has no attendance records).

  * Added meeting dates to the meeting stats overview graph.

  * Changed the name of a loop variable so as not to obscure a previous 
    local variable.

  * Don't show future meetings in the meeting stats overview.

  * Changed settings so that cache keys used by memcached includes the 
    release version, in order to avoid stale and incorrect cache content on new 
    release deployment.  Made it easier to pick out cache key composition by 
    normalizing the variable name.  Adjusted cache prefix and the composition 
    of some keys to have a unique and consistent cache key prefix.

  * Use the actual defaults from settings.py when showing the defaults on 
    /accounts/settings/.

 -- Henrik Levkowetz   28 Mar 2018 20:54:53 +0000


ietfdb (6.76.1) ietf; urgency=medium

  **IETF 101 code sprint, release 2**

  This release provides improved meeting attendance statistics (under
  /stats/meeting/overview/), improved test coverage, additional menu entries,
  html template fixes, total page count on the Last Call page, the ability
  for ADs to clear ballots themselves without going through the secretariat,
  and more.  It also changes the default cookie setting for showing or hiding
  the duplicate left-hand menu column.  (If you don't like the new default, use
  the 'User>Preferences' menu item to revert to the old look.)

  Again, many thanks to all the contributors!

  From the commit log:

  * Fixed some issues with agenda and proceedings code for meetings 
    numbered 64 and below, actualized by the introduction of meeting records 
    for such meetings for stats purposes.

  * The template filter 'keep_spacing' can interact badly with 'wordwrap'.
    Removed it from several places, and made text document display more
    consistent.  Fixes issue #2481.

  * Merged in [14902] from rjsparks@nostrum.com:
    Test coverage improvements for ietf/secr/telechat/views.  

  * Merged in [14900] from rjsparks@nostrum.com:
    Improvements to test coverage for ietf/secr/drafts/views.py.  

  * Merged in [14899] from rjsparks@nostrum.com:
    Cleaner charter factory simplifies test code.  

  * Added submenu to the Statistics menu entry, pointing at the individual 
    stats pages, and entries for NomComs and Releases.

  * Merged in [14896] from rjsparks@nostrum.com:
    Improved test coverage for ietf/doc/views_charter.py.  

  * Fixed the base template to have id='content' for the content also when 
    not showing the left-hand menu.

  * Updated the cookies tests to match the new left_menu default.

  * Changed the default setting for the datatracker left-hand menu to 
    'off'.  The top menu is always present, and works well.

  * Tweaked the document list iCalendar links to not show for presentations 
    on days in the past.

  * Merged in [14884] from adam@nostrum.com:
    Adding page count to last call page.  

  * Merged in [14883] from adam@nostrum.com:
    Adding link to documents in last call to the AD dashboard side menu.  

  * Merged in [14849] from ben@nostrum.com:
    Give ADs the red Clear Ballot button 

  * Merged in [14880] from rjsparks@nostrum.com:
    Added a Draft test suite.

  * Merged in [14879] from kivinen@iki.fi:
    Backfilled IETF meetings starting from 1 and backfilled the attendee 
    numbers for them. Modified the overview attendee statistics page to use 
    different colors for different venue continents.  

  * Tweaked the meeting registratoin import to deal more gracefully with 
    too long names and email addresses, and to fill in the total attendees 
    number for the meeting.

  * Tweaked mergeready to accept an additional hold-for-merge file outside 
    the repository.

 -- Henrik Levkowetz   26 Mar 2018 13:38:04 +0000


ietfdb (6.76.0) ietf; urgency=medium

  **IETF 101 code sprint, release 1**

  This release contains datatracker bugfixes and enhancements from the IETF-101
  Code Sprint.  It brings new views to merge duplicate person records, fixes
  an annoying bug in downref identification and a bunch of links that didn't
  point to the right document, corrects the mime-type for JSON responses, and
  brings substantial improvments to test coverage.

  Many thanks to everyone who contributed!

  * Merged in [14864] from rjsparks@nostrum.com:
    Improved test coverage for ietf.meeting.views.  

  * Updated Note Well.  Fixes issue #2474.

  * Tweaked person factory ascii name generation to not include periods 
    inside names.

  * Tweaked a factory to make duplicate mailing list names less likely in tests

  * Test data template tweak.

  * Merged in [14863] from rjsparks@nostrum.com:
    Improved test coverage for ietf.utils.draft.  

  * Merged in [14862] from rcross@amsl.com:
    Add view for merge person records.   

  * Merged in [14855] from rjsparks@nostrum.com:
    Improved test coverage for ietf.meeting.views.  

  * Added a test case for reference extraction.

  * Merged in [14854] from rjsparks@nostrum.com:
    Improved test coverage over meeting.views.  

  * Merged in [14851] from housley@vigilsec.com:
    Improve parser for references in Internet-Drafts. Fixes #2360 

  * Merged in [14850] from rjsparks@nostrum.com:
    Improvements to test coverage in ietf.meeting.views 

  * Removed premature save of upload document state (doing it after 
    verifying that the save to file worked instead).  Tweaked materials 
    documents revision extraction regexes.  Added a test on no change of 
    materials rev after failed upload.  Corrected the naming of materials in 
    make_meeting_test_data.  Refined the test crawl of meeting materials pages.

  * Added a couple of meeting material crawls to check that internal 
    material links aren't broken.

  * Check attribute isn't None before accessing sub-attributes.

  * Merged in [14845] from rcross@amsl.com:
    Remove unused roles/forms.py.   

  * Merged in [14844] from adam@nostrum.com:
    Fixing 'href' field for
    https://datatracker.ietf.org/meeting//json.  

  * Fixed some materials docname-with-rev regexes that were too narrow.

  * Merged in [14843] from adam@nostrum.com:
    Updated (deprecated, unregistered) 'text/json' with 'application/json' 
    everwhere.  

  * Merged in [14842] from rcross@amsl.com:
    Add check for deleted session when scheduling.  Fixes #2450.   

  * Fixed a bug in a RegexValidator error message string interpolation.

 -- Henrik Levkowetz   21 Mar 2018 10:58:52 +0000


ietfdb (6.75.2) ietf; urgency=medium

  **Session iCalendar links on document lists**

  This release adds session ical links to group document lists and document
  search results, fixes some issues related to upload sanitization and url
  resolution, improves performance for some pages, and adds and improves some
  tests in the test suite.  From the commit log:

  * Merged in [14830] from rjsparks@nostrum.com:
    Simple coverage tests for ietf/secr/drafts/reports.py.

  * Tweaked the document sanitizer to insert a charset meta tag after 
    sanitization.

  * Fixed a mistaken change in session urls.

  * Tweaked a test case to make it easier to get at failures.

  * Made the materials_document() view function more robust in handling 
    various materials document names.

  * Gave the 404 message a class which will make it easier to pick out the 
    message during testing.

  * Updated the meeting materials document view to handle urls with 
    revisions.

  * Updated the meeting document href patterns in settings.py to reflect 
    that we now have versioned agendas, minutes, and slides.

  * Added session ical links to document lists (WG docs, search results, 
    etc.) for documents on upcoming meeting agendas.  Reduced the query and 
    rendering times of document lists some more through additional 
    prefetch_related().

  * Added result caching for the Person.email() method.

  * Added result caching for a couple of Document methods.

  * Added a missing tag to the sanitizer whitelist (telling lxml's Cleaner 
    to not clean style with style=False is apparently not always enough).  
    Fixes issue #2470.

  * Added a debug import.

  * Updated a django patch to add origin information to query sets.

 -- Henrik Levkowetz   17 Mar 2018 10:42:11 +0000


ietfdb (6.75.1) ietf; urgency=medium

  **Modified HTML upload sanitization**

  Feedback from wgchairs@ietf.org indicated that stripping out all styling
  from uploaded files is too harsh.  This release modifies the sanitization to
  permit