ietfdb (6.81.3) ietf; urgency=medium This is a bugfix release to clear the board for the final GDPR release. From the commit log: * Merged in [15266] from rcross@amsl.com: Updated the blue sheet template. Replaced start time / end time with date. * Merged in [15267] from rcross@amsl.com: Fixed an issue with a decorator on utils.html.remove_tags(). * Tweaked the 'Related documents' part of the Document admin page to more intuitively show the relationships. * Tweaked unidecode_name() to avoid double-quotes in the ascii string. * Merged in [15248] from rjsparks@nostrum.com: Improvements to tests after converting to factories suggested by studying coverage_changes. * Merged in [15247] from rjsparks@nostrum.com: Tweaked the color palette for the agenda editor. * Removed two instances of log.assertion(); we have enough data on these now. * Changed an attribute guard to work with a reverse relationship attribute (using hasattr()). * Fixed a broken mailto link on the account edit page. * Merged in [15238] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_status_change. Repair previous commit errors in ietf.doc.tests_material. * Merged in [15228] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_downref. * Merged in [15227] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_conflict_review. * Merged in [15226] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_charter. * Tweaked unidecode_name() to not produce single-letter ascii surnames from non-ascii codepoints. The unidecode transliteration is in any case somewhat arbitrary, and in most cases a real person will tweak the ascii name of his account. When running tests, however, this tweak avoids some false test failures. And no, it's not simple to fix the draft author-extraction heuristics to deal well with single-letter surnames. * Added a guard against trying to access attributes of a None object. * Merged in [15225] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_draft. -- Henrik Levkowetz 20 Jun 2018 21:47:42 +0000 ietfdb (6.81.2) ietf; urgency=medium This is the third in a small series of releases to address compliance with the European GDPR (General Data Protection Regulation). It provides serialized (machine-readable) access to information associated with a user's account, adds photo and some links to the user account page, and improves the annotation of which information is consent-based on the account page. It adds a warning on login if consent is missing for some personal information, and requests the user to give consent. It also brings in refactoring of a number of test cases to use test object factories, giving improved test performance. * Merged in [15219] from rjsparks@nostrum.com: Use factories for ietf.doc.tests_ballots. Fixed a long-hidden bug in test_edit_verify_permissions where non-existing usernames were used and passed the tests accidently. Added list_email to the GroupFactory. * Merged in [15217], [15205], [15204], [15203], [15202], and [15201] from rjsparks@nostrum.com: Converted ietf.doc.tests to use object Factories. Created draft type specific factories. * Fixed the logic of gathering which personal information needs consent confirmation in the code that requests consent on login. * Added a request to give consent for non-LI information on user login. Addresses issue #2505. * Fixed a bug in a form clean method. * Added a missing related object to the person information export. Related to issue #2501. * Tweaked the language on the user account page. Related to issue #2501. * Added a recursive object to JSON serializer and a view which will let any logged-in user download a JSON serialized copy of the datatracker information related to his person record. Added information about this, and a link, to the account page. Related to issue #2501. * Added a requirement of oauth2client, needed by google-api-python-client but not always automatically pulled in. * Added a link to the overall IETF Statement concerning personal data to the datatracker personal-information page. * Added some text about BOF requests to the important-dates page for the session request opens dates, on request from the IESG. -- Henrik Levkowetz 08 Jun 2018 16:05:15 +0000 ietfdb (6.81.1) ietf; urgency=medium This is the second in a small series of releases to address compliance with the European GDPR (General Data Protection Regulation). This release provides two data migrations to polulate email address origin and other trace information, to make it possible to differentiate between that personal information which is to be treated as Consent-based, that which falls under Legitimate Interest. -- Henrik Levkowetz 29 May 2018 14:38:09 +0000 ietfdb (6.81.0) ietf; urgency=medium This is the first in a small series of releases to address compliance with with the European GDPR (General Data Protection Regulation). This release provides necessary GUI and internal changes, but lacks some admin utilities, and also data migration routines which takes some time to run, and therefore will be packaged in a separate release and applied separately. From the commit log: * Updated the personal information page with reviewed text from legal counsel. Fixes issue #2503. * Removed the Person.address field, which is not being used. This was a legacy from the 2001 perl-based datatracker tables. Fixes issue #2504. * Added django-simple-history and replaced the old (and unused) PersonHistory class with a history=HistoricalRecords() field on Person. Added the needed migrations and changes to admin, resources, and settings. Related to issues #2505 and #2507. * Added a new field name_from_draft to Person, to hold the name field equivalent as captured from drafts, in case name has been modified by the user and we're asked to remove that info under GDPR. Added history for Email, and also an origin field to capture from where we got an email address (draft name, username, meeting registration, etc.) Added a log.assertion() to Email.save() in order to ensure we don't create any email without setting origin. * Added origin information to all places where we create email address entries. * Removed all references to the removed Person.affiliation field. * Added email origin information to some function calls that needed it. * Overwrite earlier email origin when we've picked up the address from a submission. * Added a consent field to the Person model. * Disallow profile changes without consent given. Together with previous commits this fixes issues #2505 and #2507. * Added another category of personal information to the personal-information page, after review of personal information in the code. Completes issue #2501. * Added a dagger at the end of some fields in the account data forms to signify consent-based fields, and made the consent field required. * Simplified the email.origin assignment code for outgoing liaisons. * Fixed a long-standing bug in the liaison.name() code. * Added assingment of the person.name_from_draft field on draft submission. * Updated the edit_profile template with information about consent-based fields. Fixes issue #2502. -- Henrik Levkowetz 29 May 2018 11:10:28 +0000 ietfdb (6.80.1) ietf; urgency=medium This is a bugfix release which also clears the slate for the upcoming GDPR-related release. From the commit log: * Added a meeting helper function update_important_dates(), and a management command update_important_dates to update the important dates for a meeting if the offsets in the MeetingDateName table has been changed. * Added some text to the important dates page for 'opensched', on request from the IESG via Stephanie. * Merged in [15153] from rjsparks@nostrum.com: Changed how the mail utilities handle lists in their override and extra arguments. Added a test for send_mail_preformatted exercising override and extra. * Merged in [15152] from rjsparks@nostrum.com: Send email when the responsible AD for a conflict review is changed. Include the stream manager and stream steering group when the responsible AD changes or when the telechat date changes. Fixes #2497. * Merged in [15151] from rjsparks@nostrum.com: Provide an email group button for WG and RG documents. Fixes #2512. * Merged in [15145] from rjsparks@nostrum.com: Secretariat requested tweak to the request for minutes template. * Updated the non-WG mailing list page to include advertised lists for closed WGs, on request from the secretariat. -- Henrik Levkowetz 24 May 2018 15:53:08 +0000 ietfdb (6.80.0) ietf; urgency=medium This release fixes a number of bugs, and adds a view that makes it easier for the secretariat to generate the meeting minutes request emails. * Merged in [15125] from rjsparks@nostrum.com: Added a view that generates the 'send minutes' email for the secretariat. Fixes #2493. * Merged in [15124] from rjsparks@nostrum.com: Provide a __unicode__ member for ImportantDate. * Merged in [15123] from rjsparks@nostrum.com: Use importantdates for material submisssion cutoff and revision if they exist. Fixes #2514. * Merged in [15119] and [15120] from rjsparks@nostrum.com: Change stream state to replaced when an ISE, IAB, or IRTF document is replaced. Fixes #1963. * Merged in [15114] from rjsparks@nostrum.com: Restored a test that was unintentionally removed. Removed unused templates. Related to #2513. * Merged in [15113] from rjsparks@nostrum.com: Removed the already disabled 'Make RFC' functionality from /secr/drafts. Fixes #2513. Merged in [15112] from rjsparks@nostrum.com: Removed the 'announce' functionality from /secr/drafts. Addresses part of #2513. * Merged in [15111] from rjsparks@nostrum.com: Removed the \'replaces\' functionality from /secr/drafts. Addresses part of #2513. * Merged in [15110] from rjsparks@nostrum.com: Made settings.URL_REGEXPS.name non-greedy. Added a test to exercise the bug at #2429. Fixes #2429. * Fixed an incorrect file name in a log message. -- Henrik Levkowetz 08 May 2018 06:37:38 +0000 ietfdb (6.79.1) ietf; urgency=medium This bugfix release fixes problems with IESG ballot clearing, marks dead code as unreachable for later removal, adds new tests to increase test coverage, and fixes some issues with the test suite. It also tweaks the roles which are permitted to request document publication, loosens the need for trailing slashes for some URLs, and adds a 'Recent drafts' page. * Merged in [15084] from rjsparks@nostrum.com: Corrected the implementation of clear_ballot. Added a test for it. Restored functionality to ballot popups that was missing because a template variable was not passed in. * Added a 'recent drafts' page, showing drafs submitted the last 7 days. * Merged in [15083] from rjsparks@nostrum.com: Removed memoize from latest_event. * Merged in [15080] from rjsparks@nostrum.com: Marked a couple of functions as unreachable. * Reverted a change to UserFactory to avoid nepalese names. The issue is bigger, and needs a different resulution. Changed the url pattern that used \w to match names to instead use a more inclusive regex. * Fixed a PyQuery property access that should have been a method call. * Merged in [15076] from rjsparks@nostrum.com: Added a test for meeting/requests. * Tweaked the IPR factory to be less likely to produce invalid patent info. * Merged in [15075] from rjsparks@nostrum.com: Improved testing for docs_for_ad. * Merged in [15074] from rjsparks@nostrum.com: Simplified docs_in_iesg_process. Added a test for it. * Merged in [15073] from rjsparks@nostrum.com: Rearranged ietf/doc/urls.py to allow matching some special pages under /doc without a trailing slash. Adjusted several of those special URLs to work without the trailing slash. * Added permission for a document shepherd to request publication. Fixes issue #2509. -- Henrik Levkowetz 02 May 2018 10:42:26 +0000 ietfdb (6.79.0) ietf; urgency=medium **IPR patent info field fixes, IESG position API enhancement, and more.** This release enhances the IESG position API to send notification emails on changes, widens some IPR disclosure form fields to accept more text, and fixes a patent application number validation issue for IPR forms. It adds some useful links and a new filtering option to the past meetings page, updates document stream status indications so as to be less prone to misinterpretation, addresses an issue with session request forms, and fixes an issue with I-D expirations and another related to ordering of documents on IESG agendas. From the commit log: * Set file upload permissions to 0644 to avoid problems with photo access. * Increased the length of the patent title in IPR forms * Fixed a typo. Fixes issue #2500 which caused different document ordering on the IESG agenda compared with the IESG agenda documents page. * Increased the size of the patent_notes form field, and tweaked the validate_patent_number validator to accept US application numbers containing a series number and forward slash. * Merged in [15055] from rcross@amsl.com: Fixed an issue where user can make two session request submissions, corrupting data. Also, ignore deleted session requests in edit view. * Merged in [15047] from rcross@amsl.com: Reduced the get_meeting buffer to handle earlier shceduling requests date. * Added origin indication to a group of stats templates to make it easier to find the origin of stats configurations. * Added group links to the past meetings page, and also important dates links to IETF meetings with important-date pages. * Added IETF to past meetings filter pane, in order to be able to filter out interims. * Fixed a problem with the expire-ids code, and added logging to catch future problems. * Modified the IESG position setting API to send email messages about positions set. * Tweaked the wording of coverage_changes output. * Fixed an issue where simplified sorting of DocEvents resulted in some ballot icons not being shown on the IESG agenda documents page. * Changed incorrect use of 404 status code in api to 400. Fixes issue #2498. * Removed what could be a misleading 'RFC' from the stream indication in document search results. -- Henrik Levkowetz 23 Apr 2018 12:28:57 +0000 ietfdb (6.78.0) ietf; urgency=medium **Document search on IRTF state, performance improvements** This release brings document search support for filtering on IRTF states, courtesy of Mat Ford, and further performance improvements to the IESG agenda document page and document search pages. There are also improvements to the testing infrastructure, HTML cleanup, some added API documentation, and a number of bugfixes, as follows: * Merged in [15012] from ford@isoc.org: Added IRTF State option to Additional Search Criteria pulldown. * Included django.utils.html.escape() in the linkify() code, and removed it from various templates, to make sure escape is consistently applied before linkify. Fixes issue #2492. * Tweaked the error return from Document.get_document_content() to be more helpful during development. * Fixed various html issues. * Added more information about personal API keys to the API help page. * Tweaked the test runner to more consistently force the test coverage tests to be run last. * Changed calls to .related_that*() and .relations_that*() to use tuples rather than lists (a slight optimization). * Silenced another deprecation warning from a library. * Reduced the max number of results returned by document searches from 500 to 200, for a better user experience. A faster return and a search term tweak to pinpoint the desired search result is better than a long wait and then maybe having to tweak the search terms anyway. * Use .assertIn(...) in tests instead of .assertTrue(x in y) as the failure messages are much more informative * Changed the signature of some document methods to not accept lists (but still accept tuples) in order to make caching possible. Added @memoize to some methods. * Added another prefetch to search doc query results. * Imposed sort order (most recent docs first) on document search results, in order to retain the most recent docs in case we have to chop the result list due too many hits. * Tweaked the django_cprofile_middleware patch to use a query arg of psort instead of sort, to avoid conflict with app query args 'sort'. * Added back the on-agenda calendar links to document lists, now in an implementation with better performance characteristics. * Updated minimum requirements for bootstrap3, to work with django 1.11 * Fixed an issue where the change of slide naming between IETF 96 and 97 caused the datatracker to not provide correct link to pre-IETF-97 materials. Thanks to tjw.ietf@gmail.com for the heads-up. * Merged in [14970] from rjsparks@nostrum.com: Factories for the ipr app. Coverage of ietf.ipr.views.ajax_search, and a factory-driven test of ietf.ipr.views.edit. * Enhanced debug.py: made the mark() function indicate file and line number it was called from. Added if debug: conditions around some other function actions. * Another round of iesg page optimizations. Reduced additional instances of individual lookups to a single lookup followed by annotation of document objects. Reduced several instances of repeating the same work multiple times (queries for telechat documents, queries for telechat dates). * Fixed a number of tests to deal correctly with model method caching. * Changed the ballot_icon template tag to use an already found ballot if available, rather than doing lookups to find it (twice!). * Added a patch for debug mode, to add a filtering option to the django-cprofile-middleware. * Be more specific with bootstrap's layout classes for small devices in the base template, to avoid the bottom links and footer overlaying the bottom of the content area. Fixes a problem with entering login information on small devices. -- Henrik Levkowetz 10 Apr 2018 16:33:46 +0000 ietfdb (6.77.1) ietf; urgency=medium **Performance improvements** This release brings performance improvements, focused in particular on the IESG agenda documents page. Some of the changes will also provide performance benefits for other pages, in particular pages that lists documents, such as search result pages and WG/RG document lists. All in all, the changes speed up the rendering of /iesg/agenda/documents with a factor of about 2. The page generation time is still roughly proportional to the number of documents on the page, so a large number of documents on future IESG telechats will still give a slow page. More substantial refactoring will probably be needed for further speedups. From the commit log: * Added an index for (doc, type) on DocEvent to speed up latest_event() queries. * Changed the reviewed_by_teams list to provide acronyms directly, to avoid group lookups during template rendering. * Changed telechat_page_count() to accept a list of documents if such a list has already been generated, to avoid double work. * Provided telechat page counts to the template from the view, instead of repeated (costly) filtering through the telechat_page_count template filter in the template, which caused new, redundant document lookups every time. Removed the telechat_page_count template filter altogether, due to its cost. * Tweaked some lookups in fill_in_document_table_attributes(). * Added terms to the select_related() list for documents in IESG agenda_documents(). * Added a get_active_ads() utility function that uses the cache to reduce database access, and replaced database queries in various places with calls to get_active_ads(). * Moved some utility functions so as to reduce the likelihood of import * loops. * Changed some instances of .slug to _id, to avoid unnecessary database lookups. * Added a profiling middleware to development settings, if installed. -- Henrik Levkowetz 02 Apr 2018 11:55:07 +0000 ietfdb (6.77.0) ietf; urgency=medium **Automation API endpoint: /api/meeting/session/video/url** This release provides provides a new API endpoint to let video urls be associated with sessions, intended for use by Meetecho to update the datatracker with sesssion recording links as soon as a recording has been uploaded to the YouTube (or equivalent). It also introduces a new 'Recording Manager' role, used to authorize the API access. A short description of the API is available at https://datatracker.ietf.org/api#session-video-url-api -- Henrik Levkowetz 30 Mar 2018 19:11:56 +0000 ietfdb (6.76.2) ietf; urgency=medium This release fixes a number of blemishes and cleans the slate for merging in a meetecho API branch. * Fixed an issue which could result in server 500 errors for paths like /meeting/99/session/plenary. * Removed superfluous stats pages headers which would be repeated with different layout on the next line, as part of the graphs. * Added a test for IPR edit. * Filled in the IPR edit form, section V, with patent information derived from the patent_info field, the same way we do for the IPR update form. * Added a pip version restriction to requirements.txt, as there seems to be issues with pip 9.0.2 and 9.0.3; they seem to leave dross when down- and upgrading packages. * Don't show links to individual meeting stats pages when there's no content (meeting 71 and earlier currently has no attendance records). * Added meeting dates to the meeting stats overview graph. * Changed the name of a loop variable so as not to obscure a previous local variable. * Don't show future meetings in the meeting stats overview. * Changed settings so that cache keys used by memcached includes the release version, in order to avoid stale and incorrect cache content on new release deployment. Made it easier to pick out cache key composition by normalizing the variable name. Adjusted cache prefix and the composition of some keys to have a unique and consistent cache key prefix. * Use the actual defaults from settings.py when showing the defaults on /accounts/settings/. -- Henrik Levkowetz 28 Mar 2018 20:54:53 +0000 ietfdb (6.76.1) ietf; urgency=medium **IETF 101 code sprint, release 2** This release provides improved meeting attendance statistics (under /stats/meeting/overview/), improved test coverage, additional menu entries, html template fixes, total page count on the Last Call page, the ability for ADs to clear ballots themselves without going through the secretariat, and more. It also changes the default cookie setting for showing or hiding the duplicate left-hand menu column. (If you don't like the new default, use the 'User>Preferences' menu item to revert to the old look.) Again, many thanks to all the contributors! From the commit log: * Fixed some issues with agenda and proceedings code for meetings numbered 64 and below, actualized by the introduction of meeting records for such meetings for stats purposes. * The template filter 'keep_spacing' can interact badly with 'wordwrap'. Removed it from several places, and made text document display more consistent. Fixes issue #2481. * Merged in [14902] from rjsparks@nostrum.com: Test coverage improvements for ietf/secr/telechat/views. * Merged in [14900] from rjsparks@nostrum.com: Improvements to test coverage for ietf/secr/drafts/views.py. * Merged in [14899] from rjsparks@nostrum.com: Cleaner charter factory simplifies test code. * Added submenu to the Statistics menu entry, pointing at the individual stats pages, and entries for NomComs and Releases. * Merged in [14896] from rjsparks@nostrum.com: Improved test coverage for ietf/doc/views_charter.py. * Fixed the base template to have id='content' for the content also when not showing the left-hand menu. * Updated the cookies tests to match the new left_menu default. * Changed the default setting for the datatracker left-hand menu to 'off'. The top menu is always present, and works well. * Tweaked the document list iCalendar links to not show for presentations on days in the past. * Merged in [14884] from adam@nostrum.com: Adding page count to last call page. * Merged in [14883] from adam@nostrum.com: Adding link to documents in last call to the AD dashboard side menu. * Merged in [14849] from ben@nostrum.com: Give ADs the red Clear Ballot button * Merged in [14880] from rjsparks@nostrum.com: Added a Draft test suite. * Merged in [14879] from kivinen@iki.fi: Backfilled IETF meetings starting from 1 and backfilled the attendee numbers for them. Modified the overview attendee statistics page to use different colors for different venue continents. * Tweaked the meeting registratoin import to deal more gracefully with too long names and email addresses, and to fill in the total attendees number for the meeting. * Tweaked mergeready to accept an additional hold-for-merge file outside the repository. -- Henrik Levkowetz 26 Mar 2018 13:38:04 +0000 ietfdb (6.76.0) ietf; urgency=medium **IETF 101 code sprint, release 1** This release contains datatracker bugfixes and enhancements from the IETF-101 Code Sprint. It brings new views to merge duplicate person records, fixes an annoying bug in downref identification and a bunch of links that didn't point to the right document, corrects the mime-type for JSON responses, and brings substantial improvments to test coverage. Many thanks to everyone who contributed! * Merged in [14864] from rjsparks@nostrum.com: Improved test coverage for ietf.meeting.views. * Updated Note Well. Fixes issue #2474. * Tweaked person factory ascii name generation to not include periods inside names. * Tweaked a factory to make duplicate mailing list names less likely in tests * Test data template tweak. * Merged in [14863] from rjsparks@nostrum.com: Improved test coverage for ietf.utils.draft. * Merged in [14862] from rcross@amsl.com: Add view for merge person records. * Merged in [14855] from rjsparks@nostrum.com: Improved test coverage for ietf.meeting.views. * Added a test case for reference extraction. * Merged in [14854] from rjsparks@nostrum.com: Improved test coverage over meeting.views. * Merged in [14851] from housley@vigilsec.com: Improve parser for references in Internet-Drafts. Fixes #2360 * Merged in [14850] from rjsparks@nostrum.com: Improvements to test coverage in ietf.meeting.views * Removed premature save of upload document state (doing it after verifying that the save to file worked instead). Tweaked materials documents revision extraction regexes. Added a test on no change of materials rev after failed upload. Corrected the naming of materials in make_meeting_test_data. Refined the test crawl of meeting materials pages. * Added a couple of meeting material crawls to check that internal material links aren't broken. * Check attribute isn't None before accessing sub-attributes. * Merged in [14845] from rcross@amsl.com: Remove unused roles/forms.py. * Merged in [14844] from adam@nostrum.com: Fixing 'href' field for https://datatracker.ietf.org/meeting//json. * Fixed some materials docname-with-rev regexes that were too narrow. * Merged in [14843] from adam@nostrum.com: Updated (deprecated, unregistered) 'text/json' with 'application/json' everwhere. * Merged in [14842] from rcross@amsl.com: Add check for deleted session when scheduling. Fixes #2450. * Fixed a bug in a RegexValidator error message string interpolation. -- Henrik Levkowetz 21 Mar 2018 10:58:52 +0000 ietfdb (6.75.2) ietf; urgency=medium **Session iCalendar links on document lists** This release adds session ical links to group document lists and document search results, fixes some issues related to upload sanitization and url resolution, improves performance for some pages, and adds and improves some tests in the test suite. From the commit log: * Merged in [14830] from rjsparks@nostrum.com: Simple coverage tests for ietf/secr/drafts/reports.py. * Tweaked the document sanitizer to insert a charset meta tag after sanitization. * Fixed a mistaken change in session urls. * Tweaked a test case to make it easier to get at failures. * Made the materials_document() view function more robust in handling various materials document names. * Gave the 404 message a class which will make it easier to pick out the message during testing. * Updated the meeting materials document view to handle urls with revisions. * Updated the meeting document href patterns in settings.py to reflect that we now have versioned agendas, minutes, and slides. * Added session ical links to document lists (WG docs, search results, etc.) for documents on upcoming meeting agendas. Reduced the query and rendering times of document lists some more through additional prefetch_related(). * Added result caching for the Person.email() method. * Added result caching for a couple of Document methods. * Added a missing tag to the sanitizer whitelist (telling lxml's Cleaner to not clean style with style=False is apparently not always enough). Fixes issue #2470. * Added a debug import. * Updated a django patch to add origin information to query sets. -- Henrik Levkowetz 17 Mar 2018 10:42:11 +0000 ietfdb (6.75.1) ietf; urgency=medium **Modified HTML upload sanitization** Feedback from wgchairs@ietf.org indicated that stripping out all styling from uploaded files is too harsh. This release modifies the sanitization to permit