Commit graph

380 commits

Author SHA1 Message Date
Henrik Levkowetz d0583f4f02 Removed the repeat of the error message in the HTTP reason string. Fixes issue #2378.
- Legacy-Id: 14168
2017-09-27 11:03:25 +00:00
Henrik Levkowetz 2c1438c240 Moved unidecode_name from utils.text to person.name.
Modified UserFactory to use a new locale for each new user, instead of the
same locale for a whole test run.  This (almost) ensures the exercise of
code to deal with non-ascii names, something which would not happen if a
locale with ascii names was chosen at the start of a run.

Modified name.initials() to not use non-word characters as initials.

Modified unidecode_name() to do more normalization, to conform to the
conventions used in internet-drafts.

Added saving of the factory-boy random state in order to be able to re-run
a test suite with the same pseudo-random sequence as in a previous failed
run.

Fixed an issue with email formatting in test_api_submit_ok().

Modified the draft author extraction code to deal better with names with
embedded apostrophes.
 - Legacy-Id: 14141
2017-09-20 15:36:30 +00:00
Henrik Levkowetz 33b275b04f Added ietf.utils.text.unidecode_name() and replaced various uses of unidecode() with it, in order to normalize the generation of ascii versions of names, to avoid different practices in space stripping and space normalization in different parts of the code.
- Legacy-Id: 14128
2017-09-17 15:12:18 +00:00
Henrik Levkowetz 6aa2cfca89 Updated tests to match the submission API info page.
- Legacy-Id: 14127
2017-09-17 14:02:52 +00:00
Henrik Levkowetz 8d3c540edd Added submit API instructions and fixed a bug in error handling for the submission validity checkers.
- Legacy-Id: 14126
2017-09-17 13:04:41 +00:00
Henrik Levkowetz 3af2554b2f Added an API for draft submission, at /api/submit. Added an urls.py file under api/ to hold api urls, and moved those from ietf/urls.py. Refactored out many parts of the regular submission forms and functions in submit/forms.py and submit/views.py in order to re-use the appropriate parts for the submission API. Moved support functions to submit/utils.py. Added a new validation errors for missing docName in xml-based submissions. Updated the submission test document templates to use insert additional values. Added failure and success test cases for automated API submissions, and refactored some test utility functions.
- Legacy-Id: 14125
2017-09-16 09:35:42 +00:00
Henrik Levkowetz 135d5d946b Tweaked the submission checker shell invocation code to deal with command lines starting with environment variable settings.
- Legacy-Id: 14107
2017-09-08 12:09:22 +00:00
Henrik Levkowetz a9b259ef26 Changed the message shown when xml file parsing fails during draft submission to include the actual error message from the xml parser. Fixes issue #2353.
- Legacy-Id: 14102
2017-09-06 17:41:29 +00:00
Henrik Levkowetz 95a5400dea Changed the handling of yang validation otput to capture errors messages also when command the exit code is zero.
- Legacy-Id: 14100
2017-09-03 14:40:02 +00:00
Henrik Levkowetz d481019803 Added reverse relationships from drafts to submissions to submission checks to the Json API, to make it easier to find the yang checks for a given document.
- Legacy-Id: 13785
2017-07-07 12:55:30 +00:00
Henrik Levkowetz 25e833444c Added a check for duplicate Yang model names after xym extraction. This prevents running pyang and yanglint on one module, moving it to the module libs, and then failing to run the checks on a second module with the same name, because the file has been moved after the first check.
- Legacy-Id: 13770
2017-07-03 21:24:19 +00:00
Henrik Levkowetz 08f0338555 Tweaked the wording of the unexpected-encoding warning.
- Legacy-Id: 13760
2017-07-03 11:38:03 +00:00
Henrik Levkowetz 274548f1b1 Check the whole draft submission text when checking mime type, and catch decoding errors and raise ValidationError when converting submission form txt file to unicode for Draft().
- Legacy-Id: 13759
2017-07-02 21:32:44 +00:00
Henrik Levkowetz 7e1b09410c Upgraded xym to 0.4.2, with related changes in method calls.
- Legacy-Id: 13741
2017-06-30 15:59:40 +00:00
Henrik Levkowetz 60e9b7451c Included the temporary yang model extraction directory in the list of model librarires given to yanglint, in order to support drafts with models referencing models in the same draft. Fixes issue #2324.
- Legacy-Id: 13736
2017-06-29 21:03:26 +00:00
Henrik Levkowetz 3b7f320c29 Removed the copy of xym copied directly from repository -- the method call we use is too unstable at the moment. Now requiring xym=='0.4'.
- Legacy-Id: 13723
2017-06-28 11:53:09 +00:00
Henrik Levkowetz 13905a2a84 Modified the yang checker to work with both versions 0.3.x and 0.4.x of xym (different signatures for get_extracted_models()).
- Legacy-Id: 13719
2017-06-27 11:37:56 +00:00
Henrik Levkowetz 8eded1d8d2 Return a checker None result with exception message on xym exceptions.
- Legacy-Id: 13707
2017-06-22 15:08:01 +00:00
Henrik Levkowetz 70de94fc48 Variable naming tweak
- Legacy-Id: 13701
2017-06-21 13:12:19 +00:00
Henrik Levkowetz 768ca0b45e Catch bad module names in the Yang checker (they are not currently flagged by xym).
- Legacy-Id: 13700
2017-06-21 13:10:05 +00:00
Henrik Levkowetz 95f966c98b Changed the implementation of Submission.latest_checks() to also return None checks, so it's possible to update a Passed due to no yang modules to a None (no modules to check).
- Legacy-Id: 13698
2017-06-20 21:05:48 +00:00
Henrik Levkowetz 508bf15c6c Work around an issue in xml2rfc
- Legacy-Id: 13682
2017-06-19 19:02:42 +00:00
Henrik Levkowetz b42f1cbeb5 Replaced the use of unaccent.asciify(), which has similar functionality to unidecode.unidecode(). Changed the draft parser to work exclusively with unicode text, which both makes the removal of unaccent easier, and takes us closer to Py35 compatibility. Adjusted callers of the draft parser to send in unicode.
- Legacy-Id: 13673
2017-06-18 18:23:18 +00:00
Henrik Levkowetz 228a68348f Don't deliver spurious yang extraction errors for drafts where no yang modules were found.
- Legacy-Id: 13668
2017-06-17 20:54:52 +00:00
Henrik Levkowetz 98dc0d0092 Only run yanglint and yanglint tests if we have it configured.
- Legacy-Id: 13636
2017-06-15 22:13:43 +00:00
Henrik Levkowetz 4fc2c6f960 Merged in ^/personal/henrik/6.54.2-yang@13630:
Added a new yang checker, 'yanglint', to the existing Yang checker class, in
addition to the existing 'pyang' checker.

Added modal overlay displays showing the yang check results every place the
yin/yang symbol is shown (red or green) to indicate the presencee and result
of yang checks.  Added a Yang Validation: line in the document
meta-information section on the document's page in the datatracker.

Added the result of the xym extaction to the yang check results, to make
extration failures visible.

Added the version of the used xym, pyang, and yanglint commands to the check
results.

Added an action to move successfully extracted and validated modules to the
module library directories immediately on submission.

Added the xym and pyang repositories as svn:external components, rather than
listing them in requirements.txt, as there has been delays of many months
between essential features in the repositories, and an actual release.  We may
get occasional buildbot failures if broken code is pulled in from the
repository, but better that than the functionality failure of severely
outdated componets.

Added a new management command to re-run yang validation for active drafts for
which yang modules were found at submission time, in order to pick up imported
models which may have arrived in the model libraries after the draft's
submission.  Run daily from bin/daily.

Added a table to hold version information for external commands.  The yang
checker output should include the version information of the used checkers,
but seems unnecessary to run each command with its --version switch every
time we check a module...

Added a new management command to collect version information for external
commands on demand.  To be run daily from bin/daily.

Added tests to verify that xym, pyang and yanglint information is available
on the submission confirmation page, and updated the yang module contained in
the test document to validate under both pyang and yanglint.

Updated admin.py and resource.py files as needed.
 - Legacy-Id: 13634
2017-06-15 20:16:48 +00:00
Henrik Levkowetz 207ea51723 Added a test case for invalid yang in a draft submission.
- Legacy-Id: 13633
2017-06-15 19:22:36 +00:00
Henrik Levkowetz d98054c103 Added a new yang checker, 'yanglint', to the existing Yang checker class, in
addition to the existing 'pyang' checker.

Added modal overlay displays showing the yang check results every place the
yin/yang symbol is shown (red or green) to indicate the presencee and result
of yang checks.  Added a Yang Validation: line in the document
meta-information section on the document's page in the datatracker.

Added the result of the xym extaction to the yang check results, to make
extration failures visible.

Added the version of the used xym, pyang, and yanglint commands to the check
results.

Added an action to move successfully extracted and validated modules to the
module library directories immediately on submission.

Added the xym and pyang repositories as svn:external components, rather than
listing them in requirements.txt, as there has been delays of many months
between essential features in the repositories, and an actual release.  We may
get occasional buildbot failures if broken code is pulled in from the
repository, but better that than the functionality failure of severely
outdated componets.

Added a new management command to re-run yang validation for active drafts for
which yang modules were found at submission time, in order to pick up imported
models which may have arrived in the model libraries after the draft's
submission.  Run daily from bin/daily.

Added a table to hold version information for external commands.  The yang
checker output should include the version information of the used checkers,
but seems unnecessary to run each command with its --version switch every
time we check a module...

Added a new management command to collect version information for external
commands on demand.  To be run daily from bin/daily.

Added tests to verify that xym, pyang and yanglint information is available
on the submission confirmation page, and updated the yang module contained in
the test document to validate under both pyang and yanglint.

Updated admin.py and resource.py files as needed.
 - Legacy-Id: 13630
2017-06-15 16:09:28 +00:00
Henrik Levkowetz df2d0571c4 Narrowed a too wide try/except region in order to give more correct error messages. Changed the the strip() application to happen only on extracted author elements that actually have content. Fixes an inability to upload xml-only drafts with missing author country information.
- Legacy-Id: 13574
2017-06-09 20:12:44 +00:00
Henrik Levkowetz fb66f38afd Undid an unintentional adaptation to the xml2rfc 2.6.0 call interface change.
- Legacy-Id: 13556
2017-06-07 00:07:39 +00:00
Henrik Levkowetz a28940fc1e Name/email formatting tweaks, to complement [13536].
- Legacy-Id: 13540
Note: SVN reference [13536] has been migrated to Git commit 55d36e284d
2017-06-06 08:11:31 +00:00
Henrik Levkowetz 55d36e284d Changed the '%s <%s>'%(name,email) idiom in several places to use email.utils.formataddr() instead, to avoid future problems with names that would need quotes. Added comments in 2 places where this change would not be appropriate.
- Legacy-Id: 13536
2017-06-05 22:40:11 +00:00
Henrik Levkowetz a92c2b4ad5 Updated migration numbering to avoid collisions.
- Legacy-Id: 13480
2017-05-31 21:36:08 +00:00
Henrik Levkowetz 76628be3fd Merged in ^/branch/iola/author-stats-r13145 from olau@iola.dk, and fixed some tests in code which moved after the latest merge with trunk. The test suite passes, but the migrations are _not_ ready to run, because of numbering conflicts (again due to code changes on trunk since the latest sync).
- Legacy-Id: 13479
2017-05-31 20:59:26 +00:00
Henrik Levkowetz 4afe0b8830 Added stripping of leading and trailing whitespace from submission data (including email addresses) gleaned from submitted xml. Changed email line parsing to use email.utils.parseaddr() instead of a regex which only would handle unwuoted names (and possibly not utf-8 names) correctly.
- Legacy-Id: 13423
2017-05-25 14:10:42 +00:00
Ole Laursen 5b677dc6ba Merge author stats branch into new branch from trunk
- Legacy-Id: 13159
2017-03-28 14:36:40 +00:00
Henrik Levkowetz 14c748df98 Changed the creation of temporary test dirs to use names which include the test class name, in order to make it possible to run test cases in parallel. Disabled coverage testing when running in parallel (those classes cannot be pickled and distributed to the test runner threads, and also won't pass coverage data back). Tweaked the TestCase.tempdir() method.
- Legacy-Id: 13157
2017-03-27 21:41:18 +00:00
Lars Eggert 8a70e9773d Fix various "RemovedInDjango20Warning" warnings. Commit ready for merge.
- Legacy-Id: 13112
2017-03-25 19:16:45 +00:00
Henrik Levkowetz 814e1315b5 Added a new document tab for some documents, showing htmlized drafts and RFCs
using the htmlization code previously developed for tools.ietf.org.  As the
generation of the htmlized page is a bit too costly to do on the fly for
often-referenced drafts and RFCs, the part of each page which contains the
htmlized document is cached on file with a cache time of 2 weeks.

Changed all links which pointed to the htmlized version on tools to instead
point at the datatracker htmlized document.

Tweaked some URLs which didn't permit retrieval of intermediate-rev-charters.

Narrowed the pattern for document names to disallow dots in names, and
instead explicitly enumerated the few historical draftw with dots in the
name.

Added a file-system cache for the htmlized documents, and specified a
max_entries value for caches, overriding the default 300 entries.

Tweaked the code for new author email entries to provide a time if missing
in an updated entry.

Changed links in various email templates which pointed at tools.ietf.org
pages to instead point at datatracker pages, where appropriate.

Changed the search result rows to provide links to both the current meta-
information document pages (with a (i) info symbol) and to the new htmlized
document pages.
 - Legacy-Id: 13040
2017-03-20 14:08:52 +00:00
Henrik Levkowetz bb5096da8a Added more email validation for draft submission author emails, and blocked some baths that could lead to bad email addresses ('none') being set as document author email addresses.
- Legacy-Id: 13010
2017-03-14 14:07:21 +00:00
Henrik Levkowetz 4daf66d11c Added DocEvent rev= parameter througout the code.
- Legacy-Id: 12978
2017-03-06 15:08:21 +00:00
Henrik Levkowetz 5bb5ca727b Created a special function log.unreachable() to mark code unreachable, and changed a function name from log.affirm() to log.assertion().
- Legacy-Id: 12933
2017-02-28 20:16:52 +00:00
Henrik Levkowetz ffa19c9847 Added a log.affirm() call at the point where we may discover new flawed document author records. Fixed a place where the generation of Person.ascii from Person.name wasn't done well.
- Legacy-Id: 12903
2017-02-23 11:57:29 +00:00
Henrik Levkowetz a5290e143a Made some changes to the missing-email submission handling. This may have fixed the creation of author records containing 'none' email entries, but as I've not been able to reproduce the issue, this is uncertain. Next is to build an email notification assert statement to help capture information if this happens.
- Legacy-Id: 12898
2017-02-22 15:36:28 +00:00
Henrik Levkowetz 27351c4349 Fixed url lookups for doc.views_doc.document_main() to only use dotted-path, and removed the doc_view explicit url name. Fixes a 500 error in /submit/status/NNNN/.
- Legacy-Id: 12897
2017-02-22 11:40:55 +00:00
Henrik Levkowetz 8930d29a8e Merged in Django-1.10 upgrade work from ^/personal/henrik/6.43.1-django-1.10
- Legacy-Id: 12881
2017-02-19 18:18:00 +00:00
Henrik Levkowetz cf4a4b02a7 Reworked the email address handling in order to be able to support non-ascii names as part of email address fields. Reworked the generation of user names in the test suite to generate names from multiple non-ascii locales. Fixes issue #2080.
- Legacy-Id: 12872
2017-02-18 21:50:18 +00:00
Ole Laursen 9d8874d8b1 Add a suggestion to fix the country name when submitting a draft in
case we can't parse it and it's not empty, add page with list of
countries (to be able to refer people to it), add ISO codes as country
aliases - the country alias code is now more intelligent with respect
to case so it's easier to keep these aliases explicitly
 - Legacy-Id: 12862
2017-02-16 17:56:23 +00:00
Ole Laursen b2ff10b0f2 Add support for extracting the country line from the author addresses
to the draft parser (incorporating patch from trunk), store the
extracted country instead of trying to turn it into an ISO country
code, add country and continent name models and add initial data for
those, add helper function for cleaning the countries, add author
country and continent charts, move the affiliation models to
stats/models.py, fix a bunch of bugs.
 - Legacy-Id: 12846
2017-02-15 18:43:57 +00:00
Henrik Levkowetz f6a1fcdaf8 Added more information about submission errors and a full idnits run to the manual post email sent to the secretariat.
- Legacy-Id: 12831
2017-02-13 20:37:47 +00:00