altf4arnold/datatracker
1
0
Fork 0
Code Issues Pull requests Actions 3 Packages Projects Releases Wiki Activity

When doing test-crawling, ignore variations of the 'next=' query arg. (The code ignores other query args if 'next' is given).

Browse source 
- Legacy-Id: 18730
This commit is contained in:
Henrik Levkowetz 2020-12-04 16:04:01 +00:00
parent 9a8c6ae3f4
commit 7ee6bd4fb4
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
bin/test-crawl
View file

@ -83,6 +83,9 @@ def strip_url(url):
fragment_url = re.search("^(.+)#[a-z_.-]+$", url) fragment_url = re.search("^(.+)#[a-z_.-]+$", url)
if fragment_url: if fragment_url:
url = fragment_url.group(1) url = fragment_url.group(1)
next_url = re.search(r"^(.+)\?next=.+$", url)
if next_url:
url = next_url.group(1)
return url return url
def extract_html_urls(content): def extract_html_urls(content):