Commit 29cf863
authored
Build: Skip checking Twitter links in the hydra crawler (#1231)
Twitter pages now do 302-redirects to themselves for users without a
specific cookie set which trips the crawler; avoid checking Twitter
links by abusing the `exclude_scheme_prefixes` option of the crawler.
Since the project only accepts options in a form of a configuration
file, we also need to clone the API repo to provide such a file.1 parent 0cb2745 commit 29cf863
2 files changed
Lines changed: 16 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
27 | 36 | | |
28 | | - | |
| 37 | + | |
0 commit comments