Commit graph

310 commits

Author SHA1 Message Date
TheBeastLT
ddb452eeed [scraper] fix single anime ep imdb assignment 2021-02-01 12:06:12 +01:00
TheBeastLT
a1809d1848 [scraper] overwrite kickass provider torrents 2021-01-31 23:02:16 +01:00
TheBeastLT
d7313ecb91 [scraper] add torrent galaxy provider 2021-01-31 21:11:42 +01:00
TheBeastLT
9b96742ce8 [scraper] pass torrent link when updating contents 2021-01-31 20:07:56 +01:00
TheBeastLT
3aa36ab681 [scraper] extract isPack to a function 2021-01-31 19:19:56 +01:00
TheBeastLT
44f51d9b5c [scraper] limit number of eps parsed from file 2021-01-28 23:41:01 +01:00
TheBeastLT
16d9f7da2e [scraper] fixes anime specials imdb mapping 2021-01-26 23:27:04 +01:00
TheBeastLT
5074551c41 [scraper] disables tpb seeders update 2021-01-18 14:45:39 +01:00
TheBeastLT
2c3abf242c [scraper] re-enables tpb api seeders 2021-01-17 23:04:04 +01:00
TheBeastLT
b333607209 [scraper] ignores duplicates when creating contents 2021-01-12 12:15:05 +01:00
TheBeastLT
9f45e78f6d [scraper] fixes anime series movie ep assigment 2021-01-11 23:32:12 +01:00
TheBeastLT
4de4801b13 [scraper] update seeders for tpb from trackers not website 2021-01-11 09:57:16 +01:00
TheBeastLT
0522a5e8ab [scraper] adds retry to seeders update 2021-01-11 09:57:16 +01:00
TheBeastLT
cc2ac0ec09 [scraper] popupates anime film/special episode info 2021-01-11 09:57:16 +01:00
TheBeastLT
a297f33aef [scraper] handle zero season in parser 2021-01-11 09:57:16 +01:00
TheBeastLT
f8258c596d [scraper] updates active scrapers 2020-12-31 12:02:40 +01:00
TheBeastLT
21072dc4d7 [scraper] handles torrent files from object without index 2020-12-27 20:04:56 +01:00
TheBeastLT
59d4bfaa66 [scraper] improves concat episodes decomposition condition 2020-12-24 11:51:00 +01:00
TheBeastLT
c3c21e9dc8 [scraper] updates manual functions 2020-12-23 11:28:45 +01:00
TheBeastLT
a1a3fdba61 [scraper] improves nyaapantsu printed error 2020-12-22 11:44:27 +01:00
TheBeastLT
b39e54e2e0 [scraper] encode uri component for kitsu id query 2020-12-19 14:52:39 +01:00
TheBeastLT
0db2cca8fb [scraper] update seeders for not found providers 2020-12-14 12:21:38 +01:00
TheBeastLT
5ab9536296 [scraper] handles special episodes 2020-12-08 11:12:32 +01:00
TheBeastLT
a26bc4a55e [scraper] disables nyaasi scraper 2020-12-07 10:52:57 +01:00
TheBeastLT
a87bfc8e1a [scraper] fixes anime movie imdb id assigment 2020-12-06 17:33:44 +01:00
TheBeastLT
519799b634 [scraper] adds anime specific trackers 2020-12-02 19:30:23 +01:00
TheBeastLT
5b82ac33be [scraper] exclude nyaasi torrents from seeders query 2020-12-02 17:58:07 +01:00
TheBeastLT
cdb221a2cd [scraper] disable nyaasi seeders update request 2020-12-02 10:44:54 +01:00
TheBeastLT
63a0c39a00 [scraper] simplifies nyaa 404 error message 2020-12-01 14:01:26 +01:00
TheBeastLT
b4c217f4e7 [scraper] adds nyaa si scraper 2020-12-01 10:59:29 +01:00
TheBeastLT
705570de53 [scraper] improves subtitle matching 2020-12-01 10:54:08 +01:00
TheBeastLT
c1eb2c3164 [scraper] includes torrent link in nyaa pantsu record 2020-12-01 09:23:39 +01:00
TheBeastLT
6cb3225ccb [scraper] fixes anime movie imdb id assignment 2020-12-01 08:29:59 +01:00
TheBeastLT
b17cfd334b [scraper] fixes anime multi season handling 2020-11-30 22:46:43 +01:00
TheBeastLT
e15b1f3ce9 [scraper] small adjustments 2020-11-29 23:38:28 +01:00
TheBeastLT
f78261a889 [scraper] include underscore in redundant file regex 2020-11-24 19:22:36 +01:00
TheBeastLT
eba2e5f9cf [scraper] updates eztv url 2020-11-20 12:19:03 +01:00
TheBeastLT
a611674231 [scraper] improves anime imdb enriching 2020-11-08 21:22:38 +01:00
TheBeastLT
f0ea03b223 [scraper] filters zero episodes from season episode counts 2020-10-24 12:18:19 +02:00
TheBeastLT
0a289d9188 [scraper] adds anime extra filtering 2020-10-22 19:07:12 +02:00
TheBeastLT
10e35c1393 [scraper] updates seeders repository condition 2020-10-19 21:28:24 +02:00
TheBeastLT
225bdc7503 [scraper] increases eztv timeout 2020-10-18 12:50:19 +02:00
TheBeastLT
724676e973 [scraper] increase tpb scraper frequency 2020-10-17 22:57:51 +02:00
TheBeastLT
f893f7c01c [scraper] increases eztv timeout 2020-10-17 22:57:22 +02:00
TheBeastLT
f795f5315c [scraper] adds nyaa pantsu scraper 2020-10-17 22:26:15 +02:00
TheBeastLT
723bf38eae [scraper] fixes 5.5 sound detection for movies 2020-10-14 13:24:51 +02:00
TheBeastLT
2be400baba [scraper] do not pass date year to name-to-imdb 2020-10-03 14:35:06 +02:00
TheBeastLT
ad7ce57c36 [scraper] fixes 1337x overwrite behaviour 2020-10-02 23:06:36 +02:00
TheBeastLT
31e3b0bf5f [scraper] check if episode imdbid fro eztv 2020-10-02 14:06:51 +02:00
TheBeastLT
5426707dd9 [scraper] reduces number of calls to 1337x 2020-10-02 12:49:45 +02:00
TheBeastLT
42714d19a9 [scraper] fails google-it on empty results 2020-10-02 12:21:01 +02:00
TheBeastLT
0f86b77a24 [scraper] adds response and read timeouts fro eztv 2020-09-21 11:08:20 +02:00
TheBeastLT
1e0e99a903 [scraper] disables eztv seeders scrapeing 2020-09-20 19:24:09 +02:00
TheBeastLT
03852d76f1 [scraper] fixes to not use eztv imdb id for talk shows 2020-09-19 20:05:51 +02:00
TheBeastLT
99bfbb058a [scraper] moves timeout to provider seeders update level 2020-09-18 21:22:36 +02:00
TheBeastLT
a7b23a1313 [scraper] disables imdb id seeders update for eztv 2020-09-18 21:19:42 +02:00
TheBeastLT
36ed669e1f [scraper] improves torrent file indexing for anime 2020-09-17 21:37:07 +02:00
TheBeastLT
87161a770b [scraper] updates metadata mapping 2020-09-17 21:34:13 +02:00
TheBeastLT
d9eb5f3b35 [scraper] removes proxy for katcr 2020-09-17 21:33:38 +02:00
TheBeastLT
e9b88dd0ad [scraper] removes proxy for rarbg and updates agent name 2020-09-17 21:33:16 +02:00
TheBeastLT
2debc4ee8c [scraper] adds timeout to seeders updating 2020-09-17 21:28:47 +02:00
TheBeastLT
3a3e9a9d73 [scrapper] updates hs mapping 2020-08-27 10:11:47 +02:00
TheBeastLT
12dc20c144 [scrapper] fixes rarbg api error handling 2020-08-13 15:17:56 +02:00
TheBeastLT
ecb578bd8c [scraper] updates hs mapping 2020-07-29 10:26:17 +02:00
TheBeastLT
f886e2a078 [scraper] updates horriblesubs mapping 2020-07-20 18:45:55 +02:00
TheBeastLT
0361dc8096 [scraper] adds divx video extension 2020-05-30 14:53:13 +02:00
TheBeastLT
f0939f9943 [scraper] implements custom rargb api class 2020-05-26 23:22:08 +02:00
TheBeastLT
975673101f [scraper] filter featurettes 2020-05-20 10:54:10 +02:00
TheBeastLT
c5271c8490 [scraper] check if id is number when getting metadata 2020-05-20 10:36:06 +02:00
TheBeastLT
6c502b714a [scraper] pass kitsuId when opening contents 2020-05-19 12:36:39 +02:00
TheBeastLT
2a77abbf8d [scraper] updates featurette regex 2020-05-15 12:28:34 +02:00
TheBeastLT
d9203a88cc [scraper] disable rarbg scraper for now 2020-05-13 22:05:08 +02:00
TheBeastLT
1a7a965562 [scraper] adds 3d video extension 2020-05-13 10:48:08 +02:00
TheBeastLT
7d7f9b65a2 [scraper] fixes is disk method 2020-05-13 10:12:14 +02:00
TheBeastLT
11e20beeac [scraper] fixes trackers cache method ttl 2020-05-12 18:07:57 +02:00
TheBeastLT
f4740694a0 [scraper] detects and handles disk torrents 2020-05-12 09:15:26 +02:00
TheBeastLT
51a9d2401d [scraper] bumps rarbg-api version 2020-05-11 10:40:46 +02:00
TheBeastLT
11f3c08d17 [scraper] updates featurette regex 2020-05-11 10:36:25 +02:00
TheBeastLT
1a4a06b8e3 [scraper] improves cd detection regex 2020-05-09 21:50:41 +02:00
TheBeastLT
51144e3833 [scraper] detects 2cd movies as single one 2020-05-08 21:03:51 +02:00
TheBeastLT
0d98b6b3ae [scraper] includes Movies/x265/1080 category for rarbg 2020-05-07 13:14:05 +02:00
TheBeastLT
3b2617cd3c [scraper] fixes featurette and redundant files detection 2020-05-07 10:10:21 +02:00
TheBeastLT
6bed8bad34 [scraper] clears featurette video files info 2020-05-06 18:04:12 +02:00
TheBeastLT
9ce3f72e5d [scraper] checks for torrent contents when updating existing torrent 2020-05-06 11:59:53 +02:00
TheBeastLT
0ad7324a6a [scraper] updates samples and redundant video filtering 2020-05-06 08:34:45 +02:00
TheBeastLT
6c9313a0f4 [scraper] reverts json subtitlesIndexes column 2020-05-04 22:15:58 +02:00
TheBeastLT
d47cf4304e [scraper] assigns subtitle indexes directly to file table as json array 2020-05-04 21:46:39 +02:00
TheBeastLT
46b75f0466 [scraper] fixes subtitles assignment 2020-05-04 18:43:11 +02:00
TheBeastLT
4b31aa9024 [scraper] excludes nyaa pantsu from seeders update for now 2020-05-03 23:07:20 +02:00
TheBeastLT
129a9ed29d [scraper] retrieves update seeders torrent order by seeders and upload date 2020-05-03 22:54:49 +02:00
TheBeastLT
1be9a5f598 [scraper] moves 1337x up in order 2020-05-03 22:54:08 +02:00
TheBeastLT
9b1982abfe [scraper] limits pages per category for 1337x 2020-05-02 22:40:51 +02:00
TheBeastLT
9123c1ffd8 [scraper] reduces concurrent entry processing limit 2020-05-02 22:20:19 +02:00
TheBeastLT
452f29896c [scraper] match subtitles by equals titles instead of includes 2020-05-02 20:21:42 +02:00
TheBeastLT
4507b1bd1d [scraper] assign subtitles to videos 2020-05-02 19:48:02 +02:00
TheBeastLT
6bd8a05ba4 [scraper] upsert subtitles when updating existing file 2020-05-02 18:08:36 +02:00
TheBeastLT
2bb0570194 [scraper] adds proxy for kat and 1337x 2020-05-02 17:35:50 +02:00
TheBeastLT
cd5fac2f60 [scraper] upsert existing video 2020-05-02 17:12:28 +02:00
TheBeastLT
14b2f6b1a2 [scraper] saves opened torrent contents and videos and subs separately 2020-05-02 10:16:03 +02:00
TheBeastLT
4cf7bfb04a [scraper] create subtitle database model 2020-04-30 11:29:23 +02:00
TheBeastLT
9dba42ebfe [scraper] fixes hs api to not use redirects due to node hotfix 2020-04-30 09:43:29 +02:00
TheBeastLT
50aa4d4f1e [scraper] updates horriblesubs mapping 2020-04-28 18:33:15 +02:00
TheBeastLT
3d31d5d735 [addon] adds m4v video format 2020-04-28 13:37:07 +02:00
TheBeastLT
17a2a14f49 [scraper] escape amp html 2020-04-28 13:37:03 +02:00
TheBeastLT
529f5ba76c [scraper] simplify update seeders query 2020-04-28 09:29:40 +02:00
TheBeastLT
95c2893523 [scraper] track updated seeders statistics 2020-04-27 18:24:12 +02:00
TheBeastLT
a6f3e85c34 [scraper] adds pagination to new tpb browse api 2020-04-27 12:22:43 +02:00
TheBeastLT
33ecee4a8a [scraper] find imdbids for movie collection in seq 2020-04-27 11:01:42 +02:00
TheBeastLT
ee24dbebb8 [scraper] handles non json tpb request results 2020-04-26 11:42:24 +02:00
TheBeastLT
0be7dc4638 [scraper] cleans up repository methods 2020-04-25 14:13:19 +02:00
TheBeastLT
10615e4aed [scraper] updates 1337x search api to extend results 2020-04-25 10:37:21 +02:00
TheBeastLT
43c2e737f8 [scraper] uses bittorrent-tracker for force update seeders 2020-04-25 10:28:22 +02:00
TheBeastLT
dc35b4e982 [scraper] disable swarm seeders update for now 2020-04-24 20:31:36 +02:00
TheBeastLT
2434482d99 [scraper] increase tpb scrape frequeny 2020-04-24 09:31:22 +02:00
TheBeastLT
eb2211ddcb [scraper] add jitter for eztv retries 2020-04-23 23:22:11 +02:00
TheBeastLT
5e25bac52a [scraper] fixes eztv api search function 2020-04-23 21:19:35 +02:00
TheBeastLT
1387f38074 [scraper] refactor tpb scraper to use new api 2020-04-23 21:08:23 +02:00
TheBeastLT
d0e0d7b581 [scraper] fixes seeders update for imdb search providers 2020-04-23 20:49:04 +02:00
TheBeastLT
a6b3b164aa [scraper] schedules scrapers using cron instead of delay 2020-04-23 20:08:23 +02:00
TheBeastLT
aa0bf8da44 [scraper] fixes seeders updating if provider not found 2020-04-23 17:08:18 +02:00
TheBeastLT
9ea3932af1 [scraper] rework scraper scheduling and added seeders updating 2020-04-23 16:46:04 +02:00
TheBeastLT
d01e2c9b35 [scraper] updates rarbg dump scraper 2020-04-23 15:15:52 +02:00
TheBeastLT
3299bc2d06 [scraper] fixes periodic hs all shows scraping with disabled seeders updating 2020-04-17 19:05:55 +02:00
TheBeastLT
da2c1e5bbe [scraper] update seeders disregarding provider 2020-04-17 09:13:13 +02:00
TheBeastLT
adc78fdea7 [scraper] adds retries for rarbg 2020-04-15 16:47:53 +02:00
TheBeastLT
6d0763ba20 [scraper] include languages from tpb 2020-04-15 14:46:26 +02:00
TheBeastLT
fd7057f3dc [scraper] fixes eztv latest scraper method 2020-04-14 11:05:01 +02:00
TheBeastLT
7e959609ea [scraper] adds eztv scraper 2020-04-14 08:29:08 +02:00
TheBeastLT
a58843d75f [scraper] reduces tpb scraped pages 2020-04-13 22:48:35 +02:00
TheBeastLT
6bfba11692 [scraper] updates includes api extracted languages 2020-04-13 22:46:17 +02:00
TheBeastLT
9444b7551e [scraper] updates multiple files torrent handling 2020-04-13 22:45:18 +02:00
TheBeastLT
ac0c0b041f [scraper] adds yts scraper 2020-04-13 10:06:06 +02:00
TheBeastLT
59b7536cde [scraper] updates seeders only when providers the same 2020-04-12 21:00:07 +02:00
TheBeastLT
31e3970272 [scraper] updates hs scraper to update seeders for existing latest entries 2020-04-11 17:57:30 +02:00
TheBeastLT
f3952dcc3e [scraper] updates hs mapping 2020-04-11 16:37:24 +02:00
TheBeastLT
cf7b4411d2 [scraper] removes moch resolver from scraper 2020-04-09 13:35:11 +02:00
TheBeastLT
9ea008572d [scraper] updates hs mapping 2020-04-08 18:30:05 +02:00
TheBeastLT
ce0e330172 [scraper] adds update seeders cron 2020-04-08 16:06:02 +02:00
TheBeastLT
d3cc8c6b57 [scraper] updates hs mapping 2020-04-08 10:35:07 +02:00
TheBeastLT
6e911150f4 [scraper] updates hs mapping 2020-04-05 12:36:34 +02:00
TheBeastLT
2eefdc76ec [scraper] use same cached proxy for rd requests 2020-04-04 16:52:12 +02:00
TheBeastLT
760c577ca7 [scraper] removes proxy from kat 2020-04-03 16:28:54 +02:00
TheBeastLT
bccfe005af [scraper] removes get files from cache method 2020-04-03 14:39:57 +02:00
TheBeastLT
7dac31ae61 [scraper] fixes torrent size update method for horriblesubs 2020-04-03 14:39:23 +02:00
TheBeastLT
201ec02260 [scraper] updates created entry logging format 2020-04-03 14:17:05 +02:00
TheBeastLT
a8e76cd46e [scraper] fixes horriblesubs spread operator object 2020-04-03 14:10:31 +02:00
TheBeastLT
95505885f6 [scraper] renames cache method 2020-04-03 13:59:15 +02:00
TheBeastLT
5b07692e57 [scraper] reduce resolved url cache time 2020-04-03 13:41:52 +02:00
TheBeastLT
33d9921179 [scraper] cache rd options per ip 2020-04-03 13:27:26 +02:00
TheBeastLT
d2eb8d04c0 [scraper] updates horriblesubs mapping 2020-04-03 12:25:19 +02:00