Creepso/torrentio-scraper
1
0
Fork 0
mirror of https://github.com/TheBeastLT/torrentio-scraper.git synced 2026-04-20 18:22:03 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
516 commits 13 branches 0 tags 16 MiB
Commit graph

261 commits

Author SHA1 Message Date
TheBeastLT
5426707dd9 [scraper] reduces number of calls to 1337x 2020-10-02 12:49:45 +02:00
TheBeastLT
42714d19a9 [scraper] fails google-it on empty results 2020-10-02 12:21:01 +02:00
TheBeastLT
0f86b77a24 [scraper] adds response and read timeouts fro eztv 2020-09-21 11:08:20 +02:00
TheBeastLT
1e0e99a903 [scraper] disables eztv seeders scrapeing 2020-09-20 19:24:09 +02:00
TheBeastLT
03852d76f1 [scraper] fixes to not use eztv imdb id for talk shows 2020-09-19 20:05:51 +02:00
TheBeastLT
99bfbb058a [scraper] moves timeout to provider seeders update level 2020-09-18 21:22:36 +02:00
TheBeastLT
a7b23a1313 [scraper] disables imdb id seeders update for eztv 2020-09-18 21:19:42 +02:00
TheBeastLT
36ed669e1f [scraper] improves torrent file indexing for anime 2020-09-17 21:37:07 +02:00
TheBeastLT
87161a770b [scraper] updates metadata mapping 2020-09-17 21:34:13 +02:00
TheBeastLT
d9eb5f3b35 [scraper] removes proxy for katcr 2020-09-17 21:33:38 +02:00
TheBeastLT
e9b88dd0ad [scraper] removes proxy for rarbg and updates agent name 2020-09-17 21:33:16 +02:00
TheBeastLT
2debc4ee8c [scraper] adds timeout to seeders updating 2020-09-17 21:28:47 +02:00
TheBeastLT
3a3e9a9d73 [scrapper] updates hs mapping 2020-08-27 10:11:47 +02:00
TheBeastLT
12dc20c144 [scrapper] fixes rarbg api error handling 2020-08-13 15:17:56 +02:00
TheBeastLT
ecb578bd8c [scraper] updates hs mapping 2020-07-29 10:26:17 +02:00
TheBeastLT
f886e2a078 [scraper] updates horriblesubs mapping 2020-07-20 18:45:55 +02:00
TheBeastLT
0361dc8096 [scraper] adds divx video extension 2020-05-30 14:53:13 +02:00
TheBeastLT
f0939f9943 [scraper] implements custom rargb api class 2020-05-26 23:22:08 +02:00
TheBeastLT
975673101f [scraper] filter featurettes 2020-05-20 10:54:10 +02:00
TheBeastLT
c5271c8490 [scraper] check if id is number when getting metadata 2020-05-20 10:36:06 +02:00
TheBeastLT
6c502b714a [scraper] pass kitsuId when opening contents 2020-05-19 12:36:39 +02:00
TheBeastLT
2a77abbf8d [scraper] updates featurette regex 2020-05-15 12:28:34 +02:00
TheBeastLT
d9203a88cc [scraper] disable rarbg scraper for now 2020-05-13 22:05:08 +02:00
TheBeastLT
1a7a965562 [scraper] adds 3d video extension 2020-05-13 10:48:08 +02:00
TheBeastLT
7d7f9b65a2 [scraper] fixes is disk method 2020-05-13 10:12:14 +02:00
TheBeastLT
11e20beeac [scraper] fixes trackers cache method ttl 2020-05-12 18:07:57 +02:00
TheBeastLT
f4740694a0 [scraper] detects and handles disk torrents 2020-05-12 09:15:26 +02:00
TheBeastLT
51a9d2401d [scraper] bumps rarbg-api version 2020-05-11 10:40:46 +02:00
TheBeastLT
11f3c08d17 [scraper] updates featurette regex 2020-05-11 10:36:25 +02:00
TheBeastLT
1a4a06b8e3 [scraper] improves cd detection regex 2020-05-09 21:50:41 +02:00
TheBeastLT
51144e3833 [scraper] detects 2cd movies as single one 2020-05-08 21:03:51 +02:00
TheBeastLT
0d98b6b3ae [scraper] includes Movies/x265/1080 category for rarbg 2020-05-07 13:14:05 +02:00
TheBeastLT
3b2617cd3c [scraper] fixes featurette and redundant files detection 2020-05-07 10:10:21 +02:00
TheBeastLT
6bed8bad34 [scraper] clears featurette video files info 2020-05-06 18:04:12 +02:00
TheBeastLT
9ce3f72e5d [scraper] checks for torrent contents when updating existing torrent 2020-05-06 11:59:53 +02:00
TheBeastLT
0ad7324a6a [scraper] updates samples and redundant video filtering 2020-05-06 08:34:45 +02:00
TheBeastLT
6c9313a0f4 [scraper] reverts json subtitlesIndexes column 2020-05-04 22:15:58 +02:00
TheBeastLT
d47cf4304e [scraper] assigns subtitle indexes directly to file table as json array 2020-05-04 21:46:39 +02:00
TheBeastLT
46b75f0466 [scraper] fixes subtitles assignment 2020-05-04 18:43:11 +02:00
TheBeastLT
4b31aa9024 [scraper] excludes nyaa pantsu from seeders update for now 2020-05-03 23:07:20 +02:00
TheBeastLT
129a9ed29d [scraper] retrieves update seeders torrent order by seeders and upload date 2020-05-03 22:54:49 +02:00
TheBeastLT
1be9a5f598 [scraper] moves 1337x up in order 2020-05-03 22:54:08 +02:00
TheBeastLT
9b1982abfe [scraper] limits pages per category for 1337x 2020-05-02 22:40:51 +02:00
TheBeastLT
9123c1ffd8 [scraper] reduces concurrent entry processing limit 2020-05-02 22:20:19 +02:00
TheBeastLT
452f29896c [scraper] match subtitles by equals titles instead of includes 2020-05-02 20:21:42 +02:00
TheBeastLT
4507b1bd1d [scraper] assign subtitles to videos 2020-05-02 19:48:02 +02:00
TheBeastLT
6bd8a05ba4 [scraper] upsert subtitles when updating existing file 2020-05-02 18:08:36 +02:00
TheBeastLT
2bb0570194 [scraper] adds proxy for kat and 1337x 2020-05-02 17:35:50 +02:00
TheBeastLT
cd5fac2f60 [scraper] upsert existing video 2020-05-02 17:12:28 +02:00
TheBeastLT
14b2f6b1a2 [scraper] saves opened torrent contents and videos and subs separately 2020-05-02 10:16:03 +02:00
TheBeastLT
4cf7bfb04a [scraper] create subtitle database model 2020-04-30 11:29:23 +02:00
TheBeastLT
9dba42ebfe [scraper] fixes hs api to not use redirects due to node hotfix 2020-04-30 09:43:29 +02:00
TheBeastLT
50aa4d4f1e [scraper] updates horriblesubs mapping 2020-04-28 18:33:15 +02:00
TheBeastLT
3d31d5d735 [addon] adds m4v video format 2020-04-28 13:37:07 +02:00
TheBeastLT
17a2a14f49 [scraper] escape amp html 2020-04-28 13:37:03 +02:00
TheBeastLT
529f5ba76c [scraper] simplify update seeders query 2020-04-28 09:29:40 +02:00
TheBeastLT
95c2893523 [scraper] track updated seeders statistics 2020-04-27 18:24:12 +02:00
TheBeastLT
a6f3e85c34 [scraper] adds pagination to new tpb browse api 2020-04-27 12:22:43 +02:00
TheBeastLT
33ecee4a8a [scraper] find imdbids for movie collection in seq 2020-04-27 11:01:42 +02:00
TheBeastLT
ee24dbebb8 [scraper] handles non json tpb request results 2020-04-26 11:42:24 +02:00
TheBeastLT
0be7dc4638 [scraper] cleans up repository methods 2020-04-25 14:13:19 +02:00
TheBeastLT
10615e4aed [scraper] updates 1337x search api to extend results 2020-04-25 10:37:21 +02:00
TheBeastLT
43c2e737f8 [scraper] uses bittorrent-tracker for force update seeders 2020-04-25 10:28:22 +02:00
TheBeastLT
dc35b4e982 [scraper] disable swarm seeders update for now 2020-04-24 20:31:36 +02:00
TheBeastLT
2434482d99 [scraper] increase tpb scrape frequeny 2020-04-24 09:31:22 +02:00
TheBeastLT
eb2211ddcb [scraper] add jitter for eztv retries 2020-04-23 23:22:11 +02:00
TheBeastLT
5e25bac52a [scraper] fixes eztv api search function 2020-04-23 21:19:35 +02:00
TheBeastLT
1387f38074 [scraper] refactor tpb scraper to use new api 2020-04-23 21:08:23 +02:00
TheBeastLT
d0e0d7b581 [scraper] fixes seeders update for imdb search providers 2020-04-23 20:49:04 +02:00
TheBeastLT
a6b3b164aa [scraper] schedules scrapers using cron instead of delay 2020-04-23 20:08:23 +02:00
TheBeastLT
aa0bf8da44 [scraper] fixes seeders updating if provider not found 2020-04-23 17:08:18 +02:00
TheBeastLT
9ea3932af1 [scraper] rework scraper scheduling and added seeders updating 2020-04-23 16:46:04 +02:00
TheBeastLT
d01e2c9b35 [scraper] updates rarbg dump scraper 2020-04-23 15:15:52 +02:00
TheBeastLT
3299bc2d06 [scraper] fixes periodic hs all shows scraping with disabled seeders updating 2020-04-17 19:05:55 +02:00
TheBeastLT
da2c1e5bbe [scraper] update seeders disregarding provider 2020-04-17 09:13:13 +02:00
TheBeastLT
adc78fdea7 [scraper] adds retries for rarbg 2020-04-15 16:47:53 +02:00
TheBeastLT
6d0763ba20 [scraper] include languages from tpb 2020-04-15 14:46:26 +02:00
TheBeastLT
fd7057f3dc [scraper] fixes eztv latest scraper method 2020-04-14 11:05:01 +02:00
TheBeastLT
7e959609ea [scraper] adds eztv scraper 2020-04-14 08:29:08 +02:00
TheBeastLT
a58843d75f [scraper] reduces tpb scraped pages 2020-04-13 22:48:35 +02:00
TheBeastLT
6bfba11692 [scraper] updates includes api extracted languages 2020-04-13 22:46:17 +02:00
TheBeastLT
9444b7551e [scraper] updates multiple files torrent handling 2020-04-13 22:45:18 +02:00
TheBeastLT
ac0c0b041f [scraper] adds yts scraper 2020-04-13 10:06:06 +02:00
TheBeastLT
59b7536cde [scraper] updates seeders only when providers the same 2020-04-12 21:00:07 +02:00
TheBeastLT
31e3970272 [scraper] updates hs scraper to update seeders for existing latest entries 2020-04-11 17:57:30 +02:00
TheBeastLT
f3952dcc3e [scraper] updates hs mapping 2020-04-11 16:37:24 +02:00
TheBeastLT
cf7b4411d2 [scraper] removes moch resolver from scraper 2020-04-09 13:35:11 +02:00
TheBeastLT
9ea008572d [scraper] updates hs mapping 2020-04-08 18:30:05 +02:00
TheBeastLT
ce0e330172 [scraper] adds update seeders cron 2020-04-08 16:06:02 +02:00
TheBeastLT
d3cc8c6b57 [scraper] updates hs mapping 2020-04-08 10:35:07 +02:00
TheBeastLT
6e911150f4 [scraper] updates hs mapping 2020-04-05 12:36:34 +02:00
TheBeastLT
2eefdc76ec [scraper] use same cached proxy for rd requests 2020-04-04 16:52:12 +02:00
TheBeastLT
760c577ca7 [scraper] removes proxy from kat 2020-04-03 16:28:54 +02:00
TheBeastLT
bccfe005af [scraper] removes get files from cache method 2020-04-03 14:39:57 +02:00
TheBeastLT
7dac31ae61 [scraper] fixes torrent size update method for horriblesubs 2020-04-03 14:39:23 +02:00
TheBeastLT
201ec02260 [scraper] updates created entry logging format 2020-04-03 14:17:05 +02:00
TheBeastLT
a8e76cd46e [scraper] fixes horriblesubs spread operator object 2020-04-03 14:10:31 +02:00
TheBeastLT
95505885f6 [scraper] renames cache method 2020-04-03 13:59:15 +02:00
TheBeastLT
5b07692e57 [scraper] reduce resolved url cache time 2020-04-03 13:41:52 +02:00
TheBeastLT
33d9921179 [scraper] cache rd options per ip 2020-04-03 13:27:26 +02:00
TheBeastLT
d2eb8d04c0 [scraper] updates horriblesubs mapping 2020-04-03 12:25:19 +02:00
TheBeastLT
c2e42634b7 [scraper] uses forked rd library 2020-04-03 10:28:49 +02:00
TheBeastLT
2833c9d3a2 [scraper] adds helper class for proxy and user agent 2020-04-02 22:47:52 +02:00
TheBeastLT
db78f5873b [scraper] get torrent size for horriblesubs 2020-04-02 20:58:08 +02:00
TheBeastLT
1719a199b0 [scraper] remove updated seeders logging 2020-03-30 16:16:45 +02:00
TheBeastLT
97cbe5d6c1 [scraper] fixes update seeders method 2020-03-30 14:48:22 +02:00
TheBeastLT
04b3d8a774 [scraper] populate kitsuId for movies in series torrents 2020-03-29 12:20:27 +02:00
TheBeastLT
951add6851 [scraper] updates horriblesubs mapping 2020-03-29 11:47:55 +02:00
TheBeastLT
ee8897a088 [scraper] do not escape hyphen when searching for id 2020-03-28 23:58:31 +01:00
TheBeastLT
34486230ea [scraper] updates movie title to path 2020-03-28 23:18:51 +01:00
TheBeastLT
694f234cb2 [scraper] fixes id assignment fro metadata 2020-03-28 13:08:03 +01:00
TheBeastLT
e79064019e [scraper] fixes tpb browse request url 2020-03-27 19:51:35 +01:00
TheBeastLT
0e802c74d2 [scraper] check for if new episode when decomposing 2020-03-26 18:19:14 +01:00
TheBeastLT
184f7e23b0 [scraper] adds google-it for imdbid search 2020-03-23 20:41:07 +01:00
TheBeastLT
ed6f0969da [scraper] removes adding failed idb entry to db 2020-03-23 16:13:03 +01:00
TheBeastLT
fd3fd9c2fe [scraper] use html escape lib 2020-03-23 16:09:56 +01:00
TheBeastLT
cd9530bfa7 [scraper] updates kitsu addon url 2020-03-23 14:16:28 +01:00
TheBeastLT
83ef6e09af [scraper] fixes academia horriblesubs mapping 2020-03-21 12:43:20 +01:00
TheBeastLT
c164000bc2 [scraper] moves resolver logging to actual method 2020-03-20 16:38:19 +01:00
TheBeastLT
5f429ebdc0 [scraper] do not use imdbId for series from tpb 2020-03-20 14:25:05 +01:00
TheBeastLT
084744b35b [scraper] moves moch url resolving to scraper 2020-03-19 12:21:10 +01:00
TheBeastLT
186a8457cd [scraper] handle movie range as movie collection 2020-03-18 14:31:37 +01:00
TheBeastLT
b08d46efd4 [scraper] fixes has movies regex 2020-03-17 23:55:45 +01:00
TheBeastLT
4de9f3f013 [scraper] fixes cache option deprecation 2020-03-17 19:40:19 +01:00
TheBeastLT
d26764fce9 [scraper] move is-video function to a file 2020-03-17 19:28:43 +01:00
TheBeastLT
7948f4877e [scraper] fixes cron expression 2020-03-17 08:28:41 +01:00
TheBeastLT
8e67c5f40e [scraper] updates horriblesubs mapping 2020-03-17 08:25:31 +01:00
TheBeastLT
7e7e46a5f3 [scraper] updates tpb proxies 2020-03-16 14:48:57 +01:00
TheBeastLT
b05d4dee33 [scraper] retrieve latest entries in sequence 2020-03-16 10:53:36 +01:00
TheBeastLT
b6e2fceeee [scraper] fix null imdbid search result 2020-03-16 09:34:49 +01:00
TheBeastLT
70c279d7d7 [addon] adds filtering based on provider 2020-03-15 23:49:54 +01:00
TheBeastLT
6c56d76f98 [scraper] overwrite torrent type if differs from metadata type 2020-03-15 20:46:41 +01:00
TheBeastLT
5e4ed626e9 [scraper] includes 1337x sort api option 2020-03-15 20:45:48 +01:00
TheBeastLT
dba04c8a69 [scraper] retry different resource type for cinemeta if not found 2020-03-15 11:55:19 +01:00
TheBeastLT
e904dea5e6 [scraper] fixes 1337x size parsing 2020-03-15 11:54:55 +01:00
TheBeastLT
2ea787c0c7 [scraper] updates tpb dum scraper 2020-03-15 11:24:18 +01:00
TheBeastLT
3ca4479d8a [scraper] add html entities escape 2020-03-14 22:54:44 +01:00
TheBeastLT
326a07b82e [scraper] updates scrapers and unique index 2020-03-14 22:25:51 +01:00
TheBeastLT
1efaa0451c [scraper] check existing torrent for 1337x once infohash is defined 2020-03-13 23:15:25 +01:00
TheBeastLT
b5c58c92f0 [scraper] adds year to kitsuId search query 2020-03-13 22:47:43 +01:00
TheBeastLT
7535a4ea55 [scraper] fix season shortcut assignment for anime 2020-03-13 22:44:27 +01:00
TheBeastLT
542c96a2c3 [scraper] updates scapers limiter usage 2020-03-13 22:24:22 +01:00
TheBeastLT
c09d7cf566 [scraper] filter non mapped horriblesubs episodes 2020-03-13 18:38:57 +01:00
TheBeastLT
21338b3aee [scraper] date handling 2020-03-13 15:46:39 +01:00
TheBeastLT
2b73b56106 [scraper] improves title based decomposer 2020-03-13 15:46:39 +01:00
TheBeastLT
a47495281d [scraper] removes unique index on files 2020-03-13 15:46:39 +01:00
TheBeastLT
a2b47d39cf [scraper] disables horriblesubs scraper 2020-03-13 09:49:01 +01:00
TheBeastLT
3ec32b718b [scraper] removes unnecessary file index 2020-03-13 09:48:22 +01:00
TheBeastLT
972d4b8a43 [scraper] updates rarbg dum scraper with better logging 2020-03-13 09:48:01 +01:00
TheBeastLT
a3cc795680 adds language and resolution to torrent table 2020-03-12 14:27:55 +01:00