Not downloading all pictures/ videos

[jamesle3199]

Hey I'm trying to scrape a whole profile but it doesn't seem to be downloading all the photos and videos. Here's what I'm getting, I don't know where the db files are either so I'm unable to clear it and try again..

[?] Which area(s) would you like to scrape? (Press ENTER to continue)  [All]
Name: ✨ 𝑃𝑟𝑒𝑔𝑛𝑎𝑛𝑡 𝐿𝑖𝑣 ♡︎ 💋 | Username: vintagebarbiex | ID: 50198059 | Joined: July 12, 2           02
- 1125 posts
 -- 4195 photos
 -- 185 videos
 -- 0 audios
- 391 archived posts
 + Getting pinned media...
 + Getting timeline media...
 + Getting archived media...
 + Getting highlights...
 + Getting messages...
Progress: (1133 photos, 8 videos, 0 skipped || 1.55 GB): 100%|█████████████████████| 1143/1143 [05:02<00:00,  3.77it/s]

Using profile: main_profile
? What would you like to do?  Download content from a user
? Choose one of the following options:  Enter a username
? Enter a model's username:  vintagebarbiex
[?] Which area(s) would you like to scrape? (Press ENTER to continue)  [All]
Name: ✨ 𝑃𝑟𝑒𝑔𝑛𝑎𝑛𝑡 𝐿𝑖𝑣 ♡︎ 💋 | Username: vintagebarbiex | ID: 50198059 | Joined: July 12, 2           020
- 1125 posts
 -- 4195 photos
 -- 185 videos
 -- 0 audios
- 391 archived posts
 + Getting pinned media...
 + Getting timeline media...
 + Getting archived media...
 + Getting highlights...
 + Getting messages...
Progress: (0 photos, 0 videos, 0 skipped || 0): 100%|████████████████████████████████████| 2/2 [00:00<00:00,  4.15it/s]

[taux1c]

This is interesting, it looks like there may be an error with the others and instead of spazzing out it moves to the next download. (A recent improvement.) The only problem is with that improvement we don't get a trace back.

I'll see what I can do about this tonight.

Are you in the discord or telegram?

[jamesle3199]

I've got telegram.

[taux1c]

The links are in the readme if you want to join either group.

[taux1c]

Does the program exit at the end or does it wait? Some downloads may still run after it moves to the next user and can you verify that you are able to see those posts? I know it's difficult to stay and if you need help I can help determine if all posts are visible.

I had an issue recently where I couldn't see a post because it was removed but the base of it was still there. Just the shell of a post and it said "this post has been removed."

That's actually the reason behind the patch to stop it from spazzing out when encountering an error with a post.

[0x3333]

Looks like a problem with your database thinking that you already downloaded those files. Could you remove it? Are you on windows ou linux/mac?

[taux1c]

He was going to remove it but he can't figure out where it's stored and I looked over it and I haven't even been able to figure out where it stores db files. I thought it was the same directory.

[0x3333]

Nope. It is located in ~/.config/onlyfans-scraper/. Where ~ is the user's home folder.

[taux1c]

@jamesle3199 did you get that?

The databases are kept in ~/.config/onlyfans-scraper

Mentioned by @0x3333 in the reply above.

[taux1c]

I think I see what's going on! It's an issue with it downloading one photo from each post. I'll take a look at that and verify in a couple hours. If so it shouldn't take long to fix.

[taux1c]

@jamesle3199 I have added a debug script to the scraper. If you get the new version and install it then open constants.py at the bottom you should see:

debug = False

if you change that to:

debug = True

It will then show the total count of media items that onlyfans returns, the post id and the # of media items the scraper found. It will then also calculate the difference (if there is one) and state how many items were missed for each post. This way we can see what post ids are giving use issues or if there is just an issue with the counter at the top. #12
😄

Remove the database as mentioned above by @0x3333 and then give this a try. Let us know what happens!

[taux1c]

Disregard this, I have found what I think the issue is and will try a patch later tonight. Not sure what exactly is causing it but something is telling the program it can't download some items. I wrote a quick parser to sort through them and it comes up with the same number as onlyfans reports so I am going to try to implement it tonight.

[taux1c]

I think that the main counter is counting images that have the canView element set to false. This likely means that they are paid items that you haven't purchased or private items that your profile doesn't have permission to see.

[jamesle3199]

I’ll have to update it and test it tomorrow when I’m off work. Haven’t been able to login but when I browse her page I’m able to see all the videos as none of them are paid items or private etc. I’m physically able to view 185 videos on her page yet it’s only downloading 8.

[jamesle3199]

I’ll remove the database tomorrow and let you guys know how it goes.

[taux1c]

Okay, it's not 100% ready anyway. I found some bugs in the debug code so I moved it to a new branch. You're welcome to try it as it works with some profiles but some it doesn't. I think maybe in the most recent update they moved some flags we look for with the code and it may be causing some media to be ignored.

[taux1c]

It took a bit longer than expected but after going over this issue myself and working with other users on it I believe this is an issue with the connection as @0x3333 stated.