Welcome • User Guide
ToS • Privacy • Canary
Donate • Bugs • License

©2024 Poal.co

391

(post is archived)

You are currently inside a comment thread.
Click here to see all the comments (30).
[–] • 1 pt

Maybe I should have specified, curl -I

Ok didn't know about -I

I use requests library to get headers.

Everything works perfectly fine except for some corrupt jpeg files.

I uploaded one, you can try it with your code if you want and see if you get the same results.

https://pic8.co/sh/m2v8Y4.jpg

[–] • 0 pt

Oh and also if you are not convinced yet ;) that it really doesn't transfer the whole file:

# a download, normal GET request
curl -L https://pic8.co/sh/m2v8Y4.jpg -o "/dev/null"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100 33382  100 33382    0     0  13315      0  0:00:02  0:00:02 --:--:-- 25193

# HEAD request, received=0
curl -LI https://pic8.co/sh/m2v8Y4.jpg -o "/dev/null"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0 33382    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0

Some very stupid servers may not show content-length, and that's the ONLY time you'd want to download either the file header or the whole file.

[–] • 1 pt

urllib3 does support redirections.

Pic8 has been a little slow recently due to a bug, and because it doesn't host images but only host links it redirects too, it's normal that it takes a little more time.

I'm checking a direct link to catbox.moe and it gives me the same answer.

I can get the right 'width & height', but the size is still stuck at 31 bytes (the range/size of the headers).

I'll check that later. Got something else to do now.

Thanks for the curl suggestion.

[–] • 0 pt (edited )

One of the 'I keep forgetting everytime' with curl: it doesn't follow redirects automatically, needs the -L parameter:

curl -I https://pic8.co/sh/m2v8Y4.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
HTTP/1.1 302 Found
Server: nginx
Date: Sat, 05 Dec 2020 00:03:26 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /d/0c747999-9bdc-419e-ada4-f61f3fdad56b/
X-Frame-Options: ALLOWALL
X-Robots-Tag: noindex, nofollow, nosnippet, noarchive, noimageindex
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

but with -L it follows through the chain (oh God, I now know why pictures take so long to load):

curl -IL https://pic8.co/sh/m2v8Y4.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0 33382    0     0    0     0      0      0 --:--:--  0:00:03 --:--:--     0
HTTP/1.1 302 Found
Server: nginx
Date: Sat, 05 Dec 2020 00:03:49 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /d/0c747999-9bdc-419e-ada4-f61f3fdad56b/
X-Frame-Options: ALLOWALL
X-Robots-Tag: noindex, nofollow, nosnippet, noarchive, noimageindex
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

HTTP/1.1 302 Found
Server: nginx
Date: Sat, 05 Dec 2020 00:03:49 GMT
Content-Type: image
Content-Length: 0
Connection: keep-alive
Location: https://ist6-3.filesor.com/pimpandhost.com/1/_/_/_/1/a/g/E/F/agEFH/di-SSGKY8.jpeg
Referer: https://pimpandhost.com/static/html/wide_iframe_dynamic.php
X-Frame-Options: ALLOWALL
X-Robots-Tag: noindex, nofollow, nosnippet, noarchive, noimageindex
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block

HTTP/2 200
date: Sat, 05 Dec 2020 00:03:52 GMT
content-type: image/jpeg
content-length: 33382
last-modified: Fri, 04 Dec 2020 16:46:45 GMT
etag: "5fca67f5-8266"
expires: Thu, 31 Dec 2037 23:55:55 GMT
cache-control: max-age=315360000
cache-control: public
x-frame-options: ALLOW-FROM *
access-control-allow-origin: *
access-control-allow-credentials: true
access-control-allow-methods: GET, POST, OPTIONS
access-control-allow-headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
server: CDNCENSORED
x-CENSORED-nzt: CENSORED
x-CENSORED-nzt-ray: CENSORED
x-edge-ip: XXX.xxx.xxx.xxX
x-edge-pop: CENSORED
x-cache: MISS
accept-ranges: bytes

2 redirects, hits the image on 3rd. content-length = 33382 (just read the thing: HTTP/2 headers, while supposed to be case-insensitive, are always lower-case for better compression.

Depending on what your library does or does not do for you, you'd need to follow by the redirections, if the response mandates so.

EDIT by me: removed server IPs/nodes