Welcome • User Guide
ToS • Privacy • Canary
Donate • Bugs • License

©2025 Poal.co

983

It’s getting harder and harder for me to find good resources online. blogs get taken down and i have a hunch that in the next few years a lot of sites i frequent for research will be blocked “for my safety.” i have a large system of bookmarks for sites that i can only find via the bookmark now.

i’m thinking about downloading a collection of potentially-banned sites and then indexing them on a local private network of some kind.

i’m aware that this is probably going to cost me $1/2k to get something with enough storage.

1) has anyone done this?

2) any advice on where to start? as in, hardware to get, best way to download an entire site, software to index/search what i download, etc.

3) do you have any sites that are your go to’s for research that you don’t mind sharing with me? health topics, freedom topics, education topics, etc.

thanks!

It’s getting harder and harder for me to find good resources online. blogs get taken down and i have a hunch that in the next few years a lot of sites i frequent for research will be blocked “for my safety.” i have a large system of bookmarks for sites that i can only find via the bookmark now. i’m thinking about downloading a collection of potentially-banned sites and then indexing them on a local private network of some kind. i’m aware that this is probably going to cost me $1/2k to get something with enough storage. 1) has anyone done this? 2) any advice on where to start? as in, hardware to get, best way to download an entire site, software to index/search what i download, etc. 3) do you have any sites that are your go to’s for research that you don’t mind sharing with me? health topics, freedom topics, education topics, etc. thanks!

(post is archived)

[–] • 1 pt

anticlutch and i talked about this before. you want to be making arc or warc or equivilent files. basically a modern website archive has to "play back" the site. For some reason saving the dom is out of the question, i'm not sure why.

looks like a good place to start. I have used a , and pywayback as well. i think there is a way to make these with wget or curl as well. If you do these you can even play back video sometimes.

[–] • 0 pt

best way to download an entire site

wget -m if using linux. If using windog, you might install cygwin first, and then use wget.

sites

[–] [deleted] • 0 pt

Depends on whether you operate under windows, ios or linux. For windows there are a number of free programs available online which will download an entire website. You will need some significant storage capability as I’ve found even seemingly simple sites can take up a lot of space unless you want to spend the time and prune away unnecessary files.

[–] • 0 pt

whenever i have something i want to save i make sure i save it as a pdf. not really the best way but if you spend time to really organize it you can save quite a bit of information.

[–] • 0 pt

I recommend porting them to text only, for starters.

There are many ways to do this.

[–] • 0 pt

Thanks all! You gave me some good things to consider.