The biggest problem, IMO, is in most places you aren't allowed to live in a simple cabin any more. It's against zoning rules. You can't build under 1200 sq.ft, you have to be hooked up to electricity, water and sewer, or your structure will be condemned. You can't grow a "victory garden", the city or the HOA will fine you. And the cost of materials is nuts, and a parcel large enough that you could build with materials you harvested yourself is expensive, and property taxes will continue to eat at you.