The key idea is a combination of (a) and (b).
(a) to have a managed structure for documents. E.g. certain directories where you put them, rules for naming, classifiers, hierarchies, procedures (habits) for reviewing lists, and so on.
(b) every time you see a document in a wrong place - delete it without looking, pretend this document didn't exist.
The latter is hard (psychologically) to begin, and you'll likely lose important documents and will have to re-do some work when you're just starting to apply this principle, but it's incredibly effective, and will cause overall productivity improvement when used consistently. Long-term psychological effect is that every time you feel the document is important, you'll file it properly, it will be a no-brainer (because otherwise it will get deleted).
For transition period you can move all existing stuff into some unorganized pile, but under certain root directory, and search/pull documents from it on "as needed" basis, as you start developing a new well-managed structure.