Index all datastreams of a file

Have a suggestion for "Everything"? Please post it here.
Post Reply
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Index all datastreams of a file

Post by NotNull »

Possibly not a mainstream suggestion, but it might be useful to also index the alternate/ alternative/ named datastreams (short:ADS).

Couple of use cases:
  • Some applications store metadata and tags of documents in this ADS. That would make this metadata searchable.
  • (My main usecase:)Search for Zone.Identifier datastreams so that they can be deleted (frm within Everything).

    This Zone.Identifier datastream gets added to files you download from the internet. You have to unblock those (file properties > Unblock) before further processing is possible. Deleting the datastream does the same.
    Currently I use a script for that; it deletes all Zone.Identifier streams from all the files in my download folders.
  • This gives a more accurate representation of actual disk space usage (currently only the unnamed datastream is 'counted')
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Index all datastreams of a file

Post by void »

It's on my TODO list, currently Everything ignores named data streams.

Thanks for the suggestion.
collinchaffin
Posts: 24
Joined: Thu Feb 02, 2017 9:42 pm

Re: Index all datastreams of a file

Post by collinchaffin »

I wasn't sure if I should post here, or resurrect the decade-old thread here: viewtopic.php?f=4&t=4481

Either way after searching to see if ADS indexing had been added yet to Everything, I came across that above thread asking for a mainstream example of app(s) that actively use xattr alternate data streams and was blown away that the one that is WILDLY popular in the last decade was never mentioned since it's entire backup/restore operation (primary) depends on xattr ADS.

That tool which I'm certain void has not only heard of, but may have even used is FILEBOT. If you go grab v4.79 (the last fully freeware) although even the > 4.80beta/final (now shareware) still maintain this functionality. Filebot stores the original filename and some other metadata within ADS of the files it manipulates which it then uses as the primary to "undo". It obviously is not the only app utilizing ADS that without specialized one-off utils like Nirsoft or scripts today makes them virtually invisible to most users. ES could change all that. :)

With that said, back to the real-world example - anyone who specifically uses Filebot against large media libraries (tens-hundreds of TB+) would absolutely see Everything as a godsend to be able to have it index, store, then use it's instant-db query to report on the origin of the files etc.

Today, there are different scripted solutions (I myself am a Powershell guy and have now been doing PoSH ADS querying) but we rely on (if we even store non-volatile at all) implementing our own back-end storage methods from flat data files, to sqlite etc. I myself already use Powershell to query Everything though, too and boy since I already store all my media library standard file info from all those source drives into Everything how awesome would that be even if it was a folder-like manual refresh process to just have that data in ES not only for my scripts, but heck just for quick searches!
verviers
Posts: 34
Joined: Tue Nov 01, 2022 7:48 am

Re: Index all datastreams of a file

Post by verviers »

NotNull wrote: Sat Sep 15, 2018 2:56 pm [*](My main usecase:)Search for Zone.Identifier datastreams so that they can be deleted (frm within Everything).
void wrote: Tue Sep 18, 2018 7:47 am It's on my TODO list
I have exactly the same purpose. Five years have passed. Is it still not possible?
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Index all datastreams of a file

Post by void »

Everything 1.5 will have support for a Zone ID, Referrer URL and Host URL properties.

These properties can be indexed.

Zone ID, Referrer URL and Host URL properties
verviers
Posts: 34
Joined: Tue Nov 01, 2022 7:48 am

Re: Index all datastreams of a file

Post by verviers »

void wrote: Mon Nov 06, 2023 7:10 am These properties can be indexed.
OK. But it is still not possible to find ADS named "Zone.Identifier" and to delete them in Everything GUI? Did I understand correctly?
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Index all datastreams of a file

Post by void »

It can be done with the Alternate Data Stream Names property in Everything 1.5.

Example search:

adsname:"Zone.Identifier"

adsname:



Everything doesn't have any functionality to add, remove or modify alternate data streams.
verviers
Posts: 34
Joined: Tue Nov 01, 2022 7:48 am

Re: Index all datastreams of a file

Post by verviers »

void wrote: Mon Nov 06, 2023 7:19 am Everything doesn't have any functionality to add, remove or modify alternate data streams.
It is clear to me now. Thanks. It will be hard enough for me to delete thousands of "Zone.Identifier" streams manually. :roll:
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

I wouldn't dare to spoil your fun removing thousands of "Zone.Identifier" streams manually, but if you get RSI or cramp halfway, one method to remove them all at once (the safest method I know):
  • Search for all files with a Zone.Identifier dfatastream:

    Code: Select all

    C:|X:   alternate-data-stream-names:zone.identifier
    ( C:|X: = limit the results to your NTFS formatted volumes only)
  • Menu:File => Export
  • Save as type: TXT Text Files (*.txt)
  • Start PowerShell
  • Older versions of PowerShell did not support Unblock-File, so test first if version is OK by typing the following command (+ENTER)

    Code: Select all

    unblock-file "C:\windows\notepad.exe"
  • If this command runs without error, continue.
    If not: report back (there are more ways ..)
  • Type or paste the following command:

    Code: Select all

    gc T:\unblock.txt | %  {unblock-file $_ } 
    (where T:\unblock.txt is the name of the file with the exported filenames)
  • Done!
void
Developer
Posts: 15096
Joined: Fri Oct 16, 2009 11:31 pm

Re: Index all datastreams of a file

Post by void »

To add a right click context menu item to unblock files:
  • From the Start menu, search for:
    regedit
  • Right click the Registry Editor and click Run as administrator.
    ---
  • In the Registry Editor, navigate to:
    HKEY_CLASSES_ROOT\*\shell
  • On the right side, right click and click New -> Key
  • Set the Key name to: Unblock
    ---
  • Navigate to:
    HKEY_CLASSES_ROOT\*\shell\Unblock
  • On the right side, right click and click New -> Key
  • Set the Key name to: command
    ---
  • Navigate to:
    HKEY_CLASSES_ROOT\*\shell\Unblock\command
  • Set the (default) value data to:
    powershell.exe -noprofile -Command "& {unblock-file -path '%1'}"


Search for your files with a zone.identifier in Everything with:
C:|X: alternate-data-stream-names:zone.identifier
Select all results
Right click and click Unblock (I recommend performing this action for less than 15 files at a time*)

*edit: 15 or less files otherwise the menu item is not shown (see below)



To remove the right click context menu item to unblock files:
  • From the Start menu, search for:
    regedit
  • Right click the Registry Editor and click Run as administrator.
  • Navigate to:
    HKEY_CLASSES_ROOT\*\shell\Unblock
  • Delete this key.
Last edited by void on Fri Nov 17, 2023 12:03 am, edited 3 times in total.
Reason: powershell -Command "& {unblock-file -path '%1'}" => powershell.exe -noprofile -Command "& {unblock-file -path '%1'}" -thanks NotNull
hamid56
Posts: 21
Joined: Sun Jul 20, 2014 9:38 am

Re: Index all datastreams of a file

Post by hamid56 »

i recommend stream remover v1(made in 2018) from nice sordum website for removing ADS.
https://www.sordum.org/11263/streams-remover-v1-0/
or
https://www.sordum.org/files/download/s ... emover.zip
Last edited by hamid56 on Sat Jan 20, 2024 1:19 pm, edited 1 time in total.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

Thanks for sharing @hamid56!
verviers
Posts: 34
Joined: Tue Nov 01, 2022 7:48 am

Re: Index all datastreams of a file

Post by verviers »

Have tested three methods (NotNull, void, hamid56). void and hamid56 "right click solution" works only if I select up to 15 files.
screen_2023.11.07.001.png
screen_2023.11.07.001.png (2.13 KiB) Viewed 7512 times
If the number of selected files is greater than 15, these menu items are not shown at all. :? Windows Server 2016x64.

So the only way for me now - to follow NotNull`s guide. Thanks everyone guys.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

verviers wrote: Tue Nov 07, 2023 9:47 pm if I select up to 15 files.
Windows limitation. You can increase this number in the registry.

More info
verviers
Posts: 34
Joined: Tue Nov 01, 2022 7:48 am

Re: Index all datastreams of a file

Post by verviers »

NotNull wrote: Tue Nov 07, 2023 9:56 pm Windows limitation. You can increase this number in the registry.
Yep. Didn't get this information, because had faced to some PowerShell errors, connected with UTF-8 encoding, regarding to your and void`s sentences, and spent a lot of time trying to solve the issues. As a result, I had to modify commands as

Code: Select all

gc -encoding utf8 D:\del_zone.txt | %  {unblock-file $_ }
and

Code: Select all

powershell.exe -noprofile -Command "& {unblock-file -path '%1'}" -encoding utf8
-----------------------------------------------------------------------
I have to note, that Sordum Streams Remover looks not very suitable, because it creates notification window for every processed file from the list. So if there are about 40 000 files, it will be disaster. Not sure if we can turn off this feature.
Attachments
screen_2023.11.07.002.png
screen_2023.11.07.002.png (22.71 KiB) Viewed 7501 times
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

Well done!


Another alternative could be:
- Search for the Zone.Identifier files
- 'CTRL + A' to select them all (or select a few for testing)
- 'CTRL + SHIFT + C' to copy filenames including path to the clipboard
- In PowerShell, run the following command:
Get-Clipboard | % {unblock-file $_ }

(This command doesn't like quoted paths (like "c:\file.txt" ), so make sure Double quote copy as path is unchecked under Options > Results )



I tested Sordum Streams Remover(briefly)

The dialog boxes can be removed:
  • Start Register Editor (regedit.exe) as administrator
  • Browse to
    HKEY_CLASSES_ROOT\*\shell\ZXY-Streams-Remover\command
  • Double-click (Default) in the right pane
  • Remove the /I from the command, so the new command looks like:

    Code: Select all

    "C:\Program Files (x86)\Streams Remover\sRemover.exe"  "%1"
  • Exit Register Editor
The process was still quite slow though: removing Zone.Identifier stream from all 14 matches took quite a while. Felt like 10 seconds.
DeRobertis
Posts: 4
Joined: Thu Nov 16, 2023 3:53 pm

Re: Index all datastreams of a file

Post by DeRobertis »

hamid56 wrote: Tue Nov 07, 2023 2:29 pm i recommend stream remover v1(made in 2018) from nice sordum site for removing ADS.
Another alternative: streams64, which is part of Microsoft's excellent (and free) Sysinternals Suite https://learn.microsoft.com/en-us/sysin ... nals-suite.

The shell command I use is

Code: Select all

C:\PATH_TO\SysinternalsSuite\streams64.exe -nobanner -d "%1"
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

Thanks for sharing, DeRobertis!

I used this in the past too. It did remove *all* ADS streams however.
Some applications actually use ADS (on my system a couple of file managers)

Don't know the current situation of streams; my version is 7 years old ...
horst.epp
Posts: 1332
Joined: Fri Apr 04, 2014 3:24 pm

Re: Index all datastreams of a file

Post by horst.epp »

The actual Streams version is 1.6 from April 2020.
It deletes all streams.

You can use AlternateStreamView from Nirsoft.
It displays every stream of files and allows to just delete one named stream from them.
https://www.nirsoft.net/utils/alternate ... reams.html
Last edited by horst.epp on Thu Nov 16, 2023 6:06 pm, edited 1 time in total.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

Thanks for testing, Horst!
verviers
Posts: 34
Joined: Tue Nov 01, 2022 7:48 am

Re: Index all datastreams of a file

Post by verviers »

Finally, I`d like to prefer these methods, mentioned above by NotNull for a large number of files with zone.identifiers:

Code: Select all

Menu:File => Export
Save as type: TXT Text Files (*.txt)
Start PowerShell
gc T:\unblock.txt | %  {unblock-file $_ } 
OR

Code: Select all

- Search for the Zone.Identifier files
- 'CTRL + A' to select them all (or select a few for testing)
- 'CTRL + SHIFT + C' to copy filenames including path to the clipboard
- In PowerShell, run the following command:
Get-Clipboard | % {unblock-file $_ }
Advantages:
- no need to modify registry, right click context menu etc
- no need to download external tools
- it`s easier to detect issues in Powershell console, when command fails due to specific errors (too long path; unicode path; wildcard character pattern issue).
DeRobertis
Posts: 4
Joined: Thu Nov 16, 2023 3:53 pm

Re: Index all datastreams of a file

Post by DeRobertis »

NotNull wrote: Thu Nov 16, 2023 5:17 pm I used this in the past too. It did remove *all* ADS streams however.
To clarify: do you wish to remove all alternate data streams from any file that has a :Zone.Identifier, or do you wish to remove only :Zone.Identifier from files that contain it and other ADS?

a. if file contains(:Zone.Identifier) remove all ADS streams

b. if file contains(:Zone.Identifier) and other streams, remove only :Zone.Identifier
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

For me it is b. : Remove just Zone.Identifier streams.

Can't speak for @verviers, but by the look of it (s)he is looking for b. too.
DeRobertis
Posts: 4
Joined: Thu Nov 16, 2023 3:53 pm

Re: Index all datastreams of a file

Post by DeRobertis »

Did you check how many files you have that require such special handling?

E.g., I have hundreds of files with Filebot ADS metadata, but when I search

Code: Select all

ads-count:>1 ads-name:Zone.Identifier
I see nothing. Perhaps it depends on one's workflow. I tend to clear Zone.Identifier first, before further processing.
NotNull
Posts: 5167
Joined: Wed May 24, 2017 9:22 pm

Re: Index all datastreams of a file

Post by NotNull »

DeRobertis wrote: Fri Nov 17, 2023 2:25 am ads-count:>1
Ah, I see the miscommunication now ...
DeRobertis wrote: Thu Nov 16, 2023 9:38 pm a. if file contains(:Zone.Identifier) remove all ADS streams
b. if file contains(:Zone.Identifier) and other streams, remove only :Zone.Identifier
The answer for me (and @verviers) is:

c. if file contains (:Zone.Identifier), remove only :Zone.Identifier
DeRobertis
Posts: 4
Joined: Thu Nov 16, 2023 3:53 pm

Re: Index all datastreams of a file

Post by DeRobertis »

As described above, Powershell Unblock-File is the most straightforward way to remove :Zone.Identifier via CLI.

I avoid Powershell where alternatives exist, as it is slow, especially when working with hundreds/thousands of files at a time. IME cmd, Java, and Groovy scripts can handle file queries and transformations in a fraction of the time.

My workflow generally involves filtering via Everything, then dragging the results to a scrap window of my file manager, Zabkat xplorer². From there I can call shell commands, custom scripts, etc.

Find what works best for your common use cases and technical abilities.
Post Reply