Do an Everything search on remote FTP?

[awesnap]

Hello there. First of all, thank you for making this tool, its quite useful!

However, i'm hoping you can help me. I have a rather large (15TB) ftp server that I use for work, and searching on Filezilla is..... painfully slow.
Wouldn't it be nice to crawl the FTP server, index its contents, and then use a program like EVerything to search for files, folders, etc? I thought so too! However, I cannot seem to figure out if I can accomplish this with Everything.

WHen I attempt to add the FTP credentials in Tools>Options> ETP/FTP Server, i'm met with an error: Unable to start ETP Server: bind failed 10049

I also attempted to use a program such as WebDrive to map the FTP server to a drive letter (W:) however, this also gives me an error.

Can you tell me, is there a way that I can use Everything for my needs? I"m REALLY hoping so, as searches may take so long, that I have to run them overnight for specific files!

Thanks in advance!

[horst.epp]

You can map a drive letter to your FTP server
and use Folder indexing in Everything.

[awesnap]

Thanks for the quick reply, however in WIndows 10 you can't map a drive letter to an FTP server.

That's why I downloaded WebDrive in order to accomplish this. However, when I do access the mapped FTP letter'd drive (via WebDrive) I get an error: Unable to start ETP Server: bind failed 10049

I am able to "Add Network Location" however I am unable to add this as a folder to be scanned in Everything.

[void]

Thank you for your feedback awesnap,

Please don't use Tools>Options> ETP/FTP Server
This is for hosting an FTP server, not connecting to one.



To index an FTP server in Everything 1.5:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Folders tab on the left.
• Click Add custom....
• Type in your ftp address, for example: ftp://server/share
• Click OK.
• Click OK.

Please note: Indexing an FTP server will take a very very long time.

[awesnap]

Thank you void for your response, a few more questions for you.

1 - is there somewhere that i'd have to put my login credentials?
2 - if I scan the ftp, can I then take that database with me to another machine to speed up the process of searching on that ftp server?

[void]

1 - is there somewhere that i'd have to put my login credentials?

Please try the following custom folder:

ftp://username:password@server:port/share

if I scan the ftp, can I then take that database with me to another machine to speed up the process of searching on that ftp server?

Yes, with the Everything Server:

Install Everything on a PC that is always on.
Setup an Everything Server.
Have the Everything Server index your FTP site.
Have the Everything Server rescan your FTP site daily (this would be the default option)
Add the Everything Server to your multiple Everything clients under Tools -> Options -> Network indexes.

[awesnap]

I have a synology server thats always on, is it possible to install it on that?

[void]

No, sorry.

Everything is currently Windows only.

[awesnap]

Ok, I had added the FTP server and it is connecting, however it is not recursivly adding any folders, or files. I am looking at the debug window, but I can't understand what its telling me. It seems to be stuck on HCBT_SEWTFOCUS

[awesnap]

Alright, I spoke too soon, it seems to be recursing through the ftp, at least in the dialog box.

Question, with version 1.5a, I had to shut it down and reopen it a couple of times, however the settings that I made in the Options menu go back to defualt every time I restart the program. Is there a reason its doing this, or is there a way to save my settings so I don't have to keep doing my Excludes, and adding of the custom FTP settings?

Thanks in advance!

[horst.epp]

Alright, I spoke too soon, it seems to be recursing through the ftp, at least in the dialog box.

Question, with version 1.5a, I had to shut it down and reopen it a couple of times, however the settings that I made in the Options menu go back to defualt every time I restart the program. Is there a reason its doing this, or is there a way to save my settings so I don't have to keep doing my Excludes, and adding of the custom FTP settings?

Thanks in advance!

I guess you installed Everything under Program files
which doesn't allow to update your settings.

Set in the GUI under Options / General
Store settings and data in %APPDATA%\Everything

Or use the portable version which I prefer.

[awesnap]

Ok, i'm giving the portable version a try. Where does the portable version keep the database?

Oh, and thank you so much for going through this with me, this has been super, super helpful support

[void]

The portable version stores settings and data (the Everything.db) in the same location as your Everything.exe

Please make sure Everything will have write access to this folder.
-or-
Enable Tools -> Options -> General -> Store settings and data in %APPDATA%\Everything

[awesnap]

Ok, quick question about the database.

I can't seem to find it, even when I was using version 1.4. Is the folder that the option %APPDATA%/Everything referring to this?
C:\Users\USERNAME\AppData\Local?

Also, how can I make sure that everything.exe has permissions to write to its own directory?

I had Everything (portable 1.5) scan one folder on the FTP, and it just completed it. I don't want to shut the program just yet until I make sure it has saved its progress (the databse), and before I have it scan the FTP further. So i'd justr want to know the answers to these before I proceed.

Thanks in advance!

[void]

Everything stores its database in memory.

The database is only written to disk when you exit Everything.

Exit Everything to update the database on disk.

[awesnap]

Ok, I just exited everything, however there is no Everything.db, or any file that I can tell in the everything folder with the portable version. In addition, the dialog box has some red text saying:
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Search History-1.5a.csv.tmp
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
failed to create config file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Plugins-1.5a.ini.tmp
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
failed to create config file C:\Program Files\Everything 1.5a\Everything-1.5a.ini.tmp
processed 0 usn records in 0.000000 seconds
REFS 0000000000000000
db_save_local 5512 folders, 178657 files
CreateFileW(): 5: Failed to open file C:\Program Files\Everything 1.5a\Everything-1.5a.db.tmp
failed to save C:\Program Files\Everything 1.5a\Everything-1.5a.db.tmp 5
saved db: 0.000338 seconds
ready
DB_WAIT: _db_ready_proc waiting for _db_save_thread_proc...
DB_WAIT: _db_ready_proc waited 0.000064 seconds
_db_query_results_changed_event_proc 0000000000000000
destroy ui
fileinfo clear all

[awesnap]

Correction, there is now a Everything-1.5a.db file, however the size is only 1kb
Reopening the program does however have my settings saved from the Tools menu

[void]

Everything will not have write access to C:\Program Files\Everything

Please enable "Store settings and data in %APPDATA%\Everything" from Tools -> Options -> General.

[tuska]

... Here is an example:
 

2022-12-21_APPDATA-Everything__Save database to disk without having to close Everything.png (66.63 KiB) Viewed 6037 times

  
Based on this topic, I can save my database to disk via a shortcut (.lnk) with parameter -update without having to close Everything, e.g.
[adjust the path if necessary ...]

Code: Select all

C:\Everything\Everything64.exe -update

[awesnap]

I'm not sure why, but now Everything is not scanning the FTP at all.

The only change that I made was in the General tab, to store settings and data in %APPDATA%\Everything, but now even changing that option to be off, its just not scanning.

The noly thing the Debug console says is this:

update ftp://USER:PASS@SERVER:21/SHARE now
check for folder updates 0000000000000000
update folder start ftp://USER:PASS@SERVER:21/SHARE
scanning ftp://USER:PASS@SERVER:21/SHARE...
scan completed in 2.888309 seconds
rescan update db
rescan update db in 0.000317 seconds
next update at 21/12 3:00
leave folder update
check for folder updates 0000000000000000

[void]

Please make sure "Fast rescan" is unchecked under Tools -> Options -> Folders -> (your FTP folder)

[awesnap]

It is not checked, the only thing that is checked is "Index Subfolders"

[awesnap]

Ahhh, ok I got it. It was my mistake, I had the wrong folder it was pointing to, thats why it was not scanning any farther. Ok, I will let this run, and see what happens.

[void]

Everything is likely not seeing the root ftp folder.

Please try forcing a rebuild from Tools -> Options -> Indexes.

[awesnap]

Ok, so its been running for a bit (still running) and I typed in C:\program files\everything 1.5a>Everything64.exe -update
However, when I navigate to C:\Users\USER\AppData\Local\Everything
The database is there, and its been modified at 10:33pm. but the size is still only 1kb.

[void]

When "Store settings and data in %APPDATA%\Everything" is enabled, the Everything.db is store in:

%LOCALAPPDATA%\Everything

[awesnap]

Ok, great news. I woke up today to a database with the size 10mb, in appdata/local/everything.
And the results from the scan were in Everything when I was looking at it. I then added each folder that I wanted to scan with Everything (Another 8 directories on the FTP) and hit rescan now.

However, when I do this, the results dissapear in Everything, and on the bottom status bar, it just says "Scanning ftp://..............."

Is there a way to still see what results it has up to this point, while its scanning?

[awesnap]

Ok, the scan for the rest of the server has (for some reason) stopped. Is there some sort of maximum size of the database that is ticked somewhere?

I hit rescan, and its saying the scan is complete, but its not. It has scanned about 35-40% of the server, but i'd like it to scan the entire thing. Thanks in advance!

[void]

Everything 1.5.0.1332a fixes an issue with rescanning virtual folders.

Please try rescanning your FTP folder with this version.

Does the issue persist?

[awesnap]

Yes, unfortunately. I ran the new version you'd linked to in portable mode, and rescanned.
It seemed to stop at the same point as the last one, its currently showing 462,610 objects, and around 4.16 TB of data, which is almost exactly where it stopped last time. The total size is more than that, i'd say it scanned about 35% of the server.

Is there some sort of maximum number it can scan? Or a time out? Because it stopped right around the same point during the scan. And, when I attmept to "rescan now" , the debug is saying the scan is completed in 0.022 seconds.

As a test, i'm able to connect to the FTP via both filezilla, and my mapped frive via webdrive, and look around the folders/files, so I still have access.

[void]

Sounds like Everything is getting stuck on a particular path.

Everything has a limit of 259 characters for FTP paths.



Debug logs may help find the issue:

• Restart Everything (File -> Exit -This will clear virtual folder cache)
• In Everything, from the Tools menu, under the Debug submenu, check Verbose.
• From the Tools menu, under the Debug submenu, check Start Debug Logging.
• Rescan your virtual folder:
• From the Tools menu, Click Options.
• Click the Folders tab.
• Select your ftp folder.
• Click Rescan Now.
• ---wait for rescanning to complete (progress is shown in the top right and in the status bar)
• In Everything, from the Tools menu, under the Debug submenu, check Stop Debug Logging.
• This will open your %TEMP%\Everything Debug Log.txt in Notepad.
• Please examine this debug log for the last scanned for folder.
• What was the last scanned folder?
• -If unsure, could you please send your Everything Debug Log.txt to support@voidtools.com

Privacy



There is currently an issue with Everything using stale cache when re-scanning your virtual folders.
I am working on a fix..

[awesnap]

Ok i've done it. Its done scanning, however the last file scanned is shorter than the other files previously scanned (I can send you the debug log to examine). Using a charachter counter, its telling me there is 129 characters in the last file scanned. It did however scan the 2nd folder that i'd asked it to Rescan, which it did not do yesterday. So far its about 45-50% scanned the entire FTP, so thats an improvement, but still not the entire thing.

One other thing to note, I double checked my RAM, and I have 15.9 out of 16 gb used at the moment, not sure if that's anything to be concerned about? (Currently running MANY chrome tabs opened at the moment, but hasn't been an issue in the past, or for other programs, but figured i'd mention it anyway.)

After the last scanned file in the Debug logs, the next lines in the Debug are as follows:

2022-12-23 12:19:29.955: event post: 00007ff7ff522190 00000000011405a0
2022-12-23 12:19:29.955: processed folder fd update events in 0.569638 seconds
2022-12-23 12:19:29.955: MSG: 0000000000b61494 1401 0000000000000000 0000000000000000
2022-12-23 12:19:29.955: TRAY 00001401 0000000000000000 0000000000000000
2022-12-23 12:19:29.955: update EVENTs
2022-12-23 12:19:29.956: EVENT: 00007ff7ff522190 00000000011405a0
2022-12-23 12:19:29.956: DB_WAIT: _db_monitor_finished_process_fd_update_events_event_proc waiting for _db_monitor_folder_process_fd_update_events_thread_proc...
2022-12-23 12:19:29.956: DB_WAIT: _db_monitor_finished_process_fd_update_events_event_proc waited 0.000041 seconds
2022-12-23 12:19:29.956: _db_query_results_changed_event_proc 0000000001148070
2022-12-23 12:19:29.956: _db_monitor_process_update_event_available_event_proc


One other thing i've noted, the database size is actually SMALLER than the old one.
The first one is 115mb, and after this recent scan its only 41bm. Now showing 889,240 objects, and 7.94 TB

[awesnap]

One more issue i've come up with.

I haven't actually used Everything to download any of the files on the ftp, but just now I attempted to click and drag one of the files to a local folder, and was met with this error:

The item could not be transferred. Make sure you have permission to store files here and that the file name is valid.
Details:
No more internet handles can be allocated

However, if I navigate via webdrive to that same folder and click and drag, it works with no issues.

[void]

Thanks for the debug logs.

Everything uses the Windows Shell to list FTP folders and open/copy ftp files.

The Windows Shell is rather limited and broken as an FTP client.
It's fine for a few folders.
There seems to be handle leaks in the Windows Shell and there is only so many folders you can browse before you hit some internal limit.



Everything 1.5.0.1333a will now refresh ftp cache when scanning an ftp folders.

This might help with handle leaks? and should also help with keeping your indexes up-to-date by forcing a refresh.
Previously the Windows Shell would cache ftp folders and never refresh.

[awesnap]

Thank you for the update. I loaded the portable version, re-setup the custom folders with the ftp location to index, and hit rescan.
The status bar on the bottom says its scanning the folder, however when I pull up the debug, it seems to just sit there, it doesn't show the folders being scanned. I'll let it sit for a few hours to see if there is any changes, but it doesn't seem to want to go anywhere.

I'll post my findings here once there's an update.

[awesnap]

Ok, so Everything has stoppeddthe scanning. It has again, stopped at 4.16TB worth of files, and 462,603 objects. This was using the 1.5.0.1333a version, with the portable.exe. Not sure if this helps, but it seems that its stopping around the same point, even from the previous version.

It also didn't scan the other folder (I had put 2 of the large folders in the FTP to scan) but it stopped midway through one of the larger folders.

[void]

Thank you for the information awesnap,

What's the last reported folder scanned?

• In Everything 1333a, from the Tools menu, under the Debug submenu, check Verbose.
• From the Tools menu, under the Debug submenu, check Start Debug Logging.
• Rescan your virtual folder:
• From the Tools menu, Click Options.
• Click the Folders tab.
• Select your ftp folder.
• Click Rescan Now.
• ---wait for rescanning to complete (progress is shown in the top right and in the status bar)
• In Everything, from the Tools menu, under the Debug submenu, check Stop Debug Logging.
• This will open your %TEMP%\Everything Debug Log.txt in Notepad.
• The last folder rescanned will be shown near the bottom as:

  Code: Select all

  scan ftp://server/share/...
  scan completed in x seconds
  

Please try repeating the same rescan, is Everything dying on the same path?