voidtools forum

Hello!

I've been using Everything for so many years now and as a software developer it always felt like this is too good to be true and its developer must be cheating somehow

On the one hand Everything is incredibly fast at searching, which indicate that the in-memory data structure for the database is likely a sorted array (or multiple array chunks) for its fast lookup times, cache friendliness and great scaling with multiple threads. Maybe the array items also store the index or memory address of their parent folders, for fast path building and to save memory.

But on the other hand Everything also seems to be highly dynamic, in that there are constantly files and folders being removed or added to the database, which wouldn't work that well with an array, since inserting items sorted or deleting them involves a lot of moving memory around and presumably updating lots of references to items.

I also thought that maybe those changes would be stored in different arrays, like a database diff, but I suppose this would impact the search performance too much.

So does someone know how Everything is so fast and dynamic at the same time? Because I'm always hesitant to use arrays with lots of entries once adding and removing items becomes a common operation. Or maybe I'm just underestimating how fast computers nowadays are.

Thanks!

Would love to know more about this. The program is incredibly fast.

Even accessing its Options is instant [which when compare to FreeCommander's Settings makes it look lightspeed fast]

I think one of the keys is this is still in c++.

I actually don't even know if it uses an in-memory db or custom data structures, they must be getting flushed to disk pretty frequently too, plus it has to keep updating them with Windows fs events. I bet the current view stores a list a of affected nodes and then if there is an update, it can check if any of those nodes are impacted so they can be refreshed.

This kind of efficient low level programming by a dedicated dev is not very common these days with everyone having moved on to JS/Python/Web. Another example of an efficient app I use daily is xplorer2 (zabkat.com) which is a must have as a file explorer. I believe there is an integration too.

planeW wrote: ↑Fri Mar 19, 2021 7:23 am... it always felt like this is too good to be true and its developer must be cheating somehow ...

I agree. I wrote myself a content-search of MSWord documents, and wrote it in Word97/VBA. Blew (back then) Windows File Explorer out of the water.

I think that David, the developer is very bright and is a very good programmer.

But I suspect (and do not know) that the main advantage comes from NOT inheriting core code from the days of MSDOS and 1.2MB 5 1/4" floppy disk ideas. This Article is a starting point, but perhaps someone with a better memory than I can dig out the folklore that Bill Gates once said something like "ordinary people won't understand sub-directories".

I have seen enough corporations that carry a basic, biut stale, philosophy for dozens of years.
I suspect that some core ideas have held true right through the development of Windows, and perhaps insistence on backwards compatibility acts as a hobble on progress.

I suspect that David has not been hindered by old ideas from the 1980s; he has had to deal with those old ideas, as hurdles, but they have not been an obstacle to his progress.

A recent example is the introduction of SSDs. Out the door goes defragmentation. In the door comes even faster content searching (for all applications, not just Everything (and as I write that I am shaken by just how weird that sounds!))

I hope that helps us travel along the magic path ...
Cheers, Chris

Parts of Everything are written in assembler.

Aside from the points mentioned, my impression is that a lot of slowness and bloat in the programming world comes from using 3rd-party libraries and frameworks particularly as regards speed for tasks that are repeated rapidly. In most cases not only is coding such things specifically going to lead to much faster execution, but in many cases it's much quicker to do so than to learn how the library works, and of course more fun. d

When you see how convoluted and bloated modern frameworks it's easy to see how some asm and c code could be phenomenally fast especially if directly going to the kernel Apis at the lowest possible level.

I'd love to see some of the coding tricks David is most proud of posted here. I'm sure everyone could learn a thing or two.

Everything 1.3 and earlier had a lot of assembly.
Everything 1.4 is 100% C
Everything 1.5 is 100% C

A lot of the performance improvements in Everything 1.5 come from tight loops and keeping important things in CPU cache.

ChrisGreaves wrote: ↑Wed Feb 28, 2024 12:12 pm [...]
core code from the days of MSDOS and 1.2MB 5 1/4" floppy disk ideas. This Article
[...]

How archaic and annoyingly constraining this looks like

I must have totally forgot about such limitations from the times of me being a kid - or never had run into those limits; with the latter one being more feasible as hardware existence determines digital consciousness

Thy Grand Voidinesss wrote: ↑Fri Mar 01, 2024 9:24 amI must have totally forgot about such limitations ...

limitations? Limitations ?!!???
Good Grief what on earth are you talking about?
In The Good Old Days there were NO folders, just a large leather briefcase that could hold two boxes of punched cards (so about 4,000 cards)
In The Good Old Days, using a column as two six-bit bytes, we could fit 160 bytes on a punched card, so 640,000 bytes on two boxes of cards, so think about ten pounds. A 1.44 MB floppy weighed what? Half an ounce?
"Long filename support" we had, as long as you could hold a 2,000 card deck tight enough to write across the top with a marker pen.
Sigh!
Kids nowadays!
Cheers, Chris

ChrisGreaves wrote: ↑Fri Mar 01, 2024 1:18 pm just a large leather briefcase that could hold two boxes of punched cards

A briefcase. Now that's a good idea. The times someone tripped and dropped the box of punched cards and then had to put the code in the right order again ... Even with markings on the side that took a while.

1.44 MB disks were designed to fit in a chest pocket of a shirt. Problem solved.