Index Journal Suggestions

[adamantine]

None yet, sorry

a big thank you for what you've already done
i'll continue to wait for the improvements related to index journal

especially because i began an experiment (a weak ago) during which i don't use the cenvo-rox program at all
it's still installed but it's just in case (the service was stopped, and the exe-process is absent in the task manager)
currently i don't have plans to return to that program (which has been in a constant usage since august of 2010)

from now on, any tangible changes (in the next versions of et) (concerning the index journal) are particularly important to me

 

Does Notepad2 sort?

notepad2 can do it (but enormously slowly). a fresh today's example:

- the recent file:       index-journal-2021-07-19-1.5a.txt
- 6,47 mb
- 64 663 lines
- i launched the process of sorting:       notepad2 > main menu > sort lines > sort ascending
- it took 630 seconds to sort all the lines

not even speaking about the fact that the result of such sorting (for its part) is far from very convenient
(this has been mentioned in this thread earlier)

the positive side is that usually it's quite enough for me to have the possibilities that the index journal already provides
so tactically the things are either bearable or good or even better than they were in the cenvo-rox
but strategically the things are not great

by saying 'strategically' i mean the situations when there's no chance to get the necessary info (about the actions) using the index journal itself
(and its live search)
and instead there's the only way: to use the saved info inside the txt-files   (like index-journal-2021-07-19-1.5a.txt...)

example: each rebuild means the occurrence of such 'strategic' situation
because then the index journal clears itself, and writes all its content to the txt-file

 
 
 
added 4,5 months later:
i have never used the 'bump topic' option, so i eventually decided to try at least once
but i didn't expected that this option is so lame:
it completely deleted an important info about the real/original date of this very message:      2021-07-23       22-42-39
and in 2 places it began to display the date of this bumping:      2021-12-08       22-22-01

the normal behavior of the option should have been:
1) add the info about the date of bumping (below the post)      [done]
2) don't delete/substitute the original date (above the post)      [not done]

[adamantine]

there are several small questions concerning this fresh screen-shot:
 

2022 02 17 00 44 18.png (208.38 KiB) Viewed 8350 times

 
1) 2 lines are selected in index journal
why does the 'new name' column contain the same titles? (as in the 'name' column)
there wasn't any renaming here
so the content of the 'new name' column seems to be not logical (in such cases: shouldn't this column be empty?)

 
2) index journal has the header (ID, Date, Action...)
but it doesn't look good enough (vertically, it should be at least 1 pixel smaller)
is there an option to adjust the height of this header?
(by the way, the main window of et doesn't have this tiny visual problem)

[void]

1) 2 lines are selected in index journal
why does the 'new name' column contain the same titles? (as in the 'name' column)

The new name is always shown for a move operation.
Even when the name doesn't change.
I will look into hiding the name when it does not change.

2) index journal has the header (ID, Date, Action...)
but it doesn't look good enough (vertically, it should be at least 1 pixel smaller)
is there an option to adjust the height of this header?

The index journal dialog uses the stock listview control with the default header height.
I will consider an option to customize the height.

Thank you for the suggestions.

[adamantine]

new suggestion 1:

there are context menu items: 'open path', 'copy full path'
(in several steps) they allow:

- to copy full path of an item
- then choose 'open path'
- then (in a new appeared instance of et) select all content in the search box
- and then paste this full path

i propose to add a new context menu item: 'open item'
it will allow to open this very item in a new instance of et (it certainly may be useful from time to time)
it will also allow to escape the above-mentioned tedious steps (by performing them just in 1 step)

 
 
new suggestion 2:

move the 4 buttons (open path, undo, delete, export) from the right side to the upper side
it will allow to get more horizontal space which is very often not enough here     (the screen-shot above is a good example)

these 4 buttons may be placed horizontally (not so far from their current vertical position):
- at the same line where the 'Action: All actions' note is placed now
- or (alternatively) at the 'Search:' line (occupying the very right side of this quite long line)

 
simultaneously, this old suggestion (item 5) may be implemented:

5) the upper 'Action' line could be improved
there is a lot of free space there to the right side, and thus the drop-down list is a pure non-ergonomic evil in this case
all its 11 entries could be made available directly as small and accurate buttons: horizontally placed, 1 by 1, with a corresponding inscriptions like: all actions, folder create, folder delete...

there were already plenty of cases/situations when i was missing this possibility:

not clicking at the drop-down list of actions (and then clicking again there)
but instead have all those actions as a directly available (more convenient) buttons

(1 of these buttons is in the pressed state, while the other buttons are not in the pressed state)

[void]

i propose to add a new context menu item: 'open item'

I will consider an Open menu item.
This would be useful to quickly open a document file without having to go through Windows Explorer.

move the 4 buttons (open path, undo, delete, export) from the right side to the upper side

I will consider a toolbar to show these buttons.
I agree the space is wasted on the right.

the upper 'Action' line could be improved

Checkboxes for actions is on my TODO list.


Thank you for the suggestions.

[adamantine]

after an attempt to open the index journal, this error appears:
.\src\mem.c(1077): mem_alloc(): Fatal error: out of memory 65536
which leads to a crash of et

- the OS is 32-bit (win server 2003 r2 sp2)
- ram: 8 gb
- the up-time is more than 1 month
- currently the task manager displays: 1) commit charge (limit) 8 538 852 K       2) commit charge (peak) 7 907 584 K

i had been witnessing this error for at least the whole may (and as i remember: the end of april too) (generally: more than 10-15 times by now)

this very error (or the very similar error) was also present right after an attempt to rename any folder
- this folder-related problem had been existing for the recent weeks too
- but right now i cannot reproduce this issue anymore, for some reason: the very recent attempts (to rename some folders) were successful

  
as i found out on the 4th of march-2022, you may not answer the e-mail (which i sent you that day)
so i'm not sure if i should send you my debug log.txt now

 
(much earlier this spring) i also noticed that the  db_auto_save_on_close=0  ini-setting doesn't work at all
it's probably not related to the 'memory' error, but i decided to mention it here as well (just in case)

in other words: 1) my Everything-1.5a.db file occupies 1,77 gb
                       2) no matter what i do, et unwantedly keeps writing ~1,8-1,9 gb to my storage device (absolutely each time i close the et)
                       3) this situation already ruined the way i need to use the et-program:
                           now i'm forced not to exit it (when i need to exit it): in order to save the resources of my storage device

[void]

Hi adamantine,

It's good to hear from you.
I hope you are safe and well.

after an attempt to open the index journal, this error appears:
.\src\mem.c(1077): mem_alloc(): Fatal error: out of memory 65536
which leads to a crash of et

Everything is out of memory.
The x86 version of Everything is limited to ~2GB of ram.

Please try reducing your journal size: (from memory you have yours set to unlimited)

• In Everything, from the Tools menu, click Options.
• Click the Index tab on the left.
• Change Maximum size to: 262144 KB

-if you are unable to change this setting without getting this error:

• Ensure Everything is not running (File -> Exit)
• Open your %APPDATA%\Everything\Everything-1.5a.ini
• Change the following line:
  journal_max_size=0
  to:
  journal_max_size=268435456
• Save changes and restart Everything.

as i found out on the 4th of march-2022, you may not answer the e-mail (which i sent you that day)
so i'm not sure if i should send you my debug log.txt now

Sorry, I did not receive your email.
Could you please resend your email.
Please make a post here to let me know you sent an email.

(much earlier this spring) i also noticed that the db_auto_save_on_close=0 ini-setting doesn't work at all
it's probably not related to the 'memory' error, but i decided to mention it here as well (just in case)

Do you have Tools -> Options -> UI -> Run in background enabled?
-If disabled, Everything will save your database to disk on exit.
-If the issue persists, could you please send your %APPDATA%\Everything\Everything-1.5a.ini to support@voidtools.com

in other words: 1) my Everything-1.5a.db file occupies 1,77 gb

How many files are you indexing?
I am assuming this is your large index journal.

[adamantine]

(in ini-file) for many months i had this value:    journal_max_size=4294966272           and there were no problems

even my recent screen-shot (in this thread above) contains the indication that index journal may successfully store more than 3 million lines
and after that: something went wrong at some point, but i don't know what exactly

several days ago i tried to reduce this value radically: for example, from 4294966272 to 2294966272
but a newer value didn't solve anything

 

if you are unable to change this setting without getting this error

(during these weeks) i was able to change it in the options

several minutes ago i changed the value to 268435456:      1) the problem stays the same       2) the 1,77 gb stays the same
then i tried to change the 268435456 to 168435456:           1) the problem stays the same       2) it's 1,75 gb now (instead of 1,77)

  

Do you have Tools -> Options -> UI -> Run in background enabled?

no, i never had this option enabled

How many files are you indexing?

68 000 folders and 313 000 files     (21 tb)

what is the normal size that et should save on the disk when exiting?
(according to the more or less default settings) (when the size of index journal is not huge)

 
 

I hope you are safe and well

- physically: closer to a bearable state
- mentally: not quite good
- just during 1 day (the 9th of may): 8 missiles were launched in the direction of kiev
- as earlier: everything may be lost at any moment (relatives, home, health, life...)

Sorry, I did not receive your email.
Could you please resend your email

- i don't have a copy of that e-mail anymore (since march or april)

- i will not be able to re-write it (it wasn't very small) (it also was full of pain, while in recent weeks i rather feel some kind of emptiness,
so i cannot say that i'm continuing to try to save something:
an apathy is ruling my mind now
i'm not even sure that i will be able to buy at least several hdd's that are obviously needed to make some backup copies)

those backup copies:
- should be placed in several various districts of my city (since i cannot place them abroad)
- are mainly needed to save the collection of music (which literally saved me mentally these months)

there's another air alert began right now
(of course, not a single usual citizen counts these alerts, but i think that there were more than 400 of them already)

[void]

What is the journal size reported in Tools -> Debug -> Statistics -> Journal -> Size?



There might be a content index or property index taking up a large amount of space.
What is shown in the statistics for Database (at the top)?

(in ini-file) for many months i had this value: journal_max_size=4294966272 and there were no problems

The journal could be corrupt.
Please try forcing a rebuild from Tools -> Options -> Indexes.
A rebuild will also reset your journal.

Do you have Tools -> Options -> UI -> Run in background enabled?

no, i never had this option enabled

Please consider enabling this option.
When enabled, Everything will stay running in the system tray when all windows are closed.
You can exit Everything from the system tray icon or File -> Exit

When disabled, closing an Everything window will exit Everything.
You database will be saved to disk when you exit Everything.

How many files are you indexing?
68 000 folders and 313 000 files (21 tb)

what is the normal size that et should save on the disk when exiting?

Around 50 MB for 381,000 items.

Are you indexing content or properties under Tools -> Options -> Content / Properties?

- physically: closer to a bearable state
- mentally: not quite good
- just during 1 day (the 9th of may): the fascist state of russia launched 8 missiles in the direction of kiev
- as earlier: everything may be lost at any moment (relatives, home, health, life...)

I am sorry for your situation.

- i don't have a copy of that e-mail anymore (since march or april)

My offer for storage space still stands.
I'll send you an email.
Please let me know if you receive it.

Stay safe.

[adamantine]

this is the new info concerning the 'index journal' problem:
~3 hours ago i experimentally deleted the .db-file (1,75 gb), and launched the et to build the database again. the result is:

- a new db-file occupies 1,61 gb
- index journal is working again (it only contains less than 2000 entries right now)
- indexes > maximum size > it was 164488 KB before the rebuild          then i changed it to 262144 (index journal is still working)

 

The journal could be corrupt

it's seemingly the correct assumption. but i don't know the reason for that corruption

 

What is the journal size reported in Tools -> Debug -> Statistics -> Journal -> Size?

currently:
Size: 226 125 bytes
Max size: 268 435 456 bytes
First item ID: 1
Next item ID: 1459
Item count: 1 458

 

What is shown in the statistics for Database (at the top)?

Code: Select all

Location:	D:\Programs\Every-Thing (Portable)\Everything-1.5a.db
Indexed file properties:	Name, Path, Size, Date Modified, Length, Audio Sample Rate, Audio Channels, Audio Bits Per Sample, Content
Indexed folder properties:	Name, Path, Size, Date Modified
Fast sorts:	Name, Path, Size, Date Modified, Extension, Length, Audio Sample Rate, Audio Channels, Audio Bits Per Sample
Folder count:	67 889
File count:	314 581
Total item count:	382 470
FAT index count:	0
NTFS index count:	5
ReFS index count:	0
Network drive index count:	0
Folder index count:	0
File list index count:	0
Network index count:	0
Total index count:	5
Folder data size:	4 351 529 bytes
File data size:	1 723 077 482 bytes
Total data size:	1 727 429 011 bytes
Average folder data size:	64 bytes
Average file data size:	5477 bytes
Folder index size:	1 086 224 bytes
File index size:	11 324 916 bytes
Total index size:	12 411 140 bytes
Total size:	1 739 840 151 bytes
Folders created:	0
Folders modified:	2 635
Folders deleted:	0
Folders moved:	2
Files created:	309
Files modified:	1 052
Files deleted:	96
Files moved:	116

 

Around 50 MB for 381,000 items.

Are you indexing content or properties under Tools -> Options -> Content / Properties?

yes:
- Properties: audio bits per sample, audio channels, audio sample rate, length
- Content: include only folders (19 huge folders), include only files (*.cue;*.txt), maximum size: 0 mb

taking into account this my info (about Content / Properties), what is the normal size that et should save on the disk when exiting?
should it still be around 50 mb?
this is an important question to me, because currently et keeps writing that huge ~1,7 gb (when exiting)

 
just in case:
- my task manager displays:     Everything.exe (after the launch)     >     "mem usage" - 1 718 220 K

- right after i exit the et:    the "I/O write bytes" column began displaying a quick process of disk-writing
            (it usually lasts for ~7 seconds)     (it lasts until ~1,7--1,9 gb are written,    then the "Everything.exe" line disappears)

 

Run in background       ...      Please consider enabling this option

later i'll think about it again
- the thing is that i'm not a fan of the idea to have an exe-process, and simultaneously: not to have the corresponding program in task-bar
- there's no ''system tray'' in my OS at all: i disabled it many years ago, because it's useless for me

You database will be saved to disk when you exit Everything

sounds logically
the problem appears when the size is too large. examples:        1) around 50 mb > ok        2) around 1,8 gb > not ok at all

 

Please let me know if you receive it

yes, i received it (i'll write a short e-mail--reply in several minutes)

[void]

Thank you for the db stats.

- Content: include only folders (19 huge folders), include only files (*.cue;*.txt), maximum size: 0 mb

Content indexing is the reason for such a large database.
Consider disabling content indexing or reducing the number of files that are content indexed.

I do not recommend using Everything to index over 1GB of raw text.
(I need to add a maximum content index size setting)

Content is stored in memory and in your database on disk.
The content index will be saved to disk when you exit Everything.


The journal was most likely fine. (not corrupt)

later i'll think about it again
- the thing is that i'm not a fan of the idea to have an exe-process, and simultaneously: not to have the corresponding program in task-bar
- there's no ''system tray'' in my OS at all: i disabled it many years ago, because it's useless for me

Please consider minimize on close:

• In Everything, type in the following search and press ENTER:
  /minimize_on_close=1
• If successful, minimize_on_close=1 is shown in the status bar for a few seconds.

Now, when you close Everything, instead of being closed, Everything will minimize to the taskbar.
Use File -> Exit to exit Everything.

minimize_on_close

yes, i received it (i'll write a short e-mail--reply in several minutes)

Please make a post here after it is sent.
Just is case the email is ending up in the void.

[adamantine]

Please make a post here after it is sent

it is sent

Content indexing is the reason for such a large database

i was suspecting it several weeks ago, but 1 aspect wasn't clear to me at all:

et should index the content of ~35.000 cue/txt-files (in 19 folders)          total size of these files is very small (only ~30 mb)
so the size ~1,8 gb was a total mystery

 
1 hour ago i finally understood what was the reason for this problem:

- in early march (during several nervous processes of preparing/managing some backup copies)
i temporarily copied some folders and files (to my 19th content-indexed folder)

- (in that backup) there were almost 500 old huge txt-files: their total size is slightly more than 2 gb

- but i forgot to exclude those temporary folders/files (from the content indexing)
- sorry. it was my mistake
- the current size of db-file is great (55,12 mb) (et is usable again)

[adamantine]

is my assumption correct that currently there's no way to use 'match case', 'match path'... inside index journal?

for example, the modifier case: doesn't work

[void]

The next alpha update will support the case: modifier in the index journal search.

Thank you for the suggestion.

[adamantine]

next alpha update will support the case: modifier in the index journal search

if i understand correctly, the path: modifier will also be added at some stage in the future

 
each date my copy of et keeps creating these txt-files:
...index-journal-2022-07-23-1.5a.txt,    index-journal-2022-07-24-1.5a.txt
this is the intended behavior

index journal can display the renamed folders or renamed files
but some day in the future: will it be able to display the older of them? (perhaps in some sub-window)
i mean those renamed folders and renamed files that remained in such txt-files only (and not in the index journal window anymore)

[void]

if i understand correctly, the path: modifier will also be added at some stage in the future

It's on my TODO list.
With this you would be able to search for path:c:\folder which would limit your search to just the path part.
or, search for name:abc which would limit the search to the name only.

index journal can display the renamed folders or renamed files
but some day in the future: will it be able to display the older of them? (perhaps in some sub-window)
i mean those renamed folders and renamed files that remained in such txt-files only (and not in the index journal window anymore)

I will consider a tool to browse your journal log files.
Thank you for the suggestions.

[void]

Everything 1.5.0.1316a adds a case: search modifier to all search boxes.

[adamantine]

a case: search modifier is seemingly working, thanks

"I will consider a tool to browse your journal log files"
ok, i will be waiting for this useful tool (luckily, it's not urgent)

[adamantine]

today's date (the 13th of march) is the birth-day of ET-1.5-alpha, so i congratulate David

 
this very thread is 1 of the very first topics related to alpha-version-1.5

this thread was named ''Index Journal Suggestions'' by void himself, because originally i posted my first observations/questions/ideas
even earlier than the opening message shows you all now
(initially it was published as a reply in this official thread)

 
my copy of index journal is doing fine:
- it has 5.376.000 entries currently (1.5.0.1319a, x86, 2022)

- it keeps creating the txt-files properly (the files like:    index-journal-2023-03-12-1.5a.txt,    index-journal-2023-03-13-1.5a.txt...)

- it patiently waits for all the numerous improvements that were previously discussed here, in this thread