#then they had SERVERSIDE ERROR or something each time i tried to do their test thing to prove im human

[key-to-the-truth]

guys they fucking killed me over there

[hydrus]

Version 421

windows

zip

exe

macOS

app

linux

tar.gz

I had a good week. I fixed some important bugs, and cleaned up some core access and maintenance code. This week's changelog gets pretty technical, which you can safely ignore. Database works better™.

highlights

I fixed a problem importing files with 'only add tags that already exist' checked in tag import options. Sorry, this was a stupid typo. I added a unit test here to ensure it doesn't happen again.

When you search for potential duplicates from the duplicates page, this now happens in a non-interrupting popup in the bottom-right. You can keep searching and browsing while it works. All duplicate pages sync with each other better, too, and share more CPU work.

Thanks to a user's contribution and help, the 'discord drag and drop' BUGFIX mode, which is really a 'export files to a temp location before starting the drag and drop' mode, now supports the same 'filename pattern' you see in the export files dialog. Check it under options->gui. This is experimental, but the first step to automatic export with nice filenames. If you care about export filenames, let me know how it works for you, and what sorts of filenames you want to make in future. I really want to update the old 'filename pattern' system to something with conditional 'if' statements.

If you are a new user and had trouble syncing to the PTR in the past week with an error about an invalid tag, you got caught by my new 'invalid tag catching' code being overzealous. This is now fixed, so if your PTR automatically paused due to the error, please hit services->review services->PTR and then unpause it. When a repository gives an invalid tag (we think there are about 17 really broken tags in the PTR atm, from older days), the client now recovers.

I played around with core database modes this week. Default values are now tuned a little better for modern use. Unfortunately, the planned new mode for HDD users did not pan out due to architectural problems, but I think that HDD users will experience better write performance nonetheless. For those who want to experiment more, I have finally properly documented the different launch switches for hydrus here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

full list

misc:

thanks to a user's contribution, added the export 'filename pattern' to the discord drag and drop mode, under _options->gui_. this lets you auto-rename files in this export mode. I like how this works, but the overall pattern-based filename creation system really needs updating. let me know how this works for you, and I'll finally start the job to update filename generation

fixed a bug when importing files with the 'only add tags that already exist' filter active, and added a unit test so this should not fail due to a typo again

fixed an issue where ctrl-selecting on taglists was weird, where any mouse movement during ctrl+click would deselect. drag select and deselect can now only start when the drag crosses two indices

prototyped a basic profile mode for the client api. it is insufficient (due to the asynchronous nature of twisted), but a start

when the client catches an invalid tag with the new error handling code, when it shows you that bad tag in a popup, it now clips that to 24 characters (some PTR invalid tags are just a few hundred null characters in a row, wew lad)

the client now recovers from a repository giving it a new invalid tag definition. all such tags are, for now, called 'invalid repository tag'. a plan to auto-hide these tags clientside and fully eliminate them serverside will come later

the clipboard url watcher settings should stick a bit more firmly. those users who had trouble, please let me know how you get on

fixed an issue editing duplicate action options when they contained tag or rating preferences for services that no longer exist

I think I fixed some issues getting autocomplete results when you type the whole namespace before moving on to the subtag. when you hit 'namespace:', it should invalidate the old cache and start a new search

when the database is given content updates for services that no longer exist, those content updates filtered out of UI update broadcast

fixed an issue where URL status check could fail when the url map contained orphan hash_ids. proper orphan clearance will come later

reduced overhead of tag filtering, which should improve display speed of taglist for very large pages

parents should now work through repository processing faster. periods of 2 rows/s at the end up of updates should be up to 100 times faster

.

duplicates search improvements:

potential duplicate search now works in the background! it will not interrupt you and is easily cancellable. duplicate search pages disable their search buttons while it is going

the search distance in duplicates pages is now synchronised across all pages--when one updates, they all do

all the updates to potential search maintenance numbers are now routed through one cached manager. updates here are repeated less often

misc cleanup for duplicates page

.

database modes:

a new 'program launch arguments' help page now talks about all the available command line switches, here: https://hydrusnetwork.github.io/hydrus/help/launch_arguments.html

added the '--db_journal_mode' launch switch to set the SQLite journal mode. default is WAL, permitted values are also TRUNCATE, PERSIST, and MEMORY

ensured --db_synchronous_override was hooked up correctly

the old disk cache options under _speed and memory_ are removed, along with various deprecated disk cache load calls and code

fixed some shutdown maintenance check logic that was saying 'I think a vacuum is due' when it wasn't actually true

db_journal_mode, synchronous value, and no_db_temp_files is now shown in _help->about_

.

technical database nonsense:

PERSIST is new to hydrus, and _may_ in future versions of SQLite be boost performance for HDD drives with larger databases (e.g. those that sync to the PTR), although unfortunately in our case (which uses multiple ATTACH databases), it seems current SQLite must ultimately treat this as DELETE, as here https://sqlite.org/atomiccommit.html#_clean_up_the_rollback_journals. damn

hydrus now tries to always trim WAL (and PERSIST, if it worked) journal files down to 1GB after commits (which happen every 30 seconds), so giganto WALs should clear up promptly after big work is done

hydrus no longer refreshes the database connection every thirty minutes, meaning WAL journal files will persist (and hopefully regularly clip back to 1GB when exceeded), which should improve some elements of long-running write performance, but may result in some surprise memory issues, we'll see

in lieu of the db connection not refreshing, the memory database now reattaches every ten minutes, which _should_ stop it leaking in certain situations

when in WAL journal mode, the hydrus db now cleans up any lingering checkpointing work every half hour

after testing and feedback from users, the database is now default SQLite synchronous 1 (down from 2) when in WAL. the db is still consistent, so sudden program stop (crash, power cut) should not result in software-caused corruption, but the database may lose more than just the last 30 seconds of work. this speeds up tag processing in an SSD test environment by approx 33%

the 'no_wal' (TRUNCATE) and 'db_memory_journaling' (MEMORY) launch switches remain valid but are now deprecated

improved launch switch code generally

boosted cache size for each of the four db files to ~200MB-this will likely become a launch argument in future, along with some other specific db values

the client and server no longer disconnect from the db to check whether it is possible to vacuum databases

next week

There are two more work weeks in the year. I will continue working on small jobs and cleanup, but the main focus is now to update ancient network and service code to improve client-PTR sync and communications. I'll fix up buggy janitor tools and hopefully add some nice filters so clients and servers can better manage what they sync.