Automator partial notes: convoluted photo management

I wonder if my hive crew have been missing my incessantly nonsensical comments or if they're enjoying the serenity.

As I mentioned at the end of the last progblog I wrote two weeks ago (x_x), I finally managed to pick up a cute laptop (I'm so used to getting the highest spec'd things that getting something practically off the shelf and only upgrading storage feels so weird and wrong) and get back into the overpriced (I'm now broke for the next 6 years) and very...um..."special" Apple ecosystem that I have this love/hate relationship with (love it because it's the one that gives me the least problems and I find their devices fun to use, and hate it because it is so very overpriced and some things make me -_-).

I've since done the major setup stuff (including using my wallpaper folder so I have less generic backgrounds which are more for the lockscreen than anything else as I rarely see the desktop) and it's just little tweaks and at some stage setting up my tablet.

The reasons for getting the small rig extended past my usual needing something mobile for the time periods where I'm going to be away from the big rig.  The small rig is now also going to be housing the photo library partly as it's easier to drag around to show photos to relatives, and mostly because I've been having a lot of problems with importing photos on the big rig.  Initially I thought that the photos weren't importing and that I'd lost a batch of them, which resulted in me not deleting anything off my phone just in case.  Eventually I worked out that some of them were going into the wrong date folders but couldn't work out what I was doing wrong even after fiddling with settings.  As both shotwell and digikam were doing the same things I decided there was probably something interesting going on with gphoto which both use as a backend and continued hoarding images on my phone as I had no idea where the missing photos were ending up.

My initial thoughts were to just import everything into Photos and sync the originals folder, which once upon a time used to just be date based like everything else.

No longer.  Now it looks like

and has probably been like that for quite a while but I haven't used a Mac since my 2011 iMac.  I could have synced that but I would have never been able to find anything.

So I grabbed a selection of files and started running some tests trying to work out how the hell I was going to make what I thought would be a relatively straightforward job work.

_{Pro-tip: nothing is ever straightforward when you mix operating systems.  Different versions of the same operating systems also count as different operating systems.}

After much stuffing around, I eventually decided that the easiest thing to do would be to import everything into Photos, set up a smart album to collect the latest imports (because currently the Imports view seems to show everything that was ever imported separated by the dates they were imported which requires me being more careful about clicking than I want to be) and export everything in that album to a sorting folder (not to be confused with a sorting hat) which will then trigger an Automator folder action and sort them nicely into my cloud sync folder in the date based structure I wanted and have been using for years (since 13th of August 2004 to be precise).

A few hours playing with Automator later and I realised I was going to need a shell script.

Do you think I could find a shell script that did what I wanted?

I spent a week scouring the internet and learning just enough zsh scripting to be a total and complete hazard to myself and everyone around me get the test folder sorting nicely.

I also needed exiftool to get the file creation dates I wanted.

To make the Automator thing:

1. make the sorting folder where you like (mine is just in my home directory on the same level as the Nextcloud sync folder for no other reason than convenience)
2. make a new Folder Action in Automator
3. in Folder Action receives files and folders added to dropdown, locate your sorting folder
4. put in a Get Specified Finder Items node.  If you haven't used Automator before you'll probably get a warning dialogue at some point saying that you need it to test the folder action in Automator and should remove or disable it when using the actual folder to trigger the folder action.  However I found that for whatever reason removing or disabling the node stopped the folder action from working at all
5. select or drag and drop your sort folder into the box
6. put in a Get folder contents node
7. put in a Run shell script node
8. change Pass input to as arguments

Copy this into the script box.  If you run it in Automator it should spit out a bunch of comments to tell you what's going on while you're testing, or it will just do its thing if you just use the folder action.

ALWAYS TEST ON A TEST FOLDER WITH TEST FILES FIRST.

for f in "$@"
do
        # use exiftool to get the creation dates we want
        # sometimes all exif data is missing in which case we have to fall back
        # to a file system create date
        #
        # the multiple -s gets rid of some labels that exiftool helpfully
        # supplies for if you're using it on command line

	device=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" "$f")
	device2=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal "$f")
	exif=$(/usr/local/bin/exiftool -s -s -s -createdate "$f")
	fs=$(stat -f "%SB" $f)

        # now we're going to break down the dates into year/month/day variables
        # and populate starting from the least desirable file system create date
        # and hopefully override with the more desirable dates further down

	if [[ -z $fs ]] # -z means "empty" and if this first one is empty there's problems XD
	then
		echo wtf how does $f:t have no fs create date
	else
		y=$(stat -f "%SB" -t "%Y" "$f")
		m=$(stat -f "%SB" -t "%m" "$f")
		d=$(stat -f "%SB" -t "%d" "$f")
                
		echo $f:t has filesystem create date $y $m $d
	fi

        # some of the files I was working with had zeroed exif dates for some reason
        # the next few ifs check for empty variables or if they start with four zeroes
        # in which case it counts as empty

	if [[ -z $exif || ${exif:0:4} == "0000" ]]
	then 
		echo $f:t has no exif date
	else
		y=$(/usr/local/bin/exiftool -s -s -s -createdate -d "%Y" "$f")
		m=$(/usr/local/bin/exiftool -s -s -s -createdate -d "%m" "$f")
		d=$(/usr/local/bin/exiftool -s -s -s -createdate -d "%d" "$f")

		echo $f:t has exif date $y $m $d
	fi

	if [[ -z $device2 || ${device2:0:4} = "0000" ]]
	then
		echo "$f:t has no device2 date"
	else
		y=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal -d "%Y" "$f")
		m=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal -d "%m" "$f")
		d=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal -d "%d" "$f")

		echo $f:t has device2 date $y $m $d
	fi

	if [[ -z $device || ${device:0:4} == "0000" ]]
	then
		echo "$f:t has no device date"
	else
		y=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" -d "%Y" "$f")
		m=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" -d "%m" "$f")
		d=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" -d "%d" "$f")

		echo $f:t has device date $y $m $d
	fi

        # change this to the path you want the files to be sorted into
        # (which is obviously going to be different from your sort folder just in case
        # that wasn't clear from the outset)

	target=/path/you/want/things/sorted/to/$y/$m/$d

        # if the target directory doesn't exist
	if [[ ! -d $target ]] 
	then
		echo $target created, moving $f:t

		mkdir -p $target # make all the folders		
		mv $f $target # and move the file to the target folder

        # if the target directory does exist
	else
		echo $target exists 

                # check to see if there's a file with the same name in there              
		if [[ ! -e $target/$f:t ]]
		then
                        # if there isn't, move the file
			mv $f $target
			echo moved $f:t to $target
		else
                        # if there is, we're going to add a number to the file name
			echo file called $f:t already exists in $target
			
			n=1

                        # first we check to see if there is already a filename-nn.ext
                        # if not, we rename the file and then move it
			if [[ ! -e $target/$f:t:r-$n.$f:t:e ]]
			then
				echo n is $n
				mv $f $target/$f:t:r-$n.$f:t:e
				echo renamed to $f:t:r-$n.$f:t:e and moved to $target

                        # if there is a filename-nn already in the folder, we add 1 and try again
                        # until we get a unique filename and then rename and move the file
			else
				while [[ -e $target/$f:t:r-$n.$f:t:e ]]
				do
					echo file called $f:t:r-$n.$f:t:e already exists in $target
					(( n++ ))
					mv $f $target/$f:t:r-$n.$f:t:e
					echo renamed to $f:t:r-$n.$f:t:e and moved to $target
				done
			fi
		fi
	fi	
done

There's more information on getting bits out of the path (the $f:t etc bits) and the test operators (the -z, -e etc bits) that I found in my travels that helped immensely.

The above can be modified to a photos cleanup script by ticking the Repeat for each subfolder checkbox in the Get folder contents node and adding in a Filter Finder Items node straight after it and before the script node.  The filter you then need is Kind is not folder.

I did try this out but it turned out a lot of my older photos have incorrect date information so it didn't work out that well in my case and I'll have to find and fix problems manually on the Nextcloud server when I can be bothered.

There are probably much better ways to write this (including  theoretically adding ! to the conditionals and removing all the echos if you just wanted it to do its thing and don't care about feedback) but I suck at coding, it took me way too long to come up with this and even longer to try and fail to get the cleanup version working.

I have tested it out with a manual export (because after importing everything into Photos the export smart album of course contained a lot of things because a lot of things had been imported "in the last day") and everything seems to be working as it should.  The only minor panic I had was that it exported heic files (probably because I ticked the box to export unmodified originals), but digikam is able to read them.

Can't say the same about the Nextcloud webapp as browsers haven't caught up yet but that's fine XD

Now I can get back to art and everything else I've been neglecting while getting this sorted x_x