Category Archives: Research

GraphicConverter to the rescue!

Thorsten Lemke, author of GraphicConverter, seems to be very responsive when it comes to feature requests and bug fixes for his product. When I earlier encountered an apparent memory leak with the “JPEG -> Remove double orientation tags and reset them” feature, he quickly provided me a beta to try out, which helped somewhat with this problem. (It will still crash on tons of photos, but not for the same reason. The memory leak seems to be gone.)

Anyway, after posting to the GCMAC mailing list, I have been contacted by two folks from Lemkesoft, and now I am trying out a new beta which seems to make the routines that normal just see EXIF also see the “PictureInfo” date. In the “ExifTool” window of GC is a section for File (with modification time), then one for JFIF (resolution and such), then PictureInfo (which has the info from my old Epson camera — I specifically see the date/time and “Sierra Highland” which is a string the camera embedded in the pictures, and one you could customize with the PhotoPC software.

Using this new beta, I can now simple select some photos and do “JPEG -> Set EXIF Date to File Creation Date” which seems to honor the PictureInfo date (though the wording implies it would use the file system Creation Date, so maybe this needs to be changed).

There may still be some issues — in testing, I find it is now adding multiple “File data and time” entries to EXIF, so I’ll work with Lemkesoft on this until we get it hammered out.

Very nice response!

JFIF in iPhoto

EXIF is a specification for digital camera files that allows them to store information in the photo, like when the picture was taken and if the flash was used. Before the EXIF standard, early digital cameras (circa 1996) would use a JPEG header extension known as JFIF to do something similar. My understanding is that this was not standardized.

Here is the Wikipedia article on the JPEG, which mentions JFIF.

Older photos I took with my Epson PhotoPC do not have EXIF information in them, and the camera import software back then defaulted to a DOS-style naming convention of MMDD_NUM.JPG. (MM was the month, DD was the day, then a three digit incrementing counter. I guess they never expected someone to take more than 999 images in a day, which was difficult since the PhotoPC could only store up to 99 images before you had to download using a slow serial cable connection. But I digress…)

Because of this naming limitation, looking at a file like “0503_001.JPG” doesn’t tell you the year it was taken. Over the years, I accumulated thousands of photos taken in May, and when they are imported in to modern software like iPhoto (which supports EXIF, but does NOT see the embedded JFIF codes), the pictures all get lumped together on the date the file was last modified. (In the included screen shot, a photo I have from June 11th 1997 shows up on August 21 2001 since, I guess, that’s the last time I did any copying of these images files or something.)

To make matters worse, many of my photos got renamed to things like “DL0004.JPG” (a batch of photos from Disneyland) with no indication of the month or date. Good luck!

The problem is that modern software, such as Apple’s iPhoto or Lemkesoft’s GraphicConverter do not look at JFIF information (nor should they, since we now have the EXIF standard to support). What is needed is a way to convert files with JFIF information in to files with EXIF headers.

If you have an image without any EXIF data, GraphicConverter and iPhoto will let you modify the date and embed an EXIF header. This is required for any photos that were scanned (thus no EXIF or JFIF), or maybe had JFIF/EXIF but were manipulated with older graphics programs that did not honor it and thus destroyed it. (Any photo I rotated, for instance, lost any JFIF information, so I have thousands and thousands of photos with absolutely no indication of when they were taken inside the files.)

Nothing but manual editing can be done about files with neither JFIF or EXIF data, but for the old pics that at least have JFIF I have found a few steps, using a few programs, that will get all my old non-EXIF images converted to images with EXIF. The EXIF data will either be the correct information brought over from JFIF headers, or it will be some bogus information based on a file timestamp or whatever. In that case, manual editing is the only thing you can do.

Here is what I posted to the GCMAC (GraphicConverter) Yahoo! mailing list today documenting my steps, and it ties in to my previous post.

1) Use EXIFrenamer to set the "Creation Date" of all my files to whatever JFIF information it can find. Some
files contain none, so the creation data will be left unchanged (whatever date I copied the files over originally, I guess). Some files have bogus JFIF dates set to 1969 or other weird years.

1) I will use GraphicConverter’s "JPEG -> Set EXIF Date to File Creation Date" option to recurse through all my files. Now each JPG will contain either a valid EXIF (along with a valid JFIF?), EXIFs with bogus dates brought over from the JFIF, and EXIFs that are just whatever file system timestamp unknown files were given.

To verify, I drag the entire batch over in Mac OS X’s "Preview" app and open up the Inspector (Tools -> Inspector, or Apple-I). I switch to the center "! in a talk balloon" icon tab where I should see: "General | JFIF | Exif | TIFF" I quickly page through my images making sure all have EXIF sections now, valid or otherwise.

3) I import these images in to iPhoto, which now can rely on the EXIF tag instead of some filesystem date.

I just tested these three steps "fresh" and imported them in to an empty iPhoto Library. It seems to have worked!

Now I have three smart albums, one for 1997, one for 1998, and one for 1999 photos, then I created one called "Before 1997" and one called "After 1999" which will contain the photos I need to manually adjust.

I hope this information helps someone else! Contact me if you have any questions. I’d be happy to help.

Epson PhotoPC and EXIF JFIF dates

What you will need:

* EXIFRenamer – from qdev.de (shareware).
* GraphicConverter – from lemkesoft (shareware).
* iPhoto – from Apple (free with Mac)

Once again I find myself searching the internet to solve a particular problem that I seem to be the only one trying to solve, so I will share the problem here, and my solution, in hopes that it may help someone else.

In 1996, I purchased an Epson PhotoPC digital camera. At that time, no one had ever heard of a “digital camera.” I had to call it a “computer camera” when explaining what it was to folks who asked.

Between 1996 and 2000 I used this camera and took thousands of photos as I travelled for work, sharing them on my sites DisneyFans.com (theme parks) and AtTheFaire.com (renaissance festivals). Today, several camera models later, I have well over 100,000 photos online.

In the early days of digital photography, there was not yet a standard to how cameras were going to embed information about the photo inside the JPEG file — such as the date and time the image was captured. With no displays on the early cameras, the only way camera time was set was through a serial cable hooked up to a PC or Mac. In the case of the Epson PhotoPC, it used some form of JFIF header information for the date/time. If you try to view a picture taken by the EpconPC in modern graphics software such as GraphicConverter or iPhoto, it will not show any date information. Apparently, there just is no EXIF data in the file.

I have found that the excellent EXIFRenamer utility for the Mac can parse the JFIF information, allowing files that have it to be successfully renamed and even have the date created filestamp updated.

BUT, I found out too late that early graphics programs would destroy this information when an image was rotated. Modern utilities know about all this stuff and will preserve EXIF data, but back then, it wasn’t well supported.

So now I find myself trying to finally sort thousands of my old personal photos and get everything all updated. A huge portion of my old photos no longer contain JFIF date information (due to being rotated) and none contain EXIF. They might as well be scans from paper pictures!

I began importing my entire picture collection in to iPhoto and noticed immediately the pictures showed no date information — just a date the file was “last modified.” This caused a ton of pics to appear in year 2001 for me. I then proceeded to spend all night trying to manually adjust dates using the iPhoto feature that will do this (a very powerful option, indeed!).

The end result was hours wasted trying to figure out if “0403_004.jpg” was taken in 1997, 1998 or 1999 with some success, and many obvious failures. It was a painstaking process of visual clues — did I have eyeglasses on? Full beard or clean shaven? Which apartment was I living in at the time? After finally getting about 95% “confirmed” (and finding out I have several galleries on my photo sites labeled by the wrong year), I decided there had to be a better way before I tackled my next batch of photos.

So here is my down and dirty solution to this challenge, using iPhoto, GraphicConverter and EXIFRenamer.

1. Some photos DO contain JFIF date code information, which neither iPhoto or GC can read, but EXIFrenamer can. So, I begin by using a feature of the renamer to rename all images to a temporary name (prepending an “_” to each file) and updating the creation date of the file. Set up a simple custom rename style of %P%o%C%S%F (it really only needs %P for prefix and %O for complete filename, but I use this for other things too):

Be sure to set the Processed Files tab to “Change File Creation Date to EXIF Date”:

2. Rename using EXIFRenamer and this setting. After the files dates have been updated, I then process them through renamer again to get rid of the “_”, setting them back to normal. (I couldn’t find a way to process them without renaming.)

3. Above, “%1U%F” uses a substring then the file type extension, and below you set Substring #1 to “mid (pos)” with “1” for par1 – this starts from the first character, so it skips the “_” character.

4. I then use a feature of GraphicConverter to adjust the modification date of files to the creation date.

5. Lastly, I import things in to iPhoto, which will then honor the modification date (it does not seem to care about creation date).

From here, I can then go back to using iPhoto to adjust the images that are showing up with undefined dates (1969, for some reason, or times in 2001 which must have been the last time I copied all these files over or something).

It’s a huge amount of work, still.

What I Need

I am still looking for a tool that will recursively add EXIF information to JPGS that do not contain EXIF, by using JFIF header information (if available). There is bound to be something out there that does this. Right now, I’m doing the above steps, then using iPhoto to manually set datecode on “missing” photos, which means only they now have EXIF and this process will need to be repeated for the other files if they are ever touched in a way that loses their modification date, or sorted in another tool that relies on EXIF.

Anyone?

Some Screen Shots:
Original image using old file system date:

New image after process (ignore the long filename):

Note that the date is wrong and bogus (1970) but it has at least been updated. For photos that still have correct JFIF date information in them, it will be as accurate as the camera’s timestamp was.

EXIFRenamer settings – two things must be set to do the rename and adjust the file creation date:

EXIFRenamer settings – two more things must be set to create a pattern to rename, but ignore the leading “_” character we just added to all the files:

GraphicConverter – open the folder containing all the images you wish to process, and Apple-A to select ALL, then do a right-click “DATE->Set Modification Date to Creation Date”. (iPhoto uses the modification date, so this extra step is necessary.)

FireWire to SATA adapters for the Mac

My Situation:

I currently have 14 hard drives hooked up to my iMac. I am using three DAT Optic qBox-f enclosures that hold four IDE hard drives, each loaded with 500GB Seagates. These enclosures support FireWire 400 and 800. I hook them to my iMac (which only has FW400) and then daisy chain them through the FW800 ports on the back of the enclosures.

And the end of the chain is two bay enclosure that supports FW400 and USB. That gives me 14 active 500GB drives online for video storage. (I also have a few other 250s and 200s I swap in and out from a single case, when I need to.)

The Problem:

I’m out of disk space, and the newer 1TB-size drives do not seem to be available in IDE. The world is moving towards SATA drives. SATA has been used internally in Macs for years now, as well.

I am also out of desk space, so adding more of these qBox-f enclosures and more 500GB IDE drives isn’t really what I want to do.

The Goal:

I now want to migrate my entire system from 500GB IDE drives to 1TB+ SATA drives. If I wanted to keep my existing enclosures, DAT Optic makes an adapter that will plug on to the back of each SATA drive and give it IDE hookups. At $16/each, though, I’d rather just find new SATA enclosures and sell what I have and start fresh.

The Roadblock:

While two bay FireWire enclosures could be found for $60, and quad bay enclosures for under $200, once you step up to SATA, for some reason FireWire support is massively expensive. I’ve seen quad bay FW/SATA enclosure for well over $400, which is out of the question on my budget.

USB to SATA enclosures are abundant, and can be found for similar prices as IDE models, but USB is more CPU intensive and slower than FireWire. When dealing with multi-track audio (in GarageBand) or video (in Final Cut), I already run into situations where I get the dreaded “too many tracks to realtime preview” errors. So, even if USB only adds 5% more CPU time, and is only 1% slower, that would still impact my workflow.

The Plan:

I plan to find a way to use the FireWire ports of my iMac to hook up enclosures that contain SATA drives.

This is a log of my progress so far.

My Progress So Far:

One of the advantages SATA has is that they defined the port connectors and spacing. This allows for drive enclosures where you can slide a SATA drive in without having to deal with cables. I discovered some trayless enclosures that let you pop open the front and just slide drives in, as if they were cartridges in an old Atari 2600.

I did a search on Amazon and found the v7age220-EF 2-bay enclosure made by iStarUSA. It could hold two SATA drives, and hook up via FW800 or USB. It was also trayless, meaning you could open up the two doors on the front and just exchange drives, without using any tools.

The problem was the price – $179. Sure, it had more indicator lights, and supported some RAID stuff (without having to use a disk utility to reprogram the case, like some of my others), but that was over $100 for what I’ve found similar IDE enclosures.

The company selling the drive through Amazon was GoHardDrive.com. They had it directly from their site at $149 with free shipping, so I went ahead and placed an order. Unfortunately, after a week, I checked and noticed it still hadn’t shipped, and the product page now said they were out of stock. I contacted GoHardDrive and they apologized, and cancelled my order.

Instead, I purchased the USB version of this case – the v7age220-SAU. Instead of FW/USB (like the -EF model), the -SAU model supported USB and eSATA. And, it was only $85!

So, at least according to iStarUSA, having FW adds about $80 to the cost of an enclosure. I’m not sure I buy that, since even CompUSA (high price retail store) sold single-drive FW/USB enclosures for less than $60 (for the entire enclosure and power supply).

I received the v7age220-SAU and it worked great. It was quiet, nicely built (except for the front doors which seem pretty flimsy) and fast. But, I still wanted a way to use FireWire.

FireWire to SATA – The Quest Begins:

While I was able to find tons of USB-to-SATA adapters dirt cheap, at first I couldn’t find anything that did FireWire-to-SATA.

Eventually, I discovered two converter boxes sold by CoolDrives.com. I have not been able to find these sold anywhere else on the Internet, which is kind of bothersome.

The first is a tiny box, currently sold for just $30, that adapts USB or FireWire to a SATA port:

http://cooldrives.stores.yahoo.net/sata-to-firewire-usb-mini-converter.html

The second model was similar, but had an additional FW pass through port, so you could daisy chain. It sold for $60 (which seems unreasonably high just to add a second FW port):

http://www.cooldrives.com/satofimiad2x.html

The $30 version said it only worked with drives up to 500GB, while the $60 model claimed 1000GB (1TB). I wrote to ask if this was true, and CoolDrives.com said they would have to check. (I haven’t heard back yet.) This means the first model is instantly not usable with the modern size drives (currently shipping 1TB and 1.5TB drives) which seems odd.

But, my idea was, I could get one of these, then plug up a cheaper SATA-only enclosure, like the iStarUSA v7AGE420-ES 4-Bay Trayless eSATA RAID Box External Hard Drive Enclosure for only $165! Now we’re talking! I could actually get a better (?) four drive enclosure that was then use a cheap $30 adapter box. The end result would be FireWire performance from the iMac to cheaper SATA high capacities drives.

But would that work? The answer was NO, and I’ll explain more in a moment…

Also, a co-worker pointed me to this device: FirmTek SeriTek/SpyderHUB

It was designed for this very purpose – to hook a Mac with FireWire up to SATA enclosures. It had two SATA outputs, and accepted both FW and USB as input. But, the adapter alone was $129. It also noted it was not compatible with “Port Multiplier” enclosure. This was a new term to me, and I did some research.

Port Multiplier is the magic that lets one SATA connector then go off to multiple drives. The iStarUSA enclosures are PM devices. In JBOD (just a bunch of disk) mode, each of the drives inside would appear as a separate hard drive, but only if the SATA controller on the computer supported PM.

Long Story Short: (too late!)

It turns out, the $60 box from CoolDrives.com does not support PM devices. Plugging it up to the v7age220 USB/SATA box only shows up the top drive. (I speculate that if I ran these enclosures in a RAID or BIG mode, they would appear as one massive device to the computer, but I have not tried. I do not want this. If one drive fails, I don’t want it taking out my whole array. I’d rather have separate drives I can back up separately.)

Solution:

None yet, but if you can deal with the performance of USB, getting a hub and some of these iStarUSA enclosures may be a decent way to go (but it will cost more than similar IDE enclosures would have).

Future:

I have done extensive benchmarking of drive performance using USB to SATA adapters, the CoolDrives FW/USB adapter, and running straight to the iStarUSA enclosure (all to the same Western Digital WD1001 drive).

I will eventually be posting all the results in a nice table here (or maybe a YouTube video of all this).

If you found this page, and this is interesting to you, please drop me an e-mail. Also, if you have better answers or solutions, please let me know.

Thanks!

References:

iStarUSA – makers of the enclosures. They have yet to return my e-mails requesting information on who sells the FW models.

GoHardDrive.com – discount seller of the iStarUSA enclosures. They have been very responsive to my e-mail requests, but when I ordered, the item stayed listed as “Processing” for a week! When I checked the last day, it suddenly was “Shipped” and tracking showed it was expected that day. They actually shipped it the same day I ordered, but the log did not reflect that.

CoolDrives.com – The only place I’ve found the FW to SATA adapter. I read much bad about them, but my experience was just fine. They shipped immediately and I had no issues.

FirmTek – Makes of a Mac USB/FW to SATA converter. Note they designed this to work specifically with enclosures they sell, and do not promise it will work with any other enclosure. I have no experience with them, but if they’d build in PM support and keep the price similar, I’d buy one today!