Switching from Expression Media to Lightroom

Expression Media is a great tool. However, its future doesn’t look very bright. Since it has been bought by Microsoft and recently Phase One, no real development occurred, while on the other end Lightroom and Aperture appeared and took off, offering a much more modern user experience, including raw development, which is what I definitely need. I decided therefore to switch to Lightroom, and the challenge was to port my workflow and database that I have been defining, refining and maintaining since many years now.

My approach is the following:

  • Some fields can be taken 1:1 (like the IPTC instructions)
  • the non-standard “People” field  becomes hierarchical  keywords, like “People >> Smith John”
  • “Event” becomes hierarchical  keywords too, like “Event >> 2011 >> 12 >> 2011-12-25 Christmas”
  • EM keywords go into a Lightroom keyword sub-tree (“EMKeywords >> Flower”), with the intention to clean them up later (I didn’t use hierarchical keywords in EM)

I don’t use the Lightroom collections at all during the transfer, but I will definitely use them later in my workflow.

Here is the plugin I wrote and I used for the transfer. See the instructions at the top of the main LUA file for more information on how to use it.

Here is also most of the code inline, just for reference and the search engines:

–[[—————————————————————————-

Expression Media Importer v1.0
Olivier Croquette
ocroquette@free.fr

This Lightroom plugin will import your Expression Media data into Lightroom.

WARNING: YOU NEED TO ADAPT this plugin to your own needs !
It’s not a final product, just a basis for your own work.

Please drop me an email if this plugin was useful to you.

——————————————————————————–

I tested this plugin with iView Media Pro 2.0.2 on a Mac, but it should work for
other versions and also for iView, Phase One Media Pro, even under Windows.
I used Lightroom 4, but I believe it should work fine with Lightroom 3.

This plugin also assumes that your filenames are unique. If it’s not the case,
you will probably have to adapt it to use the full path as key for mediaitems
instead.

Lightroom is very slow at this kind of bulk changes. On my system, it is only
able to update 40 photos / second.
See http://forums.adobe.com/message/4427342

Instructions:

o in Expression Media, export your catalog(s) as XML files
File >> Export to XML

o Adapt mapEm2Lr for the simple fields you want to take over 1:1

o Adapt the code for the handling complex fields, like People, Keywords…
I transfer them as keywords in Lightroom

o Adapt the code for the handling of the location
"AnnotationFields:Country", "AnnotationFields:City", "AnnotationFields:Location"
I transfer them as keywords in Lightroom

o Select either a few photos, or select none to process all, and click on
File >> Plug-in Extras >> Expression Media Importer

Make backups ! Try on test data first !

——————————————————————————]]

require "xmlparser"

–[[
This is the mapping of simple Expression Media fields to Lightroom
I only defined the ones I need.
The EM field name is the one found in the XML file, with ":" used
as a separator between field category and field name.
For instance:

0

800

1999:09:26

Are:
AssetProperties:Rating
MediaProperties:Width
AnnotationFields:EventDate

See here for the LR field names:
http://www.robcole.com/Lightroom/SDK%203.0/API%20Reference/modules/LrPhoto.html#photo:setRawMetadata

Note that some of the EM fields can have multiple value (like People or Keyword),
and the deserve a special hard-coded handling in the code below.
–]]

local mapEm2Lr = {
["AnnotationFields:Source"] = "jobIdentifier",
["AnnotationFields:Instructions"] = "instructions",
["AnnotationFields:Author"] = "creator",
["AnnotationFields:Status"] = "title",
["AnnotationFields:EventDate"] = "dateCreated",
}

–[[
Set here the path to your XML file
]]
local xmlPath = "/path/to/file.xml"

——————————————————————————–
— You should not have to change these values
——————————————————————————–

local keywordCache = { }
local keywordIdSeparator = "::"
local progressTotalSteps = 4
local progressCurrentStep = 0
local maxNumberOfXmlTags = nil — Useful for testing

local LrDialogs = import ‘LrDialogs’
local LrLogger = import ‘LrLogger’
local LrApplication = import ‘LrApplication’
local LrTasks = import ‘LrTasks’
local LrProgressScope = import ‘LrProgressScope’

local progress

— Create the logger and enable the print function.
local myLogger = LrLogger( ‘exportLogger’ )
myLogger:enable( "print" ) — Pass either a string or a table of actions.

——————————————————————————–
— Write trace information to the logger.

local function outputToLog( message )
myLogger:trace( message )
end

— Split a string based on the given separator
function split(str,sep,n)
local sep, fields = sep or ":", {}
local pattern = string.format("([^%s]+)", sep)
str:gsub(pattern, function(c) fields[#fields+1] = c end,n)
return fields
end

— Revert an array
function revert(t)
local t2 = { }
for i,v in ipairs(t) do table.insert(t2, 1, v) end
return t2
end

— Build the media item object from the corresponding XML node
function getMediaItem(xmlnode)
mediaitem = { }

for i,subXmlNode in pairs(xmlnode.ChildNodes) do
local category = subXmlNode.Name — AssetProperties, MediaProperties or AnnotationFields
— outputToLog( "Category="..category)
for i, subXmlNode in pairs(subXmlNode.ChildNodes) do
local key = category .. ":" .. subXmlNode.Name
— outputToLog(" " .. key)

if key == "AnnotationFields:People" or key == "AnnotationFields:Keyword" then
— Array data
if mediaitem[key] == nil then mediaitem[key] = { } end
table.insert(mediaitem[key], subXmlNode.Value)
else
mediaitem[key] = subXmlNode.Value
end
end
end
— outputToLog("getMediaItem done")
return mediaitem
end

— Returns a callback used by the XML parser to cancel processing on user request
function getContinueCallback(progress)
local lastTime = os.time()
local continue = true
return function (numberOfTags)
— outputToLog(lastTime .. " " .. os.time())
if (os.time() – lastTime) >= 1 then
LrTasks.yield()
continue = not progress:isCanceled()
if numberOfTags ~= nil then
progress:setCaption("Parsing the Expression Media XML file… "..numberOfTags.." tags")
end
end
lastTime = os.time()
return continue
end
end

function incrementProgress(progress, caption)
progress:setCaption(caption)
progressCurrentStep = progressCurrentStep + 1
progress:setPortionComplete(progressCurrentStep, progressTotalSteps)
LrTasks.yield()
end

— Parse the Expression Media XML file
function parseXml()
— outputToLog( " Parsing…" )
local ccb = getContinueCallback(progress)
if not ccb() then return nil end
incrementProgress(progress, "Parsing the Expression Media XML file…")
local xmlTree = XmlParser:ParseXmlFile(xmlPath, ccb, maxNumberOfXmlTags)
if not ccb() then return nil end
local mediaitems = { }
incrementProgress(progress, "Processing the XML content…")
for i,xmlNode2 in pairs(xmlTree.ChildNodes) do
if(xmlNode2.Name=="MediaItemList") then
for i,xmlNode3 in pairs(xmlNode2.ChildNodes) do
if(xmlNode3.Name=="MediaItem") then
mediaitem = getMediaItem(xmlNode3)
— Store information by media Filename.
mediaitems[mediaitem["AssetProperties:Filename"]] = mediaitem
if not ccb() then return nil end
end
end
end
end
return mediaitems
end

— A simple wrapper that ignores nil values
function setMetadata(photo, key, value)
if not value then return end
— outputToLog(" photo:setRawMetadata " .. key .. " " .. value)
photo:setRawMetadata(key, value)
end

— Find an existing LR keyword or create it. Returns the corresponding LrKeyword object
— Hierarchical keywords are encoded into the KeywordId string using keywordIdSeparator
— as a separator
— This function caches the keywords, because Lightroom is very slow at this game.
function findOrCreateKeyword(catalog, keywordId)

— outputToLog( " findOrCreateKeyword ("..keywordId..") ; number of items in the cache: ".. # keywordCache )

if keywordCache[keywordId] ~= nil then return keywordCache[keywordId] end

— outputToLog( " findOrCreateKeyword: Item not in cache: "..tostring(keywordId) )

local idElements = split(keywordId, keywordIdSeparator)
local parentKeyword
local keywordName = table.remove(idElements, # idElements)
if # idElements > 0 then
local parentId = table.concat(idElements, keywordIdSeparator)
— outputToLog( " parentId=" .. parentId )
parentKeyword = findOrCreateKeyword(catalog, parentId)
else
parentKeyword = nil
end

local childrenOfParent
if parentKeyword == nil then
childrenOfParent = catalog:getKeywords()
elseif parentKeyword._justcreated then
childrenOfParent = { }
else
childrenOfParent = parentKeyword:getChildren()
end

— Find out if this keyword already exists in Lightroom
for _, child in ipairs(childrenOfParent) do
if ( child:getName() == keywordName ) then
keywordCache[keywordId] = child
return child
end
end

local keyword = catalog:createKeyword(keywordName , { }, true, parentKeyword )
keyword._justcreated = true
keywordCache[keywordId] = keyword
— outputToLog( " findOrCreateKeyword end number of items in the cache: ".. # keywordCache )
return keyword
end

— Call this function everytime you close a transaction, because
— the cached LrKeyword’s become invalid then.
function initializeKeywordCache(catalog, keywordList, idElements)
keywordCache = { }
return
end

function processChunk(catalog, photos, mediaitems)

initializeKeywordCache(catalog)

for _, photo in ipairs( photos ) do
if progress:isCanceled() then return end

local path = photo:getRawMetadata("path")
local filename = string.gsub(path, ".*/", "")
— outputToLog(" filename:" .. filename)

if mediaitems[filename] == nil then
outputToLog(" WARNING No information available in the XML file about " .. filename)
else
if true then
— Test and adapt to your own needs !
local locationFields = {"AnnotationFields:Country", "AnnotationFields:City", "AnnotationFields:Location"}
local locationKeywordItems = { }
for _,field in ipairs(locationFields) do
if mediaitems[filename][field] ~= nil and mediaitems[filename][field] ~= "" then
table.insert(locationKeywordItems, mediaitems[filename][field])
end
end
if # locationKeywordItems > 0 then
table.insert(locationKeywordItems, 1, "Location")
local keywordId = table.concat(locationKeywordItems, keywordIdSeparator)
local keyword = findOrCreateKeyword(catalog, keywordId)
— outputToLog(" result: ".. tostring(keyword))
photo:addKeyword(keyword)
end
end

for k,v in pairs(mediaitems[filename]) do
local k2 = mapEm2Lr[k]
if k == "AnnotationFields:People" then
for _,peopleName in pairs(v) do
— Test and adapt to your own needs !
local components = split(peopleName, " ")
local newName = table.concat(revert(components), " ")
photo:addKeyword(findOrCreateKeyword(catalog, "People" .. keywordIdSeparator .. newName))
end
elseif k == "AnnotationFields:Keyword" then
— Test and adapt to your own needs !
for _, emKeyword in pairs(v) do
photo:addKeyword(findOrCreateKeyword(catalog, "EmKeywords" .. keywordIdSeparator .. emKeyword))
end
elseif k == "AnnotationFields:Fixture" then
— You probably don’t want that code
local components = split(string.gsub(v, " ", "-", 1), "-")
while # components > 2 do table.remove(components) end — Keep only year and month
table.insert(components, 1, "Events")
table.insert(components, v)
photo:addKeyword(findOrCreateKeyword(catalog, table.concat(components, keywordIdSeparator) ))
elseif k == "AnnotationFields:Source" then
— You probably don’t want that code
local findr = { string.find(v, ‘(%a+)(%d%d)(%d%d)’) }
local year = findr[5]
if year == nil then
outputToLog(" WARNING unable to parse source for " .. filename .. " : " .. tostring(v))
else
if tonumber(year) 0 do
local currentTime = os.time()
local caption = "Updating photos (" .. completed .. "/" .. total .. ")"
if completed > 0 and startTime ~= currentTime then
caption = caption .. " " .. math.floor(completed/(currentTime – startTime)) .. " /s"
end
progress:setCaption(caption)
LrTasks.yield()

if progress:isCanceled() then return end
local batch = { }
— We create small chunks of photos to process because otherwise Lightroom becomes
— slow and unresponsive
while # photos > 0 and # batch < 1000 do
— outputToLog(" in subloop " .. # batch .. " element(s) " .. # photos .. " left")
local photo = table.remove(photos)
table.insert(batch, photo)
end

catalog:withWriteAccessDo( "Updating the LR database", function(context)
processChunk(catalog, batch, mediaitems)
end)

local endChunkTime = 0

progress:setPortionComplete(progressCurrentStep+completed*1.0/total, progressTotalSteps)
completed = completed + # batch
end

end

function startWithProgressTracking()
progress = LrProgressScope({ title = "Import from Expression Media" })

processTargetPhotos(progress)

if not progress:isDone() then progress:done() end

LrDialogs.message('Done !')

end

import 'LrTasks'.startAsyncTask( startWithProgressTracking )

Advertisements
This entry was posted in Photo. Bookmark the permalink.

2 Responses to Switching from Expression Media to Lightroom

  1. Ed Lowe says:

    Olivier-
    Search for means to transfer from EM catalog to LR4 lead me to your blog. Wanted to give your plugin a try but found the link to the plugin did not work. Is this plugin still available to help transfer the data from my EM catalog of ~ 10,000 photos into LR4?

    Thank you

    • ocroquette2 says:

      Hi Ed,

      I had deleted inadvertently this file during a cleaning action, sorry about that. I have just uploaded it again. Let me know how it works for you.

      Olivier

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s