Tag Archive for 'mysql'

How-to: e107 autogallery to Zenphoto migration

These past few days I was working on the Cool Cavemen’s photo gallery to move it to a shiny new one, powered by Zenphoto. In this post I will roughly describe how I’ve done it, code and commands included.

The old gallery was based on autogallery, a e107 plugin. We assume here that both e107 and Zenphoto are well configured and installed at the root of you web hosting space (/www in this case).

The first step is to copy the autogallery album structure, with all its content, to Zenphoto:

cd /www
cp -ax ./e107_plugins/autogallery/Gallery/* ./zenphoto/albums/

Then we delete all previews, thumbnails and XML metadatas, to keep in Zenphoto original assets only:

find ./zenphoto/albums/ -iname "*.xml" | xargs rm -f
find ./zenphoto/albums/ -iname "pv_*" | xargs rm -f
find ./zenphoto/albums/ -iname "th_*" | xargs rm -f

By now, you should be able to play with your medias using Zenphoto’s admin interface.

But if you’re unlucky as I was, you will find a strange bug which break down drag’n'drop album sorting. The fix I found was to remove, in photo filenames, the numerical prefix (and the following dot) set by autogallery to define the sort order. This operation should be performed, before the copy from autogallery to Zenphoto (= the first command in this post). By the way, if you know a one-liner to do this, please, please… share ! :)

To migrate comments, I have no automatic solution. I choose to do this manually, editing the database by hand. In my case it was the quickest way as I only had a dozen of comments to migrate.

And last but not least, if you care about measuring the popularity of your photos, you should consider migrating the view counter associated with each of your media. Don’t worry, this time I wrote a script to take care of it automagically. It will generate a bunch of SQL statements you’ll have to execute on your Zenphoto MySQL database. Here is my “e107 autogallery to Zenphoto hit counter migration script” (nice name isn’t it ? ;) ) that do the job:

#!/usr/bin/python

##############################################################################
#
# Copyright (C) 2008 Kevin Deldycke <kev@coolcavemen.com>
#
# This program is Free Software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
#
##############################################################################

"""
  Last update: 2008 aug 21
"""

########### User config ###########

AUTOGALLERY_ALBUM_PATH = "/www/e107_plugins/autogallery/Gallery"
ZENPHOTO_ALBUM_PATH    = "/www/zenphoto/albums"
ZENPHOTO_TABLE_PREFIX  = "zenphoto_"

######## End of user config #######

import os, hashlib
import xml.etree.ElementTree as ET

# Calculate hash of a given file
def getHash(path):
  # Calculate the hash from file raw data
  if not os.path.isfile(path):
    return None
  try:
    file_object = open(path, 'r')
    data = file_object.read()
  except:
    return None
  if not len(data):
    return None
  return hashlib.sha224(data).hexdigest()

# Associate each autogallery photo having a hitcounter greater than 0 with its MD5 hash
def populateHashTable(arg, dirname, names):
  global hash_table
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Get hit count for %s" % file_path
    # Check that the file as a positive hit counter associated with
    xml_file_path = "%s.xml" % file_path
    if not os.path.isfile(xml_file_path):
      continue
    try:
      tree = ET.parse(xml_file_path)
    except:
      continue
    node = tree.find("viewhits")
    if node is None:
      continue
    try:
      hits = int(node.text)
    except:
      continue
    if not hits > 0:
      continue
    # Update hash table with data we care about
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    hash_table[file_hash] = hits + hash_table.get(file_hash, 0)

# Generate hitcount SQL request for each matching file
def generateSQL(arg, dirname, names):
  global sql
  for name in names:
    file_path = os.path.join(dirname, name)
    # print "Search hitcounter matching file %s" % file_path
    file_hash = getHash(file_path)
    if file_hash is None:
      continue
    if file_hash in hash_table:
      sql += "UPDATE `%simages` SET `hitcounter`=`hitcounter`+%d WHERE `filename`=%r;\n" % (ZENPHOTO_TABLE_PREFIX, hash_table[file_hash], name)

# Core of the script
hash_table = {}
sql        = ""
# Normalize path
source_path = os.path.abspath(AUTOGALLERY_ALBUM_PATH)
dest_path   = os.path.abspath(ZENPHOTO_ALBUM_PATH)

os.path.walk(source_path, populateHashTable, None)
# print repr(hash_table)
os.path.walk(dest_path, generateSQL, None)
print sql

I think code and comments are self-explainatory. But do not forget to update in it the constants at the top of the script to match your installation paths and database naming convention.

And finally, for your information, I tested all of this on following versions:

  • e107 0.7.11
  • autogallery 2.61
  • Zenphoto 1.2
  • Python 2.5.2
  • Linux server

Website Backup Script: bug fix release

14 months after the last release, here is a new version of my website backup script. As you can see in the changelog, this version is essentially released to fix some bugs.

Changelog:

  • Check version of Python (at least v2.4 is required)
  • Rename --debug option to --verbose
  • Add a --dry-run option for testing
  • Remove use of deprecated pexpect methods
  • Add and update some error messages

Amarok 1.4.8 for Mandriva 2008.0 and repository update

amarok-148.pngI’ve just rebuild Amarok 1.4.8 for Mandriva 2008.0 with MySQL and PostgreSQL support.

This was also an opportunity for me to rebuild old packages (Interreta Televidilo, Rugg and python-icalendar) for Mandriva 2008.0. For now there is no x86_64 version of the packages (including Amarok). I plan to do it later.

Update: Amarok was recompiled to include latest libgpod version (0.6.0).

Website Backup Script: MySQL dumps and SSH supported.

Three months after the last version, here is a big update of my backup scripts for websites. The script was greatly improved and among new features, the most important is the support of backups over SSH and backups of MySQL databases.

Change log:

  • Each item of the user’s backup_list must specify the type property (FTP, FTPs, SSH, MySQLdump or MySQLdump+ssh).
  • The property previously known as site is now host.
  • File system structure changed: /ftp-mirror folders renamed to /mirror.
  • Add SSH backups.
  • The script is able to detect if a SSH connexion can be initiated without a password. This was designed for people who don’t like the idea of storing clear password in the script. Thanks to this feature, you can benefit public key authentication from OpenSSH.
  • Use of rsync whenever it’s possible for bandwidth efficiency.
  • FTP and FTPs (aka FTP over SSL) are now handled separately: this supress the default fall-back to FTP if FTPs is not supported by the remote server. This is safer as it doesn’t let lftp make the decision for you to send your clear password on the net.
  • All ports are optionnal, no need to specify it you use default ports.
  • Add MySQL backups thanks to mysqldump.
  • Two mode of MySQL backups: through SSH or direct connection to server.
  • A particular database to backup can be specified. Else, all databases are backed up.
  • Much more detailed logs that include external command’s output.
  • Auto-detect the existence of required external tools and commands at launch.
  • Use pexpect lib to simulate user password input.
  • Run all external commands in english for consistency.
  • Check that the script is running in a posix environnement.
  • Fix bug related to directory creation.

If you were using a previous version of my backup script and want to use this updated version, take care of changes, especially the ones describes in the first 3 items of the change log above.

Amarok 1.4.4 for Mandriva 2007: MusicBrainz Repaired !

amarok-144-with-musicbrainz.png

I’m happy to announce you that the latest version of Amarok for Mandriva 2007 now feature a fully functionnal MusicBrainz ! Look at the screenshot for evidences.

This build, named amarok-1.4.4-3, is exactly the same as previous one (i.e. with SQLite, MySQL and Postgresql support). Don’t forget to update the libtunepimp package from my repository and use the 5.0 version.