Release Procedure

From Open Clip Art Library Wiki

Jump to: navigation, search
This section is deprecated. You can help by updating it, or possibly deleting it..

Contents

[edit]

Creating the Release Itself

See also the old release procedures. This is the procedure jonadab tested and used starting with the 0.09 and 0.10 releases:

[edit]

Determine the new number 

new_rel_number = old_rel_num + 0.01
[edit]

Rename directories

  • Get the previous openclipart package and unpack it into a directory. Rename the directory to openclipart-relnum. This is your main working directory for generating the release. 
old_rel_num=0.17  # Change to whatever the last release number was
 new_rel_num=`perl -e "print $old_rel_num + 0.01"`
 wget http://www.openclipart.org/downloads/$old_rel_num/openclipart-$old_rel_num-full.tar.bz2 
 tar xjf openclipart-$old_rel_num-full.tar.bz2
 mv openclipart-$old_rel_num-full openclipart-$new_rel_num
  • Rename the incoming directory on the server and start a new one 
ocal_web_dir=/srv/clipart.freedesktop.org/clipart_web
ssh openclipart.org "mv ${ocal_web_dir}/incoming ${ocal_web_dir}/incoming-pre-${new_rel_num}"
ssh openclipart.org "cd ${ocal_web_dir} && mkdir incoming && chgrp clipart incoming && chmod g+rwx incoming"
ssh openclipart.org "cd ${ocal_web_dir} && getfacl incoming-pre-${new_rel_num} | setfacl --set-file=- incoming"
  • Obtain the contents of incoming-pre-relnum from the server and place them into the subdirectory incoming within the main working directory mentioned in step 3, i.e.,

Put the new incoming files into openclipart-relnum/incoming

  • Here are Bryce's notes on one way to obtain the files. Rsync down the incoming directory: 
rsync --delete -av -e ssh openclipart.org:${ocal_web_dir}/incoming-pre-${new_rel_num}
  • I just used wget. That does also get a couple extra files you don't need and can just delete, such as index.html 
wget -d -c --limit-rate=20k -r --level=1 -np http://openclipart.org/incoming-pre-relnum/
[edit]

Scripts

You will also need the scripts in the openclipart-tools package, if you don't have them already, and the SVG::Metadata module. 

wget http://www.openclipart.org/downloads/${old_rel_num}/openclipart-${old_rel_num}-tools.tar.bz2
 tar xjf openclipart-${old_rel_num}-tools.tar.bz2
 OCALBIN=$PWD/openclipart-${old_rel_num}-tools
  • If there are zipfiles or tarballs, unpack them, propagate their metadata (using the propagate-metadata.pl script), add any relevant keywords to their contents (using the keyword script), and then place their SVG files into the same incoming directory with the others.

To unpack all the zip files: 

mkdir ZIPS

for pkg in `ls *.zip 2>/dev/null`; do 

echo ${pkg}
pkgname="${pkg%.zip}"
mkdir ZIPS/${pkgname}
unzip -q ${pkg} -d ZIPS/${pkgname}

To unpack all the tarballs: 

mkdir TARBALLS

for pkg in `ls *.tgz *.tar.gz 2>/dev/null`; do 

echo $pkg
pkgname="${pkg%.tgz}"
pkgname="${pkgname%.tar.gz}"
mkdir "TARBALLS/${pkgname}"
tar xzf $pkg -C "TARBALLS/${pkgname}"
  • To add metadata to everything 
errorfile=errors-$$.log
echo "" >  $errorfile

for dirname in `ls -d ZIPS/* TARBALLS/*`; do 

echo "Processing $dirname..."
        if [ -e ${dirname}/metadata.rdf ]; then
            echo "foo" # `cd $dirname && $OCALBIN/propagate-metadata.pl 2>> $errorfile`
        else
            num_files=`find ${dirname} -type 'f' -name '*.svg' | wc -l`
            svg_check_metadata $dirname > /dev/null
            num_errors=$?
            if $num_files > 0; then
                percent_error=`perl -e "printf '%8.0f', $num_errors * 100 / $num_files"`
            else
                percent_error=0
            fi
            if [ $num_errors == $num_files ]; then
                echo "Error:  Could not process $dirname: This package does not have $dirname/metadata.rdf" >> $errorfile
            elif [ $percent_error -gt 0 ]; then
                echo "Warning:  Most of the $num_files files in $dirname are valid, but $num_errors have problems." >> $errorfile
                echo "          Run svg_check_metadata $dirname to review." >> $errorfile
            else
                echo "All files in $dirname already have valid metadata"
            fi
        fi
    done
echo
    cat $errorfile
    # rm $errorfile
  • Also, it's a good idea to convert all spaces in filenames to underscores 
find . -name "* *" | xargs rename ' ' '_'

If that command doesn't work, try 

find . -name "* *" | rename s/\ /_/g

You may need to run that multiple times if there are files with more than one space

Use this to create .svg versions, but always keep the .wmf originals with them also.
The following command should convert all .wmf files in the current directory and insert the metadata from the corresponding .rdf files: 

wmftosvg --ocal -b *.wmf
  • If there are images that are clearly the wrong format (e.g., JPG or PNG images), you can go ahead and move them aside.

We save these, in case we might do something with them later, such as release a separate package of bitmap graphics, but there's no sense trying to run them through svg_validate. I've been putting them with the failed images (see below)
Nautilus is useful for browsing the clipart, although it may choke on some of the images. Possibly a better alternative would be clipartbrowser. 

mkdir -p failed/non_svg
find . -type 'f' -not -name '*.svg' | grep -v 'failed/' | xargs -i mv "{}" failed/non_svg

mkdir -p failed/zero_bytes
find . -size 0 | xargs -i mv "{}" failed/zero_bytes/
  • If there are images with likely copyright or trademark issues, set them aside with the failed files for later review. Put these in failed/not_public_domain. Particularly look for files with 'logo' or 'mascot' in their name. Logos, trademarks, etc. are generally not going to be valid public domain material.

Note: Don't worry about getting everything perfect; we can fix up issues after the fact as we learn about them, but do make sure to get the blatantly obvious stuff. If in doubt, fail it, and it can be discussed and sorted out later.

  • Run the openclipart-fix-unwanted-transparency script over the incoming files to remove any unwanted transparency due to a (now-fixed) Inkscape bug.

(Do we still need to do this? Running this on 0.17 resulted in a lot of changes, so apparently it's still needed.) 

${OCALBIN}/openclipart-fix-unwanted-transparency
  • Keyword

Perform any desired keyword authority control on these new incoming images (using clipart-authority-control.pl). For 0.10, I did this: 

${OCALBIN}/clipart-authority-control.pl -D -c -w -s

Check the stats to see that the keyword usage matches the existing collection and make any needed adjustments, e.g., using the consolidate option to unite singular and plural versions of the same keyword. Consolidate keywords like this: 

${OCALBIN}/clipart-authority-control.pl --consolidate=animal,animals,Animal:bird,Bird:mammal,mammals:insect,Insect

Delete keywords like this: 

${OCALBIN}/clipart-authority-control.pl --remove=foo,bar,baz
  • This is also a good time to run the detect-exact-duplicates tool. 
${OCALBIN}/detect-exact-duplicates.pl
mv duplicates/ failed/duplicate
  • You may also want to update the keyword hierarchy in the convert-release-to-browse.pl script at this point.
  • Make sure any previous default-0.00 directory has been emptied of any files that are still wanted, since the next step will nuke it.

Note: This should have already been done during the last release, but I leave the info here to prevent mistakes like I made for the first release I did

  • Change to the main working directory and do this: 
svg_validate --debug=2 incoming &> LOG.txt

This copies the files that pass into directories within default-0.00 (one directory for each keyword).

  • Now do this (still from your main working directory): 
mkdir failed
${OCALBIN}/parse-svg-validate-log.pl

This parses LOG.txt and moves files that failed from incoming/ to failed/
If all is well, all of the files in incoming SHOULD now be either copied into default-0.00 or else moved to failed -- either way, you don't need them in incoming anymore; the find-missed-clipart.pl script exists to help check that all is well; if so (i.e, if the NotFound array is empty, then it's safe to remove the incoming directory, so do that: 

rm -rf incoming

Alternately, the script spits out, on standard output, a set of rm commands for removing from incoming the files that were successfully copied into default-0.00. Running these should leave incoming empty, then you can rmdir it.

  • Next you want to consolidate the accepted files from their keyword directories into the desired hierarchy. You will need to change the directory path in the script, and possibly also the content of %hierarchy, then do this: 
cd default-0.00
${OCALBIN}/convert-release-to-browse.pl

Note: I got a whole bunch of error messages about 'Bad file descriptor' when running this... Maybe this script needs more debugging?

  • Run the weed-unsorted script to remove images from default-0.00/unsorted that are also found in other (sorted) directories within default-0.00 
cd ..
$OCALBIN/weed-unsorted

Note: this script needs to be revised to not be dependent on the existance of the default-0.00 path. This script would be useful for weeding out the unsorted stuff in the main release package.

Look through default-0.00/unsorted and do any additional sorting that is easy and obvious.

If you are going to make an update release, make a copy of default-0.00 now, for that purpose. I forgot this for 0.17, so now it is documented here.

  • The files in default-0.00 are now ready to be copied into the main hierarchy of the release. So, change back to your main working directory and do this: 
cp -rf default-0.00/* openclipart-${new_rel_num}/clipart/

Optionally... 

# rm -rf default-0.00

Move the failed directory outside the main working directory. The main working directory is now the release directory, and the failed files don't belong there.

  • Run the clipart-authority-control script

Be sure to use the index option to generate keywords.idx (which will be used by the keyword search tool on the site). You also want to use the authorstats option and statistics options to create the data for statistics.txt. Also generate the xml index. While you're at it, you may also want to do keyword and/or author authority control, as seems appropriate. 

$OCALBIN/clipart-authority-control.pl -r -x -i
$OCALBIN/clipart-authority-control.pl -r -a -s > statistics.txt
  • Review everything and make sure all is well
[edit]

Move to folder clipart

  • Move the folder with the new release inside the checked out module, clipart_package into folder clipart, where there is not the release in a nested folder

Make sure the VERSION folder is updated with the lastest version number
Move tools into the tools folder inside clipart_package and run: 

:su -c 'make dist' (must be this for rpmbuild stuff)
:make nsis (if nsis is installed)
  • Upload these files to the server, somewhere within 
/srv/clipart.freedesktop.org/www/downloads

or in 

/srv/clipart.freedesktop.org/clipart_web/downloads
  • Update clipart download and index pages & deploy updated pages
  • To install the new release for the browse script, untar the latest release and place the contents inside 
/srv/clipart.freedesktop.org/OpenClipart

Ensure the permissions are correctly set so that Apache can read it all (which I think means it needs x permission on directories) and the clipart group can write and/or delete it all (in order to install the next release in its place). I believe this command sequence should do it: 

cd /srv/clipart.freedesktop.org/OpenClipart
rm -rf *
tar -xzf ../www/download/relnum/openclipart-relnum.tar.gz
mv openclipart-relnum/* ./
perl -e ' process (<*>);
 sub process { for $i (@_) {
 process(<$i/*>) if (-d $i);
 `chgrp clipart $i`;
 `chmod g+wx $i`; }}'
  • Send out email notices to clipart@freedesktop.org inkscape-user@lists.sf.net., inkscape-devel@lists.sf.net, and more (see the overall Procedure, below).

DONE

[edit]

Old rest of procedure

  • cd to the *parent* directory of the release directory and do this (where openclipart-relnum is the name of the directory): 
zip -r openclipart-relnum.zip openclipart-relnum
tar -z -c -f openclipart-relnum.tar.gz openclipart-relnum
tar -j -c -f openclipart-relnum.tar.bz2 openclipart-relnum
  • Also zip up or tar up the failed files separately, and the tools.
  • Make a copy of the directory, and run make-svgonly-releas in it. 
zip -r openclipart-relnum-svgonly.zip openclipart-relnum-svgonly
tar -z -c -f openclipart-relnum-svgonly.tar.gz openclipart-relnum-svgonly
tar -j -c -f openclipart-relnum-svgonly.tar.bz2 openclipart-relnum-svgonly
  • Upload these files to the server, somewhere within 
/srv/clipart.freedesktop.org/www/downloads

or in 

/srv/clipart.freedesktop.org/clipart_web/downloads
  • Update clipart download and index pages & deploy updated pages
  • To install the new release for the browse script, untar the latest release and place the contents inside 
/srv/clipart.freedesktop.org/OpenClipart

Ensure the permissions are correctly set so that Apache can read it all (which I think means it needs x permission on directories) and the clipart group can write and/or delete it all (in order to install the next release in its place). I believe this command sequence should do it: 

cd /srv/clipart.freedesktop.org/OpenClipart
rm -rf *
tar -xzf ../www/download/relnum/openclipart-relnum.tar.gz
mv openclipart-relnum/* ./
perl -e ' process (<*>);
 sub process { for $i (@_) {
 process(<$i/*>) if (-d $i);
 `chgrp clipart $i`;
 `chmod g+wx $i`; }}'
  • Send out email notices to clipart@freedesktop.org inkscape-user@lists.sf.net., inkscape-devel@lists.sf.net, and more (see the overall Procedure, below).
  • Update the website with releases (news and on the download page)
[edit]

Send out a Release Notice to the following sites

What Lists and Sites to Announce To

  • Inkscape
  • SodiPodi
  • TheOpenCD
  • OpenOffice.org discuss@openoffice.org (moderated list)
  • SVG.org
  • the svg-developers Yahoo! groups list
  • w3c SVG list

Major Announcements (add these)

  • freshmeat.net
  • slashdot.org
  • gnomefiles.org
  • slashdot.org
  • etc
[edit]

Consider setting a bittorent seed

Retrieved from "http://openclipart.org/wiki/Release_Procedure"
Personal tools