The whole story started with the server suddenly not delivering mails anymore – in and outbound ones. For those who know the story: yes, it’s the 15. of April. 1 of April would have been more appropriate, somehow.
http://www.clamav.net/lang/en/2009/10/05/eol-clamav-094/
The gist of the story: ClamAV wants users to upgrade to a newer version, so they stopped releases older than 0.95 artificially from working. Thus bringing this mail server (and probably quite a bunch of others, too) down.
So … the upgrade which we were planning to do anyways suddenly became necessary for the server to perform.
We were running Zimbra 5.0.11 on Debian 4, 64bit.
Zimbra 6 for Debian 4 is offered only in the 32bit flavour, which refuses to install on a 64bit platform.
So, we upgraded Debian 4 to Debian 5. (modify /etc/apt/sources.conf, run apt-get dist-upgrade. See Debian documentation for specifics).
Trying to install Zimbra 6.0.6 with the provided script failed – Perl / libperl is too “new” at version 5.10. It should be at version 5.8 – install.sh will not proceed.
Checking for suggested prerequisites…
FOUND: perl-5.8.8
After forcing a downgrade – look at this post for a how-to – the installer proceeded without problems up until a point where it failed throwing a Perl error without recovering.
/usr/bin/perl: symbol lookup error: /opt/zimbra/zimbramon/lib/x86_64-linux-gnu-thread-multi/auto/IO/IO.so: ndefined symbol: Perl_Istack_sp_ptr
# su zimbra
$ zmcontrol status
would throw the same or a similar (Perl) error afterwards:
perl: symbol lookup error: /opt/zimbra/zimbramon/lib/x86_64-linux-gnu-thread-multi/autundefined symbol: Perl_Istack_sp_ptr
Luckily, a guy named Jiménez (kudos!) posted the solution to get the upgrade running:
“a perl script (/opt/zimbra/libexec/zmsetup.pl) was run to setup some preferences to the installation before it started the server. I ran it manually, and my problems were solved.”
What Jiménez did not mention: you need to have the newest Perl for this script to work – else it will fail with similar errors. So upgrade again (unlock pinned files if you did pin them = force them to be at the version 5.8.8):
# apt-get install perl
(… installs perl – the newest version … )
# perl -v
This is perl, v5.10.0 built for x86_64-linux-gnu-thread-multi
# /opt/zimbra/libexec/zmsetup.pl
Operations logged to /tmp/zmsetup.04162010-194935.log
Installing LDAP configuration database…done.
Upgrading from 5.0.10_GA_2638 to 6.0.6_GA_2324
Stopping zimbra services…done.
Verifying /opt/zimbra/conf/my.cnf
Starting mysql…done.
This appears to be 5.0.10_GA
Installing LDAP configuration database…done.
Migrating ldap data…done.
Checking ldap status…not running.
Running zmldapapplyldif…done.
Checking ldap status…already running.
Checking ldap status…already running.
Redolog version update required.
Fri Apr 16 19:51:03 2010: Verified redolog version 1.23.
Fri Apr 16 19:51:03 2010: Updating Redolog schema version from 1.23 to 1.28.
Redolog version update finished.
Schema upgrade required from version 53 to 64.
Running /opt/zimbra/libexec/scripts/migrate20080930-MucService.pl
Fri Apr 16 19:51:07 2010: Verified schema version 53.
Fri Apr 16 19:51:13 2010: Verified schema version 53.
Fri Apr 16 19:51:13 2010: Updating DB schema version from 53 to 60.
Running /opt/zimbra/libexec/scripts/migrate20090315-MobileDevices.pl
Fri Apr 16 19:51:15 2010: Verified schema version 60.
Fri Apr 16 19:51:15 2010: Adding ZIMBRA.MOBILE_DEVICES table.
Fri Apr 16 19:51:17 2010: Verified schema version 60.
Fri Apr 16 19:51:17 2010: Updating DB schema version from 60 to 61.
Running /opt/zimbra/libexec/scripts/migrate20090406-DataSourceItemTable.pl
Fri Apr 16 19:51:20 2010: Verified schema version 61.
Fri Apr 16 19:54:16 2010: Verified schema version 61.
Fri Apr 16 19:54:16 2010: Updating DB schema version from 61 to 62.
Running /opt/zimbra/libexec/scripts/migrate20090430-highestindexed.pl
Fri Apr 16 19:54:20 2010: Verified schema version 62.
Fri Apr 16 19:54:20 2010: Adding idx_deferred_count column to Mailbox table.
Fri Apr 16 19:54:22 2010: Verified schema version 62.
Fri Apr 16 19:54:22 2010: Updating DB schema version from 62 to 63.
Running /opt/zimbra/libexec/scripts/migrate20100106-MobileDevices.pl
Fri Apr 16 19:54:24 2010: Verified schema version 63.
Fri Apr 16 19:54:24 2010: Adding policy_values column to ZIMBRA.MOBILE_DEVICES table.
Fri Apr 16 19:54:25 2010: Verified schema version 63.
Fri Apr 16 19:54:25 2010: Updating DB schema version from 63 to 64.
Stopping mysql…done.
Updating from 5.0.11_GA
Updating from 5.0.12_GA
Updating from 5.0.13_GA
Updating from 5.0.14_GA
Updating from 5.0.15_GA
Updating from 5.0.16_GA
Updating from 5.0.17_GA
Updating from 5.0.18_GA
Updating from 5.0.19_GA
Updating from 5.0.20_GA
Updating from 5.0.21_GA
Updating from 5.0.22_GA
Updating from 5.0.23_GA
Updating from 6.0.0_BETA1
Migrating all domain admins to ACL based access manager…done.
Checking for default IM conference room…not present.
Initializing default IM conference room…done.< BR>Deletedcn=mime,cn=config,cn=zimbra
Updating from 6.0.0_BETA2
Updating from 6.0.0_RC1
Updating from 6.0.0_RC2
Updating from 6.0.0_GA
Updating from 6.0.1_GA
Updating from 6.0.2_GA
Updating from 6.0.3_GA
Updating from 6.0.4_GA
Updating from 6.0.5_GA
Updating from 6.0.6_GA
Updating global config and COS’s with attributes introduced after 5.0.10_GA…done.
Stopping ldap…done.
Upgrade complete.
Running bdb db_recover…done.
Running zmldapapplyldif…done.
Checking ldap status….already running.
Setting defaults…done.
Setting defaults from existing config…done.
Checking for port conflicts
Port conflict detected: 7306 (zimbra-store)
Port conflicts detected! – Any key to continue
Setting defaults from ldap…
Port conflict detected: 7306 (zimbra-store)
Debugging this error:
# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 78.47.177.193:389 0.0.0.0:* LISTEN 3157/slapd
tcp 0 0 127.0.0.1:7306 0.0.0.0:* LISTEN 19601/mysqld
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 9494/sshd
tcp6 0 0 :::22 :::* LISTEN 9494/sshd
Apparently there’s something on the port which should not be there. At this point your mail server is back up again, although messages will probably not get delivered and queue up:
# su zimbra
$ zmcontrol status
Host mail.example.de
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
mysql.server is not running.
mta Running
snmp Running
spell Running
stats Stopped
As you can see, the mailbox is stopped and the mysql server is not running. (Aside from a talent of stating the seemingly obvious, I’m also a great problem solver. Watch me:)
# su zimbra
$ zmcontrol stop
$ zmcontrol start
$ zmcontrol status
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running
That was about the easiest part of the upgrade. Respect to the Zimbra guys for packaging the remainder of the process up so nicely! Zimbra way rocks.
Note: I do not know, whether you can directly call the upgrade script (/opt/zimbra/libexec/zmsetup.pl) without running install.sh first. I think, no:
It (install.sh) DID set up several values on the first successful run (Perl needing to be at 5.8.8, remember?):
# ./install.sh –platform-override
Operations logged to /tmp/install.log.5176
Checking for existing installation…
zimbra-ldap…FOUND zimbra-ldap-5.0.10_GA_2638.UBUNTU8_64
zimbra-logger…FOUND zimbra-logger-5.0.10_GA_2638.UBUNTU8_64
zimbra-mta…FOUND zimbra-mta-5.0.10_GA_2638.UBUNTU8_64
zimbra-snmp…FOUND zimbra-snmp-5.0.10_GA_2638.UBUNTU8_64
zimbra-store…FOUND zimbra-store-5.0.10_GA_2638.UBUNTU8_64
zimbra-apache…FOUND zimbra-apache-5.0.10_GA_2638.UBUNTU8_64
zimbra-spell…FOUND zimbra-spell-5.0.10_GA_2638.UBUNTU8_64
zimbra-convertd…NOT FOUND
zimbra-memcached…NOT FOUND
zimbra-proxy…NOT FOUND
zimbra-archiving…NOT FOUND
zimbra-cluster…NOT FOUND
zimbra-core…FOUND zimbra-core-5.0.10_GA_2638.UBUNTU8_64
ZCS upgrade from 5.0.10 to 6.0.6 will be performed.
Saving existing configuration file to /opt/zimbra/.saveconfig
PLEASE READ THIS AGREEMENT CAREFULLY BEFORE USING THE SOFTWARE.
ZIMBRA, INC. (“ZIMBRA”) WILL ONLY LICENSE THIS SOFTWARE TO YOU IF YOU
FIRST ACCEPT THE TERMS OF THIS AGREEMENT. BY DOWNLOADING OR INSTALLING
THE SOFTWARE, OR USING THE PRODUCT, YOU ARE CONSENTING TO BE BOUND BY
THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS OF THIS
AGREEMENT, THEN DO NOT DOWNLOAD, INSTALL OR USE THE PRODUCT.
License Terms for the Zimbra Collaboration Suite:
http://www.zimbra.com/license/zimbra_public_eula_2.1.html
Press Return to continue
Checking for prerequisites…
FOUND: NPTL
FOUND: sudo-1.6.9p17-2闩
FOUND: libidn11-1.8괺1
FOUND: libpcre3-7.6-2.1
FOUND: libgmp3c2-2:4.2.2痻3
FOUND: libexpat1-2.0.1-4闩
FOUND: libstdc4.4.2-9
FOUND: libperl5.8-5.8.8-7etch6
Checking for suggested prerequisites…
FOUND: perl-5.8.8
FOUND: sysstat
Prerequisite check complete.
Checking current number of databases…
df: no file systems processed
Do you want to verify message store database integrity? [Y]
Verifying integrity of message store databases. This may take a while.
mysqld is alive
No errors found
mysqld is alive
Killing mysqld with pid 5694 done.
Checking for installable packages
Found zimbra-core
Found zimbra-ldap
Found zimbra-logger
Found zimbra-mta
Found zimbra-snmp
Found zimbra-store
Found zimbra-apache
Found zimbra-spell
Found zimbra-memcached
Found zimbra-proxy
The Zimbra Collaboration Suite appears already to be installed.
It can be upgraded with no effect on existing accounts,
or the current installation can be completely removed prior
to installation for a clean install.
Do you wish to upgrade? [Y]
Select the packages to install
Upgrading zimbra-core
Upgrading zimbra-ldap
Upgrading zimbra-logger
Upgrading zimbra-mta
Upgrading zimbra-snmp
Upgrading zimbra-store
Upgrading zimbra-apache
Upgrading zimbra-spell
Install zimbra-memcached [N]
Install zimbra-proxy [N]
Checking required space for zimbra-core
checking space for zimbra-store
Installing:
zimbra-core
zimbra-ldap
zimbra-logger
zimbra-mta
zimbra-snmp
zimbra-store
zimbra-apache
zimbra-spell
You appear to be installing packages on a platform different
than the platform for which they were built.
This platform is UBUNTU8_64
Packages found: DEBIAN5_64
This may or may not work.
Using packages for a platform in which they were not designed for
may result in an installation that is NOT usable. Your support
options may be limited if you choose to continue.
Install anyway? [N] Yes
The system will be modified. Continue? [N] Yes
Shutting down zimbra mail
Backing up the ldap database…done.
Removing existing packages
zimbra-ldap…done
zimbra-logger…done
zimbra-mta…done
zimbra-snmp…done
zimbra-store…done
zimbra-spell…done
zimbra-apache…done
zimbra-core…done
Removing deployed webapp directories
Installing packages
zimbra-core……zimbra-core_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-ldap……zimbra-ldap_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-logger……zimbra-logger_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-mta……zimbra-mta_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-snmp……zimbra-snmp_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-store……zimbra-store_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-apache……zimbra-apache_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
zimbra-spell……zimbra-spell_6.0.6_GA_2324.DEBIAN5_64_amd64.deb…done
Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
HOSTNAME=mail.example.de
LDAPHOST=mail.example.de
LDAPPORT=389
SNMPTRAPHOST=mail.example.de
(…)
SNMPNOTIFY=yes
SMTPNOTIFY=yes
LDAPROOTPW=
LDAPZIMBRAPW=
LDAPPOSTPW=
LDAPREPPW=
LDAPAMAVISPW=
LDAPNGINXPW=
Restoring existing configuration file from /opt/zimbra/.saveconfig/localconfig.xml…done
/usr/bin/perl: symbol lookup error: /opt/zimbra/zimbramon/lib/x86_64-linux-gnu-thread-multi/auto/IO/IO.so: ndefined symbol: Perl_Istack_sp_ptr
Please ignore the screwed up error about us not running on Debian (which we are) – I think it’s not part of this problem with Perl.
I hope this helps you. Would be nice if you leave comments and further tips how to fix this.
G’night. – Max
