PHP and Oracle CLOB woes

Posted on August 19th, 2008

Been working with Oracle lately, trying to make CLOBs work. For those of you who dont know, CLOBs are Character Large OBjects, and BLOBs are Binary Large OBjects. (edited) Character and Binary LOcator Objects, like pointers. CLOBs and BLOBs are ways that databases can store arbitrarily large amounts of text or binary data, and overcomes the size limitation of a VARCHAR field type (4000bytes). If using an 8-bit ASCII charset, that means 4000 characters, but with the onset of UTF 8 and other multibyte charsets, Unicode and so on, that could be as few as 1000 characters per column/field.

We’re still in the middle of resolving some issues, becase the Oracle databse handles a CLOB as an IN OUT parameter, and rather than passing the data into a stored procedure, you awkwardly have to set the value of vCLOBfield = empty_clob()


CREATE OR REPLACE PACKAGE BODY PKG_CLOB AS
/***************************************

PROCEDURE UpdateClob(vCLOBfield IN OUT CLOB) IS -- cursor must be in out
BEGIN
UPDATE table_or_view_with_clob
SET clob_value = empty_clob() -- returns a reference to the LOB (think 'null pointer')
WHERE clob_id = 1 --use your own where clause here
RETURNING clob_value INTO vCLOBfield; -- returns the actual LOB reference (think 'address pointer')
END;
END PKG_CLOB;
/


However using “instanceof” OCI-Lob on the returned object will fail, as the php developers wisely named the object using invalid characters, specifically ‘-‘. Since “is_a” is deprecated, the only way to discover if the oject returned from Oracle is indeed an instance of an OCI-Lob, is perhaps something like


/*
db connection,
sql statement,
@OCIParse statements here
*/

/*
@OCIExecute,
@OCIError handling,
@OCIFreeCursor statements
*/


Now this is still very much in development here, so its by no means a complete solution to using CLOBs, but as things develop I may well come back and address this in future when I get a complete solution – so leave a comment if you are interested and want me to increase my efforts!!

TTFN.

auto create DHCP hostnames for a scope

Posted on August 14th, 2008

Following on from private IP address pools for your cluster, you might want a way to automatically create entries for all your machines in DHCP or in your /etc/hosts file – perhaps you have a Cisco router and don’t want to edit that config file by hand?

N=1; for x in $(seq -w 254); do echo -e “10.0.0.$N \t node$x”; N=$(expr $N+1); done >> /etc/hosts

sample output:
…
10.0.0.250 node250
10.0.0.251 node251
10.0.0.252 node252
10.0.0.253 node253
10.0.0.254 node254

Using echo -e interprets the escaped \t as a tab character correctly.
(I gather that tab is sometimes the recommended whitespace format e.g. used in the Windows\system32\drivers\etc\hosts file?, rather than just spaces. I also included spaces for legibility here though)

We use (seq -w 254) which iterates through all the numbers 001, 002, … 254 and the -w flag keeps the leading zeros. We use this value, assigned to $x, to generate consistent looking hostnames, ‘node001’ etc.

We also take N (starting at 1) and increase it by 1 on each iteration. We use that as the value of the node’s IP number in the echo statement.

Finally, it appends to the /etc/hosts file, although you can redirect this output anywhere if you want to check or edit it before use.

Note:
Take care if modifying the command to loop through multiple subnets, eg 10.0.$N.$M – if you are not careful you could end up with 255*65535 lines of output, or in other words, have ip addresses going from 10.0.1.1 to 10.0.1.65535, which is not what you would want!!

=)

Also Note:

If we used $x instead of $N, we may have difficulty resolving nodes < 100.
For example, ‘ping node001’ could end up trying a dns lookup on “node001.defaultsearch.domain” instead of pinging ‘10.0.0.1’ as expected.

I’ve seen this behaviour on CentOS / BlueQuartz – The leading zeros on ips lower than x.y.z.100 seemed to confuse the resolver

It had nothing to do with nsswitch.conf, “hosts: files,dns” or /etc/resolv.conf. Restarting nscd (name service caching daemon) by logging in as root and trying

/etc/init.d/nscd restart
/etc/rc.d/init.d/nscd restart (on other flavours perhaps)

to flush cached DNS entries didn’t work either, especially since nscd wasn’t running at the time…

DNS frustrations

Posted on August 11th, 2008

Imagine the situation:

You’ve just registered a domain name with a DNS registrar.
You update the DNS nameserver records to point to a couple of new nameservers, perhaps under your control, or at the ISP where your hosting is…
You’ve waited 24 hours for the migration to occur, and it seemed successful.
Now you test {www.yourdomain.xyz} with host -v {domain}, nslookup {domain}, and dig {domain}.
All the results point to your new server’s IP address. Yet your browser still goes to the old domain parking page on the registrars server…?

Well it could be that the local resolver on your PC is to blame, caching the old DNS A record while the TTL (Time-To-Live) expires. You could fix it perhaps with a reboot, or try these two Windows commands:

ipconfig /displaydns (shows the local dns cache)
ipconfig /flushdns (drops all the records and begins resolving anew)

tada!! your browser now learns the new address and off it goes to your shiny new site =)

ps if anyone knows the equivalent commands for a linux / unix OS, please post a comment!

IPv4 Address Assignments

Posted on August 6th, 2008

A picture is certainly worth a thousand words, in this case!

———————
Matt Saunders said…

XKCD visualises the same data, in less detail:

http://xkcd.com/195/

🙂
———————

Here’s a much needed shortened version of the post

A list of all the currently (un)allocated IPv4 address blocks, can be obtained from http://www.iana.org/assignments/ipv4-address-space/ (or see the comments)

The reason for posting this before a discussing on setting up your cluster / lamp in more detail is to help you determine an IP addressing scheme for your network which fits your needs. For example, I was once under the impression that:

10.x.x.x,
172.16.x.x – 172.31.x.x, and
192.168.x.x

were the only network IPs that could be used on private IP networks. However, if you really wanted to, it’s technically feasible that you could run a 1.x.x.x and a 2.x.x.x subnet on your back-end network (nb If you are in any doubt at this point, stick to using one of the ranges shown above) The reason is that there still exists some blocks marked as unassigned or reserved by IANA.

WARNING!! You should definately bear in mind that the reason for it being ill-advisable is that these address blocks could be reclaimed by the IANA in the next couple of years for use as the IPv4 addresses run out!! (eta 2010)

NOTICE!! You should also ensure that they are never directly connected to or routable over the Internet in case you cause a conflict with a valid assigned user now, or at a later date!

EXAMPLE: I sometimes use the Hamachi VPN client (and HamachiX frontend on Mac OS/X). When I join a ‘network’ by name, my client is assigned a unique 5.x.x.x IP address, and other machines on the same ‘network’ name appear as if they were on the same LAN segment with their own 5.x.x.x addresses, when in fact they could be across the Internet and on the other side of the globe. (The private 5.x.x.x addresses are tunnelled over/encapsulated inside the VPN to each endpoint, and are not ‘directly’ routed).

BTW, IPv4 addresses are really just 32 bit integers ranging from 0 to 4,294,967,295 !

Ubuntu Server VMware VA image

Posted on August 5th, 2008

You can download the same copy of Ubuntu server VA from which I will be basing my examples here. It is not the latest version, as upgrading the image will form part of my tutorial.

(Please note! I was not responsible for creating the VMware Virtual Appliance itself, any questions about it please direct to the author – see description below)

INSTRUCTIONS

http://nevynsblog.posmena.co.uk/VMwareUmbongo7.10srv.tar.gz contains:

Ubuntu7.10Server.README.txt
VMware-UbuntuServer-7.10.7z
VMware-UbuntuServer-7.10.7z.md5

If you are using Linux, download the file, and extract it into the current directory with “tar xzfmv ./VMwareUmbongo7.10srv.tar.gz”. You can “md5sum VMwareUmbongo7.10srv.tar.gz” and verify the file integrity by ensuring the output matches the contents of VMware-UbuntuServer-7.10.7z.md5 with “cat ./VMware-UbuntuServer-7.10.7z.md5”.

If you are on Windows, download WinRar from http://www.rarlabs.com/download.htm. This will handle extracting the .tar.gz and then the .7z file.

DESCRIPTION

The package also contains a readme file which you can read locally, but to spare you the 70MB download, I will repost here:

Ubuntu Server 7.10 (Gutsy Gibbon) Minimal Install
This is a standard (minimal) install of ubuntu 7.10 server, in a 69MB package. … username: toor password: password. you can issue commands as root using sudo …
www.vmware.com/appliances/directory/1060 – 9k

VMware Image – Ubuntu Server 7.10 (Gutsy Gibbon)

About:

This is a standard install of ubuntu 7.10 server, in a 75MB package.

This VM can be usefull for quickly deploying a base operating system from a small package, and with a minimal configuration time (about 2 minutes).
Some steps were taken to reduce the archive size down to a minimum:

* Some temporary data has been removed from /var (namely apt cache, installer logs, etc).
* The partition was also cloned to a new .vmdk as a part of the shrink process.
* The archive is in 7-Zip format (GPL – search you package manager or grab it from http://www.7-zip.org).

Aditional software included:
* The only extra package installed was ssh-server.

Login details:

username: toor
password: password

you can issue commands as root using ’sudo ‘ or enable the root user issuing the command ’sudo passwd root’.

VM Configuration:

Uppon login, your first action should be to run the ‘configure.sh’ script using ’sudo ./configure.sh’. this will enable you to configure:

* Keyboard,
* Timezone
* Password
* Hostname
* Network interface(s)
* DNS

Download:

Grab it here: VMware-UbuntuServer-7.10.7z. ( http://www.jcinacio.com/down/vmware/ubuntu-server-7.10-1/VMware-UbuntuServer-7.10.7z )
Also, md5 checksum ( http://www.jcinacio.com/down/vmware/ubuntu-server-7.10-1/VMware-UbuntuServer-7.10.7z.md5 ) and gpg signature ( http://www.jcinacio.com/down/vmware/ubuntu-server-7.10-1/VMware-UbuntuServer-7.10.7z.asc ) (key id: 0×146379A4).

If you find this appliance useful in any way, please link to this site.

Contact:

* Joao Inacio or at http://www.jcinacio.com

Changelog:

* 18 October 2007
– Initial release
* 20 October 2007
– Fixed network interface (eth0) name change due to new UUID

OOP criticisms

Posted on August 5th, 2008

As this blog grows, I intend to take the novice web developer from their procedural roots, and lead them by example to setup their own clustered, scalable web server platform for their own applications.

A lot of programming books only give simple examples, and avoid object oriented designs. Whilst the adage ‘Never explain beyond what is absolutely necessary to the understanding of a topic’ holds, the lack of context sometimes leave you wondering how to include examples in your own site.

Now, clustered servers might not be of use to you. You might not even use MySQL, PHP or Apache, in the end. You may not like the idea of Smarty templates. Last but not least, you may well struggle to get to grips with OOP.

Well it’s not for everybody. So here is some criticism of OOP to bring balance to this thread, and reduce any perceived bias. Feel free to read it and decide for yourself if OOP is not for you: http://www.geocities.com/tablizer/oopbad.htm

Now, I have seen a post on http://www.devx.com/opinion/Article/26776 denouncing OOP, claiming that procedural code-reuse can be acheived by copy-n-paste. Thats code duplication, not code reuse! but I agree with the author in some respects, anything that is over-administered, over-complicated and over-engineered can make for unwanted bloat, and sometimes quick-n-dirty hacks will suffice.

Even with relatively little experience of OOP myself, I see the benefits of using it, and begining to see my projects in a more abstract level. You dont need to abandon your procedural roots – but to allow your skills to develop, embed procedures in classes, and they become universally available (reusable) units that save you from reinventing the wheel continually.

Heck, getting my head around it whilst using a relatively easy language (php) has resulted in being able to follow more complicated c++ code – perhaps not completely, but certainly I can get more of a feel. You can then go into even greater depth and understand how the two languages differ, and grasp concepts like structs, references, pointers and variable type / casting. Well, thats all for now, I feel like some programming… =)

BlueQuartz::DB4

Posted on August 4th, 2008

Hi folks – just having some more fun poking around under the hood of the sausalito UI (user interface) for BlueQuartz linux. I’m actually just trying to amend one of the package names installed under the BlueLinQ tab, because I have installed Webmin 1.250, and after upgrading it to 1.420 BlueLinQ has frustratingly kept the original 1.250 version number.

I have traversed the JavaScripts, read the PHP, and found my way into the $cce objects. That led me to the discovery of the cce.so module used by the php4 admin backend webserver. it would appear that with this, the $cce object in PHP communicates with the /usr/sausalito/sbin/cced daemon via the cced.socket unix socket. (I could be slightly off here, but that seems to be vaguely representative?)

There are 2 xml files in the /usr/sausalito/ui/style folder which define the colour schemes, ‘Merlot’ and ‘trueBlue’. However, there are over 100 other .xml files in the /usr/sausalito/ui subfolders, which looked like a good place to find where BlueQuartz kept its data, but upon inspection appeared only to contain definitions for the various menus etc associated with the admin UI.

Without looking at (for) the objects, methods and function calls in the php which manipulate the cced, I investigated the cceclient /usr/sausalito/bin/cceclient. I recommend setting up an alias by using a line like “alias cceclient=’/usr/sausalito/bin/cceclient'” in your terminal session (or better, in your ~/.bashrc file) to make it easier. See my earlier post for commands you can use, and READ THE WARNINGS!

So now you’ve got access to the backend. Perhaps you can even GET and SET some values, and whilst this may be enough to solve the above problem (altering the description of an installed package), I was still curious about where it stored the data, as it wasnt in any of the .xml files found thus far, I couldn’t see any evidence in the MySQL db, and there was no sign of a postgreSQL server either.

The clue is /usr/sausalito/db4 – looks like a Berkeley db4 v4.0.14 backend, and sure enough, the apache admin instance also has dba support built in. It doesnt appear to use SQL, resembling more of a structured file like xml with key=value pairs. I’ll try and update this section shortly…

Until then, for more info see these links:

/usr/sausalito/db4/docs/index.html is the path to the (local) documentation on BlueQuartz and contains the API for C/C++, Java and Tcl. See also the docu about the supporting utilities (in db4/bin)

Adding dba support to php

How to Create a db4 Database from a Text File or Array

http://www.stanford.edu/services/pubsw/package/libraries/db4.html

“Description: Embedded database and hash file library
Source: http://www.sleepycat.com/

Berkeley DB is a programmatic toolkit that provides high-performance built-in database support for desktop and server applications and for information appliances.

The Berkeley DB access methods include B+tree, Extended Linear Hashing, Fixed and Variable-length records, and Queues. Berkeley DB provides full transactional support, database recovery, online backups, and separate access to locking, logging and shared memory caching subsystems.”

RPM In a Nutshell

Posted on August 3rd, 2008

Much like a compressed tarball, RPM uses a combination of rolling together multiple files into one archive and compression of this archive to build the bulk of the RPM package. Furthermore, additional header information is inserted. This includes pre- and post-installation scripts to prepare the system for the new package, as well as information for the database that RPM maintains. Dependencies are checked before any installation occurs, and if the appropriate flags are set for the installation they are also installed.

It is this database that makes RPM work the magic that it does. Stored in there are all of the properties of the installed packages. Should this become corrupted, it can be rebuilt using the rpm tool.
The Hows of RPM
We will now focus on the three core actions of RPM discussed in this documentation. They include installation of new packages, management of installed packages, and package removal. We will begin at the beginning, and how to add a package using RPM.
Installation Using RPM
This is the most basic RPM function, and one of the most popular: the installation of new software packages using RPM. To do this, give rpm the -i flag and point it to an RPM:

rpm -i (package)

This will install the package if all goes well and send you back to a command prompt without any messages. Pretty boring, and worse if you want to know what happened you’re out of luck. Use the -v flag to turn on some verbosity:

rpm -iv (package)

All that gets printed out is the package name, but no statistics on the progress or what it did. You can get a hash marked output of the progress is you use the -h flag. People seem to like using -ivh together to get a “pretty” output:

rpm -ivh (package)

Again, this doesn’t tell you much about what just happened, only that it hopefully did. Hence, I usually crank up the verbosity (-vv) when I install. This lets me see what’s going on:

rpm -ivv (package)

While the output usually scrolls, I can see exactly what happened and if any problems were encountered. Plus I get to see where stuff was installed.

Dependencies are handled pretty wisely by RPM, but this itself depends on a good package builder in the first place. I have seen packages that depend upon themselves, for instance, and some that seem to depend on packages that will break other things. Keep this in mind.

Sometimes RPM will whine about a dependency which is installed but isn’t registered. Perhaps you installed it not using an RPM for the package (ie OpenSSL). To get around this, you can force it to ignore dependencies:

rpm -ivv –nodeps (package)

Note that this isn’t always wise and should only be done when you know what you are getting youself into. This will rarely break installed stuff, but may mean the installed package wont work properly.

On rare occassion RPM will mess up and insist that you have a package installed when you don’t. While this is usually a sign that something is amiss, it can be worked around. Just force the installation:

rpm -ivv –force (package)

Beware. Just like when you ignored the dependencies, forcing a package to install may not be wise. Bear in mind that your machine could burst into flames or simply stop working. Caveat emptor and all that.

This probably wins the award for one of the coolest features in RPM: network installations. Sometimes, you don’t have network clients on a system but you need to install them via RPM. RPM has built in FTP and web client sowftare to handle this:

rpm -iv ftp://ftp.redhat.com/path/package.rpm
rpm -iv http://www.me.com/path/package.rpm

I don’t think it can do SSL HTTP connections, though. Debian’s packages can do this, as can BSD packages. I don’t think most commercial tools can do this, though.
Managing Your Packages
OK, you know how to install packages. But let’s say you want to work with some packages before you, either installed or not. How can you do this? Well, simply put, you can use package management features in rpm to deal with packages, ones that are installed already or ones that are not, to look inside of them. This can include verifying packages, too.

When you get a new package, you may want to examine it to see what it offers. Using query mode, you can peek inside. To simply query a package and get some generic information about it, just simply:

rpm -qp (package)

This should bring just the name of the package. Pretty boring, isnt it? A much more useful method is to get the package information from the package itself:

rpm -qip (package)

This will bring up the author, build host and date, whether it’s installed yet, etc, about a package. Also included is a summary about the package’s functionality and features.

All of this is nice, but let’s say you want to see what is really inside the package, what files are inside of it? Well, you can list the contents of a package, much like you would get the table of contents of a tar archive (using tar -tvf):

rpm -qlp (package)

This will list all of the files within the archive, using their full pathnames. I use this often to see what will be installed with a package, but most importantly where. I like to stick to conventions about putting stuff in their expected places, but some packagers do not. Lastly, to show all of the packages you have installed on your system, use:

rpm -qa

This will bring up a list of packages installed on the current system. It may be useful to sort them (by piping through sort, rpm -qa | sort). Use these names when uninstalling packages (below).

One of my favorite things about RPM is how it can verify packages. This is useful in detecting a compromised machine, or a binary that may be missing or modified due to some error on your part. To verify one package, just point rpm at it with the -V flag:

rpm -V (package)

This should bring up a brief description of wether or not the package checks out. To verify all packages installed on a system, it is quite simply:

rpm -Va

Verify mode brings up several statistics about a file. Their shorthand is as follows:

5 MD5 sum
S File size
L Symlink
T Mtime (modification time)
D Device
U User
G Group
M Mode (includes permissions and file type)

Sometimes they’re meaningless, for example if you modify your /etc/inetd.conf file it will show up as a different size and MD5 sum. However, some things shouldn’t change, like /bin/login. Hence, rpm -Va can be a useful quick security check, suggesting which things need more peering into.

One of the great things about package management we stated at the outset was the ease with which upgrading can be performed. RPM has two sometimes confusing options to upgrading packages. The first is a simple upgrade:

rpm -U (package)

What is confusing about it is it’s resulting action if the package is not yet installed. If it finds it, the package is upgraded. If it doesn’t find it, you are upgraded to it, meaning the package is installed. This can be scary sometimes if you don’t mean to install a package and an update comes out, which you blindly follow. Because of this, I suggest using “freshen” on packages you only want to ensure you have the latest version of:

rpm -F (package)

This will only update installed packages, and not install the package if it is missing.

Upgrades are done in an interesting fashion, too. The new version is first installed and the differences with the old version are noted. Then the older version is removed, but only the unique portions of it so as not to remove the new portions. Imagine if /usr/local/bin/netscape were upgraded and then removed, that would defeat the whole effort!
Uninstalling an RPM
You can install, upgrade, and manage, and hence you can definitely uninstall using RPM. The flag to use is -e, and many of the same conditions for installation apply for removal. To silently uninstall an RPM package, use:

rpm -e (package)

Note that, unlike for installations and upgrades, package here refers not to package-version.i386.rpm, but instead to package-version. These are the values reported in query mode, so use those. You should be able to get all components of a package for removal by specifying the most generic, common part of the name, ie for linuxconf and linuxconf-devel, specify linuxconf. Dependencies can also be avoided:

rpm -e –nodeps (package)

The same caveats apply here, as well, you could wind up breaking more stuff than you anticipated. Verbosity flags can also be added, just like in installations.
Some Notes about RPMs
Sometimes maintainers build rather stupid dependencies for their RPMs. Case in point, libsafe. It has a wierd dependency: itself. As such, I usually find I have to install –nodeps to get it to install properly. Other times a package will contain extra junk, and you could wind up installing more than you had planned for.

My biggest complaint are RPMs that have a name that doesn’t suit the function of the pieces. While this can be gotten around by digging around using the query tools as described above, it’s more time than I care to waste. Name your RPMs well, I suggest.
Getting RPM
You can get RPM for any Linux or UNIX variant, as it is Open Source. RPM comes native in Red Hat Linux, and some derivatives. Versions 3.0 and above are recomended for compatability, some stupid stuff went on before that that 3.0 hopes to fix. Version 4.0 reportedly has a different database format, and so I reccomend checking around for how to get around this issue before you upgrade to 4.0. I’m not sure if can simply rebuild the database in 4.0 to remedy this.

RPM is normally distributed as an RPM of itself. Cute, eh? Luckily, it also comes in a gzipped tarball and also in source form. I have RPM installed on Slackware, for example, and could install it on IRIX or Solaris if I so desired. It’s nearly useless on non-Linux platforms as rarely are packages built in RPM for other UNIX variants.

From http://linuxgazette.net/issue68/nazario.html

Samba howto – connecting from linux

Posted on August 3rd, 2008

Credits once again go to www.raqpak.com (site offline) for this. Its targeted towards Cobalt users, but it could be applicable to anyone on a linux machine trying to access a windows resource. I included it mainly for my own reference here, although I suppose I could compliment it with the counterpart – how to setup a Samba share on Cobalt / BlueQuartz / Linux for windows users to access

Mount a windows NT/2000/98 path

If you want to mount a windows filesystem onto your Qube3 use the following command:

mount -t smbfs -o username=theuser,password=thepass //servername/sharename /home/groups/home/network

Change username/password as required (If you dont need to authenticate, remove the entry and the -o option parameter completely)
Create the directory you want to mount the drive to, so in the above example create /home/groups/home/network first then do the mount statement.

If possible, before a reboot or shutdown, do a:
umount /home/groups/home/network
otherwise the reboot will take longer as it unmounts the system itself.

(For this to work i’m assuming you’ve configured your network file sharing/workgroup stuff correctly as its all related)

RaQ::Packages (manual installation)

Posted on August 3rd, 2008

one more article from www.raqpak.com (site offline)

Manually install PKGs

Incredibly simple HOWTO this one, but one that I constantly forget, to install a PKG via the ssh/command-line simply use:

/usr/local/sbin/cobalt_upgrade nameof.pkg

This is the case for RaQ3 and RaQ4.
For the Qube3 do:

/usr/sausalito/sbin/pkg_prepare.pl [-i] [-R] -f
/usr/sausalito/sbin/pkg_prepare.pl [-i] [-R] -u

Where -i means install, and R means do not automatically reboot.

– Andy