This document is a summary of a thread on the Paneris system/newbox message board. It considers where to place files for each project developed by Paneris and code which is common to many projects. We need to make sure that files on ganesh can be altered by anyone. I have set up a cvsusers group and put all of us in it. Please make all directories/files owned by this group (and alter permissions appropriately) if we all need to access them.
/where/ever/you/develop
This is where you develop and test the code. On ganesh, I suggest /home/user/cvswork or where you want on your own (remote) computer.
/usr/cvsroot/org/paneris/<project>/...
The directory actually containing the CVS repository of this module. You only really need to know this if you add a directory to the repository. You then need to chgrp cvsusers for these directories so that other people can change thier contents.
/usr/local/packages/dist/paneris/org/paneris/<project>
/usr/local/packages/dist/paneris/org/paneris/<project>-#.##
These directories contain a copy of the project's source code from the repository, or a release of the project respectively. This is only needed if we need to compile, or similar. Otherwise we can put the code straight into the equivalent inst directory.
/usr/local/packages/inst/paneris/org/paneris/<project>
/usr/local/packages/inst/paneris/org/paneris/<project>-#.##
Contain the installed, currently running version of each project (or release of a project). This is either the code straight out of the cvs (perhaps at a given revision) if there is no need for compiling, or the result of installing (make install?) the equivalent dist directory.
NB: The /org/paneris which appears above often is due to our cvs package naming conventions outlined below.
/usr/local/share/java
/usr/local/share/handlers
/usr/local/share/templates
These directories contain (links to) java bytecode, compiled java handlers for webmacro and templates for webmacro respectively.
OK, so that's a quick overview of where files might live. What do we do with them? Well, we develop in them. Here's how:
NB: If you need to do compile then you might need to extract into ../dist/.. and compile into the ../inst/.. mentioned above. See Inst & Dist below for more details.
If you are thinking of developing on ganesh, don't do it in the inst or dist directories. You will need to set the symlinks for any components of the system (http, cgi-bin, wmtemplates, etc) to point to your development area. When you commit to the repository you can either leaving everything or you can update the inst/dist directories from the cvs repository and then change the symlinks to point to this new/updated directory.
The repository contains the source for all our projects. Everything we write will be a package under the /org/paneris package. For example:
The organisation of files under these projects should be guided by:
"for any given operation on the sources, there is a directory (or subtree) which contains all and only those files. The directory structure you end up with depends on which operations you feel require the most support from it."
In particular, organise files by functionality: "That is why I propose a structure {module1,module2}/{src,doc} rather than {src,doc}/{module1,module2}. ".
Details
All the code for all the languages we use goes in the same tree, inducing an identical package namespace in Java, Perl and C++ (and eventually ML)*. If some packages are programmed in two languages then there are simply two kinds of source in the corresponding directory.
Also in the same tree goes documentation (project docs and code docs), HTML, templates, test data, any scripts needed for initialising a database, backing up a site's data, etc. The functional unit rule still holds: for instance, all the documentation lives in a subdirectory under the project (or code module, or whatever) which it documents.
org/
paneris/
utils/
*.java
*.pm
docs/
requirements/
*.{html,gif}
functional/ ...
...
guide/
*.{html,gif}
*.java (if there were examples in the user guide!)
dbadmin/
doc/ ...
handlers/
doc/ ...
wmtemplates/ ...
*.java
perlcgi/
doc/ ...
*.pm
...
sugar/
doc/ ...
module1/
doc/ ...
*.java
...
http/
doc/ ...
*.{html,gif}
handlers/
*.java
doc/ ...
wmtemplates/
*.wm
...
/usr/local/packages/dist/paneris/<cvs-module-name>
e.g. /usr/local/packages/dist/paneris/org/paneris/sugar
This directory contains the distribution of a project. This will have the same structure as the cvs repository for a particular project. You will need to compile any sources and place the resulting code in ...
/usr/local/packages/inst/paneris/org/paneris/<project>
We hope that many projects will not require the compilation step or that the compiled code can be installed directly into the right places on the machine. In this case, we can dispense with the code in the dist tree and put the code straight into the inst directory. This should be case with perl where compilation is trivial and java where we need to compile, but we can compile into the relevant system wide directory (/usr/local/share/java or /usr/local/share/handlers). You may prefer, however, to put the sources in dist, compile into inst and then set symlinks to inst. See "installation", below for more details.
Note that code in either the dist or inst trees should not be committed to cvs. You should develop in your own development directory somewhere.
Part of the idea for the organisation of our code is that any "installation" step is kept to a minimum. We hope that in most cases it will be a case of making a symlink from the appropriate directory to a directory in the inst or dist of a project. This only needs to be done once per project and not, for instance, after each change to a HTML file. Links to consider include:
In all the above it is assumes that you will create a new directory in each of the listed directories which links to a directory of yours, except for /usr/local/{bin,lib,include}.
Both /usr/local/share/handlers and /usr/local/share/java contain java bytecode. It might be easier to compile your java directly into these directories using
javac -d /usr/local/share/java *.java to compile java which lives in package and to put the bytecode in the right place under /usr/local/dev/java
javac -d /usr/local/share/handlers/project *.java to compile handlers and put the bytecode in the right place. Note that handlers cannot, at present, be in a package as far as webmacro is concerned, but the source can live very nicely inside a (for instance) org/paneris/project/webmacro/handlers module.
The alternative is to compile them into a inst directory and then link from the above directories to your inst directory. You might need to alter the CLASSPATH to take account of your new java code.
Remember to alter the CLASSPATH in /usr/local/apache/etc/jserv.properties to include the new directory containing your handlers!
I took the code that TimJ has been working on, the datadictionary and administration
interface, java utils and sugar application code. I re-arranged it according
to the module guidelines above (and made some changes) and uploaded the resulting
org/paneris/... tree it into a working directory in my user directory on ganesh
(~/cvsroot). The structure is very similar to the example
CVS structure above.
I imported it into the CVS repository using
$ cd ~/cvsroot/org
$ cvs import -m "Imported development tree" org paneris startLater you probably want to only have one project out at a time. Let's assume it's sugar and you haven't bothered with the org/paneris directories (sugar is directly in your cvsroot directory):
$ cd ~/cvsroot/sugar
$ cvs import -m "Imported sugar for the first time" /org/paneris/sugar paneris imported
You only need to do the above when you create a new module. In this case you might want to:
$ cd ~/cvsroot
$ cvs checkout CVSROOT/modules
$ pico CVSROOT/modules
and add the line
projectname /org/paneris/project$ cvs commit CVSROOT
$ cvs release -d CVSROOT (to release and -d(elete) the module/directory)This means people can cvs checkout projectname rather than cvs checkout org/paneris/project
What you do need to do is alter the permissions of any new directories you have created to be read/writeable by cvsusers:
$ cd $CVSROOT/org/paneris/
$ chgrp -R cvsusers yourprojectThis will ensure that anyone who is in the cvsusers group (all of us) can alter the repository! Remember to add new users to the cvsusers group, if appropriate!
If you're not creating a new project (you are altering an existing project), you should be able to just cvs commit. (remembering that you might have to cvs update to get any changes that others have made to the repository).
I then went to /usr/local/packages/inst/paneris and did cvs
checkout org
To "install" the project I had to compile the java org/paneris/util and put db.properties file into the global java classroot:
$ mkdir /usr/local/share/java (You won't every need to do it again!)
$ cd /usr/local/packages/inst/paneris/org/paneris/util
$ javac -d /usr/local/share/java *.javaBe nice and allow other people to compile as well by altering the permissions on any new directories (modules) and class files you produce: chgrp -R cvsusers /usr/local/share/java
$ ln -s /usr/local/packages/inst/paners/org/paneris/util/db.properties /usr/local/share/java/org/paneris/util/db.properties
I also had to install the generic db administration code:
Compile the handlers:
$ mkdir /usr/local/share/handlers/dbadmin
$ chgrp cvsusers /usr/local/share/handlers/dbadmin
$ cd /usr/local/packages/inst/paneris/org/paneris/dbadmin/handlers
$ javac -d /usr/local/share/handlers/dbadmin *.javaDon't forget to change the group for all new files to cvsusers.
I had to add a wrapper.classpath=/usr/local/share/handlers/dbadmin line to /usr/local/apache/etc/jserv.properties.
I had to add a servlet.Administration.code=org.webmacro.servlet.Reactor line to /usr/local/apache/etc/webmacro.properties for each handler.
I had to link into the templates for the dbadmin system:
ln -s /usr/local/packages/inst/paneris/org/paneris/dbadmin/templates /usr/local/share/templates/dbadmin
I then installed the sugar project by:
Compile the handlers:
$ mkdir /usr/local/share/handlers/sugar
$ cd /usr/local/packages/inst/paneris/org/paneris/sugar/handlers
$ javac -d /usr/local/share/handlers/sugar *.javaAnd obviously change the permissions.
I had to add a wrapper.classpath=/usr/local/share/handlers/sugar line to /usr/local/apache/etc/jserv.properties.
I had to link into the templates for sugar:
ln -s /usr/local/packages/inst/paneris/org/paneris/sugar/templates /usr/local/share/templates/sugarI had create the sugar database (using psql on ganesh) and set it up for the datadictionary. You can do this by running the sql scripts in .../dbadmin/scripts and .../sugar/scripts.
$ su - postgres
$ psql sugar
Welcome to the POSTGRESQL interactive sql monitor:
Please read the file COPYRIGHT for copyright terms of POSTGRESQL
type \? for help on slash commands
type \q to quit
type \g or terminate with semicolon to execute query
You are currently connected to the database: sugar
sugar=> \i /usr/local/projects/dbadmin/scripts/createdatadictionary.sql
sugar=> \i /usr/local/projects/dbadmin/scripts/createadministration.sql
sugar=> \i /usr/local/projects/sugar/scripts/createsugar.sql
I added the new database details to db.properties
I imported the data into sugar by running http://host/webmacro/Upload. Select the correct pairs of table and cvs file and press the button. I'd did this for the first 4 tables (for the datadictionary and admin stuff) and then had to reload the page. When the Upload handler ran this time, the other tables (for the sugar database) were picked up from the datadictionary and I could then upload the data for the 3 sugar tables.
Windows doesn't allow symlinks so you will have to copy some of the source around so that your system will work. Remember to copy all files back into your working copy of a project before uploading it and committing your changes into cvs.
The general plan for development is:
You can combine the FTP and CVS steps by using CVS remotely:
Just pointing a cvs client to ganesh using -d :pserver:root@paneris.org:$CVSROOT will work fine.
Instead of FTP you can use rsync:
If the uploads when you do a "commit" are very slow, then the best solution currently on offer is to maintain a mirror of your directory tree on ganesh, using rsync (which works astonshingly well!). When you want to do a "checkout" or "update", you log on to ganesh, cd to the mirror of your directory, and do "cvs update" or whatever"; then you use rsync to copy the result efficiently to your local machine. When you want to do a "commit", you rsync your ganesh mirror to your local development dir, log on to ganesh and do the "cvs commit" there.
So now all we have to do is write some code...