Wednesday, December 10, 2008

when consulting...

...there is one thing which is very important to remember: there are some clients which it is best to just walk away from.

Tuesday, November 18, 2008

NetBeans and Eclipse

Getting NetBeans and Eclipse to work together isn't all that hard. I'm now cycling through the Sakai SOPI App re-work ( spec drift ) using both IDEs against the same checkout. One just has to keep the specific project dependencies wired up correctly.

I think I've figured out the NetBeans performance issues. Watching the OS X Activity Monitor I see that Eclipse uses all the CPUs in this quad g5. NetBeans will spike one CPU and grind away.

It's still dangerous to get ahead of NetBeans rendering!

Friday, November 14, 2008


Plowing on; now the business constraints are more of an issue than uploads, roles, sections and all that.

Saturday, November 8, 2008

Uploads continued

So after building the proper multipart request and posting to the content tool I of course was granted NPEs. The content tool behaves differently than the 2.4 era access servlet, and preforms a lot of wiggling as it gathers magic pipes for accepting the upload. The short of it is that having a session isn't sufficient - the request has to be part of an overarching tool state.

I'll admit I wasn't looking forward to using the content tool this way - scrounging around to ge the tool ID seemed pretty tedious.

The good Dr. S. threw me a bone in comments here yesterday - thanks. I had examined this builds version of the Access servlet and didn't see that the Post did anything but login. I'll have to do some future archeology and see what post 2.4 version of access had the Rutgers extensions to allow uploads. The SAF app I'm building is already using the access servlet a great deal to pull media.

later doh. I should of looked at the zenly named "WebServlet" in the Access project. I had ignored it thinking that the name implied it was somebodies scratchpad. This WebServlet servlet looks more promising. it looks like it's using some deprecated methods but heck if it works off we go. I may have to dork around more; the upload has to go to an area which the the Student cannot Read, and which the Instructor sometimes cannot read, and sometimes to another site altogether. These rules are why I'll probably end up in a WS solution with sessions for both the user and an agent. anyway this WebServlet servlet is where I start my morning coding.

SteveS mentioned webservices - I see that the 2.4 ContentHosting.jws did get some upload endpoints. That will be where I go after the access servlet. Steve, is there an alternate webservice package you would recommend?

In my original plan I fall back to working with a Java WebDAV client if I can't get the access upload verion integrated or the WS to work out quickly. I expect that would be a real mess

Friday, November 7, 2008

Uploading to Sakai 2.4 foo

reverse engineering Sakai's Resource tool uploading. I'll be doing this from outside Sakai so I want to see what the URLs are like.

first it looks like there is a POST to set the folder / location

POST /portal/tool/0ed2a194-16d0-425f-8022-1250ca000fe1?panel=Main HTTP/1.1

Host: localhost:9595

User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv: Gecko/2008092414 Firefox/3.0.3

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

Referer: http://localhost:9595/portal/tool/0ed2a194-16d0-425f-8022-1250ca000fe1?panel=Main

Cookie: JSESSIONID=388ce121-9b5d-4bbe-8098-e904e98e0477.santoku

Content-Type: application/x-www-form-urlencoded

Content-Length: 217


then a POST multipart to get the stuff up:


POST /portal/tool/0ed2a194-16d0-425f-8022-1250ca000fe1/sakai.resource.type.helper.helper?sakai_action=doUpload&flow=save HTTP/1.1

Host: localhost:9595

User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv: Gecko/2008092414 Firefox/3.0.3

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive

Referer: http://localhost:9595/portal/tool/0ed2a194-16d0-425f-8022-1250ca000fe1/sakai.resource.type.helper.helper

Cookie: JSESSIONID=388ce121-9b5d-4bbe-8098-e904e98e0477.santoku

Content-Type: multipart/form-data; boundary=---------------------------126032761616950317512028061602

Content-Length: 3839


Content-Disposition: form-data; name="fileCount"



Content-Disposition: form-data; name="lastIndex"



Content-Disposition: form-data; name="exists_0"

( I cut off the rest of this... )

Wednesday, November 5, 2008

Java for quicktime; no HTTPS?

So now my application is logging into Sakai, pulling university affiliations, workgroups and sundry, and finding all the Sites bearing the mark of SOPI.

The Sakai access URLs are working fine; I'm pulling content I am supposed to be able to pull. The Quicktime Movie URLs are resolving into movies to present to the user. The user answers them - and I was about to crack the nut of uploading the recorded response movies to Sakai when I decided to use a faster remote host at the Client's facility.

So it's all going swimmingly, er, except if I use a HTTPS URL. That's not a big deal for the webservices. Nor for reading text files such as the XML manifest for the assessment media.

Apparently Apple never implemented a HTTPS transport URL resolver for quicktime.

What a headache to dig down into. I was sure that I had something screwed up with self-signed certs, certs not matching host names, and dorked with all sorts of work arounds. ( such as a custom no-op host name verifier )

I finally trawled the Apple Quicktime mailing lists and found a lone posting from a couple of years ago. Feh. An apple engineer suggested reporting a bug.

So I snork up some old IO code and securely spool the files locally only to re-read them into quicktime with a file:/// URL.

This was quite derailing.

However the national election results were a boost!

Wednesday, October 22, 2008

NetBeans IDE; huh?

I've been working in the NetBeans IDE v 6.1 for a few weeks now.

The draw is the Swing UI development tool, Matisse.

I usually work in Eclipse, with the MyEclipse extensions. Unfortunately MyEclipse hasn't seen fit to port Matisse to OS X.

Sun / NetBeans has.

But it's really vastly slower than Eclipse. I often find myself spinning back in my Aeron from the keyboard as I wait to see just what the hell NetBeans is making of my typing. The response lag is astounding.

It choosing to delete entire blocks of code isn't uncommon.

Once I have this UI nailed down enough to spin back to Eclipse I'm outta here.

Tuesday, October 21, 2008

Bad Test

I've been hacking away at this SOPI assessment tool. It is a Swing app using Quicktime for Java and the Swing Application Framework.

My first pass is to make it work. I tend to follow with a make it good, and then a make it right pass. In each pass I try to write code which is easy for the next poor person to follow. This means a lot of comments when the variable names don't help, lots of logging, lots of feedback to the end user and all that stuff. Actually I'll punt on make it right if I don't think that the next joe or jane in the organization will ever have a chance to look at the code.

So I spent a few hours, post refactoring, wondering where I had sunk the boat. All of a sudden the Quicktime movies started shuddering to a halt as the questions were being presented. I was sure I had introduced some kind of thread starvation. The end of movie handlers were still working fine, the testee could record new movies in response, but there was no guarantee that the next movie would complete.

After serial reversions I was back to looking at the data itself. As it turns out I had mis-rendered a revision to the first Quicktime movie used in the assessment. It was vastly larger then my other test cases.

As a result it was eating all the memory I had allocated to the IDE - and its spawned child processes in the debugger.


Saturday, October 4, 2008

QuickTime and iTunes

So you may recall that I'm writing a Java desktop application for delivering Language Assessments. The Application is a 'remote' Sakai tool running on the desktop but talking back and forth to Sakai.

One thing that is bugging me is this: I can create a QuickTime movie containing the desired audio track. I can annotate the QuickTime movie with a text 'track' which serves as a label for the content. It shows up when you play the movie. This is sorta kinda like captioning a movie.

I can add QuickTime metadata annotations to the movie; these show up in the QuickTime player and in the Finder just as you would expect: Artist, Album, Description, blah blah. Nice eh?

So why if I import the movie to iTunes does it forget about the metadata? the text track is still there, alabet rather small (I can fix that for sure) but none of the metadata is translated up to the iTunes attribute list.

Isn't this sorta kinda dumb?

These annotations are going to store a lot of stuff about the assessment context. But I though it would be useful if the instructor saw some of this stuff in the iTunes playlist when they imported the .mov files.

If they just download the .mov's from the Site Resources's tool then it won't matter as much. But it seems like a waste to lose automated tagging of these resources ( for example)

The Text Track I write may suffice: "UserEID Site Assessment# Question#"

( ach, I'll post some images as soon as Blogger gets over its internal errors )

Tuesday, September 23, 2008

Stanford Meltdown

Ach, it looks like the new Syllabus tool is causing this fall's Stanford Sakai meltdown.

The new Syllabus tool is exhibiting the same pattern ( of an insane # of Realms queries ) which the Profile and Roster tools did a while ago.

Lydia and crew are trawling the source code now. I've suggested that they look for the antipatterns in SAK-8434.

Monday, September 8, 2008

Summers End

Summer is over for me; kiddo is happily back to school, Mitchell is head first back into browser wars, the fridge has died and so there's no lemonade or beer available...

I'm doing fresh checkouts of Stanford's Sakai variant, CourseWork. I'm doing updates of my checkout of Sakai's trunk... There is an impedance mis-match which makes my ears ring. Doing some more re-wiring, shaking down work...

But summer. Sigh. This was a nice summer. kiddo matured in all sorts of interesting ways. got some family kayaking in. The hammock is still up. Perhaps I should knock off for the morning and have lunch out there.

Wednesday, August 6, 2008


I think that it's interesting that Google's App engine is implemented in Python. After all they've hired so many people to work on Javascript - and Javascript has a long history as a server-side platform.

Wednesday, July 9, 2008

A break from coding

I've been (greatly) enjoying a break from coding the last couple of weeks. I'm attending the Digital Media Academy's Final Cut Pro courses; Beginning and Advanced.

It's a hoot! It's on Stanford's campus. Even better my kiddo is here too - he took Web page design / Flash Animation last week and Game Design with a couple of his Peninsula buddies this week. They are gibbering with excitement over there.

The kid's program is really well run. They're hanging out in one of the Group houses on the cool and shady side of campus.

Recommended by Casey!

Wednesday, June 11, 2008

the portable Subversion server

I've been working out the details of working in my home office full time. One of the goals is to get some redundancy and recovery into this environment.

I started out with moving the Quad G5 into the next room, cutting a hole or two in the walls and running firewire & USB cables back and forth. That makes the office a lot quieter! The audio rack, HR824's, and displays remain in the office.

I wanted to create my own Subversion (svn) server, but I also don't want to open more ports on the home firewall. My server ISP, Hurricane Electric, I don't think is really into being my own SVN host. And since I expect to be wandering around, and don't want to explore git yet, I decided that I'll use my laptop, Santoku, as the house svn repo.

I found these instructions: to be foolproof. Really - don't be thrown off by the page name. The author's 'basic subversion structure' is not my cup of tea. I like the structure to be more focused on the artifact being stored - the product, and the artifacts to be under the product: /repo/BOB_THE_EDITOR/trunk as opposed to /repo/trunk/BOB_THE_EDITOR

Then a few firewire disks; one for Santoku and a couple for the G5. And a new SATA internal drive for the G5, as it's getting tight in there with all the Sakai and Native Instruments stuff - and one of those cool cable adapter kits which lets me run the SATA drive nekked while I migrate the G5's boot volume. These are multi-headed cables with adapters for IDE, ATA and SATA drives. The result is a mess of wires and a shiny hard disk sitting out exposed and running. An OS X Disk cloning tool, SuperDuper, was then used to make bootable backups for each machine. That can be a long process.

While that was running I took the time to dust off an old P3 with some SCSI drives and ran through installing Debian, Red Hat, and Ubuntu. I found that Debian had a hard time with the old oddball equipment. Red Hat - well, it's professional and all that but I found the unending user registration process to be really intrusive. And as no one I like works there anymore, feh.

Ubuntu won out. It did a better job of dealing with the oddball hardware too, even if it complained of the old BIOS. Now it'll serve as the household fileserver back in the call center. ( yes this house has a call center. I think it has a checkered past. )

This is a lot of plate juggling.

Monday, June 9, 2008

The travels of the Sakaiger

I finally had some time to deal with photographs - here is Dr. Chuck and the Sakaiger in action @ Google I/O. It was fun to see him, and really useful to close up some loose ends.

Thursday, May 29, 2008

Google I/O 2

Second day, and we're off with Melissa as she describes some principles in play at google.

"Imagination is a muscle" is the one which resounds most soundly with me.

some good quotes from this presentation:

Henry Ford: customers wld say they wanted faster horses
Sam Walton: if you don't listen to your customers, someone else will

I'm now loitering at the "GWT and client/server communication" talk. At first I've been pointedly ignoring GWT; watching folks battle JSF has been a counter example for adopting release 1.0 technologies. I'll add more as it starts up, right now they are using crowbars to fit everyone into the room.

-- later, after bandwidth was restored.

The GWT talk was inspiring. What a huge hunk of work they have been doing.

Next"Using google data APIs from non javascript languages" Vidam Spivak.

covering Search, Feed and Language AJAX APIs. pivoting on RESTful interfaces.

JScript isn't considered to work in : latency sensitive websites, Flash/Silverlight, and server-side apps.

Vidam goes through a tour of the APIs as an overview.

The AJAX language API is cool; highlight text and it tries to figure out
the source language and translate.


JS controls & UI
JS runtime
RESTful data access

this talk focuses on the RESTful layer

- read only

shows a CURL request ( bonus points )

FLASH crossdomain file is avail on google to allow
flash requsts against google - allows all requests ( silverlight too)

referrer -
specify URL where used, or your homepage or a google ID

search returns 32 results
lang 500 characters
feed 100 feed entries

--- examples,
"faster AJAX"
use DOMContentLoaded - its faster than an OnLoad handler, works cross browser
possible to have the call to G use your function to handle the content on their side before it is returned, cool.

Flash: ActionSctript 3 use HTTPService, layout via MXML, and jason parser 'corelib' for AScript.

Server side example;: Facebooj App, Google App Engine, Google AJAX.
URL Fetch API from APPs pythong
gminifb ( minimal facebook developer app in python )

in facebook translates wall postings to target language using google translation
API. nice.

examples at

--- next: server side javascript.

hah! let's see how this has come along since the old LiveWire days. I used to make a living running a small consulting firm doing LiveWire consulting. but that's another story - time to fight geeks for sandwiches.

RHINO - great lecture, highly entertaining. who the heck is this guy?


Reusing Google APIs with GWT. - migueal mendez

a specific project for the GWT.

GALGWT ( GWT is pronounced "gwit" )

opensource bindings for Google JS APIs.
Project for each API has individual JAR file, doc, etc.
supports Search, gadgets, Gears and Maps.

techniques -
containment ( java holds JS object you access )
JSIO google API interop. allws one to extend JS objects in Java etc.
JavaScriptObject in Java

in GWT 1.5 you cfan declare types which overlay JS objects.

Search overview -

condfigure SearchControlOptions
add options to the SearchControl
add SearchControl to the page
execute searches on the search control

Inspect the reults with a Search Listener . Listner callback gets updates
from search engine.

Gadgets overview -
( go to iGoogle to see a gadget )
extend Gadget class
implement each of the NeedsXXX interfaces for that it needs.
can make XHRs from gadget.
gadget suppport via Generator and Linker
need Gadget container to test against ( iGoogle, shindig, opensocial )

Gears overview
exposes Database, LocalServer and WorkerPool JS modules.
provides an Offline module which automates process of going offline.
provide GWT rebind prop to test for existence of gears, so your app can adopt if gears isnt' present.
"hosted mode" only supports IE on Windows
cannot debug WorkerPool workers currently

Maps overview
exposes Maps API w/cleanup
allows GWT widgets to be added to InfoWindows
Extend the Maps API using Java
( watch out for Maps keys issues )

( the example is dirt simple )

milestone builds:

--- Turbo GWT

First - GwtQuery
- like jQuery, query the DOM. uses what is available: XPath, W3C Selector, getElementsByClassName...
compile0time CSS3 Selector parsing

each browser gets separately compiled JS. ( neat )

!check out GWTQuery at

Second - High power graphics. Display Lists

immediate mode vrs retained mode
-Canvas vs SVG/VML
JS VM top out beween 1-10K method calls on canvas
inserting DOM objects vrs canvas calls.
idea: record and optimize desired graphics calls

Chronoscope Chart library
renders 30K+ interactive rates
supports javascript, flash, applet, servlet and android due to GWT capabilities.

demo: graphing millions of datapoints of all the google trades ( about one a second )
over the last few months. smooth scaling and zooming. the crowd goes wild as seamless zooming from months to seconds scale

Third - GWT with other environments.

GWT and servlets is a given.
Also to browser, via JS calls.
what about RPC to Flash, Android, "Gears Workers?"

write once and publish gadget ti other containers -

"Syndroid" a prototype demo is given. This guy is amazingly talented.

Wednesday, May 28, 2008

Google I/O

I managed to get to the SF Google I/O conference today. I will admit that this is rather interesting.

I missed the Guice presentation in favor of an overview of the Google App APIs and application integration. The latter started so late perhaps I made the wrong choice.

The App APIs look like an easy reach for Sakai. I'll have to go to the authentication session later today, and then look at provisioning.

Now I'm attending a "get google apps running on your mac foo" workshop. Let's see how that goes. My Python is a bit rusty.

( later ) That was too easy. there were some typos in the presentation ( all of them so far ) and the hands on code was missing some python page template content ( ! ) but it really works nicely. I still want an IDE for setting breakpoints.

ach this is all HTTP. too bad. it would have to use anonomized IDs on the google side with a resolution table in the secure side. no threads. no C using Python APIs.

-- later, some notes on the "inside the app datastore" chat.

Google app datastore: bigtable. New term for me: "sharded data." This refers to splitting data across several servers. I've done that but I didn't know the name.

this datastore is a big array. a set of arrays, when you consider the indexes. lots of scanning; they must have a special filesystem at work here. This thing just hammers across the array.

So one can build all these tables. and their indexes. and composite indexes. The presenter is really dancing around how much all this costs the developer. the presenter sez "Im not a PM, and we don't want to charge for composite indexes because that will cause you the app developer to do all sorts of crazy things."

presentation is straightforward. slides look like LATEX.

interesting bit on expando rows; you won't find records for those expandos which lack an attribute used in a compound key. suggestion; reflect across the expando before storing it. (?) I must of missed something there, I guess your app wld keep track of what new attributes had been added in the past and force those attributes on new records? hmm.

transactions - timestamp journals. related things are bundled into "entity groups"

-- later, "Auth to Google APIs"

revocable token.

ClientLogin: lives on app on hardware owned by the user. does deal with users original creds. like cell phone, cant' easily redirect. token expires after a while. one HTTPS POST, U/Pwd, service name ( which Google API) get token back from google. CAPTCHA may be involved. Auth token returned, stuff it into your Authorization header. Nothing prevents a web site from using this approach.

AuthSub: user redirect to google, they auth w/ creds, and are redirected back to you. Tokens can be long-lived. Two redirects; one over and then back. provide an endpoint for the callback. Can be tightly scoped or wide; one token for calendar, contacts & docs. ( yer token is a one time token which is easily upgraded to a long lived session )

Secure AuthSub: so far token is in clear text. to get past that a domain may be registered with Google. during registration provide a public key. it will be used by google to encrypt the token. there is an interesting handshake between you and google where they confirm you administer the domain - for example a special .html file you publish w/in your domain containing data generated during the registration process. cool.

OAuth: "Upcoming standard which looks like AuthSub." Registration, HTTPS req for scoped token, redirects back and forth. finally sign all following requests.

lots of client libraries. hmm, perhaps not; java 1.5 and X. available now.

Sunday, April 20, 2008

Sakai Uptake?

I was wandering through the vast terrain of Sakai technical documentation this weekend. There really is a lot of it. It's all presented rather flatly, if you will. Flat, and wide. And gray.

I'm giving a talk @ Stanford on the "the Sakai Architecture" for the standard set of blind men. Each of them represents a different set of stakeholders for Stanfords stovepipes. So I'm not quite sure which "Architecture" to speak to.

But enough about me.

Looking at this project I started wondering about the uptake of this codebase. There's a lot of stuff here, and it covers a wide set of concerns. A new comer may well feel as the early paleontologists did, pondering bits of fossils mixed in with contemporary remains. What to make of all this?

How many new institutions have come into the fold this last year?

The coders stand a fighting chance due to the Cafe (BOOT) work. But how do the rest of the folks reporting to the deciderators? And jeez, the deciderators?

Just asking.

Tuesday, April 15, 2008

Geek display of shell history

So inbetween building contractor consultations and spring skiing I've been dorking in a desultory fashion with Grails. And reading Planet Mozilla. huh, look at what the cool kids are doing; shell histories.

santoku:~/dev/stan_event_grail/stan_event_admin casey$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}'|sort -rn|head
132 cd
104 ls
66 grails
30 emacs
24 ../stan_scripts/
23 find
18 more
15 ./
10 cp
9 pwd

I guess that means I'm mostly lost :)

I've been building (HA! as if building describes this painless process) an administrative UI for our event buss 'buffer'. That's what the
  • cd ( "where has grails put all this stuff?")
  • the ls ( "what's in here?" )
  • the grails ( "do that again!" )

are about. Emacs is just, well, emacs.

the script is how we do our builds; it creates a common environment and set of commands for our QA folks. Sadly in the little time I had to try to jam the grails app into sakai I was too put out by all the version collisions so I punted on that. is another common environment and management script for running our sakai builds.

I took existing Hibernate config files and rolled them into this Grails stuff. The table space is dirt simple, but the experiment was so painless I'm thinking of moving to some Course Management stuff. We're always dorking around with SQL ( don't let old timers tell you you don't need SQL to deal with Sakai ) to update stuff so a little side tool is warranted.

We've got a guy (Jeff Vance) who is mongeling around some BIRT scripts for reporting against Sakai, and so I dropped this Grails app into his tomcat last night. No Problemo. A Sakai "Web Content" tool takes our Support crew over to the Grails app. Nothing fancy with Auth at this time, but I'll probably just use the Tomcat Kerberos stuff I've wrapped around the BIRT tool and extend it into this Grails app.

Monday, March 31, 2008

10 years of Mozilla

Ten years ago we where having pretty much constant house guests, faxes, email bombs and phone calls as Mitchell was setting up the launch of Mozilla. And expecting our kiddo to boot.

It hasn't really stopped. I think they're just getting started.

One of the coolest things they've done is be brave enough to create significant value for their competitors.

Wednesday, March 26, 2008

Section provisioning, de provisioning, and re provisioning

So I got to work on SAK-11320 again.

( I hope these posts sorta explain my work setting... )

We have users who get a snapshot of Registry Roster data into their sites and then TURN OFF roster synchronization by switching Section Administration to manual. I think they just want to reduce the amount of change they have to deal with.

Here's how the Groups look at the beginning of such a cycle:

Groups -fbdb0 and -e0f9a have Provider Ids allowing Sakai to go out and get folks for those groups.

We switch to Manual Section administration.

The Section Manager strips the Sakai Groupz of ProviderIds, and shuffles the current membership over to the Sakai Group. Everything works nicely.

Later on they turn the Roster Sync feed back on. Think of it as refreshing their Roster.

So in doing this last step the underlaying Section Manager impl creates new Realms. It leaves the old Realms. The new Realms are re-wired with ProviderIds, and new people pour in.Now we have 2 new groups. We also have the previous groups. I call them "orphaned" groups. The two new groups are pulling membership from whatever is at the other end of the ProviderId; that info is being used in Site Info and Section info as a source of rosters.

The "orphaned" groups also have Roster information. It may well be out of date with whatever is coming in now from the Provider.

Who cares?

Unfortunately most of the Section Aware tools stash the Group Id in private storage; often in XML in private storage.

This means that any Section ACLs created in Section aware tools are pointing at whatever Group was active when the ACL was created.

Guess what; the new Groupz are not the ones stashed away are they?

The new Groupz mostly have the same people as the old, stashed away Groupz. However the Stashed away Groupz will have people who may of dropped the Course. The stashed away Groupz will not have people who have added the course.

The Section Info tool shows these new people, and does not show the dropped people. Yet the dropped people still get announcements, still have access to section resources. New people are prohibited from getting resources.

Its a help-ticket fest.

So today I put in a small patch which restores the ProviderId for the once-disconnected Groupz. And we don't generate new Groupz for previously provided Sections.

We still have a problem of new sections lingering; but they remain invisible to all but super users... modulo the effects I describe above.

I don't know if it's 'good' to have ACLs wiht non-existing Groupz. That's for tomorrow. Probably each tool will be different.


So our upstream Course data provider, the Registry, has some upgrades in store for us.

During our testing we found a 10x degradation in response time.

The QA team and the PM on the provider side agree that this is real.

We requested an updated service profile document and response be written and sent up both sides. Our Course data provider PM agreed. After all, it is what it is, and now we need to co-ordinate a response. Perhaps something can be done on their side. As we're a clustered service we can probably roll in more boxes and parallelize the process; what I've written is ready for that.

Upper levels meet, and we hear back that the provider teams management is giving our management grief for "slowing down the rollout by not being responsive." We've been sitting on our side of the code for 6 months, btw.

A week goes by, and we have a project status meeting.

The PM from the provider side tells us he has been specifically prohibited from writing any document detailing the testing results, or even acknowledging the 10x degrading of response time.

Much eye rolling at our mushroom level. There must be an incentive bonus somewhere up there. This really is quite a day.


Be Positive: Sakai 2.5!

Hey it's out!

Cool, time for a vendor drop @ Stanford and @ Home.

And that Perl admin kit sounds really neat. More admin kits!

From the front lines

So today we're having a big discussion of the type where I'm amazed we're having the discussion.

The issues surround a tool intended to replace a legacy app - the high stakes testing app I described previously.

Our UX folks are contending that "error messages are unnecessary" and that designing the app to support providing end users with error messages is "over engineering."

I've been having discussions like this for a few years. It's like pushing water up hill.

So combine this development approach with our latest piece of fan mail from Professor K over in Civil Engineering:
"I just wanted to let you know how unimpressed I am with the new coursework. It is incredibly un-intuitive, it is very difficult to find things and it is outright frustrating to use it. Like most software (and I have been a use as well as a developer of software for more than 40 years) the geeks who developed this software obviously did not consult any of the potential users because, if they had, they would have found out that this site is a disaster of the first kind. For one thing one has to go through umpteen steps to find anything, and I mean anything. I prefer the older version which was intuitive; one did not have to spend ten minutes to figure out each and every function.

What a waste of money in developing this thing, whatever it is.

Professor K"
Our erstwhile support staff again asked what specifically was hindering her, and if she had taken a look at the training videos. The support staff is looking for specific issues which we could address...

"I did go through, but the hole point is that I never had to do that with the previous one. My time, as is the time of all faculty, is very precious and having to spend half an hour looking yet at another manual on how to use something that is suppose to make my life easy is not the idea. I believe I spent 0 time with the previous one and had no problem doing things including setting up complicated section schedules and meetings, etc. with this one I would not even attempt it.

Sorry to keep griping. I am really wondering who did develop this. I don't they had any academic experience."
wow. I left the original spelling and grammar in place. In some abstracted way we work for this person.

I suppose the professor could research in Google to find out who did develop this.

This is one of those mornings where I ponder moving to Hollywood and being a grip.

Monday, March 24, 2008

CourseWork in iTunes

In a meeting today it was mentioned that Stanford University's Registrar ( a new one, clearly ) is getting excited about publishing 'course content' to the iPhone.. sometime after an update to PeopleSoft/Oracle.

We on the Coursework team could so do that today, using the integration points I had Dana and then Jing work through over the last few years. We started off with our own for CW3, and then moved along to the Michigan code for CW5. It wouldn't be a big deal to blow CW entries into Apples iTunes U for every class, with a simple class description, as a default.

Currently this is done on request, but it's no biggie.

Stanford has a giant LeftHand RightHand problem.


More flooding at BaduCastle. This time in a sewer line. Turns out the folks who did the remodeling also didn't know plumbing. Aside from the leaky foundation ( and part of their addition has no foundation.. :( ) I have pipe problems. ugh.

Too bad the leaking occurred in the ceiling over my mighty den of computers. I did a speed unplugging of the UPSes ( in retrospect, stupid; I should of hit the breakers) and then a hasty shuffling of hardware before getting buckets.

I don't know yet if the Quad will come back up; I've spent all the weekend ripping out carpet and drywall. Oh and a marathon blowing off steam session of Starcraft with kiddo.

Monday, March 17, 2008

traffic circles

The EPGY clients came back today with a neat idea; a Flash 'gatekeeper' for Java applets. One of the recurring problems is ensurng that the Java applet has landed in the browser and has authenticated back to the host for the super secret assessment conversation that has to follow. They have a little stub which handles the timing of that conversation. This will be useful. I like how it all comes round.

The EPGY folks are doing amazing analysis of spoken language elements at the phoneme level; their applet does some of the analysis and then hands back an abbreviated form for server side analysis. @ the end the big iron on the server hands back an automated assessment of the pronunciation. wow.

Design for testing

We've an ongoing high-stakes testing which has never been well funded ( on the service side ) but continues to have high campus expectations. It's sorta typical how $$$ will roll in to fill a lab with equipment but no funding is around for building an ongoing software service support layer. It looks like Stanford is finally stepping up to the challenge of developing a solution. The team will be a mix of our own staff and outside consultants from Texas.

I'm trying to weave into the emerging design a set of capabilities which promote end-to-end testing of the necessary plumbing. I think the test administrators should be able to check out how the service is behaving at any time. Hopefully before turning the controlled test over to the students. My goal is to make it easier for the lab machines to co-ordinate mass downloads and updates through Sakai.

One of the hassles @ stanford is that the our network is essentially completely open. Only in th last couple of years have private routings for services such as Coursework been an option. One of the results has been that high-stakes testing network traffic can collide with some oddball migration of data from a linear accelerator, or a DNA dataset. Try explaining that to a part-time instructor of Hangul. Stanford is starting to separate traffic but it will be important to do a 'look ahead' from the labs.

I'm curious as to why no-one on the Sakai dev list has sensibly responded to my questions about bulk data delivery w/Sakai - whether the CHS architecture and implementation were up to the task of delivering and accepting a large number of discrete files w/in a few seconds. There was a short discussion about Streaming and / or having Apache handle the files ( we're doing that now ), but those responses missed the point about using Sakai ACLs for content access. Lydia opines that no one knows. I think that's true. I think the legacy code will barf all over the server room floor.

After experiments facilitated by the ad-hoc testing design we may have to do as we are with our PHPBB / Apache .ht* file integration, and express Sakai authorizations in a remote mechanism via web services and REMCTL calls.

Lift head

I think I got lift head this weekend past. I was skiing at a ferocious rate, and getting back to the lift every few minutes. The lifts can haul you up a couple thousand feet at a time. The GPS I was carrying was showing ridiculous amounts of total ascent. Headaches were the result. Oh but the skiing was great.

Tuesday, March 4, 2008

Musing while demolishing my yard

To address the flooding at my house I've contracted some firms to destroy the front yard, regrade it, and work on installing some pumping. fun.

New clients

I've been helping a new-to-us group, the Education Program for Gifted Youth (EPGY), asses integration with the Stanford Sakai deployment. They would like to use
Coursework to support their distance learning offering, the Online High School. This is the only of our Stanford clients for whom Coursework isn't a supplement to F2F classroom instruction.

Our first meeting ( I attended via phone ) was fun and interesting, which is a little of a surprise. They had a group of developers perhaps larger than our own, and they were ready to roll! We outlined a couple of ways to proceed:
  • They provide Stanford Registry CourseClass compliant XML documents which we would load into the CM system. After that they use the self-serve model we use with our current client base
  • They use WebServices to build / clone sites and manage the roster directly
  • They use Coursework/Sakai in a fully manual mode.
I then hunted down a dev stage machine I could use for this (we're running out of development space, as each of our Samigo/Test and Quizzes developers keeps 3 instances up for each branch at all times) and wired up a couple of samples.

Then I'm turning them loose and encouraging option B. Their development crew seems eager and capable, so in this case I decided that they could learn by doing. The WSDLs will give them a rough framework, and I've built some sample Sakai sites they can compare against. They'll hunt me down via a mailing list I set up for our collaboration, and when we get to a deeper stage in the partnership we'll open up our bugzilla instance to their QA/Developer team.

why this is going to work

In this case EPGY has a working, accepted example of Sakai; they don't need to do a full assessment analysis. Because of this they aren't asking for Sakai design documents, or database data dictionaries. They trust that we've done that analysis already. This trust releases them to act as adult learners: example and experiment.

Contrast the EPGY team with those organizations who don't have trusted peers with Sakai experience. Those folks looking to adopt Sakai and who do want to do a fuller assessment analysis have to commit to plowing through our collective ant-hill of self generated content and Javadoc. The cost of experimentation is pretty high by the time they've waded through enough to get their confidence level up.

Other bits

Just a couple of links for me to look into later.

Our WebLoad license expired sometime in the last year. The QA team wasn't really driving the use of automated testing ( the eternal fire drill problem ) and when it lapsed we were stuck. I think that Margaret did get some kind of grandfather clause and they've been poking at the newer work with it. I couldn't get them to use jMeter due to the lack of a GUI they grokked, and the need to tinker. ( I can understand that being in an eternal support firedrill leads one to not want to tinker! )

I've been wondering if Selenium would help a bit. Or if there is a way to lower the expressive boundary for writing testing scripts.

In between bobcat observations (oh, there goes another tree!) I ran across drools. That looks like what we were doing 15 years ago at IntelliCorp - rule systems and objects. This is another Rete algorithim implementation, but could be fun. It has a groovy impl too; cool. I left as the SAP cloud descended on IntelliCorp but the work done there continues to influence me today. Heck an old customer looked me up a few weeks ago to reminisce about rule systems and nuclear power plants. ( do not attempt that at home. )

Other partners

Now to see why our partners in the Registry team took 2 weeks to tell us they couldn't access our SVN repository and what's our problem. Up the management tree to boot. The problem is that they assumed we had installed the web front end, which we didn't. Hello? Just ask!

Thursday, February 14, 2008

Sakai and LDAP

Another 'Sakai and ...' I wonder if folks could use a LDAP gateway to CM / Realms&Groups too. I know that our institution (Stanford) has been unable to deploy a Course Directory, but that service is still greatly desired.

As our deployment contains a clone of the Registrar's data we're in a great position provide an LDAP view into that data.

I bet a lot of smaller institutions may have their Sakai data being the only such repository. Of course they're also probably using webservices to get the data out.

But if we cld give all the stand alone mail tools an LDAP gateway... hmm.

Wednesday, February 13, 2008

Sakai and ...

A good chat with our core IT group (CourseWork /sakai are not members of Stanford IT, but I've worked in that group.) about:

  • Sakai and Shibboleth
  • OpenId and Federated Ids
  • Sakai and Zimbra
  • the future of Remedy, our 'ticket' system
it was fun.

On my own I'm wondering about

  • Sakai and Facebook
  • Sakai as FUSE host
  • Sakai and GMail

Documentation daze.

Having an odd couple of days - many shorter blocks of useful time as
opposed to long blocks - I return to updating documentation and bumping old threads. I'm not sure how useful this always is; do people read this stuff?

Internal documentation seems to often be an exercise in self definition; in making sure that what I thought I did is what happened. That's useful, as bugs come out of the exercise.

Tuesday, February 5, 2008

Skiing in Davos

Skiing in Davos
Originally uploaded by Robert Scoble
I did get to get out in Davos and ski a couple of days. I ran into the famous Scoblizer who was gracious enough to take a few thousand pictures.

This shot was taken in Parsennes. I'm the dark figure in the foreground, a fine pharmacist named Jennifer is scooting on ahead of me. The town of Davos is in the valley below.

Cloaking Device project update

I put the Sakai User Cloaking Device project on hold sometime in December.

We had issues getting our Load Testing project into motion. For example, not enough local licenses to do load testing using our existing software and getting QA bandwidth to push it through. In our analysis we were worried about having enough local network bandwidth to do it ourselves. The concern was that if we used the PC / Mac student clusters during off-hours we would bottleneck on transport out to the server pool. So we were looking for vendors with multiple locations. They would come into the Stanford net from the big, outward facing pipes.

It's still a neat idea, and I suppose that a dedicated group could do heavy ( like 1500 simultaneous users ) testing using jmeter and a cluster of machines. As jmeter doesn't 'do javascript' I'm not so sure how that would work out.

Sakai Roadmap?

I was just reading a tasty morsel ( ) posted by the esteemed Ray Davis.

It inspired me to mutter: "Heck I have all that stuff laying around here, we use it every day!" and then to look for the Sakai release Roadmap - "...just when are we thinking 2.6 will be freezing? Perhaps I can contribute this stuff in on behalf of Stanford?"

I dash off to the Sakai project site to find out. I don't find out. It must be there.

Wednesday, January 30, 2008

posting from Python

just dorking - trying to figure out how to post from WordPy.

I should move to my next local bug, having just corrected our additions to which post sakai events for provider changes, but I wanted to take a little break.

Wednesday, January 23, 2008

crazy things you hear in davos

I made it into Davos last night.

The European rail system was a welcome relief from the rent a car drive and park drill, I watched the landscape grow whiter as I climbed into the Alps.

Mitchell & I met up for a nice Thai dinner and planned which sessions we would attend. We tend to skip the internet sessions as they are often recaps for us. Its more fun to go to sessions on topics we are not so involved in.

At an interesting session on the Arab world I fohund out that the Iraq war was orchistrated from Israel. I didnt know there were Israeli oil companies.

Something more easily believed was that the Arab and Egyptian youth are heavily wired.

tech note: the WEF stopped handing out those funky palm pilots. They're of theopinion that everyone has a handheld, like this Nokia 810 I am using. (so forgive the typos.)

Monday, January 14, 2008

When is a Realm not a Realm?

In chasing around the corners of SAK-11320 I stumbled across an odd piece of behavior.

I was pulling all the Realms for a Site with intent to refresh them. When I refreshed the Realms Section Info started to lose track of Section members. The Realms were reporting the proper membership but SI was reporting empty sets. The catalina log had a helpful message about "needing at least one Student." The CM data had the students, the Realm / Group had the students, what gives?

As I kept eyeballing the Realms in the Realm tool I finally scrolled down to the Roles display. No Roles? huh? We have PlentyOfRoles at Stanford?!

My code pulled the Realm/Groups for the Site via the getAuthzGroups(Criteria, Pager) method. It turns out that the resulting Realm Groups are not fully populated; they are missing the Roles! When the Realm refresher goes out and ... well, refreshes the realm it uses the Roles defined in the Realm as part of the refresh. Since the list accessor wasn't providing Realms/Groups with Roles the refresh was incomplete.

Section Info was looking for the Role of Student - or at least one person with the Role of Student. It didn't care that there were people in there; w/out a role what good are they,eh?

It became necessary to re-query for each Realm returned by the getAuthzGroups call individually. Then saving the Realm refreshed the data from the Provider and kept the Roles.

Thursday, January 10, 2008

Fan Mail.

Here is some fan mail from a Stanford instructor who was trying to load a MS Word document into the IMS QTI importer:

Remedy wrote:
Coursework will not upload a word documnet I have tried remanimg etc. It is only two page homework assignment. I must say that the new coursework is anythign but user friendly. I have used computers for almost 40 years and consider myself extremely knowledgeable with computer software but this new version leaves a lot to be desired. Why is it that smart computer scientists just can't get it right?

Here is the error that I keep getting. I never had a problem with this homework in previous years.

There was an error importing this assignment. Ensure that the file is correctly formatted IMS QTI. Error java.lang.NullPointerException

A timely response would be greatly appreciated.
[ name removed ]
I've left the spelling as we got it from Remedy, our help ticket system. Our Supporter pointed out that that there are more appropriate places in Sakai to upload simple documents. The testing tool, Samigo, sure isn't it. I added with a pointer to the IMS project. The thread continues in email to me, directly:

Why is it that software people insist that I need to reed tuns of material to do something as simple as posting a homework on the coursework webpage. It use to be so easy. Why did it become so complicated that now I need to spend my precious time reading how to load material on the page? You must think that faculty just have nothing better to do. I really don't care to learn another protocol or whatever this is. I just want ot upload the homework so that the student can get it. It is really rendering the website useless if I need to go through all kinds of hoops. I will just photocopy it and get it over with.
[name removed]

PS Just because other universities are using it does not mean it is something great!
I just hope this instructor uses a spelling checker before going to the photocopier.

Our UserX group made instructional videos, for Pete's sake, because they knew folks wouldn't 'reed tuns of material' before they started using Coursework/Sakai.

back to bugs.

Start of Term & New Release

The Stanford Winter term started Tuesday without Marc Brierley.

What I liked best about Marc was how he really really tried to engage you, and was dogged and admirably pleasant about it. You couldn't even shake him too much by being rude to him. He would always look for another tact. He was really getting the whole software-development picture too. Our team is really damaged right now, and there is a very low grade of functioning as people are trying to cope. I can't really imagine how his wife and parents are doing.

So the Quarter.

We didn't have the login-latency/failure and DB load issues we hit in fall. That's great, but it wasn't worth coming back from skiing for... :) ( wonderful powder in Tahoe after the winds died down.)

Operationally our issues were:
  • We didn't catch that one of the servers wasn't back up after the Holiday Upgrade. This is a 'back office' server which isn't in the end user pool, so it's not in the monitored pool. It's running with a modified Quartz configuration which provides a private Job pool running only on that machine.
  • Some of the Out Of Band (OOB) Course Site administration Quartz jobs which run on that machine were not defined to work in sync with our partners CRON jobs.
  • A return Tuesday night of an occasional lack of SSL setup from our Sakai boxes to the Registry. We failed to pull 5-600 XML Course documents that night. (Yes I built it to pause and retry...) This of course causes a great deal of confusion. Wednesday night we had not a single problem - no code change of course. I've seen SSL setup failures on low-memory conditions and in Tomcat when class loading of the jsse stuff gets mal-sequenced (really, search the tomcat newsgroups). My Operations Guy, Julian Morley and I have not yet been able to reproduce the condition, so we have an out of user-pool production Sakai server running with hoping we get a catch.

End-user issues abound due first to the Sakai UI. Users can't find their Course Sites. Here is a summary:
  • The tabs showing Course Sites are in their usual wonderful semi-chronological order, and W08 Sites are buried under the "more" pulldown. There must be dozens of these a day.
  • Not all instructors have published their Sites ( some don't know how. many don't want to know how. )
  • The student signed up for the Course 10 min ago and the data hasn't flushed through to Sakai yet.
  • Not all Stanford courses are represented in Coursework. They are all present in the Peoplesoft front end, Axcess, and so many folks expect them to be automatically present in Coursework/Sakai... which is a strict no-no from the Stanford instructional staff. (Many want no semblance of any online presence for their efforts. Check your calendars, folks.)
  • Instructors are quickly hired and it takes a few days for their records to flow through the sad slurry of tubes and pumps Stanford uses to send data around.
  • Enterprise network Ids, the SUNetIDs, are in delayed for provisioning.
  • Courses are cross-listed only in the minds of the instructors...
  • Courses IDs are different than what the Student things...
Since this is only the second quarter in Sakai for the vast majority of our users I expect the HelpSU tickets to start falling off rapidly.

There is a stickler - sometimes we still need to kick the Site Info Update Participants button to refresh the Course Site. This is after we can see the user info flow from the Registry, to Sakai's Course Management tables, and the Site Realm getting updated... and still now and again the user does not appear to be in the site.

The biggest problem is the broken Sections / Realms bug, SAK-11320 where Section Aware tools using ACLs in XML lose track of Realm Membership via Section Manager updates. (ACLs in XML? in a column? ah well.) I understand that Michigan goes in and updates the XML itself when there are problems. This effects all sorts of not-even-really-advanced uses of Sakai Sections.

Our CM Sync code cleans up orphaned Realms. I think this causes the problem to become apparent; otherwise only 'new people' will have ACL problems in the Section Aware tools, and I'm not sure how often other Sakai deployments churn their Sections. An easy thing I can do is leave that hunk of code out. This will result in an astounding growth of unused / out of date Realms. I have to think that other big deployments trim these after time.

At least then we won't see very abrupt changes in access rights.

I'm going to go work on that now. In our Stanford 2.4.x branch. Then I'll see if I can get a Sakai 2.5 build going and if the patch is appropriate for the next release.

Then I go to Davos Switzerland! whoo hoo!