Monday, March 31, 2008

10 years of Mozilla


Ten years ago we where having pretty much constant house guests, faxes, email bombs and phone calls as Mitchell was setting up the launch of Mozilla. And expecting our kiddo to boot.

It hasn't really stopped. I think they're just getting started.

One of the coolest things they've done is be brave enough to create significant value for their competitors.

Wednesday, March 26, 2008

Section provisioning, de provisioning, and re provisioning

So I got to work on SAK-11320 again.

( I hope these posts sorta explain my work setting... )

We have users who get a snapshot of Registry Roster data into their sites and then TURN OFF roster synchronization by switching Section Administration to manual. I think they just want to reduce the amount of change they have to deal with.

Here's how the Groups look at the beginning of such a cycle:

Groups -fbdb0 and -e0f9a have Provider Ids allowing Sakai to go out and get folks for those groups.

We switch to Manual Section administration.

The Section Manager strips the Sakai Groupz of ProviderIds, and shuffles the current membership over to the Sakai Group. Everything works nicely.



Later on they turn the Roster Sync feed back on. Think of it as refreshing their Roster.

So in doing this last step the underlaying Section Manager impl creates new Realms. It leaves the old Realms. The new Realms are re-wired with ProviderIds, and new people pour in.Now we have 2 new groups. We also have the previous groups. I call them "orphaned" groups. The two new groups are pulling membership from whatever is at the other end of the ProviderId; that info is being used in Site Info and Section info as a source of rosters.

The "orphaned" groups also have Roster information. It may well be out of date with whatever is coming in now from the Provider.

Who cares?

Unfortunately most of the Section Aware tools stash the Group Id in private storage; often in XML in private storage.

This means that any Section ACLs created in Section aware tools are pointing at whatever Group was active when the ACL was created.

Guess what; the new Groupz are not the ones stashed away are they?

The new Groupz mostly have the same people as the old, stashed away Groupz. However the Stashed away Groupz will have people who may of dropped the Course. The stashed away Groupz will not have people who have added the course.

The Section Info tool shows these new people, and does not show the dropped people. Yet the dropped people still get announcements, still have access to section resources. New people are prohibited from getting resources.

Its a help-ticket fest.

So today I put in a small patch which restores the ProviderId for the once-disconnected Groupz. And we don't generate new Groupz for previously provided Sections.

We still have a problem of new sections lingering; but they remain invisible to all but super users... modulo the effects I describe above.

I don't know if it's 'good' to have ACLs wiht non-existing Groupz. That's for tomorrow. Probably each tool will be different.

transparency

So our upstream Course data provider, the Registry, has some upgrades in store for us.

During our testing we found a 10x degradation in response time.

The QA team and the PM on the provider side agree that this is real.

We requested an updated service profile document and response be written and sent up both sides. Our Course data provider PM agreed. After all, it is what it is, and now we need to co-ordinate a response. Perhaps something can be done on their side. As we're a clustered service we can probably roll in more boxes and parallelize the process; what I've written is ready for that.

Upper levels meet, and we hear back that the provider teams management is giving our management grief for "slowing down the rollout by not being responsive." We've been sitting on our side of the code for 6 months, btw.

A week goes by, and we have a project status meeting.

The PM from the provider side tells us he has been specifically prohibited from writing any document detailing the testing results, or even acknowledging the 10x degrading of response time.

Much eye rolling at our mushroom level. There must be an incentive bonus somewhere up there. This really is quite a day.

onward!

Be Positive: Sakai 2.5!

Hey it's out!

Cool, time for a vendor drop @ Stanford and @ Home.

And that Perl admin kit sounds really neat. More admin kits!

From the front lines

So today we're having a big discussion of the type where I'm amazed we're having the discussion.

The issues surround a tool intended to replace a legacy app - the high stakes testing app I described previously.

Our UX folks are contending that "error messages are unnecessary" and that designing the app to support providing end users with error messages is "over engineering."

I've been having discussions like this for a few years. It's like pushing water up hill.

So combine this development approach with our latest piece of fan mail from Professor K over in Civil Engineering:
"I just wanted to let you know how unimpressed I am with the new coursework. It is incredibly un-intuitive, it is very difficult to find things and it is outright frustrating to use it. Like most software (and I have been a use as well as a developer of software for more than 40 years) the geeks who developed this software obviously did not consult any of the potential users because, if they had, they would have found out that this site is a disaster of the first kind. For one thing one has to go through umpteen steps to find anything, and I mean anything. I prefer the older version which was intuitive; one did not have to spend ten minutes to figure out each and every function.

What a waste of money in developing this thing, whatever it is.

Frustrated!
Professor K"
Our erstwhile support staff again asked what specifically was hindering her, and if she had taken a look at the training videos. The support staff is looking for specific issues which we could address...

Follow-up
=======
"I did go through, but the hole point is that I never had to do that with the previous one. My time, as is the time of all faculty, is very precious and having to spend half an hour looking yet at another manual on how to use something that is suppose to make my life easy is not the idea. I believe I spent 0 time with the previous one and had no problem doing things including setting up complicated section schedules and meetings, etc. with this one I would not even attempt it.

Sorry to keep griping. I am really wondering who did develop this. I don't they had any academic experience."
wow. I left the original spelling and grammar in place. In some abstracted way we work for this person.

I suppose the professor could research in Google to find out who did develop this.

This is one of those mornings where I ponder moving to Hollywood and being a grip.

Monday, March 24, 2008

CourseWork in iTunes

In a meeting today it was mentioned that Stanford University's Registrar ( a new one, clearly ) is getting excited about publishing 'course content' to the iPhone.. sometime after an update to PeopleSoft/Oracle.

We on the Coursework team could so do that today, using the integration points I had Dana and then Jing work through over the last few years. We started off with our own for CW3, and then moved along to the Michigan code for CW5. It wouldn't be a big deal to blow CW entries into Apples iTunes U for every class, with a simple class description, as a default.

Currently this is done on request, but it's no biggie.

Stanford has a giant LeftHand RightHand problem.

More?

More flooding at BaduCastle. This time in a sewer line. Turns out the folks who did the remodeling also didn't know plumbing. Aside from the leaky foundation ( and part of their addition has no foundation.. :( ) I have pipe problems. ugh.

Too bad the leaking occurred in the ceiling over my mighty den of computers. I did a speed unplugging of the UPSes ( in retrospect, stupid; I should of hit the breakers) and then a hasty shuffling of hardware before getting buckets.

I don't know yet if the Quad will come back up; I've spent all the weekend ripping out carpet and drywall. Oh and a marathon blowing off steam session of Starcraft with kiddo.

Monday, March 17, 2008

traffic circles

The EPGY clients came back today with a neat idea; a Flash 'gatekeeper' for Java applets. One of the recurring problems is ensurng that the Java applet has landed in the browser and has authenticated back to the host for the super secret assessment conversation that has to follow. They have a little stub which handles the timing of that conversation. This will be useful. I like how it all comes round.

The EPGY folks are doing amazing analysis of spoken language elements at the phoneme level; their applet does some of the analysis and then hands back an abbreviated form for server side analysis. @ the end the big iron on the server hands back an automated assessment of the pronunciation. wow.

Design for testing

We've an ongoing high-stakes testing which has never been well funded ( on the service side ) but continues to have high campus expectations. It's sorta typical how $$$ will roll in to fill a lab with equipment but no funding is around for building an ongoing software service support layer. It looks like Stanford is finally stepping up to the challenge of developing a solution. The team will be a mix of our own staff and outside consultants from Texas.

I'm trying to weave into the emerging design a set of capabilities which promote end-to-end testing of the necessary plumbing. I think the test administrators should be able to check out how the service is behaving at any time. Hopefully before turning the controlled test over to the students. My goal is to make it easier for the lab machines to co-ordinate mass downloads and updates through Sakai.

One of the hassles @ stanford is that the our network is essentially completely open. Only in th last couple of years have private routings for services such as Coursework been an option. One of the results has been that high-stakes testing network traffic can collide with some oddball migration of data from a linear accelerator, or a DNA dataset. Try explaining that to a part-time instructor of Hangul. Stanford is starting to separate traffic but it will be important to do a 'look ahead' from the labs.

I'm curious as to why no-one on the Sakai dev list has sensibly responded to my questions about bulk data delivery w/Sakai - whether the CHS architecture and implementation were up to the task of delivering and accepting a large number of discrete files w/in a few seconds. There was a short discussion about Streaming and / or having Apache handle the files ( we're doing that now ), but those responses missed the point about using Sakai ACLs for content access. Lydia opines that no one knows. I think that's true. I think the legacy code will barf all over the server room floor.

After experiments facilitated by the ad-hoc testing design we may have to do as we are with our PHPBB / Apache .ht* file integration, and express Sakai authorizations in a remote mechanism via web services and REMCTL calls.

Lift head

I think I got lift head this weekend past. I was skiing at a ferocious rate, and getting back to the lift every few minutes. The lifts can haul you up a couple thousand feet at a time. The GPS I was carrying was showing ridiculous amounts of total ascent. Headaches were the result. Oh but the skiing was great.

Tuesday, March 4, 2008

Musing while demolishing my yard

To address the flooding at my house I've contracted some firms to destroy the front yard, regrade it, and work on installing some pumping. fun.

New clients

I've been helping a new-to-us group, the Education Program for Gifted Youth (EPGY), asses integration with the Stanford Sakai deployment. They would like to use
Coursework to support their distance learning offering, the Online High School. This is the only of our Stanford clients for whom Coursework isn't a supplement to F2F classroom instruction.

Our first meeting ( I attended via phone ) was fun and interesting, which is a little of a surprise. They had a group of developers perhaps larger than our own, and they were ready to roll! We outlined a couple of ways to proceed:
  • They provide Stanford Registry CourseClass compliant XML documents which we would load into the CM system. After that they use the self-serve model we use with our current client base
  • They use WebServices to build / clone sites and manage the roster directly
  • They use Coursework/Sakai in a fully manual mode.
I then hunted down a dev stage machine I could use for this (we're running out of development space, as each of our Samigo/Test and Quizzes developers keeps 3 instances up for each branch at all times) and wired up a couple of samples.

Then I'm turning them loose and encouraging option B. Their development crew seems eager and capable, so in this case I decided that they could learn by doing. The WSDLs will give them a rough framework, and I've built some sample Sakai sites they can compare against. They'll hunt me down via a mailing list I set up for our collaboration, and when we get to a deeper stage in the partnership we'll open up our bugzilla instance to their QA/Developer team.

why this is going to work

In this case EPGY has a working, accepted example of Sakai; they don't need to do a full assessment analysis. Because of this they aren't asking for Sakai design documents, or database data dictionaries. They trust that we've done that analysis already. This trust releases them to act as adult learners: example and experiment.

Contrast the EPGY team with those organizations who don't have trusted peers with Sakai experience. Those folks looking to adopt Sakai and who do want to do a fuller assessment analysis have to commit to plowing through our collective ant-hill of self generated content and Javadoc. The cost of experimentation is pretty high by the time they've waded through enough to get their confidence level up.

Other bits


Just a couple of links for me to look into later.

Our WebLoad license expired sometime in the last year. The QA team wasn't really driving the use of automated testing ( the eternal fire drill problem ) and when it lapsed we were stuck. I think that Margaret did get some kind of grandfather clause and they've been poking at the newer work with it. I couldn't get them to use jMeter due to the lack of a GUI they grokked, and the need to tinker. ( I can understand that being in an eternal support firedrill leads one to not want to tinker! )

I've been wondering if Selenium would help a bit. Or if there is a way to lower the expressive boundary for writing testing scripts.

In between bobcat observations (oh, there goes another tree!) I ran across drools. That looks like what we were doing 15 years ago at IntelliCorp - rule systems and objects. This is another Rete algorithim implementation, but could be fun. It has a groovy impl too; cool. I left as the SAP cloud descended on IntelliCorp but the work done there continues to influence me today. Heck an old customer looked me up a few weeks ago to reminisce about rule systems and nuclear power plants. ( do not attempt that at home. )

Other partners

Now to see why our partners in the Registry team took 2 weeks to tell us they couldn't access our SVN repository and what's our problem. Up the management tree to boot. The problem is that they assumed we had installed the web front end, which we didn't. Hello? Just ask!