Thursday, December 16, 2010

Mediawiki foo

Some time ago Kristian Lyngstøl blogged about his wiki.sh hack which provides commandline "wikiget", "wikipost" and "wikiedit" interfaces to mediawikis. I planned to use it for a client project I'm doing right now. It turns out the restriction "supports only basic auth" was rather hard to overcome, at least for this one particular mediawiki instance I needed to make pages on.

So, in the good tradition of one-up-man-ship here is how I do mediawiki:

I was programming my project in perl in any case, so the python inferface suggested by a python-colleague would be non-optimal. As well as not included in Ubuntu afakt. But Mediawiki::API is included. The code below is mostly cut and paste from the documentation:


use Mediawiki::API;

my $wh;

sub wiki_upload {
my ($page,$content) = @_;

# First time use, generate object and log in.
if (!defined($wh)) {
$wh = MediaWiki::API->new( { api_url => $wikiapi } );
$wh->login( { lgname => $user, lgpassword => $pw } ) or
die "Wiki error: ".$wh->{error}->{code}.': '.$wh->{error}->{details};
}

my $ref = $wh->get_page( { 'title' => $page } ) or
die $wh->{error}->{code} . ': ' . $wh->{error}->{details};

die if $ref->{missing};

# At this time $ref->{*} contains the page contents

my $timestamp = $ref->{timestamp};

$wh->edit( {
'action' => 'edit',
'title' => $page,
'basetimestamp' => $timestamp,
'text' => $content
} )
or die $wh->{error}->{code} . ': ' . $wh->{error}->{details};
}

wiki_upload("Sandbox","= Sandbox=\n Foobar! Gazonk!\n");

Boom!

Tuesday, November 16, 2010

Face-mail

Years ago I had a T-shirt with the text "The Internet is full, go away!" written on it. Apparently a paraphrase over signs found between California and Oregon - some Oregonians had gotten their fill of escaping Californians.

And then of course AOL connected and so did everyone else and their grannies.

Google, Microsoft, Yahoo and some others have since wanted to dominate the Internet. It sort of looks like Facebook might, 'cause both the kids the grannies are there too and they're probably checking facebook more often than their email. And now facebook might have some kind killer email cum messaging functionality. Which may concentrate even more of the Internet's communications on Facebook.

I'm wondering if we'll see an inversion of what happened when AOL came to the Internet, a outflow from closed services onto the big open net. Will we now see the Internet convening on Facebook and disappearing into a closed platform?

Wednesday, October 27, 2010

Profiling is fun - or - getting munin-cgi-graph fast

I have worked with operations and infrastructure in and around the field of Linux for ages, and in such I job I've not gotten to program a lot. But some. Todays post may be trivial for those that program more than me, but I got a rush out of it.

I've been using some spare time to help out with the munin 2.0 alpha. So I have been profiling it to optimize performance. The original objective was to find out what munin-cgi-graph was using more than a second for when making a single graph with rrdgraph. Especially since it appeared to take significantly under a second to run the rrdgraph command.

I did the work on a site with about 150 hosts and some thousands of "services" (fewer actual plugins since there were quite a few multigraph plugins). This makes a lot of stuff heavier, the nested $config hash with all configuration in (250K lines) takes 10-20 seconds to read from a file on disk.

There are many things in Munin that can be optimized but for munin-cgi-graph the very worst single thing was unneeded use of regular expressions. Regular expressions are very very handy, but in many many cases not needed. String and sub-string matching is a lot faster than regular expressions - but more tedious to type. In the case of munin_find_field regular expressions were definitely not needed in any way. All calls to the procedure showed simple full string matches. I found a few other places with un-needed regular expressions and made some simple replacements like this:

-            next if $key =~ /^#%#/;
+ next if substr($key,0,3) eq '#%#';

This went a very long way.

But the main thing I realized was that munin_find_field was being used way too much, and after restructuring how the work given it was looped over it all went a lot faster. In the end graphing time went from over 1 second to under 0.1 second pr. single graph. That was smooth enough to look at loading in firefox that any remaining regrets I may have had about removing munin-graph from trunk some weeks ago evaporated. It's fast enough.

We're looking to put the same optimizations in Munin 1.4, tonite a guy here is going to profile a full run of munin-graph to see how it works out and what new sore thumbs might stick out. The full run he's doing normally takes 40 minutes(!) without profiling. -- Yes, I know, 40 minutes is too long, and this is why we're switching to cgi graphing. We just have to make sure the cgi graphing is fast!

For profiling I can recommend Devel::NYTprof. It was very easy to use and made it easy to see our hotspot(s).

Friday, October 8, 2010

Back up your phone - no windows needed

Clifford Stoll of " The Cuckoo's Egg" semi-fame once wrote in the rather more forgettable "Silicon Snake Oil" that no-one ever got a email that was worth keeping. I will take this as a sign of his Californian superficiality rather than a genuine insight (I know, he seems to be from Buffalo, New York, but he's an expat living in .ca.us now), as I have several emails I want to keep. And I also have MMSes and SMSes I want to keep, mostly from my wife. They are snapshots of our family life, things she wanted to show me and tell me while I was otherwise engaged. Very real and personal things that are worth keeping.

Clifford also said that e-commerce was un-viable. See, not good with the crystal ball was he.

So after using mobile phones for more than 10 years I finally get around to making backups.

I have a Nokia E52 phone. This runs Nokias Symbian, 3rd or 5th edition or some such. I recently asked my colleagues, free software users one and all, how to back up this phone. gnokii I was told. And I also found gammu/wammu which is based on the same phone interface library for Nokia Phones.

Unfortunately Nokia Symbian phones from the last 3 (? or 5 or 7 or so) years have a very limited AT command set and all in all the AT commands they support are not suitable for SMS retrieval. gnokii and friends work around this by providing a interface applet for the phone called "gnapplet". Unfortunately that does not support recent Nokia Symbian phones. So neither gnokii nor [gw]ammu works. Of the lot wammu seemed the friendliest though. If only my phone was old enough!

I booted up my tame Windows install in VMWare player and installed Nokias Ovi Suite. And it synced the phones messages. The SMS messages. No MMS, i.e. no pictures of the kids and wife. WTF? And also no way to export to any kind of file format. Not exactly safe keeping. WTF²?

Nokia has a app store too now, so I could timmidly ask "Is there an app for that?".

Yes. Just one. SyncLion. It purports to syncronize SMS, MMS and contacts with any IMAP server. First I had to fiddle with my Dovecot configuration because SyncLion was playing with namespaces and Dovecot was having none of it which resulted in "Unknown namespace" messages in SyncLion.

namespace private { separator = . prefix = Mobile. list = yes location = maildir:~/Maildir/mobile }

(I learnt what SyncLion was saying by enabeling raw logging in Dovecot). After that it started copying everything to our IMAP server. Thunderbird showed the new namespace, and I could direct it to subscribe to the three folders in the Mobile namespace: Contacts, Messages and Notes. Roundcube our webmail does not seem to see the Mobile namespace in any way and is unable to show anything from it.

Each of the three folders were full of what I expected. Including MMSes! Way! So now I can copy the messages I want to keep to some safe email folder, and keep them there forever, or until both the disk and the backup fails.

There are some issues I noticed though:

  • The Sync part of SyncLion is not complete. If I delete a message on the phone it does not disappear from the IMAP folder.
  • Nor does deleting a message in the IMAP folder remove it from the phone.
  • The emails that are composed could very well embed the image (and video) files in the emails, but the image (and video) files are attachments that I need to double-click to view.
  • Contrary-wise the contacts are easily human readable, but there is no vCard attached so I can't double-click and save.
BUT, the SMSes and MMSes are now possible to keep safe.

Friday, September 24, 2010

Uptime

A while ago I installed new DNS-servers for a client. They're sitting way behind two layers of firewalls and the environment is therefore quiet and friendly. We were just contacted to set up some new name servers - since it had been a while since the last iteration.

# uptime
12:47:42 up 1633 days, 13:15, 4 users, load average: 0.03, 0.03, 0.00
Indeed, it has been a while.

Thursday, September 2, 2010

It's just too easy

I'm often among early adopters. Not so with flat screen TVs. Just got a one of the way flat Samsung "LED" ones. My wife wanted white frame, I wanted DLNA support. Digital Living Network Alliance has made a UPnP based standard for connecting multimedia appliances to network resources. Like my (*cough* our) new TV. All our network resources run Linux though, so windows software is no good for us. One of my colleagues figured this out for us, there are multiple DLNA servers for Linux. One of the most promising was TVMOBiLi. And they make Debian and Fedora packages. To get it working with Samsungs you, right now, need a release candidate (I installed 1.4.0.2196).

Start it. Point browser to http://localhost:30888/ where there is a nice web GUI. Click the "SETTINGS" link, and set up all the different directories we use for movies, mp3s and pictures. Fire up media player app on TV, and it already knows about the tvmobili server. And browsing the My Content folder on the TV showed all our files. And Samsung firmwares supports quite a lot of video and sound codecs.

Sometimes being slow to adopt new stuff saves a lot of work. And deprived me of a learning adventure. I still don't know much about DLNA.

Happy though.

Sunday, April 4, 2010

Remote upgrade of ubuntu

I had a Ubuntu 8.04.4 (LTS) machine languishing in the basement serving various things. To make it more useful I wanted to upgrade it.

It has no CD-ROM and besides it's in the basement so I wanted to upgrade it over network. I had the hardest time with update-manager, it didn't want to do the job over SSH.

Google to the rescue: http://www.kbrandt.com/2008/11/upgrade-ubuntu-remotely.html - So I got the right iso image and away I went.

Unfortunately there were some circular dependency problems going directly from 8.04.4 to 9.10 so unfortunately I had to edit sources.list and dist-upgrade my way through 8.10 to 9.04 and then use the described method to upgrade to 9.10. In any case it was quite simple, I seem to be modifying fewer and fewer config files as time pass. It might be wisdom, or just old age.

Monday, March 15, 2010

DVDs w/o the lead-in materials

Lately a comparison of how much bother it is to watch a bought DVD vs. a priated movie has been circulating (I can't find it now...). The point being that on a DVD you have to watch the insulting "you wouldn't steal a bag would you?" thing that MPAA gets on the DVDs, , trailers for up and comming movies, the lead in before the menu, and then finally the menu. Where you can start the movie. Right after the US-federal or local copyright warning and the distributor lead in and so on. Some of these you can skip, some of them you can skip if you press the right button (i.e. Fast Forward works on some, Skip on some, and MENU on some, depending on how conscientious the DVD-issuer-staffer that did the job was).

On my DVD player at least, this is a case of "you're doing it wrong".

1. Insert DVD
2. Wait for it to spin up and start
3. Press "STOP". Wait for it to stop.
4. Press 1.

Most DVDs have the movie on track 1. I think I've seen one or two where the movie was on some other track (track 2 and 5 comes to mind), but they are few and far between.

Friday, March 5, 2010

Our most precious memories

I guess I'm in the first generation of parents that use all digital media to help us remember how it was back when the children were small.

Now we all use digital cameras and camera video recorders for this. And the most thorough of us burn it onto CDs or DVDs just to make sure we can view it in our old age. (The rest of us wait for the disk to crash, and then it'll all be lost.)

Apart from the likeliness that we'll have something that can play a DVD in 20-40 years (seems quite unlikely) it turns out a typical CD-R has a life expectancy of less than 10 years. Even the CDs I bought along with my first CD player are reaching the end of their life nowish (good thing I have backed up all my CDs then :-)

So the movies and pictures of our children growing up and all the great occasions we'd like to be able to recall and treasure for years to come are in double trouble. Fortunately we can buy high quality media to store the bits on:


I'm not sure what to do about the problem of keeping a player around for those disks for 40 years. A player that can talk to our TV in 40 years time may be even harder.

Thursday, February 25, 2010

Musikk skal bygges ut av glede

I just wanted to say that Squeezebox Duet (the black things on the right here) rocks. And the server part is written in perl and right on the box it says that Linux (and BSD and Solaris) is supported.

I ripped and put away all my CDs some years ago, and played my music from the HTPC, which was OK I guess. But in the new house I've yet to set up the HTPC (with two small children and a new house, I wonder if I'll ever get the time for it), and besides it's kind of onerous to have to fire up the TV or a latptop to play music. Having a small black box hiding in the shelves and a remote controll with a color LCD and a mp3-player style interface is so much easier.

The reason for the "duet" name had escaped me until I looked at the box and remote control yesterday evening: There is a DAC and headphone/line out jack in the remote, so both the remote and the receiver can be connected to a (or rather two different) amplifier(s). I can connect it to the line-in on the kitchen Tivoli Model Two. I should remark here that much can be said about Tivoli Audio, but they have high WAF (Wife Acceptance Factor) and the sound pleases me.

Cool trivia: The remote runs Linux and you can enable SSH on it. A colleague wrote a Munin plugin for it to graph all sorts of meaningless stuff :-) Let's see if I can get him to publish it.

My hat off to the good people at Squeeze for a excellent and reasonably priced product.

Thursday, January 28, 2010

EU-kontroll

Det begynner å synke inn at biltilsynet eller hvem det nå var har sluttet å sende ut varsler om at biler må EU-kontroleres. "Dette er noe folk selv må ta ansvar for" eller noe slikt. Naturligvis har det resultert i svært store mengder avskiltnings-varsler. Kan ikke tro at arbeidsmengden hos den relevante myndigheten blir mindre på denne måten.

Nuvel. Tankene går i rettning av å lage et hack for å minne en selv på slikt - som ellers garantert går i familiens glemmebok. Men det slipper jeg. NAF har laget en web-tjeneste hvor du kan melde på alle bilene dine og få epost eller sms når tiden for EU-kontroll kommer. Gratis.

Firefox plugins: Web-spyware

There are scores of way to monitor web users - to help target ads and do various accounting. Since I consider that the social contract between users and publishers on the web was broken way back when popup ads, animated ads, and noisy ads were introduced I feel totally justified in suppressing ads. So I've been running Adblock and Adblock-plus for ages. I've been using the adblockers to block various "spyware" scripts too, the ones normally found on Norwegian sites at least. The other day I found an other plugin: Ghostery. It tells me about the spyware it recognizes on every site and gives me the option of blocking it. Thanks! :-)

Tuesday, January 26, 2010

Podcasts

I have never listened to podcasts while "on the go". I have been listening to podcasts on and off on my laptop, my audio-playback-app (amarok) subscribes to a number of shows I like. But to plug in my mp3 player and transfer the stuff has seemed like too much of a job.

Since I moved out of Oslo (to Asker) and started taking the train I've been spending 2-3 hours commuting. I've been reading a good deal. But late in the fall I got a new phone, a Nokia E52. It has a number of features to redeem it: reasonably sized, thin, WiFi, charge-via-USB, and a standard mini-jack for the handsfree - which works well with regular headsets. And a RSS-client in the web-browser. That can download podcasts if manipulated correctly.

Turns out that mp3 players need WiFi - because it was the final hurdle to get me listening podcasts on the train. If asked before this I would have denied that and called it frivolous. But with no need to plug in the phone to get podcasts - it's passed my threshold of "easy enough". Just start the browser at home or at work and click on the podcasts I want to listen to. Now the train rides are much more enjoyable. Quality of life++.

Thursday, January 7, 2010

Munin presantasjon

På tirsdag 12 skal jeg snakke om Munin på NUUG medlemsmøte i Oslo. De har videooverføring til Stavanger. En trenger ikke å være medlem for å komme inn; møtet er åpent og gratis for alle.

Disposisjonen min ser for tiden slik ut:

  • Hva Munin er og hvorfor det er så morsomt
  • Nye features i 1.4
  • Konfigurasjon
  • Lage en enkel plugin
  • Lage en enkel multigraph plugin
  • Spørsmål og svar
Full annonsering på: http://www.nuug.no/aktiviteter/20100112-munin/