Tuesday, October 28, 2008

Skinny Times Are A'Coming

I am sitting at the dining table, pecking away at this post while wolfing down some instant noodles. I just got back from company enkai where I really understood for the first time how grave the situation is at work. What with the grim economic circumstances unfolding worldwide, even here in Sapporo, skinny times are a'coming.

It is time to bear down, re-evaluate your skills and knowledge, tap your social networks, and plan for the worst. With the sinking of Lehman Brothers just the tip of the iceberg, soaring fuel and food costs and the coming winter mean that I will have to work a bit harder and a lot more smarter to protect my family.

S0, just how do I know that my company is in for some skinny times and stormy weather?

Tonight's party menu, that's how. Or rather, the lack thereof.

Normally, an enkai for welcoming new employees should be a fairly filling affair. It is fall, so that should mean a nice, hearty nabe hot-pot, with extra dishes of skewered meats or sashimi, salads, and the like. But tonight? There was only:

  • a small nabe hot-pot, 1 serving per person, with cabbage, mushrooms, and ground chicken dumplings
  • 1 yakitori skewer
  • 1 tsukune ground chicken skewer (very tiny)
  • a sparse green salad
  • tiny, tiny fried chicken nugget-lets (I mean, tiny!)
  • vinegared sushi rice (w/out much in the way of anything added to it)
  • a dry piece of cake for dessert
Granted, it was an all-you-can-drink-for-2-hours sort of deal, but after a hard day coding, I was looking forward to sitting down to a real meal. Seeing as how such a meal could not be afforded given the financial situation, I fear that expanding the staff at this point is probably folly. And any grandiose ideas for transforming the company from a software sweatshop into a real service-oriented enterprise should have been put into execution last year.

Already, there is blood in the water for IT in Sapporo. A local Sapporo IT company called Den-nou (電脳, but their website is already down) just sank last week, and I reckon that it will not be the last. As far as I can tell, practically all IT in Sapporo have the same weak business model (bid as low as possible on jobs in Tokyo or Osaka, agree to unrealistic schedules, and work the crap outta your staff; lather, rinse, repeat). With practically nothing to discern one company from another, I expect that the same fate that befell Den-nou is what await many other Sapporo IT shops, including the one in which I am stuck.

But I am a chameleon when it comes to careers, a trapeze-swingin' kind of guy. I think it may be time to reinvent myself. Of course, I love coding way too much to give up that easy. But extraordinary times call for extraordinary measures.

Be prepared. Skinny times are a'coming.

Saturday, September 27, 2008

Weebl and Bob - Destroy (archived on YouTube)




Hilarious parody of Ladytron's Destroy video. Apparently, the creators of Weebl and Bob got flak for making this, and British laws are not permissive enough to allow this. Thank god for YouTube.

Link to thread discussing the cease and desist order here...

It's too bad that someone doesn't have a sense of humor.

Thursday, August 28, 2008

Java Implementation Loader Error in Deploying OpenOffice Add-On Project

My interest has been piqued lately: I am curious about developing applications in the OpenOffice framework, using the OpenOffice SDK.

Anyhow, I had some trouble getting the Add-On project tutorial going. Nothing serious, but I just couldn't deploy the add-on for "Could not create Java implementation loader" errors. Yeah, I didn't have a clue as to what that meant.

But with a bit of googling, I found this thread, which pointed me back to this page on the OpenOffice wiki. The solution is right there, in 2. Possible Work-Arounds. I quote:

This usually happens when one wants to debug Java in the office process and therefore has provided the debug options for Java in the options dialog. The Java service of the extension will be registered in a separate process (uno.exe) which also needs to start Java. This Java uses the same settings from the options dialog.
...
If there is also a Java running in the office process, then the creation of the virtual machine in the second process will fail. The reason is, that the latter receives the same debug settings. It appears that two virtual machines, although in separate processes, cannot use the same port for debugging. If one changes the port number before adding the extension (and after the Java has been created in the office process), then the extension will be installed properly.


Urgh. I installed OpenOffice with its own JRE, but I am using Java SE 6 Update 10 RC as the default environment in my NetBeans IDE (I am also dabbling in JavaFX!). I surmise, however, that when I use the Project View -> Project Node -> Context Menu -> Create OXT | Deploy Office Extension in NetBeans to launch the project deployer, it accordingly uses the environment settings of the NetBeans JRE/JDK the project is using, and not the JRE that comes packaged with OpenOffice.

So, to make a long story short... In WinXP, locate your user's C:\Documents and Settings\%user%\Application Data\OpenOffice.org2, and just delete it. Re-launching/re-deploying the OpenOffice Add-On project will just work. You'll have to change your Windows Explorer settings to display all hidden directories and files.

Mathematicians (nah, I'm not really a mathematician)



Which reminds me of the following joke:
An engineer, a physicist and a mathematician were all staying in a hotel, when each of their rooms individually caught fire. The engineer did some basic math, flooded the floor and said, "It is out." The physicist did more complicated math, used just precisely the amount of water needed to put out the fire and said, "It is out." The mathematician did a lot of complicated math, cried out, "I HAVE SOLVED IT!" and went back to bed.

... think I saw the above joke on /.

Wednesday, August 27, 2008

RSA Encryption: OpenSSL to Ruby to C# (and back to Ruby)

Continuing on the topic of encryption, it turns out that the work project I mentioned in my last post will require transmission of encrypted data from C# to Ruby. As this encrypted data must only be decrypt-able by the Ruby-side logic, asymmetric encryption fits the bill well. Here are some notes on working with RSA public key encryption with OpenSSL to generate the RSA key-pair; Ruby/OpenSSL to extract the public key parameters to feed the C# RSACryptoServiceProvider; C# to use the public key to encrypt an arbitrary array of bytes; and finally back to Ruby/OpenSSL to decrypt the enciphered data with the private key.

Now, I will consider that trust between the C# logic and the Ruby/OpenSSL logic is implied, so there is no need for use of X.509 certificates or what-not to distribute the public key to the C#-side. I will instead use the RSACryptoServiceProvider's FromXmlString method to initialize the provider with the key modulus and exponent. Practically speaking, I would have to obtain the key modulus and exponent (most likely using Ruby/OpenSSL), and then serialize that in an XML form that the C# class RSACryptoServiceProvider understands.

RSA Private Key Generation with OpenSSL
OK, let's start off by first using OpenSSL to generate a 512-bit RSA private key:


... with OpenSSL command-line ...

... generate RSA private key (default of 512-bits)
shino# openssl genrsa -out rsa.priv

... confirm key parameters
shino# openssl rsa -in [rsa.priv] -text -noout

Simple enough, no? And once again, yeah, yeah, yeah: a 512-bit RSA key cannot do much in the way of protecting secrets. Cracked way back in 1999. But let's just agree to go with 512-bits for the sake of this post. BTW, don't forget that you will have to take into account any steps needed to protect this RSA private key file. I leave that exercise to you, dear reader.

Obtain Key Modulus & Exponent Parameters with Ruby/OpenSSL
Next, we use Ruby/OpenSSL to read in our newly-created RSA private key and obtain Base64-encoded string values for key modulus and exponent for the RSACryptoServiceProvider's XML string (we will use those values to calculate the public key in C#-land shortly):

# with irb/ruby/openssl

# use ruby/openssl to load private key
irb(main):001:0> require 'openssl'
=> true
irb(main):002:0> require 'base64'
=> true
irb(main):003:0> privkey = OpenSSL::PKey::RSA.new(File.read('rsa.priv'))

# extract key exponent and Base64-encode
irb(main):004:0> exp = []
=> []
irb(main):005:0> pubkey = privkey.public_key
irb(main):006:0> pubkey.e
=> 65537
irb(main):007:0> (pubkey.e.num_bytes-1).downto(0) {|i| exp << ((pubkey.e >> (i*8)).to_i & 0xFF) }
=> 2
irb(main):008:0> exp
=> [1, 0, 1]
irb(main):009:0> tmp = []
=> []
irb(main):010:0> tmp[0] = exp.map {|b| sprintf("%02x" % b) }.join
=> "010001"
irb(main):011:0> Base64.encode64(tmp.pack('H*')).split("\n").join
=> "AQAB"

# extract key modulus and Base64-encode
irb(main):012:0> pubkey.n
=> 10859685957917547658301583853465231245284761442925449111820039202653087969032
48227098143938496267490334944907126548361037059959257154028677097618826672949
irb(main):013:0> (pubkey.n.num_bytes-1).downto(0) { |i| mod << ((pubkey.n >> (i*8)).to_i & 0xFF) }
=> 63
irb(main):014:0> mod
=> [207, 89, 10, 12, 14, 23, 85, 80, 200, 210, 18, 182, 229, 231, 180, 219, 103,
58, 118, 47, 162, 38, 14, 74, 94, 114, 192, 69, 174, 176, 175, 140, 177,
230, 202, 21, 102, 169, 66, 199, 161, 128, 61, 223, 167, 210, 168, 145, 51,
164, 94, 120, 73, 250, 100, 8, 7, 56, 185, 23, 28, 157, 59, 53]
irb(main):015:0> tmp = []
=> []
irb(main):016:0> tmp[0] = mod.map {|b| sprintf("%02x" % b) }.join
=>"cf590a0c0e175550c8d212b6e5e7b4db673a762fa2260e4a5e72c045aeb0af8cb1e6ca1566a942c7a1803ddfa7d2a89133a45e7849fa64080738b9171c9d3b35"
irb(main):017:0> Base64.encode64(tmp.pack('H*')).split("\n").join
=> "z1kKDA4XVVDI0hK25ee022c6di+iJg5KXnLARa6wr4yx5soVZqlCx6GAPd+n0qiRM6ReeEn6ZAgHOLkXHJ07NQ=="

A good point to bear in mind here is that for Ruby/OpenSSL, the key parameters are instances of OpenSSL::BN, which represent big numbers for OpenSSL for multiprecision integer arithmetics. OpenSSL::BN.num_bytes tells us how many bytes are used to represent a key parameter, and then in order to transform the big number into an array of bytes, we can pass a block that uses a bit mask to obtain the significant bits for each byte of the big number. Then, we transform the byte values in this array into 2-char hexadecimal strings, and concatenate into one, long string of hexadecimal chars. Stuff that into a temporary array, call pack('H*') on the array, remove any line-break chars, and then we Base64-encode the values for modulus and exponent.

Encryption Using RSA Public Key with C#
OK, so we have our 512-bit RSA private key, and we have extracted the key modulus and exponent values as Base64-encoded strings, which we can now pass to C#'s RSACryptoServiceProvider in order to obtain the public key:

using System.Security.Cryptography;
...

RSACryptoServiceProvider rsa = new RSACryptoServiceProvider();
rsa.FromXmlString("
<RSAKeyValue>
<Modulus>
z1kKDA4XVVDI0hK25ee022c6di+iJg5KXnLARa6wr4yx5soVZqlCx6GAPd+n0qiRM6ReeEn6ZAgHOLkXHJ07NQ==
</Modulus>
<Exponent>
AQAB
</Exponent>
</RSAKeyValue>");

Note that the XML string really shouldn't have any line breaks, I only broke it up to for viewing's sake.

The next step is to encrypt some arbitrary data, which we take to be an array of bytes. Be sure to Base64-encode the result, since we will pass that string over to the Ruby/OpenSSL side.

... in C# ...

byte[] cipherbytes = rsa.Encrypt(plainbytes, false);
String ciphertext = Convert.ToBase64String(cipherbytes);

Gotta love C#, the APIs are for the most part crystal clear and user-friendly.

Decrypt Ciphertext with RSA Private Key with Ruby/OpenSSL
Alright, now we just need to bring it all home, and decrypt that ciphertext string with the RSA private key, using Ruby/OpenSSL:

# with irb/ruby/openssl again

# read in private key from file
irb(main):001:0> require 'openssl'
=> true
irb(main):002:0> require 'base64'
=> true
irb(main):003:0> privkey = OpenSSL::PKey::RSA.new(File.read('rsa.priv'))

# Base64 decode the transmitted ciphertext first
irb(main):004:0> ciphertxt = Base64.decode64("RALJTqgPNRcmvZ4E09y4NODl+w/vJPRslmxi8xB9JPSrpDQ
bKjOkoeivTOqL9gT4WBpRpIlqut3jflkcUy/5Qg==")
=> "D\002\311N\250\0175\027&\275\236\004\323\334\2704\340\345\373\017\357$\364l\
226lb\363\020}$\364\253\2444\e*3\244\241\350\257L\352\213\366\004\370X\032Q\244\
211j\272\335\343~Y\034S/\371B"

# decrypt w/ private key
irb(main):005:0> plaintext = privkey.private_decrypt(ciphertxt)
=> "Ru'\035\314`\f\352"

# reconstitute as plain byte array
irb(main):006:0> plaintext.unpack('H*')[0].scan(/../).map{|b|b.hex}
=> #... results in the original array of bytes which we encrypted in C# land!


Conclusion
That wasn't so bad now, was it? We created an RSA private key with OpenSSL, used the Ruby/OpenSSL binding to manipulate the key to extract the key modulus and exponent (the public key material), and fed that into some C# code. This allowed us to encrypt an arbitrary array of bytes with the RSA public key in C#-land, and then we wrapped it up by decrypting the ciphertext with the corresponding RSA private key with Ruby/OpenSSL.

Like I mentioned in my last post, it is the lack of clear documentation for the Ruby/OpenSSL API that can trip you up. However, hackable nature of Ruby made it fairly painless to explore and experiment, and to come up with the above example of RSA asymmetric encryption using OpenSSL, Ruby and C#.

Tuesday, August 26, 2008

DES encryption: Java to OpenSSL to Ruby

Encryption has always been an interest of mine, but I really don't have mad skillz. I must admit that I ran into a bit of difficulty in trying to figure out how to properly use the OpenSSL API for Ruby, but what do you expect? The documentation is frustratingly poor.

In parallel with an ongoing project at work, I took some notes on using DES encryption with Java to generate the proper bits for a DES key; OpenSSL to actually use this key to encrypt some file; and then use Ruby/OpenSSL to decrypt. I know, I know: DES has been cracked for some time now. But this an exercise just to see if I could bring together what little acumen I possess in Java, Ruby, and OpenSSL.

Here's how I did it for DES using the Cipher Block Chain operation mode. Note that the DES key and initialization vector are being passed from Java to OpenSSL to Ruby as a string of hexadecimal char, but in reality we are dealing with bytes.

First, we create a valid DES key and set of initialization vector bytes. From the Federal Information Processing Standards Publication 46-2, the specification for the Data Encryption Standard (DES):

A key consists of 64 binary digits ("O"s or "1"s) of which 56 bits are randomly generated and used directly by the algorithm. The other 8 bits, which are not used by the algorithm, are used for error detection. The 8 error detecting bits are set to make the parity of each 8-bit byte of the key odd, i.e., there is an odd number of "1"s in each 8-bit byte.


We'll use Java and the javax.crypto.KeyGenerator factory to create a valid DES key and initialization vector:

// in Java

import javax.crypto.Cipher;
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import javax.crypto.spec.IvParameterSpec;

...

public static void main(String[] args) throws Exception {
// create a DES key
KeyGenerator kg = KeyGenerator.getInstance("DES");
SecretKey key = kg.generateKey();
byte[] keyBytes = key.getEncoded();
StringBuffer sbuf = new StringBuffer();
for (byte b : keyBytes) {
sbuf.append(String.format("%02x", (b & 0xFF)));
}
System.out.println("DES key: " + sbuf);

// create initialization parameter bytes
Cipher nonceCipher = Cipher.getInstance("DES/CBC/PKCS5padding");
nonceCipher.init(Cipher.ENCRYPT_MODE, key, new IvParameterSpec(new byte[8]));
IvParameterSpec iv = new IvParameterSpec(nonceCipher.doFinal(new byte[8]), 0, 8);
byte[] ivBytes = iv.getIV();
sbuf = new StringBuffer();
for (byte b : ivBytes) {
sbuf.append(String.format("%02x", (b & 0xFF)));
}
System.out.println("IV: " + sbuf);
}


The above Java code will output a string of 8 bytes in hexadecimal for the DES key, and also a string of 8 bytes in hexadecimal for the initialization vector to be used in CBC. We next use this as input for the openssl enc program:

... with OpenSSL command-line ...

shino# openssl des-cbc -e -in plaintext.txt -out encrypted.des-cbc \
-K [java-generated DES key string] -iv [java-generated IV string]


The above OpenSSL command will encrypt the file specified with the -in option, using DES-CBC with the given -K DES key and -iv initialization vector. The values for -K and -iv need to be a string of 8 bytes given in hex. Encrypting the plaintext file plaintext.txt will then result in the output encrypted.des-cbc. By the way, this OpenSSL command is using PKCS5 padding as default.

To decrypt with ruby/OpenSSL was a bit tricky. Like I mentioned earlier, the documentation is very, very poor. Assuming that the DES-CBC encrypted file encrypted.des-cbc exists in our current working directory, we can fire up irb to do the decryption and bring it all home:

# with irb/ruby/openssl

# given key = "838519bcc21fb0a1"
# given iv = "33bc52494428f4b6"
# given plaintext msg = "hello, encryption!\n"

irb(main):001:0> require 'openssl'
=> true
irb(main):002:0> f = File.open('encrypted.des-cbc', 'rb')
=> #<file:encrypted.des-cbc>
irb(main):003:0> data = f.readline
=> ...
irb(main):004:0> f.close
=> nil
irb(main):005:0> decrypt = OpenSSL::Cipher::Cipher.new('des-cbc')
=> #<openssl::cipher::cipher:0x4417d60>
irb(main):006:0> decrypt.decrypt
=> #<openssl::cipher::cipher:0x4417d60>
irb(main):007:0> decrypt.key = "838519bcc21fb0a1".scan(/../).map{|b|b.hex}.pack('c*')
=> "\203\205\031\274\302\037\260\241"
irb(main):008:0> decrypt.iv = "33bc52494428f4b6".scan(/../).map{|b|b.hex}.pack('c*')
=> "3\274RID(\364\266"
irb(main):009:0> decrypt.update(data) + decrypt.final
=> "hello, encryption!\n"


Here, we use irb to open the encrypted file and read in the bytes. Then, we use the ruby/OpenSSL API to create a DES-CBC cipher. The call to Cipher.decrypt sets the cipher to work in decryption mode. We next specify the DES key and IV initialization vector, as well as the string of encrypted chars as binary strings.

# ruby snippet
"838519bcc21fb0a1".scan(/../).map{|b|b.hex}.pack('c*')

In setting the key and iv, we provide a 16-char long string of hexadecimal that represents 8-byte values. The above invocation uses String.scan(/../) to break the hex string into a list of 2-char long chunks. Enumerable.map{ |b| b.hex } then maps each 2-char hex chunk back into its numerical (byte) equivalent. Array.pack('c*') lastly transforms the entire list into a single binary sequence. The documentation for Cipher.key= and Cipher.iv= is none too clear that both APIs expect a binary string. I spent quite a bit of time scratching my head as I was mistakenly passing in the plain-vanilla hexadecimal strings for the key and iv values.

Anyhoo, as you have seen, it is not all that hard to do a round-trip DES-CBC encryption using Java to generate the key and iv; OpenSSL to actually encrypt the plaintext; and ruby/OpenSSL to decrypt.

BTW, this doc page in Japanese was a bit more helpful in pointing me in the right direction.

Monday, August 25, 2008

The Hokkaido Bank predicts 57K JPY increase per household for winter expenses

A article in yesterday's Hokkaido Shimbun Press reports that the Hokkaido Bank expect an increase of 57000 JPY per household of expenditures on gasoline, kerosene, and other expenses related to winter. This figure is based on a recent Hokkaido Bank study that predicts a gross 134.6 billion JPY increase in winter expenditures for the coming 2008 season. To make things worse, this study does not take into account effects of the expected increase for electricity rates.

This seemingly downward spiral is certainly straining things for all people living in Hokkaido. Food, fuel and utilities like electricity all have been rising steadily. Personally, this has been weighing heavily on my mind as of late, as I am contemplating another career jump in order to keep up with all of the price-increases and keep my family afloat.

I never really wanted to believe it, but the IT business here Hokkaido is starting to look very ghetto to me. There are a handful of good people, but most companies are pursuing a very conservative (read that as "weak" if it cannot let their employees stay ahead of inflation!) business plans for systems integration, made-to-order development, or call center and support.

It may have been a mistake to tie myself directly to the local economy, so I am actively looking for opportunities and challenges that are more global and dynamic. Lord knows, there are a tons of such jobs in Tokyo...

Thursday, August 7, 2008

Dan Kaminsky Finally Outlines DNS Vulnerability

Dan Kaminsky finally outlined the DNS vulnerability that was reported about 4 weeks ago (article on The Register). It appears to be a DNS forgery variant of DNS cache poisoning. If an attacker of a DNS server can quickly and correctly guess a 16-bit transaction ID, then the attacker will be able to replace DNS entries with their own spoofed ones. There has been one confirmed attack so far, where the attacker caused AT&T subscribers to be re-routed to fake Google pages.

For the curious, you might want to see how safe your ISP's DNS servers are. Try the "Check My DNS" button on Dan Kaminsky's site; or the "Test My DNS" link here. The key here is to see the amount of randomness of transaction IDs and query source ports in your ISP's DNS servers.

Hacking FasTrak

I've never used the FasTrak debit system for the tollways in California when I lived there, but The Register recently reports that an exploit has been found. I can't believe that FasTrak apparently transmits its unique ID number in the clear.

Not that I use it here, but I wonder what sort of vulnerabilities and countermeasures exist for the Electronic Toll Collection (ETC) system in Japan. Or おサイフケータイ services for the mobile devices used in Japan.

Saturday, August 2, 2008

Installing OpenSolaris on VirtualBox

Finally got around to installing Sun's OpenSolaris on VirtualBox. It turned out to be easier than I expected. Just grab the latest release (2008.05 at the time of this writing), and follow thru the install just as you would any other guest OS on VirtualBox. You can find details for installing OpenSolaris on VirtualBox here and here, but believe me, it is surprisingly sweet and smooth.

The only hitch I had was getting the network interface going. Apparently, OpenSolaris ships with the Network AutoMagic, which sadly is not all that automagical when installing on VirtualBox. But the solution is quite simple. Just keep on reading...

After what should be a successful install, confirm that nwam is running:


# svcs nwam


It will be enabled by default. We don't need this, we need to configure the Intel PRO/1000 MT Desktop (NAT) e1000g0 network adapter interface, and activate it. Let's turn off nwam, and set things up:

# svcadm disable physical:nwam
# svcadm enable physical:default


Next, we configure and activate the e1000g0 connection. In the OpenSolaris' guest window menu, select System > Administration > Network. Select the e1000g0 connection, and click on the Properties button. In the Interface Properties window, check the boxes for Enable this connection and Activate on boot. Select DHCP for Configuration, and click OK. Lastly, click on the Activate button to bring e1000g0 up. You can reboot to complete the process, and hey, whoa!



Network love!

Time to install the JDK, NetBeans, and other tools...

Monday, July 28, 2008

Secure Gmail

Glad to see that Google is taking steps to protect gmail from sidejacking...

I got my gmail settings to "Always use https", how about you?

Wednesday, July 16, 2008

Schneier on Security and the iPhone

Bruce Schneier has recently written a very good article on the issue of "security" and the new iPhone. I highly suggest you click on that link and give it at least a brief read.

Technology these days seems to be making great leaps and surges, and I admit that I am mightily impressed by the sheer computing power of the iPhone. The features it offers just puts to shame the old Ericsson plastic brick I first had back in '96 when Digital Tu-Ka Hokkaido (I believe they are J-Phone now) first opened their Ericsson TDMA network. But for all the glitz and glam, there is no way I want someone to be able to track me to within a few meters using GPS. As it is, someone could use triangulation and locate me within a city block or two right now with my DoCoMo FOMA mobile.

Privacy is something that seems to be taken for granted, more and more. It is given away when consumers blindly accept technology that appears to make their lives more fun. Privacy is a privilege that comes from having freedom, and freedom is something you have to work for on an everyday basis. Shouldn't we be more vigilant?

Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety.
-- Benjamin Franklin, Historical Review of Pennsylvania, 1759

ISS: International Space Ship???

Interesting article, again linked from ./. It's about giving the International Space Station a new, real purpose by sending it to the moon.

I have more than a simple curiosity in this, since I am developing a (small) video-processing application for use on the Japan Kibo module on the ISS. One upcoming feature is to stream video data back to earth via a relay satellite, which poses the interesting problem of having only 45 minutes of satellite line-of-sight out of every 90. Transmission interruptions, interference, missing bits due to atmospheric phenomenon... wow, better have good error-correction built in!

Can you imagine what it would be like if the ISS were not parked in low earth orbit, but actually spinning around the moon? That would present some very interesting challenges, I'm sure.

But first, there are of course the logistics of moving 1,040,000 pounds of ISS the distance of roughly 250,000 miles from the earth to the moon. Just strap on some ion engines and slap together a steering module?

Thursday, June 19, 2008

Compare your US salary levels!

This place got slashdotted...

I would love to try out Glassdoor, but it doesn't accept Japan as a valid work location just yet.

Friday, May 30, 2008

Spread Firefox 3: Countdown to Download Day

UPDATE:

Got a nice certificate from Mozilla.org for participating... yeeha...








I might've missed Pi Day this year, but 3.o will do...

Join in on Download Day for Firefox 3. Maybe we can set a new world record!





Download Day - JapaneseDownload Day - English

Wednesday, May 21, 2008

Google Web Toolkit on Google App Engine: Join in a performance test!

There's an interesting experiment using the Google Web Toolkit and Google App Engine posted here. I am very curious to see what sort of performance can be had when the GWT is paired with App Engine, with no explicit, special optimizations.

If you are free and want to join in the fun, just hit http://qwerty.codathlon.com on May 23, 2008, 16:00, Paris-time. That's 23:00 here in Sapporo, Japan.

I am so in.

Monday, May 19, 2008

Japan's IT state of the nation: bleak outlook

Good thread on slashdot regarding outsourcing to China.

Here in Sapporo, the local municipal government has a deal with IT companies based in Shenyang. Having personally been involved in project management capacity with a company in Shenyang recently, as well as having worked in the past with support and development teams based in mainland China, I am rather doubtful of the present Sapporo-Shenyang partnership. More so since there are no more direct flights between Sapporo and Shenyang.

I think that the odds are stacked against you in terms of meeting a project deadline on time and with high-quality, largely due to differences in cultural expectations and the lack of a good telecommunications infrastructure in Shenyang. But even more than that, afaik, there is a sorry lack of experience and understanding about such globalized, international endeavors here in Sapporo. It really is disconcerting to see the hit-or-miss approach being taken by some companies.

I suppose that this may have a lot to do with the dwindling numbers of university graduates with technology degrees. I do not think, however, that a strategy of retaining only project managers and outsourcing the bulk of work will succeed in the end, both for the enterprise and for the local and national economies.

Wednesday, May 14, 2008

RSpec and stubbing ssl_required

Here's a quick rspec stubbing tip for those of you using RSpec and Spec::Rails with a need to stub out the controller behavior for ssl_required. I wrote this tip with mocha in mind, but you can easily apply this to whatever mocking/stubbing framework you may be using.

First, let's take a look at the source for the SslRequirement module, paying attention to the the before_filter bit:


module SslRequirement
def self.included(controller)
controller.extend(ClassMethods)
controller.before_filter(:ensure_proper_protocol)
end

...

private
def ensure_proper_protocol
return true if ssl_allowed?

if ssl_required? && !request.ssl?
redirect_to "https://" + request.host + request.request_uri
flash.keep
return false
elsif request.ssl? && !ssl_required?
redirect_to "http://" + request.host + request.request_uri
flash.keep
return false
end
end
end

You can see that it is the private method ensure_proper_protocol that is being declared as a filter for the controller. Since a controller that declares ssl_required will automatically inherit the methods listed in SslRequirement, you can stub the SSL-checking behavior in mocha-style for your controller's behavior-driven rspec test like so:

controller.stub!(:ensure_proper_protocol).and_return(:true)

Just declare the above stubbing in a setup block, and you are good to go!

Uninitialized Constant CGI::Session::ActiveRecordStore NameError

If you've found this post by keyword search, let me ask you: Are you using the Rails ActiveRecordStore along with the Ruby-GetText-Package ruby gem? And are you seeing an ugly, mysterious error that looks something this when starting Rails or even executing script/console:


[me@myserver]# mongrel_rails start
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with development environment...
/var/www/apps/argent/vendor/rails/actionpack/lib/../../activesupport/lib/active_support/dependencies.rb:478:in
`const_missing': uninitialized constant CGI::Session::ActiveRecordStore (NameError)
from /var/www/apps/argent/vendor/rails/actionpack/lib/action_controller/session_management.rb:24:in `const_get'
from /var/www/apps/argent/vendor/rails/actionpack/lib/action_controller/session_management.rb:24:in `session_store='
from /var/www/apps/argent/config/../vendor/rails/railties/lib/initializer.rb:328:in `send'
from /var/www/apps/argent/config/../vendor/rails/railties/lib/initializer.rb:328:in `initialize_framework_settings'
from /var/www/apps/argent/config/../vendor/rails/railties/lib/initializer.rb:327:in `each'
from /var/www/apps/argent/config/../vendor/rails/railties/lib/initializer.rb:327:in `initialize_framework_settings'
from /var/www/apps/argent/config/../vendor/rails/railties/lib/initializer.rb:324:in `each'
from /var/www/apps/argent/config/../vendor/rails/railties/lib/initializer.rb:324:in `initialize_framework_settings'
... 15 levels...
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/../lib/mongrel/command.rb:212:in `run'
from /usr/local/lib/ruby/gems/1.8/gems/mongrel-1.1.4/bin/mongrel_rails:281
from /usr/local/bin/mongrel_rails:19:in `load'
from /usr/local/bin/mongrel_rails:19


At the time of this post, I only saw three hits when Googling on "ruby", "rails", and "uninitialized constant CGI::Session::ActiveRecordStore (NameError)". 2 links to Japanese blog posts, the 3rd to a Rails forum in Polish. None helped me resolve the above problem.

But I was Googling in English.

When I tried changing tactics and Googling in Japanese instead, I found this link, and a hint that led me to the following solution...

There is some kind of require collision happening when trying to use both the ActiveRecordStore for session management and the gettext package for localization/internationalization. The answer is to put the require 'gettext/rails' declaration well after the default Rails environment bootstrapping. In my environment.rb, I've got my requires declarations set up like so:


# Bootstrap the Rails environment, frameworks, and default configuration
require File.join(File.dirname(__FILE__), 'boot') # default
...

Rails::Initializer.run do |config|
# Use the database for sessions instead of the cookie-based default,
# which shouldn't be used to store highly confidential information
# (create the session table with 'rake db:sessions:create')
config.action_controller.session_store = :active_record_store
...
end

# After the Rails::Initializer.run block, just before end of file
require 'jcode'
require 'gettext/rails'


In doing so, you can avoid any namespace collisions between the default Rails requires and that for gettext/rails. Hope this saves you some time and headaches.

Monday, May 12, 2008

cron jobs for PostgreSQL users without using passwords

Ever needed to run some kind of cron job on a PostgreSQL database, but was stumped for not being able to specify a database user's password with the psql utility?

In using PostgreSQL with Ruby on Rails, I often have to set up regular cron jobs to clear out stale data, such as session data when using ActiveRecord as the session storage mechanism. Instead of coding up some object to react on a timer, you can simply use cron to do your bidding. But your regular cron job to connect to the PostgreSQL database needs a user, and that user needs to provide a password when connecting. Here's how you can set this up by creating a new UNIX/PostgreSQL user and configuring this user's database access in pg_hba.conf.

First, create a new UNIX account for your user. We will use the same username when creating a database user account as well. Why? Well, PostgresSQL allows connections not only over TCP/IP, but also over UNIX domain sockets. And, you can configure pg_hba.conf so that when a certain database user attempts to connect over a UNIX domain socket, PostgreSQL will check for an existing UNIX user account. If this UNIX user is the same as the user account attempting the database connection, then PostgreSQL will treat such access as trusted, hence no explicit password passing will be required.

For example, let's say that I want to clear out stale session data from my Rails sessions table. I first create a UNIX account for my database user.


# useradd victor

Don'f forget to set up a good password to victor. You may also want to revoke permissions from groups and others to access this user's home directory. For production environments, you want to be sure to assign victor to a group with limited UNIX user privileges, since we don't want people to misuse victor and mess with our database.

Next, we create a PostgreSQL database user account for victor, using the same username. We will create this database user with limited privilege on the sessions table, for security's sake. Assuming that the we are talking about the default Rails sessions table, we have:

# psql -U [database admin] [your_database]
psql@your_database> create user victor with password 'somepassword';
psql@your_database> grant select, delete on sessions to victor;

There! Now we have a PostgreSQL database user corresponding to our newly created UNIX user account.

Next, we add this line to our $PGDATA/pg_hba.conf file:

# TYPE DATABASE USER CIDR-ADDRESS METHOD
local [your_database] victor ident sameuser


This is how we allow the new user to connect to the PostgreSQL database using a UNIX domain socket connection. Logging into your UNIX machine as victor, you can now set up a simple cron job like so:

$ crontab -e
# crontab for victor
# Let's run a simple psql command as victor to delete stale rows in sessions table
# Execution time is
0 0 * * * /full/path/to/psql -U victor -c "delete from sessions where \
updated_at < (now() - interval '6 hours')" [your_database] > /dev/null


So now we are good to go, with a UNIX/PostgreSQL user whose cron job will run daily at midnight, clearing out stale rows in the sessions table for our Rails ActiveRecord session management. And all without having to explicitly specify the password!

Friday, April 25, 2008

Apache to mongrel proxy permission denied on CentOS 5.1

Here's a brief tip for those of you about to set up Ruby on Rails, mongrel and Apache web server on CentOS 5.1: don't forget that you may have SELinux enforcement policies in place by default that prevent proxy balancers to connect correctly.

Now, you probably have proxy directives like the following in your httpd.conf file:


<Proxy balancer://mongrel_cluster>
BalancerMember http://localhost:3000
BalancerMember http://localhost:3001
BalancerMember http://localhost:3002
</Proxy>


Firing up your mongrels and kicking up httpd will result in errors like this in your Apache errors logs:

[Thu Apr 24 23:24:20 2008] [error] (13)Permission denied: proxy: HTTP: attempt to connect to localhost:3000 (myserver) failed
[Thu Apr 24 23:24:20 2008] [error] ap_proxy_connect_backend disabling worker for (myserver)
[Thu Apr 24 23:24:21 2008] [error] proxy: BALANCER: (balancer://mongrel_cluster). All workers are in error state
[Thu Apr 24 23:24:21 2008] [error] proxy: BALANCER: (balancer://mongrel_cluster). All workers are in error state
[Thu Apr 24 23:24:21 2008] [error] proxy: BALANCER: (balancer://mongrel_cluster). All workers are in error state


Hm... alright, but what does /var/log/messages have to say?

Apr 24 23:24:19 myserver setroubleshoot: SELinux is preventing the /usr/sbin/httpd from using potentially mislabeled files / (unlabeled_t). For complete SELinux messages. run sealert -l 7e096a76-b66f-48f9-a30e-d736dbb6007d


Actually running that sealert command won't tell you any more beside the fact that SELinux is denying httpd access to potentially mislabeled files. Whatever that means. But perusing section 44.2.6. Enabling or Disabling Enforcement of the CentOS 5.1 deployment guide, there is enough there to resolve this obstruction.

Executing the sestatus command lets us see what SELinux is doing to protect our server.

[root@myserver]# sestatus -b | grep httpd
...
httpd_can_network_connect off
...


Aha! This boolean property looks like it could be one that is denying the proxy connection from httpd to mongrel. So, let's enable this, shall we? The command to do this is setsebool. Reading the friendly man pages before running this command, we learn that setsebool will only set the current, running state of the boolean in question, and that if we desire to have this setting survive reboot, then we need to use the -P flag. Let us do exactly that:

[root@myserver]# setsebool -P httpd_can_network_connect=1


Now when you restart your mongrels and httpd, you will see that the proxy connections are good to go. And our configurations will live on, even after reboot. Sweet!

Wednesday, April 9, 2008

PostgreSQL 8.3.1 RPM's on CentOS 5.1

CentOS 5.1./RHEL5 comes with PostgreSQL 8.1.9, but if you are like me and want to enjoy the latest and greatest from the world's most advanced open source database, then you might want to try installing the PostgreSQL 8.3.1 RPM packages.

However, you will find that the postgresql-libs package has quite a few dependencies, not the least among them is apr-util, which is used by httpd. Deciding to go against the grain (and most common sense with respect to the entire premise of CentOS), I recently made a haphazard installation attempt whereby I blindly removed the postgresql-libs package with the rpm --nodeps option. My Apache web server refused to start, complaining about conflicts related to libpq.so.4 after I thought I successfully installed PostgreSQL 8.3.1. Luckily, this was all on VirtualBox, so recovery was not a problem.

Let me save you some time and headaches by outlining briefly the steps you can take to resolve this dependency issue by replacing the files upon which these dependencies lie by installing the compat-postgresql-libs-4, using the --replacefiles option.

First, go ahead and fetch the PostgreSQL RPMs for 8.3.1. And make sure you grab the aforementioned compat-postgresql-libs-4 RPM, too. I found all of the RPMs needed here at this archived repository.

Next, go ahead and start removing the 8.1.9 PostgreSQL packages with the rpm utility, saving the postgresql-libs package for last.


# rpm -e postgresql-contrib
# rpm -e postgresql-server
# rpm -e postgresql


Now when you attempt to remove the postgresql-libs package, you may see this:

# rpm -e postgresql-libs
error: Failed dependencies:
libpq.so.4 is needed by (installed) apr-util-...
libpq.so.4 is needed by (installed) exim-n...
libpq.so.4 is needed by (installed) mod_auth_pgsql-...
libpq.so.4 is needed by (installed) dovecot-...
libpq.so.4 is needed by (installed) perl-DBD-Pg-...
  ...


The key here is libpq.so.4, which refuses to go quietly into the night when attempting to install PostgreSQL 8.3.1. So what are we to do? Well, let's take a closer look at what happens when we try to install the compat-postgresql-libs-4 RPM.


# rpm -i compat-postgresql-libs-4-1PGDG.rhel5.i686.rpm
file /usr/lib/libpq.so.4 from install of compat-postgresql-libs-4-1PGDG.rhel5 conflicts with file from package postgresql-libs-8.1.9-1.el5
file /usr/lib/libpq.so.4.1 from install of compat-postgresql-libs-4-1PGDG.rhel5 conflicts with file from package postgresql-libs-8.1.9-1.el5


Aha! Here we've found where the conflict lies. Let's work around this by replacing the existing libpq.so.4 from the original 8.1.9 with that from compat-postgresql-libs-4-1PGDG.rhel5.i686.rpm.


# rpm -iv --replacefiles compat-postgresql-libs-4-1PGDG.rhel5.i686.rpm


We are now home free! Continue by uninstalling the old postgresql-lib package, and you can then start the 8.3.1 package install. There should be no more trouble with dependencies for libpq.so.4, since it still exists but has been swapped out with that of compat-postgresql-libs-4-1PGDG.rhel5.i686.rpm. Good to go.

Sunday, March 23, 2008

Installing VirtualBox Guest Additions for CentOS 5.1

VirtualBox, a virtual machine product from Innotek (recently acquired by Sun Microsystems), is a great development resource when you need to have access to different environments and platforms close at hand and under your full control. I have been using it for the past few months, hacking up Ruby on Rails applications while on Windows XP but targeting deployment for Linux (CentOS 5.x). I can easily set up and run multiple environments, all from my company-issued Dell laptop with the stock Windows XP operating system. For example, I've got virtual machines for CentOS 5.1 and FreeBSD 6.2. VirtualBox claims support for other guest operating systems, including Windows (NT, 2000, Server 2003, XP, Vista), many flavors of Linux distros (Ubuntu, Debian, SUSE, Fedora, Red Hat, et al.), Solaris 10, BSD-based OpenBSD and FreeBSD, and more.

It can be particularly useful if, like me, you are forced to develop on a Windows box but often deploy to *NIX environments. I have found it to be a drag, for example, when trying to build Ruby gems for mysql and postgresql on Windows, even with the aid of Cygwin, a Linux emulation layer for Windows. I highly recommend getting VirtualBox, and setting up a virtual environment over which you can exercise complete control.

However, for a virtual machine, there are certain things that are not simple and straightforward to do, like sharing folders between the host and guest operating systems. Thankfully, VirtualBox comes with the resources to install drivers and system applications into your guest operating system that make integration with your host, and your life, much easier. These resources are called VirtualBox Guest Additions. There are guest addition resources for Windows and Linux guest operating systems.

While the product documentation and user's forum for VirtualBox is quite good, I found precious little on how to install the guest additions for Linux correctly. Let me share with you how I quickly installed the guest additions on Linux for my CentOS 5.1 guest. My CentOS 5.1 guest is a server installation, so we are going to use the command line to do this.

1. Install the necessary packages and source so that you can compile the guest additions for Linux. Of course, you need to be root to do this:


# yum install gcc -y
# yum install kernel sources -y
# yum install kernel-devel -y


2. Mount the VBoxGuestAddtions.iso file as your Linux guest's virtual CD/DVD-ROM drive. There are a couple of ways to do this:
  • If your guest's virtual machine is not started, then in the VirtualBox main window, click on the your target guest Linux operating system to bring up the details and settings tabs. In the Details tab, click on CD/DVD-ROM to bring up Settings panel. Select "Mount CD/DVD Drive", and click the radio-button to select ISO Image File, making sure that the VBoxGuestAdditions.iso file is selected. Click the OK button, and start up your virtual machine.
  • Conversely, if your guest's virtual machine is already running, select Devices from the virtual machine console window's top menu. Select "Mount CD/DVD-ROM", and further select "CD/DVD-ROM image" in the sub-menu. The Virtual Disk Manager window for your guest operating system will appear on-screen. Make sure that the VBoxGuestAdditions.iso file is selected. Click the Select button.


3. As root, execute the following commands to mount the Guest Additions iso file as a directory in your guest operating system.

# mkdir /tmp/cd
# mount /dev/cdrom /tmp/cd


4. Navigate to the newly-mounted directory, and after confirming that the the guest addition files are there, compile and install the guest additions.

# cd /tmp/cd
# sh ./VBoxLinuxAdditions.run


You will need to restart your guest operating system now. Upon doing so, you will then be able to enjoy closer integration between your host and guest operating systems, letting you enjoy features like:
  • mouse pointer integration
  • improved video support
  • time synchronization
  • shared folders
  • shared clipboard
  • automated Windows logons for Windows guests


Personally, I make great use of the shared folders feature so that I can write code in my IDE on Windows, but immediately execute that code on Linux.

Sunday, March 16, 2008

森林公園温泉が閉館する! Shinrin Kouen Onsen is closing?


Ah, man... not only did I just remember that I completely forgot about Pi Day, I just found out that my favorite neighborhood onsen bath is closing down. Yeah, 森林公園温泉 (Shin'rin-kouen onsen) is closing its doors in April. It is small, but the facilities are clean and the hot spring water a great way to relax after a hard day of squinting at my screen and typing.

My family really like going there to soak, and we've been going there almost every weekend. It is one of the nicest things about living out here in this corner of Sapporo. It's been just about one year since we moved here, and it is kind of sad to think about life without 森林公園温泉 . I hope that maybe only the management will change hands, and the bath water will continue to flow.

Sunday, February 10, 2008

Sieve of Eratosthenes: Ruby and Python

Here's a bit more math history and coding nonsense, this time involving prime numbers...

Eratosthenes was a Greek mathematician and one of the greatest scholars of Alexandria (273-192 B.C). He derived an algorithm for sifting out the composite numbers in the natural series, leaving only prime numbers remaining. The algorithm works like this:


1. Start with a list of all the odd numbers in order, starting from 3.
2. Cancel the successive multiples for each element, one after another.
3. Continue iterating through the list until you reach the end.


As usual, just for fun, I coded up the Sieve of Eratosthenes in Ruby and Python. Here, we start with a list of natural numbers in order, beginning with the first prime number, 2. The sieve logic then examines each element of the list, and if the value of the list element in question is less than the square root of the maximum value in the list, then we proceed to remove all multiples of this list element, starting from the square of this element and working upwards through the end of the list. Such an approach increases efficiency a bit so that we do not have to check all elements up through to the end of the list when searching for multiples to sift out.


In Ruby


def execute(limit=100)
# set up
primes = (2..limit).to_a

# sieve logic
sieve_end = Math.sqrt(limit).floor
for i in 0...primes.size
break if primes[i] > sieve_end
primes.delete_if { |n| n >= primes[i]**2 && n%primes[i]==0 }
end

primes
end


In Python

def sublime(m):
return lambda x: x==m or (x>m and x%m!=0)

def execute(limit=100):
# set-up
primes = range(2,limit+1)

# sieve logic
sieve_end = int(sqrt(limit))

for i in range(0, len(primes)):
if primes[i] > sieve_end: break
primes[i:] = filter(sublime(primes[i]), primes)

return primes


While the code above may offend pure Rubyists and Pythonistas, it nevertheless appeals to me for still being rather clean. I tried to use more a more functional programming approach, with a nice Ruby block and a Python lambda function for sifting out the non-primes.

I haven't been coding any Java seriously for the past few months, and I don't seem to miss Java much. I must admit that functional programming seems quite fresh and interesting, and both Ruby and Python have features that allow for a functional approach mapping, reducing, and filtering lists.

Wonder if I should try this in CLisp tonight?

Wednesday, February 6, 2008

Starting Up Google Code Project

I am starting up a Google Code project at http://code.google.com/p/brooke-fujita/. It should be a good way to store/share any code I write, in an effective and easily-accessible manner. I have opted for the Apache style licensing, just for kicks. よろしく!

Sunday, February 3, 2008

Rant on Ruby On Rails

I have just about wrapped up my spike using Ruby on Rails. It was only for a very small RESTful web service, but this experience has left a somewhat bitter aftertaste. Here's a short list of things I hate about Ruby on Rails:

1. No documentation.
This one floored me, but it is true. Just have a look here. Do you see any useful specifications? User manuals? Nope. Just some dry API documentation, a handful of tutorials, and a link to a book for sale. Sorry, but does anyone find this acceptable?

2. Questionable release management
The latest Rails release broke what little library support was available for composite keys for database tables. I don't even know if this has been fixed yet. But that's not all.

The latest Rails release also apparently broke the API for setting up functional tests. Again, I don't even know if this has been addressed, not even in edge Rails.

3. Rails API for send_file and send_data leaks memory
This is not a mongrel problem. I've seen this happen with webrick, and I bet this would happen with whatever else you happen to use with Rails, but if you try to stream data or a file with the aforementioned APIs, Rails will not release the memory it uses. The current best practice is to use the HTTP header X-Sendfile if you've got to send a static file, but you are hosed if you need to do something dynamic. Oh, and I think this issue has been around since 2006.

Now, this is just a short, 3-bullet list. I get this sneaking suspicion that if I were working on a larger project, one with more enterprise-level requirements such as Service Level Agreements on performance or speed, I suppose I would find more drawbacks to using Ruby on Rails.

Which is a disappointing, for the expressiveness of Ruby and Rail's basic approach to being a fast-track framework for web apps that babysit databases actually makes for a very highly productive environment. Don't get me wrong: I would consider Ruby on Rails for future projects, but only after carefully considering the functional requirements versus what Rails can/cannot/will not do. There's way too much hype surrounding Rails right now, and that clouds better judgment.

Sunday, January 27, 2008

札幌:好調なIT産業?

Another fairly recent article in the Hokkaido Shimbun 「好調なIT産業」、2008年1月9日、木曜日、北海道新聞(経済)stated that a recent annual report on the state of IT in Hokkaido (sorry, link in Japanese) that the IT business in Hokkaido is growing at a healthy pace. The number of IT employees is growing in proportion to the annual sales figures, and has been doing so since the past 6 years.

Hm...

From my vantage point, I can understand that IT as a business in Sapporo may be doing well, and there may be a great demand for software developers. I hesitate to use the term "systems engineer", but that's a rant for some other day.

However, AFAIK, things are not as good as the newspaper paints it to be.

First off, the pay scale here in Sapporo for IT has to be the lowest in the Japan. It sure seems that way, looking at my measly paycheck. Try stretching that for a family of four, when the cost of fuel and oil is rising seemingly unchecked, and the effects rippling quickly outward to the prices of groceries and household goods. You know, I had a better salary when I was simply a junior developer back in San Francisco over 8 years ago. I am certain that there are many facets to managing and keeping human resources for a Japanese company, but I still don't get why the salaries up here in the north are so pitifully low.

The aforementioned article does get one thing right, however: it is orders from the greater Tokyo metropolitan that support IT in Sapporo. I fully understood that before deciding to move up here from Tokyo, but I didn't realize that the business model of fishing for gigs in Tokyo and then developing/testing up here in Hokkaido is practically all there is to the game up here. Seriously, there is little beyond that happening up here.

Which is a bit of a shock, coming from Silicon Valley as I did. There are lots of ideas, schemes, grand plans everywhere in the Valley. Start-ups, innovations. Not so here in this little corner of Japan. I almost choked when I overheard someone expressing their desire to market a "security product" consisting of a Javascript hack to obfuscate an email address in order to prevent a bot or script from screen-scraping. Yes, a pure WTF moment. But again, this point could lead me to write some whole other rant, so I will not delve further.

I expect that IT in Sapporo could be suffering for lack of developers due to the fact that most of the good ones probably left for Tokyo. As it is, I hear that coding is now regarded as one of the 3K jobs: difficult, dirty and dangerous. Well, that may be right, but I still feel that there is some way to create an opportunity, to re-invent myself so that I can have that "lifestyle" job and continue to live in beautiful Sapporo.

Nope, don't wanna wear a suit and tie. Nope, don't wanna have to be sitting at some office desk from 9:00 a.m. to 6:00 p.m. just to make some boss feel like he's getting his money's worth out of me. This lack of vision is stifling. Doesn't anyone value the artistry and creativity of software?

And no, hacking up Javascript to hide email addresses from bots is not security!

藤田ブルック?フジタ・ブルック?それとも Brooke Masato Fujita

I was just at the local ward office to get some papers to file with a tax declaration when I remembered to ask the person at the desk a question that has been bugging me for a while.

Can I use my family name in kanji as my official name?

Being of Japanese descent, I actually do have name that is written in kanji. I can actually trace my family all the way back to somewhere in Kumamoto prefecture on the island of Kyushu, in the south of Japan.

However, since I am of US citizenship, as the person at the ward office explained, my official name per my US passport shows "Brooke Masato Fujita". Using the alphabet. Right now, my Foreigner's registration card shows my name using the alphabet, but allows me to also have as 藤田ブルック as my alias. My alias is not official, which means my family cannot use their kanji name. Their names in the Japanese citizens registry (my wife and children have Japanese citizenship) are フジタ. Which sticks out like a sore thumb. Not too many average Japanese use katakana in their names.

So, since I am not Korean or Chinese, which would probably mean that my passport may show my name as 藤田, I have no way of using my kanji name officially. Unless I change my citizenship to become Japanese. And I really cannot see any advantage in changing my citizenship.

What's in name?

北海道 2万人転出超過

A recent article in the Hokkaido Shimbun paper (sorry, related link in Japanese) reports that for 2007, the number of people that left Hokkaido to relocate elsewhere in Japan exceeded 20000. This is an increase of almost 2000 people when compared to the same figures from 2006. Most of these people likely relocated to the greater Tokyo metropolitan, in search of jobs, as Hokkaido suffers from an increasing lack of thereof.

This is similar to the situation in Hawaii, where a brain-drain has been occurring for as long as I can remember. The best and brightest often find themselves leaving because there is not enough to keep them there.

Not that I've lived in many places in the world, but this is still a sad thing, as Hokkaido and Hawaii are two of the most beautiful places to live. For myself, it would be great if telecommuting becomes more wide-spread and acceptable, because then I wouldn't have to tie myself to the vagaries of the current local market situation here in Sapporo. The systems integration company I currently work for has a business model (if you can call it that) that is largely hand-to-mouth, survival only by finding enough gigs to build smallish web-sites and then try to charge a recurring fee for support and administration. I don't think this scales at all.

Adapt. Improvise. Evolve. It is high time to think, rethink, and reinvent.

Sunday, January 20, 2008

Pi, Probability, and Python

π and probability theory intersect more often than you would expect. The mix gets more interesting when you combine that with a quick look at math history, and toss in a short program as well.

Buffon's Problem
George Louis Leclerc, Comte de Buffon (1707-1788) was a general scientist, mathematician, and gambler, as well as a man of means with time to spare on interesting math problems. One which he put forth and subsequently solved was this: Given a needle of length L, to be tossed upon a horizontal planed with parallel straight lines separated uniformly by a distance d, what is the probability that the needle will intersect with one of the lines?

Assume that the position and orientation of the needle are two variables that are random and independent. Let the distance from the needle's center to the nearest line be x, with the orientation of the needle being the angle φ created by the needle intersecting the line in question. Consider x and φ per the needle's intersection with a single line, since all intersections with any other line will be the same.

Now from the figure above, it can be seen that the needle will intersect a line if and only if

x < ½ L sinφ

Therefore, Buffon's problem can be resolved by finding the following probability:
P(x < ½ L sinφ)

In order to find this probability, we can visually represent this by graphing with the rectangular coordinates x and φ, where
range: 0 < x < d/2
domain: 0° < φ < 180°

Given these intervals for x and φ, any point that lies within the parallelogram described by points A and B correspond to one and only one position and orientation of the needle thrown. Furthermore, since all such combinations are equally probable, the area of AB represents the sum of all probable outcomes for the thrown needle's position and orientation, with the area under the curve x < ½ L sinφ representing the sum of all possibilities where the needle intersects a line.

Hence, the probability that the thrown needle intersects a line is the ratio of the the sum of possible outcomes where the needle intersects a line, to the sum total of all possible outcomes. This can be expressed by:


Laplace and the Monte Carlo Method
Pierre Simon Laplace (1749-1827) saw Buffon's solution from a different perspective, which gave rise to a new method in computing. From that last expression above, Laplace generalized it to:

The end result was a new method whereby π could now be calculated by determining the probability P. The calculation is simplified if one sets d equal to L. But the key point is that Laplace discovered a method whereby a numerical value is found by realizing a random event many times and observing the outcomes. This method is known as the Monte Carlo method.

So where does the Python come in? I wrote a method in Python for computing π in Monte Carlo fashion:

from math import *
from random import *

def execute():
count = 0.0
step = 1000.0
for i in range(1, 10001):
for j in range(0, step):
x = 0.5 * random()
phi = randrange(0, 180, 1)
if x < 0.5 * sin(radians(phi)):
count += 1
P = count/(i * step)
print "%d: P = %0.25f, pi = %0.25f" % (i * step, P, 2.0/P)
print "all pau!"


This runs a Monte Carlo simulation of 10 million throws of the needle, computing the probability of intersection every 1000 steps of the way. One execution yielded the following output:

...
9995000: P = 0.6366198099049524827819369, pi = 3.1415924683503022585284725
9996000: P = 0.6366196478591437113436768, pi = 3.1415932680144256217147358
9997000: P = 0.6366192857857356779405222, pi = 3.1415950547767912404140134
9998000: P = 0.6366201240248049453285262, pi = 3.1415909182319108339243030
9999000: P = 0.6366206620662065995830403, pi = 3.1415882631091953669510985
10000000: P = 0.6366190000000000459579041, pi = 3.1415964650756573739442956
all pau!


Hm... only correct to 5 places after 10 million trials? I think I may have to see about calculating the probably of computing π to n places in t trials. While it may not be all that efficient for calculating π, the Monte Carlo method is nevertheless a powerful approach for resolving problems with incredibly large sets of computer-simulated trials.

With inspiration from
A History Of Pi, P. Beckmann
History of Mathematics, D.E. Smith

Saturday, January 5, 2008

Polyglot

pol·y·glot (pŏl'ē-glŏt')
adj. Speaking, writing, written in, or composed of several languages.

n.
1. A person having a speaking, reading, or writing knowledge of several languages.
2. A book, especially a Bible, containing several versions of the same text in different languages.
3. A mixture or confusion of languages.


[French polyglotte, from Greek poluglōttos : polu-, poly- + glōtta, tongue, language.]


Stevey, in a long but well-put rant on code bloat, exposes an issue which I have been wrestling for the past several months:

But you should take anything a "Java programmer" tells you with a hefty grain of salt, because an "X programmer", for any value of X, is a weak player. You have to cross-train to be a decent athlete these days. Programmers need to be fluent in multiple languages with fundamentally different "character" before they can make truly informed design decisions.

Recently I've been finding that Java is an especially bad value for X. If you absolutely must hire an X programmer, make sure it's Y.


This is absolutely true. I must admit that my résumé and work history really only show my roots in Java. But being an autodidact and wannabe polyglot, I can definitely see that I need to be cross-training. To that end, I am presently pursuing knowledge in the following languages:

  • Ruby: I have written a very small web service in Ruby and Ruby on Rails in my "day job". Rails is OK, if your requirements fit the "sweet spot" that Ruby On Rails proclaims to hit, but Lord help you if you need stuff, like, say, composite keys.
  • Python: Before Ruby, I stumbled across Python. Kevin, if you're out there reading this, thanks for a) telling me about Guido van Rossum and b) introducing me to Cygwin ("it makes Windows useful")
  • Common Lisp: from my short experiences with mathematics, I have learned to appreciate elegance, and Lisp is nothing if not elegant. David, if you're out there reading this, I apologize for not recognizing the truth in your words and conviction any earlier. But I still have not gotten the hang of Emacs yet...

Thursday, January 3, 2008

Thanks

We are 3 days into 2008 (平成20年), and already I have found a lot for which I can be grateful, a lot that make this shiny New Year special. We were driving out to Kushiro to visit my In-Law's when we spun out on an icy patch and went straight into a guard-rail on the expressway exit near Yubari. No one was hurt, and the only damage was to our tough little Toyota Sienta. It was the second ambulance ride for my poor 2-year old son in the space of a week, but thankfully, in both cases there was nothing wrong.

Family. Friends. All the people you love. That's what really matters. All else can be replaced.