Saturday, February 11, 2012

Auspicious Moment for Cogitation

Lately, I have been having a lot of fun at work drafting emails to various internal ‘stakeholders’. Being a data analyst who spends the bulk of his time crafting and running SQL queries against a ponderous leviathan of a data warehouse, there are frequent downtimes due to competing queries running simultaneously, insolent IT cretins performing in broad daylight what are intended to be nocturnal ‘cron’ jobs, or my own queries scanning and processing gigantic datasets such as call record details.

Since I refuse to ascend to the sunny uplands of my non-existent Facebook to update my status every nanosecond, I often descend with glee and gusto, as I wait for my queries to fetch desired records from the netherworld of Teradata ‘amps’, to the corporate banality of email writing.

While not compromising or clouding the messages, one of my aims in drafting emails to the mythical stakeholders who rely on data analysts for reports and analyses is to parody the imagined diction of an educated foreigner who learned English by reading Gibbon with the aid of nothing more than a hefty, well-thumbed dictionary. For good measure, I often intersperse my turgid, highfalutin prose with Latin phrases. Quid quid latine dictum sit, altum videtur.

“My final obiter dictum on the … report …”, announced one of my recent emails. Another began: “Now is a most auspicious moment to cogitate on …”. “Do you wish to circumscribe the report with a temporal boundary by prescribing an arbitrary baseline date? If yes, did the madam have a date in mind?” another inquired politely of a young marketing ‘exec’. Another finished by lavishing “most sincere thanks on the honorable gentlemen” who were implementing an IT change request.

Far be it from me to mock my stakeholders, who are really my colleagues, even though my partner warns that is how my playfulness, designed partly to alleviate ennui, could be misconstrued. In reality, I am also partly playing to the stereotype of data analysts, who inhabit, in my team’s case anyway, that crepuscular no-man’s land between the IT and marketing department.

With their Masters of the Universe mindset, some IT managers, the vast majority of whose roles furnish the modern equivalents of overseers of indentured labor in the far-flung sugarcane plantations of a benighted age, look down on data analysts as little more than middling marketing mediocrities uninitiated in the runes and rituals of information technology. Some marketing execs and product managers, on the other hand, suspect data analysts of being nothing more than number-crunching numb nuts devoid of humanizing creative impulses.

Actually, just like any other profession, “marketing analytics” attracts people from varied and storied backgrounds. My own group has, at various times, counted in its ranks analysts with degrees and backgrounds in mathematics, linguistics, literature, statistics, IT, computer science, software engineering, robotics, business, marketing, hospitality, customer service, etc.

All data analysts perform three key tasks: Scouring, sourcing and cleaning data, called “data munging” in the trade, followed by analysis and/or modeling, which can range from pivoting data in Excel to implementing sophisticated machine learning algorithms, and, finally, presenting them to stakeholders, an art that has spawned its own sub-discipline of “visualizing beautiful data”.

The profession, which is red-hot at the moment due to the exponential growth and availability of “big data”, has its share of quackery but is there one that does not?

But I have strayed far from the topic. Ipso facto, now is a most auspicious moment to shut up.

Saturday, November 12, 2011

Major Pieces in 7th Heaven

Of late, I have been pitting my modest chess skills against the AI Factory Free Chess on my mobile. Finally, I am getting the upper hand at Level 9, recording a 55% success rate.

In this game, orchestrating the Black pieces, I piled pressure on the half-open b-file and infiltrated the 7th rank with the major pieces, after which the enemy King fell swiftly on the other side of the board.

[pgn height=500 initialHalfmove=28 autoplayMode=none showMoves=justified]
[Event "Man vs Mobile"]
[Site "Kogarah, NSW"]
[Date "2011.11.12"]
[White "AI Factory Free Chess"]
[Black "RL"]
[Result "0-1"]

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. d3 d6 6. c4 Be7 7. Nc3 Bg4 8. Be3 0-0 9. Bxc6 bxc6 10. h3 Bh5 11. g4 Bg6 12. c5 Re8 13. Rc1 Bf8 14. Bg5 h6 15. Bxf6 Qxf6 16. 0-0 Qe6 17. Nd2 d5 18. Nb3 a5 19. a4 d4 20. Nb1 Reb8 21. Nb1d2 Be7 22. f4 exf4 23. Rc4 Bf6 24. Nxd4 Bxd4+ 25. Rxd4 Rxb2 26. Nf3 Rab8 27. Qc1 Qf6 28. h4 Ra2 29. h5 R8b2 30. hxg6 Qxd4+ 31. Kh1 fxg6 32. Qxf4 Qxd3 33. Rc1 Qe2 34. Rg1 Qxf3+ 35. Qxf3 Rh2++ 0-1
[/pgn]

Wednesday, October 12, 2011

Computational Model of Peace Predicts Social Violence, Harmony

"A systems model of how ethnic tensions flare into violence has passed a test in Switzerland, where harmony prevails except for one region flagged by the analysis," writes Brandom Keim at wired.com.

"The model runs census data through an assembly line of high-powered mathematical processes, but at its root is one basic assumption: that community-level violence is primarily a function of geography, modulated by the overlap of political, topographical and ethnic borders."

Click here to read more.

Sunday, June 5, 2011

Monte Carlo Simulation to Estimate pi

Imagine a person randomly throwing darts at a square board with a unit circle inscribed on it. The proportion of darts that land in the circle multiplied by 4 gives an estimate of the value of pi. The point (x, y) hit by the dart is in the circle if x^2 + y^2 <= 1.

What follows is my implementation of this idea in Teradata sql, with the value of pi estimated to two decimal places. The resource at the end of this post has a nice discourse on this topic, with Python implementations and a video on Monte Carlo simulation.
/* Create a table with a random point (x,y) on the board */
CREATE VOLATILE TABLE tblTemp AS
(SELECT RANDOM(-100,100)/100.00 AS x
,RANDOM(-100,100)/100.00 AS y
,CASE WHEN x**2+y**2 <= 1 THEN 1
ELSE 0
END AS inCirc
)WITH DATA PRIMARY INDEX (x)
ON COMMIT PRESERVE ROWS;

/* Generate a sufficiently large (but not too large!) number of random points (x,y) on the board */
CREATE VOLATILE TABLE tblRndPts AS
(WITH RECURSIVE tblRnd (lvl, x, y, inCirc) AS
(SELECT CAST(1 AS DECIMAL(18,0)) AS lvl, x, y, inCirc
FROM tblTemp

UNION ALL

SELECT lvl+1
,RANDOM(-100,100)/100.00 AS x1
,RANDOM(-100,100)/100.00 AS y1
,CASE WHEN x1**2+y1**2 <= 1 THEN 1
ELSE 0
END AS inCirc
FROM tblRnd AS t1
WHERE t1.lvl < 10000 /* total number of points (x, y) */
)
SELECT *
FROM tblRnd
) WITH DATA PRIMARY INDEX (lvl)
ON COMMIT PRESERVE ROWS;

/* Calculate pi and the diff from the actual value */
SELECT 4.00 * SUM(inCirc)/COUNT(*) AS pi, CAST(pi() - pi AS DECIMAL(4,2)) AS diff
FROM tblRndPts;

Click here to read more about this topic.

Monday, May 2, 2011

Pacquiao Was Inspired by Bruce Lee

pacquiao

On the eve of his fight with Shane Mosley this Saturday US time, The New York Times reports that the Filipino boxing sensation Manny Pacquiao was inspired by Bruce Lee.

"The boxing genius of Manny Pacquiao includes feet that belong in Riverdance, calves the size of grapefruits and deceptive power generated from his core. His movement is unorthodox, scattered and perpetual, as if designed by a jazz musician. He creates angles unlike any other fighter, past or present, appearing, disappearing, shifting, striking; on balance, off balance, even off one foot.

"It is this style — part performance art, part technical wizardry, unique to Pacquiao— that defines perhaps the best boxer of his generation. And it started with a videotape of the martial artist who became his idol. It started with Bruce Lee.

"Last month, as Pacquiao molded his style specific to Shane Mosley, his welterweight opponent on Saturday in Las Vegas, he wrapped his hands inside the dressing room at the Wild Card boxing gym here. To explain the way he fights, he settled on three words.

“Like Bruce Lee,” he said.

Saturday, April 30, 2011

Thinking About CMM

Some things learned in the academia can have practical significance. One thing I learned in a dry, theory-laden unit was the concept of Capability Maturity Model (CMM). It is summed in the following diagram.

[caption id="attachment_804" align="aligncenter" width="480" caption="Capability Maturity Model"]Capability Maturity Model matrix[/caption]

The CMM can help locate where an organization or a unit within an organization is in relation to a particular process or functional capability.

In a sense, CMM does for organizations what Maslow's hierarchy of needs does for human beings. The latter claims that lofty human needs cannot be fulfilled before satisfying baser needs.

By helping organizations assess their capabilities or lack thereof, CMM can point out where an organisation or a team needs to pour its resources before planning grand but doomed schemes.

Thursday, April 14, 2011

Thus Spake HAL

Dissociated press is a computer program that generates text based on another text. Here is an example of dissociated press based on the US presidential inaugural addresses:
Fellow - Citizens of the world. Abroad and at the same mistake. In a great river system, purified and made self. And there is a privilege which he has shown that our Government together and laugh together and save the lives of our civilization secure. In our time, a pleasing guaranty that the actual equality of rewards or possessions so long bled and suffered, we have fought not for themselves. Mindful of these States, was out of the Ohio or of rank have been distinguished by some from a fever of words

A computer would have apparently written The Book of Genesis thus:
In the day when they were both the daughters of Lot with him, at the first born; I bare the loss of it; for a possession of flocks, and she called his name Onan. And Hamor communed with him for righteousness. And her brother and her damsels, and possession of herds, and at night he shall be fruitful and multiply thee, then the other end thereof. And Terah took Abram his son with the sword? Wherefore didst thou flee away secretly, and where thou art yet alive,

I was curious how Gandhi would come out. The following text is based on a collection of Gandhi's writings at Project Gutenberg. I did not remove the Project Gutenberg copyright notice before feeding it to the computer, and the effects are pleasing:
Project Gutenberg is a grand doctrine of life. With them it is to weld India into one, he instanced Scotland's poverty of the Government whenever it needed their help. Their resistance consisted of disobedience of the copyright holder, your use and distribution of Project Gutenberg-tm eBooks are often created from several printed editions, all of them fellow cranks--if in similar circumstances I acted towards them differently from my own family. It were better to concentrate my effort upon the subject before us. I was hard put to it that the co-operative movement. " In his opinion