Briefly then, let me bring you up to date, gentle reader. It’s now the month of May, in the year two thousand and twelve. I’ve spent the past few months working on a few projects.

The first: my job. Yup, that’s a project like any other. I’m quite enjoying it, especially since I make sure to keep my geeky side alive in this vast company. It’s all too easy to start talking of ‘low-hanging fruit’, and ‘piggy-backing’ and ‘all-hands meetings’ (this last one always makes me think of a room with a bunch of hands in it…), and so forth, but I’m resisting that. I do lots of interesting stuff, from Final Cut Pro to figuring out complicated schedules and crafting roadmaps(which is a fancy way to saying “figuring out what to do next”) and what not. Y’see, I’m what they call a Senior Program Manager, and Program Managing basically means getting things done. Doing it myself when I can, and when I can’t, there are people who help out. It’s a lot of fun, and I’m learning many new things, technical and otherwise, so I’m happy.

The second: the more personal side of matters. Now, long time readers of this blog (Hi, Mom), know that I do not speak about my life in great detail. Much to your disappointment, I’m not going to start now, so you can sit back in your chair and sigh. What I can say is that I started writing this journal in 2006 (2005, actually, but I wiped out my databases and learnt the value of backups..read the first post on the blog), when I was in my fourth year of law school. Many bits have flowed through the Intertubes since then, and now it’s 2012, and many things are new under the sun. There should soon be much to announce on the personal front. Read into that what you will.

I’ve not been home in almost two years now, and I miss it quite a bit. Some things I don’t miss (the incessant din of honking being right up there.. I’m not sure why we Indians believe in the physical properties of sound so much.. the horn is NOT going to magically vanish the car in front of you), but there are many things I do. We should hopefully be travelling (and no, the collective pronoun is not a typo) to India soon, to fly some kites, and eat some food, and meet that wild and wooly collection of _interesting_ people, otherwise known as the extended family.Can’t wait!

The cool thing about being in the US is the ability to access all the stuff that we didn’t used to get back in India. I was forever hanging out on Newegg, looking at the very latest GPUs and CPUs, dreaming of the day that I would be able to buy them. And now, I’m here, and I can easily afford to buy them. So what have I bought?

Nothing, of course. Now, don’t look smug. This is not because I suddenly don’t want them anymore. Far from it. I may be almost 30 now, but in very many ways I’m exactly like I was at 15. I still read Anandtech everyday, still go to Slashdot, Arstechnica, El Reg.. Heck, I’m even making simple PHP/MySQl websites just for kicks. And of course, I’m running multiple servers in the “cloud”, on Amazon Web Services, and on other sundry clouds. It’s just that the technology market seems to have stagnated for a while in terms of GPU power. I really think we need a new generation of display technology to use all the dormant power in today’s GPUs.. And I’m waiting for the new HiDPI Macbooks to buy my new shiny toy.

But this is a topic for another day. For now, let me stop and post this, and of course, let me wish my wonderful mother a very Happy Mother’s Day! She always let me do what I wanted, and a direct result of that is the servers running in the cloud; so Mom, Amazon thanks you as well

If the blog has been archived by Google Reader, it’s pretty easy to do. No-one seems to have written a guide to this online, so I thought I’d do it.

How-To:

1. Use this special URL:

http://www.google.com/reader/atom/feed/[RSS]?n=[N]

Replace [RSS] with the URL of the RSS feed, [N] with the number of posts wanted (max. 1000)

Example:

http://www.google.com/reader/atom/feed/http://adityanag.com/journal/feed/?r=n&n=1000

N=1000 tells the API to provide the previous 1000 posts.

2. The resulting page can then be saved as an XML document, prior to importing it into Excel.

3. If you want more than 1000 posts, you have to modify the URL somewhat. First, save the XML file from the previous step. Open it up in a text editor, and look for a tag that looks like this:

gr:continuation CO6a-c7Z_awC /gr:continuation

The tag is located right in the beginning of the file, before the content of the blog starts. If you can’t see this tag, that means that either there aren’t 1000+ posts, or Google doesn’t have an archive of the earlier ones. Google only archives posts if someone starts following it in Google Reader, or uses Feedburner to create a feed. Most popular blogs are fully archived, but smaller ones (like this one) only have a few hundred entries archived.

4. Once you have the Continuation String (the text inside the gr:continuation tag), you have to craft a different URL:

http://www.google.com/reader/atom/feed/[RSS]?n=[N]&c=[C]

Replace [RSS] with the URL of the RSS feed, [N] with the number of posts wanted (max. 1000), and [C] with the Continuation String.

5. Using this method, you can retrieve 1000 posts at a time, till you reach the end of the archive. Keep saving the files as XML’s and then you can combine them later in Excel to create one large file with all the data.

Note: Excel 2011 (on the Mac) won’t work here. It opens XML as a flat text file. Excel on Windows creates an XML schema, and this allows for easy manipulation of the data. Once the schema is created in Windows, the resulting file can be saved as an XLS and then opened in Excel 2011.

When I started writing here, I didn’t know what I would write about. Now, seven years later, I still don’t know! I’ve written about math and technology and sport and books and friends and family and dogs and bikes and cars.. And it’s always been crazy fun.

Early on, I decided that I wouldn’t be editing my posts. Sure, I’d correct egregious spelling mistakes, but I would not go over what I’d written, trying to fix it. Practically all my posts are written in one session; I don’t think I’ve ever used the drafts feature. While this does not make for the most compelling prose, it does keep me honest. I always looked at this blog as an odd mashup of a private journal and a public blog; I write about stuff that I care about, but not in too much detail, just in case.

And I’m glad I did. More than anything else, this blog has become a way for me to remember my own life. Even though I may have forgotten things, reading these posts brings it all back. Even the most innocuous post is full of meaning for me, sparking associations in my mind and reminding me of what I was thinking about when I wrote it. Through law school, through moving to Bangalore and working for PC World, through moving to the US… it’s been here, waiting for me to fill the white box. Some of my posts have been excellent, others not so much.

Onwards then, to seven, nay, seventy! more years.

My own definition of Big Data is quite simple. It’s the collection, capture, and analysis of billions of data points about any subject. Sure, databases have been doing this kind of thing for ages; the difference is that Big Data is very often unstructured, and we have tools and software (Hadoop etc) that allow us to analyse this huge mass of data and make sensible inferences.

I’ve been asked if there is a clear delineation between ‘Data’ and ‘Big Data’. The best way to answer that is to paraphrase Justice Potter Stewart from the US Supreme Court: “I shall not today attempt further to define the kinds of material I understand to be embraced . . . but I know it when I see it . . . “. He made this remark in a completely different context (trying to define obscenity in the case of Jacobellis v. Ohio, 378 U.S. 184, 197 (1964)) but it’s just as relevant.

There’s no clearly defined line past which data becomes Big; no place where we can say “below this limit it’s just data, and above that it’s Big Data”. You know it when you see it is the best answer there is, and I personally believe that today’s understanding of what makes Big Data Big will change over the years. When we all have access to Petabytes of storage, Big Data will mean something else entirely.

However, that’s neither here nor there. I’m more concerned about certain aspects of big data. A major concern is the drawing of incorrect inferences. When people have access to such large amounts of data, and the tools to slice and dice the data as they see fit, the temptation to confuse correlation and causation is almost irresistible. As an example: suppose we are able to correlate instances of diabetes with school dropout rates. If there is a strong positive correlation, we can then state that education leads to less diabetes.

You do see the problem, right? A correlation of this nature does not lead to an understanding of root causes. At best, it gives as an avenue of investigation, and smart researchers will know that and dig deeper. The problem becomes more pronounced when people start to blindly confuse correlation and causation. Can’t you just see a manager saying “the data shows that X was caused by Y, so we need to fix Y. We have xxMillion data points, and the data never lies.” Yes, data never lies, but numbers, by themselves, mean nothing (or very little).

It has been said that 86% of statistics are just made up (including this one), and students of stats and data analysis will know that increasing the size of the data pool does not automatically lead to better results.

I do think that Big Data has the potential to change many things, but I’m also sure that we need new skills to properly interpret what that data is telling us. The math boffins are already coming up with new ways to analyse the data, and the hugely buzzy field of data science and data scientists is just about taking off.

Does that mean we all have to be math whizzes or data scientists to effectively use big data? Absolutely not. We just need to be careful not to become too enamored by big data. Critical thinking, reasoning, and root cause analysis aren’t going anywhere, and when applied to big data, will lead to many new breakthroughs.

I just hope that those breakthroughs are not used solely by Facebook et. al. to show us more tailored ads.

Full Disclosure & Disclaimer- I work for EMC, but everything here is my personal opinion, and should not be construed as being a corporate message.

I worked for a big tech magazine for many years, and I found out first hand how some of this stuff works. I was lucky; we had a clear separation between Editorial and Marketing, but that didn’t stop the Country Manager of asking to meet me to “discuss the discrepancy between my review and our lab tests”. They claimed an output of 300 lumens, but my tests showed that it was only about 200 lumens. That’s a pretty big difference.

So the guy came down to our labs with one of his engineers, and we ran through all the tests again. The test methodolgy proved to be correct, and my numbers were accurate. The engineer and I figured out what was going on, but the manager was not happy. Later on, I learnt that there was talk of reducing the ad spends. They didn’t do that in the end, but it was an interesting time.

The point is, a company that’s paying a lot of money to a publication expects something in return. Nothing wrong with that, but where do you draw the line? What do you guys think? If you went to a blog, what would make you question its credibility?

What’s the best way to clearly show a sponsored relationship while maintaining editorial integrity?

Personally, I think that keeping it simple is best. Clearly identify sponsors, show that they do not have editorial control, talk about your sponsor as analytically and dispassionately as you speak about the competition. The sponsor has to understand that sponsorship does not equate to purchasing content. As long as both sides keep it clear, transparent, and honest, things should work out fine.

But that’s just my opinion. What do you think?

The idea came to me this afternoon, as I used my SecurID key fob to log into my corporate network. “Hmmm”, I thought. “My server is as valuable to me (more, probably) as this network to the company”. And so it began. I googled it at work, and then came home and set it up.

I looked at Duo Security, but eventually settled on Google authenticator. Duo Security looked very nice, but I just like the idea of having all my data on my own server.

And of course, to make things even better, I added CloudFlare!. So I have 2 Factor Authentication and the power of the Cloud. Life is cloudy and nice.. at least till the next 0-day exploit arrives.

The articles that I mentioned (they’re linked at the end), had some common themes:

1. Traditional network topologies worked on a north-south pattern; all data flowed up and down a tree like network, from edge to core and back again. Networks were designed to efficiently move data, and STP helped avoid loops and made sure stuff kept moving.

2. Applications were welded to the servers. A server got set up, connected to the network, and then largely forgotten about (till something went wrong, anyway).

3. Storage used to be server specific. Even after the advent of SANs, storage was still quite tightly linked to a particular physical location and a particular physical server.

All this hummed happliy along for quite a long time. Life was good, stock valuations were up, and people wore shorts to work on Fridays. Of course, like all idyllic times, it couldn’t last. Virtualization poked its head into the picture, and messed everything up. Suddenly data was flowing in every direction; servers were jumping around from machine to machine, applications were moving from server to server.. heck no one even knew where the damn data was stored anymore. The happy little switches weren’t happy no more, oh no.

Things had to change, and indeed they are. Network fabrics are flattening out, and networks are not designed to assume data flows in fixed directions. Networks these days have to be dynamic, adapting rapidly to the changing data patterns.

So what’s all this got to do with a corporation? Naturally, you’ve spotted the parallels as well.

1. Companies worked on a siloed north south pattern. Different departments communicated up the network and then back down again. Yes, there were east west links, since people make friends (I know, weird, eh?), but the official channels of communication were up and down, not side to side.

2. Business fuctions were welded to departments/divisions/functional practices/whatever-you-want-to-call-em. A division got set up, connected to the rest of the company, and then largely forgotten about (till revenues were down, anyway).

3. Information used to be division specific. Each division knew its own area very well, but had little knowledge of the other areas.

Now, of course, things are changing. Very few companies (especially in IT) sell only products. Everyone pretty much has to talk to everyone else to figure stuff out, and the traditional corporate model is finding it hard to cope. People find workarounds to get stuff done, and many companies are realizing that they have to change how information flows inside the company. Some are doing a good job, some are wondering where the good times went..

The reason for the similarities between networks and companies is pretty simple. After all, we’re talking about transferring information in both cases. The nature of information (the content, the scope, and the amount) changes constantly, and causes everything else to change. There’s nothing new under the sun; I bet this has been happening since the first human bashed an animal on the head with a rock and went “hmmmm…”. It’s powerful too.. political systems, religions, and entire countries have come into being based on the effective dispersal of information.

May you live in interesting times. I know it’s supposed to be a Chinese curse, but aren’t all times interesting times for the people who live through them?

http://etherealmind.com/layer-2-multipath-east-west-bandwidth-switch-designs/

http://www.theregister.co.uk/2011/09/16/flattening_ethernet_expert_panel/

http://blog.ioshints.info/2011/03/data-center-fabric-architectures.html

Reamde is more of a straight up thriller in the Tom Clancy style. Still good, but I miss the old Stephenson a little. A book where you have no idea what the author is saying but still can’t put it down is a Good Thing™.

I was just thinking about how I started reading Stephenson. It was a dark and stormy night a warm summers afternoon seven years ago. I was in New Delhi for the weekend, and looking for new books to read. At that time, I was reading a lot (and I mean a LOT) of fantasy and sci-fi. Stuff like the Wheel of Time, Ursula LeGuin, The Riddle Master’s Game, the Dark Elf stuff blah blah blah. These books were not exactly flying off the shelves, and finding them in my home town was an exercise in frustration. Delhi, though, being the capital of the country, was slightly better off.

So there I was, wandering about at PVR Saket (which is kinda like a cross between a mall and Quincy Market, for those of you who have no idea what I’m talking about). There’s a nice bookstore there (there used to be, at least), and I walked in and headed straight for the fantasy and sci-fi section. I do NOT like the term speculative fiction. That’s what I use when I read the newspaper these days. But I digress. Anyway, up in the corner, I spotted three gleaming books. Literally gleaming—the Baroque Cycle paperbacks were clad in shiney gold, silver and bronze covers, and they gleamed in a stray sunbeam.

I took them off the shelf and read the blurb at the back. ‘Hmmm,’ said I, ‘this looks interesting’. And so I bought all three. Quicksilver, Confusion, and the System of the World.

It took me a month to read them all. This is rather a long time, but good god! those books were complicated. Half the time I was struggling to grasp the cast of characters, and the other half I was cursing the author. But I couldn’t put the books down. They were a combination of the Pickwick Papers, Issac Asimov, P.G Wodehouse, Jerome. K. Jerome, Mark Twain, Dostoyevsky, Sartre, and many, many more. I loved it. Those were good times, those were.

I always love discovering a great new author. I’ve discovered some more since then, but that’s a story for another day. I’ve taken my twenty minute break and need to get back to work. If you’ve read, or are reading, Reamde, gentle reader, leave a comment.

I graduated with an MBA. And hey, I did pretty well, graduating in the top 5% of my class, with a 3.89 GPA. Not too shabby, eh? Apparently, I can study, occasionally.

I’m working for EMC now. Never heard of EMC? Well, that means you’re not the kind of person who finds the idea of visiting a data center cool. Jokes aside, EMC is a huge company that does storage, big data, virtualization (they own most of VMware) and a bunch of other stuff. What am I doing there? I’m working in Global Services Strategy and Marketing, doing a little bit of this, and a little bit of that. You know how it is.

Lots of other stuff too, but this will have to suffice for now. I’ll be posting more often now, keeping this blog going. You don’t have to read, but if you do, you’re just going to get cooler.

I kid, I kid…

Random strings of jumbleness
hidden away toys of summer
remember when 21 years was old

going out to the beach with salty salty sand
wondering what to do now

course, things are zipping along,
they always do

quiet and uptight, and quite uptight
let’s go to the top of the hub

feeling stupid about writing
writing stupid about feeling

streams, and counters, and deep sea bluefish
one stroke to the edge of the shelf;
the continental shelf.

warm blue waters with droopy headed beagles

it will all work out, now won’t it
it will
it will

will it?