Month: March 2015

More Scripting

Recently, I tried to write a script to parse the output of the log file created by Cloudera Manager. Unfortunately, I had to abandon the script after a week and and quite a number of hours invested. Parsing text is hard. It is not that CM created bad log entries or anything like that. It is just what you might expect in a log, might now be there to make the decisions you want. That was the case for me, the answer I wanted couldn’t be derived easily from the logs.

Fast forward a few days later. My buddy calls me and ask me about scraping website for product info. My first question was, “is this the only option”. If we can engage with the website owner, we can get them to give us access to the products and related tables. Especially, since the project was contracted by the owner of the website.

He said consider, that this is our only option. That getting access to the actual data would mean talking to different teams, and the teams haven’t been responsive so far. I laid out for him, the details of why getting data from the primary source is preferable than trying to parse web pages.

When people look at a web page, they see structure and something that is easy to digest. But they are not seeing what the code looks like. A web page is coded so a browser can render it for the consumption by human, not by machine or another programming language. When developers want data to be consumed by machine or another programming language, it is much more efficient to use binary or some other format. Even if you want it to be human ready. Even JSON, XML, or CSV is not meant to be the presentation format for humans, it just means that it is more open.

What he was asking, wasn’t about even some text interchange format like JSON/XML, etc. but parsing a number of pages for product details. That would be such a nightmare of a project. So here to hoping that we really don’t have to do that.

Looking Up

I really don’t have anything great happened between yesterday and today. But yet, I feel great. May be it is because I laughed so much today.

My colleague was telling me about his life, and I am not going to repeat. But as he talk I was just laughing. It wasn’t anything particularly funny or terrible either. It was just about life and things that could happened to any of us.

Oh well. Here is to keep smiling.

In a Rut

I feel like I am stuck in a rut. I won’t get into details to possibly bore my other self who will read this one day. But right now, things could be very stressful and you didn’t know how to laugh.

Trying to get a house in NC. Dealing with the builder haven’t been the hard part. Getting the loan has been trying, to put it mildly. With my income, credit history, and never had a late payment on my rent, etc. I am almost the perfect person for credit. Taking into account that moving from Cali to NC would reduce just about everything we have to pay for. So again, we will have even more cash available.

Why is the loan process not being managed by humans? Well, it doesn’t feel like it is humans who are making the decisions. We are having so many issues to get a loan. I feel like I am stuck, in a rut so to speak.

Awk Script Dead End

Almost a week to the day, I had to admit to myself that my Awk script won’t take me where I wanted. I had worked on it on an over the week, may be a total of forty hours. I spend just over four hours today before I realized what should have been obvious a few days or hours into the project.

I had a question to answer, and the answer seems to lie in one or more log files. These log file can run into the hundred of thousands of line or even one million. So how to have my question answered without having to looking with my Vim editor and a number of repeated regular expressions?

Enter the idea of a script. A script that would look for a few tell tail patterns. At the end of which, I can say these things happened so this must be the conclusion. Or these things didn’t happen, so something else. It was suppose to be that simple.

I started with a shell script. Found that that with the number of lines I had to read, I would essentially need some way to loop over the files and etc line. Why do that when Sed and Awk already does that. Fortunately, Awk is a lot more flexible than Sed. I guess I could have written it in Perl too. I wanted something powerful, but didn’t want to go down the road of Python or Groovy.

Perl, I dismissed because I had written a lot of Perl code years ago and didn’t want to go back there. It was fun then, but I haven’t had use for Perl since and didn’t want to revisit using it.

I also wanted something I can give to another person to user without much headache. So while most systems have Python, I am not as deft in Python, so it would have taken a bit more time. Oh, I didn’t have much time either. So Awk was going to be it, after I didn’t get more than fifteen minutes into writing in Bash.

It turned out that Awk wasn’t the problem. It was just processing a lot of text and what you might expect to find, may not be there. Since your script is not as flexible as a full fledge language might give you. You really are just looking for a bunch of patterns and reacting. So when something isn’t there that you expect. Well…it all goes South very quickly.

Some of my early testing showed a particular setting of lines. I deduced that if I found those, then I can draw a conclusion or not. When I tested on a later data set. Many of the patterns I had coded for, weren’t even there. Plus my design hinged on finding a start pattern as a marker between distinct events. That was nowhere to be found in some actual logs. Even though, they were there in the test logs I used.

And that is what should have been clear to me early on. That log files don’t always contain what you are looking or what you want. And even when something happened and you thing that everything will be logged so you can go back and tell a story. Sometimes, you just can’t. So now I will just have to go back to using Vim, tail, head, grep/ack, less, and friends. Because writing script to part logs is a non-trivial task, especially in limited scripting language.

A Taste of C

The C programming language was one of the first computer language I learnt. As a matter of fact, it was precisely the send language. My high school offered Pascal programming, and I was taking that. When my my physics teacher at the time, suggested my friend and I learn C. Our teacher was doing his master at the time and said that C is more useful and the future. I am glad I did.

C programming opened a lot of doors for me. Even though I was an electrical engineering student, I still programmed a lot in C. Then I graduated from collected and my first job was as a software engineer. C was a large part of most of the things I did in those early eight years.

I eventually added other languages too, C++, Assembly, Perl, shell scripting, Tcl/tk, Basic and others I don’t remember at different times. C was my constant companion. It has been years now since I wrote any C code. Going on to a solid 3 years without any need to write C. The desire was there, but no work or project.

Until today. A buddy called me and asked if I would explain how functions worked to his daughter. She was in college and taking C. Well, that is surprising, because I thought they stopped teaching C in colleges. Opting mostly for Java and Python. Well, I slipped into coding again like wearing your old favorite pants.

I loved every minute of it. I was having so much fun. That I think I started helping too much on the assign. I eventually got kicked off when my buddy told me that he only wanted me to work on one, and that go out anyway. 🙁 I was having so much fun.

Don’t Know

My company is growing quickly and looking for great talent to do so. I have been helping out with interviews. Two things stood out recently that I don’t like. The two things stems from one thing. Not knowing.

Don’t Know – Looking It Up

I have people who didn’t know how to answer questions on the phone interview, look it up or tried to at lease. I can hear them in the background furiously typing as they either silently stalled for words or use fillers as they search. If a person knew something, but forgot, it is very clear from their response. Versus someone who doesn’t know, but is trying to look it up. When they find what they are looking for, they try to regurgitate what they just, and it doesn’t make sense to them nor you. And they soon figure that out and just give up.

Don’t Know – Let Me Interrupt

The other kind of person is the most annoying to me. The first type try to trick you, but if they can’t figure out or find he answer, they will eventually come clean. The one tells you up front they don’t know. But soon as you start explaining. Suddenly they know, and must interrupt you. Of course, they still don’t know, but insist on interrupting. Then goes of an tangent. Instead of staying quiet to hear what you have to say and then ask questions or make suggestions as in any normal conversation. They don’t do that. They feel the need to jump ahead of your conclusion. Always interrupting to finish your sentences.

I just don’t get either of these people. If you don’t know, you don’t know. No one knows everything. I guess it irks me because I don’t know why it isn’t apparent that they are not fooling anyone.

Cusp : Media Wars (02)

The growing rift between Puritans and Entenders didn’t stop on the street. It was in the media. Which each side having their set of supports. New outlets didn’t pretend to be unbiased any more, but planted their feet squarely on one side or the other. Some might argue it was the baiting of media and their talking head hosts and guests that whipped up the population.

It was the media that coined the terms each used today, Cyborgs or Anderthals.

Cusp : In Town – 01

Mark picked his way along a side street. He hated coming down to the city these days. It seems like every time he came, there were more new shops for Cyborgs. He mostly walked with his head down. Not wanted to see the stores or the people with who showed off their flavor of augmentation.

As he approached the corner, he heard the distinctive buzz and whirr of mechanical legs from the cross street. His anxiety raised with each step. He slowed his paced, like was he expected someone to jump out at him. As if around that corner lied in wait an hungry lion. He stopped just before the corner, and listened, as the buzz and whirr got louder.

Mark waited tensely. Soon a woman turned the corner and walked passed him. She greeted hime warmly. But he didn’t make eye contact and grumbled “freak”. In return she calmly said “idiot”. She must have been used to these kinds of insults.

His thoughts returned to how much he hated this place now. What it was becoming. How the Puritans were being pushed out, they were losing more and more of what they loved. This was is home. He lived with his mom in a small two-bedroom house on the edge of town. He used to love coming down here to meet here.

Cusp: The Dawn of the Human++ – The Pitch

Introduction

Being enhanced was nothing new. Depending on how you looked at it, humans have been enhancing themselves and their environment the moment they used a tool. While enhancements to the human body was nothing new, there were still people who won’t accept this as the norm.

The people who happily embraced the augmentation of the human body, were called Extenders. They saw it as just the next natural thing. Keeping in the line of corrections and fixing of the human when or where nature had failed. Just as glasses are used to asses the far or near sighted, so too were eye implants or stem cell sight correction used to give sight to the blind.

Extender argue, that people have been using shoes for centuries to protect their feet. Humans weren’t born with shoe on, so why stop there. What was wrong with having an artificial heart that lasted longer, beat stronger, never caused its owner an heart attack, they asked?

While all of this is happen and everyone saw the changes coming. Some people are just resistant to the idea of humans being fixed or augmented. Even for people who were in an accident or who were born physically handicap or with medical issues that could be corrected with augmentation. These people call themselves Puritans.

Puritans had not religious affiliation. They just felt very strongly that medicine was fine to help make humans healthier or to heal them. But there shouldn’t be any augmentation inside of or on the body. So no artificial lung if you had a bad lung, or need one after a crash. No prosthesis if you lost an arm or was born with a deformed leg. Corrective lens were ok, but don’t get an artificial eye that could see in the dark or infrared.

Over time, a rift developed, between Puritans and Extenders. The rift started long before each side would ascribe labels to themselves or each other. Long before the name calling and hate crimes. Long before laws had to be made to protect the rights of one side from the other. As more people extended, the more extreme the Puritans became.

Puritans saw Externders as nothing more than cyborgs. Cyborgs just happened to have a few human part, it didn’t matter it which was more. Whether there was more human than machine in a body. The same any form of internal or external extension as cyborgist.

Extenders too, had a name for Puritans. Extenders saw Puritans as Simpletons, uninformed and unevolved. Extenders argue that it was the goal of humans to be the best there is, and using technology to aid us in lifting a rock to crushing nuts. Was all the same thing, and today we take humans to the next level by bring the technology closer to the human, by adding to the human.

Mark was a typical Puritan. He was the shinning example of a Puritans. He hated everything about the idea that one would even want something the was artificial as part of them. Something you couldn’t part with, you couldn’t take off as easily as you shed your cloths, shoes, or glasses. Because, having extension and augmentations like prosthesis was more. They need permanent anchors on your body, protrusions to a fix to. It was simply not the same. Even hearing aids of the 21st century, didn’t need anything that you weren’t already born with. Same with glasses, you just put them on your nose, not drill a hook into the sides of your head.

Of course there is going to be hell to pay when Mark woke up and found that he had been fixed to save it life. He had an awful accident, an accident he doesn’t remember. He had always thought that if it came to saving his life thought augmentation, he would rather die a human than live as a twisted version of what nature intended.