Tuesday, January 17, 2017

Talend Project - Child jobs, Functions and Variables

This concluding part of Talend project video explores creating sub/child jobs, how to pass data between the two and different strategies to share the data.
Code in subroutines
// Code from https://www.youtube.com/watch?v=a7-HUU4js9E
package routines;

public class formatEmails {

    public static String formatEmail(char oldRating, char newRating, String typeOfRating) {
       String fEmailText="";
    if (newRating>oldRating) {
    fEmailText=fEmailText+""+typeOfRating +" rating reduced from " +Character.toString(oldRating) +" to "+Character.toString(newRating)  +"
" ;
    if (newRating
    fEmailText=fEmailText+""+typeOfRating +" rating improved from " +Character.toString(oldRating) +" to "+Character.toString(newRating)  +"
" ;
    return fEmailText;

It also explores subroutines/functions that helps you reduce the code and modularizes it. 

Check out the detailed video - https://www.youtube.com/watch?v=a7-HUU4js9E

tjavarow component code
/* -- Code from https://www.youtube.com/watch?v=a7-HUU4js9E ---- */ 
String wholepage;  
String ratings;
int pos=wholepage.indexOf("composite_val"); 
ratings=wholepage.substring(pos,pos+250).replaceAll("[\\[\\]\"]", "").replaceAll(" \n", " ").replaceAll(" composite_val_vgm","");    String allratingsonly="";
String[] splitratings = ratings.split("composite_val>"); 
int i=0;

Ratings for : " +context.stock +"
" ;

 for (String eachratingrow : splitratings) 
   if (eachratingrow.length()>0)  
   if (i==0){  
        context.EmailText=context.EmailText+formatEmails.formatEmail(input_row.growth_rating.charAt(0), eachratingrow.charAt(0), "Growth");
   if (i==1)
       context.EmailText=context.EmailText+formatEmails.formatEmail(input_row.momentum_rating.charAt(0), eachratingrow.charAt(0), "Momentum");  
   if (i==2)
   context.EmailText=context.EmailText+formatEmails.formatEmail(input_row.value_rating.charAt(0), eachratingrow.charAt(0), "Value"); 
   if (i==3)
    context.EmailText=context.EmailText+formatEmails.formatEmail(input_row.vgm_rating.charAt(0), eachratingrow.charAt(0), "VGM"); 
  /* - End of Code from https://www.youtube.com/watch?v=a7-HUU4js9E --*/

Thursday, January 05, 2017

Talend Project - Send mail (tSendMail component)

Talend's tSendMail component can be used to send HTML formatted emails. this Video demonstrates framing a HTML formatted text using the project we were working on.

This is the code that we used in tJavaRow component to extract the rating and create an email body that lists and changes compared to previous rating. Please follow the video get a complete understanding.

/* -- Code from https://www.youtube.com/channel/UCT3bqK2QL93j-IFYFYbvjWQ ---- */ String wholepage; String ratings; wholepage=input_row.document.toString(); int pos=wholepage.indexOf("composite_val"); ratings=wholepage.substring(pos,pos+250).replaceAll("[\\[\\]\"]", "").replaceAll(" \n", " ").replaceAll(" composite_val_vgm",""); //output_row.document = ratings; String allratingsonly=""; String[] splitratings = ratings.split("composite_val>"); int i=0;String EmailText="Ratings for : " +context.stock +"" ;for (String eachratingrow : splitratings)    if (eachratingrow.length()>0)   { allratingsonly=allratingsonly+";"+ eachratingrow.charAt(0)+"";     //allratingsonly=allratingsonly+eachratingrow+"**;";      if (i==0){  output_row.z_growth_rating=Character.toString(eachratingrow.charAt(0));  
  if ( eachratingrow.charAt(0)>input_row.growth_rating.charAt(0))  {   EmailText=EmailText+"Growth rating reduced from " +input_row.growth_rating+ " To "+ eachratingrow.charAt(0)+"
";   }
      if ( eachratingrow.charAt(0)    {   EmailText=EmailText+"Growth rating improved from " +input_row.growth_rating +" To "+ eachratingrow.charAt(0)+"
";      }  }    if (i==1){ output_row.z_momentum_rating=Character.toString(eachratingrow.charAt(0));   if ( eachratingrow.charAt(0)>input_row.momentum_rating.charAt(0))  {   EmailText=EmailText+"momentum rating reduced from " +input_row.momentum_rating+ " To "+ eachratingrow.charAt(0)+"
";   }
      if ( eachratingrow.charAt(0)    {   EmailText=EmailText+"momentum rating improved from " +input_row.momentum_rating +" To "+ eachratingrow.charAt(0)+"
";      }  }  if (i==2){ output_row.z_value_rating=Character.toString(eachratingrow.charAt(0));   if ( eachratingrow.charAt(0)>input_row.value_rating.charAt(0))  {   EmailText=EmailText+"value rating reduced from " +input_row.value_rating+ " To "+ eachratingrow.charAt(0)+"
";   }
      if ( eachratingrow.charAt(0)    {   EmailText=EmailText+"value rating improved from " +input_row.value_rating +" To "+ eachratingrow.charAt(0)+"
";       }  }     if (i==3){   output_row.z_vgm_rating=Character.toString(eachratingrow.charAt(0));   if ( eachratingrow.charAt(0)>input_row.vgm_rating.charAt(0))  {   EmailText=EmailText+"vgm rating reduced from " +input_row.vgm_rating+ " To "+ eachratingrow.charAt(0)+"
";   }
      if ( eachratingrow.charAt(0)    {   EmailText=EmailText+"vgm rating improved from " +input_row.vgm_rating +" To "+ eachratingrow.charAt(0)+"
";       }  }  i++;

output_row.EmailText=EmailText;/* - End of Code from https://www.youtube.com/channel/UCT3bqK2QL93j-IFYFYbvjWQ --*/

If you haven't visited this project from the beginning, Here is the first post about it in this blog - http://sanjaykattimani.blogspot.com/2016/12/talend-project-to-parse-webpage-zackscom.html

Saturday, December 17, 2016

Talend project to parse a webpage (Zacks.com)

Created another interesting Talend project over the weekend. This Talend job parses zacks.com webpage to extract zacks scores and then convert them to rows that can be used in other components. tHTMLParse compent use to parse the website is available in Talend's exchange (market place) for free. String manipulation consumed majority of my time on this project. I intend to extend this project in future

Here is the code that goes in tJavaRow component that extracts only ratings out of the whole page and returns a string of ratings separated by semicolon.

/* -- Code from https://www.youtube.com/channel/UCT3bqK2QL93j-IFYFYbvjWQ ---- */ 

String wholepage; 
String ratings; 
int pos=wholepage.indexOf("composite_val"); 
ratings=wholepage.substring(pos,pos+250).replaceAll("[\\[\\]\"]", "").replaceAll(" \n", " ").replaceAll(" composite_val_vgm",""); 
//output_row.document = ratings; 
String allratingsonly=""; 
String[] splitratings = ratings.split("composite_val>"); 
for (String eachratingrow : splitratings) 
   if (eachratingrow.length()>0)
   { allratingsonly=allratingsonly+";"+ eachratingrow.charAt(0)+"";     //allratingsonly=allratingsonly+eachratingrow+"**;"; 
/* - End of Code from https://www.youtube.com/channel/UCT3bqK2QL93j-IFYFYbvjWQ --*/

Monday, November 28, 2016

Creating MEAN stack application using Bitnami launchpad in AWS

My earlier Harvard course on Bigdata Analytics had given me enough exposure on AWS (Amazon Web Services) and professional and personal curiosity on emerging technologies taught me enough about MEAN stack.  MEAN stack is a full stack development tool kit that includes javascript based techologies MongoDB, Express.js, Angular.js and Node.js.

I wanted to take this learning to next level by hosting a MEAN stack application on AWS instance. While self exploring some intricate details of configuring and using AWS, much of the documentation was scattered to address individual needs and I had to connect the dots to meet my needs. Some of the documentation was old and some of the screenshots were obsolete. I decided to make a video along the way so that people who look for such information also get to learn. So here is the video of Creating MEAN stack application using Bitnami launchpad in AWS

I intend to do further videos explaining about other parts such as how to connect to the newly created VM, using Rockmongo, etc

Feel free to ask any related questions by leaving comments below this post or comments in youtube video page.

Thursday, March 20, 2014

SoapUI - A powerful web services testing tool

Recently I happened to Review/Test a few web services implemented for one of my client. 
SoapUI Pro

SoapUI is the defacto standard for validating webservices. Apart from technical deliverables, I also got to evaluate the product and compare it with free version. I downloaded free version and Pro (14 days evaluation) and was pleasantly surprised at the advancements and ease of use. I developed webservices a few years ago and hardly any useful tool was there to test it. 

SoapUI Pro interface 

Some of the notable features I liked and used are

  • Dynamic Duo/The Editors - You can visually enter data rather than searching though XML and entering the data. I used this extensively and is a very neat feature. 
  • Point and Click Testing - Almost ready to test the webservices just with few clicks. You can click on the web service response xml contents and right click and few more clicks and SoapUI writes the xpath qeury and associated test case. Its that simple. 
  • Test Debugging - Helps creating test flows, other testing related constructs like conditional execution, looping etc. I mostly utilized connecting to DB (data source) and groovy script.
  • Data Sources - Provides options to connect to most popular data sources. You can write queries/get the data from these sources and then use use the data as part of inputs to your webservice calls or compare the data to validate the webservice response.
  • Multi-Environment Support - If you have multiple environments on which you need to do same webservice testing, this is a major timesaver. I created my own config file and imported those settings as project's properties. Client code, end points. connection string, user name and such commonly used strings were stored and the imported properties could be used in test cases as variables with a few clicks(Again I did not type any of these).
  • Reporting - Provides nice reporting of all test suites/cases executed with their status. You can also customize the template of reporting. I mostly created HTML reports as they were very easy to use and store but PDF, CSV and other formats are also available. 
  • The SQL builder - Sql builder allows you to visually choose the tables and build query, our huge database with few thousand schemas was too much for SoapUI . I preferred using my ready to use queries and it worked flawlessly.
  • Stop the Hackers - XML Bomb, sql injection, malformed XML and plenty of other ready to use test cases are available under security category  - Just few clicks away.  

SoapUI had plenty of other features but these were the features that I ended up using and was pleased with the self explanatory and user friendly interface.
You can take a look at their SoapUI : 12 reasons to go pro page for more details.

SoapUI also has a free version. Free version is very limited but lets you create your test cases with basic features.

Saturday, May 18, 2013

Adobe LightRoom

Tried Adobe LightRoom software today on some of  my not so beautiful looking Raw images (of course bluebonnet flowers and kids are always beautiful)
What a difference. 
 I always wondered at those blank skies in my images and now i know how to deal with them. Pictures from Heavenly Bokeh group on Facebook and references from a few friends there inspired me to install and try out these effects. Burning midnight oil was well worth it. 

Tuesday, March 06, 2012

Youtube is scanning your videos

Yesterday I uploaded a video on Youtube that I created with photos and used some background music that i downloaded from net. In an hour, Youtube sent the following email to me.

Dear sanjaykattimani,
Your video, RishabHomeVideo.m2ts, may have content that is owned or licensed by Yash Raj Films Pvt. Ltd..  No action is required on your part; however, if you are interested in learning how this affects your video, please visit the Content ID Matches section of your account for more information.

- The YouTube Team
 © 2012 YouTube, LLC
901 Cherry Ave, San Bruno, CA 94066 

So whats Google/Youtube up to?
Youtube is scanning all the videos that you are getting uploaded and scanning for music that matches their video/audio library. In case it matches, sends out the above warning message. This is a smart way to reduce/fight copyright infringements.
May be this was part of Googles privacy policy changes. I didn't read them. Did you?

I feel this is the best for copyright owners as they are suppose to get their share. At the same time if they start going after users like us to collect royalty, we would rather drop those songs and use some general public licensed music. Or better, I may end up creating my own music?? :)

Update 1/22/2017 - Noticed that YouTube still lets you continue to upload your videos, It certainly does not let you monetize (Earn on the ads) your videos. Its not hard to find royalty free music for most of your compositions though.