Data format conversion in java

Basic need is we used to get xml date data type in different formats for data sources. However our federated search engine we are storing in a neutral format. (For simplicity sake.) After analyzing couple of formats, added this code in year 2005. Nowadays Team is using some apache package utils. But I don’t think they will something different. I am not seeing many options.

public static final String[] date_format_list = {
     "yyyy-MM-dd'T'HH:mm:ss'Z'",
     "yyyy-MM-dd'T'HH:mm:ss",
     "yyyy-MM-dd",
     "yyyy-MM-dd hh:mm:ss",
     "yyyy-MM-dd HH:mm:ss",
     "EEE MMM d hh:mm:ss z yyyy"
     /// add your own format here
   };

   public static Date parseDate(String d) throws ParseException {
     return parseInputWithFormats(d, date_format_list);
   }

   public static Date parseInputWithFormats(
           String dateValue,
           String[] formatList
   ) throws ParseException {
     if (dateValue == null || formatList == null || formatList.length == 0) {
       throw new IllegalArgumentException("dateValue is null");
     }
 
     if (dateValue.length() > 1
             && dateValue.startsWith("'")
             && dateValue.endsWith("'")
             ) {
       dateValue = dateValue.substring(1, dateValue.length() - 1);
     }

     SimpleDateFormat dateParser = null;
        for(int i=0;i < formatList.length;i++){
       String format = (String) formatList[i];
       if (dateParser == null) {
         dateParser = new SimpleDateFormat(format, Locale.US);
       } else {
         dateParser.applyPattern(format);
       }
       try {
         return dateParser.parse(dateValue);
       } catch (ParseException pe) {
        //pe.printStackTrace();
       }
     }
     throw new ParseException("Unable to parse the input date " + dateValue, 0);
   }

public static void main(String[] args) {
  String fromDt="";
  String nPattern = "yyyy-MM-dd'T'HH:mm:ss'Z'";
  SimpleDateFormat sdf = new SimpleDateFormat(nPattern);
  
  String currentValue="Fri Jul 22 04:22:14 CEST 2011";
  
   try{  
             fromDt = sdf.format(parseDate(currentValue.toString() )  );
      } catch (Exception e) {
       System.out.print("\n Case1: date format exception"+e.getMessage()+ " SOLR currentValue:"+currentValue);
       fromDt="";
      } 
      System.out.println("Case1. date as str---"+fromDt);
      
      currentValue="2011-07-21 21:22:14";
       try{
              fromDt = sdf.format(parseDate(currentValue.toString() )  );
       } catch (Exception e) {
        System.out.print("\n Case2: date format exception"+e.getMessage()+ " SOLR currentValue:"+currentValue);
        fromDt="";
       } 
      System.out.println("\n Cse2.  date as str---"+fromDt);
  }

Latest family picicture from Disney world.

This is from latest Disney world. (Photo from Disney pass port photo service. We took good number of phones with Disney theme characters. Photos are not brilliant by professional photographer standards. We must not blame them. They are shooting nonstop. However they are not bad too.)

Parsing complex Xsd file with Java code.

This one nearly 4+ year old code. Now Team is using apache xerces XSModel.
However this code is there for sanity check. Still this works fine.
(You need xsom.jar + relaxngDatatype.jar file. Google it. You will find the jars.)

public class XsdReader {
 public static void main (String args[]) 
 {
  XsdReader rr = new XsdReader();
  rr.parseSchema();
    }
 
 public void parseSchema()
 {  
  File file = new File("D:\\tmp\\books.xsd");
  try {
      XSOMParser parser = new XSOMParser();
      parser.parse(file);
      XSSchemaSet sset = parser.getResult();
      XSSchema mys = sset.getSchema(1);
      Iterator itr = sset.iterateSchema();
      while( itr.hasNext() ) {
        XSSchema s = (XSSchema)itr.next();
        System.out.println("Target namespace: "+s.getTargetNamespace());
        XSComplexType ct = mys.getComplexType("books");
        int ctr=0;
        if ( ct != null){
        Collection c = ct.getAttributeUses();
        Iterator i = c.iterator();while(i.hasNext()){
            XSAttributeDecl attributeDecl = i.next().getDecl(); 
           
            System.out.print("ctr="+ctr++ +"name:"+ attributeDecl.getName());
            System.out.print("    type: "+attributeDecl.getType());
            System.out.println("");
          }
        }
        Iterator jtr = s.iterateElementDecls();
        while( jtr.hasNext() ) {
          XSElementDecl e = (XSElementDecl)jtr.next();
          
          System.out.print( e.getName() );
          if( e.isAbstract() )
            System.out.print(" (abstract)");
          System.out.println();
        }
      }
    }
    catch (Exception exp) {
      exp.printStackTrace(System.out);
    }
  }
}

Lucene Standard Analyzer vs. Lingpipe EnglishStop Tokenizer Analyzer

This is old code from prototypes directory.
For some odd reason, I end up prototyping different analyzers for PLM space content vs 3ed party analyzers. (Basic need is which got better control on STOP words. At least based on my quick proto type, SOLR got easy constructs.)

Small sample code comparing both analyzers is included.
I did not see much difference for small input text.

public class AnalyzerTest {
 
  private static Analyzer analyzer;
   private static long perfTime = 0;

   public static void main(String[] args) {
     try {
 
         analyzer = new StandardAnalyzer(org.apache.lucene.util.Version.LUCENE_34);

         String str = "PLM technology refers to the group of software applications that create and manage the data that define a product and the process for building the product. Beyond just technology, PLM is a discipline that defines best practices for product definition, configuration management, change control, design release, and many other product and process-related procedures.";

         perfTime -= System.currentTimeMillis();
         displayTokensWithLuceneAnalyzer(analyzer, str);
         perfTime += System.currentTimeMillis();
         
         System.out.println("Lucene Analyzer: " + perfTime + " msecs.");
         
         perfTime -= System.currentTimeMillis();
         displayTokensWithLingpipeAnalyzer(str);
         perfTime += System.currentTimeMillis();
         
         System.out.println("Lingpipe Analyzer: " + perfTime + " msecs.");
         
   } catch (IOException ie) {
         System.out.println("IO Error " + ie.getMessage());
       }
       System.out.println("Time: " + perfTime + " msecs.");
       System.out.println("Ended");
        }

    private static void displayTokensWithLingpipeAnalyzer(String text)
       throws IOException {
     
     System.out.println("Inside LingpipeAnalyzer ");
     
         TokenizerFactory ieFactory  
            = IndoEuropeanTokenizerFactory.INSTANCE;
         
          TokenizerFactory factory
                = new EnglishStopTokenizerFactory(ieFactory);
               // = new IndoEuropeanTokenizerFactory();
        
     char[] cs =text.toCharArray();
     Tokenizer tokenizer = factory.tokenizer(cs, 0, cs.length);
     String[] tokens = tokenizer.tokenize();
     for (int i = 0; i < tokens.length; i++)
        System.out.println(tokens[i]);
     
     System.out.println("Total no. of Tokens: " +tokens.length );
     
    }
   private static void displayTokensWithLuceneAnalyzer(Analyzer analyzer, String text)
      throws IOException {
    System.out.println("Inside LuceneAnalyzer ");
        TokenStream tokenStream = analyzer.tokenStream("contents",new StringReader(text) );
    OffsetAttribute offsetAttribute = tokenStream.getAttribute(OffsetAttribute.class);
    CharTermAttribute charTermAttribute = tokenStream.getAttribute(CharTermAttribute.class);
       int length=0;
       
    while (tokenStream.incrementToken()) {
        int startOffset = offsetAttribute.startOffset();
        int endOffset = offsetAttribute.endOffset();
        String term = charTermAttribute.toString();
        System.out.println("term->"+term+ " start:"+startOffset+" end:"+endOffset);
        length++;
    }
       System.out.println("Total no. of Tokens: " + length);
  }
   
}

Java Set Operations

As you know, for a while I’ve been playing with Solr. Basically implementing federated search framework to aggregate different ERP, CRM, PLM space data sources.
This post is not about federated search but I keep using bunch of set operations to compare the search results, comparing the unique doc id etc.

public class SetOperations {
 public static  Set union(Set setA, Set setB) {
     Set tmp = new TreeSet(setA);
     tmp.addAll(setB);
     return tmp;
   }

   public static  Set intersection(Set setA, Set setB) {
     Set tmp = new TreeSet();
     for (T x : setA)
       if (setB.contains(x))
         tmp.add(x);
     return tmp;
   }

   public static  Set difference(Set setA, Set setB) {
     Set tmp = new TreeSet(setA);
     tmp.removeAll(setB);
     return tmp;
   }
   
   public static void main(String[] args) {
         
        SortedSet s1=  new TreeSet();
        s1.add("one");
        s1.add("two");

        SortedSet s2=  new TreeSet();
        s2.add("two");
        s2.add("three");
        s2.add("four");
         
        SortedSet result = (SortedSet) union(s1,s2);
        
        Iterator it = result.iterator();
        System.out.print("union result  -->");
         while (it.hasNext()) {
             String value = it.next();
             System.out.print(value+", ");
         }
         System.out.println("\n");
         
         result = (SortedSet) intersection(s1,s2);
         
      it = result.iterator();
      System.out.print("intersection result-->");
      while (it.hasNext()) {
              String value = it.next();
              System.out.print(value+ ", ");
      }
      System.out.println("\n");

         result = (SortedSet) difference(s1,s2);
         
      it = result.iterator();
      System.out.print("difference result-->");
      while (it.hasNext()) {
              String value = it.next();
              System.out.print(value+", ");
      }
      System.out.println("\n");
      
         /*
         SortedSet i1=  new TreeSet();
         i1.add(new Integer("1"));
         
         SortedSet i2=  new TreeSet();
         i2.add(new Integer("2"));
         
         SortedSet iresult = (SortedSet) union(i1,i2);
         
         Iterator iit = iresult.iterator();
         System.out.println("Integer union result");
          while (iit.hasNext()) {
           Integer value = iit.next();
              System.out.println(value+",");
          }
         */
   }
   
}

XSL recursion sample

In my use case, I am receiving part numbers as a big string and all are separated by comma(,), I need to split them in to array of elements.
Here is the sample code.

<xsl:template name="printChildObjects">
  <xsl:param name="inputString"/>
  <xsl:param name="delimiter"/>

  <xsl:choose>
    <xsl:when test="contains($inputString, $delimiter)">
    <xsl:variable name="aChild">
    
    <xsl:value-of select="substring-before($inputString,$delimiter)"/>
    </xsl:variable>
    <xsl:element name="field">
    <xsl:attribute name="name">
    <xsl:text> childObject</xsl:text>
     </xsl:attribute> 
     <xsl:value-of select="$aChild" /> 
     </xsl:element>
        <xsl:call-template name="printChildObjects">
           <xsl:with-param name="inputString" select="substring-after($inputString,$delimiter)"/>
           <xsl:with-param name="delimiter"
           
          select="$delimiter"/> 
            </xsl:call-template>
            </xsl:when> 
             <xsl:otherwise> 
             <xsl:choose> 
                <xsl:when test="$inputString != ''"> 
                <xsl:element name="field"> 
                <xsl:attribute name="name"> 
                <xsl:text> childObject</xsl:text> 
                     </xsl:attribute> 
                     <xsl:value-of select="$inputString" />    </xsl:element> 
             </xsl:when> 
             <xsl:otherwise>
         </xsl:otherwise> 
        </xsl:choose> 
      </xsl:otherwise> 
   </xsl:choose>
</xsl:template>

xsl:text must not contain child elements

Some info from old xsl cheetsheet.

Often there is a requirement to print a variable value in output.
Following xsl code will display compilation errors.
Simple workaround is pass it to a template

<xsl:text>

<xsl:value-of select="$objType"/> </xsl:text>

<!-- <xsl:call-template name="varValue"> <xsl:with-param name="value" select="$objType" /> </xsl:call-template> -->

<xsl:template name="varValue">

<xsl:param name="value" />

<xsl:choose> <xsl:when test="$value = ''"> <xsl:text>n/a</xsl:text> </xsl:when> <xsl:otherwise> <xsl:value-of select="$value" /> </xsl:otherwise> </xsl:choose> </xsl:template>

Typical industry search input requirements/ patterns

An object string property contains all possible values.
Now I want to filter the list with different kind of input patterns (aka single field search) In particular accepting wildcards (*,? & + etc escaping is fun in java.)
sample code:

String str="*+PATAC+*";
Pattern pat=Pattern.compile(".*\\+*\\+.*");

Matcher matcher=pat.matcher(str);
boolean flag=matcher.find(); // true;

Logger.println("1) matcher result->"+flag);
if ( flag == true)
Logger.println("pattern found"+str);

str = "adjkfh+PATAC+ajdskfhhk";

matcher=pat.matcher(str);
flag=matcher.find(); // true;

Logger.println("2) matcher result->"+flag);
if ( flag == true)
Logger.println("pattern found"+str);


str = "PATAC";

matcher=pat.matcher(str);
flag=matcher.find(); // true;

Logger.println("3) matcher result->"+flag);
if ( flag == true)
Logger.println("pattern found"+str);

str = "adjkfh+PATAC+";

matcher=pat.matcher(str);
flag=matcher.find(); // true;

Logger.println("4) matcher result->"+flag);
if ( flag == true)
Logger.println("pattern found"+str);

str = "+PATAC+testingsuffixchars";

matcher=pat.matcher(str);
flag=matcher.find(); // true;

Logger.println("5) matcher result->"+flag);
if ( flag == true)
Logger.println("pattern found"+str);

Sample code to create SOLR document from CSV file

Just one guy stopped by office & asked to index this legacy data. So quick & dirty solution is the following. (May be Perl etc are there, but this one gives me more flexibility.

public class CsvToSolrDoc
{
public String columnName(int i)
{
//workarounds workarounds
if ( i == 0) return "id";
if ( i == 1) return "what ever you want as field name";
return null;
}
public void csvToXML(String inputFile, String outputFile) throws java.io.FileNotFoundException, java.io.IOException
{
BufferedReader br = new BufferedReader(new FileReader(inputFile));
StreamTokenizer st = new StreamTokenizer(br);
String line = null;
FileWriter fw = new FileWriter(outputFile);
// Write the XML declaration and the root element
fw.write("\n");
fw.write("\n");
while ((line = br.readLine()) != null)
{
String[] values = line.split(",");
fw.write(" \n");
int i = 1;
for ( int j=0;j it is length; J++)
{
String colName = "field name=\""+columnName(j)+"\"";
fw.write("<" + colName + ">");
fw.write(values[j].trim());
fw.write( "\n");
}
fw.write(" \n");
}
// Now we're at the end of the file, so close the XML document,
// flush the buffer to disk, and close the newly-created file.
fw.write("\n");
fw.flush();
fw.close();
}

public static void main(String argv[]) throws java.io.IOException
{
CsvToSolrDoc cp = new CsvToSolrDoc();
cp.csvToXML("c:\\tmp\\m2.csv", "c:\\tmp\\m2.xml");
}

SOLR project stories. Lack of SOLR post filter support

For past few quarters, I am working on a project to implement security on object documents. My goals is Decorate every SOLR document with ACL field. This ACL field used to determine what users have access to this document. ACL syntax is something like +u (dave)-g(support) etc. My thoughts are process these ACL fields after search aka I want to subject the query component results via some kind of post filter. However SOLR is not offering any direct mechanism to specify this kind of post filter along with the search request. At Lucene level, there is an option to specify the filter however current AS IS implementation, it sucks.
It iterates entire document sets. For large documents this sucks. Also we need to SOLR distributed capabilities. Also during computing the ACL fields, I tried to encode users names etc with Base64, URLEncoder.encode etc. For small set of strings, this is working Ok. But for large sets, it is a pain. Ultimately affecting the search performance.
Another blocker.
Encode/decoder tes code.
startTime = System.currentTimeMillis();
String inputText = "Hello#$#%^#^&world";
for (int i =0;i<50000;i++)
{
String baseString = i+ " "+inputText;
encodedText = URLEncoder.encode(baseString,"UTF-8");
decodedText = URLDecoder.decode(encodedText, "UTF-8");
}
endTime = System.currentTimeMillis();

elapsedTime = endTime - startTime;

System.out.println( "\n URLEncoder/decoder Elapsed Time = " + elapsedTime + "ms");

>>>>
Elapsed Time = 2246ms

Few Inspiring Quotes from The Great Ones The Transformative Power of a Mentor

Adapt the pace of nature: her secret is patience – Emerson

Watch your thoughts; they become words
Watch you words; they become actions
Watch your actions; they become habits
Watch you habits; they become character
Watch your character; it becomes your destiny. – Unknown

I hope I shall always posse’s firmness and virtue enough to maintain what I consider the most enviable of all titles, the character of an Honest man – George Washington

The person who makes a success of living is the one who see his goal steadily and aims for it unsercingly. That is dedication – DeMille

What we say and what we do
Ultimately comes back to us
So let us own our responsibility
Place it our hands And carry it with dignity and strength -Anzaldua

Simple SOLR example code to index & search metadata

Roughly three years back, I did some a prototype to index/search unstructured content inside enterprise. At that time we are using Autonomy. So nobody cared. Basically all search engines does some kind of structured content searches & major focus on unstructured content i.e. PDF files to WS word document etc. So coming to the point, again I was asked to some prototyping. This time I am looking to solr framework because my C++ business tire makes requests to index/searches for content (mostly un structured content.) after downloading Solr, looking for some good examples. At the end, I started writing the following. I will port this kind of example with using CURL.

  protected URL solrUrl;
  public CommonsHttpSolrServer solr = null;

  public SolrServer startSolrServer() throws MalformedURLException,
  SolrServerException 
  {
     solr = new CommonsHttpSolrServer("http://localhost:8983/solr/"); 
     solr.setParser(new XMLResponseParser()); 

     return solr;
  }
   
   public TestQueryTool()
   {
          //will add more junk later
   }

 
   public static void main(String[] args)
   {
     
     TestQueryTool t = new TestQueryTool();   
     try
     {
            //1)start & work with existing SOLR instance
   t.startSolrServer();
 
   // 2)Now index content ( files later. metadata)
   t.indexBOAsDocument("uid1","bolt", "0001");
   t.indexBOAsDocument("uid2","nut", "0002");
   
   //3)now perform search
   t.performSearch("uid");

  } catch (MalformedURLException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } catch (SolrServerException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
     

 }

  
 private void indexBOAsDocument(String uid, String id, String name)throws SolrServerException 
 {
  SolrInputDocument doc = new SolrInputDocument();
  
  doc.addField("id",  uid);
  doc.addField("part_name",  name);
  doc.addField("part_id", id);

  try {
   solr.add(doc);
   solr.commit();
  } catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
 }

 
 public void performSearch(String qstring) throws SolrServerException 
 {
  SolrQuery query = new SolrQuery();
  //query.setQuery( "*:*" );
  
  query.setQuery("id:uid* OR part_name:nut");
  
        query.addSortField( "part_name", SolrQuery.ORDER.desc);
  QueryResponse response = solr.query(query);
  SolrDocumentList docs = response.getResults();
  info("docs size is ="+docs.size());
  Iterator iter =docs.iterator();

     while (iter.hasNext()) {
       SolrDocument resultDoc = iter.next();

       String part_name = (String) resultDoc.getFieldValue("part_name");
       String id = (String) resultDoc.getFieldValue("id"); //id is the uniqueKey field
        // Now you got id, part_name. End of story
   }
       
      }

Nice little book "The Great Ones" by Ridgely Goldsborough

Books come with two different stores. One side authors’ journey before, after a mentor & his own life changes and simple set of guide lines & the other side a simple fictional story. Through the tales, the author suggests importance of mentoring, benefits in ones life virtually every aspect of life. Many inspiring quotes, compact size & fictional story also effective one.

Following are simple code of conduct

1.Make a decision & commitment
2.Conceive and execute a plan
3.Take full responsibility
4.Embrace patience and temperance
5.Act with courage
6.Cultivate passion
7.Exercise discipline
8.Remain single minded
9.Demand integrity
10.Let go of past failure. ( mostly learn from them)
11.Pay the price

Interesting free book “Best Kept Secrets of Peer Code Review”

http://smartbear.com/codecollab-code-review-book.php

Order your FREE book with FREE shipping.
Also you can view sample chapters. Some of the point are interesting.
if you are doing code reviews or senior developer, take some time to read it.

I ordered one & after receiving the book, I will update this blog.

Dhanvi & Saketh still from 2009 India trip

Both Dhanvi & saketh likes the Mango a lot. However this is the first time, we took them to our own garden. Before going the garden, Dhanvi was very curious about the Mango tree & he says, will count all the mangos. At the end, after seeing the garden, he was tired in counting the number of trees itself & thrilled to see the groups of mangos handing at so low to the tree. ( see the back ground)

Long list of books waiting to be read - 1

Book title: What color is your parachute? 2010 by Richard N. Bolles

Nearly decade back, I read this book.
Now I picked the latest 2010 (Hard Times) edition.
During my Boston to Los Angeles transition, I looked at this book for some guidance.
Similar to old editions, book sticks to core ideas. However presentation & reference material was changed a lot. I am totally surprised to the information like 100 of job listing sites, starting home business etc. To some extent I am not following the latest trends. I have a job which I love & I do with tons of commitment. Because of this reason, last ten years, I never entertained any carrier change. So I am not aware of changing trends in Job markets. It does not matter if you are looking for a job or not; still it is worth reading.
Topic “Thing schools never taught us about job hunting” is always my favorite.

An interesting quote from the book
He or she who gets hired is not necessarily the one who can do the job the best; but, the one who knows the most about to get hired. – Richard Lathrop

Interesting little book “ReWealth”

Book: ReWealthby Storm Cunningham (Author)
Published by McGraw-Hill

Last weekend picked it from library. Contains very good information & what it takes to attain real sustainability.
I am not done but very interesting tales & full of quality quotes.
If I find more time, I will post more complete review here.


Give a man a fish, and you’ll feed him for a day.
Teach a man to fish, and he’ll buy a funny hat.
Talk to a hungry man about fish, and you’re a consultant.
—Scott Adams

The nation behaves well if it treats the natural resources as assets which it must turn over to the next generation INCREASED . . . in value. (emphasis added)
—Theodore Roosevelt

We really do not know how [the economy] works. . .
The old models just are not working.
—Alan Greenspan, former chairperson of the U.S.

If we become rich or poor as a nation, it’s because of water.
—Sunita Narain

Brief Kiva two years update.

I discovered Kiva in April 2007 via a PBS, tpt, Frontline program. Every month I am contributing little bit of my $ for good cause. To date, my Kiva loan portfolio contains over 225 loans, including 100 loans that have been repaid completely and then re-loaned & I reached my little goal of loaning 100 entrepreneurs always.

Birthday video

Still experimenting with my new Canon 5D & trial versions of photo story software’s. Following video is compressed from 120M in to 3.5 MB with audio added.

Eclipse Ganymede C/C++ CDT IDE platform & my first impressions

Steps

1.Make sure you have a stable JDK on your machine ( in my case, I am using JDK 1.5
2.Download MinGW from http://www.mingw.org/
& install (make sure enabling g++ installtion option)&
update your system PATH setting

After step 2, in your windows command prompt
if you type g++ -v it will display the following.
>>>
C:\a_eclipse_c_c++\eclipse-cpp-ganymede-win32\eclipse>g++ -v
Reading specs from C:/MinGW/bin/../lib/gcc/mingw32/3.4.5/specs etc. etc.
>>

3. Download eclipse & CDT environment from the following location.
http://www.eclipse.org/cdt/index.php
4. After installing eclipse, start eclipse.
5. Start with File->new->project->c++ give default name etc.
6. At this stage you are ready go with new a class or import some existing file.
7. If you are experienced user with Eclipse IDE(i.e. for java code) building & running the code is very easy. Else your have play with Run-> new configuration.
Unfortunately Eclipse-> Help Content sucks.
I finished from step(2) to step(7) in less than 30 minutes. I wrote few c & c++ programs & everything looking good. More later.