Python thread id in log statements

You can automatically add thread ids to your log statements using Python’s logging module with LogRecord. LogRecord provides 2 attributes for thread related information: threadName and thread.

%(thread)d : Thread ID (if available).
%(threadName)s : Thread name (if available).

For printing thread name:

from logging import getLogger, getLevelName, Formatter, StreamHandler

log = getLogger()
log.setLevel(getLevelName('INFO'))
log_formatter = Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s [%(threadName)s] ") # I am printing thread id here

console_handler = StreamHandler()
console_handler.setFormatter(log_formatter)
log.addHandler(console_handler)

log.info("Hi, how are you")

Sample output from above:

2014-10-19 18:18:29,644 [INFO] root: Hi, how are you [MainThread]

For printing thread id:

from logging import getLogger, getLevelName, Formatter, StreamHandler

log = getLogger()
log.setLevel(getLevelName('INFO'))
log_formatter = Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s [%(threadName)s] ") # I am printing thread id here

console_handler = StreamHandler()
console_handler.setFormatter(log_formatter)
log.addHandler(console_handler)

log.info("Hi, how are you")

Sample output from above:

2014-10-19 18:22:02,294 [INFO] root: Hi, how are you [140635865786176]

Groovy vs Java file I/O comparison, and how to improve it using withWriter.

**I learnt by making a mistake and hope to help someone with this post.

This weekend I tried to compare Java file I/O with Groovy on two fronts:

a. Code bevity.
b. Performance (time taken to execute task.)

Groovy code was 34 lines and Java came out to be 48. The groovy code could be shorter but somehow I couldn't get the multiple assignments working. Got run time error.

On the performance front Java was a winner. Java code took 425 ms and Groovy code took 37112 ms. I was expecting Groovy to be slower, but not by this margin! I was hoping I was doing something wrong.

Some code is omitted for brevity. You can download the source code from GitHub repository.

final File outputFile = new File(/D:\code\IProjects\GroovyPlayground\BigFileInGroovy.txt/)
... //Code omitted for brevity
for (final int i in 1..200000) {
    ... //Code omitted for brevity
    String line = "${accountNumber}, ${customerNumber}, ${totalMin}, ${totalMinsUsed++}, ${totalMinsAvail}, ${price}, John Smith\n"
    if (i == 1) {
        outputFile.write(line)
    } else {
        outputFile.append(line)
    }
}

So I sent an email to the Groovy user group and got prompt replies back from the users. Thanks to the awesome Groovy community I learnt what I was doing wrong. I was using the append method on the File object directly, which was creating a new reader, opening the file, writing the line at the end of the file and then closing the file and reader. Quoting Jochen Theodorou: "In your Java program you use BufferedWritter#append and in Groovy you use File#append. The big difference between those two methods is, that File#append, has to create a new reader, open the file, go to its end and there attach the line, just to close file and reader again. This takes a lot of time and burns away your performance."

So the trick to improve performance was to use withWriter on the File object that provides a buffered writer for the file in the context of the closure:

outputFile.withWriter('UTF-8') { writer ->
        String line = "${accountNumber}, ${customerNumber}, ${totalMin}, ${totalMinsUsed++}, ${totalMinsAvail}, ${price}, John Smith\n"
        writer << (line)
}

Now the execution time is 671 msec. Way better then previous 37112 msec.

Big thanks to Leonard Axelsson and Jochen Theodorou and in general to everyone who replied to my question on Groovy user group!

This weekend I tried to compare Java file I/O with Groovy on two fronts:

a. Code bevity.
b. Performance (time taken to execute task.)

Groovy code was 34 lines and Java came out to be 48. The groovy code could be shorter but somehow I couldn't get the multiple assignments working. Got run time error.

On the performance front Java was a winner. Java code took 425 ms and Groovy code took 37112 ms. I was expecting Groovy to be slower, but not by this margin!

So I sent an email to the Groovy user group and got a very prompt reply back from one of the users. He mentioned that he wasn't sure that the difference (in performance) I've measured will probably hold for other environments. I am thinking that might be true. But, I thought I would put this post out there and get other opinions about Groovy vs Java performance difference they have seen / measured and tips/tricks to get around it?

Some code is omitted for brevity. You can download the source code from GitHub repository.

for (final int i in 1..200000) {
    accountNumber = rand.nextInt(1001011)
    customerNumber = rand.nextInt(2002110)
    int totalMinsUsed = rand.nextInt(1000)
    int totalMinsAvail = totalMin - totalMinsUsed

    String line = "${accountNumber}, ${customerNumber}, ${totalMin}, ${totalMinsUsed++}, ${totalMinsAvail}, ${price}, John Smith\n"
    if (i == 1) {
        outputFile.write(line)
    } else {
        outputFile.append(line)
    }
}

NFJS Experience - What I gained from it (Part I)

I had the opportunity to be at NFJS from Jul 22-24 sponsored by Griffin Solutions Group. It was my first experience and an experience that is difficult to describe in words. There were so many great sessions, so much to learn from a myriad of speakers covering different technologies.

I aim to cover the technologies that I learnt about in more detail in future blogs. Here I will cover a few main points that spoke to me / I learnt / I noticed :

a. I noticed a lot of momentum being gained in JVM languages. There were a lot of sessions covering Groovy, Scala, Clojure and related technologies around these languages. For example: Spock, Geb (pronounced Jeb) for better unit and functional testing using Groovy.

It's exciting. I just hope that more companies adopt these technologies or at least use them in addition to Java being their primary language (doesn't have to be that way though ;-)).

b. A couple of sessions had great ideas about how to be a better programmer. Specifically the session "Hacking Your Brain for Fun and Profit" from Nathaniel Schutta blew my mind. It had a lot of useful information on how to be more productive. If you happen to be at a conference where Nathaniel is presenting the talk, please do not miss it!

c. Related to JVM based languages again: In general I feel we are evolving towards higher level languages. Java was a breath of fresh air as compared to C++. No more managing memory allocation. That in itself was worthy of gold for me :-). Now, it's time to move away from handcuffs that Java puts around us and start concentrating more on business level logic. Groovy, Scala provide a lot of freedom from writing boiler plate stuff in addition to providing powerful features like "Closures", Meta-Programming (more on them in a later post).

Personally, I gained a lot from conversing with fellow attendees and the speakers. I would highly recommend NFJS to any programmer who cares about his trade. It will reinvigorate you to learn new technologies that in turn will help you solve problems more efficiently!

In a later post, I would share a couple of things I am trying that I learnt from the conference, that are making a difference in my life!

Were you at NFJS or any other tech conference? What did you learn? What spoke to you personally? Have a different opinion than mine? Please share. I would love to hear from you!

Maven and Hibernate hell.

I am a Maven newbie and wanted to setup an example Maven + Spring + Hibernate + Mockito project. I have been fairly successful at that but not without my struggles with Maven.

Here is my hibernate dependency structure in pom.xml. Some experienced maven and hibernate users probably would spot the mistake right away. Anyway, let me go on and fix the "errors" myself.

        
            org.hibernate
            hibernate-core
            3.6.0.CR2
        
        
            org.hibernate
            hibernate-annotations
            3.5.7-SNAPSHOT
        
        
            org.hibernate
            hibernate-commons-annotations
            3.3.0.ga
        
        
            org.hibernate
            hibernate-entitymanager
            3.6.0.CR2
        

Struggle #1: Maven and transitive dependencies.

I try to deploy my application in Tomcat and get the error below:

org.springframework.beans.factory.BeanCreationException: 
  Error creating bean with name 'sessionFactory' defined in 
  ServletContext resource [/WEB-INF/spring/hibernate-config.xml]: 
  Invocation of init method failed; nested exception is
  java.lang.IncompatibleClassChangeError: Implementing class

After researching I found that the reason is multiple hibernate jars in the classpath. I check and yes there is a: hibernate.3.2.1.ga.jar and b: hibernate-core-3.6.0.CR2.jar. So now I have the task of finding out which hibernate dependency is bringing the 2 different hibernate jars.

On further research I find that hibernate-commons-annotation.3.3.0.ga.pom has dependency on hibernate.3.2.1.ga and hibernate-annotations-3.5.7-SNAPSHOT.pom has dependency on hibernate-core-3.6.0.CR2.

So that explains why I see two jars. I manually exclude the hibernate jar from hibernate-commons-annotations:

            org.hibernate
            hibernate-commons-annotations
            3.3.0.ga
            
                
                
                    org.hibernate
                    hibernate
                
            
        

And that takes care of multiple hibernate jars on the classpath. Is there is an easy way to not have transitive dependency in maven?

Struggle #2: Now on deployment I get:

Caused by: java.lang.NoClassDefFoundError:
  org/hibernate/annotations/common/reflection/MetadataProvider

hibernate-commons-annotation.3.3.0.ga.jar does not have MetadataProvider class. After googling the error I find: http://forum.springsource.org/showthread.php?t=89693

From hibernate-commons-annotation readme:

Version '3.3.0.ga' was a mistake during the release process, please disregard.
There is not such version 3.3.0.ga.

The solution then is to use hibernate-commons-3.2.0.Final

Final working dependency structure:

        
            org.hibernate
            hibernate-annotations
            3.5.7-SNAPSHOT
        
        
        
            org.hibernate
            hibernate-commons-annotations
            3.2.0.Final
        
        
            org.hibernate
            hibernate-entitymanager
            3.6.0.CR2
        

I wonder if all of this effort would have been saved had I just downloaded the jars from hibernate.org directly. Which makes me wonder if Maven is worth it for small projects?

Maven: Add Mockito Dependency.

Adding Mockito dependency to your project is pretty simple.

Simply add the dependencies below to your pom.xml.

NOTE: You will need JUnit version 4.4 or earlier. I couldn't get Mockito to work with higher JUnit versions. Also, I did not try any prior version than Junit 4.4.

    org.mockito
    mockito-all
    1.8.5
    test
  

  
    junit
    junit
    4.4
    test
  

Update: I got Mockito 1.8.5 to work with JUnit 4.7. Not sure what changed.

Maven: Add Spring dependencies and Spring MVC your project

To enable Spring and Spring MVC modules in your maven project, follow the steps below

#1: Add Spring related dependencies. The dependencies below will get all the rest of the transitive dependencies.

  
    org.springframework
    spring-context
    ${org.springframework.version}

    
      
      
        commons-logging
        commons-logging
      
    

    
      
      org.springframework
      spring-webmvc
      ${org.springframework.version}
    

The org.springframework.version property is set to 3.0.3.RELEASE:

3.0.3.RELEASE

  

#2. Setup Dispatcher Servlet in your web.xml:

The Spring MVC requires setting the dispatcher servlet in web.xml. Below we are loading the /WEB-INF/sprin/app-config.xml (the main Spring application configuration file for the project) while instantiating the Dispatcher Servlet at application start up time i.e. when the server is starting up.

NOTE: You can name your application config with any name of your liking. Just make sure it is correctly referenced in Dispatcher Servlet setting in web.xml.

    springmvcdemo
    org.springframework.web.servlet.DispatcherServlet
    
      contextConfigLocation/WEB-INF/spring/app-config.xml
      
    1
  

  
    springmvcdemo
    /app/*
  

#3. The context config location points to the spring application config file for the application (app-config.xml). There are 2 things of note here:

a. The com.mayabansi.webappdemo will be scanned for classes annotated with @Controller, @Resources. @Component, @Service etc.

b. The mvc-config.xml, that holds the MVC configuration settings is imported.

#4. The MVC config file has Spring MVC related configuration settings: