John's Technical Blog

Google API Fusion Table permissioning

2013-06-04

I recently worked on a project where I needed to update an application that leveraged Google Fusion Tables. The Google API changed significantly, and the application did not work anymore. While I found a good Fusion Table coding example of how to get the Java code changed properly, I had a lot of difficulty getting the permissioning set up.

Here is a brief summary of how I got it to work, in the hopes that it might help others who are having similar problems:

Connect the table to the Fusion Table application

  • If the table you are interested in is not already connected to Fusion Tables, click it in your Google Drive, and then click the Connect button.

Turn on the Fusion Table API Service

  • Open the Google API Console
  • Create a new project if you need to
  • In Services, turn on Fusion Tables API

Set up a Service Account

  • In the Google API Console, open API Access and click the Create an OAuth 2.0 client ID button
  • Enter a Product Name and click Next
  • Click the Service Account radio button, and then Create Client ID
  • Download the key file into your project, and rename it to whatever is appropriate for you to use in your application
  • You will also need the "Email Address", which is referred to as the "Application ID" within the API

Set permissions on the table file

This one was really difficult to figure out. If you need to do INSERTs or DELETEs into the Fusion Table, then you will need to set "writer" permissions for the Service Account. If you only need to SELECT from your application, then you can skip this step, of course.

  • Open the Google Drive SDK Permissions Page
  • Turn on the Authorize requests using OAuth 2.0 toggle (you should be prompted to authorize)
  • Enter the following information:
    • Field: [the fusion table ID]
    • role: writer
    • type: user
    • value: ["Email Address" from the Console]

Java code

That should be it. Following the example code, you will set up a credential:

credential = new GoogleCredential.Builder()
   
.setTransport(HTTP_TRANSPORT)
   
.setJsonFactory(JSON_FACTORY)
   
.setServiceAccountId(config.getAccountId())
   
.setServiceAccountScopes(Collections.singleton(FusiontablesScopes.FUSIONTABLES))
   
.setServiceAccountPrivateKeyFromP12File(keyFile)
   
.build();

Make your Fusion Table object:

fusiontables = new Fusiontables.Builder(HTTP_TRANSPORT, JSON_FACTORY, credential).build();

Run your various SQL statements:

response = fusiontables.query().sql(insertSql).execute();

Hope that helps someone!

BigDecimal.equals() vs. compareTo() Performance Test

2013-05-11


Problem


There is much debate online about the merits of using the BigDecimal.equals(Object) method. In the equals method, scale is considered in such a way that 1.0 != 1.00. The compareTo method, however, ignores scale, meaning that 1.0 == 1.00. In this specific case, however, scale is known, so either method will work. Out of curiosity, I wondered if there was a performance hit in either method.

Approach

import java.math.BigDecimal;
import java.math.MathContext;
import java.math.RoundingMode;
import java.security.SecureRandom;

import org.joda.time.DateTime;
import org.junit.Test;

/**
 <p>
 * Class to test the performance of {@link BigDecimal#equals(Object)} vs.
 {@link BigDecimal#compareTo(BigDecimal)}. The question at hand is which
 * method to use when the precision and scale of the BigDecimals being compared
 * is know to be the same.
 </p>
 <p>
 * In the equals method, scale is considered in such a way that 1.0 != 1.00. The
 * compareTo method, however, ignores scale, meaning that 1.0 == 1.00. In this
 * specific case, however, scale is known, so either method will work. Out of
 * curiosity, I wondered if there was a performance hit in either method.
 </p>
 */
public class BigDecimalPerformanceTest {

    /** The {@link SecureRandom} object. */
    public static final SecureRandom RANDOM = new SecureRandom();

    /** The number of times to loop over each comparison. */
    public static final int ITERATIONS = 100000000;

    /** The number of times to loop over all comparisons. */
    public static final int LOOPS = 3;

    /** BigDecimal precision. */
    private static final int PRECISION = 15;

    /** BigDecimal scale. */
    private static final int SCALE = 2;

    /** Equals vs. compareTo. */
    @SuppressWarnings("unused")
    @Test
    public void equalsVsCompareTo() {
        for (int j = 0; j < LOOPS; j++) {
            /* Reference object to do all the comparisons against. */
            BigDecimal testBigDecimal = makeBigDecimal();

            /*
             * Classic equals method. Assigning results to a variable for
             * consistency.
             */
            DateTime startTime = new DateTime();
            for (int i = 0; i < ITERATIONS; i++) {
                boolean test = testBigDecimal.equals(makeBigDecimal());
            }
            DateTime endTime = new DateTime();
            System.out.println("BigDecimal.equals()    : "
                               + endTime.minus(startTime.getMillis()).toString("m:s.SSS"));

            /* Plain compareTo method. */
            startTime = new DateTime();
            for (int i = 0; i < ITERATIONS; i++) {
                int test = testBigDecimal.compareTo(makeBigDecimal());
            }
            endTime = new DateTime();
            System.out.println("BigDecimal.compareTo()1: "
                               + endTime.minus(startTime.getMillis()).toString("m:s.SSS"));

            /*
             * Convert compareTo to true/false as a closer comparison with
             * equals as far as actual usage would be concerned.
             */
            startTime = new DateTime();
            for (int i = 0; i < ITERATIONS; i++) {
                boolean test = testBigDecimal.compareTo(makeBigDecimal()) == 0;
            }
            endTime = new DateTime();
            System.out.println("BigDecimal.compareTo()2: "
                               + endTime.minus(startTime.getMillis()).toString("m:s.SSS"));
        }
    }

    /**
     * Make big decimal.
     
     @return the big decimal
     */
    private BigDecimal makeBigDecimal() {
        return new BigDecimal(RANDOM.nextDouble()new MathContext(PRECISION, RoundingMode.HALF_UP))
                .setScale(SCALE, RoundingMode.HALF_UP);
    }
}

Output


BigDecimal.equals()    : 3:42.909
BigDecimal.compareTo()1: 3:44.995
BigDecimal.compareTo()2: 3:49.476
BigDecimal.equals()    : 3:48.313
BigDecimal.compareTo()1: 3:42.152
BigDecimal.compareTo()2: 3:43.889
BigDecimal.equals()    : 3:42.079
BigDecimal.compareTo()1: 3:44.639
BigDecimal.compareTo()2: 3:42.564

Conclusion


When you know the scale of a BigDecimal will be consistent, equals() and compareTo() perform equally as well. In my case, I chose to stick with using equals(), as it is immediately obvious in the code what I am trying to accomplish.

Bin2Txt - Remove Non-printable Characters from a Text File with Java

2011-10-31

I recently was dealing with some DB2 "unload" (e.g., export) files that I wanted to parse and then load into Oracle. I found that the unload files use a lot of binary characters, which makes it very difficult to parse. I wrote the following Java class to convert the unprintable characters into a tilde (which is a character that does not occur in the data). This resulted in DB2 unload files that were parsable as fixed-width data files.
The main problem this approach does not attempt to solve is that the DB2 unload files save numeric fields as the actual value, not the digit equivalent (i.e., the number 84 is unloaded as the ASCII-equivalent "T", not "84"). This code obviously does not reference the DB2 "punch" (e.g., parse instruction) files, so it makes no attempt to parse the files into fields itself - that is a separate exercise in my case. BTW, if there is a good way to import these files into Oracle automatically, please let me know, as I have not been able to find a better solution.
This code is fairly generic, and can be used for other purposes beyond converting DB2 unload files, so if you have a need to replace non-printable characters in text files, you can start with this code base.

package com.threeleaf.bin2txt;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

/**
 * Purpose is to read a file and replace non-printable characters with a given character.
 * Specifically, I want to use this to make DB2 unload files parsable with other applications so
 * that the data can be imported into Oracle.
 *
 * @author John A. Marsh
 * @since 2011-10-27
 */
public final class Bin2Txt {

    /**
     * Run this class from the command line with:
     * java Bin2Txt <pathAndFilename>.
     *
     * @param args
     *        the filename to convert
     * @throws IOException
     *         Signals that an I/O exception (e.g., file not found) has occurred.
     */
    public static void main (final String[] args) throws IOException {
        final byte ASCII_SPACE = 32;
        final byte ASCII_CR = 13;
        final byte ASCII_LF = 10;
        final byte ASCII_TILDE = 126;

        try {
            final File file = new File(args[0]);
            final InputStream inputStream = new FileInputStream(file);
            final long fileLength = file.length();

            /*
             * Array needs to be created with an int type, so need to check to ensure that file is
             * not larger than Integer.MAX_VALUE.
             */
            if (fileLength > Integer.MAX_VALUE) {
                throw new IOException("File is too big");
            }

            /* Create the byte array to hold the data */
            final byte[] bytes = new byte[(int) fileLength];

            /* Read in the bytes */
            int offset = 0;
            int numRead = 0;
            while (offset < bytes.length && (numRead = inputStream.read(bytes, offset, bytes.length - offset)) >= 0) {
                offset += numRead;
            }

            /* Ensure all the bytes have been read in */
            if (offset < bytes.length) {
                throw new IOException("Could not completely read file " + file.getName());
            }
            inputStream.close();

            for (int i = 0; i < bytes.length; i++) {
                if (bytes[i] == ASCII_CR && bytes[i + 1] == ASCII_LF) {
                    /*
                     * Preserve line breaks (carriage return + line feed) by skipping over them.
                     * Note that I don't check for end of file here because I already know my
                     * particular files will never end with a CRLF.
                     */
                    i = i + 2;
                }
                if (bytes[i] < ASCII_SPACE || bytes[i] > ASCII_TILDE) {
                    /* Replace all non-printable characters. */
                    bytes[i] = ASCII_TILDE;
                }
            }
            /* Output file name will be the same as the input, with ".out.txt" added to the end. */
            final OutputStream outputStream = new FileOutputStream(args[0] + ".out.txt");
            outputStream.write(bytes);
            outputStream.close();
        } catch (final ArrayIndexOutOfBoundsException e) {
            /*
             * If no file was passed on the command line, this exception is generated. A message
             * indicating how to the class should be called is displayed.
             */
            System.out.println("Usage: java Bin2Txt filename\n");
        }
    }
}

Here is a batch file that will convert all the files in a given directory:

:: Directory where Bin2Txt.class is located ::
cd C:\projects\workspace\bin2txt\bin\
:: Put in directory where unload files are ::
for %%f in ("C:\projects\Database\Unloads\*.txt") do call java com.threeleaf.bin2txt.Bin2Txt %%f

Solved: Firefox Tab Icons Are Missing

2011-10-13

I have been frustrated over the last few weeks because I could not figure out why the icons suddenly stopped showing up in Firefox. I have scoured the web and tried several suggestions, but none of them worked for me. I finally got clued in that it might be one of my extensions, and sure enough, going to Help ยป Restart With Add-ons Disabled caused the icons to reappear. After a little trial and error I finally found that it was the Favicon Picker 2. I wanted the functionality, so I tried Favicon Picker 3, but it caused the same problem. I finally found that Bookmark Favicon Changer (https://addons.mozilla.org/en-US/firefox/addon/bookmark-favicon-changer/) allowed me to customize my bookmark icons without causing the tabs to go blank. So, if you find that the favicons in your tabs are absent, give it a try. As of this writing, I am using Firefox 7 and Bookmark Favicon Changer 1.54.

Interning String Literals In Java

2011-09-13

A coworker and I were discussing strings, and some of the things we had heard about how the Java compiler stores and retrieves them from memory. We were particularly wondering if constants, local variables, literals, and runtime generated strings were handled differently. I wrote the following JUnit 4 test to show how all these cases are handled (Java JDK 1.6.0_27):

/**
* Test string interning affects on literal and runtime strings.
*/
public final class StringInternTest {

/**
* String intern test.
*/
@Test
public void stringIntern () {
/* literals */
assertEquals("date", "date");
assertTrue("date" == "date");
/* Compare to an external constant */
assertEquals("date", Constants.DATE);
assertTrue("date" == Constants.DATE);
/* Compare locally defined strings. */
final String myDateString = "date";
assertEquals("date", myDateString);
assertTrue("date" == myDateString);
final String myDateString2 = "date";
assertEquals(myDateString, myDateString2);
assertTrue(myDateString == myDateString2);
assertEquals(Constants.DATE, myDateString);
assertEquals(Constants.DATE, myDateString2);
assertTrue(Constants.DATE == myDateString);
assertTrue(Constants.DATE == myDateString2);
/* Create new strings at runtime. */
final String runtime1 = new String("date");
final String runtime2 = new String("date");
assertEquals("date", runtime1);
assertEquals("date", runtime2);
assertEquals(Constants.DATE, runtime1);
assertEquals(Constants.DATE, runtime2);
assertEquals(runtime1, runtime2);
assertFalse(runtime1 == runtime2); // !!!
assertFalse("date" == runtime1); // !!!
assertFalse("date" == runtime2); // !!!
assertFalse(Constants.DATE == runtime1); // !!!
assertFalse(Constants.DATE == runtime2); // !!!
/* Intern the runtime strings. */
final String interned1 = runtime1.intern();
final String interned2 = runtime2.intern();
assertTrue(interned1 == interned2);
assertTrue("date" == interned1);
assertTrue("date" == interned2);
assertTrue(Constants.DATE == interned1);
assertTrue(Constants.DATE == interned2);
}
}

If you set a break point in this test and examine the variables you will find that Constants.DATE, myDateString, myDateString2, interned1, and interned2 all have the same internal object ID. I learned online that all string literals are supposed to be interned when the application is compiled, which accounts for the variables having the same ID.

Strings stored in runtime1 and runtime2 are each given a unique object ID when instantiated at runtime, and thus return a false when == is tried with any of the interned strings.

Some have suggested that manually interning is better because using == is much faster (5x) than equals() (because comparing object IDs is faster than comparing string lengths || characters). However, others dismiss this as a minor gain at best, and I can certainly see that using == might lead to some hard-to-find bugs if a non-interned string is compared to either an interned or another non-interned string. I also learned that main argument strings (args[]) are not interned.

Interned strings are stored in the PermGen (Permanent Generation) memory, and it is possible to fill up that space with strings if one is not careful.

References: