Thursday, July 16, 2015

Hive Convert Array to String

It is simple if it is an array, there is an UDF called concat_ws(SEP, array). But concat_ws can only handle array, not for array, array. Hive doesn't support cast array to array or string. You cannot insert into a table with a string column for the array.

Run out of options? You can actually use TRANSFORM and /bin/cat to convert to string like this: when the column is passed to the script, it convert into a string same as when you run "select * from table".

create table regions(
  ...
  coordinates array<array<double>>,
  ...
)

from regions select transform(coordinates) using '/bin/cat' as (coordinates);

Tuesday, June 16, 2015

Create Fedora 22 bootable USB

  • Download netinst ISO file:
  • Log in as root:
    su -
  • Find out which device maps to the USB flash drive:
    • using
      dmesg | tail
    • using
      fdisk -l
      . NOTE: use /dev/sdd instead of /dev/sdd1 because the later one is the partition.
  • Burn the ISO file into USB flash drive. The content in the drive will be overwritten using this way.
    dd if=/home/ben/Downloads/Fedora-Workstation-netinst-i386-22.iso of=/dev/sdd
  • If it doesn't reboot the machine and you may see a blinking cursor, please change "USB Emulation Type" in your BIOS settings from Auto to Hard Disk, and try again.

My machine is a pretty old one, with a K8V-MX motherboard and AMI BIOS v02.54 (C)Copyright 1985-2003.

In my BIOS: the emulation type could be found like this:

  • press DEL to enter BIOS setup.
  • Advanced -> Chipset -> USB Configuration.
  • It will show the information like this:
    Module version -2.24.0-7.4
    USB Devices Enabled: 1 Drive
    USB Mass StorageDevice Configuration
    
  • Enter "USB Mass Storage Device Configuration", you will find "Emulation Type"

I also tried to recreate a FAT32 filesystem in my 4GB USB drive, but it didn't work even I changed to hard disk for usb emulation type. Not sure what is wrong yet."

On a Window 7, here is how to format the drive is already burned using dd: Start -> Computer -> (Right click) -> Manage -> Disk Management

  • Delete the volume of 510MB;
  • Create a new volume using 4GB;
  • Format using FAT32;

There is not a bootable partition after you finish, but after using liveusb-creator, the partition is

Monday, April 13, 2015

How to resolve "A schema cannot contain two global components with the same name"?

When I have a project using spring-data-cassandra, the XML configuration applicationContext.xml's header looks like below.


...

Eclipse always complains the following errors:

Referenced file contains errors (http://www.springframework.org/schema/beans/spring-beans.xsd).  For more information, right click on the message in the Problems View and select "Show Details..."
Referenced file contains errors (http://www.springframework.org/schema/tool/spring-tool.xsd).  For more information, right click on the message in the Problems View and select "Show Details..."

The details are shown as below.

The errors below were detected when validating the file "spring-beans.xsd" via the file "applicationContext.xml".  In most cases these errors can be detected by validating "spring-beans.xsd" directly.  However it is possible that errors will only occur when spring-beans.xsd is validated in the context of applicationContext.xml.

sch-props-correct.2: A schema cannot contain two global components with the same name; this schema contains two occurrences of 'http://www.springframwork.org/schema/beans,identifiedType'. line 43
...

The problem is caused, I believe, by the version in different XSDs. For example, spring-context.xsd imports spring-beans.xsd and spring-tools.xsd, and always uses the links with a version like this: http://www.springframework.org/schema/beans/spring-beans-4.1.xsd.




...

But spring-cassandra.xsd imports XSDs without version.




 
...

If you download the following xsd, spring-beans, spring-tool, spring-cassandra, spring-context and spring-repository, and change all schemaLocations to point a local file like "file:///tmp/spring-beans.xsd", the validation errors will be gone.

This is really annoying when you see red X. Here are my suggestions:

  • remove xsi:schemaLocation in your applicationContext.xml, there will be a warning like "No grammar"
  • Window->Preferences->XML->XMl Files->Validation->uncheck "Honor all XML schema locations"