Colin Kang

관점 : 파일포맷, 압축, 저장소

파일포맷 :

TEXTFILE, SEQUENCEFILE, RCFILE, ORC, PARQUET

압축방식 :

Deflate, GZip, Bzip2, LZO, LZ4, Snappy

선택은?

ORC with Snappy 인듯한데, 자세한내용은 아래 참조

https://www.slideshare.net/oom65/file-format-benchmarks-avro-json-orc-parquet

SerDe

SerDe는 직렬화 (Serializer)와 역직렬화 (Deserializer)의 약자

SerDe는 테이블 컬럼 데이터 타입에 직렬화와 역직렬화를 적용하여 어떻게 데이터를 처리할지를 다루는 기술

종류 :

LazySimpleSerDe: 

STORED AS TEXTFILE

ColumnarSerDe :

ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'

STORED AS RCFile 

RegexSerDe :

ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'

HBaseSerDe :

ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'

AvroSerDe :

ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'

STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro'

OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro'

ParquetHiveSerDe :

STORED AS PARQUET 

OpenCSVSerDe :

ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'

STORED AS TEXTFILE

JSONSerDe :

ROW FORMAT SERDE 

'org.apache.hive.hcatalog.data.JsonSerDe' 

STORED AS INPUTFORMAT 

'org.apache.hadoop.mapred.TextInputFormat' 

OUTPUTFORMAT 

'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'

참조 :

https://cwiki.apache.org/confluence/display/Hive/SerDe

https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-HowtoWriteYourOwnSerDe

Primitive type과 Complex type으로 나뉘며, 

Array와 Map은 자바와 동일하며, Struct의 필드는 어떤 타입이든 집합으로 가질수 있음

Complex type은 타입 중첩을 허용함.

형변환은 묵시적, 명시적

명시적은 cast(cname as TYPE)으로 변환

Numeric Types

• TINYINT (1-byte signed integer, from -128 to 127)
• SMALLINT (2-byte signed integer, from -32,768 to 32,767)

• INT/INTEGER (4-byte signed integer, from -2,147,483,648 to 2,147,483,647)

• BIGINT (8-byte signed integer, from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807)
• FLOAT (4-byte single precision floating point number)
• DOUBLE (8-byte double precision floating point number)

• DOUBLE PRECISION (alias for DOUBLE, only available starting with Hive 2.2.0)

• DECIMAL
  • Introduced in Hive 0.11.0 with a precision of 38 digits
  • Hive 0.13.0 introduced user-definable precision and scale
• NUMERIC (same as DECIMAL, starting with Hive 3.0.0)

Date/Time Types

• TIMESTAMP (Note: Only available starting with Hive 0.8.0)
• DATE (Note: Only available starting with Hive 0.12.0)
• INTERVAL (Note: Only available starting with Hive 1.2.0)

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types