Thursday, May 2, 2013
Monday, April 29, 2013
What can I do with Mahout ?
- Clustering
- Canopy
- K-Means
- Fuzzy K-Means
- Dirichlet Process
- Latent Dirichlet Allocation
- Mean-shift
- Expectation Maximization
- Spectral
- Minhash
- Top Down
- Classification
- Logistic Regression
- Bayesian
- Support Vector Machines
- Random Forests
- Decision forest
- Machine learning
- Recommendation
- Dimension reduction
- Your own business ! (If you understand how MapReduce and Mahout class work together, you can code your own logic)
Friday, April 26, 2013
Mahout !
Mahout is an incredible library to do machine learning, clustering, classification, recommendation. It works directly on top of Hadoop and MapReduce !
This is how to launch a recommender Job :
hadoop jar /usr/lib/mahout/mahout-core-0.7.0.21-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input /apps/hive/warehouse/profile_activity_text_file --output /apps/hive/warehouse/recommenderJob --similarityClassname SIMILARITY_COOCCURRENCE --booleanData true
This is how to launch a recommender Job :
hadoop jar /usr/lib/mahout/mahout-core-0.7.0.21-job.jar org.apache.mahout.cf.taste.hadoop.item.RecommenderJob --input /apps/hive/warehouse/profile_activity_text_file --output /apps/hive/warehouse/recommenderJob --similarityClassname SIMILARITY_COOCCURRENCE --booleanData true
Wednesday, April 10, 2013
Storm & Hadoop/Hive partitioning load !
I am currently working on how Storm can load data into a partitioned Hadoop/Hive table.
This is how I do :
This is how I do :
- put hadoop libs into the Storm lib directory
- add the hadoop xml conf and parse them using conf.addRessource();
- create a HDFSBolt (implements IRichBolt)
- add some private HashMap<String partition, FSDataOutputStream fsDataOutputStream >
- override execute function (if the partition already exists use current buffer else create a new one)
Tuesday, April 9, 2013
Saturday, April 6, 2013
Monday, March 25, 2013
Sunday, March 24, 2013
Friday, March 22, 2013
Storm & real-time ETL !
Storm is a amazing scalable, fault-tolerant, open-source, real-time ETL. Let's storm !
Saturday, March 16, 2013
Main Storm daemons !
- Nimbus (The Storm JobTracker)
- Supervisor (The supervisor daemon is responsible for starting and stopping worker processes)
- UI (administration website)
Thursday, March 7, 2013
HIVE-3963 : Hive & RDBMS example !
>add jar /home/hive/developpement/loadfromjdbc.jar;
>add jar /home/hive/developpement/tdgssconfig.jar;
>add jar /home/hive/developpement/terajdbc4.jar;
>create temporary function loadfromjdbc as 'mlanciau.dev.loadfromjdbc';
>SELECT result['column_name1'], result['column_name2']
FROM (
SELECT loadfromjdbc('com.teradata.jdbc.TeraDriver',
'jdbc:teradata://ip/CHARSET=UTF16',
'db_user', 'db_password',
'SELECT * FROM database_name.table_name') AS (result) FROM dual
) ssreq
We can now do join with Hadoop and database data directly from Hive.
SQL inside SQL ! Browse HIVE-3963.
>add jar /home/hive/developpement/tdgssconfig.jar;
>add jar /home/hive/developpement/terajdbc4.jar;
>create temporary function loadfromjdbc as 'mlanciau.dev.loadfromjdbc';
>SELECT result['column_name1'], result['column_name2']
FROM (
SELECT loadfromjdbc('com.teradata.jdbc.TeraDriver',
'jdbc:teradata://ip/CHARSET=UTF16',
'db_user', 'db_password',
'SELECT * FROM database_name.table_name') AS (result) FROM dual
) ssreq
We can now do join with Hadoop and database data directly from Hive.
SQL inside SQL ! Browse HIVE-3963.
Monday, February 25, 2013
Wednesday, February 13, 2013
Tuesday, February 5, 2013
HIVE-3963 : Hive & RDBMS !
I will be working soon on HIVE-3963. My goal is to allow Hive to read/write to Database thanks to JDBC. I have already tried and it works well !
Wednesday, January 16, 2013
Monday, January 14, 2013
Thursday, January 3, 2013
Tuesday, January 1, 2013
Thursday, December 20, 2012
Saturday, December 15, 2012
Wednesday, December 12, 2012
Friday, December 7, 2012
Hive and custom Map Reduce !
With perl, python or java :
ADD file reducer.pl;
FROM (
FROM t_table
SELECT c_1, c_2
WHERE c_2 = '2012-12-07'
DISTRIBUTE BY c_1
SORT BY c_2
) ssreq
REDUCE ssreq.c_1, ssreq.c_2
USING 'reducer.pl'
You can find more on Hive tutorial !
ADD file reducer.pl;
FROM (
FROM t_table
SELECT c_1, c_2
WHERE c_2 = '2012-12-07'
DISTRIBUTE BY c_1
SORT BY c_2
) ssreq
REDUCE ssreq.c_1, ssreq.c_2
USING 'reducer.pl'
You can find more on Hive tutorial !
Tuesday, December 4, 2012
Sunday, December 2, 2012
Thursday, November 15, 2012
Monday, October 29, 2012
Monday, October 22, 2012
Saturday, October 20, 2012
Monday, October 15, 2012
Hive SerDe !
SerDe means Serialisation/Deserialisation, it is one of the amazing power of Hive/Hadoop.
CREATE EXTERNAL TABLE t_access_log_part (
c_proxy STRING, c_ip STRING, c_timetaken STRING,
c_jour STRING, c_mois STRING,
c_annee STRING, c_hour STRING,
c_reste_timestamp STRING, c_commande STRING,
c_fichier STRING, c_protocole STRING,
c_code_retour STRING, c_size STRING,
c_reste STRING, c_identifiant STRING
)
PARTITIONED BY (c_date string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([a-zA-Z-0-9]*)[^\\t]*\\t(\\d{1,3}[.]\\d{1,3}[.]\\d{1,3}[.]\\d{1,3})[ ][^ ]+[ ](\\d+)[ ]+\\[(\\d+)/([0-9a-zA-Z]+)/([0-9a-zA-Z]+):(\\d+):(.*)\\][ ]\"(\\w+)[ ](.*)[ ]+([A-Za-z0-9/.]+)\"[ ]+([0-9A-Za-z]+)[ ]+([0-9A-Za-z-]+)[ ]+\"(.*)\"[ ]+\"(.*)\"",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s %10$s %11$s %12$s %13$s %14$s %15$s"
)
STORED AS TEXTFILE;
No ETL !
CREATE EXTERNAL TABLE t_access_log_part (
c_proxy STRING, c_ip STRING, c_timetaken STRING,
c_jour STRING, c_mois STRING,
c_annee STRING, c_hour STRING,
c_reste_timestamp STRING, c_commande STRING,
c_fichier STRING, c_protocole STRING,
c_code_retour STRING, c_size STRING,
c_reste STRING, c_identifiant STRING
)
PARTITIONED BY (c_date string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
"input.regex" = "([a-zA-Z-0-9]*)[^\\t]*\\t(\\d{1,3}[.]\\d{1,3}[.]\\d{1,3}[.]\\d{1,3})[ ][^ ]+[ ](\\d+)[ ]+\\[(\\d+)/([0-9a-zA-Z]+)/([0-9a-zA-Z]+):(\\d+):(.*)\\][ ]\"(\\w+)[ ](.*)[ ]+([A-Za-z0-9/.]+)\"[ ]+([0-9A-Za-z]+)[ ]+([0-9A-Za-z-]+)[ ]+\"(.*)\"[ ]+\"(.*)\"",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s %10$s %11$s %12$s %13$s %14$s %15$s"
)
STORED AS TEXTFILE;
No ETL !
Friday, October 12, 2012
Monday, October 1, 2012
Monday, September 17, 2012
Hive development !
With Hive you can
- do ETL and analysis using QL (a SQL like query language)
- create table with partition or bucketed table !
- compress your data !
- use java reflection to instantiate and call methods of objects
- build custom scalar functions (UDF's), aggregations (UDAF's), and table functions (UDTF's)
Sunday, September 16, 2012
Hive join tips !
You can specified the [biggest] table to be streamed during a join operation :
SELECT /*+ STREAMTABLE(a) */ a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key1);
And you can specified the [smallest] table to be join in a map join if you want to avoid the reducer
SELECT /*+ MAPJOIN(b) */ a.key, a.value FROM a join b on a.key = b.key;
And if you want to do ... WHERE a.key IN (SELECT ...) use :
SELECT a.key, a.val FROM a LEFT SEMI JOIN b on (a.key = b.key);
SELECT /*+ STREAMTABLE(a) */ a.val, b.val, c.val FROM a JOIN b ON (a.key = b.key1) JOIN c ON (c.key = b.key1);
And you can specified the [smallest] table to be join in a map join if you want to avoid the reducer
SELECT /*+ MAPJOIN(b) */ a.key, a.value FROM a join b on a.key = b.key;
And if you want to do ... WHERE a.key IN (SELECT ...) use :
SELECT a.key, a.val FROM a LEFT SEMI JOIN b on (a.key = b.key);
Friday, September 14, 2012
Creating Custom Hive UDFs !
I really enjoy using Hive because ETL or analysis become easy, but sometimes you need to create your own function ! Follow the link !
Sunday, September 9, 2012
Wednesday, August 15, 2012
Saturday, July 28, 2012
Wednesday, July 18, 2012
Thursday, July 5, 2012
Wednesday, July 4, 2012
Hive development !
If you read this website, you can realise how hive is an incredible tool specially if you configure your Metastore store to work with a RDBMS.
You can :
You can :
- CREATE TABLE (on HDFS or on HBase)
- SHOW, DESCRIBE, ALTER, DROP TABLE !
- LOAD DATA (from HDFS or locally)
- SELECT ... FROM ... WHERE or BETWEEN UNION ALL ... (like a RDBMS !)
- INSERT ... SELECT ...
- use well know functions like DISTINCT, COUNT, SUM, AVG and of course GROUP BY operation !
- you can [LEFT OUTER|RIGHT OUTER|FULL OUTER] JOIN multiple table !
- use built-in functions
- manipulate INT, BIGINT, FLOAT, BOOLEAN, DOUBLE, TIMESTAMP, STRING and complex types like structs, maps and arrays !
Monday, July 2, 2012
Saturday, June 30, 2012
Hadoop best practice !
- Check if your dfs.name.dir has one local value and another directory like NFS.
- No swap allowed !
- Enough memory !!
- The Secondary Name Node is not on the same machine as the NameNode.
- Time is sync !
- Fast network !
- ulimit is set
Main Hadoop daemons
- NameNode (stores HDFS metadata)
- DataNode (stores HDFS data)
- Secondary NameNode (takes snapshots of the HDFS metadata)
- JobTracker (determines the execution plan)
- TaskTracker (executes MapReduce job)
Monday, June 25, 2012
Monday, June 18, 2012
Sunday, June 17, 2012
Saturday, June 16, 2012
Hadoop platforms !
You can dowload and set up the component you want or choose to install a platform. These are the most famous for now :
Friday, June 15, 2012
Thursday, June 7, 2012
Hadoop script !
#!/bin/bash
for serveur `cat $HBASE_HOME/conf/regionservers`;
do
rsync -avz --delete --exclude='logs/*' $HBASE_HOME $serveur:$HBASE_HOME
done
for serveur `cat $HBASE_HOME/conf/regionservers`;
do
rsync -avz --delete --exclude='logs/*' $HBASE_HOME $serveur:$HBASE_HOME
done
Wednesday, June 6, 2012
Monday, May 28, 2012
Sync with firefox !
If you want to keep your bookmarks, history, passwords, add-ons and open tabs synchronize on all your firefox, try sync !
Tuesday, May 22, 2012
Sunday, May 20, 2012
Friday, May 18, 2012
So HBase !
Why : large scale (> 100 Go)
Specificity : versioned cell, column oriented, on top of Hadoop, sparse, open-source
Blog : http://hadoop-hbase.blogspot.fr/
Specificity : versioned cell, column oriented, on top of Hadoop, sparse, open-source
Blog : http://hadoop-hbase.blogspot.fr/
Thursday, May 17, 2012
Wednesday, May 16, 2012
Monday, May 14, 2012
Informatica repository tips !
Do you want to check if there is some useless workflow ? Connect to your repository and begin your analysis with REP_SESS_TBL_LOG or OPB_SWIDGINST_LOG !
Sunday, May 13, 2012
Saturday, May 12, 2012
Thursday, May 10, 2012
Informatica && Performance !
Informatica is one of the greatest ETL tool ! Mainly for performance issue. You need to improve your sessions ? Look for bottlenecks, and start with your thread statistics !
Tuesday, May 8, 2012
Saturday, May 5, 2012
Sunday, April 29, 2012
Saturday, April 28, 2012
Saturday, April 21, 2012
Wednesday, April 18, 2012
ETL !
When you want to deal with your [unstructured|semi-structured|structured] data, you need a lot of tools and specially one ! An ETL !
Tuesday, April 17, 2012
Monday, April 16, 2012
Hello !
I am here to share news and advice about Business Intelligence and other stuff !
Hope it will help someone ^^ !
Subscribe to:
Posts (Atom)