Thursday, August 29, 2013

Speculative execution && Hadoop !

I usually disable speculative execution for MapReduce task when I write to RDBMS in Hive user defined table function.

set mapred.map.tasks.speculative=false;
set mapred.reduce.tasks.speculative.execution=false;
set hive.mapred.reduce.tasks.speculative.execution=false;


And if you tune the mapred.reduce.tasks, you can control RDBMS session-running number.

It is good also to use Batch mode and control the commit !