Need some help to figure out bottlenecks

Hi everyone,

I need help to figure out what the bottleneck is in a hdfs-scan operation.

Whenever I request a query to Impalad, a bottleneck occurs in a specific
node, is named pdpds04, in my cluster. ( All nodes are identical )

I would appreciate if anyone can give me some suggestions for figuring out
what the bottleneck is.

Two query profiles of a bottleneck node and a non-bottleneck node are shown
below.

*< bottleneck node : pdpds04 >*
HDFS_SCAN_NODE (id=0):(Active: 1m, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:2/711.75 MB 1:2/711.66 MB 2:3/1.01 GB 3:3/1.04 GB 5:2/711.76 MB
File Formats: PARQUET/NONE:12
ExecOption: Codegen enabled: 0 out of 12
- AverageHdfsReadThreadConcurrency: 0.06
- HdfsReadThreadConcurrencyCountPercentage=0: 94.26
- HdfsReadThreadConcurrencyCountPercentage=1: 5.74
// max value is always 1, though the number of disks containing relevant
splits are 5
- HdfsReadThreadConcurrencyCountPercentage=2: 0.00
- HdfsReadThreadConcurrencyCountPercentage=3: 0.00
- HdfsReadThreadConcurrencyCountPercentage=4: 0.00
- HdfsReadThreadConcurrencyCountPercentage=5: 0.00
- HdfsReadThreadConcurrencyCountPercentage=6: 0.00
- HdfsReadThreadConcurrencyCountPercentage=7: 0.00
- HdfsReadThreadConcurrencyCountPercentage=8: 0.00
- AverageIoMgrQueueCapcity: 256.00
- AverageIoMgrQueueSize: 0.00
- AverageScannerThreadConcurrency: 1.00 // always less
than 1, but I think it's not I/O bound because the data is already cached
in OS buffer.
- BytesRead: 4.13 GB
- DecompressionTime: 18s860ms
- MemoryUsed: 0.00
- NumColumns: 16
- NumDisksAccessed: 5
- PerReadThreadRawHdfsThroughput: 1.59 GB/sec
- RowsRead: 82.31M (82310038)
- RowsReturned: 5
- RowsReturnedRate: 0
- ScanRangesComplete: 12
- ScannerThreadsInvoluntaryContextSwitches: 79
- ScannerThreadsTotalWallClockTime: 4m37s
- MaterializeTupleTime(*): 40s026ms
- ScannerThreadsSysTime: 30.989ms
- ScannerThreadsUserTime: 59s351ms
- ScannerThreadsVoluntaryContextSwitches: 912
- TotalRawHdfsReadTime(*): 2s595ms
- TotalReadThroughput: 68.84 MB/sec

*< non-bottleneck node>*
HDFS_SCAN_NODE (id=0):(Active: 10s828ms, % non-child: 99.98%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>):
0:2/648.07 MB 1:2/711.74 MB 2:3/1.04 GB 3:3/1.04 GB 4:1/292.44 MB
5:1/355.87 MB
File Formats: PARQUET/NONE:12
ExecOption: Codegen enabled: 0 out of 12
- AverageHdfsReadThreadConcurrency: 0.30
- HdfsReadThreadConcurrencyCountPercentage=0: 91.30
- HdfsReadThreadConcurrencyCountPercentage=1: 0.00
- HdfsReadThreadConcurrencyCountPercentage=2: 4.35
- HdfsReadThreadConcurrencyCountPercentage=3: 0.00
- HdfsReadThreadConcurrencyCountPercentage=4: 0.00
- HdfsReadThreadConcurrencyCountPercentage=5: 4.35
- HdfsReadThreadConcurrencyCountPercentage=6: 0.00
- HdfsReadThreadConcurrencyCountPercentage=7: 0.00
- HdfsReadThreadConcurrencyCountPercentage=8: 0.00
- AverageIoMgrQueueCapcity: 256.00
- AverageIoMgrQueueSize: 7.65
- AverageScannerThreadConcurrency: 7.35
- BytesRead: 4.05 GB
- DecompressionTime: 17s220ms
- MemoryUsed: 0.00
- NumColumns: 16
- NumDisksAccessed: 6
- PerReadThreadRawHdfsThroughput: 1.77 GB/sec
- RowsRead: 80.57M (80572018)
- RowsReturned: 12
- RowsReturnedRate: 1.00 /sec
- ScanRangesComplete: 12
- ScannerThreadsInvoluntaryContextSwitches: 127
- ScannerThreadsTotalWallClockTime: 1m25s
- MaterializeTupleTime(*): 1m5s
- ScannerThreadsSysTime: 73.983ms
- ScannerThreadsUserTime: 1m22s
- ScannerThreadsVoluntaryContextSwitches: 1.12K (1120)
- TotalRawHdfsReadTime(*): 2s292ms
- TotalReadThroughput: 345.37 MB/sec

Thanks,

Jung-Yup