神马文学网Hadoop NameNode内存空间计算说明
来源：百度文库 编辑：神马文学网 时间：2024/04/28 22:00:36
I've done some estimates on how much space our data structures take on the name-node per block, file and directory.
Brief overview of the data structures:
Directory tree (FSDirectory) is built of inodes.
Each INode points either to an array of blocks if it corresponds to a file or to a TreeMap of children INodes if it is a directory.
[Note: this estimates were made before Dhruba replaced the children TreeMap by ArrayList.]
Each block participates also in at least 2 more data structures.
BlocksMap contains a HashMap of all blocks mapping a Block into a BlockInfo.
DatanodeDescriptor contains a TreeMap of all blocks belonging to this data-node.
A block may or may not be contained also in other data-structures, like
UnderReplicatedBlocks
PendingReplicationBlocks
recentInvalidateSets
excessReplicateMap
Presence of a block in any of these structures is temporary and therefore I do not count them in my estimates.
The estimates can be viewed as lower bounds.
These are some classes that we are looking at here
class INode {
String name;
INode parent;
TreeMap children;
Block blocks[];
short blockReplication;
long modificationTime;
}
class Block {
long blkid;
long len;
}
class BlockInfo {
FSDirectory.INode inode;
DatanodeDescriptor[] nodes;
Block block;
}
The calculations are made for a 64-bit java vm based on that
Reference size = 8 bytes
Object header size = 16 bytes
Array header size = 24 bytes
Commonly used objects:
TreeMap.Entry = 64 bytes. It has 5 reference fields
HashMap.Entry = 48 bytes. It has 3 reference fields
String header = 64 bytes.
The size of a file includes:
Size of an empty file INode: INode.children = null, INode.blocks is a 0-length array, and file name is empty. (152 bytes)
A directory entry of the parent INode that points to this file, which is a TreeMap.Entry. (64 bytes)
file name length times 2, because String represents each name character by 2 bytes.
Reference to the outer FSDirectory class (8 bytes)
The total: 224 + 2 * fileName.length
The size of a directory includes:
Size of an empty directory INode: INode.children is an empty TreeMap, INode.blocks = null, and file name is empty. (192 bytes)
A directory entry of the parent INode that points to this file, which is a TreeMap.Entry. (64 bytes)
file name length times 2
Reference to the outer FSDirectory class (8 bytes)
The total: 264 + 2 * fileName.length
The size of a block includes:
Size of Block. (32 bytes)
Size of BlockInfo. (64 + 8*replication bytes)
Reference to the block from INode.blocks (8 bytes)
HashMap.Entry referencing the block from BlocksMap. (48 bytes)
References to the block from all DatanodeDescriptors it belongs to.
This is a TreeMap.Entry size times block replication. (64 * replication)
The total: 152 + 72 * replication
Typical object sizes:
Taking into account that typical file name is 10-15 bytes and our default replication is 3 we can say that typical sizes are
File size: 250
Directory size: 290
Block size: 368
Object size estimate (bytes) typical size (bytes)
File 224 + 2 * fileName.length 250
Directory 264 + 2 * fileName.length 290
Block 152 + 72 * replication 368
One of our clusters has
Files: 10 600 000
Dirs: 310 000
Blocks: 13 300 000
Total Size (estimate): 7,63 GB
Memory used on the name-node (actual reported by jconsole after gc): 9 GB
This means that other data structures like NetworkTopology, heartbeats, datanodeMap, Host2NodesMap,
leases, sortedLeases, and multiple block replication maps occupy ~1.4 GB, which seems to be pretty high
and need to be investigated as well.
Based on the above estimates blocks should be the main focus of the name-node memory reduction effort.
Space used by a block is 50% larger compared to a file, and there is more blocks than files or directories.
Some ideas on how we can reduce the name-node size without substantially changing the data structures.
INode.children should be an ArrayList instead of a TreeMap. Already done HADOOP-1565. (-48 bytes)
Factor out the INode class into a separate class (-8 bytes)
Create base INode class and derive file inode and directory inode classes from the base.
Directory inodes do not need to contain blocks and replication fields (-16 bytes)
File inodes do not need to contain children field (-8 bytes)
String name should be replaced by a mere byte[]. (-(40 + fileName.length) ~ -50 bytes)
Eliminate the Block object.
We should move Block fields into into BlockInfo and completely get rid of the Block object. (-16 bytes)
Block object is referenced at least 5 times in our structures for each physical block.
The number of references should be reduced to just 2. (-24)
Remove name field from INode. File or directory name is stored in the corresponding directory
entry and does need to be duplicated in the INode (-8 bytes)
Eliminate INode.parent field. INodes are accessed through the directory tree, and the parent can
be remembered in a local variable while browsing the tree. There is no need to persistently store
the parent reference for each object. (-8 bytes)
Need to optimize data-node to block map. Currently each DatanodeDescriptor holds a TreeMap of
blocks contained in the node, and we have an overhead of one TreeMap.Entry per block replica.
I expect we can reorganize datanodeMap in a way that it stores only 1 or 2 references per replica
instead of an entire TreeMap.Entry. (-48 * replication)
Note: In general TreeMaps turned out to be very expensive, we should avoid using them if possible.
Or implement a custom map structure, which would avoid using objects for each map entry.
This is what we will have after all the optimizations
Object size estimate (bytes) typical size (bytes) current typical size (bytes)
File 112 + fileName.length 125 250
Directory 144 + fileName.length 155 290
Block 112 + 24 * replication 184 368
Brief overview of the data structures:
Directory tree (FSDirectory) is built of inodes.
Each INode points either to an array of blocks if it corresponds to a file or to a TreeMap
[Note: this estimates were made before Dhruba replaced the children TreeMap by ArrayList.]
Each block participates also in at least 2 more data structures.
BlocksMap contains a HashMap
DatanodeDescriptor contains a TreeMap
A block may or may not be contained also in other data-structures, like
UnderReplicatedBlocks
PendingReplicationBlocks
recentInvalidateSets
excessReplicateMap
Presence of a block in any of these structures is temporary and therefore I do not count them in my estimates.
The estimates can be viewed as lower bounds.
These are some classes that we are looking at here
class INode {
String name;
INode parent;
TreeMap
Block blocks[];
short blockReplication;
long modificationTime;
}
class Block {
long blkid;
long len;
}
class BlockInfo {
FSDirectory.INode inode;
DatanodeDescriptor[] nodes;
Block block;
}
The calculations are made for a 64-bit java vm based on that
Reference size = 8 bytes
Object header size = 16 bytes
Array header size = 24 bytes
Commonly used objects:
TreeMap.Entry = 64 bytes. It has 5 reference fields
HashMap.Entry = 48 bytes. It has 3 reference fields
String header = 64 bytes.
The size of a file includes:
Size of an empty file INode: INode.children = null, INode.blocks is a 0-length array, and file name is empty. (152 bytes)
A directory entry of the parent INode that points to this file, which is a TreeMap.Entry. (64 bytes)
file name length times 2, because String represents each name character by 2 bytes.
Reference to the outer FSDirectory class (8 bytes)
The total: 224 + 2 * fileName.length
The size of a directory includes:
Size of an empty directory INode: INode.children is an empty TreeMap, INode.blocks = null, and file name is empty. (192 bytes)
A directory entry of the parent INode that points to this file, which is a TreeMap.Entry. (64 bytes)
file name length times 2
Reference to the outer FSDirectory class (8 bytes)
The total: 264 + 2 * fileName.length
The size of a block includes:
Size of Block. (32 bytes)
Size of BlockInfo. (64 + 8*replication bytes)
Reference to the block from INode.blocks (8 bytes)
HashMap.Entry referencing the block from BlocksMap. (48 bytes)
References to the block from all DatanodeDescriptors it belongs to.
This is a TreeMap.Entry size times block replication. (64 * replication)
The total: 152 + 72 * replication
Typical object sizes:
Taking into account that typical file name is 10-15 bytes and our default replication is 3 we can say that typical sizes are
File size: 250
Directory size: 290
Block size: 368
Object size estimate (bytes) typical size (bytes)
File 224 + 2 * fileName.length 250
Directory 264 + 2 * fileName.length 290
Block 152 + 72 * replication 368
One of our clusters has
Files: 10 600 000
Dirs: 310 000
Blocks: 13 300 000
Total Size (estimate): 7,63 GB
Memory used on the name-node (actual reported by jconsole after gc): 9 GB
This means that other data structures like NetworkTopology, heartbeats, datanodeMap, Host2NodesMap,
leases, sortedLeases, and multiple block replication maps occupy ~1.4 GB, which seems to be pretty high
and need to be investigated as well.
Based on the above estimates blocks should be the main focus of the name-node memory reduction effort.
Space used by a block is 50% larger compared to a file, and there is more blocks than files or directories.
Some ideas on how we can reduce the name-node size without substantially changing the data structures.
INode.children should be an ArrayList instead of a TreeMap. Already done HADOOP-1565. (-48 bytes)
Factor out the INode class into a separate class (-8 bytes)
Create base INode class and derive file inode and directory inode classes from the base.
Directory inodes do not need to contain blocks and replication fields (-16 bytes)
File inodes do not need to contain children field (-8 bytes)
String name should be replaced by a mere byte[]. (-(40 + fileName.length) ~ -50 bytes)
Eliminate the Block object.
We should move Block fields into into BlockInfo and completely get rid of the Block object. (-16 bytes)
Block object is referenced at least 5 times in our structures for each physical block.
The number of references should be reduced to just 2. (-24)
Remove name field from INode. File or directory name is stored in the corresponding directory
entry and does need to be duplicated in the INode (-8 bytes)
Eliminate INode.parent field. INodes are accessed through the directory tree, and the parent can
be remembered in a local variable while browsing the tree. There is no need to persistently store
the parent reference for each object. (-8 bytes)
Need to optimize data-node to block map. Currently each DatanodeDescriptor holds a TreeMap of
blocks contained in the node, and we have an overhead of one TreeMap.Entry per block replica.
I expect we can reorganize datanodeMap in a way that it stores only 1 or 2 references per replica
instead of an entire TreeMap.Entry. (-48 * replication)
Note: In general TreeMaps turned out to be very expensive, we should avoid using them if possible.
Or implement a custom map structure, which would avoid using objects for each map entry.
This is what we will have after all the optimizations
Object size estimate (bytes) typical size (bytes) current typical size (bytes)
File 112 + fileName.length 125 250
Directory 144 + fileName.length 155 290
Block 112 + 24 * replication 184 368
Hadoop NameNode内存空间计算说明
分布式计算开源框架Hadoop介绍
使用 Linux 和 Hadoop 进行分布式计算
分布式计算开源框架Hadoop介绍
Linux 和 Hadoop 进行分布式计算
分布式计算开源框架Hadoop介绍
阻抗计算说明
分布式计算开源框架Hadoop入门实践
分布式计算开源框架Hadoop入门实践1
分布式计算开源框架Hadoop入门实践2
分布式计算开源框架Hadoop入门实践111
分布式计算开源框架Hadoop介绍-系统架构
分布式计算开源框架Hadoop入门实践（一）
分布式计算开源框架Hadoop介绍-系统架构
Hadoop-- 海量文件的分布式计算处理方案 - 花钱的年华 - CSDNBlog
分布式计算开源框架Hadoop入门实践 - 文初的专栏 - CSDNBlog
分布式计算开源框架Hadoop入门实践 - 文初的专栏 - CSDNBlog
阻抗计算说明2 -- EDN电子设计技术
2.个人所得税各应税项目的计算说明
个人所得税各应税项目的计算说明
QtEmbedded对内存空间的大小需求
Hadoop、Lucene、Nutch
Hadoop项目简介信息
HADOOP的初次尝试