概述

分布式文件系统
适合：一次写入，多次读出，且不支持修改

文件块大小

128M

HDFS的shell操作（重点）

基本语法

hadoop fs 具体命令
或者
hdfs dfs 具体命名

命令大全

Usage: hadoop fs [generic options][-appendToFile <localsrc> ... <dst>] # 追加[-cat [-ignoreCrc] <src> ...]	# 查看[-checksum <src> ...][-chgrp [-R] GROUP PATH...]	# 改组[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] # 改权限[-chown [-R] [OWNER][:[GROUP]] PATH...] # 改所有者[-copyFromLocal [-f] [-p] [-l] [-d] [-t <thread count>] <localsrc> ... <dst>] # 上传[-copyToLocal [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]# 下载[-count [-q] [-h] [-v] [-t [<storage type>]] [-u] [-x] [-e] <path> ...][-cp [-f] [-p | -p[topax]] [-d] <src> ... <dst>]	# 复制[-createSnapshot <snapshotDir> [<snapshotName>]][-deleteSnapshot <snapshotDir> <snapshotName>][-df [-h] [<path> ...]][-du [-s] [-h] [-v] [-x] <path> ...]	# 统计磁盘文件大小[-expunge][-find <path> ... <expression> ...][-get [-f] [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]	# 下载[-getfacl [-R] <path>][-getfattr [-R] {-n name | -d} [-e en] <path>][-getmerge [-nl] [-skip-empty-file] <src> <localdst>][-head <file>][-help [cmd ...]][-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [-e] [<path> ...]]	# 查看列表[-mkdir [-p] <path> ...]	# 创建[-moveFromLocal <localsrc> ... <dst>]	# 剪切到hdfs[-moveToLocal <src> <localdst>]	# 剪切到本地[-mv <src> ... <dst>]	#移动[-put [-f] [-p] [-l] [-d] <localsrc> ... <dst>]	# 上传[-renameSnapshot <snapshotDir> <oldName> <newName>][-rm [-f] [-r|-R] [-skipTrash] [-safely] <src> ...]	# 删除[-rmdir [--ignore-fail-on-non-empty] <dir> ...][-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]][-setfattr {-n name [-v value] | -x name} <path>][-setrep [-R] [-w] <rep> <path> ...]	# 设置副本数[-stat [format] <path> ...][-tail [-f] <file>][-test -[defsz] <path>][-text [-ignoreCrc] <src> ...][-touch [-a] [-m] [-t TIMESTAMP ] [-c] <path> ...][-touchz <path> ...][-truncate [-w] <length> <path> ...][-usage [cmd ...]]Generic options supported are:
-conf <configuration file>        specify an application configuration file
-D <property=value>               define a value for a given property
-fs <file:///|hdfs://namenode:port> specify default filesystem URL to use, overrides 'fs.defaultFS' property from configurations.
-jt <local|resourcemanager:port>  specify a ResourceManager
-files <file1,...>                specify a comma-separated list of files to be copied to the map reduce cluster
-libjars <jar1,...>               specify a comma-separated list of jar files to be included in the classpath
-archives <archive1,...>          specify a comma-separated list of archives to be unarchived on the compute machinesThe general command line syntax is:
command [genericOptions] [commandOptions]

查看详细命令

hadoop fs -help 命令（如cat）

更改hdfs的权限

vi core-site.xml

		<property><name>hadoop.http.staticuser.user</name><value>root</value></property>

HDFS客户端API操作

Windows环境配置

1. 将Windows依赖放到文件夹，
2. 配置环境变量，添加HADOOP_HOME ，编辑Path添加%HADOOP_HOME%/bin
3. 拷贝hadoop.dll和winutils.exe到C:\Windows\System32

创建java项目

配置

编辑pom.xml

	<dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>4.12</version></dependency><dependency><groupId>org.apache.logging.log4j</groupId><artifactId>log4j-slf4j-impl</artifactId><version>2.12.0</version></dependency><dependency><groupId>org.apache.hadoop</groupId><artifactId>hadoop-client</artifactId><version>3.1.3</version></dependency></dependencies>

在src/main/resources中建立log4j2.xml

打印日志到控制台

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN"><Appenders><Console name="Console" target="SYSTEM_OUT"><PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/></Console></Appenders><Loggers><Root level="error"><AppenderRef ref="Console"/></Root></Loggers>
</Configuration>

编写代码

在/src/main/java/cn.zcx.hdfs创建TestHDFS类

public class TestHDFS {// 创建全局变量private FileSystem fs;private Configuration conf;private URI uri;private String user;// 从本地上传文件@Testpublic void testUpload() throws IOException {fs.copyFromLocalFile(false,true,new Path("F:\\Download\\使用前说明.txt"),new Path("/testhdfs"));}/** @Before 方法在@Test方法执行之前执行* */@Beforepublic void init() throws IOException, InterruptedException {uri = URI.create("hdfs://master:8020");conf = new Configuration();user = "root";fs = FileSystem.get(uri,conf,user);}/** @After方法在@Test方法结束后执行* */@Afterpublic void close() throws IOException {fs.close();}@Testpublic void testHDFS() throws IOException, InterruptedException {//1. 创建文件系统对象
/*URI uri = URI.create("hdfs://master:8020");Configuration conf = new Configuration();String user = "root";FileSystem fs = FileSystem.get(uri,conf,user);System.out.println("fs: " + fs);
*/// 2. 创建一个目录boolean b = fs.mkdirs(new Path("/testhdfs"));System.out.println(b);// 3. 关闭fs.close();}
}

参数优先级

xxx-default.xml < xxx-site.xml < IDEA中resource中创建xxx-site.xml < 在代码中通过更改Configuration 参数

文件下载

    @Testpublic void testDownload() throws IOException {fs.copyToLocalFile(false,new Path("/testhdfs/使用前说明.txt"),new Path("F:\\Download\\"),true);}

文件更改移动

//改名or移动（路径改变就可以）@Testpublic void testRename() throws IOException {boolean b = fs.rename(new Path("/testhdfs/使用前说明.txt"),new Path("/testhdfs/zcx.txt"));System.out.println(b);}

查看文件详细信息

    // 查看文件详情@Testpublic void testListFiles() throws IOException {RemoteIterator<LocatedFileStatus> listFiles = fs.listFiles(new Path("/"), true);//迭代操作while (listFiles.hasNext()){LocatedFileStatus fileStatus = listFiles.next();//获取文件详情System.out.println("文件路径："+fileStatus.getPath());System.out.println("文件权限："+fileStatus.getPermission());System.out.println("文件主人："+fileStatus.getOwner());System.out.println("文件组："+fileStatus.getGroup());System.out.println("文件大小："+fileStatus.getLen());System.out.println("文件副本数："+fileStatus.getReplication());System.out.println("文件块位置："+ Arrays.toString(fileStatus.getBlockLocations()));System.out.println("===============================");}}

文件删除

第二参数，true递归删除

    //文件删除@Testpublic void testDelete() throws IOException {boolean b = fs.delete(new Path("/testhdfs/"), true);System.out.println(b);}

NN与2NN工作原理